Investigate down / hanging nodes

threefoldfoundation / tfchain

Official implementation of the ThreeFold blockchain

https://explorer.threefoldtoken.com

Apache License 2.0

13 stars 7 forks source link

Investigate down / hanging nodes #360

Closed LeeSmet closed 4 years ago

LeeSmet commented 5 years ago

one explorer on main and standard (explorer 2) is down, while one block creator on testnet and explorer on mainnet seem to be hanging. Need to investigate what causes this

LeeSmet commented 5 years ago

Trying to install a new explorer service after wiping the down node resulted in multiple failures:

js client complaining, claiming the operation would block forever
resource busy error related to the macvlan interface for the container
'could not start tfchaind', likely because the service state was not updated

First issue does not reappear after starting the container again, issue 2 and 3 are currently blocking me from setting up a new explorer

LeeSmet commented 5 years ago

After some discussion, old zosv1 nodes will be migrated to plain ubuntu nodes to host these services, while we wait for zos V2 to be completely operational

LeeSmet commented 5 years ago

Node with halted services has been restarted, and seems to run fine for now. So only 1 node is down now, others are up and running

LeeSmet commented 4 years ago

We now have 2 linux nodes running the primary (visible) services, i.e. explorer 1 and 2 (both main and testnet), and faucet. Service on other nodes has been restored as well for now