Shard LDB reset on an s0 node

[MaxMustermann2]

A node that was moved to shard level db on 30 April 2022 seems to have reset the data to 0. Instead of ~26m blocks, it is at ~3m blocks. Issue appears to have occurred on 8 May 2022 when the first complaining alert was generated. I have found the relevant logs below, but I do not see a reason for this to have occurred. The last few lines indicate that the node has restarted from block 0.

$ grep -B 500 '"number":"0"' zerolog-harmony-2022-05-08T07-48-10.693.log | grep -v hostID
{"level":"warn","signal":"terminated","caller":"/home/sp/harmony/harmony/cmd/harmony/main.go:846","time":"2022-05-08T07:36:17.195037217Z","message":"Gracefully shutting down..."}
{"level":"info","url":"http://0.0.0.0:9500","caller":"/home/sp/harmony/harmony/rpc/rpc.go:114","time":"2022-05-08T07:36:17.195125331Z","message":"HTTP endpoint closed"}
{"level":"info","url":"http://0.0.0.0:9800","caller":"/home/sp/harmony/harmony/rpc/rpc.go:127","time":"2022-05-08T07:36:17.195161793Z","message":"WS endpoint closed"}
{"level":"info","caller":"/home/sp/harmony/harmony/node/node.go:1234","time":"2022-05-08T07:36:17.195170222Z","message":"stopping rosetta"}
{"level":"info","caller":"/home/sp/harmony/harmony/node/node.go:1239","time":"2022-05-08T07:36:17.19518868Z","message":"stopping services"}
{"level":"info","type":"Prometheus","caller":"/home/sp/harmony/harmony/api/service/manager.go:134","time":"2022-05-08T07:36:17.195196844Z","message":"Stopping service"}
{"level":"info","type":"Pprof","caller":"/home/sp/harmony/harmony/api/service/manager.go:134","time":"2022-05-08T07:36:17.197418713Z","message":"Stopping service"}
{"level":"info","type":"SupportExplorer","caller":"/home/sp/harmony/harmony/api/service/manager.go:134","time":"2022-05-08T07:36:17.197472423Z","message":"Stopping service"}
{"level":"info","caller":"/home/sp/harmony/harmony/api/service/explorer/service.go:81","time":"2022-05-08T07:36:17.197487056Z","message":"Shutting down explorer service."}
{"level":"warn","error":"http: Server closed","caller":"/home/sp/harmony/harmony/api/service/explorer/service.go:142","time":"2022-05-08T07:36:17.197597668Z","message":"[Explorer] Server error."}
{"level":"info","caller":"/home/sp/harmony/harmony/api/service/explorer/service.go:85","time":"2022-05-08T07:36:17.198955324Z","message":"Shutting down explorer server successfully"}
{"level":"info","caller":"/home/sp/harmony/harmony/node/node.go:1245","time":"2022-05-08T07:36:17.198989244Z","message":"stopping pub-sub"}
{"level":"info","caller":"/home/sp/harmony/harmony/node/node.go:1248","time":"2022-05-08T07:36:17.199006355Z","message":"stopping host"}
{"level":"info","block":"26249692","hash":"0x98f89385e771656eb4b592625af3ba223c069e02ba40755b827f23f822708968","root":"0x3058d473d675418ece9fb69d3f1c43c4626b3ae73af0a0c98d803e15b843e3d2","caller":"/home/sp/harmony/harmony/core/blockchain.go:903","time":"2022-05-08T07:36:17.204679323Z","message":"Writing cached state to disk"}
{"level":"info","block":"26249691","hash":"0x481d00a1f22507e73fe0f43760bcb1b6a5237d06cba820ab0a77a35ec9c3a9e1","root":"0xd3dfa750d3e2b33a1764794f508140b51d2cdb35de80c82d9016e862a5647204","caller":"/home/sp/harmony/harmony/core/blockchain.go:903","time":"2022-05-08T07:36:17.442451816Z","message":"Writing cached state to disk"}
{"level":"info","block":"26249565","hash":"0xa036cfd21276c6d484b5953f852906680d673c86c442ad581c87cd628bfb98f4","root":"0xa211ad69482cdaee498dfd437d625e28d6994dee21d479f490c5c35ca07b26d1","caller":"/home/sp/harmony/harmony/core/blockchain.go:903","time":"2022-05-08T07:36:17.449277761Z","message":"Writing cached state to disk"}
{"level":"info","caller":"/home/sp/harmony/harmony/core/blockchain.go:917","time":"2022-05-08T07:36:17.481528911Z","message":"Blockchain manager stopped"}
{"level":"info","self":"0.0.0.0:9000","PeerID":"QmWibkR2oaVENDvPhSWeYKDkqEthe1XcokmN2HeU6vSfsZ","PubKey":"000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000","caller":"/home/sp/harmony/harmony/p2p/host.go:202","time":"2022-05-08T07:37:38.762278025Z","message":"libp2p host ready"}
{"level":"info","publicKey":"000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000;","caller":"/home/sp/harmony/harmony/consensus/consensus.go:216","time":"2022-05-08T07:37:38.762365805Z","message":"My Public Key"}
{"level":"warn","caller":"/home/sp/harmony/harmony/cmd/harmony/main.go:659","time":"2022-05-08T07:37:38.762424496Z","message":"Blacklist setup error: open ./.hmy/blacklist.txt: no such file or directory"}
{"level":"info","shardID":0,"caller":"/home/sp/harmony/harmony/internal/shardchain/shardchains.go:89","time":"2022-05-08T07:37:39.17263511Z","message":"initializing a new chain database"}
{"level":"info","shardID":0,"caller":"/home/sp/harmony/harmony/node/node_genesis.go:41","time":"2022-05-08T07:37:39.337875824Z","message":"setting up a brand new chain database"}
{"level":"info","epoch":"0","size":43231,"caller":"/home/sp/harmony/harmony/core/rawdb/accessors_offchain.go:44","time":"2022-05-08T07:37:39.340603622Z","message":"wrote sharding state"}
{"level":"info","number":"0","hash":"0xb4d158b82ac8a653c42b78697ab1cd0c6a0d9a15ab3bc34130f0b719fb174d2a","td":"<nil>","age":"2y10mo3w","caller":"/home/sp/harmony/harmony/core/blockchain.go:386","time":"2022-05-08T07:37:39.342743771Z","message":"Loaded most recent local header"}

[PkayJava]

after checked, below partition was not mounted /mnt/disk02, /mnt/disk03, /mnt/disk04, /mnt/disk05, /mnt/disk06, /mnt/disk07

after I mounted it. now log file keep saying connected and disconnected as below.

I think somehow, its local block is messy now. we need to restore it. I am not sure how to restore it. I need help for that.

[LeoHChen]

@jhd2best can you help on the local mount?