Closed yaelmhoffman closed 3 years ago
I figured this one out. There is an underlying bug here, see linked PR for bug fix. However, there's also an exacerbating condition. This is only happening on nodes that somehow shut down without successfully saving mesh data. This is what all of the logs here have in common:
> zgrep "did not find data to recover on disc" *.zip
spacemesh-log-axylion_stuck575.zip:2021-04-11T07:50:48.312+0200 INFO 6e682.meshDb did not find data to recover on disc {"node_id": "6e682186d81ec83a4bd8fb2c7031f14d31b8ed9cd2fa9ab8c47eaed4d7a8bafb"}
spacemesh-log-axylion_stuck575.zip:2021-04-11T08:34:00.675+0200 INFO 6e682.meshDb did not find data to recover on disc {"node_id": "6e682186d81ec83a4bd8fb2c7031f14d31b8ed9cd2fa9ab8c47eaed4d7a8bafb"}
spacemesh-log_DopDD_stuck575.zip:2021-04-11T12:10:50.168+0200 INFO 7a4a0.meshDb did not find data to recover on disc {"node_id": "7a4a02370ba14f823c986e1e3b9551b671c72d3ffcf3333746bfbf20878963b6"}
...
spacemesh-log_DopDD_stuck575.zip:2021-04-11T16:51:00.767+0200 INFO 7a4a0.meshDb did not find data to recover on disc {"node_id": "7a4a02370ba14f823c986e1e3b9551b671c72d3ffcf3333746bfbf20878963b6"}
spacemesh-log-ibO_stuck575.zip:2021-04-11T00:02:32.884+0700 INFO 88888.meshDb did not find data to recover on disc {"node_id": "88888d9a6e5ba5af628c4f1180e22086a23fa6e78bf03f53d9f51b1fd3320241"}
spacemesh-log-ibO_stuck575.zip:2021-04-11T00:25:33.083+0700 INFO 88888.meshDb did not find data to recover on disc {"node_id": "88888d9a6e5ba5af628c4f1180e22086a23fa6e78bf03f53d9f51b1fd3320241"}
spacemesh-log-onursarayli_stuck575.zip:2021-04-10T23:29:56.470Z INFO 04cc1.meshDb did not find data to recover on disc {"node_id": "04cc1bde875897bd5c7dea5a815d3320fd2fadf8780042d2c66f19c5fd2fdc99"}
...
spacemesh-log-onursarayli_stuck575.zip:2021-04-11T16:04:04.894Z INFO 04cc1.meshDb did not find data to recover on disc {"node_id": "04cc1bde875897bd5c7dea5a815d3320fd2fadf8780042d2c66f19c5fd2fdc99"}
I don't know why this is happening. Either something is suddenly killing the nodes (indeed, I see logs that are suddenly truncated, with no message about shutdown), or else the mesh database files just aren't being saved correctly.
@yaelmhoffman the good news is that, if users totally delete their data files and rerun from scratch, the same error is pretty unlikely to happen again. I saw a few community members report that a reinstall and resync worked for them.
Closing as this issue has been patched, merged, and released
Sync stuck indefinitely at layer 575. spacemesh-log-axylion_stuck575.zip spacemesh-log-ibO_stuck575.zip spacemesh-log_DopDD_stuck575.zip spacemesh-log-onursarayli_stuck575.zip