search

spacemeshos / bug-reports

Issue tracking for community-generated bug reports
Creative Commons Zero v1.0 Universal
0 stars 0 forks source link

Syncing stuck at base layer (575) #1

Closed yaelmhoffman closed 3 years ago

yaelmhoffman commented 3 years ago

Sync stuck indefinitely at layer 575. spacemesh-log-axylion_stuck575.zip spacemesh-log-ibO_stuck575.zip spacemesh-log_DopDD_stuck575.zip spacemesh-log-onursarayli_stuck575.zip

lrettig commented 3 years ago

I figured this one out. There is an underlying bug here, see linked PR for bug fix. However, there's also an exacerbating condition. This is only happening on nodes that somehow shut down without successfully saving mesh data. This is what all of the logs here have in common:

> zgrep "did not find data to recover on disc" *.zip
spacemesh-log-axylion_stuck575.zip:2021-04-11T07:50:48.312+0200 INFO    6e682.meshDb            did not find data to recover on disc    {"node_id": "6e682186d81ec83a4bd8fb2c7031f14d31b8ed9cd2fa9ab8c47eaed4d7a8bafb"}
spacemesh-log-axylion_stuck575.zip:2021-04-11T08:34:00.675+0200 INFO    6e682.meshDb            did not find data to recover on disc    {"node_id": "6e682186d81ec83a4bd8fb2c7031f14d31b8ed9cd2fa9ab8c47eaed4d7a8bafb"}
spacemesh-log_DopDD_stuck575.zip:2021-04-11T12:10:50.168+0200   INFO    7a4a0.meshDb            did not find data to recover on disc    {"node_id": "7a4a02370ba14f823c986e1e3b9551b671c72d3ffcf3333746bfbf20878963b6"}
...
spacemesh-log_DopDD_stuck575.zip:2021-04-11T16:51:00.767+0200   INFO    7a4a0.meshDb            did not find data to recover on disc    {"node_id": "7a4a02370ba14f823c986e1e3b9551b671c72d3ffcf3333746bfbf20878963b6"}
spacemesh-log-ibO_stuck575.zip:2021-04-11T00:02:32.884+0700     INFO    88888.meshDb            did not find data to recover on disc    {"node_id": "88888d9a6e5ba5af628c4f1180e22086a23fa6e78bf03f53d9f51b1fd3320241"}
spacemesh-log-ibO_stuck575.zip:2021-04-11T00:25:33.083+0700     INFO    88888.meshDb            did not find data to recover on disc    {"node_id": "88888d9a6e5ba5af628c4f1180e22086a23fa6e78bf03f53d9f51b1fd3320241"}
spacemesh-log-onursarayli_stuck575.zip:2021-04-10T23:29:56.470Z INFO    04cc1.meshDb            did not find data to recover on disc    {"node_id": "04cc1bde875897bd5c7dea5a815d3320fd2fadf8780042d2c66f19c5fd2fdc99"}
...
spacemesh-log-onursarayli_stuck575.zip:2021-04-11T16:04:04.894Z INFO    04cc1.meshDb            did not find data to recover on disc    {"node_id": "04cc1bde875897bd5c7dea5a815d3320fd2fadf8780042d2c66f19c5fd2fdc99"}

I don't know why this is happening. Either something is suddenly killing the nodes (indeed, I see logs that are suddenly truncated, with no message about shutdown), or else the mesh database files just aren't being saved correctly.

lrettig commented 3 years ago

@yaelmhoffman the good news is that, if users totally delete their data files and rerun from scratch, the same error is pretty unlikely to happen again. I saw a few community members report that a reinstall and resync worked for them.

lrettig commented 3 years ago

Closing as this issue has been patched, merged, and released