MinaProtocol / mina

Mina is a cryptocurrency protocol with a constant size blockchain, improving scaling while maintaining decentralization and security.
https://minaprotocol.com
Apache License 2.0
1.98k stars 526 forks source link

Improve graceful shutdown error reporting #15572

Open nholland94 opened 2 months ago

nholland94 commented 2 months ago

When the node encounters an error, that error is bubbled up to the highest level of async, where we then initiate our graceful shutdown logic for the node. This graceful shutdown logic is supposed to log the error, generate a crash report, and shut things down in a more graceful manner. However, we are seeing that it doesn't always do this well: there seems to be some code paths in which the error will not actually get logged sufficiently before the shutdown finishes. We just end up with logs that say graceful shutdown has started with no other information.

Below is an example of what this looks like in the logs (no error or useful information prior to this).

{"timestamp":"2024-04-21 10:57:00.890593Z","level":"Info","source":{"module":"Exit_handlers","location":"File \"src/lib/exit_handlers/exit_handlers.ml\", line 31, characters 4-15"},"message":"Running async shutdown handler: $description","metadata":{"description":"Close transition frontier, if exists","host":"46.4.27.171","peer_id":"12D3KooWBtktgL7kvikZScEAioJpv36DVx6vBMECKbMRWc25rT5R","pid":18,"port":8302}}
{"timestamp":"2024-04-21 10:57:00.890632Z","level":"Info","source":{"module":"Exit_handlers","location":"File \"src/lib/exit_handlers/exit_handlers.ml\", line 31, characters 4-15"},"message":"Running async shutdown handler: $description","metadata":{"description":"Remove daemon lockfile","host":"46.4.27.171","peer_id":"12D3KooWBtktgL7kvikZScEAioJpv36DVx6vBMECKbMRWc25rT5R","pid":18,"port":8302}}
nholland94 commented 2 months ago

Let's fix this in develop for now so that we don't introduce risk to berkeley or compatible so close to the upgrade.