Forest should be able to gracefully shutdown in any situation

[elmattic]

Describe the bug

Forest should be able to gracefully shutdown in any situation when receiving SIGINT or when calling its Filecoin.Shutdown endpoint.

This is not the case when forest downloads automatically a snapshot.

There are other similar situations, i.e. when initially fetching the proofs using forest_paramfetch util or downloading actor bundles.

To Reproduce Steps to reproduce the behavior:

1. Clean mainnet db
2. Run ./target/release/forest --chain mainnet
3. See behavior

Log output

Log Output

``` $ ./target/release/forest --chain mainnet 2023-05-02T09:25:16.172948Z INFO forest: Using default mainnet config 2023-05-02T09:25:16.173467Z INFO forest::daemon: Starting Forest daemon, version 0.8.1+git.26d49baa3 2023-05-02T09:25:16.173556Z INFO forest_libp2p::service: Could not decode networking keystore! 2023-05-02T09:25:16.173886Z INFO forest_utils::io: Permissions set to 0600 on File { fd: 9, path: "/Users/guillaume/Library/Application Support/com.ChainSafe.Forest/libp2p/keypair", read: false, write: true } 2023-05-02T09:25:16.174030Z INFO forest::daemon: PeerId: 12D3KooWB1VGPK4neBDNnDaJfFVaghJgTTdFEbWRXaek35j7XoW6 2023-05-02T09:25:16.174051Z WARN forest::daemon: Warning: Keystore encryption disabled! 2023-05-02T09:25:16.174152Z INFO forest::daemon: Admin token: eyJ0eXAiOiJKV1QiLCJhbGciOiJIUzI1NiJ9.eyJBbGxvdyI6WyJyZWFkIiwid3JpdGUiLCJzaWduIiwiYWRtaW4iXSwiZXhwIjoxNjg4MjAzNTE2fQ.Qy_RiBlU4ek5Ca_UMQaRKYF3vAcWryJ0VAaa1XSoWLk 2023-05-02T09:25:16.179325Z INFO forest::daemon: Prometheus server started at 0.0.0.0:6116 2023-05-02T09:25:16.185853Z INFO forest_genesis: Initialized genesis: BlockHeader: Cid(bafy2bzacecnamqgqmifpluoeldx7zzglxcljo6oja4vrmtj7432rphldpdmm2) 2023-05-02T09:25:16.186272Z INFO forest_db::rolling::gc: Running automatic database garbage collection task 2023-05-02T09:25:16.186324Z INFO forest_db::rolling::gc: Listening on database garbage collection events 2023-05-02T09:25:16.186473Z INFO forest::daemon: Using network :: mainnet ✔ Forest needs a snapshot to sync with the network. Would you like to download one now? Required disk space 86.35 GiB. · yes 2023-05-02T09:25:25.544702Z INFO forest_daemon::bundle: Downloading actors bundle... Downloading/Importing snapshot 7.27 MB / 7.27 MB [===========================================================================================================================================================================================================] 100.00 % 109.93 GB/s 2023-05-02T09:25:26.671200Z INFO forest_daemon::bundle: Downloading actors bundle... Downloading/Importing snapshot 7.19 MB / 7.19 MB [===========================================================================================================================================================================================================] 100.00 % 120.19 GB/s 2023-05-02T09:25:28.205009Z INFO forest_utils::db: Buffered write completed: total entries: 18, total size: 7.6MB, took: 0s 2023-05-02T09:25:28.205419Z INFO forest_daemon::bundle: Loaded actors bundle with CID: bafy2bzacecsuyf7mmvrhkx2evng5gnz5canlnz2fdlzu2lvcgptiq2pzuovos 2023-05-02T09:25:28.225170Z INFO forest_utils::db: Buffered write completed: total entries: 18, total size: 7.5MB, took: 0s 2023-05-02T09:25:28.225213Z INFO forest_daemon::bundle: Loaded actors bundle with CID: bafy2bzacecnhaiwcrpyjvzl4uv4q3jzoif26okl3m66q3cijp3dfwlcxwztwo 2023-05-02T09:25:28.226745Z WARN forest_libp2p::behaviour: libp2p Forest version: 0.8.1+git.26d49baa3 2023-05-02T09:25:28.226935Z INFO forest_chain_sync::chain_muxer: Evaluating network head... 2023-05-02T09:25:28.226987Z INFO forest::daemon: JSON-RPC endpoint started at 127.0.0.1:1234 2023-05-02T09:25:28.227423Z INFO forest_rpc: Ready for RPC connections 2023-05-02T09:25:28.569346Z INFO forest_cli_shared::cli::snapshot_fetch: Snapshot url: https://snapshots.mainnet.filops.net/minimal/latest.zst 2023-05-02T09:25:29.856481Z INFO forest_cli_shared::cli::snapshot_fetch: Snapshot url: https://pub-fd31751bcb69400eb39e694385c19457.r2.dev/minimal/2823360_2023_05_02T06_00_00Z.car.zst 2023-05-02T09:25:29.856511Z INFO forest_cli_shared::cli::snapshot_fetch: Snapshot will be downloaded to /Users/guillaume/Library/Application Support/com.ChainSafe.Forest/snapshots/mainnet/filecoin_snapshot_mainnet_2023-05-02_height_2823360.car.zst 05/02 11:25:29 [NOTICE] Downloading 1 item(s) 05/02 11:25:30 [NOTICE] Allocating disk space. Use --file-allocation=none to disable it. See --file-allocation option in man page for more details. [#c9c0f5 0B/45GiB(0%) CN:1 DL:0B] [FileAlloc:#c9c0f5 5.9GiB/45GiB(13%)]^C 05/02 11:25:33 [NOTICE] Shutdown sequence commencing... Press Ctrl-C again for emergency shutdown. 05/02 11:25:33 [NOTICE] Download GID#c9c0f5a59934c7d2 not complete: /Users/guillaume/Library/Application Support/com.ChainSafe.Forest/snapshots/mainnet/filecoin_snapshot_mainnet_2023-05-02_height_2823360.car.zst Download Results: gid |stat|avg speed |path/URI ======+====+===========+======================================================= c9c0f5|INPR| 0B/s|/Users/guillaume/Library/Application Support/com.ChainSafe.Forest/snapshots/mainnet/filecoin_snapshot_mainnet_2023-05-02_height_2823360.car.zst Status Legend: (INPR):download in-progress. aria2 will resume download if the transfer is restarted. If there are any errors, then see the log file. See '-l' option in help/man page for details. 2023-05-02T09:25:33.522303Z WARN forest::daemon: Failed with exit code 7, checkout https://aria2.github.io/manual/en/html/aria2c.html#exit-status 2023-05-02T09:25:33.522313Z INFO forest::daemon: Retry attempt 1 started with delay 60s. ^C ```

Expected behaviour

Forest should shutdown gracefully in any situation.

Screenshots

Environment (please complete the following information):

• OS: all
• Rust version: 1.71.0-nightly
• Branch: 26d49baa32d922102cec97168fc62e151196cb6a

Other information and links

[LesnyRumcajs]

Open question - can we prevent this class of bugs in the future? The SIGINT bug constantly manages to get back in one form or another. https://github.com/ChainSafe/forest/issues?q=is%3Aissue+sigint+is%3Aclosed

[elmattic]

The issues you are showing in your link are more or less all different.

But I think your question is, is there a way to prevent writing new code that would not allow us to shutdown forest gracefully? Start of an answer would be to have dedicate tasks that we spawn for long waiting IO operations. I.e.

services.spawn(load_bundles(epoch, config.clone(), db.clone()));

So in this case even if internal calls to fetch_from_url takes a lot of time, we could still abort the task when SIGINT or Filecoin.Shutdown is triggered.

[lemmih]

Fixed in #2950