search

ChainSafe / forest

🌲 Rust Filecoin Node Implementation
https://forest.chainsafe.io
Apache License 2.0
637 stars 157 forks source link

Failed to validate a mainnet snapshot #3179

Closed aatifsyed closed 1 year ago

aatifsyed commented 1 year ago

I was using a recent version of forest https://github.com/ChainSafe/forest/tree/48df223b7c9abd4166240d74feaf29b1c25382b9

# ./forest-cli-rustalloc --version
forest-filecoin 0.11.1+git.48df223b7

The following command failed, which surprised me:

# FOREST_KEYSTORE_PHRASE="" ./forest-rustalloc --no-gc --import-snapshot forest_snapshot_mainnet_2023-07-11_height_3026477.car.zst --height=-10000 --halt-after-import

Here's the abridged output

2023-07-12T17:20:38.556413Z  INFO forest_filecoin::daemon::main: Using default mainnet config
2023-07-12T17:20:38.569170Z  INFO forest_filecoin::daemon: Starting Forest daemon, version 0.11.1+git.48df223b7
2023-07-12T17:20:38.569220Z  INFO forest_filecoin::libp2p::service: Networking keystore not found!
2023-07-12T17:20:38.574503Z  INFO forest_filecoin::utils::io: Permissions set to 0600 on File { fd: 9, path: "/root/.local/share/forest/libp2p/keypair", read: false, write: true }
2023-07-12T17:20:38.574571Z  INFO forest_filecoin::daemon: PeerId: 12D3KooWLMG4Y34NzBNo9KDzVasWVK3jFZVXXNmkfHuN2PZdLJgG
2023-07-12T17:20:38.575243Z  WARN forest_filecoin::key_management::keystore: Keystore does not exist, initializing new keystore at "/root/.local/share/forest/keystore"
2023-07-12T17:20:38.730717Z  INFO forest_filecoin::utils::io: Permissions set to 0600 on File { fd: 9, path: "/root/.local/share/forest/keystore", read: false, write: true }
2023-07-12T17:20:38.733140Z  INFO forest_filecoin::daemon: Admin token: eyJ0eXAiOiJKV1QiLCJhbGciOiJIUzI1NiJ9.eyJBbGxvdyI6WyJyZWFkIiwid3JpdGUiLCJzaWduIiwiYWRtaW4iXSwiZXhwIjoxNjk0MzY2NDM4fQ.DebNEEXOCXrv1WTQYQssd1-vSSPoOs1aouxNHiPpX7I
2023-07-12T17:20:38.742993Z  INFO forest_filecoin::daemon: Prometheus server started at 0.0.0.0:6116
2023-07-12T17:20:38.762984Z  INFO forest_filecoin::genesis: Initialized genesis: BlockHeader: Cid(bafy2bzacecnamqgqmifpluoeldx7zzglxcljo6oja4vrmtj7432rphldpdmm2)
2023-07-12T17:20:38.767822Z  INFO forest_filecoin::db::rolling::gc: Listening on database garbage collection events
2023-07-12T17:20:38.769088Z  INFO forest_filecoin::daemon: Using network :: mainnet
2023-07-12T17:20:38.770510Z  INFO forest_filecoin::daemon::bundle: Downloading actors bundle...
2023-07-12T17:20:38.770535Z  INFO forest_filecoin::utils::net: Downloading file: https://github.com/filecoin-project/builtin-actors/releases/download/v9.0.3/builtin-actors-mainnet.car
2023-07-12T17:20:39.382334Z  INFO forest_filecoin::daemon::bundle: Downloading actors bundle...
2023-07-12T17:20:39.382371Z  INFO forest_filecoin::utils::net: Downloading file: https://github.com/filecoin-project/builtin-actors/releases/download/v10.0.0/builtin-actors-mainnet.car
2023-07-12T17:20:40.138696Z  INFO forest_filecoin::daemon::bundle: Downloading actors bundle...
2023-07-12T17:20:40.138738Z  INFO forest_filecoin::utils::net: Downloading file: https://github.com/filecoin-project/builtin-actors/releases/download/v11.0.0/builtin-actors-mainnet.car
2023-07-12T17:20:40.728769Z  INFO forest_filecoin::daemon::bundle: Loaded actors bundle with CID: bafy2bzaceb6j6666h36xnhksu3ww4kxb6e25niayfgkdnifaqi6m6ooc66i6i
2023-07-12T17:20:40.757673Z  INFO forest_filecoin::daemon::bundle: Loaded actors bundle with CID: bafy2bzacecsuyf7mmvrhkx2evng5gnz5canlnz2fdlzu2lvcgptiq2pzuovos
2023-07-12T17:20:40.781722Z  INFO forest_filecoin::daemon::bundle: Loaded actors bundle with CID: bafy2bzacecnhaiwcrpyjvzl4uv4q3jzoif26okl3m66q3cijp3dfwlcxwztwo
2023-07-12T17:20:40.792391Z  WARN forest_filecoin::libp2p::discovery: Could not parse bootstrap addr
2023-07-12T17:20:40.792425Z  WARN forest_filecoin::libp2p::behaviour: libp2p Forest version: 0.11.1+git.48df223b7
2023-07-12T17:20:40.796023Z  INFO forest_filecoin::chain_sync::chain_muxer: Evaluating network head...
2023-07-12T17:20:40.796201Z  INFO forest_filecoin::daemon: JSON-RPC endpoint started at 127.0.0.1:1234
2023-07-12T17:20:40.796208Z  INFO forest_filecoin::genesis: Importing chain from snapshot at: forest_snapshot_mainnet_2023-07-11_height_3026477.car.zst
2023-07-12T17:20:40.796242Z  INFO forest_filecoin::utils::net: Reading file: forest_snapshot_mainnet_2023-07-11_height_3026477.car.zst
2023-07-12T17:20:40.797923Z  INFO forest_filecoin::rpc: Ready for RPC connections
2023-07-12T17:20:45.797206Z  INFO forest::progress: Loading 549027840 (elapsed: 5s, eta: 8m 8s)
...
2023-07-12T17:21:15.797900Z  INFO forest::progress: Loading 2699264000 (elapsed: 35s, eta: 11m 7s)
2023-07-12T17:21:19.206373Z  INFO parity-db: Started reindex for i00-16
2023-07-12T17:21:20.797933Z  INFO forest::progress: Loading 3044474880 (elapsed: 40s, eta: 11m 12s)
2023-07-12T17:21:25.762347Z  INFO parity-db: Completed reindex i00-16 into i00-17
2023-07-12T17:21:25.798018Z  INFO forest::progress: Loading 3316563968 (elapsed: 45s, eta: 11m 30s)
...
2023-07-12T17:56:07.644014Z  INFO forest::progress: Loading 54195503104 (elapsed: 35m 26s, eta: 0s)
2023-07-12T17:56:07.972288Z  INFO forest_filecoin::genesis: Loaded 57544880 records from .car file in 2127s
2023-07-12T17:57:05.792468Z  INFO tipset_by_height{height=0 prev=true}: forest_filecoin::chain::store::index: Resolving genesis using checkpoint tipset at height: 2986100
2023-07-12T17:57:05.793442Z  INFO forest_filecoin::genesis: Accepting [Cid(bafy2bzacebvfgqsl7dbqi4vwaiygecfcz6fwggzkjnavic353edw7njmhwu3g), Cid(bafy2bzacedbizmalbip2zi7k5wer56avlo6a7z6obxlh4wxhullzjyvglwjcy)] as new head.
2023-07-12T17:57:05.795183Z  INFO forest_filecoin::daemon: Imported snapshot in: 2184s
2023-07-12T17:57:05.803306Z  INFO forest_filecoin::utils::proofs_api::paramfetch: Fetching param file "/root/.local/share/forest/filecoin-proof-parameters/v28-fil-inner-product-v1.srs" from https://proofs.filecoin.io/ipfs/
...
2023-07-12T17:57:05.814663Z  INFO forest_filecoin::utils::proofs_api::paramfetch: Fetching param file "/root/.local/share/forest/filecoin-proof-parameters/v28-stacked-proof-of-replication-merkletree-poseidon_hasher-8-0-0-sha256_hasher-6babf46ce344ae495d558e7770a585b2382d54f225af8ed0397b8be7c3fcd472.vk" from https://proofs.filecoin.io/ipfs/
2023-07-12T17:57:08.457286Z  INFO forest_filecoin::state_manager: compute parent state height=3026476
...
2023-07-12T17:58:36.273828Z  INFO forest_filecoin::state_manager: compute parent state height=3026437
2023-07-12T17:58:36.841726Z  WARN compute_tipset_state_blocking: forest_filecoin::interpreter::vm: Message execution failed with exit code 17
2023-07-12T17:58:37.910299Z  INFO forest_filecoin::state_manager: compute parent state height=3026436
2023-07-12T17:58:38.800845Z  INFO forest_filecoin::state_manager: compute parent state height=3026435
2023-07-12T17:58:38.811549Z  INFO forest_filecoin::state_manager: compute parent state height=3026434
2023-07-12T17:58:38.895619Z  INFO forest_filecoin::state_manager: compute parent state height=3026433
2023-07-12T17:58:41.926802Z  INFO forest_filecoin::state_manager: compute parent state height=3026432
2023-07-12T17:58:43.258325Z  INFO forest_filecoin::state_manager: compute parent state height=3026431
2023-07-12T17:58:43.925562Z  WARN compute_tipset_state_blocking: forest_filecoin::interpreter::vm: Message execution failed with exit code 17
2023-07-12T17:58:44.325606Z  INFO forest_filecoin::state_manager: compute parent state height=3026430
...
2023-07-12T18:10:45.967731Z  INFO forest_filecoin::state_manager: compute parent state height=3025262
2023-07-12T18:10:46.673761Z ERROR compute_tipset_state_blocking: forest_filecoin::interpreter::fvm3: database lookup error: Can't create a valid state tree from the given root. This error may indicate unsupported version.
2023-07-12T18:10:47.377885Z ERROR compute_tipset_state_blocking: forest_filecoin::interpreter::fvm3: database lookup error: Can't create a valid state tree from the given root. This error may indicate unsupported version.
2023-07-12T18:10:48.844266Z ERROR forest_filecoin::chain_sync::chain_muxer: Evaluating the network head failed, retrying. Error = P2PEventStreamReceive("receiving on a closed channel")
2023-07-12T18:10:48.849334Z  INFO forest_filecoin::daemon::main: Shutting down tokio...
2023-07-12T18:10:49.350522Z  INFO forest_filecoin::daemon::main: Forest finish shutdown
Error: couldn't compute tipset state

Caused by:
    encountered a database lookup error

This is a clean checkout, and a clean forest, with the exception of being compiled as follows: cargo build --release --no-default-features --features rustalloc

aatifsyed commented 1 year ago

here's the full log file

aatifsyed commented 1 year ago

Validation is expected to fail when you reach the tail of the snapshot --height=-1100 is as far as you can go