search

paritytech / polkadot-sdk

The Parity Polkadot Blockchain SDK
https://polkadot.com/
1.89k stars 696 forks source link

Number of digest items must match that calculated #4588

Closed ehsan6sha closed 2 months ago

ehsan6sha commented 5 months ago

Is there an existing issue?

Experiencing problems? Have you tried our Stack Exchange first?

Description of bug

I updated the runtime code by following these instructions: https://docs.substrate.io/tutorials/build-a-blockchain/upgrade-a-running-network/ (https://github.com/functionland/sugarfunge-node/commit/b073df012951469782f14e4728161d8f92a44e50)

And now when syncing from 0, the node gives an error on the block that update happened:

2024-05-25 14:21:54 Substrate Node
2024-05-25 14:21:54 ✌️  version 0.1.0-fab981dafd8
2024-05-25 14:21:54 ❤️  by SugarFunge Developers, 2017-2024
2024-05-25 14:21:54 📋 Chain specification: Functionyard
2024-05-25 14:21:54 🏷  Node name: FulaNode
2024-05-25 14:21:54 👤 Role: AUTHORITY
2024-05-25 14:21:54 💾 Database: RocksDb at /uniondrive/chain/chains/functionyard/db/full
2024-05-25 14:21:54 ⛓  Native runtime: sugarfunge-node-101 (sugarfunge-node-1.tx1.au1)
2024-05-25 14:23:14 Using default protocol ID "sup" because none is configured in the chain specs
2024-05-25 14:23:14 🏷  Local node identity is: 12D3KooWPgLZ3Y4zPuvtDf81VfsAtvy6JWo3
egXp6Jg6WC2Nkkrd
2024-05-25 14:23:14 💻 Operating system: linux
2024-05-25 14:23:14 💻 CPU architecture: aarch64
2024-05-25 14:23:14 💻 Target environment: gnu
2024-05-25 14:23:14 💻 Memory: 7810MB
2024-05-25 14:23:14 💻 Kernel: 6.6.20+rpt-rpi-v8
2024-05-25 14:23:14 💻 Linux distribution: Ubuntu 24.04 LTS
2024-05-25 14:23:14 💻 Virtual machine: no
2024-05-25 14:23:14 📦 Highest known block at #531910
2024-05-25 14:23:14 〽️ Prometheus exporter started at 127.0.0.1:9615
2024-05-25 14:23:14 Running JSON-RPC server: addr=0.0.0.0:9945, allowed origins=["*"]
Connection to 127.0.0.1 9945 port [tcp/*] succeeded!
2024-05-25 14:23:14 Accepting new connection 1/100
2024-05-25 14:23:14 Accepting new connection 1/100
[2024-05-25T14:23:14Z INFO  actix_server::builder] starting 4 workers
[2024-05-25T14:23:14Z INFO  actix_server::server] Actix runtime found; starting in Actix runtime
2024-05-25 14:23:15 🔍 Discovered new external address for our node: /ip4/86.242.11.169/tcp/30335/p2p/12D3KooWPgLZ3Y4zPuvtDf81VfsAtvy6JWo3egXp6Jg6WC2Nkkrd
Connection to 127.0.0.1 4000 port [tcp/*] succeeded!
Connection to 127.0.0.1 5001 port [tcp/*] succeeded!
[2024-05-25T14:23:16Z INFO  actix_web::middleware::logger] 127.0.0.1 "POST /health HTTP/1.1" 200 55 "-" "curl/8.5.0" 0.013675
Waiting for Node API to have at least 3 peers...
2024-05-25 14:23:19 panicked at /home/ubuntu/.cargo/git/checkouts/substrate-7e08433d4c370a21/5e49f6e/frame/executive/src/lib.rs:588:9:
assertion `left == right` failed: Number of digest items must match that calculated.
  left: 2
 right: 1
2024-05-25 14:23:19 Block prepare storage changes error: Error at calling runtime api: Execution failed: Execution aborted due to trap: wasm trap: wasm `unreachable` instruction executed
WASM backtrace:
error while executing at wasm backtrace:
    0: 0x219d0 - <unknown>!rust_begin_unwind
    1: 0x2d4d - <unknown>!core::panicking::panic_fmt::h7d22643b0becf577
    2: 0x384b - <unknown>!core::panicking::assert_failed_inner::hf53be49d983c07ca
    3: 0x1034be - <unknown>!core::panicking::assert_failed::h81267043375e33f4
    4: 0x13ee2d - <unknown>!frame_executive::Executive<System,Block,Context,UnsignedValidator,AllPalletsWithSystem,COnRuntimeUpgrade>::execute_block::h92071aebb140cbc0
    5: 0xb1d2e - <unknown>!Core_execute_block
2024-05-25 14:23:19 💔 Error importing block 0x44433e9e7f3ea86dc1911f5f4042ce389ec2443f40927c22ba6d9f11699926db: consensus error: Import failed: Import failed: Error at calling runtime api: Execution failed: Execution aborted due to trap: wasm trap: wasm `unreachable` instruction executed
WASM backtrace:
error while executing at wasm backtrace:
    0: 0x219d0 - <unknown>!rust_begin_unwind
    1: 0x2d4d - <unknown>!core::panicking::panic_fmt::h7d22643b0becf577
    2: 0x384b - <unknown>!core::panicking::assert_failed_inner::hf53be49d983c07ca
    3: 0x1034be - <unknown>!core::panicking::assert_failed::h81267043375e33f4
    4: 0x13ee2d - <unknown>!frame_executive::Executive<System,Block,Context,UnsignedValidator,AllPalletsWithSystem,COnRuntimeUpgrade>::execute_block::h92071aebb140cbc0
    5: 0xb1d2e - <unknown>!Core_execute_block
2024-05-25 14:23:19 ⚙️  Syncing, target=#694235 (1 peers), best: #531910 (0x897f…6fa2), finalized #531910 (0x897f…6fa2), ⬇ 82.0kiB/s ⬆ 1.3kiB/s
2024-05-25 14:23:20 panicked at /home/ubuntu/.cargo/git/checkouts/substrate-7e08433d4c370a21/5e49f6e/frame/executive/src/lib.rs:588:9:
assertion `left == right` failed: Number of digest items must match that calculated.
  left: 2
 right: 1
2024-05-25 14:23:20 Block prepare storage changes error: Error at calling runtime api: Execution failed: Execution aborted due to trap: wasm trap: wasm `unreachable` instruction executed
WASM backtrace:
error while executing at wasm backtrace:
    0: 0x219d0 - <unknown>!rust_begin_unwind
    1: 0x2d4d - <unknown>!core::panicking::panic_fmt::h7d22643b0becf577
    2: 0x384b - <unknown>!core::panicking::assert_failed_inner::hf53be49d983c07ca
    3: 0x1034be - <unknown>!core::panicking::assert_failed::h81267043375e33f4
    4: 0x13ee2d - <unknown>!frame_executive::Executive<System,Block,Context,UnsignedValidator,AllPalletsWithSystem,COnRuntimeUpgrade>::execute_block::h92071aebb140cbc0
    5: 0xb1d2e - <unknown>!Core_execute_block
2024-05-25 14:23:20 💔 Error importing block 0x44433e9e7f3ea86dc1911f5f4042ce389ec2443f40927c22ba6d9f11699926db: consensus error: Import failed: Import failed: Error at calling runtime api: Execution failed: Execution aborted due to trap: wasm trap: wasm `unreachable` instruction executed
WASM backtrace:
error while executing at wasm backtrace:
    0: 0x219d0 - <unknown>!rust_begin_unwind
    1: 0x2d4d - <unknown>!core::panicking::panic_fmt::h7d22643b0becf577
    2: 0x384b - <unknown>!core::panicking::assert_failed_inner::hf53be49d983c07ca
    3: 0x1034be - <unknown>!core::panicking::assert_failed::h81267043375e33f4
    4: 0x13ee2d - <unknown>!frame_executive::Executive<System,Block,Context,UnsignedValidator,AllPalletsWithSystem,COnRuntimeUpgrade>::execute_block::h92071aebb140cbc0
    5: 0xb1d2e - <unknown>!Core_execute_block
[2024-05-25T14:23:21Z INFO  actix_web::middleware::logger] 127.0.0.1 "POST /health HTTP/1.1" 200 55 "-" "curl/8.5.0" 0.001207
Waiting for Node API to have at least 3 peers...
2024-05-25 14:23:24 💤 Idle (0 peers), best: #531910 (0x897f…6fa2), finalized #531910 (0x897f…6fa2), ⬇ 80.4kiB/s ⬆ 0.2kiB/s
[2024-05-25T14:23:26Z INFO  actix_web::middleware::logger] 127.0.0.1 "POST /health HTTP/1.1" 200 55 "-" "curl/8.5.0" 0.003389
Waiting for Node API to have at least 3 peers...
2024-05-25 14:23:29 💤 Idle (0 peers), best: #531910 (0x897f…6fa2), finalized #531910 (0x897f…6fa2), ⬇ 21 B/s ⬆ 35 B/s
2024-05-25 14:23:30 🔍 Discovered new external address for our node: /ip6/2a01:cb00:12a6:6500:2664:12f7:d79d:7bd2/tcp/30335/p2p/12D3KooWPgLZ3Y4zPuvtDf81VfsAtvy6JWo3egXp6Jg6WC2Nkkrd
[2024-05-25T14:23:31Z INFO  actix_web::middleware::logger] 127.0.0.1 "POST /health HTTP/1.1" 200 55 "-" "curl/8.5.0" 0.001223
Waiting for Node API to have at least 3 peers...
2024-05-25 14:23:34 💤 Idle (0 peers), best: #531910 (0x897f…6fa2), finalized #531910 (0x897f…6fa2), ⬇ 1.9kiB/s ⬆ 0.9kiB/s
[2024-05-25T14:23:36Z INFO  actix_web::middleware::logger] 127.0.0.1 "POST /health HTTP/1.1" 200 54 "-" "curl/8.5.0" 0.001243
Waiting for Node API to have at least 3 peers...
2024-05-25 14:23:39 💤 Idle (0 peers), best: #531910 (0x897f…6fa2), finalized #531910 (0x897f…6fa2), ⬇ 0.5kiB/s ⬆ 0.2kiB/s
[2024-05-25T14:23:41Z INFO  actix_web::middleware::logger] 127.0.0.1 "POST /health HTTP/1.1" 200 55 "-" "curl/8.5.0" 0.001217
Waiting for Node API to have at least 3 peers...

Any ideas of how to troubleshoot further? https://functionyard.fula.network/#/explorer/query/0x44433e9e7f3ea86dc1911f5f4042ce389ec2443f40927c22ba6d9f11699926db

Steps to reproduce

  1. upgrade a running node
  2. sync from scratch

However, strangely it happens on some nodes and not all

ehsan6sha commented 5 months ago

I tried this one by setting state cache size to 0 but it did not solve the issue: https://github.com/paritytech/substrate/issues/9697

bkchr commented 5 months ago

Did you increment the spec version?

ehsan6sha commented 5 months ago

Did you increment the spec version?

Yes, both spec_version and EXISTENTIAL_DEPOSIT https://github.com/functionland/sugarfunge-node/blob/main/runtime/src/lib.rs

ehsan6sha commented 5 months ago

any ideas on what is causing it or how to troubleshoot further?