search

ChainSafe / gossamer

🕸️ Go Implementation of the Polkadot Host
https://chainsafe.github.io/gossamer
GNU Lesser General Public License v3.0
427 stars 110 forks source link

bug(Westend): `Storage root must match that calculated` at #17274469 #4064

Open EclesioMeloJunior opened 6 days ago

EclesioMeloJunior commented 6 days ago

Describe the bug

Current cloudwatch log


June 26, 2024 at 03:04 (UTC-4:00) | 2024-06-26T07:04:26Z ERROR requesting max blocks from best block header: while handling workers results: while handling ready block: processing block data with header and body: handling block: failed to execute block 17274469: running runtime function: wasm error: unreachable | 3b9d053b1efe4e07b7f1162fb574a304 | westend-task
-- | -- | -- | --
June 26, 2024 at 03:04 (UTC-4:00) | wasm stack trace: | 3b9d053b1efe4e07b7f1162fb574a304 | westend-task
June 26, 2024 at 03:04 (UTC-4:00) | .rust_begin_unwind(i32) | 3b9d053b1efe4e07b7f1162fb574a304 | westend-task
June 26, 2024 at 03:04 (UTC-4:00) | ._ZN4core9panicking9panic_fmt17he4489d678d6570d5E(i32,i32) | 3b9d053b1efe4e07b7f1162fb574a304 | westend-task
June 26, 2024 at 03:04 (UTC-4:00) | ._ZN15frame_executive104Executive$LT$System$C$Block$C$Context$C$UnsignedValidator$C$AllPalletsWithSystem$C$COnRuntimeUpgrade$GT$13execute_block17h04bd9202b85e1554E(i32) | 3b9d053b1efe4e07b7f1162fb574a304 | westend-task
June 26, 2024 at 03:04 (UTC-4:00) | .Core_execute_block(i32,i32) i64 pkg=sync | 3b9d053b1efe4e07b7f1162fb574a304 | westend-task
June 26, 2024 at 03:04 (UTC-4:00) | 2024-06-26T07:04:26Z CRITICAL target=runtime message=panicked at 'Storage root must match that calculated.', /home/builder/cargo/git/checkouts/substrate-7e08433d4c370a21/7c195bf/frame/executive/src/lib.rs:602:9 ext_logging_log_version_1 pkg=runtime module=wazero

June 26, 2024 at 03:04 (UTC-4:00)
2024-06-26T07:04:26Z ERROR requesting max blocks from best block header: while handling workers results: while handling ready block: processing block data with header and body: handling block: failed to execute block 17274469: running runtime function: wasm error: unreachable
[3b9d053b1efe4e07b7f1162fb574a304](https://us-east-2.console.aws.amazon.com/ecs/v2/clusters/gossamer-stg/services/gossamer-westend-service/tasks/3b9d053b1efe4e07b7f1162fb574a304?region=us-east-2)
westend-task
June 26, 2024 at 03:04 (UTC-4:00)
wasm stack trace:
[3b9d053b1efe4e07b7f1162fb574a304](https://us-east-2.console.aws.amazon.com/ecs/v2/clusters/gossamer-stg/services/gossamer-westend-service/tasks/3b9d053b1efe4e07b7f1162fb574a304?region=us-east-2)
westend-task
June 26, 2024 at 03:04 (UTC-4:00)
.rust_begin_unwind(i32)
[3b9d053b1efe4e07b7f1162fb574a304](https://us-east-2.console.aws.amazon.com/ecs/v2/clusters/gossamer-stg/services/gossamer-westend-service/tasks/3b9d053b1efe4e07b7f1162fb574a304?region=us-east-2)
westend-task
June 26, 2024 at 03:04 (UTC-4:00)
._ZN4core9panicking9panic_fmt17he4489d678d6570d5E(i32,i32)
[3b9d053b1efe4e07b7f1162fb574a304](https://us-east-2.console.aws.amazon.com/ecs/v2/clusters/gossamer-stg/services/gossamer-westend-service/tasks/3b9d053b1efe4e07b7f1162fb574a304?region=us-east-2)
westend-task
June 26, 2024 at 03:04 (UTC-4:00)
._ZN15frame_executive104Executive$LT$System$C$Block$C$Context$C$UnsignedValidator$C$AllPalletsWithSystem$C$COnRuntimeUpgrade$GT$13execute_block17h04bd9202b85e1554E(i32)
[3b9d053b1efe4e07b7f1162fb574a304](https://us-east-2.console.aws.amazon.com/ecs/v2/clusters/gossamer-stg/services/gossamer-westend-service/tasks/3b9d053b1efe4e07b7f1162fb574a304?region=us-east-2)
westend-task
June 26, 2024 at 03:04 (UTC-4:00)
.Core_execute_block(i32,i32) i64 pkg=sync
[3b9d053b1efe4e07b7f1162fb574a304](https://us-east-2.console.aws.amazon.com/ecs/v2/clusters/gossamer-stg/services/gossamer-westend-service/tasks/3b9d053b1efe4e07b7f1162fb574a304?region=us-east-2)
westend-task
June 26, 2024 at 03:04 (UTC-4:00)
2024-06-26T07:04:26Z CRITICAL target=runtime message=panicked at 'Storage root must match that calculated.', /home/builder/cargo/git/checkouts/substrate-7e08433d4c370a21/7c195bf/frame/executive/src/lib.rs:602:9 ext_logging_log_version_1 pkg=runtime module=wazero

The problem happens at block https://westend.subscan.io/block/17274469 which contains a failed extrinsic call utility (batch_all). This call executes two others:

The failure happens in the first call of the batch (Recovery::remove_recovery) that fails with Failed(StillActive) and this is the guard present in the actual Recovery pallet, here: https://github.com/paritytech/substrate/blob/033d4e86cc7eff0066cd376b9375f815761d653c/frame/recovery/src/lib.rs#L673

One hypothesis is that Gossamer is returning none when it is expected a value, and by returning none the guard is not activated which ends up changing the state causing a mismatch in the storage root.