Open Lederstrumpf opened 4 days ago
kamilsa
commented
4 days ago @Lederstrumpf
Thanks for reporting! Seems like the issue is related to KAGOME not being able to find runtime for some parachain. Until we figured out why is that the case, can you check if https://github.com/qdrvm/kagome/pull/2281 helps?
Lederstrumpf
commented
4 days ago @Lederstrumpf
Thanks for reporting! Seems like the issue is related to KAGOME not being able to find runtime for some parachain. Until we figured out why is that the case, can you check if #2281 helps?
Thanks for the swift attention to the issue!
Until we figured out why is that the case, can you check if #2281 helps?
Unfortunately, get the same error on #2281 after rebuilding db (from #2271 - #2278 wasn't built on it).
Attached are logs both for validator & non-validator modes. kagomevalidator#2281.log kagome_nonvalidator#2281.log
kamilsa
commented
3 days ago @Lederstrumpf we are investigating the issue. Meanwhile, you may try to sync the node with Warp or Fast sync. For some of our devs problem is not reproducing after they synced to the end of the chain. If it does not help, you may try to Warp sync from genesis
Lederstrumpf
commented
2 days ago @Lederstrumpf we are investigating the issue. Meanwhile, you may try to sync the node with Warp or Fast sync. For some of our devs problem is not reproducing after they synced to the end of the chain. If it does not help, you may try to Warp sync from genesis
Thanks, fwiw, the Full sync from my backup db from 14 days ago (mentioned here: https://github.com/qdrvm/kagome/issues/2276#issuecomment-2485838632) failed today on the same block/error.
Just restarted with a Warp sync from genesis - will update on its result once completed.
Lederstrumpf
commented
2 days ago Just restarted with a Warp sync from genesis - will update on its result once completed.
Unfortunately also fails: strictly on the successor of the warp sync target block, both for validator and non-validator modes:
Nov 21 10:23:10 v1-oslo kagome[993246]: 24.11.21 10:23:10.547760 main_runner Warning Synchronizer Block #25874041 (0x521d…627b) and 104 others have been discarded: N6kagome10blockchain14BlockTreeErrorE(9) the requested block header is not found in block storage
Nov 21 10:23:35 v1-oslo kagome[993246]: 24.11.21 10:23:35.084349 main_runner Warning Synchronizer Block #25874041 (0x521d…627b) and 108 others have been discarded: N6kagome10blockchain14BlockTreeErrorE(9) the requested block header is not found in block storage
Nov 21 10:23:38 v1-oslo kagome[993246]: 24.11.21 10:23:38.752177 main_runner Warning Synchronizer Block #25874041 (0x521d…627b) and 109 others have been discarded: N6kagome10blockchain14BlockTreeErrorE(9) the requested block header is not found in block storageI've attached some logs with the initial sync and restarts of the service: on every node restart (with --sync Warp still specified), the node manages to sync up until the current sync target and then stalls the same.
The last log (kagome_warp_sync_restart_4_validator_#2280_trace.log) is trace level.
kamilsa
commented
2 days ago Sorry, I couldn't find kagome_warp_sync_restart_4_validator_#2280_trace.log log.
Also, btw the issue from kagomevalidator#2281.log kagome_nonvalidator#2281.log
could have been resolved in the master by #2285 . You may try to use master branch to check if it moves forward. (Please note we recently updated the database format, you might require --enable-db-migration flag if you run with existing database for the first time)
Lederstrumpf
commented
2 days ago Sorry, I couldn't find
kagome_warp_sync_restart_4_validator_#2280_trace.loglog.Also, btw the issue from kagomevalidator#2281.log kagome_nonvalidator#2281.log
could have been resolved in the master by #2285 . You may try to use master branch to check if it moves forward. (Please note we recently updated the database format, you might require
--enable-db-migrationflag if you run with existing database for the first time)
Ah, you're right I forgot to attach - sorry about that, here they are: kagome_warp_sync_restartnonvalidator#2280.log kagome_warp_sync_restart_4validator#2280_trace.log kagome_warp_sync_restart_3validator#2280.log kagome_warp_sync_restart_2nonvalidator#2280.log kagome_warp_syncnonvalidator#2280.log
You may try to use master branch to check if it moves forward. (Please note we recently updated the database format, you might require
--enable-db-migrationflag if you run with existing database for the first time)
Will give it a shot - I didn't have #2285 in yet. DB's already migrated though.
Lederstrumpf
commented
2 days ago You may try to use master branch to check if it moves forward. (Please note we recently updated the database format, you might require
--enable-db-migrationflag if you run with existing database for the first time)Will give it a shot - I didn't have #2285 in yet. DB's already migrated though.
Unfortunately still fails for me with #2285 included; even on a fresh Warp sync, see attached logs. It might be that Amforc's "The Spammening" happening atm is a factor.
kagome_warp_sync_fresh_6validator#2280.log kagome_warp_sync_restart_7validator#2280.log
kamilsa
commented
1 day ago I couldn't reproduce the issue exactly. I suspect there might be some undefined behaviour either with 1) wabt instrumentation or 2) wasmedge . To check 1) please try to use tmp/disable-wabt-instrumentation branch.
To check 2) please try to rebuild KAGOME with -DWASM_COMPILER=WAVM flag during cmake configuration and pass --wasm-execution Compiled flag to use Wavm instead of WasmEdge
Bug Summary
Validator crashes instantly when processing block #25830930 (0x145a…36bc) on Kusama
Bug Description
Validator crashes instantly when processing block #25830930 (0x145a…36bc) on Kusama, presumably due to inability to obtain validator list for said block:
Steps to Reproduce
Mode: Validator number of nodes: 1 Command:
kagome --chain kusama -d [...] --validator --listen-addr [...] --public-addr [...] --name [...] --rpc-port [...] --telemetry-url [...] --telemetry-url [...] --node-key-file [...]Effects of the Bug
Instantly crashes, service restarts, hits same error again.
Example error log running with
--validator:If relaunching as non-validator, get following failure:
Expected Behavior
Doesn't crash on said block. Non-validator doesn't crash but stalls on block #25830930's child (see logs below).
System Information
NixOS 24.5 with kernel 6.11.5 Compiler: gcc 13.2.0 CMake: cmake version 3.25.3
Additional Context
No response