This isn't the first time I've had this error, but I don't know where it could come from.
And it occurs at different times.
The first time, the node had been running for more than 50 days.
I had it again 2 weeks ago and again last night.
The only way I've found to quickly restart the node is to completely delete the directory containing the data.
And so, when that happens, I have to connect to the web interface of my VPS to restart it first because I no longer have access via ssh (I can't even ping the VPS) .
But it's really boring.
Have you ever had the same thing and above all, do you know how to prevent it?
(I use the 1.3.2 binary file that I downloaded, and it's an archive node)
Good morning,
This isn't the first time I've had this error, but I don't know where it could come from. And it occurs at different times. The first time, the node had been running for more than 50 days. I had it again 2 weeks ago and again last night.
The only way I've found to quickly restart the node is to completely delete the directory containing the data.
And so, when that happens, I have to connect to the web interface of my VPS to restart it first because I no longer have access via ssh (I can't even ping the VPS) .
But it's really boring.
Have you ever had the same thing and above all, do you know how to prevent it?
(I use the 1.3.2 binary file that I downloaded, and it's an archive node)
May 12 10:21:24 vmi1691458.contaboserver.net systemd[1]: ternoa-node.service: Consumed 12.273s CPU time. May 12 10:21:24 vmi1691458.contaboserver.net systemd[1]: ternoa-node.service: Scheduled restart job, restart counter is at 61. May 12 10:21:24 vmi1691458.contaboserver.net systemd[1]: Stopped Node Ternoa. May 12 10:21:24 vmi1691458.contaboserver.net systemd[1]: ternoa-node.service: Consumed 12.273s CPU time. May 12 10:21:24 vmi1691458.contaboserver.net systemd[1]: Started Node Ternoa. May 12 10:21:24 vmi1691458.contaboserver.net ternoa[5071]: 2024-05-12 10:21:24 Ternoa Node May 12 10:21:24 vmi1691458.contaboserver.net ternoa[5071]: 2024-05-12 10:21:24 ✌️ version 1.1.1 May 12 10:21:24 vmi1691458.contaboserver.net ternoa[5071]: 2024-05-12 10:21:24 ❤️ by Ternoa, 2020-2024 May 12 10:21:24 vmi1691458.contaboserver.net ternoa[5071]: 2024-05-12 10:21:24 📋 Chain specification: Ternoa Mainnet May 12 10:21:24 vmi1691458.contaboserver.net ternoa[5071]: 2024-05-12 10:21:24 🏷 Node name: NodePainNodeGain May 12 10:21:24 vmi1691458.contaboserver.net ternoa[5071]: 2024-05-12 10:21:24 👤 Role: FULL May 12 10:21:24 vmi1691458.contaboserver.net ternoa[5071]: 2024-05-12 10:21:24 💾 Database: RocksDb at /blocks/chains/mainnet-live/db/full May 12 10:21:24 vmi1691458.contaboserver.net ternoa[5071]: 2024-05-12 10:21:24 ⛓ Native runtime: mainnet-21 (mainnet-0.tx1.au1) May 12 10:21:28 vmi1691458.contaboserver.net ternoa[5071]: ==================== May 12 10:21:28 vmi1691458.contaboserver.net ternoa[5071]: Version: 1.1.1 May 12 10:21:28 vmi1691458.contaboserver.net ternoa[5071]: 0: sp_panic_handler::set::{{closure}} May 12 10:21:28 vmi1691458.contaboserver.net ternoa[5071]: 1: std::panicking::rust_panic_with_hook May 12 10:21:28 vmi1691458.contaboserver.net ternoa[5071]: at /rustc/34a6cae28e7013ff0e640026a8e46f315426829d/library/std/src/panicking.rs:702:17 May 12 10:21:28 vmi1691458.contaboserver.net ternoa[5071]: 2: std::panicking::begin_panic_handler::{{closure}} May 12 10:21:28 vmi1691458.contaboserver.net ternoa[5071]: at /rustc/34a6cae28e7013ff0e640026a8e46f315426829d/library/std/src/panicking.rs:588:13 May 12 10:21:28 vmi1691458.contaboserver.net ternoa[5071]: 3: std::sys_common::backtrace::rust_end_short_backtrace May 12 10:21:28 vmi1691458.contaboserver.net ternoa[5071]: at /rustc/34a6cae28e7013ff0e640026a8e46f315426829d/library/std/src/sys_common/backtrace.rs:138:18 May 12 10:21:28 vmi1691458.contaboserver.net ternoa[5071]: 4: rust_begin_unwind May 12 10:21:28 vmi1691458.contaboserver.net ternoa[5071]: at /rustc/34a6cae28e7013ff0e640026a8e46f315426829d/library/std/src/panicking.rs:584:5 May 12 10:21:28 vmi1691458.contaboserver.net ternoa[5071]: 5: core::panicking::panic_fmt May 12 10:21:28 vmi1691458.contaboserver.net ternoa[5071]: at /rustc/34a6cae28e7013ff0e640026a8e46f315426829d/library/core/src/panicking.rs:142:14 May 12 10:21:28 vmi1691458.contaboserver.net ternoa[5071]: 6: core::result::unwrap_failed May 12 10:21:28 vmi1691458.contaboserver.net ternoa[5071]: at /rustc/34a6cae28e7013ff0e640026a8e46f315426829d/library/core/src/result.rs:1814:5 May 12 10:21:28 vmi1691458.contaboserver.net ternoa[5071]: 7: <sp_state_machine::ext::Ext<H,B> as sp_externalities::Externalities>::storage May 12 10:21:28 vmi1691458.contaboserver.net ternoa[5071]: 8: <&mut dyn sp_externalities::Externalities as sp_io::storage::Storage>::get_version_1 May 12 10:21:28 vmi1691458.contaboserver.net ternoa[5071]: 9: sp_externalities::scope_limited::with_externalities May 12 10:21:28 vmi1691458.contaboserver.net ternoa[5071]: 10: tracing::span::Span::in_scope May 12 10:21:28 vmi1691458.contaboserver.net ternoa[5071]: 11: sp_io::storage::get_version_1 May 12 10:21:28 vmi1691458.contaboserver.net ternoa[5071]: 12: sp_io::storage::ExtStorageGetVersion1::call May 12 10:21:28 vmi1691458.contaboserver.net ternoa[5071]: 13:::with_function_context
May 12 10:21:28 vmi1691458.contaboserver.net ternoa[5071]: 14: std::panicking::try
May 12 10:21:28 vmi1691458.contaboserver.net ternoa[5071]: 15: <F as wasmtime::func::IntoFunc<T,(wasmtime::func::Caller,A1),R>>::into_func::wasm_to_host_shim
May 12 10:21:28 vmi1691458.contaboserver.net ternoa[5071]: 16:
May 12 10:21:28 vmi1691458.contaboserver.net ternoa[5071]: 17:
May 12 10:21:28 vmi1691458.contaboserver.net ternoa[5071]: 18:
May 12 10:21:28 vmi1691458.contaboserver.net ternoa[5071]: 19:
May 12 10:21:28 vmi1691458.contaboserver.net ternoa[5071]: 20: wasmtime_runtime::traphandlers::catch_traps::call_closure
May 12 10:21:28 vmi1691458.contaboserver.net ternoa[5071]: 21: wasmtime_setjmp
May 12 10:21:28 vmi1691458.contaboserver.net ternoa[5071]: 22: wasmtime_runtime::traphandlers::::with
May 12 10:21:28 vmi1691458.contaboserver.net ternoa[5071]: 23: wasmtime_runtime::traphandlers::catch_traps
May 12 10:21:28 vmi1691458.contaboserver.net ternoa[5071]: 24: wasmtime::func::invoke_wasm_and_catch_traps
May 12 10:21:28 vmi1691458.contaboserver.net ternoa[5071]: 25: wasmtime::func::typed::TypedFunc<Params,Results>::call
May 12 10:21:28 vmi1691458.contaboserver.net ternoa[5071]: 26: sc_executor_wasmtime::instance_wrapper::EntryPoint::call
May 12 10:21:28 vmi1691458.contaboserver.net ternoa[5071]: 27: sc_executor_wasmtime::runtime::perform_call
May 12 10:21:28 vmi1691458.contaboserver.net ternoa[5071]: 28: ::call_with_allocation_stats
May 12 10:21:28 vmi1691458.contaboserver.net ternoa[5071]: 29: sc_executor_common::wasm_runtime::WasmInstance::call_export
May 12 10:21:28 vmi1691458.contaboserver.net ternoa[5071]: 30: std::panicking::try
May 12 10:21:28 vmi1691458.contaboserver.net ternoa[5071]: 31: std::thread::local::LocalKey::with
May 12 10:21:28 vmi1691458.contaboserver.net ternoa[5071]: 32: sc_executor::native_executor::WasmExecutor::with_instance::{{closure}}
May 12 10:21:28 vmi1691458.contaboserver.net ternoa[5071]: 33: sc_executor::wasm_runtime::RuntimeCache::with_instance
May 12 10:21:28 vmi1691458.contaboserver.net ternoa[5071]: 34: <sc_executor::native_executor::NativeElseWasmExecutor as sp_core::traits::CodeExecutor>::call
May 12 10:21:28 vmi1691458.contaboserver.net ternoa[5071]: 35: sp_state_machine::execution::StateMachine<B,H,Exec>::execute_aux
May 12 10:21:28 vmi1691458.contaboserver.net ternoa[5071]: 36: sp_state_machine::execution::StateMachine<B,H,Exec>::execute_using_consensus_failure_handler
May 12 10:21:28 vmi1691458.contaboserver.net ternoa[5071]: 37: <sc_service::client::call_executor::LocalCallExecutor<Block,B,E> as sc_client_api::call_executor::CallExecutor>::contextual_call
May 12 10:21:28 vmi1691458.contaboserver.net ternoa[5071]: 38: <sc_service::client::client::Client<B,E,Block,RA> as sp_api::CallApiAt>::call_api_at
May 12 10:21:28 vmi1691458.contaboserver.net ternoa[5071]: 39: <mainnet_runtime::RuntimeApiImpl< SR_API_BLOCK,RuntimeApiImplCall> as sp_api::Core<__SR_API_BLOCK__>>::__runtime_api_internal_call_api_at
May 12 10:21:28 vmi1691458.contaboserver.net ternoa[5071]: 40: sp_consensus_babe::BabeApi::configuration
May 12 10:21:28 vmi1691458.contaboserver.net ternoa[5071]: 41: ternoa_service::new_partial
May 12 10:21:28 vmi1691458.contaboserver.net ternoa[5071]: 42: ternoa_service::new_full
May 12 10:21:28 vmi1691458.contaboserver.net ternoa[5071]: 43: tokio::runtime::park::CachedParkThread::block_on
May 12 10:21:28 vmi1691458.contaboserver.net ternoa[5071]: 44: tokio::runtime::scheduler::multi_thread::MultiThread::block_on
May 12 10:21:28 vmi1691458.contaboserver.net ternoa[5071]: 45: tokio::runtime::runtime::Runtime::block_on
May 12 10:21:28 vmi1691458.contaboserver.net ternoa[5071]: 46: sc_cli::runner::Runner::run_node_until_exit
May 12 10:21:28 vmi1691458.contaboserver.net ternoa[5071]: 47: ternoa_cli::command::run
May 12 10:21:28 vmi1691458.contaboserver.net ternoa[5071]: 48: ternoa::main
May 12 10:21:28 vmi1691458.contaboserver.net ternoa[5071]: 49: std::sys_common::backtrace:: rust_begin_short_backtrace
May 12 10:21:28 vmi1691458.contaboserver.net ternoa[5071]: 50: std::rt::lang_start::{{closure}}
May 12 10:21:28 vmi1691458.contaboserver.net ternoa[5071]: 51: core::ops::function::impls::<impl core::ops::function::FnOnce for &F>::call_once
May 12 10:21:28 vmi1691458.contaboserver.net ternoa[5071]: at /rustc/34a6cae28e7013ff0e640026a8e46f315426829d/library/core/src/ops/function.rs:280:13
May 12 10:21:28 vmi1691458.contaboserver.net ternoa[5071]: std::panicking::try::do_call
May 12 10:21:28 vmi1691458.contaboserver.net ternoa[5071]: at /rustc/34a6cae28e7013ff0e640026a8e46f315426829d/library/std/src/panicking.rs:492:40
May 12 10:21:28 vmi1691458.contaboserver.net ternoa[5071]: std::panicking::try
May 12 10:21:28 vmi1691458.contaboserver.net ternoa[5071]: at /rustc/34a6cae28e7013ff0e640026a8e46f315426829d/library/std/src/panicking.rs:456:19
May 12 10:21:28 vmi1691458.contaboserver.net ternoa[5071]: std::panic::catch_unwind
May 12 10:21:28 vmi1691458.contaboserver.net ternoa[5071]: at /rustc/34a6cae28e7013ff0e640026a8e46f315426829d/library/std/src/panic.rs:137:14
May 12 10:21:28 vmi1691458.contaboserver.net ternoa[5071]: std::rt::lang_start_internal::{{closure}}
May 12 10:21:28 vmi1691458.contaboserver.net ternoa[5071]: at /rustc/34a6cae28e7013ff0e640026a8e46f315426829d/library/std/src/rt.rs:128:48
May 12 10:21:28 vmi1691458.contaboserver.net ternoa[5071]: std::panicking::try::do_call
May 12 10:21:28 vmi1691458.contaboserver.net ternoa[5071]: at /rustc/34a6cae28e7013ff0e640026a8e46f315426829d/library/std/src/panicking.rs:492:40
May 12 10:21:28 vmi1691458.contaboserver.net ternoa[5071]: std::panicking::try
May 12 10:21:28 vmi1691458.contaboserver.net ternoa[5071]: at /rustc/34a6cae28e7013ff0e640026a8e46f315426829d/library/std/src/panicking.rs:456:19
May 12 10:21:28 vmi1691458.contaboserver.net ternoa[5071]: std::panic::catch_unwind
May 12 10:21:28 vmi1691458.contaboserver.net ternoa[5071]: at /rustc/34a6cae28e7013ff0e640026a8e46f315426829d/library/std/src/panic.rs:137:14
May 12 10:21:28 vmi1691458.contaboserver.net ternoa[5071]: std::rt::lang_start_internal
May 12 10:21:28 vmi1691458.contaboserver.net ternoa[5071]: at /rustc/34a6cae28e7013ff0e640026a8e46f315426829d/library/std/src/rt.rs:128:20
May 12 10:21:28 vmi1691458.contaboserver.net ternoa[5071]: 52: main
May 12 10:21:28 vmi1691458.contaboserver.net ternoa[5071]: 53:
May 12 10:21:28 vmi1691458.contaboserver.net ternoa[5071]: 54: __libc_start_main
May 12 10:21:28 vmi1691458.contaboserver.net ternoa[5071]: 55: _start
May 12 10:21:28 vmi1691458.contaboserver.net ternoa[5071]: Thread 'main' panicked at 'Externalities not allowed to fail within runtime: "Trie lookup error: Database missing expected key: 0xce06c4bff55e0fef101af3ed89f4bbda49a9b459f3d9f598fd18f28c9498c16a"', /home/peshwar9/.cargo/git/checkouts/substrate-7e08433d4c370a21/a3ed011/primitives/state-machine/src/ext.rs:189
May 12 10:21:28 vmi1691458.contaboserver.net ternoa[5071]: This is a bug. Please report it at:
May 12 10:21:28 vmi1691458.contaboserver.net ternoa[5071]: https://github.com/capsule-corp-ternoa/chain/issues
May 12 10:21:28 vmi1691458.contaboserver.net systemd[1]: ternoa-node.service: Main process exited, code=killed, status=11/SEGV
May 12 10:21:28 vmi1691458.contaboserver.net systemd[1]: ternoa-node.service: Failed with result 'signal'.