search

NethermindEth / nethermind

A robust execution client for Ethereum node operators.
https://nethermind.io/nethermind-client
GNU General Public License v3.0
1.2k stars 408 forks source link

Unable to restart AuRa node #5394

Closed zguesmi closed 1 year ago

zguesmi commented 1 year ago

Description We have an AuRa fullnode running with the this chainspec. After a restart the node fails with the error

Step InitializeBlockchainAuRaMerge failed after 306ms Nethermind.Trie.TrieException:
Failed to load key f6e43eaf2e6e18131a6cb749de30cc97dd9552aee7a5ccac4b2d197a81ef2a33
from root hash 0xaed44e75318d5f3ecffeb27918beed95e9077f0cb73292d3c481bf72ef353809

Steps to Reproduce I don't know how to reproduce the behavior.

Actual behavior The node fails to run.

Expected behavior The node should start successfully.

Screenshots If applicable, please include screenshots to help illustrate the problem.

Desktop (please complete the following information): Please provide the following information regarding your setup:

Additional context See https://github.com/NethermindEth/nethermind/pull/5005

Logs

2023-03-07 16-24-48.0351|Nethermind starting initialization.
2023-03-07 16-24-48.1038|Loading embedded plugins
2023-03-07 16-24-48.1041|  Found plugin type Nethermind.Consensus.AuRa.AuRaPlugin
2023-03-07 16-24-48.1041|  Found plugin type Nethermind.Consensus.Clique.CliquePlugin
2023-03-07 16-24-48.1041|  Found plugin type Nethermind.Consensus.Ethash.EthashPlugin
2023-03-07 16-24-48.1041|  Found plugin type Nethermind.Consensus.Ethash.NethDevPlugin
2023-03-07 16-24-48.1041|  Found plugin type Nethermind.Hive.HivePlugin
2023-03-07 16-24-48.1041|  Found plugin type Nethermind.UPnP.Plugin.UPnPPlugin
Resolved executing directory as /nethermind.
2023-03-07 16-24-48.1717|Loading 10 assemblies from /nethermind/plugins
2023-03-07 16-24-48.1719|Loading assembly Nethermind.Mev
2023-03-07 16-24-48.1838|  Found plugin type Nethermind.Mev
2023-03-07 16-24-48.1839|Loading assembly Nethermind.EthStats
2023-03-07 16-24-48.1851|  Found plugin type Nethermind.EthStats
2023-03-07 16-24-48.1851|Loading assembly Nethermind.Init
2023-03-07 16-24-48.1885|Loading assembly Nethermind.HealthChecks
2023-03-07 16-24-48.2008|  Found plugin type Nethermind.HealthChecks
2023-03-07 16-24-48.2009|Loading assembly Nethermind.Merge.AuRa
2023-03-07 16-24-48.2018|  Found plugin type Nethermind.Merge.AuRa
2023-03-07 16-24-48.2023|Loading assembly Nethermind.Merge.Plugin
2023-03-07 16-24-48.2066|  Found plugin type Nethermind.Merge.Plugin
2023-03-07 16-24-48.2068|Loading assembly Nethermind.AccountAbstraction
2023-03-07 16-24-48.2308|  Found plugin type Nethermind.AccountAbstraction
2023-03-07 16-24-48.2309|Loading assembly Nethermind.Api
2023-03-07 16-24-48.2324|Loading assembly Nethermind.Consensus.AuRa
2023-03-07 16-24-48.2349|Loading assembly Nethermind.JsonRpc.TraceStore
2023-03-07 16-24-48.2357|  Found plugin type Nethermind.JsonRpc.TraceStore
2023-03-07 16-24-48.3063|Loading standard NLog.config file from /nethermind/NLog.config.
2023-03-07 16-24-48.5922|NLog.config loaded in 285ms.
2023-03-07 16-24-48.5939|Reading config file from /nethermind/configs/bellecour.cfg
2023-03-07 16-24-48.7541|Configuration initialized.
2023-03-07 16:24:48.7934|RocksDb Version: 6.29.3 
2023-03-07 16:24:48.8043|ChainSpecPath matched an embedded resource inside the binary. Loading chainspec from embedded resources instead of file! 
2023-03-07 16:24:49.1303|CPU: Intel(R) Xeon(R) Platinum 8272CL CPU @ 2.60GHz (1C2T), 2593Mhz - 0Mhz 
2023-03-07 16:24:49.1659|Using http://ipv4.icanhazip.com to get external ip 
2023-03-07 16:24:50.0905|Setting up memory allowances 
2023-03-07 16:24:50.0905|  memory hint:        768MB 
2023-03-07 16:24:50.0905|  general memory:     32MB 
2023-03-07 16:24:50.0905|  peers memory:       50MB 
2023-03-07 16:24:50.0905|  Netty memory:       67MB 
2023-03-07 16:24:50.0905|  mempool memory:     112MB 
2023-03-07 16:24:50.0905|  fast blocks memory: 50MB 
2023-03-07 16:24:50.0905|  trie memory:        91MB 
2023-03-07 16:24:50.0957|  DB memory:          364MB 
2023-03-07 16:24:52.0672|Start block loaded from reorg boundary - 21168128 - 21168128 (0x2217f2...21fc67) 
2023-03-07 16:24:52.0733|Loaded LowestInsertedBeaconHeader:  
2023-03-07 16:24:52.0874|Numbers resolved, level = Max(21168128, 21174385), header = Max(21168128, 21174384), body = Max(21168128, 21174385) 
2023-03-07 16:24:52.0874|Detected corrupted block tree data (21174384 < 21174385) (possibly due to an unexpected shutdown). Attempting to fix by moving head backwards. This may fail and you may need to resync the node. 
2023-03-07 16:24:52.0934|Beacon Numbers resolved, level = 0, header = 0, body = 0 
2023-03-07 16:24:52.0956|Block tree initialized, last processed is 21174384 (0x6e06b4...9bd2bf), best queued is 21174384, best known is 21174385, lowest inserted header 1, body 0, lowest sync inserted block number  
2023-03-07 16:24:52.1049|Initializing 13 plugins 
2023-03-07 16:24:52.1049|  Clique by Nethermind 
2023-03-07 16:24:52.1049|  Clique by Nethermind initialized in 0ms 
2023-03-07 16:24:52.1066|  AuRa by Nethermind 
2023-03-07 16:24:52.1066|  AuRa by Nethermind initialized in 0ms 
2023-03-07 16:24:52.1066|  Ethash by Nethermind 
2023-03-07 16:24:52.1066|  Ethash by Nethermind initialized in 1ms 
2023-03-07 16:24:52.1066|  NethDev by Nethermind 
2023-03-07 16:24:52.1066|  NethDev by Nethermind initialized in 0ms 
2023-03-07 16:24:52.1066|  AuRaMerge by Nethermind 
2023-03-07 16:24:52.1183|Client started with TTD: , TTD reached: False, Terminal Block Number , FinalTotalDifficulty:  
2023-03-07 16:24:52.1214|  AuRaMerge by Nethermind initialized in 12ms 
2023-03-07 16:24:52.1214|  Merge by Nethermind 
2023-03-07 16:24:52.1214|  Merge by Nethermind initialized in 0ms 
2023-03-07 16:24:52.1214|  MEV by Nethermind 
2023-03-07 16:24:52.1228|  MEV by Nethermind initialized in 0ms 
2023-03-07 16:24:52.1228|  HealthChecks by Nethermind 
2023-03-07 16:24:52.1228|  HealthChecks by Nethermind initialized in 1ms 
2023-03-07 16:24:52.1228|  Hive by Nethermind 
2023-03-07 16:24:52.1228|  Hive by Nethermind initialized in 0ms 
2023-03-07 16:24:52.1228|  Account Abstraction by Nethermind 
2023-03-07 16:24:52.1279|  Account Abstraction Plugin: User Operation Mining Disabled 
2023-03-07 16:24:52.1279|  Account Abstraction by Nethermind initialized in 2ms 
2023-03-07 16:24:52.1279|  EthStats by Nethermind 
2023-03-07 16:24:52.1279|  EthStats by Nethermind initialized in 0ms 
2023-03-07 16:24:52.1279|  TraceStore by Nethermind 
2023-03-07 16:24:52.1279|  TraceStore by Nethermind initialized in 0ms 
2023-03-07 16:24:52.1279|  UPnP by Nethermind 
2023-03-07 16:24:52.1306|  UPnP by Nethermind initialized in 0ms 
2023-03-07 16:24:52.4394|Step InitializeBlockchainAuRaMerge failed after 306ms Nethermind.Trie.TrieException: Failed to load key f6e43eaf2e6e18131a6cb749de30cc97dd9552aee7a5ccac4b2d197a81ef2a33 from root hash 0xaed44e75318d5f3ecffeb27918beed95e9077f0cb73292d3c481bf72ef353809.
 ---> Nethermind.Trie.TrieException: Node 0xaed44e75318d5f3ecffeb27918beed95e9077f0cb73292d3c481bf72ef353809 is missing from the DB
   at Nethermind.Trie.Pruning.TrieStore.LoadRlp(Keccak keccak, IKeyValueStore keyValueStore) in /src/Nethermind/Nethermind.Trie/Pruning/TrieStore.cs:line 315
   at Nethermind.Trie.Pruning.ReadOnlyTrieStore.LoadRlp(Keccak hash) in /src/Nethermind/Nethermind.Trie/Pruning/ReadOnlyTrieStore.cs:line 28
   at Nethermind.Trie.TrieNode.ResolveNode(ITrieNodeResolver tree) in /src/Nethermind/Nethermind.Trie/TrieNode.cs:line 244
   at Nethermind.Trie.PatriciaTree.Run(Span`1 updatePath, Int32 nibblesCount, Byte[] updateValue, Boolean isUpdate, Boolean ignoreMissingDelete, Keccak startRootHash) in /src/Nethermind/Nethermind.Trie/PatriciaTree.cs:line 397
   at Nethermind.Trie.PatriciaTree.Get(Span`1 rawKey, Keccak rootHash) in /src/Nethermind/Nethermind.Trie/PatriciaTree.cs:line 310
   --- End of inner exception stack trace ---
   at Nethermind.Trie.PatriciaTree.Get(Span`1 rawKey, Keccak rootHash) in /src/Nethermind/Nethermind.Trie/PatriciaTree.cs:line 317
   at Nethermind.State.StateTree.Get(Address address, Keccak rootHash) in /src/Nethermind/Nethermind.State/StateTree.cs:line 38
   at Nethermind.State.StateProvider.GetState(Address address) in /src/Nethermind/Nethermind.State/StateProvider.cs:line 642
   at Nethermind.State.StateProvider.GetAndAddToCache(Address address) in /src/Nethermind/Nethermind.State/StateProvider.cs:line 657
   at Nethermind.State.StateProvider.GetThroughCache(Address address) in /src/Nethermind/Nethermind.State/StateProvider.cs:line 678
   at Nethermind.State.StateProvider.GetCodeHash(Address address) in /src/Nethermind/Nethermind.State/StateProvider.cs:line 294
   at Nethermind.State.IReadOnlyStateProvider.IsContract(Address address) in /src/Nethermind/Nethermind.State/IReadOnlyStateProvider.cs:line 27
   at Nethermind.Evm.TransactionProcessing.ReadOnlyTransactionProcessor.IsContractDeployed(Address address) in /src/Nethermind/Nethermind.Evm/TransactionProcessing/ReadOnlyTransactionProcessor.cs:line 44
   at Nethermind.Blockchain.Contracts.Contract.ConstantContract.CallRaw(CallInfo callInfo, IReadOnlyTransactionProcessor readOnlyTransactionProcessor) in /src/Nethermind/Nethermind.Blockchain/Contracts/Contract.ConstantContract.cs:line 96
   at Nethermind.Blockchain.Contracts.Contract.ConstantContract.Call(CallInfo callInfo) in /src/Nethermind/Nethermind.Blockchain/Contracts/Contract.ConstantContract.cs:line 89
   at Nethermind.Blockchain.Contracts.Contract.IConstantContract.Call[T](CallInfo callInfo) in /src/Nethermind/Nethermind.Blockchain/Contracts/Contract.ConstantContract.cs:line 28
   at Nethermind.Blockchain.Contracts.Contract.IConstantContract.Call[T](BlockHeader parentHeader, String functionName, Address sender, Object[] arguments) in /src/Nethermind/Nethermind.Blockchain/Contracts/Contract.ConstantContract.cs:line 37
   at Nethermind.Consensus.AuRa.Contracts.ValidatorContract.GetValidators(BlockHeader parentHeader) in /src/Nethermind/Nethermind.Consensus.AuRa/Contracts/ValidatorContract.cs:line 92
   at Nethermind.Consensus.AuRa.Validators.ContractBasedValidator.LoadValidatorsFromContract(BlockHeader parentHeader) in /src/Nethermind/Nethermind.Consensus.AuRa/Validators/ContractBasedValidator.cs:line 233
   at Nethermind.Consensus.AuRa.Validators.ContractBasedValidator.SetFinalizationManager(IAuRaBlockFinalizationManager finalizationManager, BlockHeader parentHeader) in /src/Nethermind/Nethermind.Consensus.AuRa/Validators/ContractBasedValidator.cs:line 71
   at Nethermind.Consensus.AuRa.Validators.ContractBasedValidator..ctor(IValidatorContract validatorContract, IBlockTree blockTree, IReceiptFinder receiptFinder, IValidatorStore validatorStore, IValidSealerStrategy validSealerStrategy, IAuRaBlockFinalizationManager finalizationManager, BlockHeader parentHeader, ILogManager logManager, Int64 startBlockNumber, Int64 posdaoTransition, Boolean forSealing) in /src/Nethermind/Nethermind.Consensus.AuRa/Validators/ContractBasedValidator.cs:line 58
   at Nethermind.Consensus.AuRa.AuRaValidatorFactory.<CreateValidatorProcessor>g__GetContractBasedValidator|19_2(<>c__DisplayClass19_0&) in /src/Nethermind/Nethermind.Consensus.AuRa/AuRaValidatorFactory.cs:line 91
   at Nethermind.Consensus.AuRa.AuRaValidatorFactory.CreateValidatorProcessor(Validator validator, BlockHeader parentHeader, Nullable`1 startBlock) in /src/Nethermind/Nethermind.Consensus.AuRa/AuRaValidatorFactory.cs:line 115
   at Nethermind.Consensus.AuRa.Validators.MultiValidator.CreateValidator(Int64 finalizedAtBlockNumber, Validator validatorPrototype, BlockHeader parentHeader) in /src/Nethermind/Nethermind.Consensus.AuRa/Validators/MultiValidator.cs:line 209
   at Nethermind.Consensus.AuRa.Validators.MultiValidator.SetCurrentValidator(Int64 finalizedAtBlockNumber, Validator validatorPrototype, BlockHeader parentHeader) in /src/Nethermind/Nethermind.Consensus.AuRa/Validators/MultiValidator.cs:line 191
   at Nethermind.Consensus.AuRa.Validators.MultiValidator.SetCurrentValidator(KeyValuePair`2 validatorInfo, BlockHeader parentHeader) in /src/Nethermind/Nethermind.Consensus.AuRa/Validators/MultiValidator.cs:line 183
   at Nethermind.Consensus.AuRa.Validators.MultiValidator.InitCurrentValidator(Int64 blockNumber, BlockHeader parentHeader) in /src/Nethermind/Nethermind.Consensus.AuRa/Validators/MultiValidator.cs:line 62
   at Nethermind.Consensus.AuRa.Validators.MultiValidator.SetFinalizationManager(IAuRaBlockFinalizationManager finalizationManager, BlockHeader parentHeader) in /src/Nethermind/Nethermind.Consensus.AuRa/Validators/MultiValidator.cs:line 173
   at Nethermind.Consensus.AuRa.Validators.MultiValidator..ctor(Validator validator, IAuRaValidatorFactory validatorFactory, IBlockTree blockTree, IValidatorStore validatorStore, IAuRaBlockFinalizationManager finalizationManager, BlockHeader parentHeader, ILogManager logManager, Boolean forSealing) in /src/Nethermind/Nethermind.Consensus.AuRa/Validators/MultiValidator.cs:line 53
   at Nethermind.Consensus.AuRa.AuRaValidatorFactory.CreateValidatorProcessor(Validator validator, BlockHeader parentHeader, Nullable`1 startBlock) in /src/Nethermind/Nethermind.Consensus.AuRa/AuRaValidatorFactory.cs:line 132
   at Nethermind.Consensus.AuRa.InitializationSteps.InitializeBlockchainAuRa.CreateAuRaValidator(IBlockProcessor processor, IReadOnlyTxProcessorSource readOnlyTxProcessorSource) in /src/Nethermind/Nethermind.Consensus.AuRa/InitializationSteps/InitializeBlockchainAuRa.cs:line 128
   at Nethermind.Consensus.AuRa.InitializationSteps.InitializeBlockchainAuRa.CreateBlockProcessor() in /src/Nethermind/Nethermind.Consensus.AuRa/InitializationSteps/InitializeBlockchainAuRa.cs:line 75
   at Nethermind.Init.Steps.InitializeBlockchain.InitBlockchain() in /src/Nethermind/Nethermind.Init/Steps/InitializeBlockchain.cs:line 243
   at Nethermind.Init.Steps.InitializeBlockchain.Execute(CancellationToken _) in /src/Nethermind/Nethermind.Init/Steps/InitializeBlockchain.cs:line 59
   at Nethermind.Init.Steps.EthereumStepsManager.ExecuteStep(IStep step, StepInfo stepInfo, CancellationToken cancellationToken) in /src/Nethermind/Nethermind.Init/Steps/EthereumStepsManager.cs:line 154
2023-03-07 16:24:52.4740|Step StartBlockProcessorAuRa  failed after 0ms Nethermind.Init.Steps.StepDependencyException: BlockchainProcessor
   at Nethermind.Init.Steps.StartBlockProcessor.Execute(CancellationToken _) in /src/Nethermind/Nethermind.Init/Steps/StartBlockProcessor.cs:line 24
   at Nethermind.Init.Steps.EthereumStepsManager.ExecuteStep(IStep step, StepInfo stepInfo, CancellationToken cancellationToken) in /src/Nethermind/Nethermind.Init/Steps/EthereumStepsManager.cs:line 154
2023-03-07 16:24:52.4996|Step InitializeNetwork        failed after 17ms System.ArgumentNullException: Value cannot be null. (Parameter 'BlockProcessingQueue')
   at Nethermind.Merge.Plugin.MergePlugin.InitSynchronization() in /src/Nethermind/Nethermind.Merge.Plugin/MergePlugin.cs:line 331
   at Nethermind.Init.Steps.InitializeNetwork.Initialize(CancellationToken cancellationToken) in /src/Nethermind/Nethermind.Init/Steps/InitializeNetwork.cs:line 129
   at Nethermind.Init.Steps.InitializeNetwork.Execute(CancellationToken cancellationToken) in /src/Nethermind/Nethermind.Init/Steps/InitializeNetwork.cs:line 84
   at Nethermind.Init.Steps.EthereumStepsManager.ExecuteStep(IStep step, StepInfo stepInfo, CancellationToken cancellationToken) in /src/Nethermind/Nethermind.Init/Steps/EthereumStepsManager.cs:line 154
2023-03-07 16:24:52.5065|Step DatabaseMigrations       failed after 4ms Nethermind.Init.Steps.StepDependencyException: Synchronizer
   at Nethermind.Init.Steps.Migrations.BloomMigration.Run() in /src/Nethermind/Nethermind.Init/Steps/Migrations/BloomMigration.cs:line 49
   at Nethermind.Init.Steps.DatabaseMigrations.Execute(CancellationToken cancellationToken) in /src/Nethermind/Nethermind.Init/Steps/DatabaseMigrations.cs:line 24
   at Nethermind.Init.Steps.EthereumStepsManager.ExecuteStep(IStep step, StepInfo stepInfo, CancellationToken cancellationToken) in /src/Nethermind/Nethermind.Init/Steps/EthereumStepsManager.cs:line 154
2023-03-07 16:24:52.5065|Found 1 block tree levels to review for fixes starting from 21174385 
2023-03-07 16:24:52.5160|Reviewed 0 blocks out of 1 
2023-03-07 16:24:52.5254|Loaded LowestInsertedBeaconHeader:  
2023-03-07 16:24:52.5331|Numbers resolved, level = Max(21174384, 21174385), header = Max(21174384, 21174385), body = Max(21174384, 21174385) 
2023-03-07 16:24:52.5331|Beacon Numbers resolved, level = 0, header = 0, body = 0 
2023-03-07 16:24:52.5394|Grafana / Prometheus metrics are disabled in configuration 
2023-03-07 16:24:52.5449|Step InitializeBlockProducer  failed after 1ms System.NullReferenceException: Object reference not set to an instance of an object.
   at Nethermind.Init.Steps.InitializeBlockProducer.Execute(CancellationToken _) in /src/Nethermind/Nethermind.Init/Steps/InitializeBlockProducer.cs:line 27
   at Nethermind.Init.Steps.EthereumStepsManager.ExecuteStep(IStep step, StepInfo stepInfo, CancellationToken cancellationToken) in /src/Nethermind/Nethermind.Init/Steps/EthereumStepsManager.cs:line 154
2023-03-07 16:24:52.5580|Step RegisterRpcModules       failed after 5ms Nethermind.Init.Steps.StepDependencyException: SyncModeSelector
   at Nethermind.Init.Steps.RegisterRpcModules.Execute(CancellationToken cancellationToken) in /src/Nethermind/Nethermind.Init/Steps/RegisterRpcModules.cs:line 65
   at Nethermind.Init.Steps.EthereumStepsManager.ExecuteStep(IStep step, StepInfo stepInfo, CancellationToken cancellationToken) in /src/Nethermind/Nethermind.Init/Steps/EthereumStepsManager.cs:line 154
2023-03-07 16:24:52.5603|Step StartBlockProducer       failed after 1ms System.NullReferenceException: Object reference not set to an instance of an object.
   at Nethermind.Init.Steps.StartBlockProducer.Execute(CancellationToken _) in /src/Nethermind/Nethermind.Init/Steps/StartBlockProducer.cs:line 27
   at Nethermind.Init.Steps.EthereumStepsManager.ExecuteStep(IStep step, StepInfo stepInfo, CancellationToken cancellationToken) in /src/Nethermind/Nethermind.Init/Steps/EthereumStepsManager.cs:line 154
2023-03-07 16:24:52.6836|Step StartRpc                 failed after 120ms System.ArgumentNullException: Value cannot be null. (Parameter 'implementationInstance')
   at System.ThrowHelper.Throw(String paramName)
   at Microsoft.Extensions.DependencyInjection.ServiceCollectionServiceExtensions.AddSingleton[TService](IServiceCollection services, TService implementationInstance)
   at Nethermind.Runner.JsonRpc.Bootstrap.RegisterJsonRpcServices(IServiceCollection services) in /src/Nethermind/Nethermind.Runner/JsonRpc/Bootstrap.cs:line 31
   at Nethermind.Runner.JsonRpc.Startup.ConfigureServices(IServiceCollection services) in /src/Nethermind/Nethermind.Runner/JsonRpc/Startup.cs:line 54
   at System.RuntimeMethodHandle.InvokeMethod(Object target, Void** arguments, Signature sig, Boolean isConstructor)
   at System.Reflection.MethodInvoker.Invoke(Object obj, IntPtr* args, BindingFlags invokeAttr)
   at System.Reflection.RuntimeMethodInfo.Invoke(Object obj, BindingFlags invokeAttr, Binder binder, Object[] parameters, CultureInfo culture)
   at Microsoft.AspNetCore.Hosting.ConfigureServicesBuilder.InvokeCore(Object instance, IServiceCollection services)
   at Microsoft.AspNetCore.Hosting.StartupLoader.ConfigureServicesDelegateBuilder`1.<>c__DisplayClass14_0.<ConfigureServices>g__ConfigureServicesWithContainerConfiguration|0(IServiceCollection services)
   at Microsoft.AspNetCore.Hosting.ConventionBasedStartup.ConfigureServices(IServiceCollection services)
   at Microsoft.AspNetCore.Hosting.WebHost.EnsureApplicationServices()
   at Microsoft.AspNetCore.Hosting.WebHost.Initialize()
   at Microsoft.AspNetCore.Hosting.WebHostBuilder.Build()
   at Nethermind.Runner.Ethereum.JsonRpcRunner.Start(CancellationToken cancellationToken) in /src/Nethermind/Nethermind.Runner/Ethereum/JsonRpcRunner.cs:line 67
   at Nethermind.Runner.Ethereum.Steps.StartRpc.Execute(CancellationToken cancellationToken) in /src/Nethermind/Nethermind.Runner/Ethereum/Steps/StartRpc.cs:line 87
   at Nethermind.Init.Steps.EthereumStepsManager.ExecuteStep(IStep step, StepInfo stepInfo, CancellationToken cancellationToken) in /src/Nethermind/Nethermind.Init/Steps/EthereumStepsManager.cs:line 154
2023-03-07 16:24:52.6869|Error during ethereum runner start Nethermind.Trie.TrieException: Failed to load key f6e43eaf2e6e18131a6cb749de30cc97dd9552aee7a5ccac4b2d197a81ef2a33 from root hash 0xaed44e75318d5f3ecffeb27918beed95e9077f0cb73292d3c481bf72ef353809.
 ---> Nethermind.Trie.TrieException: Node 0xaed44e75318d5f3ecffeb27918beed95e9077f0cb73292d3c481bf72ef353809 is missing from the DB
   at Nethermind.Trie.Pruning.TrieStore.LoadRlp(Keccak keccak, IKeyValueStore keyValueStore) in /src/Nethermind/Nethermind.Trie/Pruning/TrieStore.cs:line 315
   at Nethermind.Trie.Pruning.ReadOnlyTrieStore.LoadRlp(Keccak hash) in /src/Nethermind/Nethermind.Trie/Pruning/ReadOnlyTrieStore.cs:line 28
   at Nethermind.Trie.TrieNode.ResolveNode(ITrieNodeResolver tree) in /src/Nethermind/Nethermind.Trie/TrieNode.cs:line 244
   at Nethermind.Trie.PatriciaTree.Run(Span`1 updatePath, Int32 nibblesCount, Byte[] updateValue, Boolean isUpdate, Boolean ignoreMissingDelete, Keccak startRootHash) in /src/Nethermind/Nethermind.Trie/PatriciaTree.cs:line 397
   at Nethermind.Trie.PatriciaTree.Get(Span`1 rawKey, Keccak rootHash) in /src/Nethermind/Nethermind.Trie/PatriciaTree.cs:line 310
   --- End of inner exception stack trace ---
   at Nethermind.Trie.PatriciaTree.Get(Span`1 rawKey, Keccak rootHash) in /src/Nethermind/Nethermind.Trie/PatriciaTree.cs:line 317
   at Nethermind.State.StateTree.Get(Address address, Keccak rootHash) in /src/Nethermind/Nethermind.State/StateTree.cs:line 38
   at Nethermind.State.StateProvider.GetState(Address address) in /src/Nethermind/Nethermind.State/StateProvider.cs:line 642
   at Nethermind.State.StateProvider.GetAndAddToCache(Address address) in /src/Nethermind/Nethermind.State/StateProvider.cs:line 657
   at Nethermind.State.StateProvider.GetThroughCache(Address address) in /src/Nethermind/Nethermind.State/StateProvider.cs:line 678
   at Nethermind.State.StateProvider.GetCodeHash(Address address) in /src/Nethermind/Nethermind.State/StateProvider.cs:line 294
   at Nethermind.State.IReadOnlyStateProvider.IsContract(Address address) in /src/Nethermind/Nethermind.State/IReadOnlyStateProvider.cs:line 27
   at Nethermind.Evm.TransactionProcessing.ReadOnlyTransactionProcessor.IsContractDeployed(Address address) in /src/Nethermind/Nethermind.Evm/TransactionProcessing/ReadOnlyTransactionProcessor.cs:line 44
   at Nethermind.Blockchain.Contracts.Contract.ConstantContract.CallRaw(CallInfo callInfo, IReadOnlyTransactionProcessor readOnlyTransactionProcessor) in /src/Nethermind/Nethermind.Blockchain/Contracts/Contract.ConstantContract.cs:line 96
   at Nethermind.Blockchain.Contracts.Contract.ConstantContract.Call(CallInfo callInfo) in /src/Nethermind/Nethermind.Blockchain/Contracts/Contract.ConstantContract.cs:line 89
   at Nethermind.Blockchain.Contracts.Contract.IConstantContract.Call[T](CallInfo callInfo) in /src/Nethermind/Nethermind.Blockchain/Contracts/Contract.ConstantContract.cs:line 28
   at Nethermind.Blockchain.Contracts.Contract.IConstantContract.Call[T](BlockHeader parentHeader, String functionName, Address sender, Object[] arguments) in /src/Nethermind/Nethermind.Blockchain/Contracts/Contract.ConstantContract.cs:line 37
   at Nethermind.Consensus.AuRa.Contracts.ValidatorContract.GetValidators(BlockHeader parentHeader) in /src/Nethermind/Nethermind.Consensus.AuRa/Contracts/ValidatorContract.cs:line 92
   at Nethermind.Consensus.AuRa.Validators.ContractBasedValidator.LoadValidatorsFromContract(BlockHeader parentHeader) in /src/Nethermind/Nethermind.Consensus.AuRa/Validators/ContractBasedValidator.cs:line 233
   at Nethermind.Consensus.AuRa.Validators.ContractBasedValidator.SetFinalizationManager(IAuRaBlockFinalizationManager finalizationManager, BlockHeader parentHeader) in /src/Nethermind/Nethermind.Consensus.AuRa/Validators/ContractBasedValidator.cs:line 71
   at Nethermind.Consensus.AuRa.Validators.ContractBasedValidator..ctor(IValidatorContract validatorContract, IBlockTree blockTree, IReceiptFinder receiptFinder, IValidatorStore validatorStore, IValidSealerStrategy validSealerStrategy, IAuRaBlockFinalizationManager finalizationManager, BlockHeader parentHeader, ILogManager logManager, Int64 startBlockNumber, Int64 posdaoTransition, Boolean forSealing) in /src/Nethermind/Nethermind.Consensus.AuRa/Validators/ContractBasedValidator.cs:line 58
   at Nethermind.Consensus.AuRa.AuRaValidatorFactory.<CreateValidatorProcessor>g__GetContractBasedValidator|19_2(<>c__DisplayClass19_0&) in /src/Nethermind/Nethermind.Consensus.AuRa/AuRaValidatorFactory.cs:line 91
   at Nethermind.Consensus.AuRa.AuRaValidatorFactory.CreateValidatorProcessor(Validator validator, BlockHeader parentHeader, Nullable`1 startBlock) in /src/Nethermind/Nethermind.Consensus.AuRa/AuRaValidatorFactory.cs:line 115
   at Nethermind.Consensus.AuRa.Validators.MultiValidator.CreateValidator(Int64 finalizedAtBlockNumber, Validator validatorPrototype, BlockHeader parentHeader) in /src/Nethermind/Nethermind.Consensus.AuRa/Validators/MultiValidator.cs:line 209
   at Nethermind.Consensus.AuRa.Validators.MultiValidator.SetCurrentValidator(Int64 finalizedAtBlockNumber, Validator validatorPrototype, BlockHeader parentHeader) in /src/Nethermind/Nethermind.Consensus.AuRa/Validators/MultiValidator.cs:line 191
   at Nethermind.Consensus.AuRa.Validators.MultiValidator.SetCurrentValidator(KeyValuePair`2 validatorInfo, BlockHeader parentHeader) in /src/Nethermind/Nethermind.Consensus.AuRa/Validators/MultiValidator.cs:line 183
   at Nethermind.Consensus.AuRa.Validators.MultiValidator.InitCurrentValidator(Int64 blockNumber, BlockHeader parentHeader) in /src/Nethermind/Nethermind.Consensus.AuRa/Validators/MultiValidator.cs:line 62
   at Nethermind.Consensus.AuRa.Validators.MultiValidator.SetFinalizationManager(IAuRaBlockFinalizationManager finalizationManager, BlockHeader parentHeader) in /src/Nethermind/Nethermind.Consensus.AuRa/Validators/MultiValidator.cs:line 173
   at Nethermind.Consensus.AuRa.Validators.MultiValidator..ctor(Validator validator, IAuRaValidatorFactory validatorFactory, IBlockTree blockTree, IValidatorStore validatorStore, IAuRaBlockFinalizationManager finalizationManager, BlockHeader parentHeader, ILogManager logManager, Boolean forSealing) in /src/Nethermind/Nethermind.Consensus.AuRa/Validators/MultiValidator.cs:line 53
   at Nethermind.Consensus.AuRa.AuRaValidatorFactory.CreateValidatorProcessor(Validator validator, BlockHeader parentHeader, Nullable`1 startBlock) in /src/Nethermind/Nethermind.Consensus.AuRa/AuRaValidatorFactory.cs:line 132
   at Nethermind.Consensus.AuRa.InitializationSteps.InitializeBlockchainAuRa.CreateAuRaValidator(IBlockProcessor processor, IReadOnlyTxProcessorSource readOnlyTxProcessorSource) in /src/Nethermind/Nethermind.Consensus.AuRa/InitializationSteps/InitializeBlockchainAuRa.cs:line 128
   at Nethermind.Consensus.AuRa.InitializationSteps.InitializeBlockchainAuRa.CreateBlockProcessor() in /src/Nethermind/Nethermind.Consensus.AuRa/InitializationSteps/InitializeBlockchainAuRa.cs:line 75
   at Nethermind.Init.Steps.InitializeBlockchain.InitBlockchain() in /src/Nethermind/Nethermind.Init/Steps/InitializeBlockchain.cs:line 243
   at Nethermind.Init.Steps.InitializeBlockchain.Execute(CancellationToken _) in /src/Nethermind/Nethermind.Init/Steps/InitializeBlockchain.cs:line 59
   at Nethermind.Init.Steps.EthereumStepsManager.ExecuteStep(IStep step, StepInfo stepInfo, CancellationToken cancellationToken) in /src/Nethermind/Nethermind.Init/Steps/EthereumStepsManager.cs:line 154
   at Nethermind.Init.Steps.EthereumStepsManager.InitializeAll(CancellationToken cancellationToken) in /src/Nethermind/Nethermind.Init/Steps/EthereumStepsManager.cs:line 99
   at Nethermind.Runner.Ethereum.EthereumRunner.Start(CancellationToken cancellationToken) in /src/Nethermind/Nethermind.Runner/Ethereum/EthereumRunner.cs:line 36
   at Nethermind.Runner.Program.<>c__DisplayClass10_0.<<Run>b__1>d.MoveNext() in /src/Nethermind/Nethermind.Runner/Program.cs:line 194
2023-03-07 16:24:52.6869|Closing, please wait until all functions are stopped properly... 
2023-03-07 16:24:52.6927|Stopping session monitor... 
2023-03-07 16:24:52.6927|Stopping session sync mode selector... 
2023-03-07 16:24:52.6927|Stopping discovery app... 
2023-03-07 16:24:52.6927|Stopping block producer... 
2023-03-07 16:24:52.6927|Stopping sync peer pool... 
2023-03-07 16:24:52.7094|Stopping peer pool... 
2023-03-07 16:24:52.7094|Stopping peer manager... 
2023-03-07 16:24:52.7094|Stopping synchronizer... 
2023-03-07 16:24:52.7108|Stopping blockchain processor... 
2023-03-07 16:24:52.7108|Stopping rlpx peer... 
2023-03-07 16:24:52.7108|Disposing plugin Clique... 
2023-03-07 16:24:52.7108|Disposing plugin AuRa... 
2023-03-07 16:24:52.7108|Disposing plugin Ethash... 
2023-03-07 16:24:52.7108|Disposing plugin NethDev... 
2023-03-07 16:24:52.7108|Disposing plugin AuRaMerge... 
2023-03-07 16:24:52.7108|Disposing plugin Merge... 
2023-03-07 16:24:52.7145|Disposing plugin MEV... 
2023-03-07 16:24:52.7145|Disposing plugin HealthChecks... 
2023-03-07 16:24:52.7145|Disposing plugin Hive... 
2023-03-07 16:24:52.7145|Disposing plugin Account Abstraction... 
2023-03-07 16:24:52.7145|Disposing plugin EthStats... 
2023-03-07 16:24:52.7145|Disposing plugin TraceStore... 
2023-03-07 16:24:52.7145|Disposing plugin UPnP... 
2023-03-07 16:24:52.7145|Disposing Nethermind.Synchronization.Peers.SyncPeerPool... 
2023-03-07 16:24:52.7145|Disposing Nethermind.Blockchain.ReceiptCanonicalityMonitor... 
2023-03-07 16:24:52.7190|Disposing Nethermind.Trie.Pruning.TrieStore... 
2023-03-07 16:24:52.7190|Disposing Nethermind.Blockchain.TrieStoreBoundaryWatcher... 
2023-03-07 16:24:52.7190|Disposing Nethermind.Blockchain.FullPruning.PruningTriggerPersistenceStrategy... 
2023-03-07 16:24:52.7190|Disposing Nethermind.Merge.Plugin.InvalidChainTracker.InvalidChainTracker... 
2023-03-07 16:24:52.7224|Disposing Nethermind.Db.Blooms.BloomStorage... 
2023-03-07 16:24:52.7224|Disposing Nethermind.Stats.NodeStatsManager... 
2023-03-07 16:24:52.7224|Disposing Nethermind.Crypto.CryptoRandom... 
2023-03-07 16:24:52.7224|Closing DBs... 
2023-03-07 16:24:52.7224|Disposing DB BlockInfos 
2023-03-07 16:24:52.8058|Disposing DB Receipts 
2023-03-07 16:24:52.8155|Disposing DB Bloom 
2023-03-07 16:24:52.8187|Disposing DB State0 
2023-03-07 16:24:52.8267|Disposing DB Headers 
2023-03-07 16:24:52.8311|Disposing DB Code 
2023-03-07 16:24:52.8311|Disposing DB CanonicalHashTrie 
2023-03-07 16:24:52.8370|Disposing DB Witness 
2023-03-07 16:24:52.8397|Disposing DB Metadata 
2023-03-07 16:24:52.8397|Disposing DB Blocks 
2023-03-07 16:24:53.0416|All DBs closed. 
2023-03-07 16:24:53.0416|Ethereum shutdown complete... please wait for all components to close 
2023-03-07 16:24:53.0416|All done, goodbye!
benaadams commented 1 year ago

Which version of Nethermind?

zguesmi commented 1 year ago

A modified 1.14.1. Specifically this commit https://github.com/NethermindEth/nethermind/pull/5005/commits/c7d2f4fc7238ebbf4e1cf38d4187dd13cfeffa47

LukaszRozmej commented 1 year ago

regression think it is related to the regression we fixed in master.

zguesmi commented 1 year ago

We will upgrade to a recent version and let you know.

LukaszRozmej commented 1 year ago

Assumed fixed