NethermindEth / nethermind

A robust execution client for Ethereum node operators.
https://nethermind.io/nethermind-client
GNU General Public License v3.0
1.25k stars 431 forks source link

'eth_getLogs' returns empty result where OpenEthereum doesn't #2868

Closed jaensen closed 3 years ago

jaensen commented 3 years ago

Hi!

I try to use Nethermind's websocket rpc interface to query some events on xDai via web3.js but the node always returns an empty array as result.
I created a test-repo that demonstrates the problem: https://github.com/jaensen/eth_events_nethermind

Info:

Started with: 
$> ./Nethermind.Runner --config configs/xdai.cfg --Init.MemoryHint 4000000000 --Init.WebSocketsEnabled true --JsonRpc.WebSocketsPort 8545 --JsonRpc.Enabled true

======================== Nethermind initialization completed ========================
Version      : 1.10.27-0-26b6754b2-20210308/X64-Linux/5.0.3
RPC modules  : Admin, Debug, Eth, Net, Parity, Personal, Proof, Subscribe, Trace, TxPool, Web3
Mem est tx   :     41MB
Mem est DB   :   2207MB
JSON RPC     : http://127.0.0.1:8545
Genesis hash : 0x4f1dd23188aab3a76b463e4af801b52b1248ef073c648cbdc4c9333d3da79756
Chainspec    : chainspec/xdai.json
Chain ID     : 100
Chain head   : 0 (0x4f1dd2...a79756)
===================================================================================== 

Also I synced two times already (using the same 1.10.27 Installation) but keep getting a Nethermind.Trie.TrieException:
Node 0x5eec5a9708de5f41f06dbf84cd1f72f64020a7663949345048e5ea8073ff9710 is missing from the DB like someone else already expierienced in #1691.

However I used different options as above for the sync: ./Nethermind.Runner --config configs/xdai.cfg --Init.MemoryHint 64000000000. I changed it to the options above after the sync to enable RPC and to circumvent #2862.

BlockchainProcessor encountered an exception. Nethermind.Trie.TrieException: Node 0x5eec5a9708de5f41f06dbf84cd1f72f64020a7663949345048e5ea8073ff9710 is missing from the DB
   at Nethermind.Trie.Pruning.TrieStore.LoadRlp(Keccak keccak)
   at Nethermind.Trie.Pruning.ReadOnlyTrieStore.LoadRlp(Keccak hash)
   at Nethermind.Trie.TrieNode.ResolveNode(ITrieNodeResolver tree)
   at Nethermind.Trie.PatriciaTree.Run(Span`1 updatePath, Int32 nibblesCount, Byte[] updateValue, Boolean isUpdate, Boolean ignoreMissingDelete, Keccak startRootHash)
   at Nethermind.Trie.PatriciaTree.Get(Span`1 rawKey, Keccak rootHash)
   at Nethermind.State.StateTree.Get(Address address, Keccak rootHash)
   at Nethermind.State.StateProvider.GetState(Address address)
   at Nethermind.State.StateProvider.GetAndAddToCache(Address address)
   at Nethermind.State.StateProvider.GetThroughCache(Address address)
   at Nethermind.State.StateProvider.GetCodeHash(Address address)
   at Nethermind.State.IReadOnlyStateProvider.IsContract(Address address)
   at Nethermind.Evm.ReadOnlyTransactionProcessor.IsContractDeployed(Address address)
   at Nethermind.Blockchain.Contracts.Contract.ConstantContract.CallRaw(CallInfo callInfo, IReadOnlyTransactionProcessor readOnlyTransactionProcessor)
   at Nethermind.Consensus.AuRa.Contracts.TransactionPermissionContract.PermissionConstantContract.CallRaw(CallInfo callInfo, IReadOnlyTransactionProcessor readOnlyTransactionProcessor)
   at Nethermind.Blockchain.Contracts.Contract.ConstantContract.Call(CallInfo callInfo)
   at Nethermind.Blockchain.Contracts.Contract.ConstantContract.Call[T](CallInfo callInfo)
   at Nethermind.Blockchain.Contracts.Contract.ConstantContract.Call[T](BlockHeader parentHeader, String functionName, Address sender, Object[] arguments)
   at Nethermind.Consensus.AuRa.Contracts.TransactionPermissionContract.ContractVersion(BlockHeader blockHeader)
   at Nethermind.Consensus.AuRa.Contracts.VersionedContract`1.ResolveVersion(BlockHeader blockHeader)
   at Nethermind.Consensus.AuRa.Transactions.PermissionBasedTxFilter.GetVersionedContract(BlockHeader blockHeader)
   at Nethermind.Consensus.AuRa.Transactions.PermissionBasedTxFilter.GetPermissionsFromContract(Transaction tx, BlockHeader parentHeader, ValueTuple`2& key)
   at Nethermind.Consensus.AuRa.Transactions.PermissionBasedTxFilter.GetPermissions(Transaction tx, BlockHeader parentHeader)
   at Nethermind.Consensus.AuRa.Transactions.PermissionBasedTxFilter.IsAllowed(Transaction tx, BlockHeader parentHeader)
   at Nethermind.Consensus.AuRa.AuRaBlockProcessor.ValidateTxs(Block block, BlockHeader parentHeader)
   at Nethermind.Consensus.AuRa.AuRaBlockProcessor.ValidateAuRa(Block block)
   at Nethermind.Consensus.AuRa.AuRaBlockProcessor.ProcessBlock(Block block, IBlockTracer blockTracer, ProcessingOptions options)
   at Nethermind.Blockchain.Processing.BlockProcessor.ProcessOne(Block suggestedBlock, ProcessingOptions options, IBlockTracer blockTracer)
   at Nethermind.Blockchain.Processing.BlockProcessor.Process(Keccak newBranchStateRoot, List`1 suggestedBlocks, ProcessingOptions options, IBlockTracer blockTracer)
   at Nethermind.Blockchain.Processing.BlockchainProcessor.ProcessBranch(ProcessingBranch processingBranch, ProcessingOptions options, IBlockTracer tracer)
   at Nethermind.Blockchain.Processing.BlockchainProcessor.Process(Block suggestedBlock, ProcessingOptions options, IBlockTracer tracer)
   at Nethermind.Blockchain.Processing.BlockchainProcessor.RunProcessingLoop()
   at System.Threading.Tasks.Task.InnerInvoke()
   at System.Threading.Tasks.Task.<>c.<.cctor>b__277_0(Object obj)
   at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)
--- End of stack trace from previous location ---
   at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)
   at System.Threading.Tasks.Task.ExecuteWithThreadLocal(Task& currentTaskSlot, Thread threadPoolThread)

Thanks in advance!

jaensen commented 3 years ago

never mind ;)
I'm just seeing that the gateway I used for the example (xdai.poanetwork.dev) also uses Nethermind (used to be OE as far as I remember?!).
So maybe just something is wrong with my local config?!

jaensen commented 3 years ago

I restarted the node two or three times while testing and was surprised to see that my query suddenly worked. Here are some excerpts from the log. I noticed that the TrieException disappeared in the mean time:

Run x at "2021-03-08 16:49":

2021-03-08 16:49:23.4025|INFO|20|Setting up memory allowances 
2021-03-08 16:49:23.4092|INFO|20|  memory hint:        4000MB 
2021-03-08 16:49:23.4092|INFO|20|  general memory:     32MB 
2021-03-08 16:49:23.4092|INFO|20|  peers memory:       50MB 
2021-03-08 16:49:23.4092|INFO|20|  Netty memory:       268MB 
2021-03-08 16:49:23.4103|INFO|20|  mempool memory:     74MB 
2021-03-08 16:49:23.4103|INFO|20|  fast blocks memory: 357MB 
2021-03-08 16:49:23.4103|INFO|20|  trie memory:        643MB 
2021-03-08 16:49:23.4103|INFO|20|  DB memory:          2574MB 
2021-03-08 16:49:23.4378|INFO|21|Using http://ipv4.icanhazip.com to get external ip 
2021-03-08 16:49:23.7861|WARN|21|Start block loaded from HEAD - 0 (0x4f1dd2...a79756) 
2021-03-08 16:49:23.7963|INFO|21|Numbers resolved, level = Max(14819999, 14908011), header = Max(14819999, 14908011), body = Max(14819999, 14908011) 
2021-03-08 16:49:23.7963|INFO|21|Block tree initialized, last processed is 0 (0x4f1dd2...a79756), best queued is 14908011, best known is 14908011, lowest inserted header 1, body 0 
2021-03-08 16:49:23.8022|INFO|4|Initializing 4 plugins 
2021-03-08 16:49:23.8022|INFO|4|  Clique by Nethermind 
2021-03-08 16:49:23.8022|INFO|4|  Clique by Nethermind initialized in 0ms 
2021-03-08 16:49:23.8022|INFO|4|  Ethash by Nethermind 
2021-03-08 16:49:23.8022|INFO|4|  Ethash by Nethermind initialized in 0ms 
2021-03-08 16:49:23.8022|INFO|4|  NethDev by Nethermind 
2021-03-08 16:49:23.8022|INFO|4|  NethDev by Nethermind initialized in 0ms 
2021-03-08 16:49:23.8022|INFO|4|  HealthChecks by Nethermind 
2021-03-08 16:49:23.8022|INFO|4|  HealthChecks by Nethermind initialized in 0ms 
2021-03-08 16:49:23.9114|INFO|21|No block tree levels to review for fixes. All fine. 
2021-03-08 16:49:23.9169|INFO|21|Numbers resolved, level = Max(14819999, 14908011), header = Max(14819999, 14908011), body = Max(14819999, 14908011) 
2021-03-08 16:49:24.0235|INFO|7|Loaded 0 static nodes from file: /media/daniel/samsung970pro/nethermind/Data/static-nodes.json 
2021-03-08 16:49:24.0506|INFO|7|Using pivot 14820000 and barrier 1 in bodies sync 
2021-03-08 16:49:24.0506|INFO|7|Using pivot 14820000 and barrier 1 in receipts sync 
2021-03-08 16:49:24.0557|INFO|7|Now syncing nodes starting from root of block 0 
2021-03-08 16:49:24.0638|INFO|4|HeadersSyncDispatcher has finished work. 
2021-03-08 16:49:24.0638|INFO|4|Fast blocks headers task completed. 
2021-03-08 16:49:24.0705|INFO|21|Grafana / Prometheus metrics are disabled in configuration 
2021-03-08 16:49:24.2196|INFO|21|
======================== Nethermind initialization completed ========================
Version      : 1.10.27-0-26b6754b2-20210308/X64-Linux/5.0.3
This node    : enode://483b15708c80a71494f6dde75b5e1f4094556f272bf5d3811c5ba15dba0864b17db2299bef70485a234f08f61dfffe076db85ecf2b3f966432b89e1b6b649576@my-ip:30303
RPC modules  : Admin, Debug, Eth, Net, Parity, Personal, Proof, Subscribe, Trace, TxPool, Web3
Node address : 0x44335d498f2f5c28aed3d0fce2326572aaa38ac2 (do not use as an account)
Mem est tx   :     41MB
Mem est DB   :   2187MB
JSON RPC     : http://127.0.0.1:8545
Genesis hash : 0x4f1dd23188aab3a76b463e4af801b52b1248ef073c648cbdc4c9333d3da79756
External IP  : my-ip
Ethereum     : tcp://my-ip:30303
Discovery    : udp://my-ip:30303
Chainspec    : chainspec/xdai.json
Chain ID     : 100
Chain head   : 0 (0x4f1dd2...a79756)
===================================================================================== 
2021-03-08 16:49:24.5077|INFO|15|Connected to 0 bootnodes, 15 trusted/persisted nodes 
2021-03-08 16:49:24.9242|INFO|98|Peers | with known best block: 0 | all: 0 | 
2021-03-08 16:49:24.9242|INFO|98|Waiting for peers... 0s 
2021-03-08 16:49:24.9263|INFO|98|Sync peers - Initialized: 0 | All: 0 | Max: 50 
2021-03-08 16:49:25.9278|INFO|4|Waiting for peers... 2s 
2021-03-08 16:49:25.9300|INFO|109|Changing state Disconnected to Full at processed:0|state:14907958|block:14908011|header:14908011|peer block:14908024 
2021-03-08 16:49:26.1299|ERROR|105|BlockchainProcessor encountered an exception. Nethermind.Trie.TrieException: Node 0x294e00460dc102590702a2e96ff81465d0dddf0e6755fdef3214652de1e6c02d is missing from the DB
   at Nethermind.Trie.Pruning.TrieStore.LoadRlp(Keccak keccak)
   at Nethermind.Trie.Pruning.ReadOnlyTrieStore.LoadRlp(Keccak hash)
   at Nethermind.Trie.TrieNode.ResolveNode(ITrieNodeResolver tree)
   at Nethermind.Trie.PatriciaTree.Run(Span`1 updatePath, Int32 nibblesCount, Byte[] updateValue, Boolean isUpdate, Boolean ignoreMissingDelete, Keccak startRootHash)
   at Nethermind.Trie.PatriciaTree.Get(Span`1 rawKey, Keccak rootHash)
   at Nethermind.State.StateTree.Get(Address address, Keccak rootHash)
   at Nethermind.State.StateProvider.GetState(Address address)
   at Nethermind.State.StateProvider.GetAndAddToCache(Address address)
   at Nethermind.State.StateProvider.GetThroughCache(Address address)
   at Nethermind.State.StateProvider.GetCodeHash(Address address)
   at Nethermind.State.IReadOnlyStateProvider.IsContract(Address address)
   at Nethermind.Evm.ReadOnlyTransactionProcessor.IsContractDeployed(Address address)
   at Nethermind.Blockchain.Contracts.Contract.ConstantContract.CallRaw(CallInfo callInfo, IReadOnlyTransactionProcessor readOnlyTransactionProcessor)
   at Nethermind.Consensus.AuRa.Contracts.TransactionPermissionContract.PermissionConstantContract.CallRaw(CallInfo callInfo, IReadOnlyTransactionProcessor readOnlyTransactionProcessor)
   at Nethermind.Blockchain.Contracts.Contract.ConstantContract.Call(CallInfo callInfo)
   at Nethermind.Blockchain.Contracts.Contract.ConstantContract.Call[T](CallInfo callInfo)
   at Nethermind.Blockchain.Contracts.Contract.ConstantContract.Call[T](BlockHeader parentHeader, String functionName, Address sender, Object[] arguments)
   at Nethermind.Consensus.AuRa.Contracts.TransactionPermissionContract.ContractVersion(BlockHeader blockHeader)
   at Nethermind.Consensus.AuRa.Contracts.VersionedContract`1.ResolveVersion(BlockHeader blockHeader)
   at Nethermind.Consensus.AuRa.Transactions.PermissionBasedTxFilter.GetVersionedContract(BlockHeader blockHeader)
   at Nethermind.Consensus.AuRa.Transactions.PermissionBasedTxFilter.GetPermissionsFromContract(Transaction tx, BlockHeader parentHeader, ValueTuple`2& key)
   at Nethermind.Consensus.AuRa.Transactions.PermissionBasedTxFilter.GetPermissions(Transaction tx, BlockHeader parentHeader)
   at Nethermind.Consensus.AuRa.Transactions.PermissionBasedTxFilter.IsAllowed(Transaction tx, BlockHeader parentHeader)
   at Nethermind.Consensus.AuRa.AuRaBlockProcessor.ValidateTxs(Block block, BlockHeader parentHeader)
   at Nethermind.Consensus.AuRa.AuRaBlockProcessor.ValidateAuRa(Block block)
   at Nethermind.Consensus.AuRa.AuRaBlockProcessor.ProcessBlock(Block block, IBlockTracer blockTracer, ProcessingOptions options)
   at Nethermind.Blockchain.Processing.BlockProcessor.ProcessOne(Block suggestedBlock, ProcessingOptions options, IBlockTracer blockTracer)
   at Nethermind.Blockchain.Processing.BlockProcessor.Process(Keccak newBranchStateRoot, List`1 suggestedBlocks, ProcessingOptions options, IBlockTracer blockTracer)
   at Nethermind.Blockchain.Processing.BlockchainProcessor.ProcessBranch(ProcessingBranch processingBranch, ProcessingOptions options, IBlockTracer tracer)
   at Nethermind.Blockchain.Processing.BlockchainProcessor.Process(Block suggestedBlock, ProcessingOptions options, IBlockTracer tracer)
   at Nethermind.Blockchain.Processing.BlockchainProcessor.RunProcessingLoop()
   at System.Threading.Tasks.Task.InnerInvoke()
   at System.Threading.Tasks.Task.<>c.<.cctor>b__277_0(Object obj)
   at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)
--- End of stack trace from previous location ---
   at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)
   at System.Threading.Tasks.Task.ExecuteWithThreadLocal(Task& currentTaskSlot, Thread threadPoolThread)
2021-03-08 16:49:26.2690|INFO|25|Discovered new block 14908025 16:49:25 (0xd9ad43...48d136), tx count: 1 miner 0x6dc0c0be4c8b2dfe750156dc7d59faabfb5b923d, sent by [Peer|eth64|14908025|OpenEthereum/v3.0.1-stable-8ca8089-20200601/x86_64-alpine-linux-musl/rustc1.43.1|116.202.234.103:30303], with AuRa step 323043713 
2021-03-08 16:49:35.3166|INFO|25|Discovered new block 14908026 16:49:35 (0xf8bbb0...510035), tx count: 6 miner 0xa13d45301207711b7c0328c6b2b64862abfe9b7a, sent by [Peer|eth64|14908026|OpenEthereum/v3.0.1-stable-8ca8089-20200601/x86_64-alpine-linux-musl/rustc1.43.1|116.202.234.103:30303], with AuRa step 323043715 
2021-03-08 16:49:37.3944|INFO|18|Initializing WebSockets for client: ''. 
2021-03-08 16:49:39.9285|INFO|108|Peers | with known best block: 43 | all: 43 | 
2021-03-08 16:49:40.6214|INFO|25|Discovered new block 14908027 16:49:40 (0x5ede6f...73ce46), tx count: 3 miner 0x657ea4a9572dfdbfd95899eada0f6197211527be, sent by [Peer|eth64|14908027|OpenEthereum/v3.0.1-stable-8ca8089-20200601/x86_64-unknown-linux-gnu/rustc1.43.1|157.245.4.20:30303], with AuRa step 323043716 
2021-03-08 16:49:43.6164|INFO|19|Executing JSON RPC call eth_getLogs with params [{
  "fromBlock": "0xbf2f32",
  "toBlock": "latest",
  "topics": [
    "0x358ba8f768af134eb5af120e9a61dc1ef29b29f597f047b555fc3675064a0342",
    "0x000000000000000000000000de374ece6fa50e781e81aac78e811b33d16912c7"
  ],
  "address": "0x29b9a7fbb8995b2423a71cc17cf9810798f6c543"
}] 

This yielded no result for my query.

Run x at "2021-03-08 17:38":

2021-03-08 17:38:54.1125|INFO|11|Setting up memory allowances 
2021-03-08 17:38:54.1191|INFO|11|  memory hint:        4000MB 
2021-03-08 17:38:54.1191|INFO|11|  general memory:     32MB 
2021-03-08 17:38:54.1191|INFO|11|  peers memory:       50MB 
2021-03-08 17:38:54.1191|INFO|11|  Netty memory:       268MB 
2021-03-08 17:38:54.1191|INFO|11|  mempool memory:     74MB 
2021-03-08 17:38:54.1191|INFO|11|  fast blocks memory: 357MB 
2021-03-08 17:38:54.1191|INFO|11|  trie memory:        643MB 
2021-03-08 17:38:54.1229|INFO|11|  DB memory:          2574MB 
2021-03-08 17:38:54.1629|INFO|21|Using http://ipv4.icanhazip.com to get external ip 
2021-03-08 17:38:54.4856|WARN|11|Start block loaded from HEAD - 0 (0x4f1dd2...a79756) 
2021-03-08 17:38:54.4956|INFO|11|Numbers resolved, level = Max(14819999, 14908427), header = Max(14819999, 14908427), body = Max(14819999, 14908427) 
2021-03-08 17:38:54.4956|INFO|11|Block tree initialized, last processed is 0 (0x4f1dd2...a79756), best queued is 14908427, best known is 14908427, lowest inserted header 1, body 0 
2021-03-08 17:38:54.5012|INFO|12|Initializing 4 plugins 
2021-03-08 17:38:54.5012|INFO|12|  Clique by Nethermind 
2021-03-08 17:38:54.5012|INFO|12|  Clique by Nethermind initialized in 0ms 
2021-03-08 17:38:54.5012|INFO|12|  Ethash by Nethermind 
2021-03-08 17:38:54.5012|INFO|12|  Ethash by Nethermind initialized in 0ms 
2021-03-08 17:38:54.5012|INFO|12|  NethDev by Nethermind 
2021-03-08 17:38:54.5012|INFO|12|  NethDev by Nethermind initialized in 0ms 
2021-03-08 17:38:54.5020|INFO|12|  HealthChecks by Nethermind 
2021-03-08 17:38:54.5020|INFO|12|  HealthChecks by Nethermind initialized in 0ms 
2021-03-08 17:38:54.6089|INFO|11|No block tree levels to review for fixes. All fine. 
2021-03-08 17:38:54.6141|INFO|11|Numbers resolved, level = Max(14819999, 14908427), header = Max(14819999, 14908427), body = Max(14819999, 14908427) 
2021-03-08 17:38:54.7211|INFO|10|Loaded 0 static nodes from file: /media/daniel/samsung970pro/nethermind/Data/static-nodes.json 
2021-03-08 17:38:54.7481|INFO|10|Using pivot 14820000 and barrier 1 in bodies sync 
2021-03-08 17:38:54.7481|INFO|10|Using pivot 14820000 and barrier 1 in receipts sync 
2021-03-08 17:38:54.7543|INFO|10|Now syncing nodes starting from root of block 0 
2021-03-08 17:38:54.7595|INFO|7|HeadersSyncDispatcher has finished work. 
2021-03-08 17:38:54.7595|INFO|11|Fast blocks headers task completed. 
2021-03-08 17:38:54.7672|INFO|7|Grafana / Prometheus metrics are disabled in configuration 
2021-03-08 17:38:54.8996|INFO|17|
======================== Nethermind initialization completed ========================
Version      : 1.10.27-0-26b6754b2-20210308/X64-Linux/5.0.3
This node    : enode://483b15708c80a71494f6dde75b5e1f4094556f272bf5d3811c5ba15dba0864b17db2299bef70485a234f08f61dfffe076db85ecf2b3f966432b89e1b6b649576@my-ip:30303
RPC modules  : Admin, Debug, Eth, Net, Parity, Personal, Proof, Subscribe, Trace, TxPool, Web3
Node address : 0x44335d498f2f5c28aed3d0fce2326572aaa38ac2 (do not use as an account)
Mem est tx   :     41MB
Mem est DB   :   2207MB
JSON RPC     : http://127.0.0.1:8545
Genesis hash : 0x4f1dd23188aab3a76b463e4af801b52b1248ef073c648cbdc4c9333d3da79756
External IP  : my-ip
Ethereum     : tcp://my-ip:30303
Discovery    : udp://my-ip:30303
Chainspec    : chainspec/xdai.json
Chain ID     : 100
Chain head   : 0 (0x4f1dd2...a79756)
===================================================================================== 
2021-03-08 17:38:55.2501|INFO|11|Connected to 0 bootnodes, 78 trusted/persisted nodes 
2021-03-08 17:38:55.6231|INFO|14|Peers | with known best block: 0 | all: 0 | 
2021-03-08 17:38:55.6231|INFO|14|Waiting for peers... 1s 
2021-03-08 17:38:55.6231|INFO|14|Sync peers - Initialized: 0 | All: 0 | Max: 50 
2021-03-08 17:38:56.6230|INFO|9|Waiting for peers... 2s 
2021-03-08 17:38:56.6273|INFO|21|Changing state Disconnected to Full at processed:0|state:14908342|block:14908427|header:14908427|peer block:14908560 
2021-03-08 17:38:56.8862|ERROR|100|BlockchainProcessor encountered an exception. Nethermind.Trie.TrieException: Node 0x5eec5a9708de5f41f06dbf84cd1f72f64020a7663949345048e5ea8073ff9710 is missing from the DB
   at Nethermind.Trie.Pruning.TrieStore.LoadRlp(Keccak keccak)
   at Nethermind.Trie.Pruning.ReadOnlyTrieStore.LoadRlp(Keccak hash)
   at Nethermind.Trie.TrieNode.ResolveNode(ITrieNodeResolver tree)
   at Nethermind.Trie.PatriciaTree.Run(Span`1 updatePath, Int32 nibblesCount, Byte[] updateValue, Boolean isUpdate, Boolean ignoreMissingDelete, Keccak startRootHash)
   at Nethermind.Trie.PatriciaTree.Get(Span`1 rawKey, Keccak rootHash)
   at Nethermind.State.StateTree.Get(Address address, Keccak rootHash)
   at Nethermind.State.StateProvider.GetState(Address address)
   at Nethermind.State.StateProvider.GetAndAddToCache(Address address)
   at Nethermind.State.StateProvider.GetThroughCache(Address address)
   at Nethermind.State.StateProvider.GetCodeHash(Address address)
   at Nethermind.State.IReadOnlyStateProvider.IsContract(Address address)
   at Nethermind.Evm.ReadOnlyTransactionProcessor.IsContractDeployed(Address address)
   at Nethermind.Blockchain.Contracts.Contract.ConstantContract.CallRaw(CallInfo callInfo, IReadOnlyTransactionProcessor readOnlyTransactionProcessor)
   at Nethermind.Consensus.AuRa.Contracts.TransactionPermissionContract.PermissionConstantContract.CallRaw(CallInfo callInfo, IReadOnlyTransactionProcessor readOnlyTransactionProcessor)
   at Nethermind.Blockchain.Contracts.Contract.ConstantContract.Call(CallInfo callInfo)
   at Nethermind.Blockchain.Contracts.Contract.ConstantContract.Call[T](CallInfo callInfo)
   at Nethermind.Blockchain.Contracts.Contract.ConstantContract.Call[T](BlockHeader parentHeader, String functionName, Address sender, Object[] arguments)
   at Nethermind.Consensus.AuRa.Contracts.TransactionPermissionContract.ContractVersion(BlockHeader blockHeader)
   at Nethermind.Consensus.AuRa.Contracts.VersionedContract`1.ResolveVersion(BlockHeader blockHeader)
   at Nethermind.Consensus.AuRa.Transactions.PermissionBasedTxFilter.GetVersionedContract(BlockHeader blockHeader)
   at Nethermind.Consensus.AuRa.Transactions.PermissionBasedTxFilter.GetPermissionsFromContract(Transaction tx, BlockHeader parentHeader, ValueTuple`2& key)
   at Nethermind.Consensus.AuRa.Transactions.PermissionBasedTxFilter.GetPermissions(Transaction tx, BlockHeader parentHeader)
   at Nethermind.Consensus.AuRa.Transactions.PermissionBasedTxFilter.IsAllowed(Transaction tx, BlockHeader parentHeader)
   at Nethermind.Consensus.AuRa.AuRaBlockProcessor.ValidateTxs(Block block, BlockHeader parentHeader)
   at Nethermind.Consensus.AuRa.AuRaBlockProcessor.ValidateAuRa(Block block)
   at Nethermind.Consensus.AuRa.AuRaBlockProcessor.ProcessBlock(Block block, IBlockTracer blockTracer, ProcessingOptions options)
   at Nethermind.Blockchain.Processing.BlockProcessor.ProcessOne(Block suggestedBlock, ProcessingOptions options, IBlockTracer blockTracer)
   at Nethermind.Blockchain.Processing.BlockProcessor.Process(Keccak newBranchStateRoot, List`1 suggestedBlocks, ProcessingOptions options, IBlockTracer blockTracer)
   at Nethermind.Blockchain.Processing.BlockchainProcessor.ProcessBranch(ProcessingBranch processingBranch, ProcessingOptions options, IBlockTracer tracer)
   at Nethermind.Blockchain.Processing.BlockchainProcessor.Process(Block suggestedBlock, ProcessingOptions options, IBlockTracer tracer)
   at Nethermind.Blockchain.Processing.BlockchainProcessor.RunProcessingLoop()
   at System.Threading.Tasks.Task.InnerInvoke()
   at System.Threading.Tasks.Task.<>c.<.cctor>b__277_0(Object obj)
   at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)
--- End of stack trace from previous location ---
   at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)
   at System.Threading.Tasks.Task.ExecuteWithThreadLocal(Task& currentTaskSlot, Thread threadPoolThread)
2021-03-08 17:38:57.6503|INFO|18|Changing state Full to StateNodes at processed:0|state:0|block:14908560|header:14908560|peer block:14908560 
2021-03-08 17:38:57.6503|INFO|18|Sync mode changed from Full to StateNodes 
2021-03-08 17:38:57.6503|INFO|18|Starting the node data sync from the 14908560 (0xad9ee1...18ffcb) 0xd47390ea09d01aeffc64ef0b8569f547e01d37084e51474277670f7297e42891 root 
2021-03-08 17:38:57.6503|INFO|18|Setting state sync state root to 14908560 0xd47390ea09d01aeffc64ef0b8569f547e01d37084e51474277670f7297e42891 
2021-03-08 17:38:57.6503|INFO|18|Now syncing nodes starting from root of block 14908560 
2021-03-08 17:38:57.9304|INFO|17|Branch sync progress (do not extrapolate): 0.39% of block 14908560
****************************************************************
*********************************************1******************
****************************************************************
**************************************************************** 

.... .... ....

2021-03-08 17:39:03.9920|INFO|18|Branch sync progress (do not extrapolate): 100.00% of block 14908560
++++++1+1+1+++++1++++++++++++++++111++++1+++1++++++++1+++1++++++
+111++++1+1+++++++1++++1+++++++++++++++1+++++1++1+1+++++1+++++1+
+++++++++1++++++1++++++1++++++1+1+++1++++1+++1+++++++1+++1+1++1+
++++++++++1++++++++1++++++++1++1++++++++++1+1+++++++++++++++++++ 
2021-03-08 17:39:03.9920|INFO|18|Saving root 0xd47390ea09d01aeffc64ef0b8569f547e01d37084e51474277670f7297e42891 of 14908560 
2021-03-08 17:39:04.0041|INFO|18|Setting state sync state root to 14908560 0xd47390ea09d01aeffc64ef0b8569f547e01d37084e51474277670f7297e42891 
2021-03-08 17:39:04.6756|INFO|15|Changing state StateNodes to Full at processed:0|state:14908560|block:14908560|header:14908560|peer block:14908561 
2021-03-08 17:39:04.6756|INFO|15|Sync mode changed from StateNodes to Full 

.... .... .... Running in idle for quite a while before closing

2021-03-08 18:13:08.6947|INFO|157|Closing, please wait until all functions are stopped properly... 
2021-03-08 18:13:08.7050|INFO|157|Persisting trie... 
2021-03-08 18:13:08.7080|INFO|157|Stopping session monitor... 
2021-03-08 18:13:08.7080|INFO|157|Stopping discovery app... 
2021-03-08 18:13:08.7158|INFO|157|Stopping discovery udp channel 
2021-03-08 18:13:08.7194|INFO|157|Stopping block producer... 
2021-03-08 18:13:08.7194|INFO|157|Stopping sync peer pool... 
2021-03-08 18:13:08.7194|INFO|274|Discovery shutdown complete.. please wait for all components to close 
2021-03-08 18:13:08.7237|INFO|157|Stopping peer manager... 
2021-03-08 18:13:08.7260|INFO|157|Peer Manager shutdown complete.. please wait for all components to close 
2021-03-08 18:13:08.7260|INFO|157|Stopping synchronizer... 
2021-03-08 18:13:08.7260|INFO|157|Stopping blockchain processor... 
2021-03-08 18:13:08.7260|INFO|149|State sync task completed. 
2021-03-08 18:13:08.7260|INFO|138|Fast sync blocks downloader task completed. 
2021-03-08 18:13:08.7260|INFO|11|Fast blocks receipts task completed. 
2021-03-08 18:13:08.7260|INFO|286|Full sync block downloader task completed. 
2021-03-08 18:13:08.7260|INFO|141|Fast blocks bodies task completed. 
2021-03-08 18:13:08.7302|INFO|157|Stopping rlpx peer... 
2021-03-08 18:13:08.7302|INFO|141|Blockchain Processor shutdown complete.. please wait for all components to close 
2021-03-08 18:13:10.7847|INFO|181|Peers | with known best block: 35 | all: 35 | 
2021-03-08 18:13:11.2170|INFO|92|Peer update loop canceled. 
2021-03-08 18:13:13.7352|INFO|155|Local peer shutdown complete.. please wait for all components to close 
2021-03-08 18:13:13.7352|INFO|155|Disposing plugin Clique... 
2021-03-08 18:13:13.7352|INFO|155|Disposing plugin Ethash... 
2021-03-08 18:13:13.7352|INFO|155|Disposing plugin NethDev... 
2021-03-08 18:13:13.7352|INFO|155|Disposing plugin HealthChecks... 
2021-03-08 18:13:13.7352|INFO|155|Disposing Nethermind.Core.Reactive+AnonymousDisposable... 
2021-03-08 18:13:13.7395|INFO|155|Disposing Nethermind.Synchronization.SyncServer... 
2021-03-08 18:13:13.7395|INFO|155|Disposing Nethermind.Synchronization.Synchronizer... 
2021-03-08 18:13:13.7395|INFO|155|Disposing Nethermind.Consensus.AuRa.AuRaMultiSyncModeSelector... 
2021-03-08 18:13:13.7395|INFO|155|Disposing Nethermind.Synchronization.Peers.SyncPeerPool... 
2021-03-08 18:13:13.7395|INFO|155|Disposing MultiValidator [  ]... 
2021-03-08 18:13:13.7395|INFO|264|JSON RPC service stopped 
2021-03-08 18:13:13.7395|INFO|155|Disposing Nethermind.Blockchain.ReceiptCanonicalityMonitor... 
2021-03-08 18:13:13.7395|INFO|155|Disposing Nethermind.Blockchain.OnChainTxWatcher... 
2021-03-08 18:13:13.7395|INFO|155|Disposing Nethermind.Trie.Pruning.TrieStore... 
2021-03-08 18:13:13.7395|INFO|155|Disposing Nethermind.Db.Blooms.BloomStorage... 
2021-03-08 18:13:13.7395|INFO|155|Disposing Nethermind.Crypto.CryptoRandom... 
2021-03-08 18:13:13.7395|INFO|155|Closing DBs... 
2021-03-08 18:13:13.7421|INFO|155|Disposing DB Receipts 
2021-03-08 18:13:13.7476|INFO|155|Disposing DB Witness 
2021-03-08 18:13:13.7476|INFO|155|Disposing DB Blocks 
2021-03-08 18:13:13.7792|INFO|155|Disposing DB State 
2021-03-08 18:13:13.8207|INFO|155|Disposing DB Pendingtxs 
2021-03-08 18:13:13.8376|INFO|155|Disposing DB Bloom 
2021-03-08 18:13:13.8376|INFO|155|Disposing DB CanonicalHashTrie 
2021-03-08 18:13:13.8382|INFO|155|Disposing DB Code 
2021-03-08 18:13:13.8499|INFO|155|Disposing DB Headers 
2021-03-08 18:13:13.8638|INFO|155|Disposing DB BlockInfos 
2021-03-08 18:13:13.8760|INFO|155|All DBs closed. 
2021-03-08 18:13:13.8760|INFO|155|Ethereum shutdown complete... please wait for all components to close 
2021-03-08 18:13:13.8760|INFO|155|All done, goodbye! 

This run was only stitting there in idle.

Run x at "2021-03-08 18:34:48": (testing if running with low memory limit changed something for the cache so that I can start the node with more memory again -> seems so. No LruCache error on startup):

2021-03-08 18:34:48.5905|INFO|15|Setting up memory allowances 
2021-03-08 18:34:48.5971|INFO|15|  memory hint:        64000MB 
2021-03-08 18:34:48.5971|INFO|15|  general memory:     32MB 
2021-03-08 18:34:48.5971|INFO|15|  peers memory:       50MB 
2021-03-08 18:34:48.5971|INFO|15|  Netty memory:       268MB 
2021-03-08 18:34:48.5982|INFO|15|  mempool memory:     74MB 
2021-03-08 18:34:48.5982|INFO|15|  fast blocks memory: 1000MB 
2021-03-08 18:34:48.5982|INFO|15|  trie memory:        12515MB 
2021-03-08 18:34:48.5982|INFO|15|  DB memory:          50060MB 
2021-03-08 18:34:48.6185|INFO|21|Using http://ipv4.icanhazip.com to get external ip 
2021-03-08 18:34:48.9510|WARN|12|Start block loaded from HEAD - 0 (0x4f1dd2...a79756) 
2021-03-08 18:34:48.9608|INFO|12|Numbers resolved, level = Max(14819999, 14908942), header = Max(14819999, 14908942), body = Max(14819999, 14908942) 
2021-03-08 18:34:48.9608|INFO|12|Block tree initialized, last processed is 0 (0x4f1dd2...a79756), best queued is 14908942, best known is 14908942, lowest inserted header 1, body 0 
2021-03-08 18:34:48.9664|INFO|7|Initializing 4 plugins 
2021-03-08 18:34:48.9664|INFO|7|  Clique by Nethermind 
2021-03-08 18:34:48.9664|INFO|7|  Clique by Nethermind initialized in 0ms 
2021-03-08 18:34:48.9664|INFO|7|  Ethash by Nethermind 
2021-03-08 18:34:48.9664|INFO|7|  Ethash by Nethermind initialized in 0ms 
2021-03-08 18:34:48.9664|INFO|7|  NethDev by Nethermind 
2021-03-08 18:34:48.9664|INFO|7|  NethDev by Nethermind initialized in 0ms 
2021-03-08 18:34:48.9664|INFO|7|  HealthChecks by Nethermind 
2021-03-08 18:34:48.9664|INFO|7|  HealthChecks by Nethermind initialized in 0ms 
2021-03-08 18:34:49.0690|INFO|12|No block tree levels to review for fixes. All fine. 
2021-03-08 18:34:49.0741|INFO|12|Numbers resolved, level = Max(14819999, 14908942), header = Max(14819999, 14908942), body = Max(14819999, 14908942) 
2021-03-08 18:34:49.1681|INFO|4|Loaded 0 static nodes from file: /media/daniel/samsung970pro/nethermind/Data/static-nodes.json 
2021-03-08 18:34:49.1946|INFO|4|Using pivot 14820000 and barrier 1 in bodies sync 
2021-03-08 18:34:49.1946|INFO|4|Using pivot 14820000 and barrier 1 in receipts sync 
2021-03-08 18:34:49.1997|INFO|4|Now syncing nodes starting from root of block 0 
2021-03-08 18:34:49.2075|INFO|20|HeadersSyncDispatcher has finished work. 
2021-03-08 18:34:49.2075|INFO|20|Fast blocks headers task completed. 
2021-03-08 18:34:49.2124|INFO|4|Grafana / Prometheus metrics are disabled in configuration 
2021-03-08 18:34:49.3332|INFO|16|
======================== Nethermind initialization completed ========================
Version      : 1.10.27-0-26b6754b2-20210308/X64-Linux/5.0.3
This node    : enode://483b15708c80a71494f6dde75b5e1f4094556f272bf5d3811c5ba15dba0864b17db2299bef70485a234f08f61dfffe076db85ecf2b3f966432b89e1b6b649576@my-ip:30303
RPC modules  : Admin, Debug, Eth, Net, Parity, Personal, Proof, Subscribe, Trace, TxPool, Web3
Node address : 0x44335d498f2f5c28aed3d0fce2326572aaa38ac2 (do not use as an account)
Mem est tx   :     41MB
Mem est DB   :  45361MB
JSON RPC     : http://127.0.0.1:8545
Genesis hash : 0x4f1dd23188aab3a76b463e4af801b52b1248ef073c648cbdc4c9333d3da79756
External IP  : my-ip
Ethereum     : tcp://my-ip:30303
Discovery    : udp://my-ip:30303
Chainspec    : chainspec/xdai.json
Chain ID     : 100
Chain head   : 0 (0x4f1dd2...a79756)
===================================================================================== 
2021-03-08 18:34:49.5873|INFO|6|Connected to 0 bootnodes, 51 trusted/persisted nodes 
2021-03-08 18:34:50.0816|INFO|100|Peers | with known best block: 2 | all: 4 | 
2021-03-08 18:34:50.0816|INFO|100|Waiting for peers... 1s 
2021-03-08 18:34:50.0864|INFO|100|Sync peers - Initialized: 2 | All: 4 | Max: 50
   [     ][     ][Peer|eth63|14909185|Parity-Ethereum/v2.7.2-posdao-stable-2ea4265-20200226/x86_64-unknown-linux-gnu/rustc1.41.0|116.203.39.113:30303][0|19|0|0|0]
   [     ][     ][Peer|eth64|14909185|OpenEthereum/v3.0.1-stable-8ca8089-20200601/x86_64-alpine-linux-musl/rustc1.43.1|95.217.216.163:30303][0|10|0|0|0] 
2021-03-08 18:34:50.0864|INFO|10|Changing state Disconnected to Full at processed:0|state:14908816|block:14908942|header:14908942|peer block:14909185 
2021-03-08 18:34:50.4830|ERROR|103|BlockchainProcessor encountered an exception. Nethermind.Trie.TrieException: Node 0x91e29c5b95cdcd0738bacc882a5dbe6f2cd3f4aeca6b3c1baf576b1d6d4ef460 is missing from the DB
   at Nethermind.Trie.Pruning.TrieStore.LoadRlp(Keccak keccak)
   at Nethermind.Trie.Pruning.ReadOnlyTrieStore.LoadRlp(Keccak hash)
   at Nethermind.Trie.TrieNode.ResolveNode(ITrieNodeResolver tree)
   at Nethermind.Trie.PatriciaTree.Run(Span`1 updatePath, Int32 nibblesCount, Byte[] updateValue, Boolean isUpdate, Boolean ignoreMissingDelete, Keccak startRootHash)
   at Nethermind.Trie.PatriciaTree.Get(Span`1 rawKey, Keccak rootHash)
   at Nethermind.State.StateTree.Get(Address address, Keccak rootHash)
   at Nethermind.State.StateProvider.GetState(Address address)
   at Nethermind.State.StateProvider.GetAndAddToCache(Address address)
   at Nethermind.State.StateProvider.GetThroughCache(Address address)
   at Nethermind.State.StateProvider.GetCodeHash(Address address)
   at Nethermind.State.IReadOnlyStateProvider.IsContract(Address address)
   at Nethermind.Evm.ReadOnlyTransactionProcessor.IsContractDeployed(Address address)
   at Nethermind.Blockchain.Contracts.Contract.ConstantContract.CallRaw(CallInfo callInfo, IReadOnlyTransactionProcessor readOnlyTransactionProcessor)
   at Nethermind.Consensus.AuRa.Contracts.TransactionPermissionContract.PermissionConstantContract.CallRaw(CallInfo callInfo, IReadOnlyTransactionProcessor readOnlyTransactionProcessor)
   at Nethermind.Blockchain.Contracts.Contract.ConstantContract.Call(CallInfo callInfo)
   at Nethermind.Blockchain.Contracts.Contract.ConstantContract.Call[T](CallInfo callInfo)
   at Nethermind.Blockchain.Contracts.Contract.ConstantContract.Call[T](BlockHeader parentHeader, String functionName, Address sender, Object[] arguments)
   at Nethermind.Consensus.AuRa.Contracts.TransactionPermissionContract.ContractVersion(BlockHeader blockHeader)
   at Nethermind.Consensus.AuRa.Contracts.VersionedContract`1.ResolveVersion(BlockHeader blockHeader)
   at Nethermind.Consensus.AuRa.Transactions.PermissionBasedTxFilter.GetVersionedContract(BlockHeader blockHeader)
   at Nethermind.Consensus.AuRa.Transactions.PermissionBasedTxFilter.GetPermissionsFromContract(Transaction tx, BlockHeader parentHeader, ValueTuple`2& key)
   at Nethermind.Consensus.AuRa.Transactions.PermissionBasedTxFilter.GetPermissions(Transaction tx, BlockHeader parentHeader)
   at Nethermind.Consensus.AuRa.Transactions.PermissionBasedTxFilter.IsAllowed(Transaction tx, BlockHeader parentHeader)
   at Nethermind.Consensus.AuRa.AuRaBlockProcessor.ValidateTxs(Block block, BlockHeader parentHeader)
   at Nethermind.Consensus.AuRa.AuRaBlockProcessor.ValidateAuRa(Block block)
   at Nethermind.Consensus.AuRa.AuRaBlockProcessor.ProcessBlock(Block block, IBlockTracer blockTracer, ProcessingOptions options)
   at Nethermind.Blockchain.Processing.BlockProcessor.ProcessOne(Block suggestedBlock, ProcessingOptions options, IBlockTracer blockTracer)
   at Nethermind.Blockchain.Processing.BlockProcessor.Process(Keccak newBranchStateRoot, List`1 suggestedBlocks, ProcessingOptions options, IBlockTracer blockTracer)
   at Nethermind.Blockchain.Processing.BlockchainProcessor.ProcessBranch(ProcessingBranch processingBranch, ProcessingOptions options, IBlockTracer tracer)
   at Nethermind.Blockchain.Processing.BlockchainProcessor.Process(Block suggestedBlock, ProcessingOptions options, IBlockTracer tracer)
   at Nethermind.Blockchain.Processing.BlockchainProcessor.RunProcessingLoop()
   at System.Threading.Tasks.Task.InnerInvoke()
   at System.Threading.Tasks.Task.<>c.<.cctor>b__277_0(Object obj)
   at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)
--- End of stack trace from previous location ---
   at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)
   at System.Threading.Tasks.Task.ExecuteWithThreadLocal(Task& currentTaskSlot, Thread threadPoolThread)
2021-03-08 18:34:51.1101|INFO|143|Changing state Full to StateNodes at processed:0|state:0|block:14909185|header:14909185|peer block:14909185 
2021-03-08 18:34:51.1101|INFO|143|Sync mode changed from Full to StateNodes 
2021-03-08 18:34:51.1101|INFO|143|Starting the node data sync from the 14909185 (0xaf0ab8...1cf363) 0x16d8b68218f3672eabe2d8487fcc713d6998def4c515ef4455d14103fc4bd2ab root 
2021-03-08 18:34:51.1101|INFO|143|Setting state sync state root to 14909185 0x16d8b68218f3672eabe2d8487fcc713d6998def4c515ef4455d14103fc4bd2ab 
2021-03-08 18:34:51.1101|INFO|143|Now syncing nodes starting from root of block 14909185 
2021-03-08 18:34:51.3313|INFO|101|Branch sync progress (do not extrapolate): 0.39% of block 14909185
****************************************************************
****************************************************************
****************************************************************
*************************************************************1** 
...
2021-03-08 18:34:52.5108|INFO|10|Branch sync progress (do not extrapolate): 10.94% of block 14909185
***++*+*+***+***+*********++******++*1**+*********1**1**********
******1********************************1****1*******************
*************************************1**1***************1*******
*****************1*1***1*******1******1************1*+*******1** 
2021-03-08 18:34:55.3787|INFO|92|Peer update loop canceled. 
2021-03-08 18:34:57.4129|INFO|97|Local peer shutdown complete.. please wait for all components to close 
2021-03-08 18:34:57.4129|INFO|97|Disposing plugin Clique... 
2021-03-08 18:34:57.4129|INFO|97|Disposing plugin Ethash... 
2021-03-08 18:34:57.4129|INFO|97|Disposing plugin NethDev... 
2021-03-08 18:34:57.4129|INFO|97|Disposing plugin HealthChecks... 
2021-03-08 18:34:57.4141|INFO|97|Disposing Nethermind.Core.Reactive+AnonymousDisposable... 
2021-03-08 18:34:57.4193|INFO|97|Disposing Nethermind.Synchronization.SyncServer... 
2021-03-08 18:34:57.4193|INFO|97|Disposing Nethermind.Synchronization.Synchronizer... 
2021-03-08 18:34:57.4193|INFO|97|Disposing Nethermind.Consensus.AuRa.AuRaMultiSyncModeSelector... 
2021-03-08 18:34:57.4193|INFO|97|Disposing Nethermind.Synchronization.Peers.SyncPeerPool... 
2021-03-08 18:34:57.4193|INFO|97|Disposing MultiValidator [  ]... 
2021-03-08 18:34:57.4193|INFO|12|JSON RPC service stopped 
2021-03-08 18:34:57.4193|INFO|97|Disposing Nethermind.Blockchain.ReceiptCanonicalityMonitor... 
2021-03-08 18:34:57.4193|INFO|97|Disposing Nethermind.Blockchain.OnChainTxWatcher... 
2021-03-08 18:34:57.4193|INFO|97|Disposing Nethermind.Trie.Pruning.TrieStore... 
2021-03-08 18:34:57.4220|INFO|97|Disposing Nethermind.Db.Blooms.BloomStorage... 
2021-03-08 18:34:57.4220|INFO|97|Disposing Nethermind.Crypto.CryptoRandom... 
2021-03-08 18:34:57.4220|INFO|97|Closing DBs... 
2021-03-08 18:34:57.4220|INFO|97|Disposing DB Witness 
2021-03-08 18:34:57.4220|INFO|97|Disposing DB Headers 
2021-03-08 18:34:57.4538|INFO|97|Disposing DB Blocks 
2021-03-08 18:34:57.4708|INFO|97|Disposing DB CanonicalHashTrie 
2021-03-08 18:34:57.4708|INFO|97|Disposing DB Bloom 
2021-03-08 18:34:57.4708|INFO|97|Disposing DB Pendingtxs 
2021-03-08 18:34:57.4826|INFO|97|Disposing DB BlockInfos 
2021-03-08 18:34:57.4960|INFO|97|Disposing DB Receipts 
2021-03-08 18:34:57.5027|INFO|97|Disposing DB State 
2021-03-08 18:34:57.5220|INFO|97|Disposing DB Code 
2021-03-08 18:34:57.5331|INFO|97|All DBs closed. 
2021-03-08 18:34:57.5331|INFO|97|Ethereum shutdown complete... please wait for all components to close 
2021-03-08 18:34:57.5331|INFO|97|All done, goodbye! 

This time no Index was outside the bounds of the array. exception in Nethermind.Core.Caching.LruCache on startup even with high memory hint. However still a missing node in the db.

Run x at "2021-03-08 18:35:12": (running with 4GB memory hint again):

2021-03-08 18:35:12.0252|INFO|4|Setting up memory allowances 
2021-03-08 18:35:12.0319|INFO|4|  memory hint:        4000MB 
2021-03-08 18:35:12.0319|INFO|4|  general memory:     32MB 
2021-03-08 18:35:12.0319|INFO|4|  peers memory:       50MB 
2021-03-08 18:35:12.0319|INFO|4|  Netty memory:       268MB 
2021-03-08 18:35:12.0319|INFO|4|  mempool memory:     74MB 
2021-03-08 18:35:12.0319|INFO|4|  fast blocks memory: 357MB 
2021-03-08 18:35:12.0319|INFO|4|  trie memory:        643MB 
2021-03-08 18:35:12.0319|INFO|10|Using http://ipv4.icanhazip.com to get external ip 
2021-03-08 18:35:12.0357|INFO|4|  DB memory:          2574MB 
2021-03-08 18:35:12.4362|WARN|9|Start block loaded from HEAD - 0 (0x4f1dd2...a79756) 
2021-03-08 18:35:12.4460|INFO|9|Numbers resolved, level = Max(14819999, 14909185), header = Max(14819999, 14909185), body = Max(14819999, 14909185) 
2021-03-08 18:35:12.4470|INFO|9|Block tree initialized, last processed is 0 (0x4f1dd2...a79756), best queued is 14909185, best known is 14909185, lowest inserted header 1, body 0 
2021-03-08 18:35:12.4517|INFO|18|Initializing 4 plugins 
2021-03-08 18:35:12.4517|INFO|18|  Clique by Nethermind 
2021-03-08 18:35:12.4517|INFO|18|  Clique by Nethermind initialized in 0ms 
2021-03-08 18:35:12.4517|INFO|18|  Ethash by Nethermind 
2021-03-08 18:35:12.4517|INFO|18|  Ethash by Nethermind initialized in 0ms 
2021-03-08 18:35:12.4517|INFO|18|  NethDev by Nethermind 
2021-03-08 18:35:12.4517|INFO|18|  NethDev by Nethermind initialized in 0ms 
2021-03-08 18:35:12.4517|INFO|18|  HealthChecks by Nethermind 
2021-03-08 18:35:12.4517|INFO|18|  HealthChecks by Nethermind initialized in 0ms 
2021-03-08 18:35:12.5682|INFO|9|No block tree levels to review for fixes. All fine. 
2021-03-08 18:35:12.5731|INFO|9|Numbers resolved, level = Max(14819999, 14909185), header = Max(14819999, 14909185), body = Max(14819999, 14909185) 
2021-03-08 18:35:12.6788|INFO|12|Loaded 0 static nodes from file: /media/daniel/samsung970pro/nethermind/Data/static-nodes.json 
2021-03-08 18:35:12.7050|INFO|12|Using pivot 14820000 and barrier 1 in bodies sync 
2021-03-08 18:35:12.7050|INFO|12|Using pivot 14820000 and barrier 1 in receipts sync 
2021-03-08 18:35:12.7050|INFO|12|Using pivot 14820000 and barrier 1 in bodies sync 
2021-03-08 18:35:12.7050|INFO|12|Using pivot 14820000 and barrier 1 in receipts sync 
2021-03-08 18:35:12.7104|INFO|12|Now syncing nodes starting from root of block 0 
2021-03-08 18:35:12.7154|INFO|9|HeadersSyncDispatcher has finished work. 
2021-03-08 18:35:12.7154|INFO|18|Fast blocks headers task completed. 
2021-03-08 18:35:12.7246|INFO|18|Grafana / Prometheus metrics are disabled in configuration 
2021-03-08 18:35:12.8687|INFO|108|
======================== Nethermind initialization completed ========================
Version      : 1.10.27-0-26b6754b2-20210308/X64-Linux/5.0.3
This node    : enode://483b15708c80a71494f6dde75b5e1f4094556f272bf5d3811c5ba15dba0864b17db2299bef70485a234f08f61dfffe076db85ecf2b3f966432b89e1b6b649576@my-ip:30303
RPC modules  : Admin, Debug, Eth, Net, Parity, Personal, Proof, Subscribe, Trace, TxPool, Web3
Node address : 0x44335d498f2f5c28aed3d0fce2326572aaa38ac2 (do not use as an account)
Mem est tx   :     41MB
Mem est DB   :   2191MB
JSON RPC     : http://127.0.0.1:8545
Genesis hash : 0x4f1dd23188aab3a76b463e4af801b52b1248ef073c648cbdc4c9333d3da79756
External IP  : my-ip
Ethereum     : tcp://my-ip:30303
Discovery    : udp://my-ip:30303
Chainspec    : chainspec/xdai.json
Chain ID     : 100
Chain head   : 0 (0x4f1dd2...a79756)
===================================================================================== 
2021-03-08 18:35:13.1091|INFO|6|Connected to 0 bootnodes, 22 trusted/persisted nodes 
2021-03-08 18:35:13.5807|INFO|105|Peers | with known best block: 0 | all: 0 | 
2021-03-08 18:35:13.5807|INFO|105|Waiting for peers... 1s 
2021-03-08 18:35:13.5828|INFO|105|Sync peers - Initialized: 0 | All: 0 | Max: 50 
2021-03-08 18:35:14.5853|INFO|105|Waiting for peers... 2s 
2021-03-08 18:35:14.5890|INFO|6|Changing state Disconnected to FastSync, StateNodes at processed:0|state:0|block:14909185|header:14909185|peer block:14909189 
2021-03-08 18:35:14.5890|INFO|6|Sync mode changed from Disconnected to FastSync, StateNodes 
2021-03-08 18:35:14.6096|INFO|6|Starting the node data sync from the 14909185 (0xaf0ab8...1cf363) 0x16d8b68218f3672eabe2d8487fcc713d6998def4c515ef4455d14103fc4bd2ab root 
2021-03-08 18:35:14.6109|INFO|6|Setting state sync state root to 14909185 0x16d8b68218f3672eabe2d8487fcc713d6998def4c515ef4455d14103fc4bd2ab 
2021-03-08 18:35:14.6109|INFO|6|Now syncing nodes starting from root of block 14909185 
2021-03-08 18:35:14.9565|INFO|108|Branch sync progress (do not extrapolate): 0.39% of block 14909185
****************************************************************
********************************************1*******************
****************************************************************
**************************************************************** 
... ... ....

2021-03-08 18:35:21.5293|INFO|100|Branch sync progress (do not extrapolate): 99.22% of block 14909185
+**11+1+1+++1+++1+++++++++11++++++11+1++1+++++++++1++1++++++++++
++++++1++++++++++++++++++++++++++++++++1++++1+++++++++++++++++++
+++++++++++++++++++++++++++++++++++++1++1+++++++++++++++1+++++++
+++++++++++++++++1+1+++1+++++++1++++++1++++++++++++1+1+++++++1++ 
2021-03-08 18:35:21.6384|INFO|100|Changing state FastSync, StateNodes to StateNodes at processed:0|state:0|block:14909185|header:14909189|peer block:14909191 
2021-03-08 18:35:21.6384|INFO|100|Sync mode changed from FastSync, StateNodes to StateNodes 
2021-03-08 18:35:21.6615|INFO|131|Branch sync progress (do not extrapolate): 99.61% of block 14909185
+*+11+1+1+++1+++1+++++++++11++++++11+1++1+++++++++1++1++++++++++
++++++1++++++++++++++++++++++++++++++++1++++1+++++++++++++++++++
+++++++++++++++++++++++++++++++++++++1++1+++++++++++++++1+++++++
+++++++++++++++++1+1+++1+++++++1++++++1++++++++++++1+1+++++++1++ 
2021-03-08 18:35:21.7181|INFO|100|State Sync 00.01:02:01 | 4411.83MB | branches: 99.61% | kB/s:   205 | accounts 773885 | nodes 50992395 | diagnostics: 1.4.72ms 
2021-03-08 18:35:21.7814|INFO|131|Branch sync progress (do not extrapolate): 100.00% of block 14909185
+++11+1+1+++1+++1+++++++++11++++++11+1++1+++++++++1++1++++++++++
++++++1++++++++++++++++++++++++++++++++1++++1+++++++++++++++++++
+++++++++++++++++++++++++++++++++++++1++1+++++++++++++++1+++++++
+++++++++++++++++1+1+++1+++++++1++++++1++++++++++++1+1+++++++1++ 
2021-03-08 18:35:21.7814|INFO|131|Saving root 0x16d8b68218f3672eabe2d8487fcc713d6998def4c515ef4455d14103fc4bd2ab of 14909185 
2021-03-08 18:35:21.7923|INFO|15|Setting state sync state root to 14909185 0x16d8b68218f3672eabe2d8487fcc713d6998def4c515ef4455d14103fc4bd2ab 
2021-03-08 18:35:22.6422|INFO|141|Starting the node data sync from the 14909189 (0xb844f1...eeee6a) 0xb86126f4e5a392f38640352809e50894692695815f5ccd7f13bbbbcf9cca9c47 root 
2021-03-08 18:35:22.6422|INFO|141|Setting state sync state root to 14909189 0xb86126f4e5a392f38640352809e50894692695815f5ccd7f13bbbbcf9cca9c47 
2021-03-08 18:35:22.6422|INFO|141|Now syncing nodes starting from root of block 14909189 
2021-03-08 18:35:22.7045|INFO|132|Branch sync progress (do not extrapolate): 6.25% of block 14909189
****************************************************************
****************************************************************
****************1111111111111111********************************
**************************************************************** 
... ... ...

2021-03-08 18:35:23.4114|INFO|137|Branch sync progress (do not extrapolate): 100.00% of block 14909189
111111111+1111111111111++111111+111111+11++1111111111111111++1+1
+1111111111+11111111111111+11+1111+111111111111111111++111111111
1111+111111111111111111111111111111111+1111+11111111+1++111+1111
111111111+1111111111111111111++1111111111+11111111+1+11++1111111 
2021-03-08 18:35:23.4114|INFO|137|Saving root 0xb86126f4e5a392f38640352809e50894692695815f5ccd7f13bbbbcf9cca9c47 of 14909189 
2021-03-08 18:35:23.4114|INFO|21|Setting state sync state root to 14909189 0xb86126f4e5a392f38640352809e50894692695815f5ccd7f13bbbbcf9cca9c47 
2021-03-08 18:35:23.6443|INFO|21|Changing state StateNodes to Full at processed:0|state:14909189|block:14909185|header:14909189|peer block:14909191 
2021-03-08 18:35:23.6443|INFO|21|Sync mode changed from StateNodes to Full 
2021-03-08 18:35:23.8584|INFO|24|Current contract validators (16): [0x9233042b8e9e03d5dc6454bbbe5aee83818ff103, 0x6dc0c0be4c8b2dfe750156dc7d59faabfb5b923d, 0x9e41ba620feba8198369c26351063b26ec5b7c9e, 0xa13d45301207711b7c0328c6b2b64862abfe9b7a, 0x657ea4a9572dfdbfd95899eada0f6197211527be, 0x06d563905b085a6b3070c819bdb96a44e5665005, 0xdb1c683758f493cef2e7089a3640502ab306322a, 0x657e832b1a67cdef9e117afd2f419387259fa93e, 0x10aae121b3c62f3dafec9cc46c27b4c1dfe4a835, 0x1438087186fdbfd4c256fa2df446921e30e54df8, 0x0000999dc55126ca626c20377f0045946db69b6e, 0x9488f50c33e9616ee3b5b09cd3a9c603a108db4a, 0x1a740616e96e07d86203707c1619d9871614922a, 0x642c40173134f6e457a62d4c2033259433a53e8c, 0x35770ef700ff88d5f650597068e3aaf051f3d5a4, 0xb76756f95a9fb6ff9ad3e6cb41b734c1bd805103]. 
2021-03-08 18:35:23.8914|WARN|24|Saving reorg boundary 14909189 
2021-03-08 18:35:23.9015|INFO|24|Processed   14909190 |   11,335ms, mgasps    0.03 total    0.03, tps    0.35 total    0.35, bps    0.00 total    0.00, recv queue 0, proc queue 1 

No errors on startup this time.. Nice! But how?
Then.. Let's try my query again:

2021-03-08 18:44:55.5915|INFO|155|Executing JSON RPC call eth_getLogs with params [{
  "fromBlock": "0xbf2f32",
  "toBlock": "latest",
  "topics": [
    "0x358ba8f768af134eb5af120e9a61dc1ef29b29f597f047b555fc3675064a0342",
    "0x000000000000000000000000de374ece6fa50e781e81aac78e811b33d16912c7"
  ],
  "address": "0x29b9a7fbb8995b2423a71cc17cf9810798f6c543"
}] 

I get the correct result..
Can't belive. Again:

2021-03-08 18:45:46.2535|INFO|177|Executing JSON RPC call eth_getLogs with params [{
  "fromBlock": "0xbf2f32",
  "toBlock": "latest",
  "topics": [
    "0x358ba8f768af134eb5af120e9a61dc1ef29b29f597f047b555fc3675064a0342",
    "0x000000000000000000000000de374ece6fa50e781e81aac78e811b33d16912c7"
  ],
  "address": "0x29b9a7fbb8995b2423a71cc17cf9810798f6c543"
}] 

Yes, it's working now :)

Stop the node:

2021-03-08 18:45:49.7903|INFO|132|Closing WebSockets for client: ''. 
2021-03-08 18:45:50.4275|INFO|25|Discovered new block 14909306 18:45:50 (0x83a328...36f9be), tx count: 6 miner 0xdb1c683758f493cef2e7089a3640502ab306322a, sent by [Peer|eth63|14909306|Parity-Ethereum/v2.7.2-posdao-stable-2ea4265-20200226/x86_64-unknown-linux-gnu/rustc1.41.0|5.189.163.134:30303], with AuRa step 323045110 
2021-03-08 18:45:50.4565|INFO|24|Processed   14909306 |    4,513ms, mgasps    0.35 total    0.37, tps    1.99 total    1.86, bps    0.22 total    0.18, recv queue 0, proc queue 0 
2021-03-08 18:45:55.2913|INFO|25|Discovered new block 14909307 18:45:55 (0x27d163...300ca6), tx count: 4 miner 0x657e832b1a67cdef9e117afd2f419387259fa93e, sent by [Peer|eth64|14909307|OpenEthereum/v3.0.1-stable-8ca8089-20200601/x86_64-alpine-linux-musl/rustc1.43.1|37.120.245.149:30303], with AuRa step 323045111 
2021-03-08 18:45:55.3105|INFO|24|Processed   14909307 |    4,854ms, mgasps    0.32 total    0.37, tps    2.06 total    1.86, bps    0.21 total    0.18, recv queue 0, proc queue 0 
2021-03-08 18:45:56.3527|INFO|168|Closing, please wait until all functions are stopped properly... 
2021-03-08 18:45:56.3548|INFO|168|Persisting trie... 
2021-03-08 18:45:56.3548|WARN|168|Saving reorg boundary 14909243 
2021-03-08 18:45:56.3548|INFO|168|Stopping session monitor... 
2021-03-08 18:45:56.3548|INFO|168|Stopping discovery app... 
2021-03-08 18:45:56.3548|INFO|168|Stopping discovery udp channel 
2021-03-08 18:45:56.3585|INFO|168|Stopping block producer... 
2021-03-08 18:45:56.3585|INFO|168|Stopping sync peer pool... 
2021-03-08 18:45:56.3585|INFO|175|Discovery shutdown complete.. please wait for all components to close 
2021-03-08 18:45:56.3585|INFO|168|Stopping peer manager... 
2021-03-08 18:45:56.3585|INFO|168|Peer Manager shutdown complete.. please wait for all components to close 
2021-03-08 18:45:56.3585|INFO|168|Stopping synchronizer... 
2021-03-08 18:45:56.3585|INFO|148|State sync task completed. 
2021-03-08 18:45:56.3585|INFO|168|Stopping blockchain processor... 
2021-03-08 18:45:56.3585|INFO|189|Fast sync blocks downloader task completed. 
2021-03-08 18:45:56.3585|INFO|177|Fast blocks receipts task completed. 
2021-03-08 18:45:56.3585|INFO|148|Fast blocks bodies task completed. 
2021-03-08 18:45:56.3585|INFO|175|Full sync block downloader task completed. 
2021-03-08 18:45:56.3585|INFO|168|Stopping rlpx peer... 
2021-03-08 18:45:56.3622|INFO|175|Blockchain Processor shutdown complete.. please wait for all components to close 
2021-03-08 18:45:56.6706|INFO|92|Peer update loop canceled. 
2021-03-08 18:45:58.6307|INFO|181|Peers | with known best block: 42 | all: 42 | 
2021-03-08 18:46:01.3627|INFO|166|Local peer shutdown complete.. please wait for all components to close 
2021-03-08 18:46:01.3627|INFO|166|Disposing plugin Clique... 
2021-03-08 18:46:01.3627|INFO|166|Disposing plugin Ethash... 
2021-03-08 18:46:01.3627|INFO|166|Disposing plugin NethDev... 
2021-03-08 18:46:01.3627|INFO|166|Disposing plugin HealthChecks... 
2021-03-08 18:46:01.3627|INFO|166|Disposing Nethermind.Core.Reactive+AnonymousDisposable... 
2021-03-08 18:46:01.3691|INFO|166|Disposing Nethermind.Synchronization.SyncServer... 
2021-03-08 18:46:01.3691|INFO|166|Disposing Nethermind.Synchronization.Synchronizer... 
2021-03-08 18:46:01.3705|INFO|166|Disposing Nethermind.Consensus.AuRa.AuRaMultiSyncModeSelector... 
2021-03-08 18:46:01.3705|INFO|166|Disposing Nethermind.Synchronization.Peers.SyncPeerPool... 
2021-03-08 18:46:01.3705|INFO|166|Disposing MultiValidator [ ReportingContractBasedValidator [ ContractBasedValidator ] ]... 
2021-03-08 18:46:01.3705|INFO|166|Disposing Nethermind.Blockchain.ReceiptCanonicalityMonitor... 
2021-03-08 18:46:01.3705|INFO|166|Disposing Nethermind.Blockchain.OnChainTxWatcher... 
2021-03-08 18:46:01.3705|INFO|166|Disposing Nethermind.Trie.Pruning.TrieStore... 
2021-03-08 18:46:01.3705|INFO|166|Disposing Nethermind.Db.Blooms.BloomStorage... 
2021-03-08 18:46:01.3705|INFO|166|Disposing Nethermind.Crypto.CryptoRandom... 
2021-03-08 18:46:01.3705|INFO|166|Closing DBs... 
2021-03-08 18:46:01.3705|INFO|166|Disposing DB State 
2021-03-08 18:46:01.4603|INFO|166|Disposing DB Receipts 
2021-03-08 18:46:01.4668|INFO|166|Disposing DB Headers 
2021-03-08 18:46:01.4796|INFO|166|Disposing DB Code 
2021-03-08 18:46:01.4918|INFO|166|Disposing DB Blocks 
2021-03-08 18:46:01.5060|INFO|166|Disposing DB BlockInfos 
2021-03-08 18:46:01.5192|INFO|166|Disposing DB Bloom 
2021-03-08 18:46:01.5301|INFO|166|Disposing DB Pendingtxs 
2021-03-08 18:46:01.5427|INFO|166|Disposing DB CanonicalHashTrie 
2021-03-08 18:46:01.5427|INFO|166|Disposing DB Witness 
2021-03-08 18:46:01.5427|INFO|166|All DBs closed. 
2021-03-08 18:46:01.5427|INFO|166|Ethereum shutdown complete... please wait for all components to close 
2021-03-08 18:46:01.5427|INFO|166|All done, goodbye! 

Try again:

2021-03-08 18:49:38.2712|INFO|20|Setting up memory allowances 
2021-03-08 18:49:38.2778|INFO|20|  memory hint:        4000MB 
2021-03-08 18:49:38.2778|INFO|20|  general memory:     32MB 
2021-03-08 18:49:38.2778|INFO|20|  peers memory:       50MB 
2021-03-08 18:49:38.2785|INFO|20|  Netty memory:       268MB 
2021-03-08 18:49:38.2785|INFO|20|  mempool memory:     74MB 
2021-03-08 18:49:38.2785|INFO|20|  fast blocks memory: 357MB 
2021-03-08 18:49:38.2785|INFO|20|  trie memory:        643MB 
2021-03-08 18:49:38.2785|INFO|20|  DB memory:          2574MB 
2021-03-08 18:49:38.3077|INFO|18|Using http://ipv4.icanhazip.com to get external ip 
2021-03-08 18:49:38.6485|WARN|9|Start block loaded from reorg boundary - 14909243 - 14909243 (0x94a633...173fa6) 
2021-03-08 18:49:38.6555|INFO|9|Numbers resolved, level = Max(14909243, 14909307), header = Max(14909243, 14909307), body = Max(14909243, 14909307) 
2021-03-08 18:49:38.6555|INFO|9|Block tree initialized, last processed is 14909243 (0x94a633...173fa6), best queued is 14909307, best known is 14909307, lowest inserted header 1, body 0 
2021-03-08 18:49:38.6606|INFO|9|Initializing 4 plugins 
2021-03-08 18:49:38.6606|INFO|9|  Clique by Nethermind 
2021-03-08 18:49:38.6606|INFO|9|  Clique by Nethermind initialized in 0ms 
2021-03-08 18:49:38.6606|INFO|9|  Ethash by Nethermind 
2021-03-08 18:49:38.6606|INFO|9|  Ethash by Nethermind initialized in 0ms 
2021-03-08 18:49:38.6606|INFO|9|  NethDev by Nethermind 
2021-03-08 18:49:38.6606|INFO|9|  NethDev by Nethermind initialized in 0ms 
2021-03-08 18:49:38.6606|INFO|9|  HealthChecks by Nethermind 
2021-03-08 18:49:38.6622|INFO|9|  HealthChecks by Nethermind initialized in 0ms 
2021-03-08 18:49:38.8417|INFO|12|Found 64 block tree levels to review for fixes starting from 14909244 
2021-03-08 18:49:38.8451|INFO|12|Reviewed 0 blocks out of 64 
2021-03-08 18:49:38.8467|INFO|12|Reviewed 1 blocks out of 64 
2021-03-08 18:49:38.8467|INFO|12|Reviewed 2 blocks out of 64 
2021-03-08 18:49:38.8467|INFO|12|Reviewed 3 blocks out of 64 
2021-03-08 18:49:38.8467|INFO|12|Reviewed 4 blocks out of 64 
2021-03-08 18:49:38.8467|INFO|12|Reviewed 5 blocks out of 64 
2021-03-08 18:49:38.8467|INFO|12|Reviewed 6 blocks out of 64 
2021-03-08 18:49:38.8467|INFO|12|Reviewed 7 blocks out of 64 
2021-03-08 18:49:38.8467|INFO|12|Reviewed 8 blocks out of 64 
2021-03-08 18:49:38.8467|INFO|12|Reviewed 9 blocks out of 64 
2021-03-08 18:49:38.8467|INFO|12|Reviewed 10 blocks out of 64 
2021-03-08 18:49:38.8467|INFO|12|Reviewed 11 blocks out of 64 
2021-03-08 18:49:38.8467|INFO|12|Reviewed 12 blocks out of 64 
2021-03-08 18:49:38.8467|INFO|12|Reviewed 13 blocks out of 64 
2021-03-08 18:49:38.8467|INFO|12|Reviewed 14 blocks out of 64 
2021-03-08 18:49:38.8467|INFO|12|Reviewed 15 blocks out of 64 
2021-03-08 18:49:38.8467|INFO|12|Reviewed 16 blocks out of 64 
2021-03-08 18:49:38.8467|INFO|12|Reviewed 17 blocks out of 64 
2021-03-08 18:49:38.8467|INFO|12|Reviewed 18 blocks out of 64 
2021-03-08 18:49:38.8467|INFO|12|Reviewed 19 blocks out of 64 
2021-03-08 18:49:38.8467|INFO|12|Reviewed 20 blocks out of 64 
2021-03-08 18:49:38.8467|INFO|12|Reviewed 21 blocks out of 64 
2021-03-08 18:49:38.8467|INFO|12|Reviewed 22 blocks out of 64 
2021-03-08 18:49:38.8467|INFO|12|Reviewed 23 blocks out of 64 
2021-03-08 18:49:38.8467|INFO|12|Reviewed 24 blocks out of 64 
2021-03-08 18:49:38.8467|INFO|12|Reviewed 25 blocks out of 64 
2021-03-08 18:49:38.8467|INFO|12|Reviewed 26 blocks out of 64 
2021-03-08 18:49:38.8467|INFO|12|Reviewed 27 blocks out of 64 
2021-03-08 18:49:38.8467|INFO|12|Reviewed 28 blocks out of 64 
2021-03-08 18:49:38.8467|INFO|12|Reviewed 29 blocks out of 64 
2021-03-08 18:49:38.8467|INFO|12|Reviewed 30 blocks out of 64 
2021-03-08 18:49:38.8467|INFO|12|Reviewed 31 blocks out of 64 
2021-03-08 18:49:38.8467|INFO|12|Reviewed 32 blocks out of 64 
2021-03-08 18:49:38.8467|INFO|12|Reviewed 33 blocks out of 64 
2021-03-08 18:49:38.8467|INFO|12|Reviewed 34 blocks out of 64 
2021-03-08 18:49:38.8467|INFO|12|Reviewed 35 blocks out of 64 
2021-03-08 18:49:38.8467|INFO|12|Reviewed 36 blocks out of 64 
2021-03-08 18:49:38.8467|INFO|12|Reviewed 37 blocks out of 64 
2021-03-08 18:49:38.8467|INFO|12|Reviewed 38 blocks out of 64 
2021-03-08 18:49:38.8467|INFO|12|Reviewed 39 blocks out of 64 
2021-03-08 18:49:38.8467|INFO|12|Reviewed 40 blocks out of 64 
2021-03-08 18:49:38.8467|INFO|12|Reviewed 41 blocks out of 64 
2021-03-08 18:49:38.8467|INFO|12|Reviewed 42 blocks out of 64 
2021-03-08 18:49:38.8467|INFO|12|Reviewed 43 blocks out of 64 
2021-03-08 18:49:38.8467|INFO|12|Reviewed 44 blocks out of 64 
2021-03-08 18:49:38.8467|INFO|12|Reviewed 45 blocks out of 64 
2021-03-08 18:49:38.8467|INFO|12|Reviewed 46 blocks out of 64 
2021-03-08 18:49:38.8467|INFO|12|Reviewed 47 blocks out of 64 
2021-03-08 18:49:38.8467|INFO|12|Reviewed 48 blocks out of 64 
2021-03-08 18:49:38.8467|INFO|12|Reviewed 49 blocks out of 64 
2021-03-08 18:49:38.8467|INFO|12|Reviewed 50 blocks out of 64 
2021-03-08 18:49:38.8467|INFO|12|Reviewed 51 blocks out of 64 
2021-03-08 18:49:38.8467|INFO|12|Reviewed 52 blocks out of 64 
2021-03-08 18:49:38.8467|INFO|12|Reviewed 53 blocks out of 64 
2021-03-08 18:49:38.8467|INFO|12|Reviewed 54 blocks out of 64 
2021-03-08 18:49:38.8467|INFO|12|Reviewed 55 blocks out of 64 
2021-03-08 18:49:38.8467|INFO|12|Reviewed 56 blocks out of 64 
2021-03-08 18:49:38.8467|INFO|12|Reviewed 57 blocks out of 64 
2021-03-08 18:49:38.8467|INFO|12|Reviewed 58 blocks out of 64 
2021-03-08 18:49:38.8467|INFO|12|Reviewed 59 blocks out of 64 
2021-03-08 18:49:38.8467|INFO|12|Reviewed 60 blocks out of 64 
2021-03-08 18:49:38.8467|INFO|12|Reviewed 61 blocks out of 64 
2021-03-08 18:49:38.8467|INFO|12|Reviewed 62 blocks out of 64 
2021-03-08 18:49:38.8467|INFO|12|Reviewed 63 blocks out of 64 
2021-03-08 18:49:38.8517|INFO|12|Numbers resolved, level = Max(14909243, 14909307), header = Max(14909243, 14909307), body = Max(14909243, 14909307) 
2021-03-08 18:49:38.9576|INFO|18|Loaded 0 static nodes from file: /media/daniel/samsung970pro/nethermind/Data/static-nodes.json 
2021-03-08 18:49:38.9834|INFO|18|Using pivot 14820000 and barrier 1 in bodies sync 
2021-03-08 18:49:38.9834|INFO|18|Using pivot 14820000 and barrier 1 in receipts sync 
2021-03-08 18:49:38.9882|INFO|18|Now syncing nodes starting from root of block 0 
2021-03-08 18:49:38.9952|INFO|12|HeadersSyncDispatcher has finished work. 
2021-03-08 18:49:38.9952|INFO|12|Fast blocks headers task completed. 
2021-03-08 18:49:39.0019|INFO|18|Grafana / Prometheus metrics are disabled in configuration 
2021-03-08 18:49:39.1303|INFO|21|
======================== Nethermind initialization completed ========================
Version      : 1.10.27-0-26b6754b2-20210308/X64-Linux/5.0.3
This node    : enode://483b15708c80a71494f6dde75b5e1f4094556f272bf5d3811c5ba15dba0864b17db2299bef70485a234f08f61dfffe076db85ecf2b3f966432b89e1b6b649576@my-ip:30303
RPC modules  : Admin, Debug, Eth, Net, Parity, Personal, Proof, Subscribe, Trace, TxPool, Web3
Node address : 0x44335d498f2f5c28aed3d0fce2326572aaa38ac2 (do not use as an account)
Mem est tx   :     41MB
Mem est DB   :   2310MB
JSON RPC     : http://127.0.0.1:8545
Genesis hash : 0x4f1dd23188aab3a76b463e4af801b52b1248ef073c648cbdc4c9333d3da79756
External IP  : my-ip
Ethereum     : tcp://my-ip:30303
Discovery    : udp://my-ip:30303
Chainspec    : chainspec/xdai.json
Chain ID     : 100
Chain head   : 14909243 (0x94a633...173fa6)
===================================================================================== 
2021-03-08 18:49:39.3389|INFO|4|Connected to 0 bootnodes, 6 trusted/persisted nodes 
2021-03-08 18:49:39.8599|INFO|15|Peers | with known best block: 0 | all: 0 | 
2021-03-08 18:49:39.8599|INFO|15|Waiting for peers... 1s 
2021-03-08 18:49:39.8621|INFO|15|Sync peers - Initialized: 0 | All: 0 | Max: 50 
2021-03-08 18:49:40.8633|INFO|94|Waiting for peers... 2s 
2021-03-08 18:49:40.8633|INFO|14|Changing state Disconnected to Full at processed:14909243|state:14909307|block:14909307|header:14909307|peer block:14909344 
2021-03-08 18:49:41.3558|INFO|25|Discovered new block 14909345 18:49:40 (0x1d9365...025453), tx count: 6 miner 0x657ea4a9572dfdbfd95899eada0f6197211527be, sent by [Peer|eth64|14909345|OpenEthereum/v3.0.1-stable-8ca8089-20200601/x86_64-unknown-linux-gnu/rustc1.43.1|143.198.4.136:30303], with AuRa step 323045156 
2021-03-08 18:49:41.3902|INFO|24|Rerunning block after reorg or pruning: 14909307 (0x27d1637799d5b69f8a238b76e4daaa4d7713f0659a6dd48a4a12f0e800300ca6) 
2021-03-08 18:49:41.3902|INFO|24|Rerunning block after reorg or pruning: 14909306 (0x83a32821c76f02307f2458e2d58331f2a4e1b2dc0d72a44cd0eb4fb0f936f9be) 
2021-03-08 18:49:41.3902|INFO|24|Rerunning block after reorg or pruning: 14909305 (0x04f7d36b5251081555430c2e77ac6c3096bb37e424afceb7c1264516d1f24f03) 
2021-03-08 18:49:41.3902|INFO|24|Rerunning block after reorg or pruning: 14909304 (0x7e4b3dcdf615031d56213608dec11c6bfeb1597bdc44e128637443476b15634e) 
2021-03-08 18:49:41.3902|INFO|24|Rerunning block after reorg or pruning: 14909303 (0x6af95a5f6898c8d6f3dcf656a576fa71c903c7f73fb4e29a71d0b3f9e99981fe) 
2021-03-08 18:49:41.3902|INFO|24|Rerunning block after reorg or pruning: 14909302 (0x5a42e884e0a838b8aa3b41b47b0ab6892b65b492564bc9b543fce44a54ec79ba) 
2021-03-08 18:49:41.3902|INFO|24|Rerunning block after reorg or pruning: 14909301 (0xef11011078b34d28650da198eb502d34458df1987333ce4e4e8f213ac20c27bf) 
2021-03-08 18:49:41.3902|INFO|24|Rerunning block after reorg or pruning: 14909300 (0xd46af42d186152a045a18da595f8d55936ea22d9dea59cd9231cdba19a00483d) 
2021-03-08 18:49:41.3902|INFO|24|Rerunning block after reorg or pruning: 14909299 (0x4a047a5693f96261d73d1086e997fbd4741b210cd6e4291e0263bf9bde74aa0c) 
2021-03-08 18:49:41.3902|INFO|24|Rerunning block after reorg or pruning: 14909298 (0xfc67bd03aceb620fd05a342939d185ac4aeb07b0de7d5747e1cb5974e623b90e) 
2021-03-08 18:49:41.3902|INFO|24|Rerunning block after reorg or pruning: 14909297 (0x9ab78e33cdb3075a327784cafd7d26297532d7411b6eaaf4449d6d527fcf25d0) 
2021-03-08 18:49:41.3902|INFO|24|Rerunning block after reorg or pruning: 14909296 (0xbf8845d5f4a34b02ec6ac0175712582e56ba6437a9652686ee4e8b17ad0340f5) 
2021-03-08 18:49:41.3902|INFO|24|Rerunning block after reorg or pruning: 14909295 (0xcce6569245a061b5bad27f24f6af538b8f28ad727a20f85b894f36528a7a7ea0) 
2021-03-08 18:49:41.3902|INFO|24|Rerunning block after reorg or pruning: 14909294 (0xeb5e189afdd2792464db28da4b56b07b155e33b027224a6afb07ade58983c3c7) 
2021-03-08 18:49:41.3902|INFO|24|Rerunning block after reorg or pruning: 14909293 (0x1633155d444902c249604b94d859c8ff4aaba1a47bae6c9d33a40e9c830d32cd) 
2021-03-08 18:49:41.3902|INFO|24|Rerunning block after reorg or pruning: 14909292 (0x7c9ee4b3de0dede92703dcfedb75df197d35536b50a99a30f3a01fdf7af4990a) 
2021-03-08 18:49:41.3902|INFO|24|Rerunning block after reorg or pruning: 14909291 (0xe8c6152f848f9ea360bdad950160ffbaaf0fd4d1960330ec8b4d6c1aa237cdef) 
2021-03-08 18:49:41.3902|INFO|24|Rerunning block after reorg or pruning: 14909290 (0x8094bd5213605514a8f93c5d9cb5c8a3bd2376c9b1307475c35e94c545f7658a) 
2021-03-08 18:49:41.3902|INFO|24|Rerunning block after reorg or pruning: 14909289 (0x7ebc4be63505ca9b690a4f9a7aa1689dc6318617c8ea1c94aa0dfabc2044b649) 
2021-03-08 18:49:41.3902|INFO|24|Rerunning block after reorg or pruning: 14909288 (0x20da5ffb67be209189fc76fecce565e74dc48b61eb2dc380b540cefe626a0b06) 
2021-03-08 18:49:41.3902|INFO|24|Rerunning block after reorg or pruning: 14909287 (0xd51a53aff4c3b68e1998bfbd0356e2410a93f93b5e614943eaf7cdb2182a5fdf) 
2021-03-08 18:49:41.3902|INFO|24|Rerunning block after reorg or pruning: 14909286 (0xbdbcab8c7410d90349bdafeb89ace37d068bc11d95315c9b5f0698b15d742812) 
2021-03-08 18:49:41.3902|INFO|24|Rerunning block after reorg or pruning: 14909285 (0xaf47904fc431a20477a3d3f602dcc9b8860f93d01cf9712faa7b48ffbeab531a) 
2021-03-08 18:49:41.3902|INFO|24|Rerunning block after reorg or pruning: 14909284 (0x81f371e6d18d23c8cae6deb587c6801a9c494e896cc555646f42502bad96606b) 
2021-03-08 18:49:41.3902|INFO|24|Rerunning block after reorg or pruning: 14909283 (0x1b71f3329a4bb70042b6a514cff2ab73d859f21decb10ea9699d4765d0430d3f) 
2021-03-08 18:49:41.3902|INFO|24|Rerunning block after reorg or pruning: 14909282 (0x5803bde04be67cd733f0e4d9b6a243c3cb3dc096c8fd109a00e63afeb011768a) 
2021-03-08 18:49:41.3902|INFO|24|Rerunning block after reorg or pruning: 14909281 (0x5318fb292f53e37ef7d024f20af0c3569c4158de9df23b6cb170baf4c7a14c06) 
2021-03-08 18:49:41.3902|INFO|24|Rerunning block after reorg or pruning: 14909280 (0xd1f9ad43349fa4ed080796603dd806005078515536140d9ad0fa1b6f149e4d78) 
2021-03-08 18:49:41.3902|INFO|24|Rerunning block after reorg or pruning: 14909279 (0xd5fc04fd93d5e91c66e729c0625f6c4f9b2f115f709949cd8925a33334682251) 
2021-03-08 18:49:41.3902|INFO|24|Rerunning block after reorg or pruning: 14909278 (0xe83500350f7ffcbf487183af2fe048c7bc00db6e932edc8d8735b7553bc6c056) 
2021-03-08 18:49:41.3902|INFO|24|Rerunning block after reorg or pruning: 14909277 (0x0b9c5080ec601358cd2c981ea50bb0a21977a128029edffdc3c783b5e748e02e) 
2021-03-08 18:49:41.3902|INFO|24|Rerunning block after reorg or pruning: 14909276 (0x71f1d62fe37a286b2b9a2364da916208f0477dedabc15596876afcc04f4ee837) 
2021-03-08 18:49:41.3902|INFO|24|Rerunning block after reorg or pruning: 14909275 (0xcfbd2f71b4d5238034c3ceb1b57bc9eb07c00575ddd7e650bbb6624ea1dd6fd2) 
2021-03-08 18:49:41.3902|INFO|24|Rerunning block after reorg or pruning: 14909274 (0x3ff501201140b4ebed3c101505897dbbd06e5cb405fab1abd6407d940ebe197d) 
2021-03-08 18:49:41.3902|INFO|24|Rerunning block after reorg or pruning: 14909273 (0x1ab066e27a220a277daf9758c5e48b1ff4a4302458b23c5791a0bc873fa39cd2) 
2021-03-08 18:49:41.3902|INFO|24|Rerunning block after reorg or pruning: 14909272 (0x04e67ff6cb7fd535d77aed6f65a01ecc14da3e6c96878e650d7a1aa8f87d0e83) 
2021-03-08 18:49:41.3902|INFO|24|Rerunning block after reorg or pruning: 14909271 (0xb8e6f37e1a79a79e85adcf23eca198ed7a59bff98d641f29dd0c60d81fc3e4bd) 
2021-03-08 18:49:41.3902|INFO|24|Rerunning block after reorg or pruning: 14909270 (0x1c9ace455480b3f1e2723b808c85ed8110f7d600201dc9e1d40602273071a90f) 
2021-03-08 18:49:41.3902|INFO|24|Rerunning block after reorg or pruning: 14909269 (0xc456241ed2ef8a94c9b93c97ec123e5b03ba05f1a8e38c86f0be0bf3004be66d) 
2021-03-08 18:49:41.3902|INFO|24|Rerunning block after reorg or pruning: 14909268 (0x174efebedbbf1cc5b8a5822fb1836d35dd82e2a35bbacab5765b504155ca7c3e) 
2021-03-08 18:49:41.3902|INFO|24|Rerunning block after reorg or pruning: 14909267 (0x0aaf54bb1f8997fb9ee59a932f979739a78793c6644e903cc5a038e22d248fb4) 
2021-03-08 18:49:41.3902|INFO|24|Rerunning block after reorg or pruning: 14909266 (0x93871f9dadded28b0f574c5872bdda7919d70af1ef0b60c3155fd58534598268) 
2021-03-08 18:49:41.3902|INFO|24|Rerunning block after reorg or pruning: 14909265 (0x5b98fb4c648be27ff0b4e34718e73962ede0cfcfe8d4a389e6c653523272b029) 
2021-03-08 18:49:41.3902|INFO|24|Rerunning block after reorg or pruning: 14909264 (0xfacdb9e913581723bf7a0219f07932f8444a6616111db6191aeaddb7e19450b2) 
2021-03-08 18:49:41.3902|INFO|24|Rerunning block after reorg or pruning: 14909263 (0x08acdc30b8e7c2eea4c5ded558d2131ad52cadeac48bb60d786eae3d8f2fe48e) 
2021-03-08 18:49:41.3902|INFO|24|Rerunning block after reorg or pruning: 14909262 (0x9caa97b3251c80ffeddb27d73fd508e9fb31595e34d580a3f2a3b7576fec47db) 
2021-03-08 18:49:41.3902|INFO|24|Rerunning block after reorg or pruning: 14909261 (0xf6da3eecf68a10f77ae7fb5e870d2f0e4b04696b9d58b681d4b9dacad4580a18) 
2021-03-08 18:49:41.3902|INFO|24|Rerunning block after reorg or pruning: 14909260 (0xc6a36bc754a257e847c73b7ce74f25e8999b990c625591a65c6680e7ab15f062) 
2021-03-08 18:49:41.3902|INFO|24|Rerunning block after reorg or pruning: 14909259 (0xfbde2808bd5fb8ec1cf24eaa235efb04bad70a8c4410a24ad61161c91cb2cc7f) 
2021-03-08 18:49:41.3902|INFO|24|Rerunning block after reorg or pruning: 14909258 (0xdd73c9853d50a1885f695634c1e2f1087c7f3ff724f52544866fb30c301ea3f6) 
2021-03-08 18:49:41.3902|INFO|24|Rerunning block after reorg or pruning: 14909257 (0x949b6e8d5b06e6dbcaab35a71370aa6e502566e9d920d320091d33c9bfbbf01a) 
2021-03-08 18:49:41.3902|INFO|24|Rerunning block after reorg or pruning: 14909256 (0x7c6b88c16d904e27f3035c2c6049c084dd40fffdc3baa781c7107854c5f8359f) 
2021-03-08 18:49:41.3902|INFO|24|Rerunning block after reorg or pruning: 14909255 (0x360856b02bd58387e582c4c651e1ab3ce21f1583983cb8084c465cac7c8be26c) 
2021-03-08 18:49:41.3902|INFO|24|Rerunning block after reorg or pruning: 14909254 (0x1db07eb9f56120fcd45d6b2da46cc158834c53cd0e583ceedf31496d0ef41f30) 
2021-03-08 18:49:41.3902|INFO|24|Rerunning block after reorg or pruning: 14909253 (0x7921b96ca1699188a56f0c47822fa4be2a61f091b60f431995106627c13a1857) 
2021-03-08 18:49:41.3902|INFO|24|Rerunning block after reorg or pruning: 14909252 (0x951fef4f45bd305598ffe7a16073dd8ba72484bc401da1ff9c455e794cac1851) 
2021-03-08 18:49:41.3902|INFO|24|Rerunning block after reorg or pruning: 14909251 (0x324b7ae5218c76e1402c814cecfe34bdbc7d702a941b94f3034be3d0644e3ed8) 
2021-03-08 18:49:41.3902|INFO|24|Rerunning block after reorg or pruning: 14909250 (0x0b504a093ab58e14fdf722b323760bb535d7b80100d8487b7b41e69adf5b9078) 
2021-03-08 18:49:41.3902|INFO|24|Rerunning block after reorg or pruning: 14909249 (0xbe13f6825b9914da2a71cc30eca69ae7d747dff7c7eacebe1f7368f37d0ee04e) 
2021-03-08 18:49:41.3902|INFO|24|Rerunning block after reorg or pruning: 14909248 (0x453cdc5c2ab262b3468fa8cd3ec2be60306208c5dbc3f3c96cb4623c706796b7) 
2021-03-08 18:49:41.3902|INFO|24|Rerunning block after reorg or pruning: 14909247 (0x420299a91610d4254abe5f111424146decf2df729ad4d2564fab0e8e9df21592) 
2021-03-08 18:49:41.3902|INFO|24|Rerunning block after reorg or pruning: 14909246 (0x4ca0231c258e8f7734b97ef71a738e86dc6860084b8bf494f13ae4d3e4f9cd41) 
2021-03-08 18:49:41.3902|INFO|24|Rerunning block after reorg or pruning: 14909245 (0x912181ef6ec57a34f1e093a286fb533471ae9f7f9f8ef1eb1b2b4af65dc51de0) 
2021-03-08 18:49:41.3902|INFO|24|Rerunning block after reorg or pruning: 14909244 (0xa08930a4082d66cad8dbad94076f8b08c3c47eba9813db9fefcd16afc6918ea0) 
2021-03-08 18:49:41.4170|INFO|24|Processing part of a long blocks branch 0/65 
2021-03-08 18:49:41.4222|INFO|24|Current contract validators (16): [0x9233042b8e9e03d5dc6454bbbe5aee83818ff103, 0x6dc0c0be4c8b2dfe750156dc7d59faabfb5b923d, 0x9e41ba620feba8198369c26351063b26ec5b7c9e, 0xa13d45301207711b7c0328c6b2b64862abfe9b7a, 0x657ea4a9572dfdbfd95899eada0f6197211527be, 0x06d563905b085a6b3070c819bdb96a44e5665005, 0xdb1c683758f493cef2e7089a3640502ab306322a, 0x657e832b1a67cdef9e117afd2f419387259fa93e, 0x10aae121b3c62f3dafec9cc46c27b4c1dfe4a835, 0x1438087186fdbfd4c256fa2df446921e30e54df8, 0x0000999dc55126ca626c20377f0045946db69b6e, 0x9488f50c33e9616ee3b5b09cd3a9c603a108db4a, 0x1a740616e96e07d86203707c1619d9871614922a, 0x642c40173134f6e457a62d4c2033259433a53e8c, 0x35770ef700ff88d5f650597068e3aaf051f3d5a4, 0xb76756f95a9fb6ff9ad3e6cb41b734c1bd805103]. 
2021-03-08 18:49:41.5004|WARN|24|Saving reorg boundary 14909243 
2021-03-08 18:49:41.6243|INFO|24|Processing part of a long blocks branch 8/65 
2021-03-08 18:49:41.7428|INFO|24|Processing part of a long blocks branch 16/65 
2021-03-08 18:49:41.8707|INFO|24|Processing part of a long blocks branch 24/65 
2021-03-08 18:49:41.9994|INFO|24|Processing part of a long blocks branch 32/65 
2021-03-08 18:49:42.0855|INFO|24|Processing part of a long blocks branch 40/65 
2021-03-08 18:49:42.2512|INFO|24|Processing part of a long blocks branch 48/65 
2021-03-08 18:49:42.3741|INFO|24|Processing part of a long blocks branch 56/65 
2021-03-08 18:49:42.5115|INFO|24|Processing part of a long blocks branch 64/65 
2021-03-08 18:49:42.5515|INFO|24|Processed   14909308 |    3,711ms, mgasps    0.52 total    0.52, tps    3.23 total    3.23, bps    0.00 total    0.00, recv queue 0, proc queue 36 
2021-03-08 18:49:45.6673|INFO|25|Discovered new block 14909346 18:49:45 (0x136ce5...b1924d), tx count: 1 miner 0x06d563905b085a6b3070c819bdb96a44e5665005, sent by [Peer|eth64|14909346|OpenEthereum/v3.0.1-stable-8ca8089-20200601/x86_64-unknown-linux-gnu/rustc1.43.1|168.119.79.89:20303], with AuRa step 323045157 
2021-03-08 18:49:45.6673|INFO|24|Processed   14909346 |    3,118ms, mgasps   42.59 total   19.73, tps  233.78 total  108.50, bps   12.19 total    5.56, recv queue 0, proc queue 0 
2021-03-08 18:49:50.9444|INFO|25|Discovered new block 14909347 18:49:50 (0x84b61e...163a5e), tx count: 5 miner 0xdb1c683758f493cef2e7089a3640502ab306322a, sent by [Peer|eth64|14909347|OpenEthereum/xdai-friends/v3.0.1-stable-8ca8089-20200601/x86_64-unknown-linux-gnu/rustc1.43.1|95.168.163.52:30303], with AuRa step 323045158 
2021-03-08 18:49:50.9563|INFO|24|Processed   14909347 |    5,287ms, mgasps    0.29 total   11.24, tps    1.13 total   61.65, bps    0.19 total    3.22, recv queue 0, proc queue 0 
2021-03-08 18:49:54.8634|INFO|137|Peers | with known best block: 33 | all: 33 | 
2021-03-08 18:49:55.3474|INFO|25|Discovered new block 14909348 18:49:55 (0x7927f4...10ad15), tx count: 11 miner 0x657e832b1a67cdef9e117afd2f419387259fa93e, sent by [Peer|eth64|14909348|OpenEthereum/v3.0.1-stable-8ca8089-20200601/x86_64-alpine-linux-musl/rustc1.43.1|37.120.245.149:30303], with AuRa step 323045159 
2021-03-08 18:49:55.3723|INFO|24|Processed   14909348 |    4,416ms, mgasps    0.91 total    8.48, tps    3.62 total   46.15, bps    0.23 total    2.42, recv queue 0, proc queue 0 
2021-03-08 18:50:00.3687|INFO|25|Discovered new block 14909349 18:50:00 (0x7f220a...ecdba0), tx count: 5 miner 0x10aae121b3c62f3dafec9cc46c27b4c1dfe4a835, sent by [Peer|eth63|14909349|Parity-Ethereum/v2.7.2-posdao-stable-2ea4265-20200226/x86_64-unknown-linux-gnu/rustc1.41.0|116.203.39.113:30303], with AuRa step 323045160 
2021-03-08 18:50:00.3814|INFO|24|Processed   14909349 |    5,009ms, mgasps    0.83 total    6.70, tps    3.19 total   36.16, bps    0.20 total    1.90, recv queue 0, proc queue 0 
2021-03-08 18:50:05.7093|INFO|25|Discovered new block 14909350 18:50:05 (0x3a8f35...9aaa10), tx count: 5 miner 0x1438087186fdbfd4c256fa2df446921e30e54df8, sent by [Peer|eth64|14909350|OpenEthereum/xdai-friends/v3.0.1-stable-8ca8089-20200601/x86_64-unknown-linux-gnu/rustc1.43.1|95.168.163.52:30303], with AuRa step 323045161 
2021-03-08 18:50:05.7163|INFO|24|Processed   14909350 |    5,335ms, mgasps    0.43 total    5.46, tps    1.87 total   29.36, bps    0.19 total    1.56, recv queue 0, proc queue 0 
2021-03-08 18:50:07.2784|INFO|103|Initializing WebSockets for client: ''. 

Still works?

2021-03-08 18:50:07.8581|INFO|17|Executing JSON RPC call eth_getLogs with params [{
  "fromBlock": "0xbf2f32",
  "toBlock": "latest",
  "topics": [
    "0x358ba8f768af134eb5af120e9a61dc1ef29b29f597f047b555fc3675064a0342",
    "0x000000000000000000000000de374ece6fa50e781e81aac78e811b33d16912c7"
  ],
  "address": "0x29b9a7fbb8995b2423a71cc17cf9810798f6c543"
}] 

Yay!

Stopping the node:

2021-03-08 19:13:47.2175|INFO|99|Persisting trie... 
2021-03-08 19:13:47.2212|WARN|99|Saving reorg boundary 14909544 
2021-03-08 19:13:47.2212|INFO|99|Stopping session monitor... 
2021-03-08 19:13:47.2234|INFO|99|Stopping discovery app... 
2021-03-08 19:13:47.2308|INFO|99|Stopping discovery udp channel 
2021-03-08 19:13:47.2348|INFO|99|Stopping block producer... 
2021-03-08 19:13:47.2348|INFO|99|Stopping sync peer pool... 
2021-03-08 19:13:47.2348|INFO|152|Discovery shutdown complete.. please wait for all components to close 
2021-03-08 19:13:47.2391|INFO|99|Stopping peer manager... 
2021-03-08 19:13:47.2391|INFO|99|Peer Manager shutdown complete.. please wait for all components to close 
2021-03-08 19:13:47.2391|INFO|99|Stopping synchronizer... 
2021-03-08 19:13:47.2430|INFO|104|State sync task completed. 
2021-03-08 19:13:47.2430|INFO|146|Fast sync blocks downloader task completed. 
2021-03-08 19:13:47.2430|INFO|99|Stopping blockchain processor... 
2021-03-08 19:13:47.2430|INFO|104|Fast blocks receipts task completed. 
2021-03-08 19:13:47.2430|INFO|146|Fast blocks bodies task completed. 
2021-03-08 19:13:47.2430|INFO|152|Full sync block downloader task completed. 
2021-03-08 19:13:47.2462|INFO|99|Stopping rlpx peer... 
2021-03-08 19:13:47.2462|INFO|146|Blockchain Processor shutdown complete.. please wait for all components to close 
2021-03-08 19:13:47.3590|INFO|92|Peer update loop canceled. 
2021-03-08 19:13:52.2520|INFO|18|Local peer shutdown complete.. please wait for all components to close 
2021-03-08 19:13:52.2520|INFO|18|Disposing plugin Clique... 
2021-03-08 19:13:52.2520|INFO|18|Disposing plugin Ethash... 
2021-03-08 19:13:52.2520|INFO|18|Disposing plugin NethDev... 
2021-03-08 19:13:52.2520|INFO|18|Disposing plugin HealthChecks... 
2021-03-08 19:13:52.2520|INFO|18|Disposing Nethermind.Core.Reactive+AnonymousDisposable... 
2021-03-08 19:13:52.2562|INFO|18|Disposing Nethermind.Synchronization.SyncServer... 
2021-03-08 19:13:52.2562|INFO|18|Disposing Nethermind.Synchronization.Synchronizer... 
2021-03-08 19:13:52.2562|INFO|18|Disposing Nethermind.Consensus.AuRa.AuRaMultiSyncModeSelector... 
2021-03-08 19:13:52.2562|INFO|18|Disposing Nethermind.Synchronization.Peers.SyncPeerPool... 
2021-03-08 19:13:52.2562|INFO|18|Disposing MultiValidator [ ReportingContractBasedValidator [ ContractBasedValidator ] ]... 
2021-03-08 19:13:52.2562|INFO|18|Disposing Nethermind.Blockchain.ReceiptCanonicalityMonitor... 
2021-03-08 19:13:52.2562|INFO|18|Disposing Nethermind.Blockchain.OnChainTxWatcher... 
2021-03-08 19:13:52.2562|INFO|18|Disposing Nethermind.Trie.Pruning.TrieStore... 
2021-03-08 19:13:52.2562|INFO|18|Disposing Nethermind.Db.Blooms.BloomStorage... 
2021-03-08 19:13:52.2583|INFO|18|Disposing Nethermind.Crypto.CryptoRandom... 
2021-03-08 19:13:52.2583|INFO|18|Closing DBs... 
2021-03-08 19:13:52.2583|INFO|18|Disposing DB Witness 
2021-03-08 19:13:52.2583|INFO|18|Disposing DB Pendingtxs 
2021-03-08 19:13:52.2877|INFO|18|Disposing DB Blocks 
2021-03-08 19:13:52.3031|INFO|18|Disposing DB CanonicalHashTrie 
2021-03-08 19:13:52.3031|INFO|18|Disposing DB Receipts 
2021-03-08 19:13:52.3073|INFO|18|Disposing DB Bloom 
2021-03-08 19:13:52.3192|INFO|18|Disposing DB State 
2021-03-08 19:13:52.4109|INFO|18|Disposing DB Headers 
2021-03-08 19:13:52.4245|INFO|18|Disposing DB BlockInfos 
2021-03-08 19:13:52.4373|INFO|18|Disposing DB Code 
2021-03-08 19:13:52.4487|INFO|18|All DBs closed. 
2021-03-08 19:13:52.4487|INFO|18|Ethereum shutdown complete... please wait for all components to close 
2021-03-08 19:13:52.4487|INFO|18|All done, goodbye! 

So then, let's try again with 64 GB. Worked before so why not:

2021-03-08 19:14:00.9871|INFO|21|Setting up memory allowances 
2021-03-08 19:14:00.9938|INFO|21|  memory hint:        64000MB 
2021-03-08 19:14:00.9938|INFO|21|  general memory:     32MB 
2021-03-08 19:14:00.9940|INFO|21|  peers memory:       50MB 
2021-03-08 19:14:00.9940|INFO|21|  Netty memory:       268MB 
2021-03-08 19:14:00.9940|INFO|21|  mempool memory:     74MB 
2021-03-08 19:14:00.9940|INFO|21|  fast blocks memory: 1000MB 
2021-03-08 19:14:00.9940|INFO|21|  trie memory:        12515MB 
2021-03-08 19:14:00.9940|INFO|21|  DB memory:          50060MB 
2021-03-08 19:14:01.0226|INFO|20|Using http://ipv4.icanhazip.com to get external ip 
2021-03-08 19:14:01.3872|WARN|19|Start block loaded from reorg boundary - 14909544 - 14909544 (0x76dd31...9061f2) 
2021-03-08 19:14:01.3943|INFO|19|Numbers resolved, level = Max(14909544, 14909608), header = Max(14909544, 14909608), body = Max(14909544, 14909608) 
2021-03-08 19:14:01.3943|INFO|19|Block tree initialized, last processed is 14909544 (0x76dd31...9061f2), best queued is 14909608, best known is 14909608, lowest inserted header 1, body 0 
2021-03-08 19:14:01.3995|INFO|19|Initializing 4 plugins 
2021-03-08 19:14:01.3995|INFO|19|  Clique by Nethermind 
2021-03-08 19:14:01.3995|INFO|19|  Clique by Nethermind initialized in 0ms 
2021-03-08 19:14:01.3995|INFO|19|  Ethash by Nethermind 
2021-03-08 19:14:01.3995|INFO|19|  Ethash by Nethermind initialized in 0ms 
2021-03-08 19:14:01.3995|INFO|19|  NethDev by Nethermind 
2021-03-08 19:14:01.3995|INFO|19|  NethDev by Nethermind initialized in 0ms 
2021-03-08 19:14:01.3995|INFO|19|  HealthChecks by Nethermind 
2021-03-08 19:14:01.3995|INFO|19|  HealthChecks by Nethermind initialized in 1ms 
2021-03-08 19:14:01.5145|ERROR|7|Step InitializeBlockchainAuRa failed after 112ms System.IndexOutOfRangeException: Index was outside the bounds of the array.
   at Nethermind.Core.Caching.LruCache`2.NodeRemove(Int32 i, Boolean cleanValues)
   at Nethermind.Core.Caching.LruCache`2.Replace(TKey key, TValue value)
   at Nethermind.Core.Caching.LruCache`2.Set(TKey key, TValue val)
   at Nethermind.Trie.CachingStore.get_Item(Byte[] key)
   at Nethermind.Trie.Pruning.TrieStore.LoadRlp(Keccak keccak)
   at Nethermind.Trie.Pruning.ReadOnlyTrieStore.LoadRlp(Keccak hash)
   at Nethermind.Trie.TrieNode.ResolveNode(ITrieNodeResolver tree)
   at Nethermind.Trie.PatriciaTree.Run(Span`1 updatePath, Int32 nibblesCount, Byte[] updateValue, Boolean isUpdate, Boolean ignoreMissingDelete, Keccak startRootHash)
   at Nethermind.Trie.PatriciaTree.Get(Span`1 rawKey, Keccak rootHash)
   at Nethermind.State.StateTree.Get(Address address, Keccak rootHash)
   at Nethermind.State.StateProvider.GetState(Address address)
   at Nethermind.State.StateProvider.GetAndAddToCache(Address address)
   at Nethermind.State.StateProvider.GetThroughCache(Address address)
   at Nethermind.State.StateProvider.GetCodeHash(Address address)
   at Nethermind.State.IReadOnlyStateProvider.IsContract(Address address)
   at Nethermind.Evm.ReadOnlyTransactionProcessor.IsContractDeployed(Address address)
   at Nethermind.Blockchain.Contracts.Contract.ConstantContract.CallRaw(CallInfo callInfo, IReadOnlyTransactionProcessor readOnlyTransactionProcessor)
   at Nethermind.Blockchain.Contracts.Contract.ConstantContract.Call(CallInfo callInfo)
   at Nethermind.Blockchain.Contracts.Contract.ConstantContract.Call[T](CallInfo callInfo)
   at Nethermind.Blockchain.Contracts.Contract.ConstantContract.Call[T](BlockHeader parentHeader, String functionName, Address sender, Object[] arguments)
   at Nethermind.Consensus.AuRa.Contracts.ValidatorContract.GetValidators(BlockHeader parentHeader)
   at Nethermind.Consensus.AuRa.Validators.ContractBasedValidator.LoadValidatorsFromContract(BlockHeader parentHeader)
   at Nethermind.Consensus.AuRa.Validators.ContractBasedValidator.SetFinalizationManager(IBlockFinalizationManager finalizationManager, BlockHeader parentHeader)
   at Nethermind.Consensus.AuRa.Validators.ContractBasedValidator..ctor(IValidatorContract validatorContract, IBlockTree blockTree, IReceiptFinder receiptFinder, IValidatorStore validatorStore, IValidSealerStrategy validSealerStrategy, IBlockFinalizationManager finalizationManager, BlockHeader parentHeader, ILogManager logManager, Int64 startBlockNumber, Int64 posdaoTransition, Boolean forSealing)
   at Nethermind.Consensus.AuRa.AuRaValidatorFactory.<CreateValidatorProcessor>g__GetContractBasedValidator|17_2(<>c__DisplayClass17_0& )
   at Nethermind.Consensus.AuRa.AuRaValidatorFactory.CreateValidatorProcessor(Validator validator, BlockHeader parentHeader, Nullable`1 startBlock)
   at Nethermind.Consensus.AuRa.Validators.MultiValidator.CreateValidator(Int64 finalizedAtBlockNumber, Validator validatorPrototype, BlockHeader parentHeader)
   at Nethermind.Consensus.AuRa.Validators.MultiValidator.SetCurrentValidator(Int64 finalizedAtBlockNumber, Validator validatorPrototype, BlockHeader parentHeader)
   at Nethermind.Consensus.AuRa.Validators.MultiValidator.SetCurrentValidator(KeyValuePair`2 validatorInfo, BlockHeader parentHeader)
   at Nethermind.Consensus.AuRa.Validators.MultiValidator.InitCurrentValidator(Int64 blockNumber, BlockHeader parentHeader)
   at Nethermind.Consensus.AuRa.Validators.MultiValidator.SetFinalizationManager(IBlockFinalizationManager finalizationManager, BlockHeader parentHeader)
   at Nethermind.Consensus.AuRa.Validators.MultiValidator..ctor(Validator validator, IAuRaValidatorFactory validatorFactory, IBlockTree blockTree, IValidatorStore validatorStore, IBlockFinalizationManager finalizationManager, BlockHeader parentHeader, ILogManager logManager, Boolean forSealing)
   at Nethermind.Consensus.AuRa.AuRaValidatorFactory.CreateValidatorProcessor(Validator validator, BlockHeader parentHeader, Nullable`1 startBlock)
   at Nethermind.Runner.Ethereum.Steps.InitializeBlockchainAuRa.CreateAuRaValidator(IBlockProcessor processor, IReadOnlyTxProcessorSource readOnlyTxProcessorSource)
   at Nethermind.Runner.Ethereum.Steps.InitializeBlockchainAuRa.CreateBlockProcessor()
   at Nethermind.Runner.Ethereum.Steps.InitializeBlockchain.InitBlockchain()
   at Nethermind.Runner.Ethereum.Steps.InitializeBlockchain.Execute(CancellationToken _)
2021-03-08 19:14:01.5239|ERROR|6|Error during ethereum runner start Nethermind.Runner.Ethereum.Steps.StepDependencyException: BlockchainProcessor
   at Nethermind.Runner.Ethereum.Steps.StartBlockProcessor.Execute(CancellationToken _)
   at Nethermind.Runner.Ethereum.Steps.EthereumStepsManager.RunOneRoundOfInitialization(CancellationToken cancellationToken)
   at Nethermind.Runner.Ethereum.Steps.EthereumStepsManager.InitializeAll(CancellationToken cancellationToken)
   at Nethermind.Runner.Ethereum.EthereumRunner.Start(CancellationToken cancellationToken)
2021-03-08 19:14:06.4154|INFO|24|Closing, please wait until all functions are stopped properly... 
2021-03-08 19:14:06.4263|INFO|24|Persisting trie... 
2021-03-08 19:14:06.4306|INFO|24|Stopping session monitor... 
2021-03-08 19:14:06.4306|INFO|24|Stopping discovery app... 
2021-03-08 19:14:06.4306|INFO|24|Stopping block producer... 
2021-03-08 19:14:06.4306|INFO|24|Stopping sync peer pool... 
2021-03-08 19:14:06.4306|INFO|24|Stopping peer manager... 
2021-03-08 19:14:06.4341|INFO|24|Stopping synchronizer... 
2021-03-08 19:14:06.4341|INFO|24|Stopping blockchain processor... 
2021-03-08 19:14:06.4341|INFO|24|Stopping rlpx peer... 
2021-03-08 19:14:06.4341|INFO|24|Disposing plugin Clique... 
2021-03-08 19:14:06.4341|INFO|24|Disposing plugin Ethash... 
2021-03-08 19:14:06.4341|INFO|24|Disposing plugin NethDev... 
2021-03-08 19:14:06.4341|INFO|24|Disposing plugin HealthChecks... 
2021-03-08 19:14:06.4341|INFO|24|Disposing Nethermind.Blockchain.ReceiptCanonicalityMonitor... 
2021-03-08 19:14:06.4392|INFO|24|Disposing Nethermind.Blockchain.OnChainTxWatcher... 
2021-03-08 19:14:06.4392|INFO|24|Disposing Nethermind.Trie.Pruning.TrieStore... 
2021-03-08 19:14:06.4392|INFO|24|Disposing Nethermind.Db.Blooms.BloomStorage... 
2021-03-08 19:14:06.4392|INFO|24|Disposing Nethermind.Crypto.CryptoRandom... 
2021-03-08 19:14:06.4421|INFO|24|Closing DBs... 
2021-03-08 19:14:06.4421|INFO|24|Disposing DB Blocks 
2021-03-08 19:14:06.4792|INFO|24|Disposing DB Witness 
2021-03-08 19:14:06.4792|INFO|24|Disposing DB State 
2021-03-08 19:14:06.4873|INFO|24|Disposing DB Headers 
2021-03-08 19:14:06.4873|INFO|24|Disposing DB Pendingtxs 
2021-03-08 19:14:06.4873|INFO|24|Disposing DB BlockInfos 
2021-03-08 19:14:06.4913|INFO|24|Disposing DB Code 
2021-03-08 19:14:06.4913|INFO|24|Disposing DB Bloom 
2021-03-08 19:14:06.4913|INFO|24|Disposing DB Receipts 
2021-03-08 19:14:06.4983|INFO|24|Disposing DB CanonicalHashTrie 
2021-03-08 19:14:06.4983|INFO|24|All DBs closed. 
2021-03-08 19:14:06.4983|INFO|24|Ethereum shutdown complete... please wait for all components to close 
2021-03-08 19:14:06.4983|INFO|24|All done, goodbye! 

Nah. But events work. So I'm fine for now :)

tkstanczak commented 3 years ago

@jaensen we are fixing the LRU Cache in the today's release. Some other problems with AuRa block processing may persist.

Thank you for the logs - they will be very, very helpful in further investigation.

So to clarify:

this is fixed in the next release today:

2021-03-08 19:14:01.5145|ERROR|7|Step InitializeBlockchainAuRa failed after 112ms System.IndexOutOfRangeException: Index was outside the bounds of the array. at Nethermind.Core.Caching.LruCache2.NodeRemove(Int32 i, Boolean cleanValues) at Nethermind.Core.Caching.LruCache2.Replace(TKey key, TValue value) at Nethermind.Core.Caching.LruCache`2.Set(TKey key, TValue val)

tkstanczak commented 3 years ago

This is not fixed yet: 2021-03-08 18:34:50.4830|ERROR|103|BlockchainProcessor encountered an exception. Nethermind.Trie.TrieException: Node 0x91e29c5b95cdcd0738bacc882a5dbe6f2cd3f4aeca6b3c1baf576b1d6d4ef460 is missing from the DB at Nethermind.Trie.Pruning.TrieStore.LoadRlp(Keccak keccak) at Nethermind.Trie.Pruning.ReadOnlyTrieStore.LoadRlp(Keccak hash)

jaensen commented 3 years ago

Just tried the new release and can confirm that the LruCache issue is resolved. Also I synced again with a build of 1.10.31 and had no issues with missing nodes in the DB this time.

tkstanczak commented 3 years ago

Hi @jaensen - 1.10.41 brings more fixes related to the issues that you have reported

LukaszRozmej commented 3 years ago

@jaensen I belive everything is working fine now with never versions? Feel free to reopen if there is still some issue.