Closed longfin closed 1 year ago
Detailed stack trace about this issue pointed a little bit different part than imagine 😓
System.Collections.Generic.KeyNotFoundException: The given key was not present in the dictionary
at LruCacheNet.LruCache`2.Get(TKey key)
at LruCacheNet.LruCache`2.get_Item(TKey key)
at Libplanet.RocksDBStore.RocksDBStore.IterateIndexes(Guid chainId, Int32 offset, Nullable`1 limit) in /app/Lib9c/.Libplanet/Libplanet.RocksDBStore/RocksDBStore.cs:line 544
at Libplanet.Explorer.Queries.ExplorerQuery`1.ListBlocks(Boolean desc, Int64 offset, Nullable`1 limit, Boolean excludeEmptyTxs, Nullable`1 miner)+MoveNext() in /app/Lib9c/.Libplanet/Libplanet.Explorer/Queries/ExplorerQuery.cs:line 102
at GraphQL.Execution.ExecutionStrategy.SetArrayItemNodes(ExecutionContext context, ArrayExecutionNode parent) in /_/src/GraphQL/Execution/ExecutionStrategy.cs:line 372
at GraphQL.Execution.ExecutionStrategy.CompleteNode(ExecutionContext context, ExecutionNode node) in /_/src/GraphQL/Execution/ExecutionStrategy.cs:line 505
KEY_NOT_FOUND
(perhaps KeyNotFoundException
) had been thrown for LRUCache
in RocksDBStore
while running RockDBStore.AppendIndex()
and RockDBStore.IterateIndexes()
simultaneously. RockDBStore.AppendIndex()
seemed to been called from BlockChain<T>.Append()
and RockDBStore.IterateIndexes()
from ExplorerQuery.ListBlocks()
. in this case, adjusting RockDBStore.AppendIndex()
position can't be solution. KeyNotFoundException
as first aid, but we may need to consider other perspectives too.Related PR: #1676
(Note: Confirmation via regression needed)
Summary
When appending a new block to the chain, some API(like
BlocksQuery
in GraphQL)s might fail as below.Analysis
I guess that this issue is a sort of "dirty reads". Basically, we guarantee the chain-level isolation, using
BlockChain<T>._rwlock
for.Append()
. butExplorerQuery.ListBlocks()
requires lock only forBlockChain<T>.Tip
.How to reproduce
TBD
Possible approaches
If this problem is really due to "dirty-reads", there are currently two considerable approchies to fix it.
ExplorerQuery.ListBlock()
: force that APIs that traverse the chain index, such asExplorerQuery.ListBlock()
requireBlockChain<T>._rwlocks
.BlockChain<T>
externally or modify it to use APIs ofBlockChain<T>
instead ofIStore
.IStore.AppendIndex()
in.Append()
to run after any other ops finish.