`ExplorerQuery.ListBlocks()` may fails, while `BlockChain<T>.Append()` executing.

[longfin]

(Note: Confirmation via regression needed)

Summary

When appending a new block to the chain, some API(like BlocksQuery in GraphQL)s might fail as below.

{
  "errors": [
    {
      "message": "Error trying to resolve field 'blocks'.",
      "locations": [
        {
          "line": 1,
          "column": 18
        }
      ],
      "path": [
        "blockQuery",
        "blocks"
      ],
      "extensions": {
        "code": "KEY_NOT_FOUND",
        "codes": [
          "KEY_NOT_FOUND"
        ]
      }
    }
  ],
  "data": {
    "blockQuery": null
  },
  "extensions": {
    "tracing": {
      "version": 1,
      "startTime": "2023-05-02T06:30:42.1517439Z",
      "endTime": "2023-05-02T06:30:43.0376972Z",
      "duration": 885953400,
      "parsing": {
        "startOffset": 600,
        "duration": 20500
      },
      "validation": {
        "startOffset": 21500,
        "duration": 95000
      },
      "execution": {
        "resolvers": []
      }
    }
  }
}

Analysis

I guess that this issue is a sort of "dirty reads". Basically, we guarantee the chain-level isolation, using BlockChain<T>._rwlock for .Append(). but ExplorerQuery.ListBlocks() requires lock only for BlockChain<T>.Tip.

How to reproduce

TBD

Possible approaches

If this problem is really due to "dirty-reads", there are currently two considerable approchies to fix it.

• Secure isolation via locks: ExplorerQuery.ListBlock(): force that APIs that traverse the chain index, such as ExplorerQuery.ListBlock() require BlockChain<T>._rwlocks.
  • Expose the lock of BlockChain<T> externally or modify it to use APIs of BlockChain<T> instead of IStore.
• Adjust index write timing: move IStore.AppendIndex() in .Append() to run after any other ops finish.

[longfin]

Detailed stack trace about this issue pointed a little bit different part than imagine 😓

System.Collections.Generic.KeyNotFoundException: The given key was not present in the dictionary
  at LruCacheNet.LruCache`2.Get(TKey key)
  at LruCacheNet.LruCache`2.get_Item(TKey key)
  at Libplanet.RocksDBStore.RocksDBStore.IterateIndexes(Guid chainId, Int32 offset, Nullable`1 limit) in /app/Lib9c/.Libplanet/Libplanet.RocksDBStore/RocksDBStore.cs:line 544
  at Libplanet.Explorer.Queries.ExplorerQuery`1.ListBlocks(Boolean desc, Int64 offset, Nullable`1 limit, Boolean excludeEmptyTxs, Nullable`1 miner)+MoveNext() in /app/Lib9c/.Libplanet/Libplanet.Explorer/Queries/ExplorerQuery.cs:line 102
  at GraphQL.Execution.ExecutionStrategy.SetArrayItemNodes(ExecutionContext context, ArrayExecutionNode parent) in /_/src/GraphQL/Execution/ExecutionStrategy.cs:line 372
  at GraphQL.Execution.ExecutionStrategy.CompleteNode(ExecutionContext context, ExecutionNode node) in /_/src/GraphQL/Execution/ExecutionStrategy.cs:line 505

• KEY_NOT_FOUND (perhaps KeyNotFoundException) had been thrown for LRUCache in RocksDBStore while running RockDBStore.AppendIndex() and RockDBStore.IterateIndexes() simultaneously.
• RockDBStore.AppendIndex() seemed to been called from BlockChain<T>.Append() and RockDBStore.IterateIndexes() from ExplorerQuery.ListBlocks(). in this case, adjusting RockDBStore.AppendIndex() position can't be solution.
• I'll fix caching logic to avoid KeyNotFoundException as first aid, but we may need to consider other perspectives too.

[longfin]

Related PR: #1676