Duplicated transactions

[spilin]

There is an error in indexer that periodically appears when trying to parse 2 consecutive blocks, splitted by a skip block, that have similar identical transaction.

Indexing transaction 0xc9a8089a8328d11b3ef6d9e1b34deae2ad97e071ce6f5fffc95579fa4e3b1ecd at #75996231:0...
...
Indexing block #75996232...
Indexing transaction 0xc9a8089a8328d11b3ef6d9e1b34deae2ad97e071ce6f5fffc95579fa4e3b1ecd at #75996233:0...
indexTransaction error: duplicate key value violates unique constraint "transaction_hash_key"
    ....
    at TCP.onStreamRead (node:internal/stream_base_commons:199:23) {
  length: 273,
  severity: 'ERROR',
  code: '23505',
  detail: 'Key (hash)=(\\xc9a8089a8328d11b3ef6d9e1b34deae2ad97e071ce6f5fffc95579fa4e3b1ecd) already exists.',
  ...
  constraint: 'transaction_hash_key',
  file: 'nbtinsert.c',
  line: '649',
  routine: '_bt_check_unique'
}

To check blocks in questions: tx: 0xc9a8089a8328d11b3ef6d9e1b34deae2ad97e071ce6f5fffc95579fa4e3b1ecd http post https://rpc.testnet.near.org jsonrpc=2.0 id=dontcare method=block params:='{"block_id": 75996231}' http post https://rpc.testnet.near.org jsonrpc=2.0 id=dontcare method=block params:='{"block_id": 75996232}' - block mising http post https://rpc.testnet.near.org jsonrpc=2.0 id=dontcare method=block params:='{"block_id": 75996233}'

75996231 and 75996233 have identical chunks

Another example: http post https://rpc.testnet.near.org jsonrpc=2.0 id=dontcare method=block params:='{"block_id": 76060419}' http post https://rpc.testnet.near.org jsonrpc=2.0 id=dontcare method=block params:='{"block_id": 76060420}' - block mising http post https://rpc.testnet.near.org jsonrpc=2.0 id=dontcare method=block params:='{"block_id": 76060421}'

76060419 and 76060421 have identical chunks

[bowenwang1996]

Every block contains the same number of chunks. However, what you cares about is whether it contains a new chunk. You can checks this by looking at the chunk_mask in the block header. If for some shard it is false, then there is no new chunk for the shard in this block and it should be ignored for indexing purposes.

[mfornet]

@bowenwang1996 👍 I confirmed that the mask for the second block is all False.

@spilin @0x3bfc So the issue most likely will be while parsing a block, we iterate through the chunks and fetch all txs for this chunk, but if the mask is false for a particular chunk, then it will try to parse all txs for a previous block.