Error when loading last N blocks using JSONRPCReorganisationMonitor

[bryaan]

How do you only load the last N blocks? This code wants to loads only last 5 blocks i believe, however it errors when adding new blocks.

    reorg_mon = JSONRPCReorganisationMonitor(web3, check_depth=30)

    reorg_mon.load_initial_block_headers(block_count=5)

    while True:
        try:
            # Figure out the next good unscanned block range,
            # and fetch block headers and timestamps for this block range
            chain_reorg_resolution = reorg_mon.update_chain()

            if chain_reorg_resolution.reorg_detected:
                logger.info(f"Chain reorganisation data updated: {chain_reorg_resolution}")

            # Read specified events in block range
            for log_result in read_events(
                web3,
                start_block=chain_reorg_resolution.latest_block_with_good_data + 1,
                end_block=chain_reorg_resolution.last_live_block,
                filter=filter,
                notify=None,
                chunk_size=100,
                context=token_cache,
                extract_timestamps=None,
                reorg_mon=reorg_mon,
            ):
               pass

INFO:eth_defi.event_reader.reorganisation_monitor:figure_reorganisation_and_new_blocks(), range 17,285,423 - 17,285,443, last block we have is 17,285,443, check depth is 20
ERROR: LoadError: Python: AssertionError: Blocks must be added in order. Last block we have: 17285443, the new record is: BlockHeader(block_number=17285423, block_hash='0x8d481922bd607150c9f3299004a113e44955327770ab04ed10de115e2172d6fe', timestamp=1684400615)
Python stacktrace:
 [1] add_block
   @ eth_defi.event_reader.reorganisation_monitor ~/Library/Caches/pypoetry/virtualenvs/cryptopy-5siZoxZ4-py3.10/lib/python3.10/site-packages/eth_defi/event_reader/reorganisation_monitor.py:324
 [2] figure_reorganisation_and_new_blocks
   @ eth_defi.event_reader.reorganisation_monitor ~/Library/Caches/pypoetry/virtualenvs/cryptopy-5siZoxZ4-py3.10/lib/python3.10/site-packages/eth_defi/event_reader/reorganisation_monitor.py:396
 [3] update_chain

[hieuh25]

How do you only load the last N blocks? This code wants to loads only last 5 blocks i believe, however it errors when adding new blocks.

    reorg_mon = JSONRPCReorganisationMonitor(web3, check_depth=30)

    reorg_mon.load_initial_block_headers(block_count=5)

    while True:
        try:
            # Figure out the next good unscanned block range,
            # and fetch block headers and timestamps for this block range
            chain_reorg_resolution = reorg_mon.update_chain()

            if chain_reorg_resolution.reorg_detected:
                logger.info(f"Chain reorganisation data updated: {chain_reorg_resolution}")

            # Read specified events in block range
            for log_result in read_events(
                web3,
                start_block=chain_reorg_resolution.latest_block_with_good_data + 1,
                end_block=chain_reorg_resolution.last_live_block,
                filter=filter,
                notify=None,
                chunk_size=100,
                context=token_cache,
                extract_timestamps=None,
                reorg_mon=reorg_mon,
            ):
               pass

INFO:eth_defi.event_reader.reorganisation_monitor:figure_reorganisation_and_new_blocks(), range 17,285,423 - 17,285,443, last block we have is 17,285,443, check depth is 20
ERROR: LoadError: Python: AssertionError: Blocks must be added in order. Last block we have: 17285443, the new record is: BlockHeader(block_number=17285423, block_hash='0x8d481922bd607150c9f3299004a113e44955327770ab04ed10de115e2172d6fe', timestamp=1684400615)
Python stacktrace:
 [1] add_block
   @ eth_defi.event_reader.reorganisation_monitor ~/Library/Caches/pypoetry/virtualenvs/cryptopy-5siZoxZ4-py3.10/lib/python3.10/site-packages/eth_defi/event_reader/reorganisation_monitor.py:324
 [2] figure_reorganisation_and_new_blocks
   @ eth_defi.event_reader.reorganisation_monitor ~/Library/Caches/pypoetry/virtualenvs/cryptopy-5siZoxZ4-py3.10/lib/python3.10/site-packages/eth_defi/event_reader/reorganisation_monitor.py:396
 [3] update_chain

Thank you for reporting this to us!

The reason your script failed was the check_depth is 30, which is higher than initial loaded block count 5. So what happened was:

• The reorganisation monitor loaded 5 block headers into it's block map
• The while loop started, it ran reorg_mon.update_chain(), which tried to check the specified depth of 30 last read blocks: https://github.com/tradingstrategy-ai/web3-ethereum-defi/blob/d47802caa1f1579d497cd87bb400b503828299bb/eth_defi/event_reader/reorganisation_monitor.py#L383-L386
• The block range it tried to read didn't exist in block map, so it tried to add and failed

This is more or less undocumented feature, so I will either add a warning for this particular case or add some assert to prevent it to happen

[miohtama]

I just close this as there has not been any further comments.