search

dipdup-io / dipdup

Modular framework for creating selective indexers and featureful backends for dapps
https://dipdup.io
MIT License
92 stars 52 forks source link

Getting level of big map batch must be higher than index state level error again #389

Closed Karantezsure closed 2 years ago

Karantezsure commented 2 years ago

Steps to reproduce:

What did you expect to happen:

What actually happened:

Environment

INFO     dipdup.index         assetMap_bigmap: 2434891: `on_asset_update` handler matched!
INFO     dipdup.index         assetMap_bigmap: 2434891: `on_asset_update` handler matched!
INFO     dipdup.index         assetMap_bigmap: 2434891: `on_asset_update` handler matched!
INFO     dipdup.tzkt          tzkt_mainnet: Realtime message received: head, DATA, 2434893 -> 2434894
INFO     dipdup.tzkt          tzkt_mainnet: Realtime message received: big_map, DATA, 2434893 -> 2434894
INFO     dipdup.tzkt          tzkt_mainnet: Realtime message received: head, DATA, 2434894 -> 2434895
INFO     dipdup.tzkt          tzkt_mainnet: Realtime message received: big_map, DATA, 2434894 -> 2434895
INFO     dipdup.tzkt          tzkt_mainnet: Realtime message received: head, DATA, 2434895 -> 2434896
INFO     dipdup.tzkt          tzkt_mainnet: Realtime message received: big_map, DATA, 2434895 -> 2434896
INFO     dipdup.tzkt          tzkt_mainnet: Realtime message received: head, DATA, 2434896 -> 2434897
INFO     dipdup.tzkt          tzkt_mainnet: Realtime message received: big_map, DATA, 2434896 -> 2434897
INFO     dipdup.tzkt          tzkt_mainnet: Realtime message received: head, DATA, 2434897 -> 2434898
INFO     dipdup.tzkt          tzkt_mainnet: Realtime message received: big_map, DATA, 2434897 -> 2434898
INFO     dipdup.tzkt          tzkt_mainnet: Realtime message received: head, DATA, 2434898 -> 2434899
INFO     dipdup.tzkt          tzkt_mainnet: Realtime message received: big_map, DATA, 2434898 -> 2434899
INFO     dipdup.tzkt          tzkt_mainnet: Realtime message received: head, DATA, 2434899 -> 2434900
INFO     dipdup.tzkt          tzkt_mainnet: Realtime message received: big_map, DATA, 2434899 -> 2434900
INFO     dipdup.tzkt          tzkt_mainnet: Realtime message received: head, DATA, 2434900 -> 2434901
INFO     dipdup.tzkt          tzkt_mainnet: Realtime message received: big_map, DATA, 2434900 -> 2434901
INFO     dipdup.tzkt          tzkt_mainnet: Realtime message received: head, DATA, 2434901 -> 2434902
INFO     dipdup.tzkt          tzkt_mainnet: Realtime message received: big_map, DATA, 2434901 -> 2434902
INFO     dipdup.tzkt          tzkt_mainnet: Realtime message received: head, DATA, 2434902 -> 2434903
INFO     dipdup.tzkt          tzkt_mainnet: Realtime message received: big_map, DATA, 2434902 -> 2434903
INFO     dipdup.tzkt          tzkt_mainnet: Realtime message received: head, DATA, 2434903 -> 2434904
INFO     dipdup.tzkt          tzkt_mainnet: Realtime message received: big_map, DATA, 2434903 -> 2434904
INFO     dipdup.tzkt          tzkt_mainnet: Realtime message received: head, DATA, 2434904 -> 2434905
INFO     dipdup.tzkt          tzkt_mainnet: Realtime message received: big_map, DATA, 2434904 -> 2434905
INFO     dipdup.tzkt          tzkt_mainnet: Realtime message received: head, DATA, 2434905 -> 2434906
INFO     dipdup.tzkt          tzkt_mainnet: Realtime message received: big_map, DATA, 2434905 -> 2434906
INFO     dipdup.tzkt          tzkt_mainnet: Realtime message received: operation, DATA, 2434891 -> 2434906
INFO     dipdup.tzkt          tzkt_mainnet: Realtime message received: head, DATA, 2434906 -> 2434907
INFO     dipdup.tzkt          tzkt_mainnet: Realtime message received: big_map, DATA, 2434906 -> 2434907
INFO     dipdup.tzkt          tzkt_mainnet: Realtime message received: head, DATA, 2434907 -> 2434908
INFO     dipdup.index         assetMap_bigmap: 2434906: `on_asset_update` handler matched!
INFO     dipdup.index         assetMap_bigmap: 2434906: `on_asset_update` handler matched!
INFO     dipdup.index         assetMap_bigmap: 2434906: `on_asset_update` handler matched!
INFO     dipdup.index         assetMap_bigmap: 2434906: `on_asset_update` handler matched!
INFO     dipdup.index         assetMap_bigmap: 2434906: `on_asset_update` handler matched!
INFO     dipdup.index         assetMap_bigmap: 2434906: `on_asset_update` handler matched!
INFO     dipdup.index         assetMap_bigmap: 2434906: `on_asset_update` handler matched!
INFO     dipdup.index         assetMap_bigmap: 2434906: `on_asset_update` handler matched!
INFO     dipdup.index         assetMap_bigmap: 2434906: `on_asset_update` handler matched!
INFO     dipdup.index         assetMap_bigmap: 2434906: `on_asset_update` handler matched!
INFO     dipdup.tzkt          tzkt_mainnet: Realtime message received: big_map, DATA, 2434907 -> 2434908
INFO     dipdup.tzkt          tzkt_mainnet: Realtime message received: head, DATA, 2434908 -> 2434909
INFO     dipdup.tzkt          tzkt_mainnet: Realtime message received: big_map, DATA, 2434908 -> 2434909
INFO     dipdup.tzkt          tzkt_mainnet: Realtime message received: head, DATA, 2434909 -> 2434910
INFO     dipdup.tzkt          tzkt_mainnet: Realtime message received: big_map, DATA, 2434909 -> 2434910
INFO     dipdup.tzkt          tzkt_mainnet: Realtime message received: head, DATA, 2434910 -> 2434911
INFO     dipdup.tzkt          tzkt_mainnet: Realtime message received: big_map, DATA, 2434910 -> 2434911
INFO     dipdup.tzkt          tzkt_mainnet: Realtime message received: head, DATA, 2434911 -> 2434912
INFO     dipdup.tzkt          tzkt_mainnet: Realtime message received: big_map, DATA, 2434911 -> 2434912
INFO     dipdup.tzkt          tzkt_mainnet: Realtime message received: head, DATA, 2434912 -> 2434913
INFO     dipdup.tzkt          tzkt_mainnet: Realtime message received: big_map, DATA, 2434912 -> 2434913
INFO     dipdup.tzkt          tzkt_mainnet: Realtime message received: head, DATA, 2434913 -> 2434914
INFO     dipdup.tzkt          tzkt_mainnet: Realtime message received: big_map, DATA, 2434913 -> 2434914
INFO     dipdup.tzkt          tzkt_mainnet: Realtime message received: head, DATA, 2434914 -> 2434915
INFO     dipdup.tzkt          tzkt_mainnet: Realtime message received: big_map, DATA, 2434914 -> 2434915
INFO     dipdup.tzkt          tzkt_mainnet: Realtime message received: head, DATA, 2434915 -> 2434916
INFO     dipdup.tzkt          tzkt_mainnet: Realtime message received: big_map, DATA, 2434915 -> 2434916
INFO     dipdup.tzkt          tzkt_mainnet: Realtime message received: head, DATA, 2434916 -> 2434917
INFO     dipdup.tzkt          tzkt_mainnet: Realtime message received: big_map, DATA, 2434916 -> 2434917
INFO     dipdup.tzkt          tzkt_mainnet: Realtime message received: head, DATA, 2434917 -> 2434918
INFO     dipdup.tzkt          tzkt_mainnet: Realtime message received: big_map, DATA, 2434917 -> 2434918
INFO     dipdup.tzkt          tzkt_mainnet: Realtime message received: head, DATA, 2434918 -> 2434919
INFO     dipdup.tzkt          tzkt_mainnet: Realtime message received: big_map, DATA, 2434918 -> 2434919
INFO     dipdup.tzkt          tzkt_mainnet: Realtime message received: head, DATA, 2434919 -> 2434920
INFO     dipdup.tzkt          tzkt_mainnet: Realtime message received: big_map, DATA, 2434919 -> 2434920
INFO     dipdup.tzkt          tzkt_mainnet: Realtime message received: head, DATA, 2434920 -> 2434921
INFO     dipdup.tzkt          tzkt_mainnet: Realtime message received: big_map, DATA, 2434920 -> 2434921
INFO     dipdup.tzkt          tzkt_mainnet: Realtime message received: head, DATA, 2434921 -> 2434922
INFO     dipdup.tzkt          tzkt_mainnet: Realtime message received: big_map, DATA, 2434921 -> 2434922
INFO     dipdup.tzkt          tzkt_mainnet: Realtime message received: operation, DATA, 2434906 -> 2434922
INFO     dipdup.tzkt          tzkt_mainnet: Realtime message received: head, DATA, 2434922 -> 2434923
INFO     dipdup.tzkt          tzkt_mainnet: Realtime message received: big_map, DATA, 2434922 -> 2434923
INFO     dipdup.tzkt          tzkt_mainnet: Realtime message received: head, DATA, 2434923 -> 2434924
INFO     dipdup.index         assetMap_bigmap: 2434922: `on_asset_update` handler matched!
INFO     dipdup.index         assetMap_bigmap: 2434922: `on_asset_update` handler matched!
INFO     dipdup.index         assetMap_bigmap: 2434922: `on_asset_update` handler matched!
INFO     dipdup.index         assetMap_bigmap: 2434922: `on_asset_update` handler matched!
INFO     dipdup.index         assetMap_bigmap: 2434922: `on_asset_update` handler matched!
INFO     dipdup.index         assetMap_bigmap: 2434922: `on_asset_update` handler matched!
INFO     dipdup.index         assetMap_bigmap: 2434922: `on_asset_update` handler matched!
INFO     dipdup.tzkt          tzkt_mainnet: Realtime message received: big_map, DATA, 2434923 -> 2434924
INFO     dipdup.tzkt          tzkt_mainnet: Realtime message received: head, DATA, 2434924 -> 2434925
INFO     dipdup.tzkt          tzkt_mainnet: Realtime message received: big_map, DATA, 2434924 -> 2434925
INFO     dipdup.tzkt          tzkt_mainnet: Realtime message received: head, DATA, 2434925 -> 2434926
INFO     dipdup.tzkt          tzkt_mainnet: Realtime message received: big_map, DATA, 2434925 -> 2434926
WARNING  pysignalr.transport  Connection closed: no close frame received or sent
INFO     pysignalr.transport  State change: connected -> reconnecting
INFO     dipdup.tzkt          tzkt_mainnet: Realtime connection lost
INFO     dipdup               Datasource disconnected, dropping realtime queues
INFO     dipdup.tzkt          tzkt_mainnet: Subscribing to 6 channels
INFO     pysignalr.transport  State change: reconnecting -> disconnected
INFO     dipdup.tzkt          tzkt_mainnet: Realtime connection lost
INFO     dipdup               Datasource disconnected, dropping realtime queues
INFO     pysignalr.transport  State change: disconnected -> connecting
INFO     pysignalr.transport  Performing negotiation, URL: `https://api.tzkt.io/v1/events/negotiate?id=6Mp6kuHZjJ0sR-eg4d0d-A`
INFO     pysignalr.transport  Negotiation completed
INFO     pysignalr.transport  Sending handshake to server
INFO     pysignalr.transport  Awaiting handshake from server
INFO     pysignalr.transport  State change: connecting -> connected
INFO     dipdup.tzkt          tzkt_mainnet: Realtime connection established
INFO     dipdup.tzkt          tzkt_mainnet: Subscribed to 6 channels
INFO     dipdup.index         tzbtc_holders_mainnet: Index is behind datasource level, syncing: 2434882 -> 2434927
INFO     dipdup.index         tzbtc_holders_mainnet: Synchronizing index to level 2434927
INFO     dipdup.index         ovens_bigmap: Index is behind datasource level, syncing: 2434924 -> 2434927
INFO     dipdup.index         ovens_bigmap: Synchronizing index to level 2434927
INFO     dipdup.index         assetMap_bigmap: Index is behind datasource level, syncing: 2434924 -> 2434927
INFO     dipdup.index         assetMap_bigmap: Synchronizing index to level 2434927
INFO     dipdup.index         tzbtc_holders_mainnet: Fetching operations from level 2434883 to 2434927
INFO     dipdup.index         ovens_bigmap: Fetching big map diffs from level 2434925 to 2434927
INFO     dipdup.index         assetMap_bigmap: Fetching big map diffs from level 2434925 to 2434927
INFO     dipdup.tzkt          tzkt_mainnet: Realtime message received: head, DATA, 2434926 -> 2434927
INFO     dipdup.index         ovens_bigmap: Index is synchronized to level 2434927
INFO     dipdup.index         assetMap_bigmap: Index is synchronized to level 2434927
INFO     dipdup.tzkt          tzkt_mainnet: Realtime message received: big_map, DATA, 2434926 -> 2434927
INFO     dipdup.index         tzbtc_holders_mainnet: Index is synchronized to level 2434927
INFO     apscheduler.scheduler Scheduler has been shut down
Traceback (most recent call last):
  File "/usr/local/bin/dipdup", line 8, in <module>
    sys.exit(cli())
  File "/usr/local/lib/python3.10/site-packages/asyncclick/core.py", line 1150, in __call__
    return anyio.run(self._main, main, args, kwargs, **({"backend":_anyio_backend} if _anyio_backend is not None else {}))
  File "/usr/local/lib/python3.10/site-packages/anyio/_core/_eventloop.py", line 70, in run
    return asynclib.run(func, *args, **backend_options)
  File "/usr/local/lib/python3.10/site-packages/anyio/_backends/_asyncio.py", line 292, in run
    return native_run(wrapper(), debug=debug)
  File "/usr/local/lib/python3.10/asyncio/runners.py", line 44, in run
    return loop.run_until_complete(main)
  File "/usr/local/lib/python3.10/asyncio/base_events.py", line 646, in run_until_complete
    return future.result()
  File "/usr/local/lib/python3.10/site-packages/anyio/_backends/_asyncio.py", line 287, in wrapper
    return await func(*args)
  File "/usr/local/lib/python3.10/site-packages/asyncclick/core.py", line 1153, in _main
    return await main(*args, **kwargs)
  File "/usr/local/lib/python3.10/site-packages/asyncclick/core.py", line 1074, in main
    rv = await self.invoke(ctx)
  File "/usr/local/lib/python3.10/site-packages/asyncclick/core.py", line 1684, in invoke
    return await _process_result(await sub_ctx.command.invoke(sub_ctx))
  File "/usr/local/lib/python3.10/site-packages/asyncclick/core.py", line 1420, in invoke
    return await ctx.invoke(self.callback, **ctx.params)
  File "/usr/local/lib/python3.10/site-packages/asyncclick/core.py", line 774, in invoke
    rv = await rv
  File "/usr/local/lib/python3.10/site-packages/dipdup/cli.py", line 89, in wrapper
    await fn(*args, **kwargs)
  File "/usr/local/lib/python3.10/site-packages/dipdup/cli.py", line 229, in run
    await dipdup.run()
  File "/usr/local/lib/python3.10/site-packages/dipdup/dipdup.py", line 423, in run
    await gather(*tasks)
  File "/usr/local/lib/python3.10/site-packages/dipdup/dipdup.py", line 85, in run
    await gather(*tasks)
  File "/usr/local/lib/python3.10/site-packages/dipdup/dipdup.py", line 113, in _run
    await gather(*tasks)
  File "/usr/local/lib/python3.10/site-packages/dipdup/index.py", line 233, in process
    await self._process_queue()
  File "/usr/local/lib/python3.10/site-packages/dipdup/index.py", line 661, in _process_queue
    await self._process_level_big_maps(big_maps)
  File "/usr/local/lib/python3.10/site-packages/dipdup/index.py", line 752, in _process_level_big_maps
    raise RuntimeError(f'Level of big map batch must be higher than index state level: {batch_level} <= {index_level}')
RuntimeError: Level of big map batch must be higher than index state level: 2434925 <= 2434927
________________________________________________________________________________

An unexpected error has occurred!

Please file a bug report at https://github.com/dipdup-net/dipdup/issues

Skipping virtualenv creation, as specified in config file.
droserasprout commented 2 years ago

Hi! I've made another attempt to fix this bug in 5.1.6, now filtering queue directly within the index. Should work fine. Please, update DipDup.

Karantezsure commented 2 years ago

ok

droserasprout commented 2 years ago

Closing as duplicate. Feel free to reopen if the problem persists.