search

dipdup-io / dipdup

Modular framework for creating selective indexers and featureful backends for dapps
https://dipdup.io
MIT License
92 stars 52 forks source link

Crash while fetching data from Tzkt #316

Closed pravin-d closed 2 years ago

pravin-d commented 2 years ago

Steps to reproduce:

What did you expect to happen:

What actually happened:

Environment

Error callstack

indexer_1 | WARNING dipdup.http HTTP request attempt 1/inf failed: [Errno 1] [SSL: APPLICATION_DATA_AFTER_CLOSE_NOTIFY] application data after close notify (_ssl.c:2672) indexer_1 | INFO dipdup.http Waiting 1.0 seconds before retry indexer_1 | Traceback (most recent call last): indexer_1 | File "/usr/local/bin/dipdup", line 5, in indexer_1 | cli() indexer_1 | File "/usr/local/lib/python3.10/site-packages/asyncclick/core.py", line 1150, in call indexer_1 | return anyio.run(self._main, main, args, kwargs, ({"backend":_anyio_backend} if _anyio_backend is not None else {})) indexer_1 | File "/usr/local/lib/python3.10/site-packages/anyio/_core/_eventloop.py", line 56, in run indexer_1 | return asynclib.run(func, *args, backend_options) indexer_1 | File "/usr/local/lib/python3.10/site-packages/anyio/_backends/_asyncio.py", line 233, in run indexer_1 | return native_run(wrapper(), debug=debug) indexer_1 | File "/usr/local/lib/python3.10/asyncio/runners.py", line 44, in run indexer_1 | return loop.run_until_complete(main) indexer_1 | File "/usr/local/lib/python3.10/asyncio/base_events.py", line 646, in run_until_complete indexer_1 | return future.result() indexer_1 | File "/usr/local/lib/python3.10/site-packages/anyio/_backends/_asyncio.py", line 228, in wrapper indexer_1 | return await func(args) indexer_1 | File "/usr/local/lib/python3.10/site-packages/asyncclick/core.py", line 1153, in _main indexer_1 | return await main(args, kwargs) indexer_1 | File "/usr/local/lib/python3.10/site-packages/asyncclick/core.py", line 1074, in main indexer_1 | rv = await self.invoke(ctx) indexer_1 | File "/usr/local/lib/python3.10/site-packages/asyncclick/core.py", line 1684, in invoke indexer_1 | return await _process_result(await sub_ctx.command.invoke(sub_ctx)) indexer_1 | File "/usr/local/lib/python3.10/site-packages/asyncclick/core.py", line 1420, in invoke indexer_1 | return await ctx.invoke(self.callback, *ctx.params) indexer_1 | File "/usr/local/lib/python3.10/site-packages/asyncclick/core.py", line 774, in invoke indexer_1 | rv = await rv indexer_1 | File "/home/dipdup/source/src/dipdup/cli.py", line 87, in wrapper indexer_1 | await fn(args, kwargs) indexer_1 | File "/home/dipdup/source/src/dipdup/cli.py", line 205, in run indexer_1 | await dipdup.run() indexer_1 | File "/home/dipdup/source/src/dipdup/dipdup.py", line 372, in run indexer_1 | await gather(tasks) indexer_1 | File "/home/dipdup/source/src/dipdup/dipdup.py", line 85, in run indexer_1 | await gather(tasks) indexer_1 | File "/home/dipdup/source/src/dipdup/dipdup.py", line 113, in _run indexer_1 | await gather(tasks) indexer_1 | File "/home/dipdup/source/src/dipdup/index.py", line 204, in process indexer_1 | await self._synchronize(sync_level) indexer_1 | File "/home/dipdup/source/src/dipdup/index.py", line 326, in _synchronize indexer_1 | await self._process_level_operations(operation_subgroups) indexer_1 | File "/home/dipdup/source/src/dipdup/index.py", line 370, in _process_level_operations indexer_1 | matched_handlers += await self._match_operation_subgroup(operation_subgroup) indexer_1 | File "/home/dipdup/source/src/dipdup/index.py", line 451, in _match_operation_subgroup indexer_1 | args = await self._prepare_handler_args(handler_config, matched_operations) indexer_1 | File "/home/dipdup/source/src/dipdup/index.py", line 488, in _prepare_handler_args indexer_1 | storage = deserialize_storage(operation_data, storage_type) indexer_1 | File "/home/dipdup/source/src/dipdup/datasources/tzkt/models.py", line 182, in deserialize_storage indexer_1 | operation_data.storage = _process_storage( indexer_1 | File "/home/dipdup/source/src/dipdup/datasources/tzkt/models.py", line 167, in _process_storage indexer_1 | storage[key] = _process_storage(value, value_type, bigmap_diffs) indexer_1 | File "/home/dipdup/source/src/dipdup/datasources/tzkt/models.py", line 166, in _process_storage indexer_1 | value_type = get_dict_value_type(storage_type, key) # type: ignore indexer_1 | File "/home/dipdup/source/src/dipdup/datasources/tzkt/models.py", line 79, in get_dict_value_type indexer_1 | fields = dict_type.fields indexer_1 | File "/usr/local/lib/python3.10/typing.py", line 984, in getattr indexer_1 | raise AttributeError(attr) indexer_1 | AttributeError: fields indexer_1 | ____ indexer_1 | indexer_1 | An unexpected error has occurred! indexer_1 | {context} indexer_1 | Please file a bug report at https://github.com/dipdup-net/dipdup/issues and attach the following: indexer_1 | indexer_1 | dipdup.yml config. Make sure to remove sensitive information. indexer_1 | * Reasonable amount of logs before the crash. indexer_1 |

droserasprout commented 2 years ago

Pravin, [4/27/22 6:04 PM] contract KT1Q82hNv6bG3PiDK5GKuxqmwc7F3jQC2PZ6 Hash: onnJXhXvYhP8JEU9GNRoq5TvGBwXFmmM9Kp3FggrDZ8otWGzMUi

Pravin, [4/27/22 6:05 PM] entrypoint: transfer

droserasprout commented 2 years ago

Can't reproduce with the following generic config:

spec_version: 1.2
package: test
contracts:
  test:
    typename: test
    address: KT1E8Qzgx3C5AAE4iGuXvqSQjdd21LK2aXAk
  test2:
    typename: test2
    address: KT1Q82hNv6bG3PiDK5GKuxqmwc7F3jQC2PZ6
datasources:
  tzkt:
    kind: tzkt
    url: https://api.tzkt.io/
indexes:
  test:
    kind: operation
    datasource: tzkt
    contracts:
      - test
      - test2
    handlers:
      - callback: on_test
        pattern:
          - destination: test
            entrypoint: cancel
          - destination: test2
            entrypoint: transfer
INFO     dipdup.index         test: Processing operations of level 2262035
INFO     dipdup.index         test: Processing operations of level 2279844
INFO     dipdup.index         test: Processing operations of level 2314771
INFO     dipdup.index         test: onnJXhXvYhP8JEU9GNRoq5TvGBwXFmmM9Kp3FggrDZ8otWGzMUi: `on_test` handler matched!
INFO     dipdup.index         test: Processing operations of level 2314829
INFO     dipdup.index         test: Index is synchronized to level 2322048
WARNING  dipdup.callback      Skipping SQL hook `on_synchronized`: not supported on SQLite
INFO     dipdup.tzkt          tzkt: Subscribing to 2 channels
INFO     dipdup               Spawning datasources
INFO     dipdup.tzkt          tzkt: Creating websocket client
INFO     dipdup.tzkt          tzkt: Establishing realtime connection
INFO     pysignalr.transport  State change: disconnected -> connecting
INFO     pysignalr.transport  Performing negotiation, URL: `https://api.tzkt.io/v1/events/negotiate`
INFO     pysignalr.transport  Negotiation completed
INFO     pysignalr.transport  Sending handshake to server
INFO     pysignalr.transport  Awaiting handshake from server
INFO     pysignalr.transport  State change: connecting -> connected
INFO     dipdup.tzkt          tzkt: Realtime connection established
INFO     dipdup.tzkt          tzkt: Subscribed to 2 channels
INFO     dipdup.tzkt          tzkt: Realtime message received: head, DATA, 2322048 -> 2322049
INFO     dipdup.tzkt          tzkt: Realtime message received: head, DATA, 2322049 -> 2322050
droserasprout commented 2 years ago

@pravin-d, please, run dipdup init --overwrite-types to ensure your codegened types are correct and try to run indexing again.