Crash while fetching transaction from objkt minter factory

[pravin-d]

Steps to reproduce: Run

dipdup init with the attached dipdup.yml file

comment out the origination related content in the file. Reason being is that origination operation takes a lot of time.

• comment line 32
• comment lines from 36 to 39 (both inclusive)

Run dipdup run What did you expect to happen: Should run without any error What actually happened: Results in a crash

Log

File "home/.pyenv/versions/myproject/bin/dipdup", line 8, in <module>
    sys.exit(cli())
  File "home/.pyenv/versions/3.10.7/envs/myproject/lib/python3.10/site-packages/asyncclick/core.py", line 1157, in __call__
    return anyio.run(self._main, main, args, kwargs, **opts)
  File "home/.pyenv/versions/3.10.7/envs/myproject/lib/python3.10/site-packages/anyio/_core/_eventloop.py", line 70, in run
    return asynclib.run(func, *args, **backend_options)
  File "home/.pyenv/versions/3.10.7/envs/myproject/lib/python3.10/site-packages/anyio/_backends/_asyncio.py", line 292, in run
    return native_run(wrapper(), debug=debug)
  File "home/.pyenv/versions/3.10.7/lib/python3.10/asyncio/runners.py", line 44, in run
    return loop.run_until_complete(main)
  File "home/.pyenv/versions/3.10.7/lib/python3.10/asyncio/base_events.py", line 646, in run_until_complete
    return future.result()
  File "home/.pyenv/versions/3.10.7/envs/myproject/lib/python3.10/site-packages/anyio/_backends/_asyncio.py", line 287, in wrapper
    return await func(*args)
  File "home/.pyenv/versions/3.10.7/envs/myproject/lib/python3.10/site-packages/asyncclick/core.py", line 1160, in _main
    return await main(*args, **kwargs)
  File "home/.pyenv/versions/3.10.7/envs/myproject/lib/python3.10/site-packages/asyncclick/core.py", line 1076, in main
    rv = await self.invoke(ctx)
  File "home/.pyenv/versions/3.10.7/envs/myproject/lib/python3.10/site-packages/asyncclick/core.py", line 1687, in invoke
    return await _process_result(await sub_ctx.command.invoke(sub_ctx))
  File "home/.pyenv/versions/3.10.7/envs/myproject/lib/python3.10/site-packages/asyncclick/core.py", line 1434, in invoke
    return await ctx.invoke(self.callback, **ctx.params)
  File "home/.pyenv/versions/3.10.7/envs/myproject/lib/python3.10/site-packages/asyncclick/core.py", line 780, in invoke
    rv = await rv
  File "home/.pyenv/versions/3.10.7/envs/myproject/lib/python3.10/site-packages/dipdup/cli.py", line 109, in wrapper
    await fn(*args, **kwargs)
  File "home/.pyenv/versions/3.10.7/envs/myproject/lib/python3.10/site-packages/dipdup/cli.py", line 309, in run
    await dipdup.run()
  File "home/.pyenv/versions/3.10.7/envs/myproject/lib/python3.10/site-packages/dipdup/dipdup.py", line 393, in run
    await gather(*tasks)
  File "home/.pyenv/versions/3.10.7/envs/myproject/lib/python3.10/site-packages/dipdup/dipdup.py", line 103, in run
    indexes_processed = await gather(*tasks)
  File "home/.pyenv/versions/3.10.7/envs/myproject/lib/python3.10/site-packages/dipdup/index.py", line 232, in process
    await self._synchronize(sync_level)
  File "home/.pyenv/versions/3.10.7/envs/myproject/lib/python3.10/site-packages/dipdup/index.py", line 371, in _synchronize
    await self._process_level_operations(operation_subgroups, sync_level)
  File "home/.pyenv/versions/3.10.7/envs/myproject/lib/python3.10/site-packages/dipdup/index.py", line 391, in _process_level_operations
    matched_handlers += await self._match_operation_subgroup(operation_subgroup)
  File "home/.pyenv/versions/3.10.7/envs/myproject/lib/python3.10/site-packages/dipdup/index.py", line 474, in _match_operation_subgroup
    args = await self._prepare_handler_args(handler_config, matched_operations)
  File "home/.pyenv/versions/3.10.7/envs/myproject/lib/python3.10/site-packages/dipdup/index.py", line 511, in _prepare_handler_args
    storage = deserialize_storage(operation_data, storage_type)
  File "home/.pyenv/versions/3.10.7/envs/myproject/lib/python3.10/site-packages/dipdup/datasources/tzkt/models.py", line 189, in deserialize_storage
    operation_data.storage = _process_storage(
  File "home/.pyenv/versions/3.10.7/envs/myproject/lib/python3.10/site-packages/dipdup/datasources/tzkt/models.py", line 173, in _process_storage
    value_type = get_dict_value_type(storage_type, key)
  File "home/.pyenv/versions/3.10.7/envs/myproject/lib/python3.10/site-packages/dipdup/datasources/tzkt/models.py", line 96, in get_dict_value_type
    raise KeyError(f'Field `{key}` not found in {dict_type}')
KeyError: "Field `mods` not found in <class 'refPrtcl.types.objkt_minting_factory.storage.ObjktMintingFactoryStorage'>"

dipdup.yml

spec_version: 1.2
package: refPrtcl

database:
  kind: sqlite
  path: demo_tzbtc.sqlite3

contracts:
  OBJKT_minting_factory:
    address: KT1Aq4wWmVanpQhq4TTfjZXB5AjFpx15iQMM
    typename: objkt_minting_factory

datasources:
  tzkt_mainnet:
    kind: tzkt
    url: ${TZKT_URL:-https://api.tzkt.io}

advanced:
  postpone_jobs: True
  rollback_depth: 4
  reindex:
    config_modified: ignore

indexes:
  objkt_minting_factory_mainnet:
    kind: operation
    datasource: tzkt_mainnet
    types:
      - transaction
      - origination
    contracts:
      - OBJKT_minting_factory
    handlers:
      - callback: on_objkt_minter_factory_originations
        pattern:
          - type: origination
            source: OBJKT_minting_factory
      - callback: on_mint_artist
        pattern:
          - type: transaction
            destination: OBJKT_minting_factory
            entrypoint: mint_artist

Environment

• Python version: 3.10.7
• DipDup version or commit hash: 6.1.3
• Database engine version:
• Hasura version:
• Are you in docker? No
• Can you reproduce this issue on master? Yes

[pravin-d]

The reason for the crash is that the storage that we get while fetching a transaction is different from storage that we get when fetching origination. Please check the following links Transaction Link Origination Link

[droserasprout]

Hey, @pravin-d, and sorry for the late reply. I was not sure what we should do with this one, keeping in mind current limitations.

Initial warmup taking so long is DipDup fetching originated contracts' addresses when using the source field. This filter implies being used within index factories to call ctx.add_index inside the origination callback. Child indexes then match their originations by the originated_contract filter. So, until indexing for contract families is not implemented, the only way to do what you're trying to achieve is to use index factories.

Codegen should fail early on such configurations; leaving this open.

[m-kus]

Similar to transactions, DipDup should also allow untyped originations in patterns if none of the originated_contract or similar_to fields are present. source has not to be used to codegen origination storage, as it is incorrect.