search

dipdup-io / dipdup

Modular framework for creating selective indexers and featureful backends for dapps
https://dipdup.io
MIT License
92 stars 52 forks source link

Datasources raise asyncio.TimeoutError on connection_timeout instead of retry #699

Closed TristanAllaire closed 1 year ago

TristanAllaire commented 1 year ago

Steps to reproduce: The indexer was working properly with Dipdup 6.5.5 but it started to return asyncio timeouts when I updated it to 6.5.6. Downgrading to 6.5.5 fixed the issue so this error seems related to the new dipdup version (6.5.6).

I saw this error when using the ctx.datasource.get_contract_summary built-in function. If other built-in functions use the asyncio library too, it could also happen but I couldn't test it.

What did you expect to happen: Get a contract summary.

What actually happened: The indexer crashed return an asyncio timeout error. Trace: 

Traceback (most recent call last):
  File "/usr/local/bin/dipdup", line 8, in <module>
    sys.exit(cli())
  File "/usr/local/lib/python3.10/site-packages/asyncclick/core.py", line 1157, in __call__
    return anyio.run(self._main, main, args, kwargs, **opts)
  File "/usr/local/lib/python3.10/site-packages/anyio/_core/_eventloop.py", line 70, in run
    return asynclib.run(func, *args, **backend_options)
  File "/usr/local/lib/python3.10/site-packages/anyio/_backends/_asyncio.py", line 292, in run
    return native_run(wrapper(), debug=debug)
  File "/usr/local/lib/python3.10/asyncio/runners.py", line 44, in run
    return loop.run_until_complete(main)
  File "/usr/local/lib/python3.10/asyncio/base_events.py", line 646, in run_until_complete
    return future.result()
  File "/usr/local/lib/python3.10/site-packages/anyio/_backends/_asyncio.py", line 287, in wrapper
    return await func(*args)
  File "/usr/local/lib/python3.10/site-packages/asyncclick/core.py", line 1160, in _main
    return await main(*args, **kwargs)
  File "/usr/local/lib/python3.10/site-packages/asyncclick/core.py", line 1076, in main
    rv = await self.invoke(ctx)
  File "/usr/local/lib/python3.10/site-packages/asyncclick/core.py", line 1687, in invoke
    return await _process_result(await sub_ctx.command.invoke(sub_ctx))
  File "/usr/local/lib/python3.10/site-packages/asyncclick/core.py", line 1434, in invoke
    return await ctx.invoke(self.callback, **ctx.params)
  File "/usr/local/lib/python3.10/site-packages/asyncclick/core.py", line 780, in invoke
    rv = await rv
  File "/usr/local/lib/python3.10/site-packages/dipdup/cli.py", line 66, in wrapper
    raise e
  File "/usr/local/lib/python3.10/site-packages/dipdup/cli.py", line 57, in wrapper
    await fn(ctx, *args, **kwargs)
  File "/usr/local/lib/python3.10/site-packages/dipdup/cli.py", line 205, in run
    await dipdup.run()
  File "/usr/local/lib/python3.10/site-packages/dipdup/dipdup.py", line 464, in run
    await gather(*tasks)
  File "/usr/local/lib/python3.10/site-packages/dipdup/dipdup.py", line 112, in run
    indexes_processed = await gather(*tasks)
  File "/usr/local/lib/python3.10/site-packages/dipdup/index.py", line 154, in process
    await self._synchronize(sync_level)
  File "/usr/local/lib/python3.10/site-packages/dipdup/indexes/operation/index.py", line 241, in _synchronize
    async for level, operations in fetcher.fetch_by_level():
  File "/usr/local/lib/python3.10/site-packages/dipdup/indexes/operation/fetcher.py", line 418, in fetch_by_level
    await min_channel.fetch()
  File "/usr/local/lib/python3.10/site-packages/dipdup/indexes/operation/fetcher.py", line 227, in fetch
    transactions = await self._datasource.get_transactions(
  File "/usr/local/lib/python3.10/site-packages/dipdup/datasources/tzkt/datasource.py", line 587, in get_transactions
    raw_transactions = await self.request(
  File "/usr/local/lib/python3.10/site-packages/dipdup/http.py", line 65, in request
    return await self._http.request(method, url, weight, **kwargs)
  File "/usr/local/lib/python3.10/site-packages/dipdup/http.py", line 258, in request
    return await self._retry_request(method, url, weight, **kwargs)
  File "/usr/local/lib/python3.10/site-packages/dipdup/http.py", line 146, in _retry_request
    return await self._request(
  File "/usr/local/lib/python3.10/site-packages/dipdup/http.py", line 198, in _request
    async with self._session.request(
  File "/usr/local/lib/python3.10/site-packages/aiohttp/client.py", line 1141, in __aenter__
    self._resp = await self._coro
  File "/usr/local/lib/python3.10/site-packages/aiohttp/client.py", line 560, in _request
    await resp.start(conn)
  File "/usr/local/lib/python3.10/site-packages/aiohttp/client_reqrep.py", line 894, in start
    with self._timer:
  File "/usr/local/lib/python3.10/site-packages/aiohttp/helpers.py", line 721, in __exit__
    raise asyncio.TimeoutError from None
asyncio.exceptions.TimeoutError
________________________________________________________________________________
An unexpected error has occurred! Most likely it's a framework bug.
Please, tell us about it: https://github.com/dipdup-io/dipdup/issues

Environment

droserasprout commented 1 year ago

Hi @TristanAllaire! 6.5.6 release have changed the way http.connection_timeout config option behaves. Now it applies to the total request time instead of just establishing connection. Ideally, this option should be called total_timeout or something, but I didn't want to introduce new stuff in 6.5 branch. So, get_contract_summary request took more than one minute. Maybe TzKT was down that time. But the actual issue is that the TimeoutError is not catched in retry_request as it supposed to. Thanks for the report!