stellar / django-polaris

An extendable Django app for building modular Stellar services
https://django-polaris.readthedocs.io
Apache License 2.0
94 stars 66 forks source link

Autorestart watch_transactions #703

Open yuriescl opened 10 months ago

yuriescl commented 10 months ago

Sometimes when there's a network connection issue between the Anchor and horizon, the watch_transaction script crashes due to an exception:

File "/usr/lib/python3.10/asyncio/base_events.py", line 646, in run_until_complete
    return future.result()
  File "/home/app/.venv/lib/python3.10/site-packages/polaris/management/commands/watch_transactions.py", line 77, in watch_transactions
    await asyncio.gather(
  File "/home/app/.venv/lib/python3.10/site-packages/polaris/management/commands/watch_transactions.py", line 117, in _for_account
    async for response in endpoint.stream():
  File "/home/app/.venv/lib/python3.10/site-packages/typeguard/__init__.py", line 954, in asend
    value = await self.__wrapped.asend(obj)
  File "/home/app/.venv/lib/python3.10/site-packages/stellar_sdk/call_builder/call_builder_async/base_call_builder.py", line 69, in stream
    yield await stream.__anext__()
  File "/home/app/.venv/lib/python3.10/site-packages/typeguard/__init__.py", line 954, in asend
    value = await self.__wrapped.asend(obj)
  File "/home/app/.venv/lib/python3.10/site-packages/stellar_sdk/client/aiohttp_client.py", line 217, in stream
    raise StreamClientError(
stellar_sdk.exceptions.StreamClientError: Failed to get stream message.

But once watch_transaction crashes, it doesn't start automatically and requires manual intervention. At the moment the Anchor has to setup some kind of watcher to restart the script automatically, but this is not ideal, Anchor should not need to worry about this.

Polaris should handle those network crashes gracefully by either restarting the stream automatically.

JakeUrban commented 10 months ago

I think this makes sense. We'll add a WARN-level log message when the stream is restarted due this error.