search

voiio / Sam

Sam – cuz your company is nothing without Sam
https://code.voiio.de/Sam/
BSD 3-Clause "New" or "Revised" License
8 stars 1 forks source link

Task was destroyed but it is pending! #85

Open sentry-io[bot] opened 5 months ago

sentry-io[bot] commented 5 months ago

Sentry Issue: VOIIO-SAM-9

Task was destroyed but it is pending!
task: <Task pending name='Task-4668' coro=<AsyncioListenerRunner.run.<locals>.run_ack_function_asynchronously() done, defined at /app/.heroku/python/lib/python3.11/site-packages/slack_bolt/listener/asyncio_runner.py:111> wait_for=<Future pending cb=[Task.task_wakeup()]>>
codingjoe commented 5 months ago

Graceful shutdown or restarts are an issue. If you have a ongoing inference run, a SIGTERM will leave it hanging. Which can lead to errors, if you have a run that is pending function execution. Users will not be able to add new messages to a thread or start another run for 10 minutes (run timeout).

amureki commented 5 months ago

Graceful shutdown or restarts are an issue. If you have a ongoing inference run, a SIGTERM will leave it hanging. Which can lead to errors, if you have a run that is pending function execution. Users will not be able to add new messages to a thread or start another run for 10 minutes (run timeout).

So, what I could imagine - on a graceful shutdown - we should keep it in the queue and come back to it as soon as the server is back up. Or what would you expect here?

codingjoe commented 5 months ago

That's a great idea, how do we know if a run is still in the queue, though? Or rather, how do we remember which runs we should poll again? Listing runs for all threads on redis seems like an overkill. We would store runs in a queue and acknowledge them upon completion.