search

bihealth / sodar-server

SODAR: System for Omics Data Access and Retrieval
https://github.com/bihealth/sodar-server
MIT License
14 stars 3 forks source link

Uncaught exception if project is locked by another operation #1842

Open mikkonie opened 11 months ago

mikkonie commented 11 months ago

I noticed this in our production server logs. This obviously should not go uncaught, I need to investigate.

sodar-celeryd-default_1  | [2023-11-30 10:36:59,320: ERROR/ForkPoolWorker-15] Task taskflowbackend.tasks_celery.submit_flow_task[6c57ffa9-4ee9-42ca-92fb-4322a8d23d2d] raised unexpected: FlowSubmitException('Unable to acquire project lock: Project is locked by another operation')
sodar-celeryd-default_1  | Traceback (most recent call last):
sodar-celeryd-default_1  |   File "/usr/src/app/taskflowbackend/api.py", line 149, in run_flow
sodar-celeryd-default_1  |     lock_api.acquire(lock)
sodar-celeryd-default_1  |   File "/usr/src/app/taskflowbackend/lock_api.py", line 82, in acquire
sodar-celeryd-default_1  |     raise LockAcquireException('Project is locked by another operation')
sodar-celeryd-default_1  | taskflowbackend.lock_api.LockAcquireException: Project is locked by another operation
sodar-celeryd-default_1  | 
sodar-celeryd-default_1  | During handling of the above exception, another exception occurred:
sodar-celeryd-default_1  | 
sodar-celeryd-default_1  | Traceback (most recent call last):
sodar-celeryd-default_1  |   File "/usr/local/lib/python3.8/site-packages/celery/app/trace.py", line 451, in trace_task
sodar-celeryd-default_1  |     R = retval = fun(*args, **kwargs)
sodar-celeryd-default_1  |   File "/usr/local/lib/python3.8/site-packages/sentry_sdk/integrations/celery.py", line 207, in _inner
sodar-celeryd-default_1  |     reraise(*exc_info)
sodar-celeryd-default_1  |   File "/usr/local/lib/python3.8/site-packages/sentry_sdk/_compat.py", line 60, in reraise
sodar-celeryd-default_1  |     raise value
sodar-celeryd-default_1  |   File "/usr/local/lib/python3.8/site-packages/sentry_sdk/integrations/celery.py", line 202, in _inner
sodar-celeryd-default_1  |     return f(*args, **kwargs)
sodar-celeryd-default_1  |   File "/usr/local/lib/python3.8/site-packages/celery/app/trace.py", line 734, in __protected_call__
sodar-celeryd-default_1  |     return self.run(*args, **kwargs)
sodar-celeryd-default_1  |   File "/usr/src/app/taskflowbackend/tasks_celery.py", line 29, in submit_flow_task
sodar-celeryd-default_1  |     taskflow.run_flow(
sodar-celeryd-default_1  |   File "/usr/src/app/taskflowbackend/api.py", line 151, in run_flow
sodar-celeryd-default_1  |     cls._raise_flow_exception(
sodar-celeryd-default_1  |   File "/usr/src/app/taskflowbackend/api.py", line 63, in _raise_flow_exception
sodar-celeryd-default_1  |     raise cls.FlowSubmitException(ex_msg)
mikkonie commented 11 months ago

It turns out adding exception handling here causes some API view breaking problems due to how we handle lock problems. I'm postponing fixing this until the whole handling has been refactored. I'll also open the appropriate issue(s) for the problem.