bihealth / sodar-docker-compose

Setup SODAR using Docker Compose
MIT License
2 stars 3 forks source link

Make celery and celerybeat wait for sodar-web startup #10

Open mikkonie opened 2 years ago

mikkonie commented 2 years ago

Right now, celery may attempt to execute tasks before database migrations have been completed. This, obviously, results in errors. Example:

sodar-web_1              |   Applying landingzones.0005_rename_uuid... OK
sodar-web_1              |   Applying landingzones.0006_update_uuid... OK
sodar-web_1              |   Applying landingzones.0007_update_jsonfields... OK
sodar-celeryd-default_1  | [2022-05-11 14:12:59,967: INFO/MainProcess] mingle: all alone
sodar-celeryd-default_1  | [2022-05-11 14:12:59,976: INFO/MainProcess] celery@05c8e7428056 ready.
sodar-web_1              |   Applying landingzones.0008_landingzone_user_message... OK
sodar-celeryd-default_1  | [2022-05-11 14:12:59,997: INFO/MainProcess] Task samplesheets.tasks.sheet_sync_task[299e07e5-0bb5-4fc8-8391-f27e6e15699f] received
sodar-celeryd-default_1  | [2022-05-11 14:12:59,999: INFO/MainProcess] Task landingzones.tasks.trigger_zone_move[73cd0e5d-f356-49e8-9ba2-ea38cc7dd1dc] received
postgres_1               | 2022-05-11 12:13:00.008 UTC [98] ERROR:  column projectroles_project.has_public_children does not exist at character 418
sodar-celeryd-default_1  | psycopg2.errors.UndefinedColumn: column projectroles_project.has_public_children does not exist
(Some time later...)
sodar-web_1              |   Applying projectroles.0020_project_has_public_children... OK

This will happen only on first startup and when the SODAR database migrations are updated and in most cases should only result in a single failure of a periodic task which will be retried after a while anyway. Still, should fix it nonetheless to avoid logging redundant errors.