Fixes an issue in which task status is not properly tracked, which caused request status to never fully complete.
Adds a result_backend configuration option for Celery in order to persist celery task metadata in Redis.
Adds a worker_deduplicate_successful_tasks configuration option to Celery to avoid duplicate messages of successful tasks.
Adds a task_reject_on_worker_lost configuration option to Celery so tasks can be re-queued if a worker abruptly exists (e.g. SIGTERM/SIGKILL signal sent from GKE or OS process to celery worker)
Adds a celery_id attribute to Turbinia tasks for traceability between Turbinia and Celery tasks.
Enables Celery mingle in the configuration to be able to handle revoked tasks between workers and worker clock synchronization.
adds some additional exception handling when queueing tasks.
aarontp
commented
1 month ago
Description of the change
This PR
result_backendconfiguration option for Celery in order to persist celery task metadata in Redis.worker_deduplicate_successful_tasksconfiguration option to Celery to avoid duplicate messages of successful tasks.task_reject_on_worker_lostconfiguration option to Celery so tasks can be re-queued if a worker abruptly exists (e.g. SIGTERM/SIGKILL signal sent from GKE or OS process to celery worker)celery_idattribute to Turbinia tasks for traceability between Turbinia and Celery tasks.Celery minglein the configuration to be able to handle revoked tasks between workers and worker clock synchronization.Applicable issues
Additional information
Checklist