Make Celery worker the root process

hacktobeer commented 1 month ago

Description of the change

This PR strips away layers of complexity and makes the Celery worker the root process in the container. This will benefit the reliability and stability of the system.

will eliminate the race condition we had in k8s
will eliminate the need for scaledown locks in both Turbinia code and helm charts (follow-up PR to remove that code)
Will in general make debugging easier as there are less Turbinia wrapper classes involved.

This PR also:

adds hot reloading of the Celery worker through watchdog without the need for Python binary injection hacks for development purposes (see Dockerfile)
set default log level for the worker to INFO which can be changed to DEBUG by the setting the TURBINIA_DEBUG=1 environment variable.

Applicable issues

-fixes #

Additional information

Checklist

[x] All tests were successful.
[ ] Unit tests added.
[ ] Documentation updated.

hacktobeer commented 1 month ago

@jleaniz @wajihyassine Besides the normal code review I would like you to review this PR with a bigger scope in your mind as well if you have time.

E.g. Do we need to cleanup move code or documents? Can we tune the Celery settings? Can we introduce different task types (as we only hard code one task (task-runner) for all jobs :(.

hacktobeer commented 1 month ago

@wajihyassine Do we use any of the arguments from start.sh in the helm charts? As with this PR we are not using start.sh and are not passing in those args. https://github.com/google/turbinia/blob/master/docker/worker/start.sh

google / turbinia