Girder 5

[manthey]

There are some jobs that are run locally (so that they share the python environment of the girder server), and some that are run remotely (so that they use the environment of girder_worker). This has been a virtue for, example, converting images to a more efficient format without needing to have a girder_worker node spun up. If we convert all of these jobs to celery jobs, we might want to retain a deployment mode with a local queue that is distinct from the girder_worker queue.

[zachmullen]

In Girder 5, anything that needs to be done asynchronously will be offloaded from the Girder HTTP server, with celery being the built-in choice for doing so. To accommodate that change, the idea is to make the celery environment as similar as possible to the server environment, chiefly by giving it access to the mongo database.

we might want to retain a deployment mode with a local queue that is distinct from the girder_worker queue.

Indeed, this is the goal; I've started replacing these tasks with raw celery tasks rather than girder_worker tasks. The deployment mode now uses multiple processes and eschews background threads. See https://github.com/girder/histomics-dev-env/blob/main/docker-compose.yml for an example of a dev environment that reflects this.