conda-incubator / conda-store

Data science environments, for collaboration. ✨
https://conda.store
BSD 3-Clause "New" or "Revised" License
139 stars 45 forks source link

Terminate worker subprocess in standalone mode #707

Closed nkaretnikov closed 6 months ago

nkaretnikov commented 7 months ago

Context

When running with --standalone, conda-store spins up a celery worker subprocess. That subprocess is not terminated when you Ctrl-C the main process. This needs to be done in a way that would work on Linux, Windows, macOS.

conda-store-server/conda_store_server/server/app.py

        # start worker if in standalone mode
        if self.standalone:
            import multiprocessing

            multiprocessing.set_start_method("spawn")

            from conda_store_server.worker.app import CondaStoreWorker

            process = multiprocessing.Process(target=CondaStoreWorker.launch_instance)
            process.start()

https://github.com/conda-incubator/conda-store/issues/513#issuecomment-1766776871

If we do want it, it's possible to forcibly control-c out of the standalone server in a way that leaves the worker processes intact. See Revisit the implementation of --standalone as well as the assumptions made #513 (comment). However, I've also noticed that this is somewhat of an issue with celery itself, even when the worker is run separately, that cancellation doesn't work very smoothly. The difference is that when workers are run separately, the process doesn't end until the workers are actually shutdown. Not clear how important this issue is. It comes up during development, but I don't know if it comes up in deployment.

Value and/or benefit

Improved UX.

Anything else?

No response