Stop using `airflow celery stop` command

[rafidka]

Overview

TL;DR - The airflow celery stop command doesn't work when the worker is not running in daemon mode, which is the case for us.

The implementation of the celery stop command relies on the existence of the airflow-worker.pid file; if it doesn't exist, you end up getting None here, and nothing happens. Now, the airflow-worker.pid file is only generated if we pass the --daemon option when we call airflow celery worker; see help snapshot below. Since we are not using daemon mode when we create the worker, no airflow-worker.pid file ends up being created, and hence calls to airflow celery stop ends up doing nothing. I have verified this by logging into the Docker container itself and executing airflow celery stop manually, and it didn't do nothing.

Acceptance Criteria

• Avoid all reliance on airflow celery stop calls to stop Celery workers. We should instead just interrupt the worker ourselves.
• Potentially, the above is a bug in Airflow Celery Provider, so we should also create an issue to their repository to get it fixed.

Additional Info

[rafidka]

We are not using this anymore, and instead we are just SIGTERMing the process we create.