Airflow tasks are killed on Kubernetes

[isichei]

What happened?

Airflow tasks (being ran as Pods on Kubernetes) stop (or are killed) unexpectedly (logs state SIGTERM is sent to task). This process is intermittent (no common theme around when it happens) so far we have not been able to map it to any specific script, package or docker image. The SIGTERM seems to be sent to multiple tasks if they are currently running when the SIGTERM is sent (even if the tasks are related to different DAGs). This makes us believe that something outside of Airflow is sending the SIGTERMs to the Pods (although not certain).

Example of airflow task logs:

[2020-05-15 07:31:52,314] {logging_mixin.py:95} INFO - [2020-05-15 07:31:52,314] {jobs.py:2536} ERROR - Received SIGTERM. Terminating subprocesses
[2020-05-15 07:31:52,315] {helpers.py:281} INFO - Sending Signals.SIGTERM to GPID 15
[2020-05-15 07:31:52,315] {__init__.py:1416} ERROR - Received SIGTERM. Terminating subprocesses.
[2020-05-15 07:31:53,024] {__init__.py:1580} ERROR - Pod Launching failed: Task received SIGTERM signal

Further Notes

Potential guesses for cause of SIGTERM:

• k8s networking failed
• whatever gets the logs failed (does the node's kubelet do this?)
• k8s decided to kill the worker pod because it was over its memory limit
• k8s decided to kill the worker pod for outside reasons (node was full or something)

Additional findings:

• Airflow scheduler doesn't seem to be affected by SIGTERM.
• Logs from pods that received SIGTERM could not be recovered.

[davidread]

This most recently occurred 7.31 UTC on Friday 15th, as per the log snippet above. Example logs of it occurring at the exact same time on more than one pod:

• https://airflow.tools.alpha.mojanalytics.xyz/log?execution_date=2020-05-15T07%3A12%3A34.422715%2B00%3A00&task_id=all-offence&dag_id=xhibit_derived_db
• https://airflow.tools.alpha.mojanalytics.xyz/log?task_id=disposal-summary&dag_id=xhibit_derived_db&execution_date=2020-05-15T07%3A12%3A34.422715%2B00%3A00

Chat thread: https://mojdt.slack.com/archives/C58G63XK5/p1589535763346400