Closed isichei closed 1 year ago
This most recently occurred 7.31 UTC on Friday 15th, as per the log snippet above. Example logs of it occurring at the exact same time on more than one pod:
Chat thread: https://mojdt.slack.com/archives/C58G63XK5/p1589535763346400
What happened?
Airflow tasks (being ran as Pods on Kubernetes) stop (or are killed) unexpectedly (logs state
SIGTERM
is sent to task). This process is intermittent (no common theme around when it happens) so far we have not been able to map it to any specific script, package or docker image. The SIGTERM seems to be sent to multiple tasks if they are currently running when the SIGTERM is sent (even if the tasks are related to different DAGs). This makes us believe that something outside of Airflow is sending the SIGTERMs to the Pods (although not certain).Example of airflow task logs:
Further Notes
Potential guesses for cause of SIGTERM:
Additional findings: