taylorsilva
commented
4 years ago I tried a quick patch with your suggestion:
diff --git a/templates/worker-prestop-configmap.yaml b/templates/worker-prestop-configmap.yaml
index 9d5dd31..9f43a76 100644
--- a/templates/worker-prestop-configmap.yaml
+++ b/templates/worker-prestop-configmap.yaml
@@ -11,5 +11,5 @@ data:
pre-stop-hook.sh: |
#!/bin/bash
kill -s {{ .Values.concourse.worker.shutdownSignal }} 1
- while [ -e /proc/1 ]; do sleep 1; done
+ timeout -k {{ .Values.worker.terminationGracePeriodSeconds }} {{ .Values.worker.terminationGracePeriodSeconds }} /bin/bash -c 'while [ -e /proc/1 ]; do sleep 1; done'The script still exits with a non-zero exit code, 124 in this case:
Warning FailedPreStopHook 1s kubelet, gke-topgun-topgun-worker-2c49df4e-qwh6 Exec lifecycle hook ([/bin/bash /pre-stop-hook.sh]) for Container "issue81-worker" in Pod "issue81-worker-0_issue81(4ad690c9-d362-48d8-9e5a-c5e873b5571e)" failed - error: command '/bin/bash /pre-stop-hook.sh' exited with 124: , message: ""Not sure what a good solution for this one is 🤔
to reproduce this I installed the helm chart with default settings and started this long running job:
---
jobs:
- name: simple-job
plan:
- task: simple-task
config:
platform: linux
image_resource:
type: registry-image
source: {repository: busybox}
run:
path: /bin/sh
args:
- -c
- |
#!/bin/sh
sleep 1hI then deleted the pod
$ kubectl delete pod -n issue81 issue81-worker-0and kept describing the pod until I saw the relevant error:
$ k describe pod -n issue81 issue81-worker-0 | tail -n 10
smoke
skreddy6673
vineethNaroju
Using the following command stucks for too much time:
When I describe the POD it is clear that the PreStop Hook did not exit clean:
So the only workaround is to now force delete the pod:
May be/pre-stop-hook.shshould be patched to handle (trap) the relevant signals (e.g. SIGTERM, SIGINT, SIGHUP) and exit cleanly. I assume when the dumb-init is signaled, it on its own tries to cleanly terminate the/pre-stop-hook.shand given it does not terminate cleanly - it gets killed with the exit code 137 that then blocks K8S.I will give it a try and will update the ticket, hopefully with a PR.Actually K8S just waits for the PreStop Hook only for a
terminationGracePeriodSecondsamount of time and then sends a SIGTERM the containers and then SIGKILL all the running processes after 2 more seconds as per https://github.com/kubernetes/kubernetes/issues/39170#issuecomment-448195287 and https://kubernetes.io/docs/concepts/workloads/pods/pod/#termination-of-podsHowever strange thing is the POD is left in terminating state for many more minutes and doesn't seem to restart.
So may be the best course of action would be to use
timeout -k {.Values.worker.terminationGracePeriodSeconds} bash -c 'while [ -e /proc/1 ]; do sleep 1; done'or something similar I guess. This way at least the delete command will not be blocked.Also it is important to increase the .Values.worker.terminationGracePeriodSeconds to something that makes sense for your own Pipelines.