search

kubernetes / kubernetes

Production-Grade Container Scheduling and Management
https://kubernetes.io
Apache License 2.0
110.88k stars 39.61k forks source link

A pod with a still running postStart lifecycle hook that is deleted is not terminated even after terminationGracePeriod #116032

Open jgoeres opened 1 year ago

jgoeres commented 1 year ago

What happened?

We introduced a postStart hook to one of our pods. Eventually, we noticed these pods hanging in "Terminating" state indefinitely when we try to delete them. Trying kubectl logsyielded Error from server (BadRequest): container "reproducer" in pod "reproducer" is waiting to start: ContainerCreating

That was the first hint that the postStart hook was the problem (since a pod is considered PodInitializing until its postStart hook exits).

We can exec into the container just fine, there we found that indeed our postStart hook was stuck. Killing all processes of the postStart hook made the pod terminate. Looking at the logs of our PID 1 process (luckily, we also write that to a file...) showed no sign of it receiving a SIGTERM (we have a shutdown hook in that application that would log if it were invoked).

From our observations, we conclude that while the pod is still not done with the postStart hook, K8s does not send a SIGTERM to the container's PID 1 process. To make matters worse, even after the terminationGracePeriod expired, no kill happens, leading to the observed behaviour.

What did you expect to happen?

The pod terminates - if not with a SIGTERM to the PID 1 process, then at least with a kill after the terminationGracePeriod expired.

How can we reproduce it (as minimally and precisely as possible)?

Apply the following pod spec:

apiVersion: v1
kind: Pod
metadata:
  name: reproducer
spec:
  containers:
    - command:
        - /bin/bash
        - -c
        - |
          trap "echo SIGTERM received, setting exit file | tee -a /log.txt; touch /exit" SIGTERM;
          while [ ! -f "/exit" ]; do
            echo "$(date) Faking work" | tee -a /log.txt;
            sleep 10;
          done

      image: ubuntu
      imagePullPolicy: IfNotPresent
      lifecycle:
        postStart:
          exec:
            command:
              - /bin/bash
              - -c
              - | 
                while [ ! -f "/postStartDone" ]; do 
                  sleep 5;  
                done; 
                echo "poststart hook done" | tee -a /log.txt

      name: reproducer
  terminationGracePeriodSeconds: 30

(Note that we explicitly set the terminationGracePeriodSeconds to the default 30 for clarity.) In one terminal, run watch -d kubectl get pods. After applying, the pod will show up, but since the postStart hook will not exit, the pod will stay in ContainerCreating. Confirm that you cannot see the logs with kubectl logs reproducer (error message Error from server (BadRequest): container "reproducer" in pod "reproducer" is waiting to start: ContainerCreating)

Exec into the pod with kubectl exec -it reproducer -- /bin/bash Confirm that both the root process and the post start hook are running with ps aux:

root@reproducer:/# ps aux
USER         PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
root           1  0.0  0.0   4360  3320 ?        Ss   09:55   0:00 /bin/bash -c trap "echo SIGTERM received, setting exit file | tee -a /log.txt; touch /exit" SIGTERM; while [ ! -f "/exit"
root          11  0.0  0.0   4360  3364 ?        Ss   09:55   0:00 /bin/bash -c while [ ! -f "/postStartDone" ]; do    sleep 5;   done;  echo "poststart hook done" | tee -a /log.txt
root          19  0.0  0.0   4628  3796 pts/0    Ss   09:55   0:00 /bin/bash
root          31  0.0  0.0   2788  1096 ?        S    09:55   0:00 sleep 10
root          32  0.0  0.0   2788  1108 ?        S    09:55   0:00 sleep 5
root          33  0.0  0.0   7060  1592 pts/0    R+   09:55   0:00 ps aux

Exit. Delete the pod with kubectl delete reproducer. Observe that the pod goes from Container Creating to Terminating and stays in that state indefinitely (and stays that way indefinitely, even after the termination grace period of 30secs). Exec into the pod again, run tail -f log.txt, observe the log output... you should only see lines Faking work. In another shell, create the file that will make the postStart hook exit successfully with kubectl exec reproducer -- /bin/touch /postStartDone poststart hook donewill be logged and eventually, the pod will terminate.

Anything else we need to know?

No response

Kubernetes version

Originally observed on EKS v1.24.8-eks-ffeb93d, reproduced on AKS v1.23.8 and v1.25.2, Minikube v1.24.10 and v1.25.6

Cloud provider

See above AWS ( EKS v1.24.8-eks-ffeb93d) and Azure AKS v1.23.8 and v1.25.2

OS version

```console # On Linux: $ cat /etc/os-release # paste output here $ uname -a # paste output here # On Windows: C:\> wmic os get Caption, Version, BuildNumber, OSArchitecture # paste output here ```

Install tools

Container runtime (CRI) and version (if applicable)

Related plugins (CNI, CSI, ...) and versions (if applicable)

HirazawaUi commented 1 year ago

/sig node

xyz-li commented 1 year ago

Related issue #113606 . The pod worker cannot cancel the context. https://github.com/kubernetes/kubernetes/blob/7efa62dfdf96890f7f3cf95d957c7561e09055c4/pkg/kubelet/pod_workers.go#L776

https://github.com/kubernetes/kubernetes/blob/7efa62dfdf96890f7f3cf95d957c7561e09055c4/pkg/kubelet/kubelet.go#L1606-L1609

smarterclayton commented 1 year ago

113606 blocks and should resolve this issue.

mariida commented 1 year ago

/cc

SergeyKanzhelev commented 1 year ago

/triage accepted /assign @smarterclayton

k8s-triage-robot commented 7 months ago

This issue has not been updated in over 1 year, and should be re-triaged.

You can:

For more details on the triage process, see https://www.kubernetes.dev/docs/guide/issue-triage/

/remove-triage accepted

k8s-triage-robot commented 4 months ago

The Kubernetes project currently lacks enough contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

You can:

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

k8s-triage-robot commented 3 months ago

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

You can:

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle rotten

k8s-triage-robot commented 2 months ago

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues according to the following rules:

You can:

Please send feedback to sig-contributor-experience at kubernetes/community.

/close not-planned

k8s-ci-robot commented 2 months ago

@k8s-triage-robot: Closing this issue, marking it as "Not Planned".

In response to [this](https://github.com/kubernetes/kubernetes/issues/116032#issuecomment-2282853800): >The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. > >This bot triages issues according to the following rules: >- After 90d of inactivity, `lifecycle/stale` is applied >- After 30d of inactivity since `lifecycle/stale` was applied, `lifecycle/rotten` is applied >- After 30d of inactivity since `lifecycle/rotten` was applied, the issue is closed > >You can: >- Reopen this issue with `/reopen` >- Mark this issue as fresh with `/remove-lifecycle rotten` >- Offer to help out with [Issue Triage][1] > >Please send feedback to sig-contributor-experience at [kubernetes/community](https://github.com/kubernetes/community). > >/close not-planned > >[1]: https://www.kubernetes.dev/docs/guide/issue-triage/ Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md). If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes-sigs/prow](https://github.com/kubernetes-sigs/prow/issues/new?title=Prow%20issue:) repository.
HirazawaUi commented 2 months ago

/reopen

k8s-ci-robot commented 2 months ago

@HirazawaUi: Reopened this issue.

In response to [this](https://github.com/kubernetes/kubernetes/issues/116032#issuecomment-2293896767): >/reopen > Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md). If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes-sigs/prow](https://github.com/kubernetes-sigs/prow/issues/new?title=Prow%20issue:) repository.
kannon92 commented 2 months ago

/triage accepted /priority backlog

kannon92 commented 2 months ago

This is a known issue with pod lifecycle but it was not yet solved. Leaving it open.