Quick way to kill PID 1 in a pod

[rocketraman]

Is your feature request related to a problem? Please describe. I sometimes wish to restart a container in a pod, without deleting the entire pod itself.

This causes K8S to restart a container in a pod, which can be very useful if the pod has some local state that you wish to maintain across the restart, or the pod name is significant in some way (a Kafka consumer using the static consumer group feature with the pod name as instance.id comes to mind here), or if you wish to avoid potentially expensive rescheduling of a StatefulSet pod. Obviously this isn't something applications should rely on for correctness, but can be useful as a shortcut.

Currently, the right / only way to do this seems to be to shell into a container in the pod and execute "kill 1" but I'm wondering if this might be something useful that k9s could support directly?

Describe the solution you'd like A shortcut to kill PID 1 of one container of a pod. This could be a shortcut "k" at the container list level, similar to the shortcut "k" at the pod list level.

Describe alternatives you've considered I haven't looked into it deeply yet, but I may be able to script this via the new command scripting functionality.

Additional context This is somewhat related to my comment on the "kill pod" feature here: https://github.com/derailed/k9s/issues/272#issuecomment-520608702.

[derailed]

@rocketraman Thank you for this report! I do see your point regarding container cycling without nuking the pod. I'll need to do some research here as the one obvious way to achieve this outside the cluster if to exec + kill. My worry tho is some container may not offer the kill command. Not sure how else we could do it? I know a few folks have offered ways to cycle pods when updating associated cms/secrets. Not quiet sure how they went about it so I'll have to check it out.

As for the pod deletion aka kill I think we should be ok as this operation will be on the container and not the pod itself.

[rocketraman]

@mcristina422 Care to explain the downvote?

[brunohms]

@derailed I would like to get working on this issue, did a quick search and found this saying basically that the options are:

• kubectl exec -it [POD_NAME] -c [CONTAINER_NAME] -- /bin/sh -c "kill 1"
• kubectl exec POD_NAME -c CONTAINER_NAME /sbin/killall5

Can you explain further when those would not be available in a pod?

[brunohms]

@rocketraman AFAIK where I work everytime we had to do something that is pod-name dependend we used replicaset. This way the pod would always be recreated with the same name.

Does this help you in any way?

[rocketraman]

@brunohms

Can you explain further when those would not be available in a pod?

It all depends what the container contains. If the container contains the commands /bin/sh or /sbin/killall5 then those commands will work, otherwise they won't. A container isn't guaranteed to contain a shell at /bin/sh, though most containers of course do. IMO, its fine for a kill container command to produce an error if /bin/sh doesn't exist -- this is the same logic as the shell ("s") command as well, which will only work if a shell is available in the container and otherwise will not.

@rocketraman AFAIK where I work everytime we had to do something that is pod-name dependend we used replicaset. This way the pod would always be recreated with the same name.

I think you're talking something different here. Killing a pod is not the same thing as killing a container inside a pod.

If you kill the pod, no, a ReplicaSet will not recreate the pod with the same name. It will create a new pod with a new name. ReplicaSet pods are meant to be ephemeral so this is OK (and any k8s-based applications that rely on any other behavior is broken). If you want k8s to recreate the pod with the same name, you have to use a StatefulSet.

So if pods are meant to be ephemeral, why support a kill container function? In some cases its nice to take a little shortcut of killing the container inside the pod instead, knowing that killing the container is a faster way to achieve the same end goal by avoiding all of the K8S rescheduling machinery and requiring the new pod to rebuild its ephemeral state, because K8S will just restart the killed container instead of recreating and scheduling a new pod.

This is also true for StatefulSet's as well -- even though a StatefulSet will be rescheduled with the same name, since killing the pod goes through the rescheduling machinery, and the underlying persistent volumes attached to a StatefulSet may actually have to be detached and re-attached, this can be a slow operation (especially on Azure). Killing a container inside a StatefulSet pod is a nice shortcut, because it quicly restarts the app inside the pod without changing anything about the pod and therefore the attached volumes.

All that being said, looking at this issue again, I think this is probably too confusing and is likely something that should be up to users to script themselves if they want it. I don't know if the scripting language is powerful enough to achieve this though.

[brunohms]

@rocketraman

If you kill the pod, no, a ReplicaSet will not recreate the pod with the same name. It will create a new pod with a new name. ReplicaSet pods are meant to be ephemeral so this is OK (and any k8s-based applications that rely on any other behavior is broken). BUT, in some cases its nice to take a little shortcut of killing the container inside the pod instead, knowing that killing the container is a faster way to achieve the same end goal by avoiding all of the K8S rescheduling machinery and requiring the new pod to rebuild its ephemeral state, because K8S will just restart the killed container instead of recreating and scheduling a new pod.

Woops, wrong name, I wanted to mean StatefulSet not ReplicaSet. You're right, recplicaset would recreate the pod with a diferent name.

All that being said, looking at this issue again, I think this is probably too confusing and is likely something that should be up to users to script themselves if they want it. I don't know if the scripting language is powerful enough to achieve this though.

I do not find that confusing at all, just wanted some more clarification. I feel like /bin/sh or /sbin/killall5 should work most of the time.

[derailed]

@rocketraman @brunohms Thank you both for replying and commenting on this issue!! I think it's pretty dicey as it may yield to unexpected behavior on a given pod. My take here, I believe the rollout restart that @mcristina422 put into K9s is a much better/safer option to cycle managed pods. @rocketraman is correct in terms of scheduling but that what sts were designed for ie stable identity and storage. I believe that given container security more folks might leverage scratch images and thus no shell cmds on a container. Also killing and reloading a container in a pod might yield to cm/sec reload hence if they happen to change, you may find yourself with a pod pool that would no longer service requests in uniform manner which could yield debugging session nightmares. To be honest I am not sure if this one off situation would be useful for most K9s users. That said I am, as always, open for suggestions here...

[rocketraman]

@derailed I can't disagree. Its not really an intended UX for Kubernetes, and really requires the user understand the implications of taking this shortcut if they're gonna take it. I'm fine with closing this issue.

[brunohms]

@derailed from my point of view this enhancement would be more useful if it is a way to run a custom comand inside the pod.

For @rocketraman this would be a /bin/sh -c "kill 1" but more users would benefit from it.

[derailed]

How about we focus on beefing up the K9s plugins ? This is really in the spirit of what a plugin already does. I think with a bit of TLC @rocketraman could achieve his goal and as a bonus we could make a K9s plugin shared repo to highlight and share these plugins? I was thinking about this as we currently don't track the associations but we could totally do this and extend the current K9s env vars. In this case the pod name from where the selected container comes from. I think this is totally doable and would provide an affordance for these one shot commands.

[brunohms]

@derailed do you have any plans for K9s plugins? I would like to help in that front, up util now I didn't know that k9s had any plugin related functionality.

[guettli]

I am willing to pay for that feature to get implemented.

[KevinGimbel]

@guettli this can be done with a plugin, something like:

# ~/.config/k9s/plugins.yaml
plugins:
  kubectl-force-kill-pid1:
    shortCut: Shift-K
    confirm: true
    description: Force kill PID 1 in container
    scopes:
      - pod
    background: false
    command: kubectl
    args:
      - exec
      - $COL-NAME
      - --
      - /bin/sh
      - -c
      - "kill -9 1"

The problem is that not all containers have /bin/sh or kill. It depends on your setup and your containers; Generally I think a plugin config for everyone who wants to use this sort of shortcode is preferred to implementing it in k9s directly.