search

hashicorp / nomad-driver-podman

A nomad task driver plugin for sandboxing workloads in podman containers
https://developer.hashicorp.com/nomad/plugins/drivers/podman
Mozilla Public License 2.0
226 stars 62 forks source link

Draining a node does not remove old pods #230

Open jdoss opened 1 year ago

jdoss commented 1 year ago

While working on a systemd unit to drain my client nodes of jobs before shutdown, I noticed that if you have a job that uses podman pods, it does not remove the pods as part of the drain:

These are left over after a few drains testing the unit.

# podman ps -a
CONTAINER ID  IMAGE                 COMMAND     CREATED       STATUS                         PORTS                                                     NAMES
b6868d8e0485  k8s.gcr.io/pause:3.1              40 hours ago  Exited (0) 37 hours ago        100.71.2.23:30608->8080/tcp, 100.71.2.23:30608->8080/udp  pod-76dd69dc-694c-2549-e988-38f3f4ab4ea2
e462e4822b71  k8s.gcr.io/pause:3.1              37 hours ago  Exited (0) 26 hours ago        100.71.2.23:29984->8080/tcp, 100.71.2.23:29984->8080/udp  pod-67df082b-d322-d5f0-64d2-d097f587e09c
1dae89051a88  k8s.gcr.io/pause:3.1              5 hours ago   Exited (0) About a minute ago  100.71.2.23:25426->8080/tcp, 100.71.2.23:25426->8080/udp  pod-1cbb8973-df27-7f93-dae5-a464bc0f6073
towe75 commented 1 year ago

"draining" means nomad node drain, right?

Can you share a example job definition?

jdoss commented 1 year ago

Sorry for the late reply. Yes, I mean nomad node drain. If you use the example https://github.com/hashicorp/nomad-driver-podman/blob/main/examples/jobs/nats_pod.nomad and then issue a nomad node drainit should leave the pod container from the job behind.

If the example doesn't reproduce the issue, I can fix up the job file that I have for Netbox so I can share it, but I believe the nats pod example should reprio the issue as well.