hashicorp / nomad-driver-podman

A nomad task driver plugin for sandboxing workloads in podman containers
https://developer.hashicorp.com/nomad/plugins/drivers/podman
Mozilla Public License 2.0
224 stars 61 forks source link

All nomad containers receive SIGTERM after ~10 minutes #346

Open ColtonHerrod opened 1 month ago

ColtonHerrod commented 1 month ago

I've been troubleshooting a strange issue off and on for a while. All nomad containers I run receive a SIGTERM after about 10 minutes of runtime and reboot as appropriate for their configuration. This appears to be the case regardless of the image being run or if health checks are enabled. None of the containers approach their memory or CPU limits and the single node I'm running is nowhere near its resource limits. This pattern seems to persist as long as the job runs.

There is nothing seemingly obvious in the logs that points to a specific issue. Is this something that could be an issue with the podman driver or with my local configuration? Happy to provide additional information as needed.

System Information