search

hashicorp / nomad

Nomad is an easy-to-use, flexible, and performant workload orchestrator that can deploy a mix of microservice, batch, containerized, and non-containerized applications. Nomad is easy to operate and scale and has native Consul and Vault integrations.
https://www.nomadproject.io/
Other
14.84k stars 1.95k forks source link

After uninstalling Nomad file locks on the "alloc secrets" directory persists #8814

Open idrennanvmware opened 4 years ago

idrennanvmware commented 4 years ago

Nomad version

Nomad v0.12.0 (8f7fbc8e7b5a4ed0d0209968faf41b238e6d5817)

Operating system and Environment details

PhotonOS3

Issue

After draining a node of allocations, and stopping the Nomad agent entirely - file locks persist on the secrets folders for allocs.

We verified this by logging in to the vm and we are unable to remove the directories from even the command line (see output)

rm: cannot remove '/opt/nomad/alloc/41cf69f3-6a66-9b68-c100-b207e61d9bcf/telegraf-statsd-logging-task/secrets': Device or resource busy rm: cannot remove '/opt/nomad/alloc/41cf69f3-6a66-9b68-c100-b207e61d9bcf/run-telegraf-with-statsd/secrets': Device or resource busy rm: cannot remove '/opt/nomad/alloc/45375c6f-ee38-6ef0-73a1-f1dae791046a/fabio/secrets': Device or resource busy rm: cannot remove '/opt/nomad/alloc/946bc1fa-d8ea-6c05-ed37-b258a4d64b26/journalbeat-task/secrets': Device or resource busy rm: cannot remove '/opt/nomad/alloc/946bc1fa-d8ea-6c05-ed37-b258a4d64b26/filebeat-task/secrets': Device or resource busy

tgross commented 4 years ago

Hi @idrennanvmware! This looks to be another case of https://github.com/hashicorp/nomad/issues/7848. If the client node doesn't get a chance to GC before it's shut down, it's leaving these tmpfs mounts for secrets behind.

idrennanvmware commented 4 years ago

Gotcha! Thanks @tgross