search

hashicorp / nomad

Nomad is an easy-to-use, flexible, and performant workload orchestrator that can deploy a mix of microservice, batch, containerized, and non-containerized applications. Nomad is easy to operate and scale and has native Consul and Vault integrations.
https://www.nomadproject.io/
Other
14.94k stars 1.96k forks source link

Cgroups with `raw_exec` also enable eBPF filtering on devices #17200

Closed gudmundur closed 1 year ago

gudmundur commented 1 year ago

Nomad version

Output from 1.5.1

Operating system and Environment details

Linux with cgroups v2 enabled.

Issue

My expectation with cgroups enabled for raw_exec is that it only manages cgroups for the task, and not provide any additional confinement. When we upgraded from cgroups v1 to v2 a little while back, we found that we had to use no_cgroups as Nomad applies eBPF filtering on devices.

Reproduction steps

Using raw_exec and no_cgroups = false, try accessing any devices that are not allowed in specconv.AllowedDevices.

Expected Result

Only cgroups applied, no device filtering or any confinement as raw_exec implies.

Actual Result

Permission denied error when attempting to access for example /dev/mapper/<Device created by udev>.

tgross commented 1 year ago

Hi @gudmundur! I think you just figured out what I was having trouble figuring out in https://github.com/hashicorp/nomad/issues/13538. The fix isn't immediately obvious, as that executor is also used by the exec and java drivers, so we'll need to make sure it's configured differently for those two cases, and we'll probably also want to provide better guidance to the exec driver docs about how to open up to a larger set of devices when needed.

shoenig commented 1 year ago

As of https://github.com/hashicorp/nomad/pull/18371 (Nomad 1.7) raw_exec should no longer trigger the eBFP filtering for devices.