hashicorp / nomad-driver-podman

A nomad task driver plugin for sandboxing workloads in podman containers
https://developer.hashicorp.com/nomad/plugins/drivers/podman
Mozilla Public License 2.0
224 stars 61 forks source link

Can't mount CSI volume in unprivileged container #226

Open p1u3o opened 1 year ago

p1u3o commented 1 year ago

I'm having an issue where when attempting to use a volume_mount and volume stanza as follows below, the volume can not be mounted.

If I add privileged = true to the container config, the mount works, but I don't see this as wise in production.

rpc error: code = Unknown desc = failed to start task, could not start container: cannot start container, status code: 500: {"cause":"operation not supported","message":"lsetxattr /opt/nomad/data/client/csi/node/juicefs0/per-alloc/34725522-7bb4-8d4a-6f06-0d7646902b75/mxp-swift/rw-file-system-multi-node-multi-writer: operation not supported","response":500}

volume "cache-volume" {
      type            = "csi"
      source          = "mxp-swift"
      read_only = false
      attachment_mode = "file-system"
      access_mode     = "multi-node-multi-writer"
    }
  volume_mount {
    volume      = "cache-volume"
    destination = "/data/job"
  }
jdoss commented 1 year ago

What distro are you running this on and is SELinux set to enforcing?

p1u3o commented 1 year ago
$ cat /etc/os-release
NAME="Rocky Linux"
VERSION="9.1 (Blue Onyx)"
ID="rocky"
ID_LIKE="rhel centos fedora"
$ getenforce
Permissive
$ podman version
Client:       Podman Engine
Version:      4.5.0-dev
API Version:  4.5.0-dev
Go Version:   go1.18.9
Built:        Thu Jan  1 01:00:00 1970
OS/Arch:      linux/amd64

I am not sure if relevant, the the CSI Driver I am using is JuiceFS with extended attributes turned off.

jdoss commented 1 year ago

What is the nomad-driver-podman version? v0.4.2 addresses some of the issues you are seeing which might help but the last time I tried to use JuiceFS via the CSI driver, I always had to run the job as privileged to get the mounts to work correctly.

p1u3o commented 1 year ago

@jdoss I am using 0.4.2 specifically because it worked with the JuiceFS driver

I suppose a work around is to mount the volume in a sidecar in the alloc directory, but I worry about unforseen issues with doing something like that.

I also tried to manually edit the Podman seccomp policy to allow the unallowed function call, but it failed. Perhaps it does the function call only when not running privileged?

I will try running the JuiceFS mount with extended attributes enabled and see if that helps.

Edit: Nope, no difference.

lgfa29 commented 7 months ago

Hi @p1u3o 👋

Apologies for the delay here, somehow this issue fell through the cracks of my GitHub notifications.

Which task driver are you using to run the CSI plugin? Would you be able to share that job with us?