Cannot open raw block device provisioned by kubernetes when running as non root user

[kormide]

I'm using kubernetes on GCP to deploy buildbarn and, unlike the example deployment, I'm provisioning entire block devices for the CAS, AC, and KLMs rather than creating files on a device for the remote cache. However, I'm running into a permissions issue when running the container as a non-root user. For example, my securityContext looks like:

securityContext:
    fsGroup: 2000
    runAsUser: 1000
    runAsGroup: 2000
    runAsNonRoot: true

Kubernetes mounts the block devices with the following permissions on the device descriptors. Note that the owner is root and the group is disk (2000).

0 brw-rw---- 1 root disk 8, 64 May 5 20:03 /dev/cas
0 brw-rw---- 1 root disk 8, 64 May 5 20:03 /dev/cas-klm
0 brw-rw---- 1 root disk 8, 64 May 5 20:03 /dev/ac
0 brw-rw---- 1 root disk 8, 64 May 5 20:03 /dev/ac-klm

When I set runAsNonRoot to true, and runAsGroup to 2000, I get the following error when the container is deployed.

2023/05/08 20:32:06 Fatal error: rpc error: code = Unknown desc = Failed to create Content Addressable Storage: Failed to open blocks block device: Failed to open device node "/dev/cas": permission denied

The error is triggered from here.

fd, err := unix.Open(path, unix.O_RDWR, 0)
if err != nil {
    return nil, 0, 0, util.StatusWrapf(err, "Failed to open device node %#v", }

I"m wondering if the parameters need to be changed on that call somehow to allow for a member of the permissions group to read and write rather than just the owner. I'm not aware of a way to change the owner from root when mounting the device in k8s. I'm also relatively new to k8s, so I may be missing something.

[EdSchouten]

Is there a way you could place yourself in the disk group?

[kormide]

Is there a way you could place yourself in the disk group?

I believe that's what the runAsGroup: 2000 should be doing.

[EdSchouten]

I'm going to close this issue. Though it may be the case that you're still running into this issue, addressing it is outside the scope of the Buildbarn project. Getting this fixed pertains to Linux kernel security policies, Kubernetes, containerd, etc., cluster configuration/administration, not Buildbarn.