Allow configuring udev rules

[ammmze]

Feature Request

Description

I've got a cluster where I am running Plex Media Server on bare metal nodes running Intel processors with Intel QuickSync. In order for Plex to be able to use that for hardware transcoding, the user plex is running as needs rw permissions to the /dev/dri/* (maybe just the /dev/dri/renderD* ... /dev/dri/renderD128 in my case). It looks like by default the permissions for those devices are 600 and owned as root:root. This is fine IF the plex container is running as root. However, in my case the image I prefer to run is running as a different user with some supplemental groups assigned to the user. One of those groups is named video, which typically is gid 44. So ideally we'd be able to run with a user with the video supplemental group and be able to use this.

The typical way to configure this is with udev rules. However, it doesn't appear we support udev rules in Talos yet.

As a workaround for now, i've just updated my container to run as root, and change owner and permissions on the /dev/dri/* devices to root:video and 660 and then I invoke the entrypoint as the user we normally would have run as. This still isn't ideal though because I can easily exec into the container and immediately I am root.

securityContext:
  runAsUser: 0

command:
  - sh
  - -c
  - |-
    set -eux
    chown root:video /dev/dri/*
    chmod g+rw /dev/dri/*
    exec runuser --user kah --group kah --supp-group video --supp-group users -- /entrypoint.sh

[ammmze]

Just came up with a better workaround...

It turns out the actual location of the rules.d directory that udevd is looking for is at /usr/etc/udev/rules.d rather than /etc/udev/rules.d AND we have /usr/etc/udev mounted as an overlay on top of the read-only filesystem. This allowed me to write the rules file...with a little bit of hacking. Essentially I just created a Job that mounts the directory and writes the file. Then I rebooted the node and was able to confirm that udev rules configured the device(s) and Plex hardware transcoding works!

Ultimately, I think we can support this in the files in the Machine Configuration. However, at present, when I attempted this, I'm told that it can only write to /var and so I get left in a boot loop. Perhaps we can allow /usr/etc/udev/rules.d in the allowed paths to write files to?

FWIW, here's the Job I used:

---
apiVersion: batch/v1
kind: Job
metadata:
  name: udev-rules
  namespace: default
spec:
  template:
    spec:
      restartPolicy: Never
      volumes:
        - name: rulesd
          hostPath:
            path: /usr/etc/udev/rules.d
      affinity:
        nodeAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            nodeSelectorTerms:
              - matchExpressions:
                - key: kubernetes.io/hostname
                  operator: In
                  values:
                  - talos-192-168-32-4
      containers:
        - name: write-udev-rules
          image: alpine:3.14.0
          resources:
            requests:
              memory: 30Mi
              cpu: 20m
            limits:
              memory: 30Mi
              cpu: 20m
          volumeMounts:
            - mountPath: /usr/etc/udev/rules.d
              name: rulesd
          command:
            - sh
            - -c
            - |-
              set -eux
              echo 'SUBSYSTEM=="drm", KERNEL=="renderD*", GROUP="44", MODE="0660"' > /usr/etc/udev/rules.d/99-intel-gpu.rules
              echo 'SUBSYSTEM=="drm", KERNEL=="card*", GROUP="44", MODE="0660"' >> /usr/etc/udev/rules.d/99-intel-gpu.rules
  backoffLimit: 0

[ammmze]

after finding the relevant code, i realized I can actually still use the files machine config by making the path relative to /var. obviously quite hacky and potential for breakage in the future. The downside here is that the file gets written after udev has already started its thing. So for already present devices (like the intel gpu), it won’t get picked up until you reboot again after having the boot process write the file.

files:
    - path: /var/../usr/etc/udev/rules.d/99-intel-gpu.rules
        permissions: 0o644
        op: create
        content: |-
        SUBSYSTEM=="drm", KERNEL=="card*", GROUP="44", MODE="0660"
        SUBSYSTEM=="drm", KERNEL=="renderD*", GROUP="44", MODE="0660"

[sergelogvinov]

Nice hack, thank you for that!

/var/../usr/...

[smira]

we should disallow hacks like that lol :)

[ammmze]

I'm new to GO, but i'm willing to take a crack at implementing this as a first class citizen.

First step is probably deciding what the config would look like. Do you think it makes sense to support multiple rules files? Or would we just collect the rules themselves and just write them to a single file? I'm not sure there is a need for multiple files TBH. But this is also my first experience with udev rules...so...maybe there is more to it and just taking the rules and writing them out to a single file.

I'm thinking under the MachineConfig, we'd have something like:

udev:
  rules:
    - SUBSYSTEM=="drm", KERNEL=="card*", GROUP="44", MODE="0660"
    - SUBSYSTEM=="drm", KERNEL=="renderD*", GROUP="44", MODE="0660"

Which would write those rules on newlines to /usr/etc/udev/rules.d/99-machine.rules. Also not sure if there is a preference to that numeric value in there (which typically is just for ordering...but typically we'd just have the one file, so it probably doesn't matter).

Otherwise if we want to support different files, we could do something like:

udev:
  rules:
    - name: 99-intel-gpu
      rules:
        - SUBSYSTEM=="drm", KERNEL=="card*", GROUP="44", MODE="0660"
        - SUBSYSTEM=="drm", KERNEL=="renderD*", GROUP="44", MODE="0660"

As far as actual implementation goes, it looks like i'd need to:

1. Update the machine config interface here
2. Add a method like this but obviously using the udev rules config
3. Add the above method as a step between mounting overlay fs and starting udev here

Of course adding tests where ever possible. Anything that needs to be updated for reading the config file or anything?

And need to figure out the process of building talos to do a full end-to-end test.

[ammmze]

Oh another question I had...

For udev rules that span multiple lines they require a backslash before each newline. Presumably we would want to do replace all new lines with backslash + newline when writing out the file? Or would we just expect whoever is writing the machine config to add the backslashes.

For example:

udev:
  rules:
    - |-
      SUBSYSTEM=="drm",
      KERNEL=="card*",
      GROUP="44",
      MODE="0660"

versus

udev:
  rules:
    - |-
      SUBSYSTEM=="drm", \
      KERNEL=="card*", \
      GROUP="44", \
      MODE="0660"

[ammmze]

Personally I would lean towards allowing it to be configured without backslashes and we would add backslashes when writing the file.

udev:
  rules:
    - |-
      SUBSYSTEM=="drm",
      KERNEL=="card*",
      GROUP="44",
      MODE="0660"