search

squat / generic-device-plugin

A Kubernetes device plugin to schedule generic Linux devices
Apache License 2.0
208 stars 23 forks source link

/dev/ttyUSB0 not detected by device plugin #60

Closed Xiol closed 11 months ago

Xiol commented 11 months ago

Hello.

I have a 1-wire USB temperature sensor that sets up /dev/ttyUSB0 as a serial port:

root@excession:~# lsusb | grep Prolific
Bus 001 Device 002: ID 067b:2303 Prolific Technology, Inc. PL2303 Serial Port / Mobile Phone Data Cable

root@excession:~# ls -l /dev/ttyUSB0 
crw-rw---- 1 root dialout 188, 0 Dec  7 08:29 /dev/ttyUSB0

dmesg:

[77172.100651] usb 1-2: new full-speed USB device number 2 using xhci-hcd
[77172.249691] usb 1-2: New USB device found, idVendor=067b, idProduct=2303, bcdDevice= 3.00
[77172.249701] usb 1-2: New USB device strings: Mfr=1, Product=2, SerialNumber=0
[77172.249705] usb 1-2: Product: USB-Serial Controller
[77172.249708] usb 1-2: Manufacturer: Prolific Technology Inc.
[77172.255622] pl2303 1-2:1.0: pl2303 converter detected
[77172.256440] usb 1-2: pl2303 converter now attached to ttyUSB0

The sensor works fine directly on the host (Raspberry Pi 5 with k3s) using digitemp, however the device plugin isn't recognising /dev/ttyUSB0. If I set up the USB vendor/product codes, they get picked up and passed through successfully, but the container requires /dev/ttyUSB0 rather than using the USB device directly.

I've tried only with /dev/ttyUSB0, with /dev/ttyUSB*, with both USB and /dev/ttyUSB0 entries, and none of them will cause the plugin to detect /dev/ttyUSB0.

FWIW the USB device is detected and passed through successfully to the container, but unfortunately I need the serial device instead.

Plugin manifest:

apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: generic-device-plugin
  namespace: kube-system
  labels:
    app.kubernetes.io/name: generic-device-plugin
spec:
  selector:
    matchLabels:
      app.kubernetes.io/name: generic-device-plugin
  template:
    metadata:
      labels:
        app.kubernetes.io/name: generic-device-plugin
    spec:
      priorityClassName: system-node-critical
      tolerations:
      - operator: "Exists"
        effect: "NoExecute"
      - operator: "Exists"
        effect: "NoSchedule"
      containers:
      - image: squat/generic-device-plugin
        args:
        - --device
        - |
          name: serial
          groups:
            - paths:
                - path: /dev/ttyUSB0
                  mountPath: /dev/ttyUSB0
        - --device
        - |
          name: temperature
          groups:
            - usb:
                - product: 2303
                  vendor: 067b
        - --log-level=debug
        name: generic-device-plugin
        resources:
          requests:
            cpu: 50m
            memory: 10Mi
          limits:
            cpu: 50m
            memory: 20Mi
        ports:
        - containerPort: 8080
          name: http
        securityContext:
          privileged: true
        volumeMounts:
        - name: device-plugin
          mountPath: /var/lib/kubelet/device-plugins
        - name: dev
          mountPath: /dev
      volumes:
      - name: device-plugin
        hostPath:
          path: /var/lib/kubelet/device-plugins
      - name: dev
        hostPath:
          path: /dev
  updateStrategy:
    type: RollingUpdate

kubectl describe node:

...snip...
Allocated resources:
  (Total limits may be over 100 percent, i.e., overcommitted.)
  Resource              Requests    Limits
  --------              --------    ------
  cpu                   250m (6%)   50m (1%)
  memory                150Mi (1%)  190Mi (2%)
  ephemeral-storage     0 (0%)      0 (0%)
  squat.ai/serial       0           0
  squat.ai/temperature  1           1

Pod logs for device plugin:

root@excession:~# kubectl -n kube-system logs generic-device-plugin-4rl4k
{"caller":"main.go:218","msg":"Starting the generic-device-plugin for \"squat.ai/serial\".","ts":"2023-12-07T10:26:39.925508384Z"}
{"caller":"main.go:218","msg":"Starting the generic-device-plugin for \"squat.ai/temperature\".","ts":"2023-12-07T10:26:39.92548018Z"}
{"caller":"plugin.go:114","level":"info","msg":"listening on Unix socket","resource":"squat.ai/serial","socket":"/var/lib/kubelet/device-plugins/gdp-c3F1YXQuYWkvc2VyaWFs-1701944799.sock","ts":"2023-12-07T10:26:39.925755144Z"}
{"caller":"plugin.go:114","level":"info","msg":"listening on Unix socket","resource":"squat.ai/temperature","socket":"/var/lib/kubelet/device-plugins/gdp-c3F1YXQuYWkvdGVtcGVyYXR1cmU=-1701944799.sock","ts":"2023-12-07T10:26:39.925851347Z"}
{"caller":"plugin.go:122","level":"info","msg":"starting gRPC server","resource":"squat.ai/serial","ts":"2023-12-07T10:26:39.925926033Z"}
{"caller":"plugin.go:176","level":"info","msg":"waiting for the gRPC server to be ready","resource":"squat.ai/serial","ts":"2023-12-07T10:26:39.92594757Z"}
{"caller":"plugin.go:176","level":"info","msg":"waiting for the gRPC server to be ready","resource":"squat.ai/temperature","ts":"2023-12-07T10:26:39.925969774Z"}
{"caller":"plugin.go:122","level":"info","msg":"starting gRPC server","resource":"squat.ai/temperature","ts":"2023-12-07T10:26:39.925993218Z"}
{"caller":"plugin.go:188","level":"info","msg":"the gRPC server is ready","resource":"squat.ai/temperature","ts":"2023-12-07T10:26:40.023834494Z"}
{"caller":"plugin.go:226","level":"info","msg":"registering plugin with kubelet","resource":"squat.ai/temperature","ts":"2023-12-07T10:26:40.023891568Z"}
{"caller":"plugin.go:188","level":"info","msg":"the gRPC server is ready","resource":"squat.ai/serial","ts":"2023-12-07T10:26:40.023910124Z"}
{"caller":"plugin.go:226","level":"info","msg":"registering plugin with kubelet","resource":"squat.ai/serial","ts":"2023-12-07T10:26:40.023953827Z"}
{"caller":"generic.go:232","level":"info","msg":"starting listwatch","resource":"squat.ai/temperature","ts":"2023-12-07T10:26:40.123648957Z"}
{"caller":"generic.go:232","level":"info","msg":"starting listwatch","resource":"squat.ai/serial","ts":"2023-12-07T10:26:40.124360291Z"}
{"caller":"usb.go:269","level":"debug","msg":"USB device match","path":"/dev/bus/usb/001/002","resource":"squat.ai/temperature","ts":"2023-12-07T10:26:40.323616625Z","usbdevice":"067b:2303"}
{"caller":"usb.go:269","level":"debug","msg":"USB device match","path":"/dev/bus/usb/001/002","resource":"squat.ai/temperature","ts":"2023-12-07T10:26:45.325038652Z","usbdevice":"067b:2303"}
(...repeats...)
squat commented 11 months ago

Hi @Xiol 

Allocated resources:
  (Total limits may be over 100 percent, i.e., overcommitted.)
  Resource              Requests    Limits
  --------              --------    ------
  cpu                   250m (6%)   50m (1%)
  memory                150Mi (1%)  190Mi (2%)
  ephemeral-storage     0 (0%)      0 (0%)
  squat.ai/serial       0           0
  squat.ai/temperature  1           1

This snippet indicates that the Kubelet hasn't allocated any serial devices to any pod, which could happen for several reasons: no one is requesting it, none have been discovered, the plugin refused to allocate it, or some combination.

Can you please share more of the node description? I would particularly like to see the part the output that lists the capacity for the node, e.g. total memory, total serial devices, total temperature devices.

Could you also share your pod manifest?

Xiol commented 11 months ago

Ahh, it's working now - of course it is.

So when I took the node description for the above, I had the pod getting a squat.ai/temperature as I was in the process of testing the USB device passthrough after failing with the serial one for some time. Switching this back to squat.ai/serial now has worked, and the device has been passed through successfully. Having also removed the USB configuration and only left the serial configuration, this also now works.

So, the problem seems to have only existed between the chair and keyboard, although what I was doing wrong is still a mystery to me. Sorry for wasting your time!

squat commented 11 months ago

I'm glad you got it working ❤️