search

squat / generic-device-plugin

A Kubernetes device plugin to schedule generic Linux devices
Apache License 2.0
208 stars 23 forks source link

Question: delay getting access to devices #46

Closed aledbf closed 1 year ago

aledbf commented 1 year ago

I see an unexpected behavior (permission denied) when I try to access a device immediately after starting a pod. Waiting for ~2s before trying to access the device solves the issue. Do you know if this is expected? Or maybe I need to do something differently? Thanks

squat commented 1 year ago

Hi @aledbf I've never seen that before! AFAIK devices are always ready to be used immediately. I'm not sure what that could be due to. Mounting the device is done by the container runtime and should be immediate.

Hmm you say that the device is present, it's just the permissions that are wrong initially. I wonder how the container runtime sets the permissions on the device and if there's a delay between mounting+starting the pod and setting permissions.

Is this with /dev/kvm?

Setting up a repro should be quite straightforward. I wonder if it has to do with your host, that specific device, or if it's common across all devices when using this plugin.

squat commented 1 year ago

Hi @aledbf do you have any update on the issue you were experiencing?

aledbf commented 1 year ago

@squat I am sorry for not updating the issue. The source of the delay was not related to the plugin but a security agent installed in a subset of cluster nodes. I'm sorry for the noise.

squat commented 1 year ago

@aledbf nice, I'm glad to hear that the plugin was working as intended and that you figured out the issue!