DualCoder
commented
3 years ago Ok, first of all, this error:
Verify all devices in group 25 are bound to vfio-<bus> or pci-stub and not already in use
start failed: QEMU exited with code 1Is reported by QEMU for almost all driver issues. In this case dmesg will also show start failed. status: 0x1. If you see any of these you should check journalctl -u nvidia-vgpu-mgr for any error reported by the driver (which you provided for approach one only).
Second of all, you are using vGPU version 13 (470.X), I have not tested version 13 myself yet (it is quite new) so there is the possibility that it contains bugs or is not working properly with vgpu_unlock. If the issue persists I would recommend trying vGPU versions 11 (450.X) or 12 (460.X) which are known to work.
And also, I am not very familiar with Proxmox, but I will do my best to answer your questions.
Regarding the first approach:
then in Proxmox conf i added the line
args: -device 'vfio-pci,sysfsdev=/sys/bus/mdev/devices/0b5fd3fb-2389-4a22-ba70-52969a26b9d5,display=off,id=hostpci0.0,bus=ich9-pcie-port-1,addr=0x0,x-pci-vendor-id=0x10de,x-pci-device-id=0x1b38,x-pci-sub-vendor-id=0x10de,x-pci-sub-device-id=0x11A0' -uuid 0b5fd3fb-2389-4a22-ba70-52969a26b9d5
Where did you get the subsystem ID (x-pci-sub-device-id=0x11A0) from? For P40-2Q (which I believe is type-47) it should be 0x11E9. Also compared to NVIDIA's example for RHEL you are adding alot of parameters, so maybe try with only vfio-pci and sysfsdev:
https://docs.nvidia.com/grid/13.0/grid-vgpu-user-guide/index.html#adding-vgpu-to-red-hat-el-kvm-vm-qemu-cli
Then the lowest level error seems to be this:
error: vmiop_log: (0x0): Timed out (6001 ms) trying to syncUnfortunately I can't say what is causing this without further debugging.
Regarding the second approach:
i create and define the mdev device like i did above and use that uuid in the Proxmox conf.
args: -uuid 0b5fd3fb-2389-4a22-ba70-52969a26b9d5
This seems unnecessary or wrong since the VM ended up using UUID 00000000-0000-0000-0000-000000000205 instead of the one you provided. What is the intended purpose of adding this UUID to the Proxmox conf?
Then i start up the machine (qm start 205) which results in this output
kvm: -device vfio-pci,sysfsdev=/sys/bus/pci/devices/0000:2b:00.0/00000000-0000-0000-0000-000000000205,id=hostpci0,bus=pci.0,addr=0x10: warning: vfio 00000000-0000-0000-0000-000000000205: Could not enable error recovery for the deviceIt works and sees the vGPU as a Tesla P40, which is correct.
I'm assuming "It works" means that the VM booted and was able to load the driver for the vGPU, this is good since it rules out any driver issue on the guest side.
But when powering down and starting it back up again gives the(more or less) same error as the first approach
You should provide the output of journalctl -u nvidia-vgpu-mgr for this case as well.
I'm clearly missing something here. So my question is, which one is the correct approach and how do i get rid of this error
It seems to me that both ways are sane and I would expect both to work. However the first case allows for more in-depth control of individual parameters (such as PCI-IDs and UUIDs), while the second case seems more "user-friendly" since it relies on the web-interface. I would prefer the first case.
wvthoog
smurmann
Hi,
i guess this is a question about how to use mdevctl properly. I folowed two tutorials on how to set up a vGPU in Proxmox. One from Craft Computing and the other found on this github page. The first one starts and defines the mdev devices prior to creating a VM, the latter uses the uuid of a previously created mdev device but uses the Proxmox web gui to assign a mdev device
My setup is:
Output of nvidia-smi
output of dmesg|grep -i vpgu
So when i follow the first approach i create the mdev device using
and define it
then in Proxmox conf i added the line
after starting up the machine (qm start 205) i've got this error
output of dmesg
output of journalctl -u nvidia-vgpu-mgr
When i try the second approach
i create and define the mdev device like i did above and use that uuid in the Proxmox conf.
then i i've assigned an mdev device to the VM using the web interface which adds the following line to the VM's conf
Then i start up the machine (qm start 205) which results in this output
It works and sees the vGPU as a Tesla P40, which is correct.
But when powering down and starting it back up again gives the(more or less) same error as the first approach
I'm clearly missing something here. So my question is, which one is the correct approach and how do i get rid of this error
Thanks