kthcloud / console

kthcloud web console
https://cloud.cbh.kth.se
MIT License
3 stars 4 forks source link

GPUs not connecting to VM smoothly #324

Closed sreyan-ghosh closed 4 months ago

sreyan-ghosh commented 4 months ago

Describe the bug GPUs not connecting to VM. The NVIDIA RTX A-Series in the Flemingburg zone is not connecting to the VMs smoothly. The other GPUs namely NVIDIA Quadro K420 and NVIDIA GRID K1 are recognised by the UNIX system as PCI devices, but nvidia-smi does not recognise them due to a signing-key mismatch.

To Reproduce Steps to reproduce the behaviour:

  1. Create a Ubuntu VM in the Flemingsburg zone in the Gold tier.
  2. Attach any of the available GPUs (except NVIDIA Quadro RTX 5000 this was not available during my testing, so its behaviour is inconclusive)
  3. After installing the latest production drivers, run the nvidia-smi command to see a failed output.
  4. Check the lspci | grep -i vga command and the lspci | grep -i nvidia command to obtain connected GPUs in the K420 and K1 cases and empty outputs in the A-Series' cases.
  5. Check the dmesg | grep -i nvidia command to see the errors occurring with the exact details.
  6. Check dpkg -l | grep -i nvidia to see that the modules are installed but running the lsmod | grep -i nvidia gives a blank output signalling that they are not loaded. Manual loading of the modules also fails.

Expected behavior Essentially on leasing a GPU through the web interface and installing the drivers and rebooting the system should flawlessly attach the GPUs. But I understand there is a scope of access for each GPU and if there is something of that sort, then it should be made clear in the web interface or in the documentation as to which tier has access to which GPUs.

System Configurations:

pierrelefevre commented 4 months ago

Hi! This sounds like a go-deploy issue https://github.com/kthcloud/go-deploy and not directly related to the console (web UI)

sreyan-ghosh commented 4 months ago

I see. I have raised the issue in go-deploy and am closing it here.