search

lxc / incus

Powerful system container and virtual machine manager
https://linuxcontainers.org/incus
Apache License 2.0
2.73k stars 225 forks source link

Ubuntu 24.04LTS host+guest hangs (at start) when a Nvidia gpu is added to the guest vm. #858

Closed pythonmobile closed 5 months ago

pythonmobile commented 6 months ago

Required information

config: {} api_extensions:

Issue description

a 24.04LTS VM fails to start if i add gpu. These are the commands to reproduce the issue:

incus admin init --minimal
incus launch images:ubuntu/24.04 ubuntu-vm --vm
incus stop ubuntu-vm
incus config device add ubuntu-vm gpu gpu gputype=physical pci=0000:01:00.0
incus start ubuntu-vm # Hangs forever

The command : # incus info --show-log ubuntu-vm
Name: ubuntu-vm Status: STOPPED Type: virtual-machine Architecture: x86_64 Created: 2024/05/08 21:20 EDT Last Used: 2024/05/08 21:20 EDT Error: open /var/log/incus/ubuntu-vm/qemu.log: no such file or directory

Information to attach

Nothing looks relevant except:

[ 1575.928610] incusbr0: port 1(tap23eea69f) entered blocking state
[ 1575.928620] incusbr0: port 1(tap23eea69f) entered disabled state
[ 1575.928638] tap23eea69f: entered allmulticast mode
[ 1575.928757] tap23eea69f: entered promiscuous mode
[ 1576.067184] VFIO - User Level meta-driver version: 0.3
[ 1576.086626] NVRM: Attempting to remove device 0000:01:00.0 with non-zero usage count!```

 - [x] Container log (`incus info NAME --show-log`)

Same as shown in above command - incus info --show-log ubuntu-vm

 - [x] Container configuration (`incus config show NAME --expanded`)
```architecture: x86_64
config:
  image.architecture: amd64
  image.description: Ubuntu noble amd64 (20240507_07:42)
  image.os: Ubuntu
  image.release: noble
  image.requirements.cgroup: v2
  image.serial: "20240507_07:42"
  image.type: disk-kvm.img
  image.variant: default
  volatile.base_image: 6993d53a868865f2b9dfe40c8fa9f260b145ad6a0d6cf5cffbfeef1ee03a8e50
  volatile.cloud-init.instance-id: b23617b2-65a0-4329-90fa-2a73d3dc7580
  volatile.eth0.host_name: tap23eea69f
  volatile.eth0.hwaddr: 00:16:3e:ee:da:56
  volatile.last_state.power: STOPPED
  volatile.last_state.ready: "false"
  volatile.uuid: 49401d0f-181a-4cc9-ab17-cebd18a1f8f4
  volatile.uuid.generation: 49401d0f-181a-4cc9-ab17-cebd18a1f8f4
  volatile.vsock_id: "110305049"
devices:
  eth0:
    name: eth0
    network: incusbr0
    type: nic
  gpu:
    gputype: physical
    pci: "0000:01:00.0"
    type: gpu
  root:
    path: /
    pool: default
    type: disk
ephemeral: false
profiles:
- default
stateful: false
description: ""```

 - [x] Main daemon log (at /var/log/incus/incusd.log)
 nothing with start or delete commands
 - [ ] Output of the client with --debug
The command: incus --debug ubuntu-vm start # hangs on the following log 
DEBUG  [2024-05-08T21:34:44-04:00] Sending request to Incus                      etag= method=GET url="http://unix.socket/1.0/operations/61bfe7a1-70d4-4fc9-be50-566540734303"
DEBUG  [2024-05-08T21:34:44-04:00] Got response struct from Incus
DEBUG  [2024-05-08T21:34:44-04:00]
    {
        "id": "61bfe7a1-70d4-4fc9-be50-566540734303",
        "class": "task",
        "description": "Starting instance",
        "created_at": "2024-05-08T21:34:44.662064351-04:00",
        "updated_at": "2024-05-08T21:34:44.662064351-04:00",
        "status": "Running",
        "status_code": 103,
        "resources": {
            "instances": [
                "/1.0/instances/ubuntu-vm"
            ]
        },
        "metadata": null,
        "may_cancel": false,
        "err": "",
        "location": "none"
    }

The command: incus --debug ubuntu-vm delete  hangs on the following end of log
DEBUG  [2024-05-08T21:35:53-04:00] Sending request to Incus                      etag= method=GET url="http://unix.socket/1.0/operations/5ef81344-a494-4d09-a5a7-34142f51254e"
DEBUG  [2024-05-08T21:35:53-04:00] Got response struct from Incus
DEBUG  [2024-05-08T21:35:53-04:00]
    {
        "id": "5ef81344-a494-4d09-a5a7-34142f51254e",
        "class": "task",
        "description": "Deleting instance",
        "created_at": "2024-05-08T21:35:53.465845247-04:00",
        "updated_at": "2024-05-08T21:35:53.465845247-04:00",
        "status": "Running",
        "status_code": 103,
        "resources": {
            "instances": [
                "/1.0/instances/ubuntu-vm"
            ]
        },
        "metadata": null,
        "may_cancel": false,
        "err": "",
        "location": "none"
    }

 - [x] Output of the daemon with --debug (alternatively output of `incus monitor --pretty` while reproducing the issue)
```DEBUG  [2024-05-08T21:40:12-04:00] Event listener server handler started         id=259d4acc-02ef-421e-9ebc-4217e703fbf6 local=/var/lib/incus/unix.socket remote=@
DEBUG  [2024-05-08T21:40:15-04:00] Event listener server handler stopped         listener=c4aeb100-3090-4938-a94a-e61f97b69ccf local=/var/lib/incus/unix.socket remote=@
DEBUG  [2024-05-08T21:40:18-04:00] Handling API request                          ip=@ method=GET protocol=unix url=/1.0 username=root
DEBUG  [2024-05-08T21:40:18-04:00] Handling API request                          ip=@ method=GET protocol=unix url=/1.0/events username=root
DEBUG  [2024-05-08T21:40:18-04:00] Event listener server handler started         id=6e922280-920f-4aec-8896-dca50fb9b8ac local=/var/lib/incus/unix.socket remote=@
DEBUG  [2024-05-08T21:40:18-04:00] Handling API request                          ip=@ method=PUT protocol=unix url=/1.0/instances/ubuntu-vm/state username=root
DEBUG  [2024-05-08T21:40:18-04:00] Started operation                             class=task description="Stopping instance" operation=b743d98c-6ad2-45ff-afa4-c52049f5db78 project=default
INFO   [2024-05-08T21:40:18-04:00] ID: b743d98c-6ad2-45ff-afa4-c52049f5db78, Class: task, Description: Stopping instance  CreatedAt="2024-05-08 21:40:18.300714719 -0400 EDT" Err= Location=none MayCancel=false Metadata="map[]" Resources="map[instances:[/1.0/instances/ubuntu-vm]]" Status=Pending StatusCode=Pending UpdatedAt="2024-05-08 21:40:18.300714719 -0400 EDT"
DEBUG  [2024-05-08T21:40:18-04:00] New operation                                 class=task description="Stopping instance" operation=b743d98c-6ad2-45ff-afa4-c52049f5db78 project=default
DEBUG  [2024-05-08T21:40:18-04:00] Shutdown started                              instance=ubuntu-vm instanceType=virtual-machine project=default timeout=10m0s
DEBUG  [2024-05-08T21:40:18-04:00] Shutdown finished                             instance=ubuntu-vm instanceType=virtual-machine project=default timeout=10m0s
INFO   [2024-05-08T21:40:18-04:00] ID: b743d98c-6ad2-45ff-afa4-c52049f5db78, Class: task, Description: Stopping instance  CreatedAt="2024-05-08 21:40:18.300714719 -0400 EDT" Err="The instance is already stopped" Location=none MayCancel=false Metadata="map[]" Resources="map[instances:[/1.0/instances/ubuntu-vm]]" Status=Failure StatusCode=Failure UpdatedAt="2024-05-08 21:40:18.300714719 -0400 EDT"
DEBUG  [2024-05-08T21:40:18-04:00] Failure for operation                         class=task description="Stopping instance" err="The instance is already stopped" operation=b743d98c-6ad2-45ff-afa4-c52049f5db78 project=default
INFO   [2024-05-08T21:40:18-04:00] ID: b743d98c-6ad2-45ff-afa4-c52049f5db78, Class: task, Description: Stopping instance  CreatedAt="2024-05-08 21:40:18.300714719 -0400 EDT" Err= Location=none MayCancel=false Metadata="map[]" Resources="map[instances:[/1.0/instances/ubuntu-vm]]" Status=Running StatusCode=Running UpdatedAt="2024-05-08 21:40:18.300714719 -0400 EDT"
DEBUG  [2024-05-08T21:40:18-04:00] Handling API request                          ip=@ method=GET protocol=unix url=/1.0/operations/b743d98c-6ad2-45ff-afa4-c52049f5db78 username=root
DEBUG  [2024-05-08T21:40:18-04:00] Event listener server handler stopped         listener=6e922280-920f-4aec-8896-dca50fb9b8ac local=/var/lib/incus/unix.socket remote=@
DEBUG  [2024-05-08T21:40:19-04:00] Handling API request                          ip=@ method=GET protocol=unix url=/1.0 username=root
DEBUG  [2024-05-08T21:40:19-04:00] Handling API request                          ip=@ method=GET protocol=unix url=/1.0/instances/ubuntu-vm username=root
DEBUG  [2024-05-08T21:40:19-04:00] Handling API request                          ip=@ method=GET protocol=unix url=/1.0/instances/ubuntu-vm username=root
DEBUG  [2024-05-08T21:40:19-04:00] Handling API request                          ip=@ method=GET protocol=unix url=/1.0/events username=root
DEBUG  [2024-05-08T21:40:19-04:00] Event listener server handler started         id=9225b0e9-abb2-4c26-b07f-1390bc2b3a09 local=/var/lib/incus/unix.socket remote=@
DEBUG  [2024-05-08T21:40:19-04:00] Handling API request                          ip=@ method=DELETE protocol=unix url=/1.0/instances/ubuntu-vm username=root
INFO   [2024-05-08T21:40:19-04:00] ID: 9ab0810d-8b82-4cef-b513-f8561aca8432, Class: task, Description: Deleting instance  CreatedAt="2024-05-08 21:40:19.587499443 -0400 EDT" Err= Location=none MayCancel=false Metadata="map[]" Resources="map[instances:[/1.0/instances/ubuntu-vm]]" Status=Running StatusCode=Running UpdatedAt="2024-05-08 21:40:19.587499443 -0400 EDT"
DEBUG  [2024-05-08T21:40:19-04:00] Started operation                             class=task description="Deleting instance" operation=9ab0810d-8b82-4cef-b513-f8561aca8432 project=default
DEBUG  [2024-05-08T21:40:19-04:00] New operation                                 class=task description="Deleting instance" operation=9ab0810d-8b82-4cef-b513-f8561aca8432 project=default
INFO   [2024-05-08T21:40:19-04:00] ID: 9ab0810d-8b82-4cef-b513-f8561aca8432, Class: task, Description: Deleting instance  CreatedAt="2024-05-08 21:40:19.587499443 -0400 EDT" Err= Location=none MayCancel=false Metadata="map[]" Resources="map[instances:[/1.0/instances/ubuntu-vm]]" Status=Pending StatusCode=Pending UpdatedAt="2024-05-08 21:40:19.587499443 -0400 EDT"
DEBUG  [2024-05-08T21:40:19-04:00] Handling API request                          ip=@ method=GET protocol=unix url=/1.0/operations/9ab0810d-8b82-4cef-b513-f8561aca8432 username=root
stgraber commented 6 months ago

Not seeing any sign of an issue on the Incus side, looks like it's been asking the kernel to switch driver ahead of passing the GPU through vfio but the kernel is refusing/hanging due to the device being in use for some reason?

Just to check, can you confirm that:

Can you show the output of incus query /1.0/resources?

pythonmobile commented 6 months ago

I boot Ubuntu in text mode with the login prompt. 

$ lspci | grep -E "VGA|3D|Display"
0000:00:02.0 Display controller: Intel Corporation Raptor Lake-S GT1 [UHD Graphics 770] (rev 04)
0000:01:00.0 VGA compatible controller: NVIDIA Corporation AD102 [GeForce RTX 4090] (rev a1)

I don't see VT-d here or in the BIOS (will try to figure that out today). The system is a Dell XPS 8960.

For your third bullet: 

# cat /proc/cmdline
BOOT_IMAGE=/boot/vmlinuz-6.8.0-31-generic root=UUID=cd99a160-bd3e-469d-812e-516d6cbdf49b ro quiet splash iommu=pt intel_iommu=on systemd.unit=multi-user.target vt.handoff=7

Can you show the output of incus query /1.0/resources?

incus query /1.0/resources

{
    "cpu": {
        "architecture": "x86_64",
        "sockets": [
            {
                "cache": [
                    {
                        "level": 1,
                        "size": 49152,
                        "type": "Data"
                    },
                    {
                        "level": 1,
                        "size": 32768,
                        "type": "Instruction"
                    },
                    {
                        "level": 2,
                        "size": 2097152,
                        "type": "Unified"
                    },
                    {
                        "level": 3,
                        "size": 31457280,
                        "type": "Unified"
                    }
                ],
                "cores": [
                    {
                        "core": 0,
                        "die": 0,
                        "frequency": 1099,
                        "threads": [
                            {
                                "id": 0,
                                "isolated": false,
                                "numa_node": 0,
                                "online": true,
                                "thread": 0
                            },
                            {
                                "id": 1,
                                "isolated": false,
                                "numa_node": 0,
                                "online": true,
                                "thread": 1
                            }
                        ]
                    },
                    {
                        "core": 4,
                        "die": 0,
                        "frequency": 799,
                        "threads": [
                            {
                                "id": 2,
                                "isolated": false,
                                "numa_node": 0,
                                "online": true,
                                "thread": 0
                            },
                            {
                                "id": 3,
                                "isolated": false,
                                "numa_node": 0,
                                "online": true,
                                "thread": 1
                            }
                        ]
                    },
                    {
                        "core": 8,
                        "die": 0,
                        "frequency": 800,
                        "threads": [
                            {
                                "id": 4,
                                "isolated": false,
                                "numa_node": 0,
                                "online": true,
                                "thread": 0
                            },
                            {
                                "id": 5,
                                "isolated": false,
                                "numa_node": 0,
                                "online": true,
                                "thread": 1
                            }
                        ]
                    },
                    {
                        "core": 12,
                        "die": 0,
                        "frequency": 1055,
                        "threads": [
                            {
                                "id": 6,
                                "isolated": false,
                                "numa_node": 0,
                                "online": true,
                                "thread": 0
                            },
                            {
                                "id": 7,
                                "isolated": false,
                                "numa_node": 0,
                                "online": true,
                                "thread": 1
                            }
                        ]
                    },
                    {
                        "core": 16,
                        "die": 0,
                        "frequency": 1150,
                        "threads": [
                            {
                                "id": 8,
                                "isolated": false,
                                "numa_node": 0,
                                "online": true,
                                "thread": 0
                            },
                            {
                                "id": 9,
                                "isolated": false,
                                "numa_node": 0,
                                "online": true,
                                "thread": 1
                            }
                        ]
                    },
                    {
                        "core": 20,
                        "die": 0,
                        "frequency": 1079,
                        "threads": [
                            {
                                "id": 10,
                                "isolated": false,
                                "numa_node": 0,
                                "online": true,
                                "thread": 0
                            },
                            {
                                "id": 11,
                                "isolated": false,
                                "numa_node": 0,
                                "online": true,
                                "thread": 1
                            }
                        ]
                    },
                    {
                        "core": 24,
                        "die": 0,
                        "frequency": 1043,
                        "threads": [
                            {
                                "id": 12,
                                "isolated": false,
                                "numa_node": 0,
                                "online": true,
                                "thread": 0
                            },
                            {
                                "id": 13,
                                "isolated": false,
                                "numa_node": 0,
                                "online": true,
                                "thread": 1
                            }
                        ]
                    },
                    {
                        "core": 28,
                        "die": 0,
                        "frequency": 800,
                        "threads": [
                            {
                                "id": 14,
                                "isolated": false,
                                "numa_node": 0,
                                "online": true,
                                "thread": 0
                            },
                            {
                                "id": 15,
                                "isolated": false,
                                "numa_node": 0,
                                "online": true,
                                "thread": 1
                            }
                        ]
                    },
                    {
                        "core": 32,
                        "die": 0,
                        "frequency": 1399,
                        "threads": [
                            {
                                "id": 16,
                                "isolated": false,
                                "numa_node": 0,
                                "online": true,
                                "thread": 0
                            }
                        ]
                    },
                    {
                        "core": 33,
                        "die": 0,
                        "frequency": 1401,
                        "threads": [
                            {
                                "id": 17,
                                "isolated": false,
                                "numa_node": 0,
                                "online": true,
                                "thread": 0
                            }
                        ]
                    },
                    {
                        "core": 34,
                        "die": 0,
                        "frequency": 1400,
                        "threads": [
                            {
                                "id": 18,
                                "isolated": false,
                                "numa_node": 0,
                                "online": true,
                                "thread": 0
                            }
                        ]
                    },
                    {
                        "core": 35,
                        "die": 0,
                        "frequency": 800,
                        "threads": [
                            {
                                "id": 19,
                                "isolated": false,
                                "numa_node": 0,
                                "online": true,
                                "thread": 0
                            }
                        ]
                    },
                    {
                        "core": 36,
                        "die": 0,
                        "frequency": 1400,
                        "threads": [
                            {
                                "id": 20,
                                "isolated": false,
                                "numa_node": 0,
                                "online": true,
                                "thread": 0
                            }
                        ]
                    },
                    {
                        "core": 37,
                        "die": 0,
                        "frequency": 1397,
                        "threads": [
                            {
                                "id": 21,
                                "isolated": false,
                                "numa_node": 0,
                                "online": true,
                                "thread": 0
                            }
                        ]
                    },
                    {
                        "core": 38,
                        "die": 0,
                        "frequency": 1400,
                        "threads": [
                            {
                                "id": 22,
                                "isolated": false,
                                "numa_node": 0,
                                "online": true,
                                "thread": 0
                            }
                        ]
                    },
                    {
                        "core": 39,
                        "die": 0,
                        "frequency": 800,
                        "threads": [
                            {
                                "id": 23,
                                "isolated": false,
                                "numa_node": 0,
                                "online": true,
                                "thread": 0
                            }
                        ]
                    }
                ],
                "frequency": 1113,
                "frequency_minimum": 800,
                "frequency_turbo": 5100,
                "name": "13th Gen Intel(R) Core(TM) i7-13700",
                "socket": 0,
                "vendor": "GenuineIntel"
            }
        ],
        "total": 24
    },
    "gpu": {
        "cards": [
            {
                "driver": "simple-framebuffer",
                "driver_version": "6.8.0-31-generic",
                "drm": {
                    "card_device": "226:0",
                    "card_name": "card0",
                    "control_device": "226:0",
                    "control_name": "controlD64",
                    "id": 0
                },
                "numa_node": 0
            },
            {
                "driver": "i915",
                "driver_version": "6.8.0-31-generic",
                "drm": {
                    "card_device": "226:1",
                    "card_name": "card1",
                    "control_device": "226:1",
                    "control_name": "controlD65",
                    "id": 1,
                    "render_device": "226:128",
                    "render_name": "renderD128"
                },
                "numa_node": 0,
                "pci_address": "0000:00:02.0",
                "product": "Raptor Lake-S GT1 [UHD Graphics 770]",
                "product_id": "a780",
                "sriov": {
                    "current_vfs": 0,
                    "maximum_vfs": 7,
                    "vfs": null
                },
                "vendor": "Intel Corporation",
                "vendor_id": "8086"
            },
            {
                "driver": "nvidia",
                "driver_version": "535.171.04",
                "drm": {
                    "card_device": "226:2",
                    "card_name": "card2",
                    "control_device": "226:2",
                    "control_name": "controlD66",
                    "id": 2,
                    "render_device": "226:129",
                    "render_name": "renderD129"
                },
                "numa_node": 0,
                "nvidia": {
                    "architecture": "8.9",
                    "brand": "GeForce",
                    "card_device": "195:0",
                    "card_name": "nvidia0",
                    "cuda_version": "12.2",
                    "model": "NVIDIA GeForce RTX 4090",
                    "nvrm_version": "535.171.04",
                    "uuid": "GPU-f2effe4a-f03d-0a88-097b-cd67c655ed90"
                },
                "pci_address": "0000:01:00.0",
                "product": "AD102 [GeForce RTX 4090]",
                "product_id": "2684",
                "vendor": "NVIDIA Corporation",
                "vendor_id": "10de"
            }
        ],
        "total": 3
    },
    "memory": {
        "hugepages_size": 2097152,
        "hugepages_total": 0,
        "hugepages_used": 0,
        "nodes": [
            {
                "hugepages_total": 0,
                "hugepages_used": 0,
                "numa_node": 0,
                "total": 34225520640,
                "used": 2047164416
            }
        ],
        "total": 34225520640,
        "used": 1917054976
    },
    "network": {
        "cards": [
            {
                "driver": "r8169",
                "driver_version": "6.8.0-31-generic",
                "firmware_version": "rtl8125b-2_0.0.2 07/13/20",
                "numa_node": 0,
                "pci_address": "0000:03:00.0",
                "ports": [
                    {
                        "address": "4c:d7:17:7e:d4:c6",
                        "auto_negotiation": true,
                        "id": "enp3s0",
                        "link_detected": true,
                        "link_duplex": "full",
                        "link_speed": 1000,
                        "port": 0,
                        "port_type": "twisted pair",
                        "protocol": "ethernet",
                        "supported_modes": [
                            "10baseT/Half",
                            "10baseT/Full",
                            "100baseT/Half",
                            "100baseT/Full",
                            "1000baseT/Full",
                            "2500baseT/Full"
                        ],
                        "supported_ports": [
                            "twisted pair",
                            "media-independent"
                        ],
                        "transceiver_type": "external"
                    }
                ],
                "product": "Killer E3000 2.5GbE Controller",
                "product_id": "3000",
                "vendor": "Realtek Semiconductor Co., Ltd.",
                "vendor_id": "10ec"
            },
            {
                "driver": "iwlwifi",
                "driver_version": "6.8.0-31-generic",
                "firmware_version": "86.fb5c9aeb.0 so-a0-gf-a0-86.uc",
                "numa_node": 0,
                "pci_address": "0000:00:14.3",
                "ports": [
                    {
                        "address": "f0:d4:15:df:23:57",
                        "auto_negotiation": false,
                        "id": "wlo1",
                        "link_detected": false,
                        "port": 0,
                        "protocol": "ethernet"
                    }
                ],
                "product": "Alder Lake-S PCH CNVi WiFi",
                "product_id": "7af0",
                "vendor": "Intel Corporation",
                "vendor_id": "8086"
            }
        ],
        "total": 2
    },
    "pci": {
        "devices": [
            {
                "driver": "",
                "driver_version": "",
                "iommu_group": 1,
                "numa_node": 0,
                "pci_address": "0000:00:00.0",
                "product": "",
                "product_id": "a703",
                "vendor": "Intel Corporation",
                "vendor_id": "8086",
                "vpd": {}
            },
            {
                "driver": "pcieport",
                "driver_version": "6.8.0-31-generic",
                "iommu_group": 2,
                "numa_node": 0,
                "pci_address": "0000:00:01.0",
                "product": "",
                "product_id": "a70d",
                "vendor": "Intel Corporation",
                "vendor_id": "8086",
                "vpd": {}
            },
            {
                "driver": "i915",
                "driver_version": "6.8.0-31-generic",
                "iommu_group": 0,
                "numa_node": 0,
                "pci_address": "0000:00:02.0",
                "product": "Raptor Lake-S GT1 [UHD Graphics 770]",
                "product_id": "a780",
                "vendor": "Intel Corporation",
                "vendor_id": "8086",
                "vpd": {}
            },
            {
                "driver": "proc_thermal_pci",
                "driver_version": "6.8.0-31-generic",
                "iommu_group": 3,
                "numa_node": 0,
                "pci_address": "0000:00:04.0",
                "product": "Raptor Lake Dynamic Platform and Thermal Framework Processor Participant",
                "product_id": "a71d",
                "vendor": "Intel Corporation",
                "vendor_id": "8086",
                "vpd": {}
            },
            {
                "driver": "",
                "driver_version": "",
                "iommu_group": 4,
                "numa_node": 0,
                "pci_address": "0000:00:08.0",
                "product": "GNA Scoring Accelerator module",
                "product_id": "a74f",
                "vendor": "Intel Corporation",
                "vendor_id": "8086",
                "vpd": {}
            },
            {
                "driver": "vmd",
                "driver_version": "0.6",
                "iommu_group": 5,
                "numa_node": 0,
                "pci_address": "0000:00:0e.0",
                "product": "Volume Management Device NVMe RAID Controller Intel Corporation",
                "product_id": "a77f",
                "vendor": "Intel Corporation",
                "vendor_id": "8086",
                "vpd": {}
            },
            {
                "driver": "xhci_hcd",
                "driver_version": "6.8.0-31-generic",
                "iommu_group": 6,
                "numa_node": 0,
                "pci_address": "0000:00:14.0",
                "product": "Alder Lake-S PCH USB 3.2 Gen 2x2 XHCI Controller",
                "product_id": "7ae0",
                "vendor": "Intel Corporation",
                "vendor_id": "8086",
                "vpd": {}
            },
            {
                "driver": "",
                "driver_version": "",
                "iommu_group": 6,
                "numa_node": 0,
                "pci_address": "0000:00:14.2",
                "product": "Alder Lake-S PCH Shared SRAM",
                "product_id": "7aa7",
                "vendor": "Intel Corporation",
                "vendor_id": "8086",
                "vpd": {}
            },
            {
                "driver": "iwlwifi",
                "driver_version": "6.8.0-31-generic",
                "iommu_group": 7,
                "numa_node": 0,
                "pci_address": "0000:00:14.3",
                "product": "Alder Lake-S PCH CNVi WiFi",
                "product_id": "7af0",
                "vendor": "Intel Corporation",
                "vendor_id": "8086",
                "vpd": {}
            },
            {
                "driver": "intel-lpss",
                "driver_version": "6.8.0-31-generic",
                "iommu_group": 8,
                "numa_node": 0,
                "pci_address": "0000:00:15.0",
                "product": "Alder Lake-S PCH Serial IO I2C Controller #0",
                "product_id": "7acc",
                "vendor": "Intel Corporation",
                "vendor_id": "8086",
                "vpd": {}
            },
            {
                "driver": "mei_me",
                "driver_version": "6.8.0-31-generic",
                "iommu_group": 9,
                "numa_node": 0,
                "pci_address": "0000:00:16.0",
                "product": "Alder Lake-S PCH HECI Controller #1",
                "product_id": "7ae8",
                "vendor": "Intel Corporation",
                "vendor_id": "8086",
                "vpd": {}
            },
            {
                "driver": "",
                "driver_version": "",
                "iommu_group": 10,
                "numa_node": 0,
                "pci_address": "0000:00:17.0",
                "product": "RST VMD Managed Controller",
                "product_id": "09ab",
                "vendor": "Intel Corporation",
                "vendor_id": "8086",
                "vpd": {}
            },
            {
                "driver": "pcieport",
                "driver_version": "6.8.0-31-generic",
                "iommu_group": 11,
                "numa_node": 0,
                "pci_address": "0000:00:1b.0",
                "product": "",
                "product_id": "7ac3",
                "vendor": "Intel Corporation",
                "vendor_id": "8086",
                "vpd": {}
            },
            {
                "driver": "pcieport",
                "driver_version": "6.8.0-31-generic",
                "iommu_group": 12,
                "numa_node": 0,
                "pci_address": "0000:00:1c.0",
                "product": "Alder Lake-S PCH PCI Express Root Port #5",
                "product_id": "7abc",
                "vendor": "Intel Corporation",
                "vendor_id": "8086",
                "vpd": {}
            },
            {
                "driver": "",
                "driver_version": "",
                "iommu_group": 13,
                "numa_node": 0,
                "pci_address": "0000:00:1f.0",
                "product": "Z690 Chipset LPC/eSPI Controller",
                "product_id": "7a84",
                "vendor": "Intel Corporation",
                "vendor_id": "8086",
                "vpd": {}
            },
            {
                "driver": "snd_hda_intel",
                "driver_version": "6.8.0-31-generic",
                "iommu_group": 13,
                "numa_node": 0,
                "pci_address": "0000:00:1f.3",
                "product": "Alder Lake-S HD Audio Controller",
                "product_id": "7ad0",
                "vendor": "Intel Corporation",
                "vendor_id": "8086",
                "vpd": {}
            },
            {
                "driver": "i801_smbus",
                "driver_version": "6.8.0-31-generic",
                "iommu_group": 13,
                "numa_node": 0,
                "pci_address": "0000:00:1f.4",
                "product": "Alder Lake-S PCH SMBus Controller",
                "product_id": "7aa3",
                "vendor": "Intel Corporation",
                "vendor_id": "8086",
                "vpd": {}
            },
            {
                "driver": "intel-spi",
                "driver_version": "6.8.0-31-generic",
                "iommu_group": 13,
                "numa_node": 0,
                "pci_address": "0000:00:1f.5",
                "product": "Alder Lake-S PCH SPI Controller",
                "product_id": "7aa4",
                "vendor": "Intel Corporation",
                "vendor_id": "8086",
                "vpd": {}
            },
            {
                "driver": "nvidia",
                "driver_version": "535.171.04",
                "iommu_group": 14,
                "numa_node": 0,
                "pci_address": "0000:01:00.0",
                "product": "AD102 [GeForce RTX 4090]",
                "product_id": "2684",
                "vendor": "NVIDIA Corporation",
                "vendor_id": "10de",
                "vpd": {}
            },
            {
                "driver": "snd_hda_intel",
                "driver_version": "6.8.0-31-generic",
                "iommu_group": 14,
                "numa_node": 0,
                "pci_address": "0000:01:00.1",
                "product": "AD102 High Definition Audio Controller",
                "product_id": "22ba",
                "vendor": "NVIDIA Corporation",
                "vendor_id": "10de",
                "vpd": {}
            },
            {
                "driver": "rtsx_pci",
                "driver_version": "6.8.0-31-generic",
                "iommu_group": 15,
                "numa_node": 0,
                "pci_address": "0000:02:00.0",
                "product": "RTS525A PCI Express Card Reader",
                "product_id": "525a",
                "vendor": "Realtek Semiconductor Co., Ltd.",
                "vendor_id": "10ec",
                "vpd": {}
            },
            {
                "driver": "r8169",
                "driver_version": "6.8.0-31-generic",
                "iommu_group": 16,
                "numa_node": 0,
                "pci_address": "0000:03:00.0",
                "product": "Killer E3000 2.5GbE Controller",
                "product_id": "3000",
                "vendor": "Realtek Semiconductor Co., Ltd.",
                "vendor_id": "10ec",
                "vpd": {}
            },
            {
                "driver": "ahci",
                "driver_version": "3.0",
                "iommu_group": 5,
                "numa_node": 0,
                "pci_address": "10000:e0:17.0",
                "product": "Alder Lake-S PCH SATA Controller [AHCI Mode]",
                "product_id": "7ae2",
                "vendor": "Intel Corporation",
                "vendor_id": "8086",
                "vpd": {}
            },
            {
                "driver": "",
                "driver_version": "",
                "iommu_group": 5,
                "numa_node": 0,
                "pci_address": "10000:e0:1b.0",
                "product": "RST VMD Managed Controller",
                "product_id": "09ab",
                "vendor": "Intel Corporation",
                "vendor_id": "8086",
                "vpd": {}
            },
            {
                "driver": "pcieport",
                "driver_version": "6.8.0-31-generic",
                "iommu_group": 5,
                "numa_node": 0,
                "pci_address": "10000:e0:1b.4",
                "product": "Alder Lake-S PCH PCI Express Root Port #21",
                "product_id": "7ac4",
                "vendor": "Intel Corporation",
                "vendor_id": "8086",
                "vpd": {}
            },
            {
                "driver": "nvme",
                "driver_version": "1.0",
                "iommu_group": 5,
                "numa_node": 0,
                "pci_address": "10000:e1:00.0",
                "product": "NVMe SSD Controller PM9A1/PM9A3/980PRO",
                "product_id": "a80a",
                "vendor": "Samsung Electronics Co Ltd",
                "vendor_id": "144d",
                "vpd": {}
            }
        ],
        "total": 26
    },
    "storage": {
        "disks": [
            {
                "block_size": 512,
                "device": "259:0",
                "device_id": "nvme-eui.36483330581143050025384600000001",
                "device_path": "pci-0000:00:0e.0-pci-10000:e1:00.0-nvme-1",
                "firmware_version": "36310229",
                "id": "nvme0n1",
                "model": "PM9A1 NVMe Samsung 512GB",
                "numa_node": 0,
                "partitions": [
                    {
                        "device": "259:1",
                        "id": "nvme0n1p1",
                        "partition": 1,
                        "read_only": false,
                        "size": 209715200
                    },
                    {
                        "device": "259:2",
                        "id": "nvme0n1p2",
                        "partition": 2,
                        "read_only": false,
                        "size": 134217728
                    },
                    {
                        "device": "259:3",
                        "id": "nvme0n1p3",
                        "partition": 3,
                        "read_only": false,
                        "size": 170422960128
                    },
                    {
                        "device": "259:4",
                        "id": "nvme0n1p4",
                        "partition": 4,
                        "read_only": false,
                        "size": 1038090240
                    },
                    {
                        "device": "259:5",
                        "id": "nvme0n1p5",
                        "partition": 5,
                        "read_only": false,
                        "size": 24051187712
                    },
                    {
                        "device": "259:6",
                        "id": "nvme0n1p6",
                        "partition": 6,
                        "read_only": false,
                        "size": 1665138688
                    },
                    {
                        "device": "259:7",
                        "id": "nvme0n1p7",
                        "partition": 7,
                        "read_only": false,
                        "size": 314572800000
                    }
                ],
                "pci_address": "10000:e1:00.0",
                "read_only": false,
                "removable": false,
                "rpm": 0,
                "serial": "S6H3NF0X114305",
                "size": 512110190592,
                "type": "nvme",
                "wwn": "eui.36483330581143050025384600000001"
            }
        ],
        "total": 8
    },
    "system": {
        "chassis": {
            "serial": "9J3SF24",
            "type": "Desktop",
            "vendor": "Dell Inc.",
            "version": "Not Specified"
        },
        "family": "XPS",
        "firmware": {
            "date": "12/11/2023",
            "vendor": "Dell Inc.",
            "version": "2.3.0"
        },
        "motherboard": {
            "product": "0XD433",
            "serial": ".9J3SF24.CNFCW003B3018Z.",
            "vendor": "Dell Inc.",
            "version": "A01"
        },
        "product": "XPS 8960",
        "serial": "9J3SF24",
        "sku": "0BC0",
        "type": "physical",
        "uuid": "4c4c4544-004a-3310-8053-b9c04f463234",
        "vendor": "Dell Inc.",
        "version": ""
    },
    "usb": {
        "devices": [
            {
                "bus_address": 1,
                "device_address": 4,
                "interfaces": [
                    {
                        "class": "Wireless",
                        "class_id": 224,
                        "driver": "btusb",
                        "driver_version": "0.8",
                        "number": 0,
                        "subclass": "Radio Frequency",
                        "subclass_id": 1
                    },
                    {
                        "class": "Wireless",
                        "class_id": 224,
                        "driver": "btusb",
                        "driver_version": "0.8",
                        "number": 1,
                        "subclass": "Radio Frequency",
                        "subclass_id": 1
                    }
                ],
                "product": "AX211 Bluetooth",
                "product_id": "0033",
                "serial": "",
                "speed": 12,
                "vendor": "Intel Corp.",
                "vendor_id": "8087"
            },
            {
                "bus_address": 1,
                "device_address": 2,
                "interfaces": [
                    {
                        "class": "Human Interface Device",
                        "class_id": 3,
                        "driver": "usbhid",
                        "driver_version": "6.8.0-31-generic",
                        "number": 0,
                        "subclass": "Boot Interface Subclass",
                        "subclass_id": 1
                    },
                    {
                        "class": "Human Interface Device",
                        "class_id": 3,
                        "driver": "usbhid",
                        "driver_version": "6.8.0-31-generic",
                        "number": 1,
                        "subclass": "",
                        "subclass_id": 0
                    }
                ],
                "product": "USB Keyboard",
                "product_id": "1603",
                "serial": "",
                "speed": 1.5,
                "vendor": "Holtek Semiconductor, Inc.",
                "vendor_id": "04d9"
            },
            {
                "bus_address": 1,
                "device_address": 3,
                "interfaces": [
                    {
                        "class": "Human Interface Device",
                        "class_id": 3,
                        "driver": "usbhid",
                        "driver_version": "6.8.0-31-generic",
                        "number": 0,
                        "subclass": "Boot Interface Subclass",
                        "subclass_id": 1
                    },
                    {
                        "class": "Human Interface Device",
                        "class_id": 3,
                        "driver": "usbhid",
                        "driver_version": "6.8.0-31-generic",
                        "number": 1,
                        "subclass": "",
                        "subclass_id": 0
                    }
                ],
                "product": "USB Receiver",
                "product_id": "c52f",
                "serial": "",
                "speed": 12,
                "vendor": "Logitech, Inc.",
                "vendor_id": "046d"
            }
        ],
        "total": 3
    }
}
stgraber commented 6 months ago

Try removing the NVIDIA driver or at least blacklisting it so it doesn't get loaded on boot.

Since the card is getting moved to the VM, the driver doesn't need to be present on the host and may be causing the issue here.

stgraber commented 6 months ago

You could put:

blacklist nvidia
blacklist nouveau

In a .conf file alongside the others in /etc/modprobe.d/ and then run update-initramfs -u.

Rebooting after that should have the system boot with no driver tied to the card, avoiding any potential situation where it can't be released by the host.

pythonmobile commented 6 months ago

Thanks! That solved the issue. Now I can run ubuntu 24.04LTS vm inside ubuntu 24.04LTS base OS if I install the nvidia-drivers inside the Guest VM (This also hangs at times randomly : see below). But when I try to run ubuntu 22.04LTS with the same driver installation (version 535), I get the following error:

+-------------+---------+------+------+-----------------+-----------+
|    NAME     |  STATE  | IPV4 | IPV6 |      TYPE       | SNAPSHOTS |
+-------------+---------+------+------+-----------------+-----------+
| ubuntu22-vm | STOPPED |      |      | VIRTUAL-MACHINE | 0         |
+-------------+---------+------+------+-----------------+-----------+
| ubuntu-vm   | STOPPED |      |      | VIRTUAL-MACHINE | 0         |
+-------------+---------+------+------+-----------------+-----------+
root@singlegpu1:~# incus start ubuntu22-vm
Error: Failed to start device "gpu": Failed to detect requested GPU device
Try `incus info --show-log ubuntu22-vm` for more info

In dmesg I see:

[ 1069.508343] incusbr0: port 1(tap8df141eb) entered blocking state
[ 1069.508355] incusbr0: port 1(tap8df141eb) entered disabled state
[ 1069.508375] tap8df141eb: entered allmulticast mode
[ 1069.508473] tap8df141eb: entered promiscuous mode
[ 1069.583701] tap8df141eb: left allmulticast mode
[ 1069.583712] tap8df141eb: left promiscuous mode
[ 1069.583771] incusbr0: port 1(tap8df141eb) entered disabled state

The same error pops up with Ubuntu 20.04LTS VM as well! 

root@singlegpu1:~# incus start ubuntu-vm
root@singlegpu1:~# incus stop ubuntu-vm

The last command hanged and when I ctrl+c it:

root@singlegpu1:~# incus stop ubuntu-vm
Error: User signaled us three times, exiting. The remote operation will keep running
Try `incus info --show-log ubuntu-vm` for more info
root@singlegpu1:~# ^C
root@singlegpu1:~# incus stop ubuntu-vm
root@singlegpu1:~#

This time it worked! Not sure what is going on.

stgraber commented 5 months ago

incus stop asks the VM to perform a clean shutdown through ACPI. It's possible that the VM was not yet capable of handling the ACPI event at that stage, which is why it was waiting seemingly indefinitely.

You can use incus stop -f if you want to just forcefully stop the VM or as you've noticed, run incus stop again to get a new event sent which hopefully will have the VM react.

stgraber commented 5 months ago

I'm a bit confused about the state of this issue though. Given the proper module blacklist entries, are you still running into issues?

pythonmobile commented 5 months ago

@stgraber Can you please close this issue. stop -f worked for me. If I have other issues, will post as new issues. thanks.