stgraber
commented
3 weeks ago It looks like something got passed at least, can you show:
- incus exec gputest -- ls -lh /dev/dri/
- incus info --resources
Closed svenhendrikx closed 3 weeks ago
stgraber
commented
3 weeks ago It looks like something got passed at least, can you show:
svenhendrikx
commented
3 weeks ago Hi Stephane,
Thanks for the quick response, here's the outputs:
incus exec gputest -- ls -lh /dev/dri/
total 0
crw-rw---- 1 root root 226, 0 Oct 19 18:52 card0
crw-rw-rw- 1 root root 226, 128 Oct 19 18:52 renderD128incus info --resources
System:
UUID: 9f061de9-d9d3-e9c3-0562-fc3497c07ee9
Vendor: ASUS
Product: System Product Name
Family: To be filled by O.E.M.
Version: System Version
SKU: SKU
Serial: System Serial Number
Type: physical
Chassis:
Vendor: Default string
Type: Desktop
Version: Default string
Serial: Default string
Motherboard:
Vendor: ASUSTeK COMPUTER INC.
Product: PRIME H510M-A
Serial: 210484246000126
Version: Rev 1.xx
Firmware:
Vendor: American Megatrends Inc.
Version: 0406
Date: 03/17/2021
Load:
Processes: 512
Average: 0.03 0.16 0.16
CPU:
Architecture: x86_64
Vendor: GenuineIntel
Name: Intel(R) Core(TM) i5-10400 CPU @ 2.90GHz
Caches:
- Level 1 (type: Data): 32KiB
- Level 1 (type: Instruction): 32KiB
- Level 2 (type: Unified): 256KiB
- Level 3 (type: Unified): 12MiB
Cores:
- Core 0
Frequency: 3715Mhz
Threads:
- 0 (id: 0, online: true, NUMA node: 0)
- 1 (id: 6, online: true, NUMA node: 0)
- Core 1
Frequency: 3811Mhz
Threads:
- 0 (id: 1, online: true, NUMA node: 0)
- 1 (id: 7, online: true, NUMA node: 0)
- Core 2
Frequency: 1979Mhz
Threads:
- 0 (id: 2, online: true, NUMA node: 0)
- 1 (id: 8, online: true, NUMA node: 0)
- Core 3
Frequency: 3602Mhz
Threads:
- 0 (id: 3, online: true, NUMA node: 0)
- 1 (id: 9, online: true, NUMA node: 0)
- Core 4
Frequency: 3613Mhz
Threads:
- 0 (id: 4, online: true, NUMA node: 0)
- 1 (id: 10, online: true, NUMA node: 0)
- Core 5
Frequency: 3704Mhz
Threads:
- 0 (id: 5, online: true, NUMA node: 0)
- 1 (id: 11, online: true, NUMA node: 0)
Frequency: 3404Mhz (min: 800Mhz, max: 4300Mhz)
Memory:
Free: 12.13GiB
Used: 3.87GiB
Total: 16.00GiB
GPU:
NUMA node: 0
Vendor: NVIDIA Corporation (10de)
Product: GA104 [GeForce RTX 3060 Ti Lite Hash Rate] (2489)
PCI address: 0000:01:00.0
Driver: nvidia (560.35.03)
DRM:
ID: 0
Card: card0 (226:0)
Control: controlD64 (226:0)
Render: renderD128 (226:128)
NVIDIA information:
Architecture:
Brand: NVIDIA
Model: NVIDIA GeForce RTX 3060 Ti
CUDA Version:
NVRM Version:
UUID:
NIC:
NUMA node: 0
Vendor: Intel Corporation (8086)
Product: Ethernet Connection (14) I219-V (15fa)
PCI address: 0000:00:1f.6
Driver: e1000e (6.11.4-arch1-1)
Ports:
- Port 0 (ethernet)
ID: eno1
Address: fc:34:97:c0:7e:e9
Supported modes: 10baseT/Half, 10baseT/Full, 100baseT/Half, 100baseT/Full, 1000baseT/Full
Supported ports: twisted pair
Port type: twisted pair
Transceiver type: internal
Auto negotiation: true
Link detected: true
Link speed: 1000Mbit/s (full duplex)
Disks:
Disk 0:
NUMA node: 0
ID: nvme0n1
Device: 259:0
Model: Samsung SSD 980 1TB
Type: nvme
Size: 931.51GiB
WWN: eui.002538d821a1cb0c
Read-Only: false
Removable: false
Partitions:
- Partition 1
ID: nvme0n1p1
Device: 259:1
Read-Only: false
Size: 1022.00MiB
- Partition 2
ID: nvme0n1p2
Device: 259:2
Read-Only: false
Size: 4.00GiB
- Partition 3
ID: nvme0n1p3
Device: 259:3
Read-Only: false
Size: 922.51GiB
- Partition 4
ID: nvme0n1p4
Device: 259:4
Read-Only: false
Size: 4.00GiB
Disk 1:
NUMA node: 0
ID: sda
Device: 8:0
Model: WDC WD40EFPX-68C6CN0
Type: sata
Size: 3.64TiB
Read-Only: false
Removable: false
Disk 2:
NUMA node: 0
ID: sdb
Device: 8:16
Model: WDC WD40EFAX-68JH4N1
Type: sata
Size: 3.64TiB
Read-Only: false
Removable: false
USB devices:
Device 0:
Vendor:
Vendor ID: 413c
Product: DELL USB Keyboard
Product ID: 2005
Bus Address: 1
Device Address: 2
Device 1:
Vendor:
Vendor ID: 0b05
Product: AURA LED Controller
Product ID: 19af
Bus Address: 1
Device Address: 3
PCI devices:
Device 0:
Address: 0000:00:00.0
Vendor: Intel Corporation
Vendor ID: 8086
Product: Comet Lake-S 6c Host Bridge/DRAM Controller
Product ID: 9b53
NUMA node: 0
IOMMU group: 0
Driver: skl_uncore
Device 1:
Address: 0000:00:01.0
Vendor: Intel Corporation
Vendor ID: 8086
Product: 6th-10th Gen Core Processor PCIe Controller (x16)
Product ID: 1901
NUMA node: 0
IOMMU group: 1
Driver: pcieport
Device 2:
Address: 0000:00:14.0
Vendor: Intel Corporation
Vendor ID: 8086
Product: Tiger Lake-H USB 3.2 Gen 2x1 xHCI Host Controller
Product ID: 43ed
NUMA node: 0
IOMMU group: 2
Driver: xhci_hcd
Device 3:
Address: 0000:00:14.2
Vendor: Intel Corporation
Vendor ID: 8086
Product: Tiger Lake-H Shared SRAM
Product ID: 43ef
NUMA node: 0
IOMMU group: 2
Driver:
Device 4:
Address: 0000:00:15.0
Vendor: Intel Corporation
Vendor ID: 8086
Product: Tiger Lake-H Serial IO I2C Controller #0
Product ID: 43e8
NUMA node: 0
IOMMU group: 3
Driver: intel-lpss
Device 5:
Address: 0000:00:16.0
Vendor: Intel Corporation
Vendor ID: 8086
Product: Tiger Lake-H Management Engine Interface
Product ID: 43e0
NUMA node: 0
IOMMU group: 4
Driver: mei_me
Device 6:
Address: 0000:00:17.0
Vendor: Intel Corporation
Vendor ID: 8086
Product:
Product ID: 43d2
NUMA node: 0
IOMMU group: 5
Driver: ahci
Device 7:
Address: 0000:00:1c.0
Vendor: Intel Corporation
Vendor ID: 8086
Product: Tiger Lake-H PCI Express Root Port #5
Product ID: 43bc
NUMA node: 0
IOMMU group: 6
Driver: pcieport
Device 8:
Address: 0000:00:1f.0
Vendor: Intel Corporation
Vendor ID: 8086
Product: H510 LPC/eSPI Controller
Product ID: 4388
NUMA node: 0
IOMMU group: 7
Driver:
Device 9:
Address: 0000:00:1f.3
Vendor: Intel Corporation
Vendor ID: 8086
Product:
Product ID: f0c8
NUMA node: 0
IOMMU group: 7
Driver: snd_hda_intel
Device 10:
Address: 0000:00:1f.4
Vendor: Intel Corporation
Vendor ID: 8086
Product: Tiger Lake-H SMBus Controller
Product ID: 43a3
NUMA node: 0
IOMMU group: 7
Driver: i801_smbus
Device 11:
Address: 0000:00:1f.5
Vendor: Intel Corporation
Vendor ID: 8086
Product: Tiger Lake-H SPI Controller
Product ID: 43a4
NUMA node: 0
IOMMU group: 7
Driver: intel-spi
Device 12:
Address: 0000:00:1f.6
Vendor: Intel Corporation
Vendor ID: 8086
Product: Ethernet Connection (14) I219-V
Product ID: 15fa
NUMA node: 0
IOMMU group: 7
Driver: e1000e
Device 13:
Address: 0000:01:00.0
Vendor: NVIDIA Corporation
Vendor ID: 10de
Product: GA104 [GeForce RTX 3060 Ti Lite Hash Rate]
Product ID: 2489
NUMA node: 0
IOMMU group: 1
Driver: nvidia
Device 14:
Address: 0000:01:00.1
Vendor: NVIDIA Corporation
Vendor ID: 10de
Product: GA104 High Definition Audio Controller
Product ID: 228b
NUMA node: 0
IOMMU group: 1
Driver: snd_hda_intel
Device 15:
Address: 0000:02:00.0
Vendor: Samsung Electronics Co Ltd
Vendor ID: 144d
Product: NVMe SSD Controller 980 (DRAM-less)
Product ID: a809
NUMA node: 0
IOMMU group: 8
Driver: nvme
Okay, so the card was passed through, its DRI/DRM devices anyways.
The remaining NVIDIA-specific devices likely aren't passed through because there's something wrong with your system which is preventing it from getting the needed information:
GPU:
NUMA node: 0
Vendor: NVIDIA Corporation (10de)
Product: GA104 [GeForce RTX 3060 Ti Lite Hash Rate] (2489)
PCI address: 0000:01:00.0
Driver: nvidia (560.35.03)
DRM:
ID: 0
Card: card0 (226:0)
Control: controlD64 (226:0)
Render: renderD128 (226:128)
NVIDIA information:
Architecture:
Brand: NVIDIA
Model: NVIDIA GeForce RTX 3060 Ti
CUDA Version:
NVRM Version:
UUID:Looks like you may need to install https://archlinux.org/packages/extra/x86_64/nvidia-container-toolkit/
svenhendrikx
commented
3 weeks ago That fixed it! It also resolved the issue I was having with containers not booting when setting nvidia.runtime=true, so no more (failing at) alligning nvidia runtime driver versions. Very happy with that.
Thank you @stgraber, very quick and helpful as usual.
6.11.4-arch1-1Issue description
When trying to pass through my gpu to my development instance (
dev), which I have done before succesfully, it does not show up when listing/dev. When settingnvidia.runtime=true, the container does not startSteps to reproduce
incus launch images:archlinux/cloud --profile default gputestincus config device add gputest mygpu gpuls /devoutput of
incus info --show-log gputestResources: Processes: 19 CPU usage: CPU usage (in seconds): 1 Memory usage: Memory (current): 54.97MiB Network usage: eth0: Type: broadcast State: UP Host interface: veth68fb752c MAC address: 00:16:3e:88:9a:6d MTU: 1500 Bytes received: 354B Bytes sent: 775B Packets received: 1 Packets sent: 7 IP addresses: inet: 10.198.92.110/24 (global) inet6: fe80::216:3eff:fe88:9a6d/64 (link) lo: Type: loopback State: UP MTU: 65536 Bytes received: 0B Bytes sent: 0B Packets received: 0 Packets sent: 0 IP addresses: inet: 127.0.0.1/8 (local) inet6: ::1/128 (local)
Log:
Error: Failed to run: /usr/bin/incusd forkstart gputest /var/lib/incus/containers /run/incus/gputest/lxc.conf: exit status 1 Try
incus info --show-log gputestfor more infoName: gputest Status: STOPPED Type: container Architecture: x86_64 Created: 2024/10/19 20:23 CEST Last Used: 2024/10/19 20:29 CEST
Log:
lxc gputest 20241019182947.171 ERROR utils - ../src/lxc/utils.c:run_buffer:571 - Script exited with status 1 lxc gputest 20241019182947.171 ERROR conf - ../src/lxc/conf.c:lxc_setup:3940 - Failed to run mount hooks lxc gputest 20241019182947.171 ERROR start - ../src/lxc/start.c:do_start:1273 - Failed to setup container "gputest" lxc gputest 20241019182947.171 ERROR sync - ../src/lxc/sync.c:sync_wait:34 - An error occurred in another process (expected sequence number 4) lxc gputest 20241019182947.179 WARN network - ../src/lxc/network.c:lxc_delete_network_priv:3674 - Failed to rename interface with index 0 from "eth0" to its initial name "vethb99950f1" lxc gputest 20241019182947.179 ERROR start - ../src/lxc/start.c:__lxc_start:2114 - Failed to spawn container "gputest" lxc gputest 20241019182947.179 ERROR lxccontainer - ../src/lxc/lxccontainer.c:wait_on_daemonized_start:837 - Received container state "ABORTING" instead of "RUNNING" lxc gputest 20241019182947.180 WARN start - ../src/lxc/start.c:lxc_abort:1037 - No such process - Failed to send SIGKILL via pidfd 17 for process 23303 lxc 20241019182947.325 ERROR af_unix - ../src/lxc/af_unix.c:lxc_abstract_unix_recv_fds_iov:218 - Connection reset by peer - Failed to receive response lxc 20241019182947.326 ERROR commands - ../src/lxc/commands.c:lxc_cmd_rsp_recv_fds:128 - Failed to receive file descriptors for command "get_init_pid"
architecture: x86_64 config: image.architecture: amd64 image.description: Archlinux current amd64 (20241018_04:18) image.os: Archlinux image.release: current image.requirements.secureboot: "false" image.serial: "20241018_04:18" image.type: squashfs image.variant: cloud nvidia.runtime: "false" security.secureboot: "false" volatile.base_image: 81330bba5b9337ec0ec8852efb4e437487312d088657bb2a0845f74299bb49df volatile.cloud-init.instance-id: 7861d41e-76fe-4be5-b143-b265f395dd84 volatile.eth0.host_name: veth68fb752c volatile.eth0.hwaddr: 00:16:3e:88:9a:6d volatile.idmap.base: "0" volatile.idmap.current: '[{"Isuid":true,"Isgid":false,"Hostid":100000,"Nsid":0,"Maprange":65536},{"Isuid":false,"Isgid":true,"Hostid":100000,"Nsid":0,"Maprange":65536}]' volatile.idmap.next: '[{"Isuid":true,"Isgid":false,"Hostid":100000,"Nsid":0,"Maprange":65536},{"Isuid":false,"Isgid":true,"Hostid":100000,"Nsid":0,"Maprange":65536}]' volatile.last_state.idmap: '[]' volatile.last_state.power: RUNNING volatile.last_state.ready: "false" volatile.uuid: b127eb57-3645-435a-81d7-a9340a349b84 volatile.uuid.generation: b127eb57-3645-435a-81d7-a9340a349b84 devices: eth0: name: eth0 network: lxdbr0 type: nic mygpu: type: gpu root: path: / pool: default type: disk ephemeral: false profiles:
default stateful: false description: ""
DEBUG [2024-10-19T20:51:26+02:00] Connecting to a local Incus over a Unix socket DEBUG [2024-10-19T20:51:26+02:00] Sending request to Incus etag= method=GET url="http://unix.socket/1.0" DEBUG [2024-10-19T20:51:26+02:00] Got response struct from Incus
DEBUG [2024-10-19T20:51:26+02:00] { "config": { "core.https_address": "10.198.92.1:8443" }, "api_extensions": [ "storage_zfs_remove_snapshots", "container_host_shutdown_timeout", "container_stop_priority", "container_syscall_filtering", "auth_pki", "container_last_used_at", "etag", "patch", "usb_devices", "https_allowed_credentials", "image_compression_algorithm", "directory_manipulation", "container_cpu_time", "storage_zfs_use_refquota", "storage_lvm_mount_options", "network", "profile_usedby", "container_push", "container_exec_recording", "certificate_update", "container_exec_signal_handling", "gpu_devices", "container_image_properties", "migration_progress", "id_map", "network_firewall_filtering", "network_routes", "storage", "file_delete", "file_append", "network_dhcp_expiry", "storage_lvm_vg_rename", "storage_lvm_thinpool_rename", "network_vlan", "image_create_aliases", "container_stateless_copy", "container_only_migration", "storage_zfs_clone_copy", "unix_device_rename", "storage_lvm_use_thinpool", "storage_rsync_bwlimit", "network_vxlan_interface", "storage_btrfs_mount_options", "entity_description", "image_force_refresh", "storage_lvm_lv_resizing", "id_map_base", "file_symlinks", "container_push_target", "network_vlan_physical", "storage_images_delete", "container_edit_metadata", "container_snapshot_stateful_migration", "storage_driver_ceph", "storage_ceph_user_name", "resource_limits", "storage_volatile_initial_source", "storage_ceph_force_osd_reuse", "storage_block_filesystem_btrfs", "resources", "kernel_limits", "storage_api_volume_rename", "network_sriov", "console", "restrict_dev_incus", "migration_pre_copy", "infiniband", "dev_incus_events", "proxy", "network_dhcp_gateway", "file_get_symlink", "network_leases", "unix_device_hotplug", "storage_api_local_volume_handling", "operation_description", "clustering", "event_lifecycle", "storage_api_remote_volume_handling", "nvidia_runtime", "container_mount_propagation", "container_backup", "dev_incus_images", "container_local_cross_pool_handling", "proxy_unix", "proxy_udp", "clustering_join", "proxy_tcp_udp_multi_port_handling", "network_state", "proxy_unix_dac_properties", "container_protection_delete", "unix_priv_drop", "pprof_http", "proxy_haproxy_protocol", "network_hwaddr", "proxy_nat", "network_nat_order", "container_full", "backup_compression", "nvidia_runtime_config", "storage_api_volume_snapshots", "storage_unmapped", "projects", "network_vxlan_ttl", "container_incremental_copy", "usb_optional_vendorid", "snapshot_scheduling", "snapshot_schedule_aliases", "container_copy_project", "clustering_server_address", "clustering_image_replication", "container_protection_shift", "snapshot_expiry", "container_backup_override_pool", "snapshot_expiry_creation", "network_leases_location", "resources_cpu_socket", "resources_gpu", "resources_numa", "kernel_features", "id_map_current", "event_location", "storage_api_remote_volume_snapshots", "network_nat_address", "container_nic_routes", "cluster_internal_copy", "seccomp_notify", "lxc_features", "container_nic_ipvlan", "network_vlan_sriov", "storage_cephfs", "container_nic_ipfilter", "resources_v2", "container_exec_user_group_cwd", "container_syscall_intercept", "container_disk_shift", "storage_shifted", "resources_infiniband", "daemon_storage", "instances", "image_types", "resources_disk_sata", "clustering_roles", "images_expiry", "resources_network_firmware", "backup_compression_algorithm", "ceph_data_pool_name", "container_syscall_intercept_mount", "compression_squashfs", "container_raw_mount", "container_nic_routed", "container_syscall_intercept_mount_fuse", "container_disk_ceph", "virtual-machines", "image_profiles", "clustering_architecture", "resources_disk_id", "storage_lvm_stripes", "vm_boot_priority", "unix_hotplug_devices", "api_filtering", "instance_nic_network", "clustering_sizing", "firewall_driver", "projects_limits", "container_syscall_intercept_hugetlbfs", "limits_hugepages", "container_nic_routed_gateway", "projects_restrictions", "custom_volume_snapshot_expiry", "volume_snapshot_scheduling", "trust_ca_certificates", "snapshot_disk_usage", "clustering_edit_roles", "container_nic_routed_host_address", "container_nic_ipvlan_gateway", "resources_usb_pci", "resources_cpu_threads_numa", "resources_cpu_core_die", "api_os", "container_nic_routed_host_table", "container_nic_ipvlan_host_table", "container_nic_ipvlan_mode", "resources_system", "images_push_relay", "network_dns_search", "container_nic_routed_limits", "instance_nic_bridged_vlan", "network_state_bond_bridge", "usedby_consistency", "custom_block_volumes", "clustering_failure_domains", "resources_gpu_mdev", "console_vga_type", "projects_limits_disk", "network_type_macvlan", "network_type_sriov", "container_syscall_intercept_bpf_devices", "network_type_ovn", "projects_networks", "projects_networks_restricted_uplinks", "custom_volume_backup", "backup_override_name", "storage_rsync_compression", "network_type_physical", "network_ovn_external_subnets", "network_ovn_nat", "network_ovn_external_routes_remove", "tpm_device_type", "storage_zfs_clone_copy_rebase", "gpu_mdev", "resources_pci_iommu", "resources_network_usb", "resources_disk_address", "network_physical_ovn_ingress_mode", "network_ovn_dhcp", "network_physical_routes_anycast", "projects_limits_instances", "network_state_vlan", "instance_nic_bridged_port_isolation", "instance_bulk_state_change", "network_gvrp", "instance_pool_move", "gpu_sriov", "pci_device_type", "storage_volume_state", "network_acl", "migration_stateful", "disk_state_quota", "storage_ceph_features", "projects_compression", "projects_images_remote_cache_expiry", "certificate_project", "network_ovn_acl", "projects_images_auto_update", "projects_restricted_cluster_target", "images_default_architecture", "network_ovn_acl_defaults", "gpu_mig", "project_usage", "network_bridge_acl", "warnings", "projects_restricted_backups_and_snapshots", "clustering_join_token", "clustering_description", "server_trusted_proxy", "clustering_update_cert", "storage_api_project", "server_instance_driver_operational", "server_supported_storage_drivers", "event_lifecycle_requestor_address", "resources_gpu_usb", "clustering_evacuation", "network_ovn_nat_address", "network_bgp", "network_forward", "custom_volume_refresh", "network_counters_errors_dropped", "metrics", "image_source_project", "clustering_config", "network_peer", "linux_sysctl", "network_dns", "ovn_nic_acceleration", "certificate_self_renewal", "instance_project_move", "storage_volume_project_move", "cloud_init", "network_dns_nat", "database_leader", "instance_all_projects", "clustering_groups", "ceph_rbd_du", "instance_get_full", "qemu_metrics", "gpu_mig_uuid", "event_project", "clustering_evacuation_live", "instance_allow_inconsistent_copy", "network_state_ovn", "storage_volume_api_filtering", "image_restrictions", "storage_zfs_export", "network_dns_records", "storage_zfs_reserve_space", "network_acl_log", "storage_zfs_blocksize", "metrics_cpu_seconds", "instance_snapshot_never", "certificate_token", "instance_nic_routed_neighbor_probe", "event_hub", "agent_nic_config", "projects_restricted_intercept", "metrics_authentication", "images_target_project", "images_all_projects", "cluster_migration_inconsistent_copy", "cluster_ovn_chassis", "container_syscall_intercept_sched_setscheduler", "storage_lvm_thinpool_metadata_size", "storage_volume_state_total", "instance_file_head", "instances_nic_host_name", "image_copy_profile", "container_syscall_intercept_sysinfo", "clustering_evacuation_mode", "resources_pci_vpd", "qemu_raw_conf", "storage_cephfs_fscache", "network_load_balancer", "vsock_api", "instance_ready_state", "network_bgp_holdtime", "storage_volumes_all_projects", "metrics_memory_oom_total", "storage_buckets", "storage_buckets_create_credentials", "metrics_cpu_effective_total", "projects_networks_restricted_access", "storage_buckets_local", "loki", "acme", "internal_metrics", "cluster_join_token_expiry", "remote_token_expiry", "init_preseed", "storage_volumes_created_at", "cpu_hotplug", "projects_networks_zones", "network_txqueuelen", "cluster_member_state", "instances_placement_scriptlet", "storage_pool_source_wipe", "zfs_block_mode", "instance_generation_id", "disk_io_cache", "amd_sev", "storage_pool_loop_resize", "migration_vm_live", "ovn_nic_nesting", "oidc", "network_ovn_l3only", "ovn_nic_acceleration_vdpa", "cluster_healing", "instances_state_total", "auth_user", "security_csm", "instances_rebuild", "numa_cpu_placement", "custom_volume_iso", "network_allocations", "zfs_delegate", "storage_api_remote_volume_snapshot_copy", "operations_get_query_all_projects", "metadata_configuration", "syslog_socket", "event_lifecycle_name_and_project", "instances_nic_limits_priority", "disk_initial_volume_configuration", "operation_wait", "image_restriction_privileged", "cluster_internal_custom_volume_copy", "disk_io_bus", "storage_cephfs_create_missing", "instance_move_config", "ovn_ssl_config", "certificate_description", "disk_io_bus_virtio_blk", "loki_config_instance", "instance_create_start", "clustering_evacuation_stop_options", "boot_host_shutdown_action", "agent_config_drive", "network_state_ovn_lr", "image_template_permissions", "storage_bucket_backup", "storage_lvm_cluster", "shared_custom_block_volumes", "auth_tls_jwt", "oidc_claim", "device_usb_serial", "numa_cpu_balanced", "image_restriction_nesting", "network_integrations", "instance_memory_swap_bytes", "network_bridge_external_create", "network_zones_all_projects", "storage_zfs_vdev", "container_migration_stateful", "profiles_all_projects", "instances_scriptlet_get_instances", "instances_scriptlet_get_cluster_members", "instances_scriptlet_get_project", "network_acl_stateless", "instance_state_started_at", "networks_all_projects", "network_acls_all_projects", "storage_buckets_all_projects", "resources_load", "instance_access", "project_access", "projects_force_delete", "resources_cpu_flags", "disk_io_bus_cache_filesystem", "instance_oci", "clustering_groups_config", "instances_lxcfs_per_instance", "clustering_groups_vm_cpu_definition", "disk_volume_subpath", "projects_limits_disk_pool", "network_ovn_isolated", "qemu_raw_qmp", "network_load_balancer_health_check", "oidc_scopes", "network_integrations_peer_name", "qemu_scriptlet", "instance_auto_restart", "storage_lvm_metadatasize", "ovn_nic_promiscuous", "ovn_nic_ip_address_none", "instances_state_os_info", "network_load_balancer_state", "instance_nic_macvlan_mode", "storage_lvm_cluster_create", "network_ovn_external_interfaces" ], "api_status": "stable", "api_version": "1.0", "auth": "trusted", "public": false, "auth_methods": [ "tls" ], "auth_user_name": "sven", "auth_user_method": "unix", "environment": { "addresses": [ "10.198.92.1:8443" ], "architectures": [ "x86_64", "i686" ], "certificate": "-----BEGIN CERTIFICATE-----\nMIICETCCAZagAwIBAgIRAM3a5K9UDi9Stg8H3o3DjxcwCgYIKoZIzj0EAwMwODEc\nMBoGA1UEChMTbGludXhjb250YWluZXJzLm9yZzEYMBYGA1UEAwwPcm9vdEAyMFYt\nc2VydmVyMB4XDTIzMDEyMTE2NTkxNVoXDTMzMDExODE2NTkxNVowODEcMBoGA1UE\nChMTbGludXhjb250YWluZXJzLm9yZzEYMBYGA1UEAwwPcm9vdEAyMFYtc2VydmVy\nMHYwEAYHKoZIzj0CAQYFK4EEACIDYgAEVR9olmnff90vCq1dhmID5DAK+cM6oreB\nyiD1lFMvW+O//lyxlv0MEvLhssCWaWWQV1z4PqkRd1Cq5YNeiUbOamWPmaH/0dMl\nF+UjvfPEMfWy5j7OvaKz94gWj1K8+eWTo2QwYjAOBgNVHQ8BAf8EBAMCBaAwEwYD\nVR0lBAwwCgYIKwYBBQUHAwEwDAYDVR0TAQH/BAIwADAtBgNVHREEJjAkggoyMFYt\nc2VydmVyhwR/AAABhxAAAAAAAAAAAAAAAAAAAAABMAoGCCqGSM49BAMDA2kAMGYC\nMQDea3GUfhfjBcvrN14JtTuQUHXNcnCd1pkGzPEKYN7YHscI+gEI27DGD9dq/NG7\nhjICMQCa2myQyO6Yb+ksgRnHPM/Ko+b38YyACQpHVPUhr2EX5nu5ASFIDXUDJPRT\nWX7klHE=\n-----END CERTIFICATE-----\n", "certificate_fingerprint": "0166677795b317b950fe9513f9afcb26ec5486ec40eb350b9d5b7299435aea5a", "driver": "lxc | qemu", "driver_version": "6.0.2 | 9.1.0", "firewall": "nftables", "kernel": "Linux", "kernel_architecture": "x86_64", "kernel_features": { "idmapped_mounts": "true", "netnsid_getifaddrs": "true", "seccomp_listener": "true", "seccomp_listener_continue": "true", "uevent_injection": "true", "unpriv_binfmt": "true", "unpriv_fscaps": "true" }, "kernel_version": "6.11.4-arch1-1", "lxc_features": { "cgroup2": "true", "core_scheduling": "true", "devpts_fd": "true", "idmapped_mounts_v2": "true", "mount_injection_file": "true", "network_gateway_device_route": "true", "network_ipvlan": "true", "network_l2proxy": "true", "network_phys_macvlan_mtu": "true", "network_veth_router": "true", "pidfd": "true", "seccomp_allow_deny_syntax": "true", "seccomp_notify": "true", "seccomp_proxy_send_notify_fd": "true" }, "os_name": "Arch Linux", "os_version": "", "project": "default", "server": "incus", "server_clustered": false, "server_event_mode": "full-mesh", "server_name": "20V-server", "server_pid": 590, "server_version": "6.6", "storage": "dir | btrfs", "storage_version": "1 | 6.11", "storage_supported_drivers": [ { "Name": "dir", "Version": "1", "Remote": false }, { "Name": "btrfs", "Version": "6.11", "Remote": false } ] } } DEBUG [2024-10-19T20:51:26+02:00] Sending request to Incus etag= method=GET url="http://unix.socket/1.0/instances/gputest" DEBUG [2024-10-19T20:51:26+02:00] Got response struct from Incus
DEBUG [2024-10-19T20:51:26+02:00] { "architecture": "x86_64", "config": { "image.architecture": "amd64", "image.description": "Archlinux current amd64 (20241018_04:18)", "image.os": "Archlinux", "image.release": "current", "image.requirements.secureboot": "false", "image.serial": "20241018_04:18", "image.type": "squashfs", "image.variant": "cloud", "nvidia.runtime": "false", "volatile.base_image": "81330bba5b9337ec0ec8852efb4e437487312d088657bb2a0845f74299bb49df", "volatile.cloud-init.instance-id": "7861d41e-76fe-4be5-b143-b265f395dd84", "volatile.eth0.hwaddr": "00:16:3e:88:9a:6d", "volatile.idmap.base": "0", "volatile.idmap.current": "[{\"Isuid\":true,\"Isgid\":false,\"Hostid\":100000,\"Nsid\":0,\"Maprange\":65536},{\"Isuid\":false,\"Isgid\":true,\"Hostid\":100000,\"Nsid\":0,\"Maprange\":65536}]", "volatile.idmap.next": "[{\"Isuid\":true,\"Isgid\":false,\"Hostid\":100000,\"Nsid\":0,\"Maprange\":65536},{\"Isuid\":false,\"Isgid\":true,\"Hostid\":100000,\"Nsid\":0,\"Maprange\":65536}]", "volatile.last_state.idmap": "[]", "volatile.last_state.power": "STOPPED", "volatile.last_state.ready": "false", "volatile.uuid": "b127eb57-3645-435a-81d7-a9340a349b84", "volatile.uuid.generation": "b127eb57-3645-435a-81d7-a9340a349b84" }, "devices": { "mygpu": { "type": "gpu" } }, "ephemeral": false, "profiles": [ "default" ], "stateful": false, "description": "", "created_at": "2024-10-19T18:23:44.583903568Z", "expanded_config": { "image.architecture": "amd64", "image.description": "Archlinux current amd64 (20241018_04:18)", "image.os": "Archlinux", "image.release": "current", "image.requirements.secureboot": "false", "image.serial": "20241018_04:18", "image.type": "squashfs", "image.variant": "cloud", "nvidia.runtime": "false", "security.secureboot": "false", "volatile.base_image": "81330bba5b9337ec0ec8852efb4e437487312d088657bb2a0845f74299bb49df", "volatile.cloud-init.instance-id": "7861d41e-76fe-4be5-b143-b265f395dd84", "volatile.eth0.hwaddr": "00:16:3e:88:9a:6d", "volatile.idmap.base": "0", "volatile.idmap.current": "[{\"Isuid\":true,\"Isgid\":false,\"Hostid\":100000,\"Nsid\":0,\"Maprange\":65536},{\"Isuid\":false,\"Isgid\":true,\"Hostid\":100000,\"Nsid\":0,\"Maprange\":65536}]", "volatile.idmap.next": "[{\"Isuid\":true,\"Isgid\":false,\"Hostid\":100000,\"Nsid\":0,\"Maprange\":65536},{\"Isuid\":false,\"Isgid\":true,\"Hostid\":100000,\"Nsid\":0,\"Maprange\":65536}]", "volatile.last_state.idmap": "[]", "volatile.last_state.power": "STOPPED", "volatile.last_state.ready": "false", "volatile.uuid": "b127eb57-3645-435a-81d7-a9340a349b84", "volatile.uuid.generation": "b127eb57-3645-435a-81d7-a9340a349b84" }, "expanded_devices": { "eth0": { "name": "eth0", "network": "lxdbr0", "type": "nic" }, "mygpu": { "type": "gpu" }, "root": { "path": "/", "pool": "default", "type": "disk" } }, "name": "gputest", "status": "Stopped", "status_code": 102, "last_used_at": "2024-10-19T18:30:09.260728063Z", "location": "none", "type": "container", "project": "default" } DEBUG [2024-10-19T20:51:26+02:00] Connected to the websocket: ws://unix.socket/1.0/events DEBUG [2024-10-19T20:51:26+02:00] Sending request to Incus etag= method=PUT url="http://unix.socket/1.0/instances/gputest/state" DEBUG [2024-10-19T20:51:26+02:00] { "action": "start", "timeout": 0, "force": false, "stateful": false } DEBUG [2024-10-19T20:51:26+02:00] Got operation from Incus
DEBUG [2024-10-19T20:51:26+02:00] { "id": "9f251da8-730b-40c1-b0b6-75c20b883109", "class": "task", "description": "Starting instance", "created_at": "2024-10-19T20:51:26.685062064+02:00", "updated_at": "2024-10-19T20:51:26.685062064+02:00", "status": "Running", "status_code": 103, "resources": { "instances": [ "/1.0/instances/gputest" ] }, "metadata": null, "may_cancel": false, "err": "", "location": "none" } DEBUG [2024-10-19T20:51:26+02:00] Sending request to Incus etag= method=GET url="http://unix.socket/1.0/operations/9f251da8-730b-40c1-b0b6-75c20b883109" DEBUG [2024-10-19T20:51:26+02:00] Got response struct from Incus
DEBUG [2024-10-19T20:51:26+02:00] { "id": "9f251da8-730b-40c1-b0b6-75c20b883109", "class": "task", "description": "Starting instance", "created_at": "2024-10-19T20:51:26.685062064+02:00", "updated_at": "2024-10-19T20:51:26.685062064+02:00", "status": "Running", "status_code": 103, "resources": { "instances": [ "/1.0/instances/gputest" ] }, "metadata": null, "may_cancel": false, "err": "", "location": "none" }
DEBUG [2024-10-19T20:52:26+02:00] Event listener server handler started id=65c95b05-59d7-4d8a-a611-86f1cd9f0a2e local=/var/lib/incus/unix.socket remote=@ DEBUG [2024-10-19T20:52:33+02:00] Handling API request ip=@ method=GET protocol=unix url=/1.0 username=sven DEBUG [2024-10-19T20:52:33+02:00] Handling API request ip=@ method=GET protocol=unix url=/1.0/instances/gputest username=sven DEBUG [2024-10-19T20:52:33+02:00] Handling API request ip=@ method=GET protocol=unix url=/1.0/events username=sven DEBUG [2024-10-19T20:52:33+02:00] Event listener server handler started id=18fe56fc-d909-4a41-8066-9ac7192c6c6d local=/var/lib/incus/unix.socket remote=@ DEBUG [2024-10-19T20:52:33+02:00] Handling API request ip=@ method=PUT protocol=unix url=/1.0/instances/gputest/state username=sven DEBUG [2024-10-19T20:52:33+02:00] Start started instance=gputest instanceType=container project=default stateful=false DEBUG [2024-10-19T20:52:33+02:00] Started operation class=task description="Starting instance" operation=2598a12b-2e76-4337-9faa-84d7abc47a99 project=default DEBUG [2024-10-19T20:52:33+02:00] New operation class=task description="Starting instance" operation=2598a12b-2e76-4337-9faa-84d7abc47a99 project=default DEBUG [2024-10-19T20:52:33+02:00] Instance operation lock created action=start instance=gputest project=default reusable=false INFO [2024-10-19T20:52:33+02:00] ID: 2598a12b-2e76-4337-9faa-84d7abc47a99, Class: task, Description: Starting instance CreatedAt="2024-10-19 20:52:33.399068909 +0200 CEST" Err= Location=none MayCancel=false Metadata="map[]" Resources="map[instances:[/1.0/instances/gputest]]" Status=Running StatusCode=Running UpdatedAt="2024-10-19 20:52:33.399068909 +0200 CEST" INFO [2024-10-19T20:52:33+02:00] ID: 2598a12b-2e76-4337-9faa-84d7abc47a99, Class: task, Description: Starting instance CreatedAt="2024-10-19 20:52:33.399068909 +0200 CEST" Err= Location=none MayCancel=false Metadata="map[]" Resources="map[instances:[/1.0/instances/gputest]]" Status=Pending StatusCode=Pending UpdatedAt="2024-10-19 20:52:33.399068909 +0200 CEST" INFO [2024-10-19T20:52:33+02:00] Starting instance action=start created="2024-10-19 18:23:44.583903568 +0000 UTC" ephemeral=false instance=gputest instanceType=container project=default stateful=false used="2024-10-19 18:51:26.781674435 +0000 UTC" DEBUG [2024-10-19T20:52:33+02:00] Handling API request ip=@ method=GET protocol=unix url=/1.0/operations/2598a12b-2e76-4337-9faa-84d7abc47a99 username=sven DEBUG [2024-10-19T20:52:33+02:00] MountInstance started driver=dir instance=gputest pool=default project=default DEBUG [2024-10-19T20:52:33+02:00] MountInstance finished driver=dir instance=gputest pool=default project=default DEBUG [2024-10-19T20:52:33+02:00] Starting device device=eth0 instance=gputest instanceType=container project=default type=nic DEBUG [2024-10-19T20:52:33+02:00] Starting device device=root instance=gputest instanceType=container project=default type=disk DEBUG [2024-10-19T20:52:33+02:00] Starting device device=mygpu instance=gputest instanceType=container project=default type=gpu DEBUG [2024-10-19T20:52:33+02:00] UpdateInstanceBackupFile started driver=dir instance=gputest pool=default project=default DEBUG [2024-10-19T20:52:33+02:00] UpdateInstanceBackupFile finished driver=dir instance=gputest pool=default project=default DEBUG [2024-10-19T20:52:33+02:00] Skipping unmount as in use driver=dir pool=default refCount=1 volName=gputest DEBUG [2024-10-19T20:52:33+02:00] Handling API request ip=@ method=GET protocol=unix url="/internal/containers/gputest/onstart?project=default" username=root DEBUG [2024-10-19T20:52:33+02:00] Scheduler: container gputest started: re-balancing DEBUG [2024-10-19T20:52:33+02:00] Start finished instance=gputest instanceType=container project=default stateful=false INFO [2024-10-19T20:52:33+02:00] Started instance action=start created="2024-10-19 18:23:44.583903568 +0000 UTC" ephemeral=false instance=gputest instanceType=container project=default stateful=false used="2024-10-19 18:51:26.781674435 +0000 UTC" DEBUG [2024-10-19T20:52:33+02:00] Success for operation class=task description="Starting instance" operation=2598a12b-2e76-4337-9faa-84d7abc47a99 project=default DEBUG [2024-10-19T20:52:33+02:00] Instance operation lock finished action=start err="" instance=gputest project=default reusable=false
INFO [2024-10-19T20:52:33+02:00] Action: instance-started, Source: /1.0/instances/gputest, Requestor: unix/sven (@)
INFO [2024-10-19T20:52:33+02:00] ID: 2598a12b-2e76-4337-9faa-84d7abc47a99, Class: task, Description: Starting instance CreatedAt="2024-10-19 20:52:33.399068909 +0200 CEST" Err= Location=none MayCancel=false Metadata="map[]" Resources="map[instances:[/1.0/instances/gputest]]" Status=Success StatusCode=Success UpdatedAt="2024-10-19 20:52:33.399068909 +0200 CEST"
DEBUG [2024-10-19T20:52:33+02:00] Event listener server handler stopped listener=18fe56fc-d909-4a41-8066-9ac7192c6c6d local=/var/lib/incus/unix.socket remote=@