search

mdevctl / mdevctl

A mediated device management and persistence utility
GNU Lesser General Public License v2.1
60 stars 19 forks source link

mdevctl start fails with Invalid argument (os error 22) #115

Closed Pesc0 closed 6 months ago

Pesc0 commented 6 months ago

Hi, I am trying to get nvidia vgpu running (using vgpu unlock hack, https://github.com/VGPU-Community-Drivers/vGPU-Unlock-patcher)

I have a gtx1070, and the driver seems to be working:

$ mdevctl types
0000:02:00.0
  nvidia-156
    Available instances: 12
    Device API: vfio-pci
    Name: GRID P40-2B
    Description: num_heads=4, frl_config=45, framebuffer=2048M, max_resolution=5120x2880, max_instance=12
  nvidia-215
    Available instances: 12
    Device API: vfio-pci
    Name: GRID P40-2B4
    Description: num_heads=4, frl_config=45, framebuffer=2048M, max_resolution=5120x2880, max_instance=12
  nvidia-241
    Available instances: 24
    Device API: vfio-pci
    Name: GRID P40-1B4
    Description: num_heads=4, frl_config=45, framebuffer=1024M, max_resolution=5120x2880, max_instance=24
  nvidia-283
    Available instances: 6
    Device API: vfio-pci
    Name: GRID P40-4C
    Description: num_heads=1, frl_config=60, framebuffer=4096M, max_resolution=4096x2400, max_instance=6
  nvidia-284
    Available instances: 4
    Device API: vfio-pci
    Name: GRID P40-6C
    Description: num_heads=1, frl_config=60, framebuffer=6144M, max_resolution=4096x2400, max_instance=4
  nvidia-285
    Available instances: 3
    Device API: vfio-pci
    Name: GRID P40-8C
    Description: num_heads=1, frl_config=60, framebuffer=8192M, max_resolution=4096x2400, max_instance=3
  nvidia-286
    Available instances: 2
    Device API: vfio-pci
    Name: GRID P40-12C
    Description: num_heads=1, frl_config=60, framebuffer=12288M, max_resolution=4096x2400, max_instance=2
  nvidia-287
    Available instances: 1
    Device API: vfio-pci
    Name: GRID P40-24C
    Description: num_heads=1, frl_config=60, framebuffer=24576M, max_resolution=4096x2400, max_instance=1
  nvidia-46
    Available instances: 24
    Device API: vfio-pci
    Name: GRID P40-1Q
    Description: num_heads=4, frl_config=60, framebuffer=1024M, max_resolution=5120x2880, max_instance=24
  nvidia-47
    Available instances: 12
    Device API: vfio-pci
    Name: GRID P40-2Q
    Description: num_heads=4, frl_config=60, framebuffer=2048M, max_resolution=7680x4320, max_instance=12
  nvidia-48
    Available instances: 8
    Device API: vfio-pci
    Name: GRID P40-3Q
    Description: num_heads=4, frl_config=60, framebuffer=3072M, max_resolution=7680x4320, max_instance=8
  nvidia-49
    Available instances: 6
    Device API: vfio-pci
    Name: GRID P40-4Q
    Description: num_heads=4, frl_config=60, framebuffer=4096M, max_resolution=7680x4320, max_instance=6
  nvidia-50
    Available instances: 4
    Device API: vfio-pci
    Name: GRID P40-6Q
    Description: num_heads=4, frl_config=60, framebuffer=6144M, max_resolution=7680x4320, max_instance=4
  nvidia-51
    Available instances: 3
    Device API: vfio-pci
    Name: GRID P40-8Q
    Description: num_heads=4, frl_config=60, framebuffer=8192M, max_resolution=7680x4320, max_instance=3
  nvidia-52
    Available instances: 2
    Device API: vfio-pci
    Name: GRID P40-12Q
    Description: num_heads=4, frl_config=60, framebuffer=12288M, max_resolution=7680x4320, max_instance=2
  nvidia-53
    Available instances: 1
    Device API: vfio-pci
    Name: GRID P40-24Q
    Description: num_heads=4, frl_config=60, framebuffer=24576M, max_resolution=7680x4320, max_instance=1
  nvidia-54
    Available instances: 24
    Device API: vfio-pci
    Name: GRID P40-1A
    Description: num_heads=1, frl_config=60, framebuffer=1024M, max_resolution=1280x1024, max_instance=24
  nvidia-55
    Available instances: 12
    Device API: vfio-pci
    Name: GRID P40-2A
    Description: num_heads=1, frl_config=60, framebuffer=2048M, max_resolution=1280x1024, max_instance=12
  nvidia-56
    Available instances: 8
    Device API: vfio-pci
    Name: GRID P40-3A
    Description: num_heads=1, frl_config=60, framebuffer=3072M, max_resolution=1280x1024, max_instance=8
  nvidia-57
    Available instances: 6
    Device API: vfio-pci
    Name: GRID P40-4A
    Description: num_heads=1, frl_config=60, framebuffer=4096M, max_resolution=1280x1024, max_instance=6
  nvidia-58
    Available instances: 4
    Device API: vfio-pci
    Name: GRID P40-6A
    Description: num_heads=1, frl_config=60, framebuffer=6144M, max_resolution=1280x1024, max_instance=4
  nvidia-59
    Available instances: 3
    Device API: vfio-pci
    Name: GRID P40-8A
    Description: num_heads=1, frl_config=60, framebuffer=8192M, max_resolution=1280x1024, max_instance=3
  nvidia-60
    Available instances: 2
    Device API: vfio-pci
    Name: GRID P40-12A
    Description: num_heads=1, frl_config=60, framebuffer=12288M, max_resolution=1280x1024, max_instance=2
  nvidia-61
    Available instances: 1
    Device API: vfio-pci
    Name: GRID P40-24A
    Description: num_heads=1, frl_config=60, framebuffer=24576M, max_resolution=1280x1024, max_instance=1
  nvidia-62
    Available instances: 24
    Device API: vfio-pci
    Name: GRID P40-1B
    Description: num_heads=4, frl_config=45, framebuffer=1024M, max_resolution=5120x2880, max_instance=24

$ nvidia-smi
Sat Mar 16 16:46:42 2024       
+---------------------------------------------------------------------------------------+
| NVIDIA-SMI 535.129.03             Driver Version: 535.129.03   CUDA Version: 12.2     |
|-----------------------------------------+----------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |         Memory-Usage | GPU-Util  Compute M. |
|                                         |                      |               MIG M. |
|=========================================+======================+======================|
|   0  NVIDIA GeForce GTX 1070        On  | 00000000:02:00.0 Off |                  N/A |
|  0%   43C    P8              12W / 151W |      5MiB /  8192MiB |      0%      Default |
|                                         |                      |                  N/A |
+-----------------------------------------+----------------------+----------------------+

+---------------------------------------------------------------------------------------+
| Processes:                                                                            |
|  GPU   GI   CI        PID   Type   Process name                            GPU Memory |
|        ID   ID                                                             Usage      |
|=======================================================================================|
|    0   N/A  N/A      1512      G   /usr/bin/gnome-shell                          2MiB |
+---------------------------------------------------------------------------------------+

$ nvidia-smi vgpu
Sat Mar 16 16:46:51 2024       
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 535.129.03             Driver Version: 535.129.03                |
|---------------------------------+------------------------------+------------+
| GPU  Name                       | Bus-Id                       | GPU-Util   |
|      vGPU ID     Name           | VM ID     VM Name            | vGPU-Util  |
|=================================+==============================+============|
|   0  NVIDIA GeForce GTX 1070    | 00000000:02:00.0             |   0%       |
+---------------------------------+------------------------------+------------+

However when I run 

$ sudo MDEVCTL_LOG=debug mdevctl start -p 0000:02:00.0 -t nvidia-51
[2024-03-16T15:48:37Z DEBUG mdevctl] Starting up
[2024-03-16T15:48:37Z DEBUG mdevctl] DefaultEnvironment { rootpath: "/", callout_scripts: Mutex { data: CalloutScriptCache { callouts: [] }, poisoned: false, .. } }
[2024-03-16T15:48:37Z DEBUG mdevctl::environment] checking that the environment is sane
[2024-03-16T15:48:37Z DEBUG mdevctl] Starting device 'None'
[2024-03-16T15:48:37Z DEBUG mdevctl::callouts] Looking up callout script for mdev type '"nvidia-51"'
[2024-03-16T15:48:37Z DEBUG mdevctl::callouts] Callout script lookup failed. Start searching for mdev type '"nvidia-51"' and parent "0000:02:00.0"
[2024-03-16T15:48:37Z DEBUG mdevctl::callouts] get-capabilities: looking for a matching callout script for dev type 'nvidia-51' in "/usr/lib/mdevctl/scripts.d/callouts"
[2024-03-16T15:48:37Z DEBUG mdevctl::callouts] No callout script with version support found
[2024-03-16T15:48:37Z DEBUG mdevctl::callouts] pre-start: looking for a matching callout script for dev type 'nvidia-51' in "/usr/lib/mdevctl/scripts.d/callouts"
[2024-03-16T15:48:37Z DEBUG mdevctl::mdev] Creating mdev 48df3410-5c21-4c86-8951-7d67705cea21
[2024-03-16T15:48:37Z DEBUG mdevctl::mdev] Loading device '48df3410-5c21-4c86-8951-7d67705cea21' from sysfs
[2024-03-16T15:48:37Z DEBUG mdevctl::mdev] loaded device MDev { uuid: 48df3410-5c21-4c86-8951-7d67705cea21, active: false, autostart: false, parent: None, mdev_type: None, attrs: [], env: Environment { mdev_base: "/sys/bus/mdev/devices", config_base: "/etc/mdevctl.d", parent_base: "/sys/class/mdev_bus" } }
[2024-03-16T15:48:37Z DEBUG mdevctl::mdev] Checking parent for mdev support: "/sys/class/mdev_bus/0000:02:00.0/mdev_supported_types"
[2024-03-16T15:48:37Z DEBUG mdevctl::mdev] Checking parent for mdev type nvidia-51: "/sys/class/mdev_bus/0000:02:00.0/mdev_supported_types/nvidia-51"
[2024-03-16T15:48:37Z DEBUG mdevctl::mdev] Checking available instances: "/sys/class/mdev_bus/0000:02:00.0/mdev_supported_types/nvidia-51/available_instances"
[2024-03-16T15:48:37Z DEBUG mdevctl::mdev] Available instances: 3
[2024-03-16T15:48:37Z DEBUG mdevctl::mdev] Creating mediated device: 48df3410-5c21-4c86-8951-7d67705cea21 -> "/sys/class/mdev_bus/0000:02:00.0/mdev_supported_types/nvidia-51/create"
[2024-03-16T15:48:37Z DEBUG mdevctl::callouts] post-start: looking for a matching callout script for dev type 'nvidia-51' in "/usr/lib/mdevctl/scripts.d/callouts"
[2024-03-16T15:48:37Z DEBUG mdevctl::callouts] notify-start: executing notification scripts for device 48df3410-5c21-4c86-8951-7d67705cea21
Error: Failed to create mdev 48df3410-5c21-4c86-8951-7d67705cea21, type nvidia-51 on 0000:02:00.0

Caused by:
    Invalid argument (os error 22)

$sudo dmesg
[ 3389.569550] ------------[ cut here ]------------
[ 3389.569552] WARNING: CPU: 0 PID: 38526 at drivers/vfio/vfio_main.c:273 __vfio_register_dev+0xbf/0x110 [vfio]
[ 3389.569560] Modules linked in: nvidia_uvm(OE) rfcomm snd_seq_dummy snd_seq_midi snd_hrtimer snd_seq_midi_event snd_seq xt_CHECKSUM xt_MASQUERADE xt_conntrack ipt_REJECT nf_reject_ipv4 xt_tcpudp nft_compat nft_chain_nat nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nf_tables libcrc32c bridge stp llc nvidia_vgpu_vfio(OE) mdev uhid cmac algif_hash algif_skcipher af_alg bnep nvidia_drm(POE) nvidia_modeset(POE) snd_sof_pci_intel_cnl intel_rapl_msr snd_sof_intel_hda_common intel_rapl_common soundwire_intel snd_sof_intel_hda_mlink soundwire_cadence snd_sof_intel_hda snd_sof_pci snd_sof_xtensa_dsp snd_sof snd_sof_utils intel_tcc_cooling soundwire_generic_allocation soundwire_bus x86_pkg_temp_thermal intel_powerclamp snd_soc_skl snd_soc_hdac_hda snd_hda_ext_core coretemp vfat snd_soc_sst_ipc snd_soc_sst_dsp fat kvm_intel snd_soc_acpi_intel_match snd_soc_acpi kvm snd_soc_core nvidia(OE) snd_compress ac97_bus snd_pcm_dmaengine crct10dif_pclmul crc32_pclmul snd_hda_codec_realtek polyval_clmulni snd_hda_codec_generic
[ 3389.569586]  snd_hda_codec_hdmi polyval_generic gf128mul snd_usb_audio snd_hda_intel uvcvideo ghash_clmulni_intel snd_intel_dspcfg btusb sha512_ssse3 snd_intel_sdw_acpi snd_usbmidi_lib eeepc_wmi snd_hda_codec btrtl sha256_ssse3 asus_wmi iTCO_wdt snd_ump videobuf2_vmalloc sha1_ssse3 btintel intel_pmc_bxt ee1004 snd_hda_core aesni_intel snd_rawmidi uvc iTCO_vendor_support videobuf2_memops ledtrig_audio btbcm videobuf2_v4l2 mei_hdcp snd_seq_device snd_hwdep mei_pxp sparse_keymap btmtk platform_profile crypto_simd bluetooth videodev e1000e snd_pcm i8042 cryptd videobuf2_common rapl snd_timer ecdh_generic serio intel_cstate mc mousedev joydev intel_uncore snd rfkill spi_nor pcspkr ptp i2c_i801 mei_me wmi_bmof intel_wmi_thunderbolt mtd pps_core soundcore i2c_smbus mei intel_pmc_core pinctrl_cannonlake acpi_tad acpi_pad mac_hid vboxnetflt(OE) vboxnetadp(OE) vboxdrv(OE) pkcs8_key_parser i2c_dev crypto_user fuse dm_mod loop nfnetlink ip_tables x_tables ext4 crc32c_generic crc16 mbcache jbd2 hid_generic usbhid amdgpu amdxcp
[ 3389.569618]  i2c_algo_bit drm_ttm_helper ttm drm_exec mxm_wmi gpu_sched drm_suballoc_helper crc32c_intel drm_buddy spi_intel_pci spi_intel drm_display_helper xhci_pci xhci_pci_renesas video cec wmi vfio_pci vfio_pci_core irqbypass vfio_iommu_type1 vfio iommufd
[ 3389.569626] CPU: 0 PID: 38526 Comm: mdevctl Tainted: P        W IOE      6.7.9-arch1-1 #1 ad54415bbff2f0801422a3b76df850f68e71ecab
[ 3389.569628] Hardware name: ASUS System Product Name/PRIME Z390-A, BIOS 2004 11/02/2021
[ 3389.569629] RIP: 0010:__vfio_register_dev+0xbf/0x110 [vfio]
[ 3389.569634] Code: 00 4c 89 e6 e8 62 cc 8d f9 85 c0 74 29 48 89 df 89 44 24 04 e8 12 1b 00 00 8b 44 24 04 48 83 c4 08 5b 5d 41 5c c3 cc cc cc cc <0f> 0b b8 ea ff ff ff c3 cc cc cc cc c7 83 d0 03 00 00 01 00 00 00
[ 3389.569635] RSP: 0018:ffff9e260a477d08 EFLAGS: 00010246
[ 3389.569636] RAX: ffffffffc58ffc00 RBX: ffff8f1a2bbb6000 RCX: 0000000000000000
[ 3389.569637] RDX: ffffffffbc0dc678 RSI: 0000000000000001 RDI: ffff8f1a2bbb6000
[ 3389.569638] RBP: ffff8f19e5e73400 R08: 0000000000000000 R09: 0000000000000000
[ 3389.569638] R10: ffff8f1ae6e20c40 R11: 0000000000000001 R12: ffffffffc58f2590
[ 3389.569640] R13: 0000000000000000 R14: ffff8f19e5e73400 R15: ffff8f1b51cd49e0
[ 3389.569640] FS:  0000763b921327c0(0000) GS:ffff8f211dc00000(0000) knlGS:0000000000000000
[ 3389.569641] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 3389.569642] CR2: 0000000000000000 CR3: 000000010177a001 CR4: 00000000003706f0
[ 3389.569643] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 3389.569644] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[ 3389.569645] Call Trace:
[ 3389.569645]  <TASK>
[ 3389.569646]  ? __vfio_register_dev+0xbf/0x110 [vfio 93f49971511c4fb03b6cdcd910186b6091f4f5ed]
[ 3389.569651]  ? __warn+0x81/0x130
[ 3389.569653]  ? __vfio_register_dev+0xbf/0x110 [vfio 93f49971511c4fb03b6cdcd910186b6091f4f5ed]
[ 3389.569658]  ? report_bug+0x171/0x1a0
[ 3389.569661]  ? handle_bug+0x3c/0x80
[ 3389.569663]  ? exc_invalid_op+0x17/0x70
[ 3389.569664]  ? asm_exc_invalid_op+0x1a/0x20
[ 3389.569681]  ? __vfio_register_dev+0xbf/0x110 [vfio 93f49971511c4fb03b6cdcd910186b6091f4f5ed]
[ 3389.569686]  nv_vfio_mdev_probe+0x7c/0xf0 [nvidia_vgpu_vfio 86d9d7ab81227435290222677a8aae426758c411]
[ 3389.569694]  really_probe+0x19b/0x3e0
[ 3389.569697]  __driver_probe_device+0x78/0x160
[ 3389.569698]  device_driver_attach+0x45/0xa0
[ 3389.569700]  mdev_device_create+0x1d9/0x280 [mdev 0584c228d0733e09ba61936cd1c298174bc8b427]
[ 3389.569704]  create_store+0xad/0xe0 [mdev 0584c228d0733e09ba61936cd1c298174bc8b427]
[ 3389.569707]  kernfs_fop_write_iter+0x133/0x1d0
[ 3389.569710]  vfs_write+0x23a/0x400
[ 3389.569713]  ksys_write+0x6f/0xf0
[ 3389.569715]  do_syscall_64+0x61/0xe0
[ 3389.569716]  ? do_syscall_64+0x70/0xe0
[ 3389.569717]  entry_SYSCALL_64_after_hwframe+0x6e/0x76
[ 3389.569719] RIP: 0033:0x763b92231184
[ 3389.569737] Code: c7 00 16 00 00 00 b8 ff ff ff ff c3 66 2e 0f 1f 84 00 00 00 00 00 f3 0f 1e fa 80 3d c5 3e 0e 00 00 74 13 b8 01 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 54 c3 0f 1f 00 48 83 ec 28 48 89 54 24 18 48
[ 3389.569738] RSP: 002b:00007fffb0648788 EFLAGS: 00000202 ORIG_RAX: 0000000000000001
[ 3389.569740] RAX: ffffffffffffffda RBX: 0000000000000024 RCX: 0000763b92231184
[ 3389.569741] RDX: 0000000000000024 RSI: 00005d2ab5729720 RDI: 0000000000000003
[ 3389.569741] RBP: 0000000000000003 R08: 8080808080808080 R09: 01010000000f0303
[ 3389.569742] R10: 00000000000001b6 R11: 0000000000000202 R12: 7fffffffffffffff
[ 3389.569743] R13: 0000763b92231170 R14: 00005d2ab5729720 R15: 00005d2ab4815098
[ 3389.569744]  </TASK>
[ 3389.569745] ---[ end trace 0000000000000000 ]---
[ 3389.569746] [nvidia-vgpu-vfio] 48df3410-5c21-4c86-8951-7d67705cea21: Failed to probe mdev device, ret: -22
[ 3389.569751] nvidia-vgpu-vfio: probe of 48df3410-5c21-4c86-8951-7d67705cea21 failed with error -22

It might be that this is the result of my own incompetence, in which case I apologize, however I can't seem to get past this error.

awilliam commented 6 months ago

The kernel oops indicates this block of code:

        if (WARN_ON(IS_ENABLED(CONFIG_IOMMUFD) &&
                    (!device->ops->bind_iommufd ||
                     !device->ops->unbind_iommufd ||
                     !device->ops->attach_ioas ||
                     !device->ops->detach_ioas)))
                return -EINVAL;

The NVIDIA driver isn't providing callbacks which are considered required when CONFIG_IOMMUFD is enabled. Rebuild your kernel with CONFIG_IOMMUFD disabled or complain to NVIDIA for a newer driver.

Pesc0 commented 6 months ago

Thank you!