search

snu-csl / nvmevirt

NVMeVirt: A Versatile Software-defined Virtual NVMe Device
Other
182 stars 54 forks source link

Error occurred when testing P2PDMA with CUDA GPUDirect Storage #44

Closed xiurui-pan closed 4 months ago

xiurui-pan commented 4 months ago

I was trying to test P2PDMA with CUDA GPUDirect Storage with NVM SSD simulated by NVMeVirt. My OS and kernel version is Ubuntu 20.04 5.15, and my GPU card is RTX A6000 with CUDA version 12.1 and nvidia driver 550.90.07.

While everything went well with fio tests, P2PDMA with GPU and NVMeVirt-simulated SSD failed. I install the NVMeVirt module with the following commands:

sudo insmod ./nvmev.ko \ 
memmap_start=64G \
memmap_size=32G \
cpus=7,8

sudo mount /dev/nvme3n1 -o data=ordered /mnt/gds
sudo chmod 777 /mnt/gds

The code I have tested for GDS is: https://github.com/NVIDIA/MagnumIO/blob/main/gds/samples/cufile_sample_002.cc . And I have got the following error messgae:

$ ./cufile_sample_002 /mnt/gds/test-002 0  
opening file /mnt/gds/test-002
allocating device memory of size :1048576 gpu id: 0
writing from gpuid: 0
write failed : Input/output error

I have tested the GDS sample codes with real SSDs and everything worked well. I wonder if you have any ideas about GDS with NVMeVirt. Thanks again.

Seong-yeop commented 4 months ago

Can I see the dmesg log?

xiurui-pan commented 4 months ago

Can I see the dmesg log?

dmesg.log

The full dmesg log is attached in the file. I have opened CONFIG_NVMEV_VERBOSE, CONFIG_NVMEV_DEBUG, and CONFIG_NVMEV_DEBUG_VERBOSE in nvmev.h for more detailed output. The data transfer began at 18394.942079. It seems a page fault occurred:

[18396.290282] NVMeVirt: [W] 270640 + 255, prp 101fe04c1000 45b2b0000
[18396.290282] BUG: unable to handle page fault for address: ff12fccb204a0000
[18396.290284] #PF: supervisor read access in kernel mode
[18396.290285] NVMeVirt: nvmev_io_worker_0/5394[1], sq 1 cq 1, entry 274, 18396290282571 + 49
[18396.290287] NVMeVirt: [W] 270896 + 255, prp 101fe04e1000 45b2b0400
[18396.290288] #PF: error_code(0x0000) - not-present page
.......
[18458.176659] NVMeVirt: irq: int 15, vector 38
[18519.583181] nvme nvme3: I/O 23 QID 0 timeout, disable controller
[18519.583187] NVMeVirt: [R] 0x0, size: 4, val: 0x1100c51
[18519.583191] NVMeVirt: nvmev_proc_bars: cc 0x460001:0 -> 0x464001:1
[18519.585222] NVMeVirt: [R] 0x68, size: 2, val: 0x10f
[18519.585224] NVMeVirt: [W] 0x68, mask: 0xffffffff, val: 0x10f -> 0x100, size: 2, new: 0x100
[18519.585226] NVMeVirt: [R] 0x4, size: 2, val: 0x2
[18519.585247] nvme nvme3: Identify Controller failed (-4)
[18519.585260] nvme nvme3: Disabling device after reset failure: -5
[18519.599163] blk_update_request: I/O error, dev nvme3n1, sector 29622336 op 0x1:(WRITE) flags 0x800 phys_seg 7 prio class 0
[18519.599172] blk_update_request: I/O error, dev nvme3n1, sector 272176 op 0x1:(WRITE) flags 0x8800 phys_seg 1 prio class 0
[18519.599174] blk_update_request: I/O error, dev nvme3n1, sector 271920 op 0x1:(WRITE) flags 0xc800 phys_seg 1 prio class 0
[18519.599177] blk_update_request: I/O error, dev nvme3n1, sector 271664 op 0x1:(WRITE) flags 0xc800 phys_seg 1 prio class 0
[18519.599179] blk_update_request: I/O error, dev nvme3n1, sector 271408 op 0x1:(WRITE) flags 0xc800 phys_seg 1 prio class 0
[18519.599181] blk_update_request: I/O error, dev nvme3n1, sector 271152 op 0x1:(WRITE) flags 0xc800 phys_seg 1 prio class 0
[18519.599182] blk_update_request: I/O error, dev nvme3n1, sector 270896 op 0x1:(WRITE) flags 0xc800 phys_seg 1 prio class 0
[18519.599183] blk_update_request: I/O error, dev nvme3n1, sector 270640 op 0x1:(WRITE) flags 0xc800 phys_seg 2 prio class 0
[18519.599185] blk_update_request: I/O error, dev nvme3n1, sector 270384 op 0x1:(WRITE) flags 0xc800 phys_seg 1 prio class 0
[18519.599189] blk_update_request: I/O error, dev nvme3n1, sector 270376 op 0x1:(WRITE) flags 0x8800 phys_seg 1 prio class 0
[18519.599222] nvidia-fs:write IO failed :-5
[18519.599277] Aborting journal on device nvme3n1-8.
[18519.599296] Buffer I/O error on dev nvme3n1, logical block 3702784, lost sync page write
[18519.599293] nvme3n1: detected capacity change from 67106816 to 0
[18519.599299] JBD2: Error -5 detected when updating journal superblock for nvme3n1-8.
Seong-yeop commented 4 months ago

I want to see the dmesg log when NVMeVirt is initialized. Is it possible?

xiurui-pan commented 4 months ago

Of course. I only left CONFIG_NVMEV_DEBUG open, otherwise the log would be too long.

[  162.459558] NVMeVirt: Version 1.10 for >> NVM SSD <<
[  162.459568] NVMeVirt: Storage: 0x1000100000-0x1800000000 (32767 MiB)
[  162.495384] NVMeVirt: ns 0/1: size 32767 MiB
[  162.495453] PCI host bridge to bus 0001:10
[  162.495456] pci_bus 0001:10: root bus resource [io  0x0000-0xffff]
[  162.495460] pci_bus 0001:10: root bus resource [mem 0x00000000-0x3fffffffffff]
[  162.495462] pci_bus 0001:10: root bus resource [bus 00-ff]
[  162.495472] pci 0001:10:00.0: [0c51:0110] type 00 class 0x010802
[  162.495477] pci 0001:10:00.0: reg 0x10: [mem 0x1000000000-0x1000003fff 64bit]
[  162.495480] pci 0001:10:00.0: enabling Extended Tags
[  162.495836] NVMeVirt: Virtual PCI bus created (node 0)
[  162.496661] NVMeVirt: nvmev_io_worker_0 started on cpu 8 (node 0)
[  162.497020] NVMeVirt: nvmev_dispatcher started on cpu 7 (node 0)
[  162.497780] nvme nvme3: pci function 0001:10:00.0
[  162.497864] ------------[ cut here ]------------
[  162.497867] WARNING: CPU: 0 PID: 585 at include/linux/msi.h:262 pci_msi_setup_msi_irqs.isra.0+0x33/0x40
[  162.497885] Modules linked in: nvmev(OE+) xt_conntrack xt_MASQUERADE nf_conntrack_netlink nfnetlink xfrm_user xfrm_algo iptable_nat nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 libcrc32c xt_addrtype iptable_filter bpfilter br_netfilter bridge stp llc tcp_diag inet_diag aufs uio_pci_generic uio cuse overlay rdma_ucm(OE) rdma_cm(OE) iw_cm(OE) ib_ipoib(OE) ib_cm(OE) ib_umad(OE) nls_iso8859_1 ipmi_ssif intel_rapl_msr intel_rapl_common nvidia_uvm(OE) i10nm_edac nfit snd_hda_codec_realtek x86_pkg_temp_thermal snd_hda_codec_generic intel_powerclamp ledtrig_audio snd_hda_codec_hdmi snd_hda_intel snd_intel_dspcfg snd_intel_sdw_acpi snd_hda_codec kvm_intel snd_hda_core snd_hwdep snd_pcm binfmt_misc kvm snd_seq_midi ast snd_seq_midi_event nvidia_drm(OE) drm_vram_helper snd_rawmidi drm_ttm_helper crct10dif_pclmul ttm ghash_clmulni_intel snd_seq sha256_ssse3 drm_kms_helper sha1_ssse3 nvidia_modeset(OE) mlx5_ib(OE) aesni_intel cec snd_seq_device rndis_host crypto_simd ib_uverbs(OE) rc_core
[  162.498022]  snd_timer cryptd cdc_ether joydev input_leds fb_sys_fops snd syscopyarea usbnet rapl sysfillrect mei_me isst_if_mbox_pci isst_if_mmio mii intel_cstate ib_core(OE) sysimgblt isst_if_common mei soundcore ioatdma intel_pch_thermal sch_fq_codel acpi_ipmi ipmi_si acpi_pad acpi_power_meter nvidia_fs(OE) mac_hid nvidia(OE) knem(OE) hwmon_vid coretemp ipmi_devintf ipmi_msghandler msr parport_pc ppdev lp parport ramoops drm reed_solomon pstore_blk pstore_zone efi_pstore ip_tables x_tables autofs4 hid_generic usbhid hid mlx5_core(OE) pci_hyperv_intf mlxdevm(OE) crc32_pclmul mlxfw(OE) nvme(OE) igb psample atlantic nvme_core(OE) tls i2c_i801 ahci i2c_algo_bit macsec mlx_compat(OE) i2c_smbus xhci_pci dca libahci xhci_pci_renesas intel_pmt wmi
[  162.498150] CPU: 0 PID: 585 Comm: kworker/0:3 Tainted: G           OE     5.15.0-107-generic #117~20.04.1-Ubuntu
[  162.498157] Hardware name: Supermicro Super Server/X12SPA-TF, BIOS 1.1 06/21/2021
[  162.498161] Workqueue: events work_for_cpu_fn
[  162.498173] RIP: 0010:pci_msi_setup_msi_irqs.isra.0+0x33/0x40
[  162.498180] Code: 02 00 00 4d 85 c0 74 22 41 f6 40 28 01 74 1b 55 89 f2 48 8d b7 d0 00 00 00 4c 89 c7 48 89 e5 e8 83 e9 a7 ff 5d c3 cc cc cc cc <0f> 0b b8 ed ff ff ff c3 cc cc cc cc 90 0f 1f 44 00 00 48 8b 87 00
[  162.498187] RSP: 0018:ff672b3dc29afc90 EFLAGS: 00010246
[  162.498192] RAX: 0000000000000001 RBX: 0000000000000000 RCX: ff672b3dc0c0d000
[  162.498196] RDX: ff27222fce96f300 RSI: 0000000000000001 RDI: ff27222fce96f000
[  162.498200] RBP: ff672b3dc29afd00 R08: 0000000000000000 R09: ff27222f6cd16a00
[  162.498203] R10: ff27222e801e97b8 R11: ff672b3dc29afbcc R12: 0000000000000001
[  162.498206] R13: ff27222fce96f000 R14: ff27222fce96f0d0 R15: 0000000000000000
[  162.498209] FS:  0000000000000000(0000) GS:ff27224d9f400000(0000) knlGS:0000000000000000
[  162.498213] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[  162.498217] CR2: 00007f601e75a340 CR3: 0000000bd0c10004 CR4: 0000000000771ef0
[  162.498221] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[  162.498224] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[  162.498227] PKRU: 55555554
[  162.498230] Call Trace:
[  162.498233]  <TASK>
[  162.498238]  ? show_regs.cold+0x1a/0x1f
[  162.498248]  ? pci_msi_setup_msi_irqs.isra.0+0x33/0x40
[  162.498253]  ? __warn+0x8b/0xe0
[  162.498263]  ? pci_msi_setup_msi_irqs.isra.0+0x33/0x40
[  162.498270]  ? report_bug+0xd5/0x110
[  162.498279]  ? handle_bug+0x39/0x90
[  162.498290]  ? exc_invalid_op+0x19/0x70
[  162.498297]  ? asm_exc_invalid_op+0x1b/0x20
[  162.498308]  ? pci_msi_setup_msi_irqs.isra.0+0x33/0x40
[  162.498313]  ? __pci_enable_msix+0x36d/0x490
[  162.498321]  pci_alloc_irq_vectors_affinity+0xed/0x1a0
[  162.498331]  nvme_pci_enable+0x7b/0x400 [nvme]
[  162.498345]  nvme_probe.cold+0x44/0x2a0 [nvme]
[  162.498359]  local_pci_probe+0x48/0x90
[  162.498367]  work_for_cpu_fn+0x17/0x30
[  162.498373]  process_one_work+0x228/0x3d0
[  162.498382]  worker_thread+0x21d/0x3f0
[  162.498389]  ? process_one_work+0x3d0/0x3d0
[  162.498396]  kthread+0x127/0x150
[  162.498403]  ? set_kthread_struct+0x50/0x50
[  162.498410]  ret_from_fork+0x1f/0x30
[  162.498423]  </TASK>
[  162.498425] ---[ end trace b2f1ddc2f767359c ]---
[  162.498444] ------------[ cut here ]------------
[  162.498446] WARNING: CPU: 0 PID: 585 at include/linux/msi.h:268 free_msi_irqs+0x74/0x140
[  162.498455] Modules linked in: nvmev(OE+) xt_conntrack xt_MASQUERADE nf_conntrack_netlink nfnetlink xfrm_user xfrm_algo iptable_nat nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 libcrc32c xt_addrtype iptable_filter bpfilter br_netfilter bridge stp llc tcp_diag inet_diag aufs uio_pci_generic uio cuse overlay rdma_ucm(OE) rdma_cm(OE) iw_cm(OE) ib_ipoib(OE) ib_cm(OE) ib_umad(OE) nls_iso8859_1 ipmi_ssif intel_rapl_msr intel_rapl_common nvidia_uvm(OE) i10nm_edac nfit snd_hda_codec_realtek x86_pkg_temp_thermal snd_hda_codec_generic intel_powerclamp ledtrig_audio snd_hda_codec_hdmi snd_hda_intel snd_intel_dspcfg snd_intel_sdw_acpi snd_hda_codec kvm_intel snd_hda_core snd_hwdep snd_pcm binfmt_misc kvm snd_seq_midi ast snd_seq_midi_event nvidia_drm(OE) drm_vram_helper snd_rawmidi drm_ttm_helper crct10dif_pclmul ttm ghash_clmulni_intel snd_seq sha256_ssse3 drm_kms_helper sha1_ssse3 nvidia_modeset(OE) mlx5_ib(OE) aesni_intel cec snd_seq_device rndis_host crypto_simd ib_uverbs(OE) rc_core
[  162.498563]  snd_timer cryptd cdc_ether joydev input_leds fb_sys_fops snd syscopyarea usbnet rapl sysfillrect mei_me isst_if_mbox_pci isst_if_mmio mii intel_cstate ib_core(OE) sysimgblt isst_if_common mei soundcore ioatdma intel_pch_thermal sch_fq_codel acpi_ipmi ipmi_si acpi_pad acpi_power_meter nvidia_fs(OE) mac_hid nvidia(OE) knem(OE) hwmon_vid coretemp ipmi_devintf ipmi_msghandler msr parport_pc ppdev lp parport ramoops drm reed_solomon pstore_blk pstore_zone efi_pstore ip_tables x_tables autofs4 hid_generic usbhid hid mlx5_core(OE) pci_hyperv_intf mlxdevm(OE) crc32_pclmul mlxfw(OE) nvme(OE) igb psample atlantic nvme_core(OE) tls i2c_i801 ahci i2c_algo_bit macsec mlx_compat(OE) i2c_smbus xhci_pci dca libahci xhci_pci_renesas intel_pmt wmi
[  162.498666] CPU: 0 PID: 585 Comm: kworker/0:3 Tainted: G        W  OE     5.15.0-107-generic #117~20.04.1-Ubuntu
[  162.498671] Hardware name: Supermicro Super Server/X12SPA-TF, BIOS 1.1 06/21/2021
[  162.498674] Workqueue: events work_for_cpu_fn
[  162.498681] RIP: 0010:free_msi_irqs+0x74/0x140
[  162.498687] Code: d0 00 00 00 e8 3d e8 a7 ff 49 c7 85 78 09 00 00 00 00 00 00 49 8b bd e8 02 00 00 48 85 ff 74 0a f6 47 28 01 0f 85 b0 00 00 00 <0f> 0b 4d 8b ad 00 03 00 00 49 bf 00 01 00 00 00 00 ad de 4d 8b 75
[  162.498692] RSP: 0018:ff672b3dc29afc60 EFLAGS: 00010246
[  162.498697] RAX: ff27222fce96f300 RBX: ff27222fce96f300 RCX: 0000000000000000
[  162.498700] RDX: ff27222fce96f300 RSI: 0000000000000000 RDI: 0000000000000000
[  162.498703] RBP: ff672b3dc29afc88 R08: 0000000000000000 R09: ff27222f6cd16a00
[  162.498706] R10: ff27222e801e97b8 R11: ff672b3dc29afbcc R12: ff27222fce96f300
[  162.498709] R13: ff27222fce96f000 R14: ff27222fce96f0d0 R15: 00000000ffffffed
[  162.498712] FS:  0000000000000000(0000) GS:ff27224d9f400000(0000) knlGS:0000000000000000
[  162.498715] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[  162.498718] CR2: 00007f601e75a340 CR3: 0000000bd0c10004 CR4: 0000000000771ef0
[  162.498722] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[  162.498724] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[  162.498727] PKRU: 55555554
[  162.498729] Call Trace:
[  162.498730]  <TASK>
[  162.498732]  ? show_regs.cold+0x1a/0x1f
[  162.498738]  ? free_msi_irqs+0x74/0x140
[  162.498743]  ? __warn+0x8b/0xe0
[  162.498751]  ? free_msi_irqs+0x74/0x140
[  162.498756]  ? report_bug+0xd5/0x110
[  162.498763]  ? handle_bug+0x39/0x90
[  162.498770]  ? exc_invalid_op+0x19/0x70
[  162.498778]  ? asm_exc_invalid_op+0x1b/0x20
[  162.498785]  ? free_msi_irqs+0x74/0x140
[  162.498791]  __pci_enable_msix+0x43d/0x490
[  162.498797]  pci_alloc_irq_vectors_affinity+0xed/0x1a0
[  162.498805]  nvme_pci_enable+0x7b/0x400 [nvme]
[  162.498815]  nvme_probe.cold+0x44/0x2a0 [nvme]
[  162.498825]  local_pci_probe+0x48/0x90
[  162.498832]  work_for_cpu_fn+0x17/0x30
[  162.498838]  process_one_work+0x228/0x3d0
[  162.498846]  worker_thread+0x21d/0x3f0
[  162.498853]  ? process_one_work+0x3d0/0x3d0
[  162.498860]  kthread+0x127/0x150
[  162.498866]  ? set_kthread_struct+0x50/0x50
[  162.498872]  ret_from_fork+0x1f/0x30
[  162.498881]  </TASK>
[  162.498883] ---[ end trace b2f1ddc2f767359d ]---
[  162.498911] NVMeVirt: nvmev_proc_bars: aqa 0x0 -> 0x1f001f
[  162.498919] NVMeVirt: nvmev_proc_bars: asq 0x0 -> 0x1ec744000
[  162.498925] NVMeVirt: nvmev_proc_bars: acq 0x0 -> 0x24eae1000
[  162.498928] NVMeVirt: nvmev_proc_bars: cc 0x0:0 -> 0x460001:0
[  162.501180] NVMeVirt: __nvmev_proc_admin_req: 0 0x6 0x18
[  162.501561] NVMeVirt: __nvmev_proc_admin_req: 1 0x2 0x19
[  162.501972] NVMeVirt: __nvmev_proc_admin_req: 2 0x9 0x1a
[  162.502495] NVMeVirt: __nvmev_proc_admin_req: 3 0x5 0x1b
[  162.502750] NVMeVirt: __nvmev_proc_admin_req: 4 0x1 0x1018
[  162.502939] nvme nvme3: 1/0/0 default/read/poll queues
[  162.503215] NVMeVirt: __nvmev_proc_admin_req: 5 0x6 0x10
[  162.503281] NVMeVirt: __nvmev_proc_admin_req: 6 0x6 0x4
[  162.503575] NVMeVirt: __nvmev_proc_admin_req: 7 0x6 0x5
[  162.505846] NVMeVirt: Virtual NVMe device created
[  162.573622] EXT4-fs (nvme3n1): mounted filesystem with ordered data mode. Opts: data=ordered. Quota mode: none.
Seong-yeop commented 4 months ago

It seems that NVMeVirt module is not loaded properly. In normal cases, such msi related warnings should not appear.

Could you apply intremap setting below? 

GRUB_CMDLINE_LINUX="memmap=64G\\\$128G intremap=off"

and also please check if your I/OAT DMA engine was initialized. (If it's not initialized, you can see "Cannot use DMA engine, Fall back to memcpy" in your dmesg)

xiurui-pan commented 4 months ago

I tried to apply the intremap setting, but it still didn't work. "Cannot use DMA engine, Fall back to memcpy" did not show in the dmesg log, so I guess the I/OAT DMA engine was initialized.

$ sudo dmesg
......
[ 3837.006839] NVMeVirt: Version 1.10 for >> NVM SSD <<
[ 3837.006850] NVMeVirt: Storage: 0x1000100000-0x1800000000 (32767 MiB)
[ 3837.042325] NVMeVirt: ns 0/1: size 32767 MiB
[ 3837.042393] PCI host bridge to bus 0001:10
[ 3837.042396] pci_bus 0001:10: root bus resource [io  0x0000-0xffff]
[ 3837.042399] pci_bus 0001:10: root bus resource [mem 0x00000000-0x3fffffffffff]
[ 3837.042402] pci_bus 0001:10: root bus resource [bus 00-ff]
[ 3837.042411] pci 0001:10:00.0: [0c51:0110] type 00 class 0x010802
[ 3837.042416] pci 0001:10:00.0: reg 0x10: [mem 0x1000000000-0x1000003fff 64bit]
[ 3837.042419] pci 0001:10:00.0: enabling Extended Tags
[ 3837.042766] NVMeVirt: Virtual PCI bus created (node 0)
[ 3837.043586] NVMeVirt: nvmev_io_worker_0 started on cpu 8 (node 0)
[ 3837.043842] NVMeVirt: nvmev_dispatcher started on cpu 7 (node 0)
[ 3837.044428] nvme nvme3: pci function 0001:10:00.0
[ 3837.044571] NVMeVirt: nvmev_proc_bars: aqa 0x0 -> 0x1f001f
[ 3837.044579] NVMeVirt: nvmev_proc_bars: asq 0x0 -> 0x20748c000
[ 3837.044584] NVMeVirt: nvmev_proc_bars: acq 0x0 -> 0x257bd9000
[ 3837.044588] NVMeVirt: nvmev_proc_bars: cc 0x0:0 -> 0x460001:0
......
[ 3927.554421] BUG: unable to handle page fault for address: ff3fba70a04a0000
[ 3927.554425] #PF: supervisor read access in kernel mode
[ 3927.554426] #PF: error_code(0x0000) - not-present page
[ 3927.554428] PGD 496001067 P4D 0 
[ 3927.554430] Oops: 0000 [#1] SMP NOPTI
[ 3927.554432] CPU: 8 PID: 4418 Comm: nvmev_io_worker Tainted: G           OE     5.15.0-107-generic #117~20.04.1-Ubuntu
[ 3927.554434] Hardware name: Supermicro Super Server/X12SPA-TF, BIOS 1.1 06/21/2021
[ 3927.554435] RIP: 0010:nvmev_io_worker+0x791/0xb00 [nvmev]
[ 3927.554440] Code: e0 06 48 01 f8 e9 55 ff ff ff 48 8b 3d 60 85 00 00 48 01 ce 89 c1 48 8b bf 58 01 00 00 4c 8b 44 1f 10 4d 01 e0 83 f8 08 72 5e <48> 8b 0e 49 89 08 89 c1 48 8b 7c 0e f8 49 89 7c 08 f8 49 8d 78 08
[ 3927.554441] RSP: 0018:ff7fec8980f0be80 EFLAGS: 00010212
[ 3927.554443] RAX: 0000000000001000 RBX: 0000000000000000 RCX: 0000000000001000
[ 3927.554444] RDX: ff3faa5231d0a000 RSI: ff3fba70a04a0000 RDI: ff3faa52465a35c0
[ 3927.554445] RBP: ff7fec8980f0bf08 R08: ff7fec89c8405000 R09: 0000000000001000
[ 3927.554446] R10: ff3faa53208b0040 R11: 0000000000000001 R12: 0000000008405000
[ 3927.554447] R13: ff3faa5231d0a000 R14: 0000000000000000 R15: 0000000000001000
[ 3927.554448] FS:  0000000000000000(0000) GS:ff3faa70df600000(0000) knlGS:0000000000000000
[ 3927.554449] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 3927.554450] CR2: ff3fba70a04a0000 CR3: 0000000494610006 CR4: 0000000000771ee0
[ 3927.554452] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 3927.554452] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[ 3927.554453] PKRU: 55555554
[ 3927.554454] Call Trace:
[ 3927.554455]  <TASK>
[ 3927.554457]  ? show_regs.cold+0x1a/0x1f
[ 3927.554461]  ? __die_body+0x20/0x70
[ 3927.554465]  ? __die+0x2b/0x37
[ 3927.554467]  ? page_fault_oops+0x136/0x2c0
[ 3927.554469]  ? search_bpf_extables+0x63/0x80
[ 3927.554473]  ? nvmev_io_worker+0x791/0xb00 [nvmev]
[ 3927.554475]  ? search_exception_tables+0x5f/0x70
[ 3927.554477]  ? kernelmode_fixup_or_oops+0xa2/0x120
[ 3927.554479]  ? __bad_area_nosemaphore+0x15d/0x1a0
[ 3927.554481]  ? bad_area_nosemaphore+0x16/0x20
[ 3927.554482]  ? do_kern_addr_fault+0x62/0x80
[ 3927.554484]  ? exc_page_fault+0xe7/0x170
[ 3927.554486]  ? asm_exc_page_fault+0x27/0x30
[ 3927.554489]  ? nvmev_io_worker+0x791/0xb00 [nvmev]
[ 3927.554492]  ? __allocate_work_queue_entry+0x90/0x90 [nvmev]
[ 3927.554494]  kthread+0x127/0x150
[ 3927.554496]  ? set_kthread_struct+0x50/0x50
[ 3927.554498]  ret_from_fork+0x1f/0x30
[ 3927.554502]  </TASK>
[ 3927.554503] Modules linked in: nvmev(OE) tcp_diag inet_diag xt_conntrack xt_MASQUERADE nf_conntrack_netlink nfnetlink xfrm_user xfrm_algo iptable_nat nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 libcrc32c xt_addrtype iptable_filter bpfilter br_netfilter bridge stp llc aufs uio_pci_generic uio cuse overlay rdma_ucm(OE) rdma_cm(OE) iw_cm(OE) ib_ipoib(OE) ib_cm(OE) ib_umad(OE) nls_iso8859_1 ipmi_ssif snd_hda_codec_realtek intel_rapl_msr snd_hda_codec_generic intel_rapl_common ledtrig_audio i10nm_edac snd_hda_codec_hdmi nfit x86_pkg_temp_thermal nvidia_uvm(OE) intel_powerclamp kvm_intel snd_hda_intel binfmt_misc snd_intel_dspcfg snd_intel_sdw_acpi snd_hda_codec kvm snd_hda_core snd_hwdep ast snd_pcm crct10dif_pclmul ghash_clmulni_intel sha256_ssse3 snd_seq_midi nvidia_drm(OE) sha1_ssse3 drm_vram_helper snd_seq_midi_event aesni_intel drm_ttm_helper snd_rawmidi ttm crypto_simd snd_seq cryptd nvidia_modeset(OE) drm_kms_helper snd_seq_device mlx5_ib(OE) rndis_host cec snd_timer cdc_ether
[ 3927.554540]  snd rapl usbnet rc_core ib_uverbs(OE) mei_me fb_sys_fops syscopyarea isst_if_mbox_pci isst_if_mmio sysfillrect joydev mii input_leds intel_cstate ib_core(OE) sch_fq_codel sysimgblt isst_if_common soundcore mei intel_pch_thermal ioatdma nvidia_fs(OE) acpi_ipmi ipmi_si acpi_pad acpi_power_meter mac_hid nvidia(OE) knem(OE) hwmon_vid coretemp ipmi_devintf ipmi_msghandler msr parport_pc ppdev lp parport drm ramoops pstore_blk reed_solomon pstore_zone efi_pstore ip_tables x_tables autofs4 hid_generic usbhid hid mlx5_core(OE) pci_hyperv_intf mlxdevm(OE) crc32_pclmul igb nvme(OE) mlxfw(OE) psample atlantic nvme_core(OE) i2c_algo_bit tls i2c_i801 macsec ahci mlx_compat(OE) xhci_pci i2c_smbus dca libahci xhci_pci_renesas intel_pmt wmi
[ 3927.554577] CR2: ff3fba70a04a0000
[ 3927.554578] ---[ end trace 1394ff10587746e2 ]---
[ 3927.673068] RIP: 0010:nvmev_io_worker+0x791/0xb00 [nvmev]
[ 3927.673076] Code: e0 06 48 01 f8 e9 55 ff ff ff 48 8b 3d 60 85 00 00 48 01 ce 89 c1 48 8b bf 58 01 00 00 4c 8b 44 1f 10 4d 01 e0 83 f8 08 72 5e <48> 8b 0e 49 89 08 89 c1 48 8b 7c 0e f8 49 89 7c 08 f8 49 8d 78 08
[ 3927.673078] RSP: 0018:ff7fec8980f0be80 EFLAGS: 00010212
[ 3927.673081] RAX: 0000000000001000 RBX: 0000000000000000 RCX: 0000000000001000
[ 3927.673082] RDX: ff3faa5231d0a000 RSI: ff3fba70a04a0000 RDI: ff3faa52465a35c0
[ 3927.673083] RBP: ff7fec8980f0bf08 R08: ff7fec89c8405000 R09: 0000000000001000
[ 3927.673084] R10: ff3faa53208b0040 R11: 0000000000000001 R12: 0000000008405000
[ 3927.673086] R13: ff3faa5231d0a000 R14: 0000000000000000 R15: 0000000000001000
[ 3927.673087] FS:  0000000000000000(0000) GS:ff3faa70df600000(0000) knlGS:0000000000000000
[ 3927.673088] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 3927.673090] CR2: ff3fba70a04a0000 CR3: 000000010e5e4005 CR4: 0000000000771ee0
[ 3927.673091] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 3927.673092] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[ 3927.673094] PKRU: 55555554
[ 3958.223796] nvme nvme3: I/O 193 (I/O Cmd) QID 14 timeout, aborting
[ 3958.223804] nvme nvme3: I/O 194 (I/O Cmd) QID 14 timeout, aborting
[ 3958.223806] nvme nvme3: I/O 195 (I/O Cmd) QID 14 timeout, aborting
[ 3958.223807] nvme nvme3: I/O 196 (I/O Cmd) QID 14 timeout, aborting
[ 3958.223827] NVMeVirt: __nvmev_proc_admin_req: 14 0x8 0x0
[ 3958.223831] NVMeVirt: __nvmev_proc_admin_req: 15 0x8 0x1
[ 3958.223832] NVMeVirt: __nvmev_proc_admin_req: 16 0x8 0x2
[ 3958.223833] NVMeVirt: __nvmev_proc_admin_req: 17 0x8 0x3
[ 3988.943758] nvme nvme3: I/O 193 QID 14 timeout, reset controller
......
[ 3988.944477] NVMeVirt: nvmev_proc_bars: cc 0x460001:0 -> 0x460000:1
[ 3988.946823] nvme nvme3: Abort status: 0x371
[ 3988.946826] nvme nvme3: Abort status: 0x371
[ 3988.946827] nvme nvme3: Abort status: 0x371
[ 3988.946827] nvme nvme3: Abort status: 0x371
[ 3988.959889] NVMeVirt: nvmev_proc_bars: cc 0x460000:0 -> 0x460001:0
[ 3988.961941] NVMeVirt: __nvmev_proc_admin_req: 26 0x4 0x2013
[ 3988.961942] NVMeVirt: __nvmev_proc_admin_req: 27 0x4 0x2014
[ 3988.961943] NVMeVirt: __nvmev_proc_admin_req: 28 0x4 0x2015
[ 3988.961944] NVMeVirt: __nvmev_proc_admin_req: 29 0x4 0x2016
[ 3988.961945] NVMeVirt: __nvmev_proc_admin_req: 30 0x4 0x2017
[ 3988.961945] NVMeVirt: __nvmev_proc_admin_req: 31 0x4 0x201d
[ 3988.961946] NVMeVirt: __nvmev_proc_admin_req: 0 0x6 0x4014
[ 4050.383749] nvme nvme3: I/O 20 QID 0 timeout, disable controller
[ 4050.383765] NVMeVirt: nvmev_proc_bars: cc 0x460001:0 -> 0x464001:1
[ 4050.385841] nvme nvme3: Identify Controller failed (-4)
[ 4050.385846] nvme nvme3: Disabling device after reset failure: -5
[ 4050.399702] blk_update_request: I/O error, dev nvme3n1, sector 272176 op 0x1:(WRITE) flags 0x8800 phys_seg 1 prio class 0
[ 4050.399707] blk_update_request: I/O error, dev nvme3n1, sector 29622336 op 0x1:(WRITE) flags 0x800 phys_seg 7 prio class 0
[ 4050.399708] blk_update_request: I/O error, dev nvme3n1, sector 271920 op 0x1:(WRITE) flags 0xc800 phys_seg 1 prio class 0
[ 4050.399712] blk_update_request: I/O error, dev nvme3n1, sector 271664 op 0x1:(WRITE) flags 0xc800 phys_seg 1 prio class 0
[ 4050.399715] blk_update_request: I/O error, dev nvme3n1, sector 271408 op 0x1:(WRITE) flags 0xc800 phys_seg 1 prio class 0
[ 4050.399717] blk_update_request: I/O error, dev nvme3n1, sector 271152 op 0x1:(WRITE) flags 0xc800 phys_seg 1 prio class 0
[ 4050.399719] blk_update_request: I/O error, dev nvme3n1, sector 270896 op 0x1:(WRITE) flags 0xc800 phys_seg 1 prio class 0
[ 4050.399721] blk_update_request: I/O error, dev nvme3n1, sector 270640 op 0x1:(WRITE) flags 0xc800 phys_seg 1 prio class 0
[ 4050.399723] blk_update_request: I/O error, dev nvme3n1, sector 270384 op 0x1:(WRITE) flags 0xc800 phys_seg 2 prio class 0
[ 4050.399726] Aborting journal on device nvme3n1-8.
[ 4050.399730] blk_update_request: I/O error, dev nvme3n1, sector 270376 op 0x1:(WRITE) flags 0x8800 phys_seg 1 prio class 0
[ 4050.399739] EXT4-fs error (device nvme3n1): ext4_journal_check_start:83: comm cufile_sample_0: Detected aborted journal
[ 4050.399989] nvme3n1: detected capacity change from 67106816 to 0
[ 4050.399991] Buffer I/O error on dev nvme3n1, logical block 3702784, lost sync page write
[ 4050.399993] JBD2: Error -5 detected when updating journal superblock for nvme3n1-8.
[ 4050.400025] Buffer I/O error on dev nvme3n1, logical block 0, lost sync page write
[ 4050.400030] EXT4-fs (nvme3n1): I/O error while writing superblock
[ 4050.400033] EXT4-fs (nvme3n1): Remounting filesystem read-only
[ 4050.400040] nvidia-fs:write IO failed :-5

$ cat /etc/default/grub | grep GRUB_CMDLINE_LINUX                                                                                                                
GRUB_CMDLINE_LINUX_DEFAULT="quiet splash intel_iommu=off"
GRUB_CMDLINE_LINUX="memmap=32G\\\$64G intremap=off"
Seong-yeop commented 4 months ago

Please check if the machine is a NUMA node. I suspect there might be an issue because the PCIe root is allocated at the location of NUMA 1.

xiurui-pan commented 4 months ago

Thank you for your kind reply, but the machine does not contain a NUMA node. I'm afraid there exists some other issues.

$ numactl -H
available: 1 nodes (0)
node 0 cpus: 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51
node 0 size: 96155 MB
node 0 free: 69182 MB
node distances:
node   0 
  0:  10
jaehshim commented 4 months ago

Have you turned on the io_using_dma? Check https://github.com/snu-csl/nvmevirt/blob/07acdd24e03d699bc88b272966d083bad21c82c6/main.c#L76C5-L76C17

xiurui-pan commented 4 months ago

Thank you for your help! This works for me.

jaehshim commented 4 months ago

Good to hear that it works! Using DMA engine is very important when running P2P DMA between real PCIe device and NVMeVirt.