search

nutanix / libvfio-user

framework for emulating devices in userspace
BSD 3-Clause "New" or "Revised" License
166 stars 51 forks source link

QEMU aborted when doing reboot inside the VM #439

Open changpe1 opened 3 years ago

changpe1 commented 3 years ago

Environment:

QEMU: vfio-user-v0.6 libvfiouser: latest master with commit ba6abc4ce1

Got the following errors when doing reboot inside VM:

38512@1618242898.230520:vfio_listener_region_add_ram region_add [ram] 0xc0000 - 0xbfffffff [0x7fcaa7ec0000]
qemu-system-x86_64: kvm_set_user_memory_region: KVM_SET_USER_MEMORY_REGION failed, slot=4, start=0xc0000, size=0xbff40000: File exists
kvm_set_phys_mem: error registering slot: File exists
changpe1 commented 3 years ago

Here the vfio trace in the QEMU side:

49519@1619090661.362566:vfio_listener_region_add_ram region_add [ram] 0x0 - 0x9ffff [0x7f50b3e00000]
49519@1619090661.362794:vfio_listener_region_add_skip SKIPPING region_add 0xa0000 - 0xbffff
49519@1619090661.362806:vfio_listener_region_add_ram region_add [ram] 0xc0000 - 0xdffff [0x7f50b3200000]
49519@1619090661.362855:vfio_listener_region_add_ram region_add [ram] 0xe0000 - 0xfffff [0x7f50b3420000]
49519@1619090661.362904:vfio_listener_region_add_ram region_add [ram] 0x100000 - 0xbfffffff [0x7f50b3f00000]
49519@1619090661.454098:vfio_listener_region_add_skip SKIPPING region_add 0xfec00000 - 0xfec00fff
49519@1619090661.454123:vfio_listener_region_add_skip SKIPPING region_add 0xfed00000 - 0xfed003ff
49519@1619090661.454131:vfio_listener_region_add_skip SKIPPING region_add 0xfee00000 - 0xfeefffff
49519@1619090661.454138:vfio_listener_region_add_ram region_add [ram] 0xfffc0000 - 0xffffffff [0x7f50b3400000]
49519@1619090661.454216:vfio_listener_region_add_ram region_add [ram] 0x100000000 - 0x13fffffff [0x7f5173e00000]
49519@1619090661.485242:vfio_region_sparse_mmap_header Device VFIO user </var/run/muser/domain/muser0/8/cntrl> region 0: 1 sparse mmap entries
49519@1619090661.485265:vfio_region_sparse_mmap_entry sparse entry 0 [0x1000 - 0x2000]
49519@1619090661.485274:vfio_region_setup Device VFIO user </var/run/muser/domain/muser0/8/cntrl>, region 0 "VFIO user </var/run/muser/domain/muser0/8/cntrl> BAR 0", flags: 0xf, offset: 0x0, size: 0x4000
49519@1619090661.485331:vfio_region_setup Device VFIO user </var/run/muser/domain/muser0/8/cntrl>, region 1 "VFIO user </var/run/muser/domain/muser0/8/cntrl> BAR 1", flags: 0x0, offset: 0x10000000000, size: 0x0
49519@1619090661.485383:vfio_region_setup Device VFIO user </var/run/muser/domain/muser0/8/cntrl>, region 2 "VFIO user </var/run/muser/domain/muser0/8/cntrl> BAR 2", flags: 0x0, offset: 0x20000000000, size: 0x0
49519@1619090661.485433:vfio_region_setup Device VFIO user </var/run/muser/domain/muser0/8/cntrl>, region 3 "VFIO user </var/run/muser/domain/muser0/8/cntrl> BAR 3", flags: 0x0, offset: 0x30000000000, size: 0x0
49519@1619090661.485496:vfio_region_setup Device VFIO user </var/run/muser/domain/muser0/8/cntrl>, region 4 "VFIO user </var/run/muser/domain/muser0/8/cntrl> BAR 4", flags: 0x3, offset: 0x40000000000, size: 0x1000
49519@1619090661.485559:vfio_region_setup Device VFIO user </var/run/muser/domain/muser0/8/cntrl>, region 5 "VFIO user </var/run/muser/domain/muser0/8/cntrl> BAR 5", flags: 0x3, offset: 0x50000000000, size: 0x1000
49519@1619090661.485607:vfio_populate_device_config Device VFIO user </var/run/muser/domain/muser0/8/cntrl> config:
  size: 0x1000, offset: 0x70000000000, flags: 0x3
49519@1619090661.485715:vfio_msix_early_setup VFIO user </var/run/muser/domain/muser0/8/cntrl> PCI MSI-X CAP @0x84, BAR 4, offset 0x0, entries 64
49519@1619090661.486354:vfio_region_mmap Region VFIO user </var/run/muser/domain/muser0/8/cntrl> BAR 0 mmaps[0] [0x1000 - 0x1fff]
49519@1619090661.487971:vfio_intx_enable_kvm  (VFIO user </var/run/muser/domain/muser0/8/cntrl>) KVM INTx accel enabled
49519@1619090661.487985:vfio_intx_enable  (VFIO user </var/run/muser/domain/muser0/8/cntrl>)
49519@1619090661.488147:vfio_get_dev_region VFIO user </var/run/muser/domain/muser0/8/cntrl> index 9, 00000003/18
49519@1619090661.488160:vfio_region_setup Device VFIO user </var/run/muser/domain/muser0/8/cntrl>, region 9 "migration", flags: 0x8, offset: 0x90000000000, size: 0x0
qemu-system-x86_64: -device vfio-user-pci,socket=/var/run/muser/domain/muser0/8/cntrl: VFIO user </var/run/muser/domain/muser0/8/cntrl>: Invalid zero-sized VFIO migration region 9
49519@1619090661.505461:vfio_intx_disable_kvm  (VFIO user </var/run/muser/domain/muser0/8/cntrl>) KVM INTx accel disabled
49519@1619090661.505526:vfio_region_mmaps_set_enabled Region VFIO user </var/run/muser/domain/muser0/8/cntrl> BAR 0 mmaps enabled: 1
49519@1619090661.505535:vfio_region_mmaps_set_enabled Region VFIO user </var/run/muser/domain/muser0/8/cntrl> BAR 4 mmaps enabled: 1
49519@1619090661.505546:vfio_region_mmaps_set_enabled Region VFIO user </var/run/muser/domain/muser0/8/cntrl> BAR 5 mmaps enabled: 1
49519@1619090661.505565:vfio_intx_disable  (VFIO user </var/run/muser/domain/muser0/8/cntrl>)
49519@1619090661.505618:vfio_pci_read_config  (VFIO user </var/run/muser/domain/muser0/8/cntrl>, @0x44, len=0x2) 0x8
49519@1619090661.505668:vfio_pci_read_config  (VFIO user </var/run/muser/domain/muser0/8/cntrl>, @0x4, len=0x2) 0x0
49519@1619090661.505679:vfio_pci_write_config  (VFIO user </var/run/muser/domain/muser0/8/cntrl>, @0x4, 0x0, len=0x2)
49519@1619090661.505782:vfio_intx_enable  (VFIO user </var/run/muser/domain/muser0/8/cntrl>)
49526@1619090661.511185:vfio_listener_region_del region_del 0xc0000 - 0xdffff
49526@1619090661.511405:vfio_listener_region_del region_del 0xe0000 - 0xfffff
49526@1619090661.512559:vfio_listener_region_del region_del 0x100000 - 0xbfffffff
49526@1619090661.551021:vfio_listener_region_add_ram region_add [ram] 0xc0000 - 0xcffff [0x7f50b3ec0000]
49526@1619090661.551261:vfio_listener_region_add_ram region_add [ram] 0xd0000 - 0xdffff [0x7f50b3210000]
49526@1619090661.551367:vfio_listener_region_add_ram region_add [ram] 0xe0000 - 0xeffff [0x7f50b3420000]
49526@1619090661.551455:vfio_listener_region_add_ram region_add [ram] 0xf0000 - 0xbfffffff [0x7f50b3ef0000]
49526@1619090661.649249:vfio_listener_region_del region_del 0xc0000 - 0xcffff
49520@1619090661.649625:vfio_listener_region_del region_del 0xc0000 - 0xcffff
49526@1619090661.649715:vfio_listener_region_del region_del 0xd0000 - 0xdffff
qemu-system-x86_64: vfio_dma_unmap(0x55b8b810c000, 0xc0000, 0x10000) = -2 (Success)
49520@1619090661.649796:vfio_listener_region_del region_del 0xd0000 - 0xdffff
49526@1619090661.649874:vfio_listener_region_del region_del 0xe0000 - 0xeffff
qemu-system-x86_64: vfio_dma_unmap(0x55b8b810c000, 0xd0000, 0x10000) = -2 (Success)
49520@1619090661.649915:vfio_listener_region_del region_del 0xe0000 - 0xeffff
49526@1619090661.650983:vfio_listener_region_del region_del 0xf0000 - 0xbfffffff
qemu-system-x86_64: vfio_dma_unmap(0x55b8b810c000, 0xe0000, 0x10000) = -2 (Success)
49520@1619090661.651106:vfio_listener_region_del region_del 0xf0000 - 0xbfffffff
49526@1619090661.689651:vfio_listener_region_add_ram region_add [ram] 0xc0000 - 0xbfffffff [0x7f50b3ec0000]
qemu-system-x86_64: vfio_dma_unmap(0x55b8b810c000, 0xf0000, 0xbff10000) = -2 (Success)
49520@1619090661.689718:vfio_listener_region_add_ram region_add [ram] 0xc0000 - 0xbfffffff [0x7f50b3ec0000]
qemu-system-x86_64: kvm_set_user_memory_region: KVM_SET_USER_MEMORY_REGION failed, slot=4, start=0xc0000, size=0xbff40000: File exists
kvm_set_phys_mem: error registering slot: File exists
tmakatos commented 3 years ago

I can't reproduce this in my nested setup (KVM is enabled):

qemu-system-x86_64 -cpu host -smp 1 -enable-kvm -nographic -m 2G -object memory-backend-file,id=mem0,size=2G,mem-path=/dev/hugepages,share=on,prealloc=yes, -numa node,memdev=mem0 -kernel bionic-server-cloudimg-amd64-vmlinuz-generic -initrd bionic-server-cloudimg-amd64-initrd-generic -append console=ttyS0 root=/dev/sda1 single -hda bionic-server-cloudimg-amd64-0.raw -device vfio-user-pci,socket=/var/run/muser/domain/muser0/8/cntrl -trace enable=vfio*

This might be a KVM bug. What's your kernel version? I'm testing with 5.4.

changpe1 commented 3 years ago

@tmakatos I found the issue can only happen when I used the physical NVMe SSDs as the backend, the difference is SPDK will call spdk_mem_register() to register VM's memory regions to the kernel vfio driver(if the physical NVMe SSD is attached to vfio-pci driver).

For your test case, it should work well.

changpe1 commented 3 years ago

I can confirm that when the issue happened, there are still memory region hold by kernel vfio-pci driver(should be unregistered), there maybe mismatch between QEMU/libvfio-user/SPDK, I can still not confirm in which part.

tmakatos commented 3 years ago

I see, looks like I need a very specific setup which will take me some time to prepare. Does this repro reliably? Also, does the passed through disk to SPDK have to be NVMe? Does it still repro with a SCSI disk?

Also, can you provide the SPDK and QEMU commits you're using?

changpe1 commented 3 years ago

Yeah, it's very easy to reproduce, even without rebooting inside VM, I can still reproduce it.

  1. Start QEMU
  2. Kill QEMU
  3. Restart QEMU
changpe1 commented 3 years ago 
49526@1619090661.689651:vfio_listener_region_add_ram region_add [ram] 0xc0000 - 0xbfffffff [0x7f50b3ec0000]
qemu-system-x86_64: vfio_dma_unmap(0x55b8b810c000, 0xf0000, 0xbff10000) = -2 (Success)
49520@1619090661.689718:vfio_listener_region_add_ram region_add [ram] 0xc0000 - 0xbfffffff [0x7f50b3ec0000]

it's not quite clear for me that why the same memory region was added twice, that's the reason why KVM failed with EEXIST.

changpe1 commented 3 years ago

For a test comparison, I started VMs with "-device vfio-pci,sysfsdev=/sys/bus/pci/devices/0000:d8:00.0" and "-device vfio-user-pci,socket=/var/run/muser/domain/muser0/8/cntrl", the test with QEMU vfio-pci driver worked well for me, so I can narrow down this issue should related with libvfio-user or SPDK, looks like there are some actions are missed.

Is it related with PCI_RESET? I didn't add that function in SPDK yet.

tmakatos commented 3 years ago

Thanks for providing more information, the reset issue might be related. I'm looking at fixing the migration bug(s) now, can this wait for a bit?

tmakatos commented 3 years ago

@changpe1 with https://github.com/nutanix/libvfio-user/pull/464 fixed, can you try reproing with latest libvfio-user?

changpe1 commented 3 years ago

Tested and it passed, closing it for now.

tmakatos commented 3 years ago

Re-opening as I've just hit this bug. This bug triggers on the destination host when trying to migrate a guest:

qemu-system-x86_64: kvm_set_user_memory_region: KVM_SET_USER_MEMORY_REGION failed, slot=10, start=0xfebd1000, size=0x1000: File exists

VFIO trace:

vfio_listener_region_add_ram region_add [ram] 0xfebd1000 - 0xfebd1fff [0x7fa916d1c000]
vfio_listener_region_add_skip SKIPPING region_add 0xfebd0000 - 0xfebd0fff
vfio_listener_region_add_ram region_add [ram] 0xfebd1000 - 0xfebd1fff [0x7fa916d1c000]

This isn't 100% reproducible. Reloading kvm_intel.ko seems to fix it for while. Removing -cpu host -enable-kvm also fixes it. I think that this is something that the mp-qemu folks should look at: https://github.com/oracle/qemu/issues/9