search

nutanix / libvfio-user

framework for emulating devices in userspace
BSD 3-Clause "New" or "Revised" License
166 stars 51 forks source link

QEMU maps overlapping DMA regions #463

Open tmakatos opened 3 years ago

tmakatos commented 3 years ago

Discovered while working on https://github.com/nutanix/libvfio-user/issues/218, see https://github.com/nutanix/libvfio-user/issues/218#issuecomment-831994511:

On the destination SPDK of the migration:

[2021-05-04 10:25:25.311034] vfio_user.c:1476:vfio_user_log: *DEBUG*: /var/tmp: adding DMA region [0xc4000, 0xe0000) offset=0 prot=0x1 flags=0
...
[2021-05-04 10:25:25.312026] vfio_user.c:1476:vfio_user_log: *DEBUG*: /var/tmp: adding DMA region [0xc0000, 0xc1000) offset=0xc0000 prot=0x1 flags=0x1
[2021-05-04 10:25:25.312064] vfio_user.c:1476:vfio_user_log: *DEBUG*: /var/tmp: mapped DMA region iova=[0xc0000, 0xc1000) vaddr=0x7f269d4c0000 page_size=0x200000 mapping=[0x7f269d400000, 0x7f269d600000)
[2021-05-04 10:25:25.312079] vfio_user.c:1166:memory_region_add_cb: *DEBUG*: /var/tmp: map IOVA 0x7f269d400000-0x7f269d600000
[2021-05-04 10:25:25.312209] vfio_user.c:1476:vfio_user_log: *DEBUG*: /var/tmp: adding DMA region [0xc4000, 0xe8000) offset=0xc4000 prot=0x1 flags=0x1
[2021-05-04 10:25:25.312234] vfio_user.c:1478:vfio_user_log: *INFO*: /var/tmp: new DMA region [0xc4000, 0xe8000) fd=501 offset=0xc4000 prot=0x1 overlaps with DMA region [0xc4000, 0xe0000)
[2021-05-04 10:25:25.312248] vfio_user.c:1484:vfio_user_log: *ERROR*: /var/tmp: failed to add DMA region [0xc4000, 0xe8000) offset=0xc4000 prot=0x1 flags=0x1: Invalid argument

On the destination QEMU:

33461@1620138325.310587:vfio_listener_region_add_ram region_add [ram] 0xc4000 - 0xdffff [0x7f6775604000]
...
33457@1620138325.311915:vfio_listener_region_add_ram region_add [ram] 0xc0000 - 0xc0fff [0x7f66ac6c0000]
33457@1620138325.312133:vfio_listener_region_add_ram region_add [ram] 0xc4000 - 0xe7fff [0x7f66ac6c4000]
2021-05-04T14:25:25.312351Z qemu-system-x86_64: vfio_dma_map(0x559d8de93800, 0xc4000, 0x24000, 0x7f66ac6c4000) = -22 (Resource temporarily unavaiqemu: hardware error: vfio: DMA mapping failed, unable to continue

This is the first time we see this.

EDIT: Seen again, in SPDK:

[2021-05-05 05:38:16.156976] vfio_user.c:1476:vfio_user_log: *DEBUG*: /var/tmp: adding DMA region [0xc0000, 0xc1000) offset=0 prot=0x1 flags=0
[2021-05-05 05:38:16.157137] vfio_user.c:1476:vfio_user_log: *DEBUG*: /var/tmp: removing DMA region [0xc4000, 0xe0000) offset=0 prot=0 flags=0
[2021-05-05 05:38:16.157285] vfio_user.c:1476:vfio_user_log: *DEBUG*: /var/tmp: adding DMA region [0xc1000, 0xc4000) offset=0xc1000 prot=0x3 flags=0x1
[2021-05-05 05:38:16.157311] vfio_user.c:1166:memory_region_add_cb: *DEBUG*: /var/tmp: map IOVA 0x7f5961200000-0x7f5961400000
[2021-05-05 05:38:16.157332] vfio_user.c:1175:memory_region_add_cb: *ERROR*: Memory region register 0x7f5961200000-0x7f5961400000 failed: Resource temporarily unavailable
[2021-05-05 05:38:16.157389] vfio_user.c:1476:vfio_user_log: *DEBUG*: /var/tmp: removing DMA region [0xe0000, 0x100000) offset=0 prot=0 flags=0
[2021-05-05 05:38:16.157557] vfio_user.c:1476:vfio_user_log: *DEBUG*: /var/tmp: adding DMA region [0xc4000, 0xe0000) offset=0 prot=0x1 flags=0
[2021-05-05 05:38:16.157646] vfio_user.c:1476:vfio_user_log: *DEBUG*: /var/tmp: adding DMA region [0xc0000, 0xc1000) offset=0xc0000 prot=0x1 flags=0x1
[2021-05-05 05:38:16.157670] vfio_user.c:1484:vfio_user_log: *ERROR*: /var/tmp: bad offset for new DMA region [0xc0000, 0xc1000) fd=501 offset=0xc0000 prot=0x1; existing=0
[2021-05-05 05:38:16.157691] vfio_user.c:1484:vfio_user_log: *ERROR*: /var/tmp: failed to add DMA region [0xc0000, 0xc1000) offset=0xc0000 prot=0x1 flags=0x1: Invalid argument
[2021-05-05 05:38:16.157715] vfio_user.c:1484:vfio_user_log: *ERROR*: /var/tmp: msg0x3c: cmd 2 failed: Invalid argument

In QEMU:

55480@1620207496.156448:vfio_listener_region_add_ram region_add [ram] 0xc0000 - 0xc0fff [0x7f6702c00000]
55477@1620207496.157087:vfio_listener_region_del region_del 0xc4000 - 0xdffff
55480@1620207496.157141:vfio_listener_region_add_ram region_add [ram] 0xc1000 - 0xc3fff [0x7f66406c1000]
55477@1620207496.157280:vfio_listener_region_del region_del 0xe0000 - 0xfffff
55480@1620207496.157392:vfio_listener_region_add_ram region_add [ram] 0xc4000 - 0xdffff [0x7f6702c04000]
55477@1620207496.157559:vfio_listener_region_add_ram region_add [ram] 0xc0000 - 0xc0fff [0x7f66406c0000]
2021-05-05T09:38:16.158864Z qemu-system-x86_64: vfio_dma_map(0x557b8fd281b0, 0xc0000, 0x1000, 0x7f66406c0000) = -22 (Resource temporarily unavailable)
tmakatos commented 3 years ago

I've sent an email to qemu-devel.

EDIT: Heard back from Alex, this shouldn't be happening, apparently there's a bug somehwere, need to take a look at QEMU. https://lists.gnu.org/archive/html/qemu-devel/2021-05/msg02114.html