nvmf_subsystem_add_listener return error when CONFIG_ARM64_64K_PAGES=y

[peng6662001]

I run the commands in the dir of spdk on Ampere Altra.

sudo scripts/setup.sh
rm ~/images/test-disk.raw
truncate ~/images/test-disk.raw -s 128M
mkfs.ext4  ~/images/test-disk.raw
killall reactor_0
sleep 2
./build/bin/nvmf_tgt -i 0 -e 0xFFFF -m 0x1 &
sleep 2
sudo ./scripts/rpc.py nvmf_create_transport -t VFIOUSER
sudo rm -rf /tmp/nvme-vfio-user
sudo mkdir -p /tmp/nvme-vfio-user
sudo ./scripts/rpc.py bdev_aio_create ~/images/test-disk.raw test 512
sudo ./scripts/rpc.py nvmf_create_subsystem nqn.2019-07.io.spdk:cnode -a -s test
sudo ./scripts/rpc.py nvmf_subsystem_add_ns nqn.2019-07.io.spdk:cnode test
sudo ./scripts/rpc.py nvmf_subsystem_add_listener nqn.2019-07.io.spdk:cnode -t VFIOUSER -a /tmp/nvme-vfio-user -s 0

The last command returns error:

• sudo ./scripts/rpc.py nvmf_subsystem_add_listener nqn.2019-07.io.spdk:cnode -t VFIOUSER -a /tmp/nvme-vfio-user -s 0 [2023-04-25 09:14:12.193544] vfio_user.c:4454:nvmf_vfio_user_listen: ERROR: /tmp/nvme-vfio-user: error to mmap file /tmp/nvme-vfio-user/bar0: Invalid argument. [2023-04-25 09:14:12.193687] nvmf.c: 684:spdk_nvmf_tgt_listen_ext: ERROR: Unable to listen on address '/tmp/nvme-vfio-user'

The relative code is:

lib/nvmf/vfio_user.c
nvmf_vfio_user_listen
    endpoint->bar0_doorbells = mmap(NULL, NVMF_VFIO_USER_DOORBELLS_SIZE,
                    PROT_READ | PROT_WRITE, MAP_SHARED, endpoint->devmem_fd, NVME_DOORBELLS_OFFSET);

It works well when CONFIG_ARM64_4K_PAGES=y. How to resolve this issue?

./build/bin/nvmf_tgt -v SPDK v23.05-pre git sha1 487069501

uname -a Linux adam-ubuntu 6.3.0-rc4 #9 SMP Fri Apr 21 07:14:57 UTC 2023 aarch64 aarch64 aarch64 GNU/Linux

lsb_release -a No LSB modules are available. Distributor ID: Ubuntu Description: Ubuntu 22.04.1 LTS Release: 22.04 Codename: jammy

Please find more details in https://github.com/spdk/spdk/issues/2994 Does libvfio-user support 64K pages?

[jlevon]

As per my comment on SPDK issue, I don't think this is vfio-user related at all.

[tmakatos]

NVMF_VFIO_USER_DOORBELLS_SIZE should be 4K in your case, that's why it breaks, and as John said this is not a libvfio-user issue.

How to resolve this issue?

You need to align NVMF_VFIO_USER_DOORBELLS_SIZE to the system page size (currently we assume 4K) and possibly adapt relevant code. I bet this isn't the only place where we assume a 4K page size in nvmf/vfio-user.

Does libvfio-user support 64K pages?

I believe it does.

[jlevon]

As this isn't a libvfio-user issue, I'm closing this.

@peng6662001 please feel free to open further issues as and when you discover issues with the library (or the protocol!). Thanks.