Hi,
I'm using Soft RoCE running on regular Broadcom 1G on-board NIC, and trying to connect to NVMe over RoCEv2 target. I have two types of NVMe/RoCEv2 target and one Linux software base NVMe/RoCEv2 target works fine so my Soft-RoCE setup should be OK but other NVMe/RoCEv2 target doesn't work like below
% nvme discover -t rdma -a 192.168.1.4 -s 4420
Failed to write to /dev/nvme-fabrics: Connection timed out
NVMe/RoCEv2 target device supplier is saying, it could be because Soft-RoCE specifies 0x80 in Responder Resources in ConnectRequest packet. It can only supports up to 64 (0x40). So if I can change the parameter for Responder Resources for ConnectRequest packet in SoftRoCE, it's likely that connection to this target works fine.
I tried to set parameter from nvme-cli like below but it doesn't change anything on wireshark.
nvme discover -t rdma -a -s 4420 -Q 64
nvme discover -t rdma -a -s 4420 -i 64
And target supplier is saying it should be parameter in Soft-RoCE.
Based on the following website, there're some parameters that we can set, I think it should one of parameter like below.
http://manpages.ubuntu.com/manpages/bionic/man7/rxe.7.html
/sys/module/rdma_rxe/parameters/max_qp_wr
but somehow I was not able to set/change parameter for it.
It doesn't allow me to edit file even I change the access privilege of directory.
root@xxxxxxx:/sys/module/rdma_rxe/parameters# chmod 777 .
root@xxxxxxx:/sys/module/rdma_rxe/parameters# ls -la
total 0
drwxrwxrwx 2 root root 0 Feb 12 04:57 .
drwxr-xr-x 6 root root 0 Feb 12 04:57 ..
--w------- 1 root root 4096 Feb 6 09:43 add
--w------- 1 root root 4096 Feb 12 04:57 remove
root@ xxxxxxx:/sys/module/rdma_rxe/parameters# touch max_qp
touch: cannot touch 'max_qp': Permission denied
So I'd like to know how I can modify parameters for Soft-RoCE.
And if possible I'd like to know how I can change Responder Resources field in ConnectRequest packet using Soft-RoCE (I could be wrong to change parameter for rdma_rxe...)
I attached screenshot of wireshark packet.
You can see "Response Resources" is set to 0x80 for ConnectRequest packet and
NVMe/RoCEv2 target device supplier is saying it needs to be 0x40 (64byte) or less.
Hi, I'm using Soft RoCE running on regular Broadcom 1G on-board NIC, and trying to connect to NVMe over RoCEv2 target. I have two types of NVMe/RoCEv2 target and one Linux software base NVMe/RoCEv2 target works fine so my Soft-RoCE setup should be OK but other NVMe/RoCEv2 target doesn't work like below
% nvme discover -t rdma -a 192.168.1.4 -s 4420 Failed to write to /dev/nvme-fabrics: Connection timed out
NVMe/RoCEv2 target device supplier is saying, it could be because Soft-RoCE specifies 0x80 in Responder Resources in ConnectRequest packet. It can only supports up to 64 (0x40). So if I can change the parameter for Responder Resources for ConnectRequest packet in SoftRoCE, it's likely that connection to this target works fine. I tried to set parameter from nvme-cli like below but it doesn't change anything on wireshark.
nvme discover -t rdma -a -s 4420 -Q 64
nvme discover -t rdma -a -s 4420 -i 64
And target supplier is saying it should be parameter in Soft-RoCE. Based on the following website, there're some parameters that we can set, I think it should one of parameter like below. http://manpages.ubuntu.com/manpages/bionic/man7/rxe.7.html /sys/module/rdma_rxe/parameters/max_qp_wr but somehow I was not able to set/change parameter for it. It doesn't allow me to edit file even I change the access privilege of directory. root@xxxxxxx:/sys/module/rdma_rxe/parameters# chmod 777 . root@xxxxxxx:/sys/module/rdma_rxe/parameters# ls -la total 0 drwxrwxrwx 2 root root 0 Feb 12 04:57 . drwxr-xr-x 6 root root 0 Feb 12 04:57 .. --w------- 1 root root 4096 Feb 6 09:43 add --w------- 1 root root 4096 Feb 12 04:57 remove root@ xxxxxxx:/sys/module/rdma_rxe/parameters# touch max_qp touch: cannot touch 'max_qp': Permission denied
So I'd like to know how I can modify parameters for Soft-RoCE. And if possible I'd like to know how I can change Responder Resources field in ConnectRequest packet using Soft-RoCE (I could be wrong to change parameter for rdma_rxe...)
I attached screenshot of wireshark packet. You can see "Response Resources" is set to 0x80 for ConnectRequest packet and NVMe/RoCEv2 target device supplier is saying it needs to be 0x40 (64byte) or less.
Thanks in advance, Shiroyd