search

SoftRoCE / rxe-dev

Development Repository for RXE
Other
130 stars 56 forks source link

SoftRoce cannot work with NVMe-oF target #70

Open optimistyzy opened 7 years ago

optimistyzy commented 7 years ago

Test SPDK NVMe-oF target with Software Roce. While increasing the queue depth, we always have the following issue while calling ibv_poll_cq: nvmf_tgt.c: 255:spdk_nvmf_startup: NOTICE: Acceptor running on core 0 on socket 0 rdma.c:1707:spdk_nvmf_rdma_qpair_poll: ERROR: CQ error on CQ 0x1a69230, Request 0x27695128 (12): transport retry counter exceeded

divyanks commented 6 years ago

I was setting up SPDK NVMe-OF with softRoCE on linux kernel 4.15.0. I am not able to discover the nvme target from another linux client. However, I can see that its getting discovered on nvme target ie using same initiator as the target very well. Used the following conf ======snip==== NQN nqn.2016-06.io.spdk:cnode2 Listen RDMA 192.168.106.163:4420 AllowAnyHost Yes Host nqn.2016-06.io.spdk:init SN SPDK00000000000002 Namespace Malloc0 Namespace Malloc1 Namespace AIO0 Namespace AIO1

===snip

Error am getting at the client side is root@amit-VirtualBox:/RCoE/targets/nvme-cli# ./nvme discover -t rdma -a 192.168.106.163 -s 4420 Failed to write to /dev/nvme-fabrics: Connection reset by peer

But request did not reach the nvme target as I had started it in debug mode. Any help would be appreciated.