search

SoftRoCE / librxe-dev

Development repository for RXE user space code.
Other
66 stars 33 forks source link

soft-roce can't connect successfully with hw-ROCE #6

Open janes09 opened 8 years ago

janes09 commented 8 years ago

I test that it is ok that hw-roce connect hw-roce, also it is ok that soft-roce connect soft-roce. but hw-roce connect failure. 

i trace the message: try to find the message difference,  i found that hw-roce's RC Acknowledge.s BTH header: MigReq is TRUE, but soft-roce's value is FALSE.

Because i can't check whether this is the root cause, so i raise the issue here.

monis410 commented 8 years ago

Which hardware do you use for HardRoCE? And which test?

janes09 commented 8 years ago

ConnectX-3_Pro, use rsocket feature in librdmacm.

monis410 commented 8 years ago

I ran some tests of Soft vs. Hard (ConnectX-3) successfully and without any issue Can you please try other utils?

  1. ibv_rc-pingpong
  2. ucmatose
  3. rping

ALso, please send full tcpdump (or even better, ibdump) of a test that fails

janes09 commented 8 years ago

Base Transport Header Opcode: 11 0... .... = Solicited Event: False .1.. .... = MigReq: True //here is different between soft-roce and hw-roce. Hw-roce is 1, soft-roce is 0. .... 0000 = Header Version: 0

janes09 commented 8 years ago

i use ROCE V2, not ROCE V1

monis410 commented 8 years ago

SoftRoCE supports only RoCEv2

janes09 commented 8 years ago

Does soft-ROCE ROCE V1? i install soft-ROCE, the default is ROCE V2. Now i don't know how to set soft-ROCE to support ROCE V1, can you tell me the method for setting?

janes09 commented 8 years ago

ok. but you said： “I ran some tests of Soft vs. Hard (ConnectX-3)”， the ConnectX-3 don't support ROCE V2, only ConnectX-3_pro support ROCE V2

janes09 commented 8 years ago

I use rping to test, it is fine. But i don't know why rsocket is not ok.

janes09 commented 8 years ago

soft-roce don't support qperf test, also don't have OFED package, it isn't very convenient to use.

monis410 commented 8 years ago

To all your questions

  1. I wrote ConnectX-3 but meant to say ConnectX-3 Pro. SoftRoCE supports only V1 so compatability with is only with RoCEv2 capable hardware.
  2. I will try with rsocket and let you know
  3. Not sure what you mean by soft-roce don't support qperf test. Did you run a test that failed? Which one?
  4. SoftRoCE is planned for OFED but I can't say when exactly
janes09 commented 8 years ago

qperf tool will check some configuration: in file: configure.in, it do:

"AC_CHECK_LIB(ibverbs, ibv_open_device, RDMA=1) AC_CHECK_LIB(ibverbs, ibv_open_xrc_domain, HAS_XRC=1)"

finally after install qperf, don't have RDMA test function, i analyze it, and find the RDMA is compiler macro in qperf source code, it should be that don't open the RDMA macro when install qperf. but HW-ROCE is fine to install qperf RDMA test function.

Housin9 commented 7 years ago

Please can we install soft roce over ethernet adapter card?