Invalid parameter when calling ibv_post_send

[Luo-Liang]

Hi! I'm trying soft roce and I'm having an issue:

I have a work request that looks like this:

{wr_id = 1, next = 0x0, sg_list = 0x5555558312e0, num_sge = 1, opcode = IBV_WR_RDMA_WRITE_WITH_IMM,
  send_flags = 2, {imm_data = 4194304, invalidate_rkey = 4194304}, wr = {rdma = {remote_addr = 94268250094656,
      rkey = 785}, atomic = {remote_addr = 94268250094656, compare_add = 785, swap = 0, rkey = 0}, ud = {
      ah = 0x55bc898f7c40, remote_qpn = 785, remote_qkey = 0}}, qp_type = {xrc = {remote_srqn = 0}}, {bind_mw = {
      mw = 0x0, rkey = 0, bind_info = {mr = 0x0, addr = 0, length = 0, mw_access_flags = 0}}, tso = {hdr = 0x0,
      hdr_sz = 0, mss = 0}}}

Which works fine with InfiniBand hardware, but throws -EINVALID (-22) invalid argument with Soft Roce.

I have verified SoftRoce works and ib_write_bw and other utilities work.

Is there anything that's obviously wrong/unsupported in SoftRoCE?

[Luo-Liang]

Further information: I found the error is from

681 if (write(ibqp->context->cmd_fd, &cmd, sizeof(cmd)) != sizeof(cmd))

This is the post_send_db function in rxe.c

in rxe.c, where cmd is:

(gdb) p cmd
$6 = {hdr = {command = 28, in_words = 8, out_words = 1}, {{response = 140737271903836, qp_handle = 4, wr_count = 0, sge_count = 0, wqe_size = 128, send_wr = 0x7ffff3193680}, core_payload = {
      response = 140737271903836, qp_handle = 4, wr_count = 0, sge_count = 0, wqe_size = 128, send_wr = 0x7ffff3193680}}}

[Luo-Liang]

The problem is on my side!

The GRH is set incorrectly.

Thanks!