Failed to modify UD QP to INIT on mlx5_bond_0: Invalid argument warning showing in output

Describe the bug

When running a job through OpenMPI and UCX, a warning/error of Failed to modify UD QP to INIT on mlx5_bond_0: Invalid argument shows up in the output.

It doesn't happen everytime

[scrosby@spartan-bm035 OpenMPI]$ mpirun -np 2 ./mpi-pingpong spartan-bm035.hpc.unimelb.edu.au:rank0.mpi-pingpong: Failed to modify UD QP to INIT on mlx5_bond_0: Invalid argument spartan-bm035.hpc.unimelb.edu.au:rank1.mpi-pingpong: Failed to modify UD QP to INIT on mlx5_bond_0: Invalid argument spartan-bm035.hpc.unimelb.edu.au:rank1.mpi-pingpong: Failed to modify UD QP to INIT on mlx5_bond_0: Invalid argument spartan-bm035.hpc.unimelb.edu.au:rank1.mpi-pingpong: Failed to modify UD QP to INIT on mlx5_bond_0: Invalid argument spartan-bm035.hpc.unimelb.edu.au:rank0.mpi-pingpong: Failed to modify UD QP to INIT on mlx5_bond_0: Invalid argument spartan-bm035.hpc.unimelb.edu.au:rank0.mpi-pingpong: Failed to modify UD QP to INIT on mlx5_bond_0: Invalid argument Hello from 1 of 2 Hello from 0 of 2 Timer accuracy of ~0.188000 usecs

   8 bytes took         6 usec (   2.789 MB/sec)
  16 bytes took         3 usec (   9.762 MB/sec)
  32 bytes took         1 usec ( 118.519 MB/sec)
  64 bytes took         0 usec ( 288.288 MB/sec)
 128 bytes took        21 usec (  12.388 MB/sec)
 256 bytes took         3 usec ( 169.480 MB/sec)
 512 bytes took        14 usec (  75.611 MB/sec)
1024 bytes took         9 usec ( 236.162 MB/sec)
2048 bytes took        10 usec ( 397.709 MB/sec)
4096 bytes took        11 usec ( 770.214 MB/sec)
8192 bytes took        14 usec (1184.328 MB/sec)

16384 bytes took 26 usec (1269.881 MB/sec) 32768 bytes took 39 usec (1692.300 MB/sec) 65536 bytes took 66 usec (1996.253 MB/sec) 131072 bytes took 130 usec (2020.533 MB/sec) 262144 bytes took 336 usec (1560.780 MB/sec) 524288 bytes took 299 usec (3512.594 MB/sec) 1048576 bytes took 467 usec (4493.143 MB/sec)

Asynchronous ping-pong

   8 bytes took         1 usec (  12.251 MB/sec)
  16 bytes took         0 usec (  79.208 MB/sec)
  32 bytes took         1 usec (  99.533 MB/sec)
  64 bytes took         1 usec ( 215.852 MB/sec)
 128 bytes took         2 usec ( 147.891 MB/sec)
 256 bytes took         7 usec (  73.903 MB/sec)
 512 bytes took         1 usec ( 947.271 MB/sec)
1024 bytes took        13 usec ( 159.950 MB/sec)
2048 bytes took         2 usec (2442.457 MB/sec)
4096 bytes took         8 usec (1034.343 MB/sec)
8192 bytes took         7 usec (2272.084 MB/sec)

16384 bytes took 8 usec (4124.355 MB/sec) 32768 bytes took 9 usec (6980.827 MB/sec) 65536 bytes took 30 usec (4420.939 MB/sec) 131072 bytes took 111 usec (2353.135 MB/sec) 262144 bytes took 54 usec (9758.371 MB/sec) 524288 bytes took 82 usec (12848.779 MB/sec) 1048576 bytes took 171 usec (12228.577 MB/sec)

Bi-directional asynchronous ping-pong

   8 bytes took         1 usec (  30.476 MB/sec)
  16 bytes took         0 usec (  71.910 MB/sec)
  32 bytes took         0 usec ( 132.505 MB/sec)
  64 bytes took         1 usec ( 228.980 MB/sec)
 128 bytes took         1 usec ( 247.582 MB/sec)
 256 bytes took         1 usec ( 455.922 MB/sec)
 512 bytes took         1 usec ( 767.616 MB/sec)
1024 bytes took         1 usec (1684.211 MB/sec)
2048 bytes took         2 usec (2142.259 MB/sec)
4096 bytes took         2 usec (3457.999 MB/sec)
8192 bytes took         4 usec (4397.209 MB/sec)

16384 bytes took 7 usec (4946.860 MB/sec) 32768 bytes took 38 usec (1711.748 MB/sec) 65536 bytes took 38 usec (3450.989 MB/sec) 131072 bytes took 50 usec (5225.531 MB/sec) 262144 bytes took 155 usec (3389.304 MB/sec) 524288 bytes took 165 usec (6345.661 MB/sec) 1048576 bytes took 290 usec (7230.711 MB/sec)

If I rerun it a couple of times immediately after a failure, the command seems to run without warning

Steps to Reproduce

mpirun -np 2 ./mpi-pingpong

$ ucx_info -v

# Version 1.13.1
# Git branch '', revision 09f27c0
# Configured with: --disable-logging --disable-debug --disable-assertions --disable-params-check --prefix=/apps/easybuild-2022/easybuild/software/Compiler/GCCcore/11.3.0/UCX/1.13.1 --enable-optimizations --enable-cma --enable-mt --with-verbs --without-java --without-go --disable-doxygen-doc

UCX_DC_MLX5_OOO_RW=y
UCX_DC_MLX5_TM_ENABLE=y
UCX_IB_GID_INDEX=5
UCX_NET_DEVICES=mlx5_bond_0:1
UCX_RC_MAX_NUM_EPS=1024
UCX_RC_MLX5_OOO_RW=y
UCX_RC_MLX5_TM_ENABLE=y
UCX_RC_VERBS_OOO_RW=y
UCX_RC_VERBS_RX_MAX_BUFS=128000
UCX_WARN_UNUSED_ENV_VARS=n

Setup and versions

Red Hat Enterprise Linux release 9.2 (Plow) 5.14.0-284.18.1.el9_2.x86_64
OFED 5.8-3.0.7.0

$ ibv_devinfo -vv

hca_id: mlx5_bond_0
transport:          InfiniBand (0)
fw_ver:             22.38.1002
node_guid:          08c0:eb03:006f:c062
sys_image_guid:         08c0:eb03:006f:c062
vendor_id:          0x02c9
vendor_part_id:         4125
hw_ver:             0x0
board_id:           MT_0000000359
phys_port_cnt:          1
max_mr_size:            0xffffffffffffffff
page_size_cap:          0xfffffffffffff000
max_qp:             131072
max_qp_wr:          32768
device_cap_flags:       0xed721c36
                BAD_PKEY_CNTR
                BAD_QKEY_CNTR
                AUTO_PATH_MIG
                CHANGE_PHY_PORT
                PORT_ACTIVE_EVENT
                SYS_IMAGE_GUID
                RC_RNR_NAK_GEN
                MEM_WINDOW
                XRC
                MEM_MGT_EXTENSIONS
                MEM_WINDOW_TYPE_2B
                RAW_IP_CSUM
                MANAGED_FLOW_STEERING
                Unknown flags: 0xC8400000
max_sge:            30
max_sge_rd:         30
max_cq:             16777216
max_cqe:            4194303
max_mr:             16777216
max_pd:             8388608
max_qp_rd_atom:         16
max_ee_rd_atom:         0
max_res_rd_atom:        2097152
max_qp_init_rd_atom:        16
max_ee_init_rd_atom:        0
atomic_cap:         ATOMIC_HCA (1)
max_ee:             0
max_rdd:            0
max_mw:             16777216
max_raw_ipv6_qp:        0
max_raw_ethy_qp:        0
max_mcast_grp:          2097152
max_mcast_qp_attach:        240
max_total_mcast_qp_attach:  503316480
max_ah:             2147483647
max_fmr:            0
max_srq:            8388608
max_srq_wr:         32767
max_srq_sge:            31
max_pkeys:          128
local_ca_ack_delay:     16
general_odp_caps:
                ODP_SUPPORT
                ODP_SUPPORT_IMPLICIT
rc_odp_caps:
                SUPPORT_SEND
                SUPPORT_RECV
                SUPPORT_WRITE
                SUPPORT_READ
                SUPPORT_ATOMIC
                SUPPORT_SRQ
uc_odp_caps:
                NO SUPPORT
ud_odp_caps:
                SUPPORT_SEND
xrc_odp_caps:
                SUPPORT_SEND
                SUPPORT_WRITE
                SUPPORT_READ
                SUPPORT_ATOMIC
                SUPPORT_SRQ
completion timestamp_mask:          0x7fffffffffffffff
hca_core_clock:         1000000kHZ
raw packet caps:
                C-VLAN stripping offload
                Scatter FCS offload
                IP csum offload
                Delay drop
device_cap_flags_ex:        0x30000054ED721C36
                RAW_SCATTER_FCS
                PCI_WRITE_END_PADDING
                Unknown flags: 0x3000004000000000
tso_caps:
    max_tso:            262144
    supported_qp:
                SUPPORT_RAW_PACKET
rss_caps:
    max_rwq_indirection_tables:         524288
    max_rwq_indirection_table_size:         2048
    rx_hash_function:               0x1
    rx_hash_fields_mask:                0x800000FF
    supported_qp:
                SUPPORT_RAW_PACKET
max_wq_type_rq:         8388608
packet_pacing_caps:
    qp_rate_limit_min:  1kbps
    qp_rate_limit_max:  100000000kbps
    supported_qp:
                SUPPORT_RAW_PACKET
tag matching not supported

cq moderation caps:
    max_cq_count:   65535
    max_cq_period:  4095 us

maximum available device memory:    131072Bytes

num_comp_vectors:       63
    port:   1
        state:          PORT_ACTIVE (4)
        max_mtu:        4096 (5)
        active_mtu:     4096 (5)
        sm_lid:         0
        port_lid:       0
        port_lmc:       0x00
        link_layer:     Ethernet
        max_msg_sz:     0x40000000
        port_cap_flags:     0x04010000
        port_cap_flags2:    0x0000
        max_vl_num:     invalid value (0)
        bad_pkey_cntr:      0x0
        qkey_viol_cntr:     0x0
        sm_sl:          0
        pkey_tbl_len:       1
        gid_tbl_len:        255
        subnet_timeout:     0
        init_type_reply:    0
        active_width:       4X (2)
        active_speed:       25.0 Gbps (32)
        phys_state:     LINK_UP (5)
        GID[  0]:       fe80:0000:0000:0000:0ac0:ebff:fe6f:c062, RoCE v1
        GID[  1]:       fe80::ac0:ebff:fe6f:c062, RoCE v2
        GID[  2]:       0000:0000:0000:0000:0000:ffff:ac1a:6f22, RoCE v1
        GID[  3]:       ::ffff:172.26.111.34, RoCE v2
        GID[  4]:       0000:0000:0000:0000:0000:ffff:ac1a:5c4a, RoCE v1
        GID[  5]:       ::ffff:172.26.92.74, RoCE v2
        GID[  6]:       fe80:0000:0000:0000:0c3b:c0ff:fe1f:f5e7, RoCE v1
        GID[  7]:       fe80::c3b:c0ff:fe1f:f5e7, RoCE v2

Additional information (depending on the issue)

OpenMPI 4.1.4

$ ucx_info -d

#
# Memory domain: self
#     Component: self
#             register: unlimited, cost: 0 nsec
#           remote key: 0 bytes
#
#      Transport: self
#         Device: memory0
#           Type: loopback
#  System device: <unknown>
#
#      capabilities:
#            bandwidth: 0.00/ppn + 6911.00 MB/sec
#              latency: 0 nsec
#             overhead: 10 nsec
#            put_short: <= 4294967295
#            put_bcopy: unlimited
#            get_bcopy: unlimited
#             am_short: <= 8K
#             am_bcopy: <= 8K
#               domain: cpu
#           atomic_add: 32, 64 bit
#           atomic_and: 32, 64 bit
#            atomic_or: 32, 64 bit
#           atomic_xor: 32, 64 bit
#          atomic_fadd: 32, 64 bit
#          atomic_fand: 32, 64 bit
#           atomic_for: 32, 64 bit
#          atomic_fxor: 32, 64 bit
#          atomic_swap: 32, 64 bit
#         atomic_cswap: 32, 64 bit
#           connection: to iface
#      device priority: 0
#     device num paths: 1
#              max eps: inf
#       device address: 0 bytes
#        iface address: 8 bytes
#       error handling: ep_check
#
#
# Memory domain: tcp
#     Component: tcp
#             register: unlimited, cost: 0 nsec
#           remote key: 0 bytes
#
#      Transport: tcp
#         Device: bond0.3027
#           Type: network
#  System device: <unknown>
#
#      capabilities:
#            bandwidth: 23636.10/ppn + 0.00 MB/sec
#              latency: 5203 nsec
#             overhead: 50000 nsec
#            put_zcopy: <= 18446744073709551590, up to 6 iov
#  put_opt_zcopy_align: <= 1
#        put_align_mtu: <= 0
#             am_short: <= 8K
#             am_bcopy: <= 8K
#             am_zcopy: <= 64K, up to 6 iov
#   am_opt_zcopy_align: <= 1
#         am_align_mtu: <= 0
#            am header: <= 8037
#           connection: to ep, to iface
#      device priority: 0
#     device num paths: 1
#              max eps: 256
#       device address: 6 bytes
#        iface address: 2 bytes
#           ep address: 10 bytes
#       error handling: peer failure, ep_check, keepalive
#
#      Transport: tcp
#         Device: lo
#           Type: network
#  System device: <unknown>
#
#      capabilities:
#            bandwidth: 11.91/ppn + 0.00 MB/sec
#              latency: 10960 nsec
#             overhead: 50000 nsec
#            put_zcopy: <= 18446744073709551590, up to 6 iov
#  put_opt_zcopy_align: <= 1
#        put_align_mtu: <= 0
#             am_short: <= 8K
#             am_bcopy: <= 8K
#             am_zcopy: <= 64K, up to 6 iov
#   am_opt_zcopy_align: <= 1
#         am_align_mtu: <= 0
#            am header: <= 8037
#           connection: to ep, to iface
#      device priority: 1
#     device num paths: 1
#              max eps: 256
#       device address: 18 bytes
#        iface address: 2 bytes
#           ep address: 10 bytes
#       error handling: peer failure, ep_check, keepalive
#
#      Transport: tcp
#         Device: eno145
#           Type: network
#  System device: <unknown>
#
#      capabilities:
#            bandwidth: 113.16/ppn + 0.00 MB/sec
#              latency: 5776 nsec
#             overhead: 50000 nsec
#            put_zcopy: <= 18446744073709551590, up to 6 iov
#  put_opt_zcopy_align: <= 1
#        put_align_mtu: <= 0
#             am_short: <= 8K
#             am_bcopy: <= 8K
#             am_zcopy: <= 64K, up to 6 iov
#   am_opt_zcopy_align: <= 1
#         am_align_mtu: <= 0
#            am header: <= 8037
#           connection: to ep, to iface
#      device priority: 1
#     device num paths: 1
#              max eps: 256
#       device address: 6 bytes
#        iface address: 2 bytes
#           ep address: 10 bytes
#       error handling: peer failure, ep_check, keepalive
#
#      Transport: tcp
#         Device: bond0
#           Type: network
#  System device: <unknown>
#
#      capabilities:
#            bandwidth: 23636.10/ppn + 0.00 MB/sec
#              latency: 5203 nsec
#             overhead: 50000 nsec
#            put_zcopy: <= 18446744073709551590, up to 6 iov
#  put_opt_zcopy_align: <= 1
#        put_align_mtu: <= 0
#             am_short: <= 8K
#             am_bcopy: <= 8K
#             am_zcopy: <= 64K, up to 6 iov
#   am_opt_zcopy_align: <= 1
#         am_align_mtu: <= 0
#            am header: <= 8037
#           connection: to ep, to iface
#      device priority: 1
#     device num paths: 1
#              max eps: 256
#       device address: 6 bytes
#        iface address: 2 bytes
#           ep address: 10 bytes
#       error handling: peer failure, ep_check, keepalive
#
#
# Connection manager: tcp
#      max_conn_priv: 2064 bytes
#
# Memory domain: sysv
#     Component: sysv
#             allocate: unlimited
#           remote key: 12 bytes
#           rkey_ptr is supported
#
#      Transport: sysv
#         Device: memory
#           Type: intra-node
#  System device: <unknown>
#
#      capabilities:
#            bandwidth: 0.00/ppn + 12179.00 MB/sec
#              latency: 80 nsec
#             overhead: 10 nsec
#            put_short: <= 4294967295
#            put_bcopy: unlimited
#            get_bcopy: unlimited
#             am_short: <= 100
#             am_bcopy: <= 8256
#               domain: cpu
#           atomic_add: 32, 64 bit
#           atomic_and: 32, 64 bit
#            atomic_or: 32, 64 bit
#           atomic_xor: 32, 64 bit
#          atomic_fadd: 32, 64 bit
#          atomic_fand: 32, 64 bit
#           atomic_for: 32, 64 bit
#          atomic_fxor: 32, 64 bit
#          atomic_swap: 32, 64 bit
#         atomic_cswap: 32, 64 bit
#           connection: to iface
#      device priority: 0
#     device num paths: 1
#              max eps: inf
#       device address: 8 bytes
#        iface address: 8 bytes
#       error handling: ep_check
#
#
# Memory domain: posix
#     Component: posix
#             allocate: <= 395239128K
#           remote key: 24 bytes
#           rkey_ptr is supported
#
#      Transport: posix
#         Device: memory
#           Type: intra-node
#  System device: <unknown>
#
#      capabilities:
#            bandwidth: 0.00/ppn + 12179.00 MB/sec
#              latency: 80 nsec
#             overhead: 10 nsec
#            put_short: <= 4294967295
#            put_bcopy: unlimited
#            get_bcopy: unlimited
#             am_short: <= 100
#             am_bcopy: <= 8256
#               domain: cpu
#           atomic_add: 32, 64 bit
#           atomic_and: 32, 64 bit
#            atomic_or: 32, 64 bit
#           atomic_xor: 32, 64 bit
#          atomic_fadd: 32, 64 bit
#          atomic_fand: 32, 64 bit
#           atomic_for: 32, 64 bit
#          atomic_fxor: 32, 64 bit
#          atomic_swap: 32, 64 bit
#         atomic_cswap: 32, 64 bit
#           connection: to iface
#      device priority: 0
#     device num paths: 1
#              max eps: inf
#       device address: 8 bytes
#        iface address: 8 bytes
#       error handling: ep_check
#
#
# Memory domain: mlx5_bond_0
#     Component: ib
#             register: unlimited, cost: 180 nsec
#           remote key: 8 bytes
#           local memory handle is required for zcopy
#           memory invalidation is supported
#
#      Transport: dc_mlx5
#         Device: mlx5_bond_0:1
#           Type: network
#  System device: mlx5_bond_0 (0)
#
#      capabilities:
#            bandwidth: 11664.63/ppn + 0.00 MB/sec
#              latency: 860 nsec
#             overhead: 40 nsec
#            put_short: <= 2K
#            put_bcopy: <= 8256
#            put_zcopy: <= 1G, up to 11 iov
#  put_opt_zcopy_align: <= 512
#        put_align_mtu: <= 4K
#            get_bcopy: <= 8256
#            get_zcopy: 65..1G, up to 11 iov
#  get_opt_zcopy_align: <= 512
#        get_align_mtu: <= 4K
#             am_short: <= 2046
#             am_bcopy: <= 8254
#             am_zcopy: <= 8254, up to 3 iov
#   am_opt_zcopy_align: <= 512
#         am_align_mtu: <= 4K
#            am header: <= 138
#               domain: device
#           atomic_add: 32, 64 bit
#           atomic_and: 32, 64 bit
#            atomic_or: 32, 64 bit
#           atomic_xor: 32, 64 bit
#          atomic_fadd: 32, 64 bit
#          atomic_fand: 32, 64 bit
#           atomic_for: 32, 64 bit
#          atomic_fxor: 32, 64 bit
#          atomic_swap: 32, 64 bit
#         atomic_cswap: 32, 64 bit
#           connection: to iface
#      device priority: 60
#     device num paths: 2
#              max eps: 1K
#       device address: 17 bytes
#        iface address: 5 bytes
#       error handling: buffer (zcopy), remote access, peer failure, ep_check
#
#
#      Transport: rc_verbs
#         Device: mlx5_bond_0:1
#           Type: network
#  System device: mlx5_bond_0 (0)
#
#      capabilities:
#            bandwidth: 11664.63/ppn + 0.00 MB/sec
#              latency: 800 + 1.000 * N nsec
#             overhead: 75 nsec
#            put_short: <= 124
#            put_bcopy: <= 8256
#            put_zcopy: <= 1G, up to 5 iov
#  put_opt_zcopy_align: <= 512
#        put_align_mtu: <= 4K
#            get_bcopy: <= 8256
#            get_zcopy: 65..1G, up to 5 iov
#  get_opt_zcopy_align: <= 512
#        get_align_mtu: <= 4K
#             am_short: <= 123
#             am_bcopy: <= 8255
#             am_zcopy: <= 8255, up to 4 iov
#   am_opt_zcopy_align: <= 512
#         am_align_mtu: <= 4K
#            am header: <= 127
#               domain: device
#           atomic_add: 64 bit
#          atomic_fadd: 64 bit
#         atomic_cswap: 64 bit
#           connection: to ep
#      device priority: 60
#     device num paths: 2
#              max eps: 1K
#       device address: 17 bytes
#           ep address: 5 bytes
#       error handling: peer failure, ep_check
#
#
#      Transport: rc_mlx5
#         Device: mlx5_bond_0:1
#           Type: network
#  System device: mlx5_bond_0 (0)
#
#      capabilities:
#            bandwidth: 11664.63/ppn + 0.00 MB/sec
#              latency: 800 + 1.000 * N nsec
#             overhead: 40 nsec
#            put_short: <= 2K
#            put_bcopy: <= 8256
#            put_zcopy: <= 1G, up to 14 iov
#  put_opt_zcopy_align: <= 512
#        put_align_mtu: <= 4K
#            get_bcopy: <= 8256
#            get_zcopy: 65..1G, up to 14 iov
#  get_opt_zcopy_align: <= 512
#        get_align_mtu: <= 4K
#             am_short: <= 2046
#             am_bcopy: <= 8254
#             am_zcopy: <= 8254, up to 3 iov
#   am_opt_zcopy_align: <= 512
#         am_align_mtu: <= 4K
#            am header: <= 186
#               domain: device
#           atomic_add: 32, 64 bit
#           atomic_and: 32, 64 bit
#            atomic_or: 32, 64 bit
#           atomic_xor: 32, 64 bit
#          atomic_fadd: 32, 64 bit
#          atomic_fand: 32, 64 bit
#           atomic_for: 32, 64 bit
#          atomic_fxor: 32, 64 bit
#          atomic_swap: 32, 64 bit
#         atomic_cswap: 32, 64 bit
#           connection: to ep
#      device priority: 60
#     device num paths: 2
#              max eps: 1K
#       device address: 17 bytes
#           ep address: 7 bytes
#       error handling: buffer (zcopy), remote access, peer failure, ep_check
#
#
#      Transport: ud_verbs
#         Device: mlx5_bond_0:1
#           Type: network
#  System device: mlx5_bond_0 (0)
#
#      capabilities:
#            bandwidth: 11664.63/ppn + 0.00 MB/sec
#              latency: 830 nsec
#             overhead: 105 nsec
#             am_short: <= 116
#             am_bcopy: <= 4088
#             am_zcopy: <= 4088, up to 5 iov
#   am_opt_zcopy_align: <= 512
#         am_align_mtu: <= 4K
#            am header: <= 3952
#           connection: to ep, to iface
#      device priority: 60
#     device num paths: 2
#              max eps: inf
#       device address: 17 bytes
#        iface address: 3 bytes
#           ep address: 6 bytes
#       error handling: peer failure, ep_check
#
#
#      Transport: ud_mlx5
#         Device: mlx5_bond_0:1
#           Type: network
#  System device: mlx5_bond_0 (0)
#
#      capabilities:
#            bandwidth: 11664.63/ppn + 0.00 MB/sec
#              latency: 830 nsec
#             overhead: 80 nsec
#             am_short: <= 180
#             am_bcopy: <= 4088
#             am_zcopy: <= 4088, up to 3 iov
#   am_opt_zcopy_align: <= 512
#         am_align_mtu: <= 4K
#            am header: <= 132
#           connection: to ep, to iface
#      device priority: 60
#     device num paths: 2
#              max eps: inf
#       device address: 17 bytes
#        iface address: 3 bytes
#           ep address: 6 bytes
#       error handling: peer failure, ep_check
#
#
# Connection manager: rdmacm
#      max_conn_priv: 54 bytes
#
# Memory domain: cma
#     Component: cma
#             register: unlimited, cost: 9 nsec
#
#      Transport: cma
#         Device: memory
#           Type: intra-node
#  System device: <unknown>
#
#      capabilities:
#            bandwidth: 0.00/ppn + 11145.00 MB/sec
#              latency: 80 nsec
#             overhead: 2000 nsec
#            put_zcopy: unlimited, up to 16 iov
#  put_opt_zcopy_align: <= 1
#        put_align_mtu: <= 1
#            get_zcopy: unlimited, up to 16 iov
#  get_opt_zcopy_align: <= 1
#        get_align_mtu: <= 1
#           connection: to iface
#      device priority: 0
#     device num paths: 1
#              max eps: inf
#       device address: 8 bytes
#        iface address: 4 bytes
#       error handling: peer failure, ep_check

openucx / ucx