UCX ERROR ibv_modify_qp(DCI, RTR) failed : Invalid argument

gregfi commented 2 years ago

Describe the bug

Running ring_c yields output like the following on our "next gen" systems:

[1665103135.306110] [hpap13n2:7391 :0]         dc_mlx5.c:456  UCX  ERROR ibv_modify_qp(DCI, RTR) failed : Invalid argument
[hpap13n2:07330] ../../../../../openmpi-4.1.4/ompi/mca/pml/ucx/pml_ucx.c:309  Error: Failed to create UCP worker

The same installation works fine on mlx4 hardware. I tried running with UCX_TLS=sm,self,rc,ud,tcp (disabling dc), but then I get timeout errors.

Steps to Reproduce

mpirun -np 128 --mca pml ucx ./ring_c

UCX version 1.13.0

Setup and versions

OS version (SLES12SP5) + CPU architecture (x86_64)

uname -a output: Linux hpap13n1 4.12.14-122.106-default #1 SMP Mon Jan 3 22:02:42 UTC 2022 (cca30be) x86_64 x86_64 x86_64 GNU/Linux

Driver version:

rdma-core from MLNX_OFED_SRC-4.9-5.1.0.0

Kernel modules from the stock SuSE kernel:

(base) fischega@hpap13n1:~> /usr/sbin/modinfo mlx5_core
filename:       /lib/modules/4.12.14-122.106-default/kernel/drivers/net/ethernet/mellanox/mlx5/core/mlx5_core.ko
version:        5.0-0
license:        Dual BSD/GPL
description:    Mellanox 5th generation network adapters (ConnectX series) core driver
author:         Eli Cohen <eli@mellanox.com>
suserelease:    SLE12-SP5

HW information from ibstat:

CA 'mlx5_0'
CA type: MT4123
Number of ports: 1
Firmware version: 20.27.6202
Hardware version: 0
Node GUID: 0x88e9a4ffff1cfe74
System image GUID: 0x88e9a4ffff1cfe74
Port 1:
State: Active
Physical state: LinkUp
Rate: 100
Base lid: 0
LMC: 0
SM lid: 0
Capability mask: 0x00010000
Port GUID: 0x8ae9a4fffe1cfe74
Link layer: Ethernet
CA 'mlx5_1'
CA type: MT4123
Number of ports: 1
Firmware version: 20.27.6202
Hardware version: 0
Node GUID: 0x88e9a4ffff1cfe75
System image GUID: 0x88e9a4ffff1cfe74
Port 1:
State: Down
Physical state: Disabled
Rate: 40
Base lid: 0
LMC: 0
SM lid: 0
Capability mask: 0x00010000
Port GUID: 0x8ae9a4fffe1cfe75
Link layer: Ethernet
CA 'mlx5_2'
CA type: MT4123
Number of ports: 1
Firmware version: 20.27.6202
Hardware version: 0
Node GUID: 0x88e9a4ffff1cfe34
System image GUID: 0x88e9a4ffff1cfe34
Port 1:
State: Active
Physical state: LinkUp
Rate: 100
Base lid: 0
LMC: 0
SM lid: 0
Capability mask: 0x00010000
Port GUID: 0x8ae9a4fffe1cfe74
Link layer: Ethernet
CA 'mlx5_3'
CA type: MT4123
Number of ports: 1
Firmware version: 20.27.6202
Hardware version: 0
Node GUID: 0x88e9a4ffff1cfe35
System image GUID: 0x88e9a4ffff1cfe34
Port 1:
State: Down
Physical state: Disabled
Rate: 40
Base lid: 0
LMC: 0
SM lid: 0
Capability mask: 0x00010000
Port GUID: 0x8ae9a4fffe1cfe35
Link layer: Ethernet

Additional information (depending on the issue)

OpenMPI version: 4.1.4

Output of ucx_info -d:

#
# Memory domain: self
#     Component: self
#             register: unlimited, cost: 0 nsec
#           remote key: 0 bytes
#
#      Transport: self
#         Device: memory0
#           Type: loopback
#  System device: <unknown>
#
#      capabilities:
#            bandwidth: 0.00/ppn + 6911.00 MB/sec
#              latency: 0 nsec
#             overhead: 10 nsec
#            put_short: <= 4294967295
#            put_bcopy: unlimited
#            get_bcopy: unlimited
#             am_short: <= 8K
#             am_bcopy: <= 8K
#               domain: cpu
#           atomic_add: 32, 64 bit
#           atomic_and: 32, 64 bit
#            atomic_or: 32, 64 bit
#           atomic_xor: 32, 64 bit
#          atomic_fadd: 32, 64 bit
#          atomic_fand: 32, 64 bit
#           atomic_for: 32, 64 bit
#          atomic_fxor: 32, 64 bit
#          atomic_swap: 32, 64 bit
#         atomic_cswap: 32, 64 bit
#           connection: to iface
#      device priority: 0
#     device num paths: 1
#              max eps: inf
#       device address: 0 bytes
#        iface address: 8 bytes
#       error handling: ep_check
#
#
# Memory domain: tcp
#     Component: tcp
#             register: unlimited, cost: 0 nsec
#           remote key: 0 bytes
#
#      Transport: tcp
#         Device: lo
#           Type: network
#  System device: <unknown>
#
#      capabilities:
#            bandwidth: 11.91/ppn + 0.00 MB/sec
#              latency: 10960 nsec
#             overhead: 50000 nsec
#            put_zcopy: <= 18446744073709551590, up to 6 iov
#  put_opt_zcopy_align: <= 1
#        put_align_mtu: <= 0
#             am_short: <= 8K
#             am_bcopy: <= 8K
#             am_zcopy: <= 64K, up to 6 iov
#   am_opt_zcopy_align: <= 1
#         am_align_mtu: <= 0
#            am header: <= 8037
#           connection: to ep, to iface
#      device priority: 1
#     device num paths: 1
#              max eps: 256
#       device address: 18 bytes
#        iface address: 2 bytes
#           ep address: 10 bytes
#       error handling: peer failure, ep_check, keepalive
#
#      Transport: tcp
#         Device: team0
#           Type: network
#  System device: <unknown>
#
#      capabilities:
#            bandwidth: 11.32/ppn + 0.00 MB/sec
#              latency: 10960 nsec
#             overhead: 50000 nsec
#            put_zcopy: <= 18446744073709551590, up to 6 iov
#  put_opt_zcopy_align: <= 1
#        put_align_mtu: <= 0
#             am_short: <= 8K
#             am_bcopy: <= 8K
#             am_zcopy: <= 64K, up to 6 iov
#   am_opt_zcopy_align: <= 1
#         am_align_mtu: <= 0
#            am header: <= 8037
#           connection: to ep, to iface
#      device priority: 0
#     device num paths: 1
#              max eps: 256
#       device address: 6 bytes
#        iface address: 2 bytes
#           ep address: 10 bytes
#       error handling: peer failure, ep_check, keepalive
#
#
# Connection manager: tcp
#      max_conn_priv: 2064 bytes
#
# Memory domain: sysv
#     Component: sysv
#             allocate: unlimited
#           remote key: 12 bytes
#           rkey_ptr is supported
#
#      Transport: sysv
#         Device: memory
#           Type: intra-node
#  System device: <unknown>
#
#      capabilities:
#            bandwidth: 0.00/ppn + 12179.00 MB/sec
#              latency: 80 nsec
#             overhead: 10 nsec
#            put_short: <= 4294967295
#            put_bcopy: unlimited
#            get_bcopy: unlimited
#             am_short: <= 100
#             am_bcopy: <= 8256
#               domain: cpu
#           atomic_add: 32, 64 bit
#           atomic_and: 32, 64 bit
#            atomic_or: 32, 64 bit
#           atomic_xor: 32, 64 bit
#          atomic_fadd: 32, 64 bit
#          atomic_fand: 32, 64 bit
#           atomic_for: 32, 64 bit
#          atomic_fxor: 32, 64 bit
#          atomic_swap: 32, 64 bit
#         atomic_cswap: 32, 64 bit
#           connection: to iface
#      device priority: 0
#     device num paths: 1
#              max eps: inf
#       device address: 8 bytes
#        iface address: 8 bytes
#       error handling: ep_check
#
#
# Memory domain: posix
#     Component: posix
#             allocate: <= 528141124K
#           remote key: 24 bytes
#           rkey_ptr is supported
#
#      Transport: posix
#         Device: memory
#           Type: intra-node
#  System device: <unknown>
#
#      capabilities:
#            bandwidth: 0.00/ppn + 12179.00 MB/sec
#              latency: 80 nsec
#             overhead: 10 nsec
#            put_short: <= 4294967295
#            put_bcopy: unlimited
#            get_bcopy: unlimited
#             am_short: <= 100
#             am_bcopy: <= 8256
#               domain: cpu
#           atomic_add: 32, 64 bit
#           atomic_and: 32, 64 bit
#            atomic_or: 32, 64 bit
#           atomic_xor: 32, 64 bit
#          atomic_fadd: 32, 64 bit
#          atomic_fand: 32, 64 bit
#           atomic_for: 32, 64 bit
#          atomic_fxor: 32, 64 bit
#          atomic_swap: 32, 64 bit
#         atomic_cswap: 32, 64 bit
#           connection: to iface
#      device priority: 0
#     device num paths: 1
#              max eps: inf
#       device address: 8 bytes
#        iface address: 8 bytes
#       error handling: ep_check
#
#
# Memory domain: mlx5_0
#     Component: ib
#             register: unlimited, cost: 180 nsec
#           remote key: 8 bytes
#           local memory handle is required for zcopy
#
#      Transport: dc_mlx5
#         Device: mlx5_0:1
#           Type: network
#  System device: mlx5_0 (0)
[1665104310.103285] [hpap13n1:39047:0]         dc_mlx5.c:456  UCX  ERROR ibv_modify_qp(DCI, RTR) failed : Invalid argument
#   < failed to open interface >
#
#      Transport: rc_verbs
#         Device: mlx5_0:1
#           Type: network
#  System device: mlx5_0 (0)
#
#      capabilities:
#            bandwidth: 10957.84/ppn + 0.00 MB/sec
#              latency: 1000 + 1.000 * N nsec
#             overhead: 75 nsec
#            put_short: <= 124
#            put_bcopy: <= 8256
#            put_zcopy: <= 1G, up to 5 iov
#  put_opt_zcopy_align: <= 512
#        put_align_mtu: <= 1K
#            get_bcopy: <= 8256
#            get_zcopy: 65..1G, up to 5 iov
#  get_opt_zcopy_align: <= 512
#        get_align_mtu: <= 1K
#             am_short: <= 123
#             am_bcopy: <= 8255
#             am_zcopy: <= 8255, up to 4 iov
#   am_opt_zcopy_align: <= 512
#         am_align_mtu: <= 1K
#            am header: <= 127
#               domain: device
#           atomic_add: 64 bit
#          atomic_fadd: 64 bit
#         atomic_cswap: 64 bit
#           connection: to ep
#      device priority: 50
#     device num paths: 1
#              max eps: 256
#       device address: 18 bytes
#           ep address: 4 bytes
#       error handling: peer failure, ep_check
#
#
#      Transport: rc_mlx5
#         Device: mlx5_0:1
#           Type: network
#  System device: mlx5_0 (0)
#
#      capabilities:
#            bandwidth: 10957.84/ppn + 0.00 MB/sec
#              latency: 1000 + 1.000 * N nsec
#             overhead: 40 nsec
#            put_short: <= 2K
#            put_bcopy: <= 8256
#            put_zcopy: <= 1G, up to 14 iov
#  put_opt_zcopy_align: <= 512
#        put_align_mtu: <= 1K
#            get_bcopy: <= 8256
#            get_zcopy: 65..1G, up to 14 iov
#  get_opt_zcopy_align: <= 512
#        get_align_mtu: <= 1K
#             am_short: <= 2046
#             am_bcopy: <= 8254
#             am_zcopy: <= 8254, up to 3 iov
#   am_opt_zcopy_align: <= 512
#         am_align_mtu: <= 1K
#            am header: <= 186
#               domain: device
#           atomic_add: 64 bit
#          atomic_fadd: 64 bit
#         atomic_cswap: 64 bit
#           connection: to ep
#      device priority: 50
#     device num paths: 1
#              max eps: 256
#       device address: 18 bytes
#           ep address: 7 bytes
#       error handling: buffer (zcopy), remote access, peer failure, ep_check
#
#
#      Transport: ud_verbs
#         Device: mlx5_0:1
#           Type: network
#  System device: mlx5_0 (0)
#
#      capabilities:
#            bandwidth: 10957.84/ppn + 0.00 MB/sec
#              latency: 1030 nsec
#             overhead: 105 nsec
#             am_short: <= 116
#             am_bcopy: <= 1016
#             am_zcopy: <= 1016, up to 5 iov
#   am_opt_zcopy_align: <= 512
#         am_align_mtu: <= 1K
#            am header: <= 880
#           connection: to ep, to iface
#      device priority: 50
#     device num paths: 1
#              max eps: inf
#       device address: 18 bytes
#        iface address: 3 bytes
#           ep address: 6 bytes
#       error handling: peer failure, ep_check
#
#
#      Transport: ud_mlx5
#         Device: mlx5_0:1
#           Type: network
#  System device: mlx5_0 (0)
#
#      capabilities:
#            bandwidth: 10957.84/ppn + 0.00 MB/sec
#              latency: 1030 nsec
#             overhead: 80 nsec
#             am_short: <= 180
#             am_bcopy: <= 1016
#             am_zcopy: <= 1016, up to 3 iov
#   am_opt_zcopy_align: <= 512
#         am_align_mtu: <= 1K
#            am header: <= 132
#           connection: to ep, to iface
#      device priority: 50
#     device num paths: 1
#              max eps: inf
#       device address: 18 bytes
#        iface address: 3 bytes
#           ep address: 6 bytes
#       error handling: peer failure, ep_check
#
#
# Memory domain: mlx5_1
#     Component: ib
#             register: unlimited, cost: 180 nsec
#           remote key: 8 bytes
#           local memory handle is required for zcopy
#   < no supported devices found >
#
# Memory domain: mlx5_2
#     Component: ib
#             register: unlimited, cost: 180 nsec
#           remote key: 8 bytes
#           local memory handle is required for zcopy
#
#      Transport: dc_mlx5
#         Device: mlx5_2:1
#           Type: network
#  System device: mlx5_2 (2)
[1665104310.200076] [hpap13n1:39047:0]         dc_mlx5.c:456  UCX  ERROR ibv_modify_qp(DCI, RTR) failed : Invalid argument
#   < failed to open interface >
#
#      Transport: rc_verbs
#         Device: mlx5_2:1
#           Type: network
#  System device: mlx5_2 (2)
#
#      capabilities:
#            bandwidth: 10957.84/ppn + 0.00 MB/sec
#              latency: 1000 + 1.000 * N nsec
#             overhead: 75 nsec
#            put_short: <= 124
#            put_bcopy: <= 8256
#            put_zcopy: <= 1G, up to 5 iov
#  put_opt_zcopy_align: <= 512
#        put_align_mtu: <= 1K
#            get_bcopy: <= 8256
#            get_zcopy: 65..1G, up to 5 iov
#  get_opt_zcopy_align: <= 512
#        get_align_mtu: <= 1K
#             am_short: <= 123
#             am_bcopy: <= 8255
#             am_zcopy: <= 8255, up to 4 iov
#   am_opt_zcopy_align: <= 512
#         am_align_mtu: <= 1K
#            am header: <= 127
#               domain: device
#           atomic_add: 64 bit
#          atomic_fadd: 64 bit
#         atomic_cswap: 64 bit
#           connection: to ep
#      device priority: 50
#     device num paths: 1
#              max eps: 256
#       device address: 18 bytes
#           ep address: 4 bytes
#       error handling: peer failure, ep_check
#
#
#      Transport: rc_mlx5
#         Device: mlx5_2:1
#           Type: network
#  System device: mlx5_2 (2)
#
#      capabilities:
#            bandwidth: 10957.84/ppn + 0.00 MB/sec
#              latency: 1000 + 1.000 * N nsec
#             overhead: 40 nsec
#            put_short: <= 2K
#            put_bcopy: <= 8256
#            put_zcopy: <= 1G, up to 14 iov
#  put_opt_zcopy_align: <= 512
#        put_align_mtu: <= 1K
#            get_bcopy: <= 8256
#            get_zcopy: 65..1G, up to 14 iov
#  get_opt_zcopy_align: <= 512
#        get_align_mtu: <= 1K
#             am_short: <= 2046
#             am_bcopy: <= 8254
#             am_zcopy: <= 8254, up to 3 iov
#   am_opt_zcopy_align: <= 512
#         am_align_mtu: <= 1K
#            am header: <= 186
#               domain: device
#           atomic_add: 64 bit
#          atomic_fadd: 64 bit
#         atomic_cswap: 64 bit
#           connection: to ep
#      device priority: 50
#     device num paths: 1
#              max eps: 256
#       device address: 18 bytes
#           ep address: 7 bytes
#       error handling: buffer (zcopy), remote access, peer failure, ep_check
#
#
#      Transport: ud_verbs
#         Device: mlx5_2:1
#           Type: network
#  System device: mlx5_2 (2)
#
#      capabilities:
#            bandwidth: 10957.84/ppn + 0.00 MB/sec
#              latency: 1030 nsec
#             overhead: 105 nsec
#             am_short: <= 116
#             am_bcopy: <= 1016
#             am_zcopy: <= 1016, up to 5 iov
#   am_opt_zcopy_align: <= 512
#         am_align_mtu: <= 1K
#            am header: <= 880
#           connection: to ep, to iface
#      device priority: 50
#     device num paths: 1
#              max eps: inf
#       device address: 18 bytes
#        iface address: 3 bytes
#           ep address: 6 bytes
#       error handling: peer failure, ep_check
#
#
#      Transport: ud_mlx5
#         Device: mlx5_2:1
#           Type: network
#  System device: mlx5_2 (2)
#
#      capabilities:
#            bandwidth: 10957.84/ppn + 0.00 MB/sec
#              latency: 1030 nsec
#             overhead: 80 nsec
#             am_short: <= 180
#             am_bcopy: <= 1016
#             am_zcopy: <= 1016, up to 3 iov
#   am_opt_zcopy_align: <= 512
#         am_align_mtu: <= 1K
#            am header: <= 132
#           connection: to ep, to iface
#      device priority: 50
#     device num paths: 1
#              max eps: inf
#       device address: 18 bytes
#        iface address: 3 bytes
#           ep address: 6 bytes
#       error handling: peer failure, ep_check
#
#
# Memory domain: mlx5_3
#     Component: ib
#             register: unlimited, cost: 180 nsec
#           remote key: 8 bytes
#           local memory handle is required for zcopy
#   < no supported devices found >
#
# Connection manager: rdmacm
#      max_conn_priv: 54 bytes
#
# Memory domain: cma
#     Component: cma
#             register: unlimited, cost: 9 nsec
#
#      Transport: cma
#         Device: memory
#           Type: intra-node
#  System device: <unknown>
#
#      capabilities:
#            bandwidth: 0.00/ppn + 11145.00 MB/sec
#              latency: 80 nsec
#             overhead: 2000 nsec
#            put_zcopy: unlimited, up to 16 iov
#  put_opt_zcopy_align: <= 1
#        put_align_mtu: <= 1
#            get_zcopy: unlimited, up to 16 iov
#  get_opt_zcopy_align: <= 1
#        get_align_mtu: <= 1
#           connection: to iface
#      device priority: 0
#     device num paths: 1
#              max eps: inf
#       device address: 8 bytes
#        iface address: 4 bytes
#       error handling: peer failure, ep_check
#

yosefe commented 2 years ago

@gregfi can you pls build UCX with debug info and run UCX_LOG_LEVEL=data ucx_info -d -t dc_mlx5 and upload the output?

gregfi commented 2 years ago

debug.txt

yosefe commented 2 years ago

@gregfi UCX no longer supports MLNX_OFED 4.9. pls see https://github.com/openucx/ucx#supported-transports I'd advise to install latest MLNX_OFED 5.7 on the system.

gregfi commented 2 years ago

So if I have a system that still needs to support mlx4 hardware, which version of UCX would you advise using?

yosefe commented 2 years ago

it should use the UCX version that comes with MLNX_OFED 4.9 (AFAIR it's 1.8) for the system with mlx5 it's recommended to use latest MLNX_OFED 5.x

gregfi commented 2 years ago

when I do that (use UCX 1.8.0 / OpenMPI 4.0.3rc4), I get:

ib_mlx5_dv.c:223 UCX ERROR mlx5dv_devx_qp_modify(502) failed, syndrome 0: Invalid argument

UCX_LOG_LEVEL=data ucx_info -d attached.

ucx_info_1.8.0.txt

yosefe commented 2 years ago

@gregfi can you pls follow the instructions on https://github.com/openucx/ucx/wiki/How-to-get-FW-syndrome-when-using-DEVX to get the FW syndrome from dmesg?

gregfi commented 2 years ago

Here's the dmesg.log that we captured: dmesg.log

My admins tell me that two ethernet interfaces (eth4 and eth6) are "teamed" together to team0 to provide (in theory) 200G throughput:

hpap13n1:/tmp # ifconfig | tail -8
team0     Link encap:Ethernet  HWaddr 88:E9:A4:1C:FE:74  
          inet addr:10.179.33.218  Bcast:10.179.33.255  Mask:255.255.255.0
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:7028393 errors:0 dropped:34 overruns:0 frame:0
          TX packets:7939350 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:5405896968 (5155.4 Mb)  TX bytes:4180773337 (3987.0 Mb)

Does that seem connected to the failure here?

yosefe commented 2 years ago

@gregfi are you sure echo 'func mlx5_cmd_check +p' | sudo tee /sys/kernel/debug/dynamic_debug/control was done ? was it done before running ucx_info? seems the errors in dmesg are not exactly what i was looking for, but it could be related.

gregfi commented 2 years ago

The dmesg output was produced after re-executing the ring test, not ucx_info. Let me see if my admin changed it back already...

gregfi commented 2 years ago

Here's what I get with reproducer=UCX_LOG_LEVEL=data ucx_info -d:

dmesg.2.log

Artemy-Mellanox commented 2 years ago

@gregfi could you please add some more dynamic debug

echo 1 | sudo tee /sys/module/mlx5_core/parameters/debug_mask
echo 'func dump_buf +p' | sudo tee /sys/kernel/debug/dynamic_debug/control
echo 'func dump_command +p' | sudo tee /sys/kernel/debug/dynamic_debug/control
echo 'func mlx5_cmd_check +p' | sudo tee /sys/kernel/debug/dynamic_debug/control

and re-execute the reproducer

yosefe commented 2 years ago

@gregfi ping

gregfi commented 2 years ago

Attached.

ucx_info.txt

dmesg.3.log

Artemy-Mellanox commented 2 years ago

@gregfi this debug prints a lot of data on your setup so the dmesg buffer can hold only ~20sec of the history could you please try to capture the dmesg output immediately after the reproducer so the line

RST2INIT_QP(0x502) op_mod(0x0) failed, status bad resource state(0x9), syndrome (0xc7051)

still will be in the buffer. something like

dmesg -C ; UCX_LOG_LEVEL=data ucx_info -d ; dmesg -c > dmesg.log

gregfi commented 2 years ago

dmesg.4.log

yosefe commented 2 years ago

@gregfi can you try latest UCX master branch with MLNX_OFED 5.x? perhaps #8677 could fix the issue

gregfi commented 2 years ago

I compiled the latest master branch. I haven't re-compiled OpenMPI and tried the original reproducer, but ucx_info -d still produces:

[1667590544.077395] [hpap13n1:36263:0] dc_mlx5.c:509 UCX ERROR ibv_modify_qp(DCI, RTR) failed : Invalid argument

gleon99 commented 2 years ago

@gregfi can you try to run some IB test, e.g ib_send_lat with DC? Check whether it works and provide the output.

yosefe commented 1 year ago

@gregfi ping

openucx / ucx