"CM private data buffer is too small to pack UCP endpoint info" - a warning that should be heeded?

[rrgargeya]

Describe the bug

I see the messages "wireup_cm.c:333 UCX DIAG CM private data buffer is too small to pack UCP endpoint info, ep 0xXXX service data version 0, size 11, address length 70, cm 0xXXXX max_conn_priv 54"

Is this a message a bug or is it a warning to which one should pay attention?

Steps to Reproduce

• Command line

• UCX version used (from github branch XX or release YY) + UCX configure flags (can be checked by ucx_info -v)

• $ ucx_info -v Version 1.13.0 Git branch '', revision 6765970 Configured with: --disable-logging --disable-debug --disable-assertions --disable-params-check --with-rdmacm

• Any UCX environment variables used

UCX_LOG_LEVEL=INFO

Setup and versions

• OS version (e.g Linux distro) + CPU architecture (x86_64/aarch64/ppc64le/...)

  • cat /etc/issue or cat /etc/redhat-release + uname -a $ cat /etc/redhat-release Red Hat Enterprise Linux Server release 7.4 (Maipo) $ uname -a Linux host1 3.10.0-693.17.1.rt56.636.el7.x86_64 #1 SMP PREEMPT RT Tue Jan 16 16:25:18 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux

  • For Nvidia Bluefield SmartNIC include cat /etc/mlnx-release (the string identifies software and firmware setup)

• For RDMA/IB/RoCE related issues:

  • Driver version:

    • rpm -q rdma-core or rpm -q libibverbs
    • or: MLNX_OFED version ofed_info -s $ ofed_info -s MLNX_OFED_LINUX-5.5-1.0.3.2:

  • HW information from ibstat or ibv_devinfo -vv command

$ ibstat CA 'mlx5_0' CA type: MT4119 Number of ports: 1 Firmware version: 16.32.1010 Hardware version: 0 Node GUID: 0xec0d9a0300431604 System image GUID: 0xec0d9a0300431604 Port 1: State: Active Physical state: LinkUp Rate: 100 Base lid: 17 LMC: 0 SM lid: 1 Capability mask: 0x2651e848 Port GUID: 0xec0d9a0300431604 Link layer: InfiniBand CA 'mlx5_1' CA type: MT4119 Number of ports: 1 Firmware version: 16.32.1010 Hardware version: 0 Node GUID: 0xec0d9a0300431605 System image GUID: 0xec0d9a0300431604 Port 1: State: Active Physical state: LinkUp Rate: 100 Base lid: 104 LMC: 0 SM lid: 1 Capability mask: 0x2651e848 Port GUID: 0xec0d9a0300431605 Link layer: InfiniBand CA 'mlx5_2' CA type: MT4119 Number of ports: 1 Firmware version: 16.32.1010 Hardware version: 0 Node GUID: 0xec0d9a0300431624 System image GUID: 0xec0d9a0300431624 Port 1: State: Active Physical state: LinkUp Rate: 100 Base lid: 0 LMC: 0 SM lid: 0 Capability mask: 0x00010000 Port GUID: 0xee0d9afffe431624 Link layer: Ethernet CA 'mlx5_3' CA type: MT4119 Number of ports: 1 Firmware version: 16.32.1010 Hardware version: 0 Node GUID: 0xec0d9a0300431625 System image GUID: 0xec0d9a0300431624 Port 1: State: Active Physical state: LinkUp Rate: 100 Base lid: 0 LMC: 0 SM lid: 0 Capability mask: 0x00010000 Port GUID: 0xee0d9afffe431625 Link layer: Ethernet

• For GPU related issues:
  • GPU type
  • Cuda:
    • Drivers version
    • Check if peer-direct is loaded: lsmod|grep nv_peer_mem and/or gdrcopy: lsmod|grep gdrdrv

Additional information (depending on the issue)

• OpenMPI version

• Output of ucx_info -d to show transports and devices recognized by UCX $ ucx_info -d

  Memory domain: self Component: self register: unlimited, cost: 0 nsec remote key: 0 bytes

  Transport: self Device: memory0 Type: loopback System device:

  capabilities: bandwidth: 0.00/ppn + 6911.00 MB/sec latency: 0 nsec overhead: 10 nsec put_short: <= 4294967295 put_bcopy: unlimited get_bcopy: unlimited am_short: <= 8K am_bcopy: <= 8K domain: cpu atomic_add: 32, 64 bit atomic_and: 32, 64 bit atomic_or: 32, 64 bit atomic_xor: 32, 64 bit atomic_fadd: 32, 64 bit atomic_fand: 32, 64 bit atomic_for: 32, 64 bit atomic_fxor: 32, 64 bit atomic_swap: 32, 64 bit atomic_cswap: 32, 64 bit connection: to iface device priority: 0 device num paths: 1 max eps: inf device address: 0 bytes iface address: 8 bytes error handling: ep_check

  Memory domain: tcp Component: tcp register: unlimited, cost: 0 nsec remote key: 0 bytes

  Transport: tcp Device: lo Type: network System device:

  capabilities: bandwidth: 11.91/ppn + 0.00 MB/sec latency: 10960 nsec overhead: 50000 nsec put_zcopy: <= 18446744073709551590, up to 6 iov put_opt_zcopy_align: <= 1 put_align_mtu: <= 0 am_short: <= 8K am_bcopy: <= 8K am_zcopy: <= 64K, up to 6 iov am_opt_zcopy_align: <= 1 am_align_mtu: <= 0 am header: <= 8037 connection: to ep, to iface device priority: 1 device num paths: 1 max eps: 256 device address: 18 bytes iface address: 2 bytes ep address: 10 bytes error handling: peer failure, ep_check, keepalive

  Transport: tcp Device: ib0 Type: network System device:

  capabilities: bandwidth: 11142.51/ppn + 0.00 MB/sec latency: 5206 nsec overhead: 50000 nsec put_zcopy: <= 18446744073709551590, up to 6 iov put_opt_zcopy_align: <= 1 put_align_mtu: <= 0 am_short: <= 8K am_bcopy: <= 8K am_zcopy: <= 64K, up to 6 iov am_opt_zcopy_align: <= 1 am_align_mtu: <= 0 am header: <= 8037 connection: to ep, to iface device priority: 1 device num paths: 1 max eps: 256 device address: 6 bytes iface address: 2 bytes ep address: 10 bytes error handling: peer failure, ep_check, keepalive

  Transport: tcp Device: ib1 Type: network System device:

  capabilities: bandwidth: 11142.51/ppn + 0.00 MB/sec latency: 5206 nsec overhead: 50000 nsec put_zcopy: <= 18446744073709551590, up to 6 iov put_opt_zcopy_align: <= 1 put_align_mtu: <= 0 am_short: <= 8K am_bcopy: <= 8K am_zcopy: <= 64K, up to 6 iov am_opt_zcopy_align: <= 1 am_align_mtu: <= 0 am header: <= 8037 connection: to ep, to iface device priority: 1 device num paths: 1 max eps: 256 device address: 6 bytes iface address: 2 bytes ep address: 10 bytes error handling: peer failure, ep_check, keepalive

  Transport: tcp Device: net4 Type: network System device:

  capabilities: bandwidth: 11818.05/ppn + 0.00 MB/sec latency: 5206 nsec overhead: 50000 nsec put_zcopy: <= 18446744073709551590, up to 6 iov put_opt_zcopy_align: <= 1 put_align_mtu: <= 0 am_short: <= 8K am_bcopy: <= 8K am_zcopy: <= 64K, up to 6 iov am_opt_zcopy_align: <= 1 am_align_mtu: <= 0 am header: <= 8037 connection: to ep, to iface device priority: 1 device num paths: 1 max eps: 256 device address: 6 bytes iface address: 2 bytes ep address: 10 bytes error handling: peer failure, ep_check, keepalive

  Transport: tcp Device: net5 Type: network System device:

  capabilities: bandwidth: 11818.05/ppn + 0.00 MB/sec latency: 5206 nsec overhead: 50000 nsec put_zcopy: <= 18446744073709551590, up to 6 iov put_opt_zcopy_align: <= 1 put_align_mtu: <= 0 am_short: <= 8K am_bcopy: <= 8K am_zcopy: <= 64K, up to 6 iov am_opt_zcopy_align: <= 1 am_align_mtu: <= 0 am header: <= 8037 connection: to ep, to iface device priority: 1 device num paths: 1 max eps: 256 device address: 6 bytes iface address: 2 bytes ep address: 10 bytes error handling: peer failure, ep_check, keepalive

  Transport: tcp Device: net0 Type: network System device:

  capabilities: bandwidth: 113.16/ppn + 0.00 MB/sec latency: 5776 nsec overhead: 50000 nsec put_zcopy: <= 18446744073709551590, up to 6 iov put_opt_zcopy_align: <= 1 put_align_mtu: <= 0 am_short: <= 8K am_bcopy: <= 8K am_zcopy: <= 64K, up to 6 iov am_opt_zcopy_align: <= 1 am_align_mtu: <= 0 am header: <= 8037 connection: to ep, to iface device priority: 1 device num paths: 1 max eps: 256 device address: 6 bytes iface address: 2 bytes ep address: 10 bytes error handling: peer failure, ep_check, keepalive

  Connection manager: tcp max_conn_priv: 2064 bytes

  Memory domain: sysv Component: sysv allocate: unlimited remote key: 12 bytes rkey_ptr is supported

  Transport: sysv Device: memory Type: intra-node System device:

  capabilities: bandwidth: 0.00/ppn + 12179.00 MB/sec latency: 80 nsec overhead: 10 nsec put_short: <= 4294967295 put_bcopy: unlimited get_bcopy: unlimited am_short: <= 100 am_bcopy: <= 8256 domain: cpu atomic_add: 32, 64 bit atomic_and: 32, 64 bit atomic_or: 32, 64 bit atomic_xor: 32, 64 bit atomic_fadd: 32, 64 bit atomic_fand: 32, 64 bit atomic_for: 32, 64 bit atomic_fxor: 32, 64 bit atomic_swap: 32, 64 bit atomic_cswap: 32, 64 bit connection: to iface device priority: 0 device num paths: 1 max eps: inf device address: 8 bytes iface address: 8 bytes error handling: ep_check

  Memory domain: posix Component: posix allocate: <= 97650420K remote key: 24 bytes rkey_ptr is supported

  Transport: posix Device: memory Type: intra-node System device:

  capabilities: bandwidth: 0.00/ppn + 12179.00 MB/sec latency: 80 nsec overhead: 10 nsec put_short: <= 4294967295 put_bcopy: unlimited get_bcopy: unlimited am_short: <= 100 am_bcopy: <= 8256 domain: cpu atomic_add: 32, 64 bit atomic_and: 32, 64 bit atomic_or: 32, 64 bit atomic_xor: 32, 64 bit atomic_fadd: 32, 64 bit atomic_fand: 32, 64 bit atomic_for: 32, 64 bit atomic_fxor: 32, 64 bit atomic_swap: 32, 64 bit atomic_cswap: 32, 64 bit connection: to iface device priority: 0 device num paths: 1 max eps: inf device address: 8 bytes iface address: 8 bytes error handling: ep_check

  Memory domain: mlx5_0 Component: ib register: unlimited, cost: 180 nsec remote key: 8 bytes local memory handle is required for zcopy

  Transport: dc_mlx5 Device: mlx5_0:1 Type: network System device: mlx5_0 (0)

  capabilities: bandwidth: 11794.23/ppn + 0.00 MB/sec latency: 660 nsec overhead: 40 nsec put_short: <= 2K put_bcopy: <= 8256 put_zcopy: <= 1G, up to 11 iov put_opt_zcopy_align: <= 512 put_align_mtu: <= 4K get_bcopy: <= 8256 get_zcopy: 65..1G, up to 11 iov get_opt_zcopy_align: <= 512 get_align_mtu: <= 4K am_short: <= 2046 am_bcopy: <= 8254 am_zcopy: <= 8254, up to 3 iov am_opt_zcopy_align: <= 512 am_align_mtu: <= 4K am header: <= 138 domain: device atomic_add: 32, 64 bit atomic_and: 32, 64 bit atomic_or: 32, 64 bit atomic_xor: 32, 64 bit atomic_fadd: 32, 64 bit atomic_fand: 32, 64 bit atomic_for: 32, 64 bit atomic_fxor: 32, 64 bit atomic_swap: 32, 64 bit atomic_cswap: 32, 64 bit connection: to iface device priority: 38 device num paths: 1 max eps: inf device address: 3 bytes iface address: 5 bytes error handling: buffer (zcopy), remote access, peer failure, ep_check

  Transport: rc_verbs Device: mlx5_0:1 Type: network System device: mlx5_0 (0)

  capabilities: bandwidth: 11794.23/ppn + 0.00 MB/sec latency: 600 + 1.000 * N nsec overhead: 75 nsec put_short: <= 124 put_bcopy: <= 8256 put_zcopy: <= 1G, up to 5 iov put_opt_zcopy_align: <= 512 put_align_mtu: <= 4K get_bcopy: <= 8256 get_zcopy: 65..1G, up to 5 iov get_opt_zcopy_align: <= 512 get_align_mtu: <= 4K am_short: <= 123 am_bcopy: <= 8255 am_zcopy: <= 8255, up to 4 iov am_opt_zcopy_align: <= 512 am_align_mtu: <= 4K am header: <= 127 domain: device atomic_add: 64 bit atomic_fadd: 64 bit atomic_cswap: 64 bit connection: to ep device priority: 38 device num paths: 1 max eps: 256 device address: 3 bytes ep address: 5 bytes error handling: peer failure, ep_check

  Transport: rc_mlx5 Device: mlx5_0:1 Type: network System device: mlx5_0 (0)

  capabilities: bandwidth: 11794.23/ppn + 0.00 MB/sec latency: 600 + 1.000 * N nsec overhead: 40 nsec put_short: <= 2K put_bcopy: <= 8256 put_zcopy: <= 1G, up to 14 iov put_opt_zcopy_align: <= 512 put_align_mtu: <= 4K get_bcopy: <= 8256 get_zcopy: 65..1G, up to 14 iov get_opt_zcopy_align: <= 512 get_align_mtu: <= 4K am_short: <= 2046 am_bcopy: <= 8254 am_zcopy: <= 8254, up to 3 iov am_opt_zcopy_align: <= 512 am_align_mtu: <= 4K am header: <= 186 domain: device atomic_add: 32, 64 bit atomic_and: 32, 64 bit atomic_or: 32, 64 bit atomic_xor: 32, 64 bit atomic_fadd: 32, 64 bit atomic_fand: 32, 64 bit atomic_for: 32, 64 bit atomic_fxor: 32, 64 bit atomic_swap: 32, 64 bit atomic_cswap: 32, 64 bit connection: to ep device priority: 38 device num paths: 1 max eps: 256 device address: 3 bytes ep address: 7 bytes error handling: buffer (zcopy), remote access, peer failure, ep_check

  Transport: ud_verbs Device: mlx5_0:1 Type: network System device: mlx5_0 (0)

  capabilities: bandwidth: 11794.23/ppn + 0.00 MB/sec latency: 630 nsec overhead: 105 nsec am_short: <= 116 am_bcopy: <= 4088 am_zcopy: <= 4088, up to 5 iov am_opt_zcopy_align: <= 512 am_align_mtu: <= 4K am header: <= 3952 connection: to ep, to iface device priority: 38 device num paths: 1 max eps: inf device address: 3 bytes iface address: 3 bytes ep address: 6 bytes error handling: peer failure, ep_check

  Transport: ud_mlx5 Device: mlx5_0:1 Type: network System device: mlx5_0 (0)

  capabilities: bandwidth: 11794.23/ppn + 0.00 MB/sec latency: 630 nsec overhead: 80 nsec am_short: <= 180 am_bcopy: <= 4088 am_zcopy: <= 4088, up to 3 iov am_opt_zcopy_align: <= 512 am_align_mtu: <= 4K am header: <= 132 connection: to ep, to iface device priority: 38 device num paths: 1 max eps: inf device address: 3 bytes iface address: 3 bytes ep address: 6 bytes error handling: peer failure, ep_check

  Memory domain: mlx5_1 Component: ib register: unlimited, cost: 180 nsec remote key: 8 bytes local memory handle is required for zcopy

  Transport: dc_mlx5 Device: mlx5_1:1 Type: network System device: mlx5_1 (1)

  capabilities: bandwidth: 11794.23/ppn + 0.00 MB/sec latency: 660 nsec overhead: 40 nsec put_short: <= 2K put_bcopy: <= 8256 put_zcopy: <= 1G, up to 11 iov put_opt_zcopy_align: <= 512 put_align_mtu: <= 4K get_bcopy: <= 8256 get_zcopy: 65..1G, up to 11 iov get_opt_zcopy_align: <= 512 get_align_mtu: <= 4K am_short: <= 2046 am_bcopy: <= 8254 am_zcopy: <= 8254, up to 3 iov am_opt_zcopy_align: <= 512 am_align_mtu: <= 4K am header: <= 138 domain: device atomic_add: 32, 64 bit atomic_and: 32, 64 bit atomic_or: 32, 64 bit atomic_xor: 32, 64 bit atomic_fadd: 32, 64 bit atomic_fand: 32, 64 bit atomic_for: 32, 64 bit atomic_fxor: 32, 64 bit atomic_swap: 32, 64 bit atomic_cswap: 32, 64 bit connection: to iface device priority: 38 device num paths: 1 max eps: inf device address: 3 bytes iface address: 5 bytes error handling: buffer (zcopy), remote access, peer failure, ep_check

  Transport: rc_verbs Device: mlx5_1:1 Type: network System device: mlx5_1 (1)

  capabilities: bandwidth: 11794.23/ppn + 0.00 MB/sec latency: 600 + 1.000 * N nsec overhead: 75 nsec put_short: <= 124 put_bcopy: <= 8256 put_zcopy: <= 1G, up to 5 iov put_opt_zcopy_align: <= 512 put_align_mtu: <= 4K get_bcopy: <= 8256 get_zcopy: 65..1G, up to 5 iov get_opt_zcopy_align: <= 512 get_align_mtu: <= 4K am_short: <= 123 am_bcopy: <= 8255 am_zcopy: <= 8255, up to 4 iov am_opt_zcopy_align: <= 512 am_align_mtu: <= 4K am header: <= 127 domain: device atomic_add: 64 bit atomic_fadd: 64 bit atomic_cswap: 64 bit connection: to ep device priority: 38 device num paths: 1 max eps: 256 device address: 3 bytes ep address: 5 bytes error handling: peer failure, ep_check

  Transport: rc_mlx5 Device: mlx5_1:1 Type: network System device: mlx5_1 (1)

  capabilities: bandwidth: 11794.23/ppn + 0.00 MB/sec latency: 600 + 1.000 * N nsec overhead: 40 nsec put_short: <= 2K put_bcopy: <= 8256 put_zcopy: <= 1G, up to 14 iov put_opt_zcopy_align: <= 512 put_align_mtu: <= 4K get_bcopy: <= 8256 get_zcopy: 65..1G, up to 14 iov get_opt_zcopy_align: <= 512 get_align_mtu: <= 4K am_short: <= 2046 am_bcopy: <= 8254 am_zcopy: <= 8254, up to 3 iov am_opt_zcopy_align: <= 512 am_align_mtu: <= 4K am header: <= 186 domain: device atomic_add: 32, 64 bit atomic_and: 32, 64 bit atomic_or: 32, 64 bit atomic_xor: 32, 64 bit atomic_fadd: 32, 64 bit atomic_fand: 32, 64 bit atomic_for: 32, 64 bit atomic_fxor: 32, 64 bit atomic_swap: 32, 64 bit atomic_cswap: 32, 64 bit connection: to ep device priority: 38 device num paths: 1 max eps: 256 device address: 3 bytes ep address: 7 bytes error handling: buffer (zcopy), remote access, peer failure, ep_check

  Transport: ud_verbs Device: mlx5_1:1 Type: network System device: mlx5_1 (1)

  capabilities: bandwidth: 11794.23/ppn + 0.00 MB/sec latency: 630 nsec overhead: 105 nsec am_short: <= 116 am_bcopy: <= 4088 am_zcopy: <= 4088, up to 5 iov am_opt_zcopy_align: <= 512 am_align_mtu: <= 4K am header: <= 3952 connection: to ep, to iface device priority: 38 device num paths: 1 max eps: inf device address: 3 bytes iface address: 3 bytes ep address: 6 bytes error handling: peer failure, ep_check

  Transport: ud_mlx5 Device: mlx5_1:1 Type: network System device: mlx5_1 (1)

  capabilities: bandwidth: 11794.23/ppn + 0.00 MB/sec latency: 630 nsec overhead: 80 nsec am_short: <= 180 am_bcopy: <= 4088 am_zcopy: <= 4088, up to 3 iov am_opt_zcopy_align: <= 512 am_align_mtu: <= 4K am header: <= 132 connection: to ep, to iface device priority: 38 device num paths: 1 max eps: inf device address: 3 bytes iface address: 3 bytes ep address: 6 bytes error handling: peer failure, ep_check

  Memory domain: mlx5_2 Component: ib register: unlimited, cost: 180 nsec remote key: 8 bytes local memory handle is required for zcopy

  Transport: dc_mlx5 Device: mlx5_2:1 Type: network System device: mlx5_2 (2)

  capabilities: bandwidth: 11664.63/ppn + 0.00 MB/sec latency: 860 nsec overhead: 40 nsec put_short: <= 2K put_bcopy: <= 8256 put_zcopy: <= 1G, up to 11 iov put_opt_zcopy_align: <= 512 put_align_mtu: <= 4K get_bcopy: <= 8256 get_zcopy: 65..1G, up to 11 iov get_opt_zcopy_align: <= 512 get_align_mtu: <= 4K am_short: <= 2046 am_bcopy: <= 8254 am_zcopy: <= 8254, up to 3 iov am_opt_zcopy_align: <= 512 am_align_mtu: <= 4K am header: <= 138 domain: device atomic_add: 32, 64 bit atomic_and: 32, 64 bit atomic_or: 32, 64 bit atomic_xor: 32, 64 bit atomic_fadd: 32, 64 bit atomic_fand: 32, 64 bit atomic_for: 32, 64 bit atomic_fxor: 32, 64 bit atomic_swap: 32, 64 bit atomic_cswap: 32, 64 bit connection: to iface device priority: 38 device num paths: 1 max eps: inf device address: 17 bytes iface address: 5 bytes error handling: buffer (zcopy), remote access, peer failure, ep_check

  Transport: rc_verbs Device: mlx5_2:1 Type: network System device: mlx5_2 (2)

  capabilities: bandwidth: 11664.63/ppn + 0.00 MB/sec latency: 800 + 1.000 * N nsec overhead: 75 nsec put_short: <= 124 put_bcopy: <= 8256 put_zcopy: <= 1G, up to 5 iov put_opt_zcopy_align: <= 512 put_align_mtu: <= 4K get_bcopy: <= 8256 get_zcopy: 65..1G, up to 5 iov get_opt_zcopy_align: <= 512 get_align_mtu: <= 4K am_short: <= 123 am_bcopy: <= 8255 am_zcopy: <= 8255, up to 4 iov am_opt_zcopy_align: <= 512 am_align_mtu: <= 4K am header: <= 127 domain: device atomic_add: 64 bit atomic_fadd: 64 bit atomic_cswap: 64 bit connection: to ep device priority: 38 device num paths: 1 max eps: 256 device address: 17 bytes ep address: 5 bytes error handling: peer failure, ep_check

  Transport: rc_mlx5 Device: mlx5_2:1 Type: network System device: mlx5_2 (2)

  capabilities: bandwidth: 11664.63/ppn + 0.00 MB/sec latency: 800 + 1.000 * N nsec overhead: 40 nsec put_short: <= 2K put_bcopy: <= 8256 put_zcopy: <= 1G, up to 14 iov put_opt_zcopy_align: <= 512 put_align_mtu: <= 4K get_bcopy: <= 8256 get_zcopy: 65..1G, up to 14 iov get_opt_zcopy_align: <= 512 get_align_mtu: <= 4K am_short: <= 2046 am_bcopy: <= 8254 am_zcopy: <= 8254, up to 3 iov am_opt_zcopy_align: <= 512 am_align_mtu: <= 4K am header: <= 186 domain: device atomic_add: 32, 64 bit atomic_and: 32, 64 bit atomic_or: 32, 64 bit atomic_xor: 32, 64 bit atomic_fadd: 32, 64 bit atomic_fand: 32, 64 bit atomic_for: 32, 64 bit atomic_fxor: 32, 64 bit atomic_swap: 32, 64 bit atomic_cswap: 32, 64 bit connection: to ep device priority: 38 device num paths: 1 max eps: 256 device address: 17 bytes ep address: 7 bytes error handling: buffer (zcopy), remote access, peer failure, ep_check

  Transport: ud_verbs Device: mlx5_2:1 Type: network System device: mlx5_2 (2)

  capabilities: bandwidth: 11664.63/ppn + 0.00 MB/sec latency: 830 nsec overhead: 105 nsec am_short: <= 116 am_bcopy: <= 4088 am_zcopy: <= 4088, up to 5 iov am_opt_zcopy_align: <= 512 am_align_mtu: <= 4K am header: <= 3952 connection: to ep, to iface device priority: 38 device num paths: 1 max eps: inf device address: 17 bytes iface address: 3 bytes ep address: 6 bytes error handling: peer failure, ep_check

  Transport: ud_mlx5 Device: mlx5_2:1 Type: network System device: mlx5_2 (2)

  capabilities: bandwidth: 11664.63/ppn + 0.00 MB/sec latency: 830 nsec overhead: 80 nsec am_short: <= 180 am_bcopy: <= 4088 am_zcopy: <= 4088, up to 3 iov am_opt_zcopy_align: <= 512 am_align_mtu: <= 4K am header: <= 132 connection: to ep, to iface device priority: 38 device num paths: 1 max eps: inf device address: 17 bytes iface address: 3 bytes ep address: 6 bytes error handling: peer failure, ep_check

  Memory domain: mlx5_3 Component: ib register: unlimited, cost: 180 nsec remote key: 8 bytes local memory handle is required for zcopy

  Transport: dc_mlx5 Device: mlx5_3:1 Type: network System device: mlx5_3 (3)

  capabilities: bandwidth: 11664.63/ppn + 0.00 MB/sec latency: 860 nsec overhead: 40 nsec put_short: <= 2K put_bcopy: <= 8256 put_zcopy: <= 1G, up to 11 iov put_opt_zcopy_align: <= 512 put_align_mtu: <= 4K get_bcopy: <= 8256 get_zcopy: 65..1G, up to 11 iov get_opt_zcopy_align: <= 512 get_align_mtu: <= 4K am_short: <= 2046 am_bcopy: <= 8254 am_zcopy: <= 8254, up to 3 iov am_opt_zcopy_align: <= 512 am_align_mtu: <= 4K am header: <= 138 domain: device atomic_add: 32, 64 bit atomic_and: 32, 64 bit atomic_or: 32, 64 bit atomic_xor: 32, 64 bit atomic_fadd: 32, 64 bit atomic_fand: 32, 64 bit atomic_for: 32, 64 bit atomic_fxor: 32, 64 bit atomic_swap: 32, 64 bit atomic_cswap: 32, 64 bit connection: to iface device priority: 38 device num paths: 1 max eps: inf device address: 17 bytes iface address: 5 bytes error handling: buffer (zcopy), remote access, peer failure, ep_check

  Transport: rc_verbs Device: mlx5_3:1 Type: network System device: mlx5_3 (3)

  capabilities: bandwidth: 11664.63/ppn + 0.00 MB/sec latency: 800 + 1.000 * N nsec overhead: 75 nsec put_short: <= 124 put_bcopy: <= 8256 put_zcopy: <= 1G, up to 5 iov put_opt_zcopy_align: <= 512 put_align_mtu: <= 4K get_bcopy: <= 8256 get_zcopy: 65..1G, up to 5 iov get_opt_zcopy_align: <= 512 get_align_mtu: <= 4K am_short: <= 123 am_bcopy: <= 8255 am_zcopy: <= 8255, up to 4 iov am_opt_zcopy_align: <= 512 am_align_mtu: <= 4K am header: <= 127 domain: device atomic_add: 64 bit atomic_fadd: 64 bit atomic_cswap: 64 bit connection: to ep device priority: 38 device num paths: 1 max eps: 256 device address: 17 bytes ep address: 5 bytes error handling: peer failure, ep_check

  Transport: rc_mlx5 Device: mlx5_3:1 Type: network System device: mlx5_3 (3)

  capabilities: bandwidth: 11664.63/ppn + 0.00 MB/sec latency: 800 + 1.000 * N nsec overhead: 40 nsec put_short: <= 2K put_bcopy: <= 8256 put_zcopy: <= 1G, up to 14 iov put_opt_zcopy_align: <= 512 put_align_mtu: <= 4K get_bcopy: <= 8256 get_zcopy: 65..1G, up to 14 iov get_opt_zcopy_align: <= 512 get_align_mtu: <= 4K am_short: <= 2046 am_bcopy: <= 8254 am_zcopy: <= 8254, up to 3 iov am_opt_zcopy_align: <= 512 am_align_mtu: <= 4K am header: <= 186 domain: device atomic_add: 32, 64 bit atomic_and: 32, 64 bit atomic_or: 32, 64 bit atomic_xor: 32, 64 bit atomic_fadd: 32, 64 bit atomic_fand: 32, 64 bit atomic_for: 32, 64 bit atomic_fxor: 32, 64 bit atomic_swap: 32, 64 bit atomic_cswap: 32, 64 bit connection: to ep device priority: 38 device num paths: 1 max eps: 256 device address: 17 bytes ep address: 7 bytes error handling: buffer (zcopy), remote access, peer failure, ep_check

  Transport: ud_verbs Device: mlx5_3:1 Type: network System device: mlx5_3 (3)

  capabilities: bandwidth: 11664.63/ppn + 0.00 MB/sec latency: 830 nsec overhead: 105 nsec am_short: <= 116 am_bcopy: <= 4088 am_zcopy: <= 4088, up to 5 iov am_opt_zcopy_align: <= 512 am_align_mtu: <= 4K am header: <= 3952 connection: to ep, to iface device priority: 38 device num paths: 1 max eps: inf device address: 17 bytes iface address: 3 bytes ep address: 6 bytes error handling: peer failure, ep_check

  Transport: ud_mlx5 Device: mlx5_3:1 Type: network System device: mlx5_3 (3)

  capabilities: bandwidth: 11664.63/ppn + 0.00 MB/sec latency: 830 nsec overhead: 80 nsec am_short: <= 180 am_bcopy: <= 4088 am_zcopy: <= 4088, up to 3 iov am_opt_zcopy_align: <= 512 am_align_mtu: <= 4K am header: <= 132 connection: to ep, to iface device priority: 38 device num paths: 1 max eps: inf device address: 17 bytes iface address: 3 bytes ep address: 6 bytes error handling: peer failure, ep_check

  Connection manager: rdmacm max_conn_priv: 54 bytes

  Memory domain: cma Component: cma register: unlimited, cost: 9 nsec

  Transport: cma Device: memory Type: intra-node System device:

  capabilities: bandwidth: 0.00/ppn + 11145.00 MB/sec latency: 80 nsec overhead: 2000 nsec put_zcopy: unlimited, up to 16 iov put_opt_zcopy_align: <= 1 put_align_mtu: <= 1 get_zcopy: unlimited, up to 16 iov get_opt_zcopy_align: <= 1 get_align_mtu: <= 1 connection: to iface device priority: 0 device num paths: 1 max eps: inf device address: 8 bytes iface address: 4 bytes error handling: peer failure, ep_check

  Memory domain: knem Component: knem register: unlimited, cost: 180 nsec remote key: 16 bytes

  Transport: knem Device: memory Type: intra-node System device:

  capabilities: bandwidth: 13862.00/ppn + 0.00 MB/sec latency: 80 nsec overhead: 2000 nsec put_zcopy: unlimited, up to 16 iov put_opt_zcopy_align: <= 1 put_align_mtu: <= 1 get_zcopy: unlimited, up to 16 iov get_opt_zcopy_align: <= 1 get_align_mtu: <= 1 connection: to iface device priority: 0 device num paths: 1 max eps: inf device address: 8 bytes iface address: 0 bytes error handling: none

• Configure result - config.log

• Log file - configure UCX with "--enable-logging" - and run with "UCX_LOG_LEVEL=data"

[brminich]

Is there any failure or another symptom? This message is not a bug, because it is diag trace (not an error). UCX is able to shrink the address when this situation happens.

[rrgargeya]

No, there is no failure. But I was wondering if the DIAG trace is indicating a possible problem and if I should do something about it. From your response am I to gather that this trace message is harmless and UCX is taking counter measures?