Nightly Tests for ucx-1.17 from 2024-10-24 22:00: all completed tests passed; 3 timeouts; 20/36 scenarios with performance regressions

[pentschev]

Test results

Timeouts

dask-cuda-test timed out, full logs. local-cudf-merge-ucx-nvlink timed out, full logs. local-cupy-transpose-sum-ucx-nvlink timed out, full logs.

Passes

All tests passed in ucx-py-ib-rdmacm-debug-test, full logs. All tests passed in ucx-py-ib-test, full logs. All tests passed in ucx-py-libs-ib-test, full logs. All tests passed in ucx-py-libs-nvlink-test, full logs. All tests passed in ucx-py-libs-tcp-test, full logs. All tests passed in ucx-py-nvlink-test, full logs. All tests passed in ucx-py-tcp-test, full logs.

Performance results

Failures

Scenario numpy-TAG-Core-TCP SM expected_bw=4.0GB/s failed with bw 3.6 GB/s; has never passed

Scenario numpy-TAG-Core-RC expected_bw=16.0GB/s failed with bw 8.2 GB/s; has never passed

Scenario numpy-AM-Core-TCP SM expected_bw=2.8GB/s failed with bw 2.62 GB/s; has never passed

Scenario numpy-AM-Core-RC expected_bw=3.0GB/s failed with bw 2.1 GB/s; has never passed

Scenario numpy-TAG-Async-TCP SM expected_bw=4.0GB/s failed with bw 3.61 GB/s; has never passed

Scenario numpy-TAG-Async-RC expected_bw=16.0GB/s failed with bw 8.07 GB/s; has never passed

Scenario numpy-AM-Async-TCP SM expected_bw=2.8GB/s failed with bw 2.6 GB/s; has never passed

Scenario numpy-AM-Async-RC expected_bw=3.0GB/s failed with bw 2.1 GB/s; has never passed

Scenario cupy-TAG-Core-TCP SM expected_bw=3.0GB/s failed with bw 2.43 GB/s; has never passed

Scenario cupy-TAG-Core-TCP expected_bw=3.0GB/s failed with bw 2.43 GB/s; has never passed

Scenario cupy-TAG-Core-RC, expected_bw=12.0GB/s failed with bw 11.29GB/s; last pass on 2024-10-20T22:00:00 (UCX-Py version 0.40.0; UCX commit 4ef9a09)

Scenario cupy-AM-Core-TCP SM expected_bw=3.0GB/s failed with bw 2.43 GB/s; has never passed

Scenario cupy-AM-Core-TCP expected_bw=3.0GB/s failed with bw 2.41 GB/s; has never passed

Scenario cupy-AM-Core-RC, expected_bw=12.0GB/s failed with bw 11.35GB/s; last pass on 2024-09-27T22:00:00 (UCX-Py version 0.40.0; UCX commit 770b5a6)

Scenario cupy-TAG-Async-TCP SM expected_bw=3.0GB/s failed with bw 2.39 GB/s; has never passed

Scenario cupy-TAG-Async-TCP expected_bw=3.0GB/s failed with bw 2.37 GB/s; has never passed

Scenario cupy-TAG-Async-RC, expected_bw=12.0GB/s failed with bw 11.31GB/s; last pass on 2024-10-20T22:00:00 (UCX-Py version 0.40.0; UCX commit 4ef9a09)

Scenario cupy-AM-Async-TCP SM expected_bw=3.0GB/s failed with bw 2.38 GB/s; has never passed

Scenario cupy-AM-Async-TCP expected_bw=3.0GB/s failed with bw 2.38 GB/s; has never passed

Scenario cupy-AM-Async-RC expected_bw=12.0GB/s failed with bw 11.29 GB/s; has never passed

Passes

Scenario numpy-TAG-Core-TCP, expected_bw=2.8GB/s passed with bw 3.49GB/s

Scenario numpy-AM-Core-TCP, expected_bw=2.3GB/s passed with bw 2.62GB/s

Scenario numpy-TAG-Async-TCP, expected_bw=2.8GB/s passed with bw 3.63GB/s

Scenario numpy-AM-Async-TCP, expected_bw=2.3GB/s passed with bw 2.59GB/s

Scenario cupy-TAG-Core-CUDA_IPC_SELF, expected_bw=370.0GB/s passed with bw 372.78GB/s

Scenario cupy-TAG-Core-CUDA_IPC_NV2, expected_bw=48.0GB/s passed with bw 48.44GB/s

Scenario cupy-TAG-Core-CUDA_IPC_NV1, expected_bw=24.0GB/s passed with bw 24.23GB/s

Scenario cupy-AM-Core-CUDA_IPC_SELF, expected_bw=370.0GB/s passed with bw 369.73GB/s

Scenario cupy-AM-Core-CUDA_IPC_NV2, expected_bw=48.0GB/s passed with bw 48.37GB/s

Scenario cupy-AM-Core-CUDA_IPC_NV1, expected_bw=24.0GB/s passed with bw 24.22GB/s

Scenario cupy-TAG-Async-CUDA_IPC_SELF, expected_bw=325.0GB/s passed with bw 325.37GB/s

Scenario cupy-TAG-Async-CUDA_IPC_NV2, expected_bw=48.0GB/s passed with bw 47.3GB/s

Scenario cupy-TAG-Async-CUDA_IPC_NV1, expected_bw=24.0GB/s passed with bw 23.93GB/s

Scenario cupy-AM-Async-CUDA_IPC_SELF, expected_bw=325.0GB/s passed with bw 324.26GB/s

Scenario cupy-AM-Async-CUDA_IPC_NV2, expected_bw=48.0GB/s passed with bw 47.21GB/s

Scenario cupy-AM-Async-CUDA_IPC_NV1, expected_bw=24.0GB/s passed with bw 23.93GB/s