search

pentschev / ucx-py-ci

UCX-Py CI Issue Tracker
1 stars 1 forks source link

Nightly Tests for ucx-master from 2024-09-12 22:00: 6 failures; no timeouts; 20/36 scenarios with performance regressions #3041

Open pentschev opened 1 week ago

pentschev commented 1 week ago

Test results

Failures

1 failures in `dask-cuda`, [full logs](https://raw.githack.com/pentschev/ucx-py-ci/test-results/assets/ucx-master-dask-cuda-202409122200.html). Failed tests: - `dask_cuda.tests.test_proxify_host_file::test_spill_on_demand`
5 failures in `ucx-py-ib-test`, [full logs](https://raw.githack.com/pentschev/ucx-py-ci/test-results/assets/ucx-master-ucx-py-ib-test-202409122200.html). Failed tests: - `tests.test_probe::test_message_probe[am]` - `tests.test_probe::test_message_probe[tag]` - `tests.test_reset::test_reset` - `tests.test_ucx_getters::test_get_ucp_worker` - `tests.test_ucx_getters::test_get_endpoint`

Passes

All tests passed in ucx-py-ib-rdmacm-debug-test, full logs. All tests passed in ucx-py-libs-ib-test, full logs. All tests passed in ucx-py-libs-nvlink-test, full logs. All tests passed in ucx-py-libs-tcp-test, full logs. All tests passed in ucx-py-nvlink-test, full logs. All tests passed in ucx-py-tcp-test, full logs.

Performance results

Failures

Scenario numpy-TAG-Core-TCP SM expected_bw=4.0GB/s failed with bw 3.41 GB/s; has never passed

Scenario numpy-TAG-Core-RC expected_bw=16.0GB/s failed with bw 8.2 GB/s; has never passed

Scenario numpy-AM-Core-TCP SM expected_bw=2.8GB/s failed with bw 2.54 GB/s; has never passed

Scenario numpy-AM-Core-RC expected_bw=3.0GB/s failed with bw 2.08 GB/s; has never passed

Scenario numpy-TAG-Async-TCP SM expected_bw=4.0GB/s failed with bw 3.5 GB/s; has never passed

Scenario numpy-TAG-Async-RC expected_bw=16.0GB/s failed with bw 8.16 GB/s; has never passed

Scenario numpy-AM-Async-TCP SM expected_bw=2.8GB/s failed with bw 2.53 GB/s; has never passed

Scenario numpy-AM-Async-RC expected_bw=3.0GB/s failed with bw 2.09 GB/s; has never passed

Scenario cupy-TAG-Core-TCP SM expected_bw=3.0GB/s failed with bw 2.37 GB/s; has never passed

Scenario cupy-TAG-Core-TCP expected_bw=3.0GB/s failed with bw 2.36 GB/s; has never passed

Scenario cupy-TAG-Core-RC, expected_bw=12.0GB/s failed with bw 11.27GB/s; last pass on 2024-09-08T22:00:00 (UCX-Py version 0.40.0; UCX commit fd9deb4)

Scenario cupy-AM-Core-TCP SM expected_bw=3.0GB/s failed with bw 2.36 GB/s; has never passed

Scenario cupy-AM-Core-TCP expected_bw=3.0GB/s failed with bw 2.37 GB/s; has never passed

Scenario cupy-AM-Core-RC, expected_bw=12.0GB/s failed with bw 11.35GB/s; last pass on 2024-09-05T22:00:00 (UCX-Py version 0.40.0; UCX commit 2a78fca)

Scenario cupy-TAG-Async-TCP SM expected_bw=3.0GB/s failed with bw 2.34 GB/s; has never passed

Scenario cupy-TAG-Async-TCP expected_bw=3.0GB/s failed with bw 2.33 GB/s; has never passed

Scenario cupy-TAG-Async-RC, expected_bw=12.0GB/s failed with bw 11.31GB/s; last pass on 2024-05-08T05:00:00 (UCX-Py version 0.38.0; UCX commit 63f85a9)

Scenario cupy-AM-Async-TCP SM expected_bw=3.0GB/s failed with bw 2.34 GB/s; has never passed

Scenario cupy-AM-Async-TCP expected_bw=3.0GB/s failed with bw 2.33 GB/s; has never passed

Scenario cupy-AM-Async-RC, expected_bw=12.0GB/s failed with bw 11.23GB/s; last pass on 2024-05-09T05:00:00 (UCX-Py version 0.38.0; UCX commit f02bc54)

Passes

Scenario numpy-TAG-Core-TCP, expected_bw=2.8GB/s passed with bw 3.4GB/s

Scenario numpy-AM-Core-TCP, expected_bw=2.3GB/s passed with bw 2.54GB/s

Scenario numpy-TAG-Async-TCP, expected_bw=2.8GB/s passed with bw 3.53GB/s

Scenario numpy-AM-Async-TCP, expected_bw=2.3GB/s passed with bw 2.53GB/s

Scenario cupy-TAG-Core-CUDA_IPC_SELF, expected_bw=370.0GB/s passed with bw 372.35GB/s

Scenario cupy-TAG-Core-CUDA_IPC_NV2, expected_bw=48.0GB/s passed with bw 48.44GB/s

Scenario cupy-TAG-Core-CUDA_IPC_NV1, expected_bw=24.0GB/s passed with bw 24.23GB/s

Scenario cupy-AM-Core-CUDA_IPC_SELF, expected_bw=370.0GB/s passed with bw 369.45GB/s

Scenario cupy-AM-Core-CUDA_IPC_NV2, expected_bw=48.0GB/s passed with bw 48.34GB/s

Scenario cupy-AM-Core-CUDA_IPC_NV1, expected_bw=24.0GB/s passed with bw 24.21GB/s

Scenario cupy-TAG-Async-CUDA_IPC_SELF, expected_bw=325.0GB/s passed with bw 330.16GB/s

Scenario cupy-TAG-Async-CUDA_IPC_NV2, expected_bw=48.0GB/s passed with bw 47.27GB/s

Scenario cupy-TAG-Async-CUDA_IPC_NV1, expected_bw=24.0GB/s passed with bw 23.96GB/s

Scenario cupy-AM-Async-CUDA_IPC_SELF, expected_bw=325.0GB/s passed with bw 323.57GB/s

Scenario cupy-AM-Async-CUDA_IPC_NV2, expected_bw=48.0GB/s passed with bw 47.27GB/s

Scenario cupy-AM-Async-CUDA_IPC_NV1, expected_bw=24.0GB/s passed with bw 23.94GB/s