search

pentschev / ucx-py-ci

UCX-Py CI Issue Tracker
1 stars 1 forks source link

Nightly Tests for ucx-1.17 from 2024-09-16 22:00: 45 failures; no timeouts; 20/36 scenarios with performance regressions #3054

Open pentschev opened 1 month ago

pentschev commented 1 month ago

Test results

Failures

8 failures in `dask-cuda`, [full logs](https://raw.githack.com/pentschev/ucx-py-ci/test-results/assets/ucx-1.17-dask-cuda-202409162200.html). Failed tests: - `dask_cuda.tests.test_dask_cuda_worker::test_cuda_visible_devices_and_memory_limit_and_nthreads` - `dask_cuda.tests.test_dask_cuda_worker::test_rmm_pool` - `dask_cuda.tests.test_dask_cuda_worker::test_rmm_managed` - `dask_cuda.tests.test_dask_cuda_worker::test_rmm_async` - `dask_cuda.tests.test_dask_cuda_worker::test_rmm_async_with_maximum_pool_size` - `dask_cuda.tests.test_dask_cuda_worker::test_cudf_spill` - `dask_cuda.tests.test_dask_cuda_worker::test_dashboard_address` - `dask_cuda.tests.test_proxify_host_file::test_spill_on_demand`
1 failures in `ucx-py-ib-rdmacm-debug-test`, [full logs](https://raw.githack.com/pentschev/ucx-py-ci/test-results/assets/ucx-1.17-ucx-py-ib-rdmacm-debug-test-202409162200.html). Failed tests: - `::debug-tests.test_send_recv_many_workers`
12 failures in `ucx-py-libs-ib-test`, [full logs](https://raw.githack.com/pentschev/ucx-py-ci/test-results/assets/ucx-1.17-ucx-py-libs-ib-test-202409162200.html). Failed tests: - `::ucp._libs.tests.test_address_object` - `::ucp._libs.tests.test_arr` - `::ucp._libs.tests.test_cancel` - `::ucp._libs.tests.test_config` - `::ucp._libs.tests.test_endpoint` - `::ucp._libs.tests.test_listener` - `::ucp._libs.tests.test_mem` - `::ucp._libs.tests.test_peer_send_recv` - `::ucp._libs.tests.test_probe` - `::ucp._libs.tests.test_rma` - `::ucp._libs.tests.test_server_client` - `::ucp._libs.tests.test_server_client_am`
12 failures in `ucx-py-libs-nvlink-test`, [full logs](https://raw.githack.com/pentschev/ucx-py-ci/test-results/assets/ucx-1.17-ucx-py-libs-nvlink-test-202409162200.html). Failed tests: - `::ucp._libs.tests.test_address_object` - `::ucp._libs.tests.test_arr` - `::ucp._libs.tests.test_cancel` - `::ucp._libs.tests.test_config` - `::ucp._libs.tests.test_endpoint` - `::ucp._libs.tests.test_listener` - `::ucp._libs.tests.test_mem` - `::ucp._libs.tests.test_peer_send_recv` - `::ucp._libs.tests.test_probe` - `::ucp._libs.tests.test_rma` - `::ucp._libs.tests.test_server_client` - `::ucp._libs.tests.test_server_client_am`
12 failures in `ucx-py-libs-tcp-test`, [full logs](https://raw.githack.com/pentschev/ucx-py-ci/test-results/assets/ucx-1.17-ucx-py-libs-tcp-test-202409162200.html). Failed tests: - `::ucp._libs.tests.test_address_object` - `::ucp._libs.tests.test_arr` - `::ucp._libs.tests.test_cancel` - `::ucp._libs.tests.test_config` - `::ucp._libs.tests.test_endpoint` - `::ucp._libs.tests.test_listener` - `::ucp._libs.tests.test_mem` - `::ucp._libs.tests.test_peer_send_recv` - `::ucp._libs.tests.test_probe` - `::ucp._libs.tests.test_rma` - `::ucp._libs.tests.test_server_client` - `::ucp._libs.tests.test_server_client_am`

Performance results

Failures

Scenario numpy-TAG-Core-TCP SM expected_bw=4.0GB/s failed with bw 3.46 GB/s; has never passed

Scenario numpy-TAG-Core-RC expected_bw=16.0GB/s failed with bw 8.21 GB/s; has never passed

Scenario numpy-AM-Core-TCP SM expected_bw=2.8GB/s failed with bw 2.54 GB/s; has never passed

Scenario numpy-AM-Core-RC expected_bw=3.0GB/s failed with bw 2.07 GB/s; has never passed

Scenario numpy-TAG-Async-TCP SM expected_bw=4.0GB/s failed with bw 3.52 GB/s; has never passed

Scenario numpy-TAG-Async-RC expected_bw=16.0GB/s failed with bw 8.1 GB/s; has never passed

Scenario numpy-AM-Async-TCP SM expected_bw=2.8GB/s failed with bw 2.51 GB/s; has never passed

Scenario numpy-AM-Async-RC expected_bw=3.0GB/s failed with bw 2.09 GB/s; has never passed

Scenario cupy-TAG-Core-TCP SM expected_bw=3.0GB/s failed with bw 2.37 GB/s; has never passed

Scenario cupy-TAG-Core-TCP expected_bw=3.0GB/s failed with bw 2.36 GB/s; has never passed

Scenario cupy-TAG-Core-RC, expected_bw=12.0GB/s failed with bw 11.36GB/s; last pass on 2024-09-07T22:00:00 (UCX-Py version 0.40.0; UCX commit 770b5a6)

Scenario cupy-AM-Core-TCP SM expected_bw=3.0GB/s failed with bw 2.38 GB/s; has never passed

Scenario cupy-AM-Core-TCP expected_bw=3.0GB/s failed with bw 2.36 GB/s; has never passed

Scenario cupy-AM-Core-RC, expected_bw=12.0GB/s failed with bw 11.35GB/s; last pass on 2024-09-09T22:00:00 (UCX-Py version 0.40.0; UCX commit 770b5a6)

Scenario cupy-TAG-Async-TCP SM expected_bw=3.0GB/s failed with bw 2.34 GB/s; has never passed

Scenario cupy-TAG-Async-TCP expected_bw=3.0GB/s failed with bw 2.32 GB/s; has never passed

Scenario cupy-TAG-Async-RC, expected_bw=12.0GB/s failed with bw 11.32GB/s; last pass on 2024-07-18T05:00:00 (UCX-Py version 0.39.0; UCX commit 770b5a6)

Scenario cupy-AM-Async-TCP SM expected_bw=3.0GB/s failed with bw 2.09 GB/s; has never passed

Scenario cupy-AM-Async-TCP expected_bw=3.0GB/s failed with bw 2.31 GB/s; has never passed

Scenario cupy-AM-Async-RC expected_bw=12.0GB/s failed with bw 11.22 GB/s; has never passed

Passes

Scenario numpy-TAG-Core-TCP, expected_bw=2.8GB/s passed with bw 3.36GB/s

Scenario numpy-AM-Core-TCP, expected_bw=2.3GB/s passed with bw 2.54GB/s

Scenario numpy-TAG-Async-TCP, expected_bw=2.8GB/s passed with bw 3.51GB/s

Scenario numpy-AM-Async-TCP, expected_bw=2.3GB/s passed with bw 2.52GB/s

Scenario cupy-TAG-Core-CUDA_IPC_SELF, expected_bw=370.0GB/s passed with bw 372.02GB/s

Scenario cupy-TAG-Core-CUDA_IPC_NV2, expected_bw=48.0GB/s passed with bw 48.43GB/s

Scenario cupy-TAG-Core-CUDA_IPC_NV1, expected_bw=24.0GB/s passed with bw 24.25GB/s

Scenario cupy-AM-Core-CUDA_IPC_SELF, expected_bw=370.0GB/s passed with bw 369.78GB/s

Scenario cupy-AM-Core-CUDA_IPC_NV2, expected_bw=48.0GB/s passed with bw 48.37GB/s

Scenario cupy-AM-Core-CUDA_IPC_NV1, expected_bw=24.0GB/s passed with bw 24.22GB/s

Scenario cupy-TAG-Async-CUDA_IPC_SELF, expected_bw=325.0GB/s passed with bw 324.91GB/s

Scenario cupy-TAG-Async-CUDA_IPC_NV2, expected_bw=48.0GB/s passed with bw 47.58GB/s

Scenario cupy-TAG-Async-CUDA_IPC_NV1, expected_bw=24.0GB/s passed with bw 24.02GB/s

Scenario cupy-AM-Async-CUDA_IPC_SELF, expected_bw=325.0GB/s passed with bw 324.66GB/s

Scenario cupy-AM-Async-CUDA_IPC_NV2, expected_bw=48.0GB/s passed with bw 47.48GB/s

Scenario cupy-AM-Async-CUDA_IPC_NV1, expected_bw=24.0GB/s passed with bw 23.99GB/s