pentschev
commented
1 month ago Indeed that doesn't look right. I do not have immediate access to a system with H100s but I can try to do that tomorrow. In the meantime I was able to run that on a DGX-1 and the results I see are much more in line with what we would expect, although I had to reduce to chunk size to 100M due to the amount of memory in the V100s:
2 GPUs
``` $ python -m ucp.benchmarks.cudf_merge --devs 0,1 --chunk-size 100_000_000 --iter 10 cuDF merge benchmark -------------------------------------------------------------------------------------------------------------- Device(s) | [0, 1] Chunks per device | 1 Rows per chunk | 100000000 Total data processed | 59.60 GiB Data processed per iter | 5.96 GiB Row matching fraction | 0.3 ============================================================================================================== Wall-clock | 3.11 s Bandwidth | 19.62 GiB/s Throughput | 19.16 GiB/s ============================================================================================================== Run | Wall-clock | Bandwidth | Throughput 0 | 310.42 ms | 19.59 GiB/s | 19.20 GiB/s 1 | 309.82 ms | 19.70 GiB/s | 19.24 GiB/s 2 | 309.94 ms | 19.64 GiB/s | 19.23 GiB/s 3 | 309.69 ms | 19.66 GiB/s | 19.25 GiB/s 4 | 310.31 ms | 19.51 GiB/s | 19.21 GiB/s 5 | 310.90 ms | 19.64 GiB/s | 19.17 GiB/s 6 | 310.08 ms | 19.69 GiB/s | 19.22 GiB/s 7 | 312.31 ms | 19.68 GiB/s | 19.09 GiB/s 8 | 310.86 ms | 19.58 GiB/s | 19.17 GiB/s 9 | 310.12 ms | 19.49 GiB/s | 19.22 GiB/s ```4 GPUs
``` $ python -m ucp.benchmarks.cudf_merge --devs 0,1,2,3 --chunk-size 100_000_000 --iter 10 cuDF merge benchmark -------------------------------------------------------------------------------------------------------------- Device(s) | [0, 1, 2, 3] Chunks per device | 1 Rows per chunk | 100000000 Total data processed | 119.21 GiB Data processed per iter | 11.92 GiB Row matching fraction | 0.3 ============================================================================================================== Wall-clock | 2.59 s Bandwidth | 54.69 GiB/s Throughput | 45.95 GiB/s ============================================================================================================== Run | Wall-clock | Bandwidth | Throughput 0 | 259.75 ms | 52.47 GiB/s | 45.89 GiB/s 1 | 258.23 ms | 54.83 GiB/s | 46.16 GiB/s 2 | 257.99 ms | 55.27 GiB/s | 46.21 GiB/s 3 | 259.56 ms | 54.79 GiB/s | 45.93 GiB/s 4 | 260.32 ms | 53.85 GiB/s | 45.79 GiB/s 5 | 258.22 ms | 55.15 GiB/s | 46.16 GiB/s 6 | 258.11 ms | 55.54 GiB/s | 46.19 GiB/s 7 | 258.34 ms | 54.97 GiB/s | 46.14 GiB/s 8 | 258.46 ms | 55.07 GiB/s | 46.12 GiB/s 9 | 258.89 ms | 55.12 GiB/s | 46.05 GiB/s ```8 GPUs
``` $ python -m ucp.benchmarks.cudf_merge --devs 0,1,2,3,4,5,6,7 --chunk-size 100_000_000 -- iter 10 cuDF merge benchmark -------------------------------------------------------------------------------------------------------------- Device(s) | [0, 1, 2, 3, 4, 5, 6, 7] Chunks per device | 1 Rows per chunk | 100000000 Total data processed | 238.42 GiB Data processed per iter | 23.84 GiB Row matching fraction | 0.3 ============================================================================================================== Wall-clock | 5.18 s Bandwidth | 12.45 GiB/s Throughput | 45.99 GiB/s ============================================================================================================== Run | Wall-clock | Bandwidth | Throughput 0 | 519.35 ms | 12.41 GiB/s | 45.91 GiB/s 1 | 521.78 ms | 12.41 GiB/s | 45.69 GiB/s 2 | 519.69 ms | 12.42 GiB/s | 45.88 GiB/s 3 | 514.90 ms | 12.45 GiB/s | 46.30 GiB/s 4 | 517.55 ms | 12.46 GiB/s | 46.07 GiB/s 5 | 515.59 ms | 12.51 GiB/s | 46.24 GiB/s 6 | 521.08 ms | 12.43 GiB/s | 45.75 GiB/s 7 | 514.37 ms | 12.41 GiB/s | 46.35 GiB/s 8 | 516.32 ms | 12.48 GiB/s | 46.18 GiB/s 9 | 516.35 ms | 12.47 GiB/s | 46.17 GiB/s ```Based on the affinity reported by your system as the output of nvidia-smi topo -m I suspect this is a only a partition of a node, is that right? Are you able to get a full node allocation to test it as well? Could you also try to disable InfiniBand (UCX_TLS=^rc) and later NVLink (UCX_TLS=cuda_ipc) to see whether the errors and the variability go away? Could you also report what cat /proc/cpuinfo shows (I want that we confirm that only the cores with matching affinity are available to your node partition, but I'm not entirely sure only /proc/cpuinfo will provide us with that information)?
And finally, is it expected that the benchmark saturate the available bandwidth between the GPUs?
That is a good question, I'm not sure we have done such testing in the past but for the 2 GPU case we get 85-90% of expected, with ~19.5GiB/s where ucx_perftest alone reports ~22.5GiB/s (note ucx_perftest reports GB/s and not GiB/s, I've done the conversion on my own so we're comparing at right scale):
ucx_perftest 2xV100
``` $ ucx_perftest -t tag_bw -m cuda -s 1000000000 -n 1000 & ucx_perftest -t tag_bw -m cuda -s 1000000000 -n 1000 localhost [1] 1875533 [1729719349.121679] [dgx13:1875534:0] perftest.c:793 UCX WARN CPU affinity is not set (bound to 80 cpus). Performance may be impacted. [1729719349.121731] [dgx13:1875533:0] perftest.c:793 UCX WARN CPU affinity is not set (bound to 80 cpus). Performance may be impacted. Waiting for connection... Accepted connection from 127.0.0.1:33582 +----------------------------------------------------------------------------------------------------------+ +--------------+--------------+------------------------------+---------------------+-----------------------+ | API: protocol layer | | | | overhead (usec) | bandwidth (MB/s) | message rate (msg/s) | | Test: tag match bandwidth | +--------------+--------------+----------+---------+---------+----------+----------+-----------+-----------+ | Data layout: (automatic) | | Stage | # iterations | 50.0%ile | average | overall | average | overall | average | overall | | Send memory: cuda | +--------------+--------------+----------+---------+---------+----------+----------+-----------+-----------+ | Recv memory: cuda | | Message size: 1000000000 | | Window size: 32 | +----------------------------------------------------------------------------------------------------------+ [thread 0] 57 3.855 18078.963 18078.963 52750.50 52750.50 55 55 [thread 0] 82 41114.179 41216.364 25133.049 23138.24 37945.03 24 40 [thread 0] 107 41118.761 41216.316 28890.821 23138.27 33009.60 24 35 [thread 0] 132 41119.304 41216.402 31225.212 23138.22 30541.80 24 32 [thread 0] 157 41119.606 41216.240 32816.140 23138.31 29061.14 24 30 [thread 0] 182 41119.606 41216.278 33970.005 23138.29 28074.01 24 29 [thread 0] 207 41119.798 41216.326 34845.164 23138.27 27368.91 24 29 [thread 0] 232 41119.923 41216.316 35531.711 23138.27 26840.09 24 28 [thread 0] 257 41120.042 41216.516 36084.708 23138.16 26428.77 24 28 [thread 0] 282 41120.065 41216.278 36539.634 23138.29 26099.72 24 27 [thread 0] 307 41120.199 41216.364 36920.475 23138.24 25830.50 24 27 [thread 0] 332 41120.195 41216.278 37243.954 23138.29 25606.15 24 27 [thread 0] 357 41120.195 41216.478 37522.142 23138.18 25416.31 24 27 [thread 0] 382 41120.199 41216.326 37763.908 23138.27 25253.59 24 26 [thread 0] 407 41120.263 41216.354 37975.975 23138.25 25112.57 24 26 [thread 0] 432 41120.337 41216.364 38163.497 23138.24 24989.17 24 26 [thread 0] 457 41120.364 41216.316 38330.501 23138.27 24880.30 24 26 [thread 0] 482 41120.434 41216.440 38480.186 23138.20 24783.52 24 26 [thread 0] 507 41120.435 41216.288 38615.103 23138.29 24696.93 24 26 [thread 0] 532 41120.435 41216.316 38737.340 23138.27 24618.99 24 26 [thread 0] 557 41120.453 41216.402 38848.609 23138.22 24548.48 24 26 [thread 0] 582 41120.453 41216.316 38950.314 23138.27 24484.38 24 26 [thread 0] 607 41120.510 41216.278 39043.641 23138.29 24425.86 24 26 [thread 0] 632 41120.543 41216.326 39129.585 23138.27 24372.21 24 26 [thread 0] 657 41120.543 41216.402 39208.992 23138.22 24322.85 24 26 [thread 0] 682 41120.550 41216.240 39282.572 23138.31 24277.29 24 25 [thread 0] 707 41120.577 41216.316 39350.950 23138.27 24235.10 24 25 [thread 0] 732 41120.577 41216.478 39414.664 23138.18 24195.93 24 25 [thread 0] 757 41120.568 41216.240 39474.161 23138.31 24159.46 24 25 [thread 0] 782 41120.568 41216.326 39529.857 23138.27 24125.42 24 25 [thread 0] 807 41120.567 41216.316 39582.102 23138.27 24093.57 24 25 [thread 0] 832 41120.553 41216.478 39631.211 23138.18 24063.72 24 25 [thread 0] 857 41120.577 41216.240 39677.449 23138.31 24035.68 24 25 [thread 0] 882 41120.581 41216.402 39721.070 23138.22 24009.28 24 25 [thread 0] 907 41120.581 41216.316 39762.284 23138.27 23984.39 24 25 [thread 0] 932 41120.581 41216.326 39801.288 23138.27 23960.89 24 25 [thread 0] 957 41120.591 41216.316 39838.253 23138.27 23938.66 24 25 [thread 0] 982 41120.599 41216.316 39873.336 23138.27 23917.60 24 25 Final: 1000 41120.606 114493.397 41216.497 8329.51 23138.17 9 24 ```Also note that a DGX-1 doesn't have NVSwitch connecting all GPUs, so when scaling to all of them we expect to be limited by the Connect-X 4 bandwidth.
orliac
Hi there, I'm facing issue when trying to run the cudf_merge benchmark locally on a node that hosts 4 h100:
I can run the benchmark over any pair of GPUs with no issue:
python -m ucp.benchmarks.cudf_merge --devs 0,1 --chunk-size 200_000_000 --iter 10But it fails to run over the 4 devices:
python -m ucp.benchmarks.cudf_merge --devs 0,1,2,3 --chunk-size 200_000_000 --iter 10My environment:
Any idea?
Also, I'm surprised by the variability of the benchmark over the successive 10 iterations.
And finally, is it expected that the benchmark saturate the available bandwidth between the GPUs?