[GH200] Unexpected Low Host-to-Device Bandwidth

vitduck commented 2 months ago

Hi,

We observed an unexpected low host-to-device bandwidth on GH200 Superchip.

Specs
- GH200 (480GB LPDDR5X + 96GB HBM3)
- OS: Rocky Linux 9.3 (Blue Onyx)
- nvbandwith: v5.0

nvidia-smi topology:

        GPU0    NIC0    NIC1    CPU Affinity    NUMA Affinity   GPU NUMA ID
GPU0     X  SYS SYS 0-71    0       1
NIC0    SYS  X  PIX             
NIC1    SYS PIX  X

output: nvbandwidth-gh200.log
- SM is expected to give high bandwidth that CE in general since the latter is limited by DMA engine.
- Bandwidth anomaly:
  - Host-to-device: CE (346 GB/s) vs SM (342 GB/s)
  - From NVIDIA reference result[, there should be ~ 15% difference between them.
  - Is there some hardware/kernel settings that negatively affect the SM-variant memcpy ?
- Bandwidth asymmetry:
  - For GH200, H2D is approximately 15% faster than D2H, is this due to an intrinsic property of ATS ?
ref:
- https://docs.nvidia.com/gh200-superchip-benchmark-guide.pdf

Regards.

deepakcu commented 2 months ago

Are the GPU clocks locked? To what value? Can you attach the output of nvidia-smi -q?

vitduck commented 2 months ago

@deepakcu

I have attached here the output of lscpu and nvidia-smi -q per your requested. I believe everything is running at stock.

Below is addition information

Linux kernel with 64k page table

$ uname -a 
Linux gpu51 5.14.0-362.8.1.el9_3.aarch64+64k #1 SMP PREEMPT_DYNAMIC Thu Nov 9 05:07:41 UTC 2023 aarch64 aarch64 aarch64 GNU/Linux

NVIDIA driver: 550.54.14
Addition kernel module installed: nvidia_peermem, and gdrdrv

I also check dmesg immediately after running nvbandwidth and didn't observe and warning or error.

Thanks.

deepakcu commented 2 months ago

Can you repeat your test locking clocks at max (1980MHz)

sudo nvidia-smi --lock-gpu-clocks=1980,1980

NVIDIA / nvbandwidth

[GH200] Unexpected Low Host-to-Device Bandwidth #23