search

tinygrad / open-gpu-kernel-modules

NVIDIA Linux open GPU with P2P support
Other
772 stars 57 forks source link

Small BAR Size Support? #13

Open Qubitium opened 3 weeks ago

Qubitium commented 3 weeks ago

NVIDIA Open GPU Kernel Modules Version

550.90.07

Operating System and Version

Ubuntu 22.04

Kernel Release

6.8.9

Hardware: GPU

4090

Describe the bug

We have tested the modified kernel on two systems: 1x intel desktop (with full BAR=32GB), 1x amd server (without resizable_bar, BAR=256MB,512MB)

On the intel with full 32GB BAR size for the 2x4090, NCCL/P2P test is passing with modded driver.

However on the amd server platform where bios doesn't support resizable_bar, nvidia-smi is only showing 256MB and 512MB bar sizes for the 2x4090. On this amd server, even with the this modded nvidia driver, NCCL/P2P tests failed. The amd server also has lots of pcie devices so it may be running out of pcie map space to assign the large 32GB bars that 4090 support.

So my question is, is the current P2P+4090 code only working if BAR size >= full 4090 GPU vram size? Thank you!

puppetm4st3r commented 1 week ago

same question here =)

ex3ndr commented 2 days ago

What motherboard is AMD? I just updated bios on mine and it got Re-Size Bar support