Open cyclinder opened 1 month ago
that is a kubernetes feature. you can configure device manager and check the topology type
https://kubernetes.io/docs/tasks/administer-cluster/topology-manager/#policy-single-numa-node
Thanks for your reply, I think even if GPU and Nic are in the same NUMA nodes, they may still cross the PCIe bridge, as shown in the figure above, GPU0 and mlx5_3, so in this case, we cannot enable GPU Direct RDMA. The same NUMA nodes may be a large distance, we may need a smaller distance.
currently there is no solution that im aware of which takes into account PCIe topology.
DRA (Dynamic Resource Allocation) aims to solve that, but there is still a way to go....
If the gpu and nic are on the same PCIe bridge or their topology distance is at least
PHB
, then communication between them can be accelerated by enabling GPU Direct RDMA.