Closed dmagdavector closed 2 months ago
Hum, that would not work on our clusters. The content and the existence of
/sys/class/infiniband/mlx5_0/ports/1/link_layer
on the login nodes is quite variable... I found one cluster that has Ethernet in that file, two that have InfiniBand, and one for which that file does not exist on login nodes, while all of them have InfiniBand.
However, this function is only used to define RSNT_INTERCONNECT
if and only if it is not already defined. If you define RSNT_INTERCONNECT=Ethernet
before our profile scripts are sourced, it will honour the value that you have defined and not try to detect it.
To account for multiple port cards, and different generations, we probably would need to do something like
grep InfiniBand /sys/class/infiniband/*/ports/*/link_layer
to figure out if there is InfiniBand on the node.
In the
SitePackage.lua
file there is a check for a particular directory:But the presence of that directory—which indicates that a particular kernel module is loaded—does not necessarily mean that the network is using Infiniband. That module (
ib_core
) is loaded by the Mellanox driver regardless of what the link layer type is, even if the Mellanox NICs are Ethernet:NVidia's documentation suggests a different way to check the network type: