During the NHR course week a participant got an unexpected result when running pinthreads(:numa) inside of SLURM allocation that didn't comprise an entire compute node (128 cores) but only a fraction of it (32 cores). In this case, SLURM (rightfully) restricted the access to the first 32 cores. This led to the situation that pinthreads(:numa), while not showing any error or similar, was actually only pinning to the first two NUMA domains (corresponding to the first 32 cores).
Maybe this is fine. But we could also think about adding "SLURM awareness" in some form. For example, we could check SLURM environment variables to figure out A) if we are running within a SLURM allocation and B) what cores are actually accessible (we should be able to figure this out from the mask). Wrt. TP.jl we could then
leave things as they are but show a clear warning about this issue
in threadinfo() only show the accessible cores (maybe grey out the inaccessible ones?)
During the NHR course week a participant got an unexpected result when running
pinthreads(:numa)
inside of SLURM allocation that didn't comprise an entire compute node (128 cores) but only a fraction of it (32 cores). In this case, SLURM (rightfully) restricted the access to the first 32 cores. This led to the situation thatpinthreads(:numa)
, while not showing any error or similar, was actually only pinning to the first two NUMA domains (corresponding to the first 32 cores).Maybe this is fine. But we could also think about adding "SLURM awareness" in some form. For example, we could check SLURM environment variables to figure out A) if we are running within a SLURM allocation and B) what cores are actually accessible (we should be able to figure this out from the mask). Wrt. TP.jl we could then
threadinfo()
only show the accessible cores (maybe grey out the inaccessible ones?)