Deal with heterogeneous nodes

simonsobs-uk / data-centre

This tracks the issues in the baseline design of the SO:UK Data Centre at Blackett

https://souk-data-centre.readthedocs.io

BSD 3-Clause "New" or "Revised" License

2 stars 1 forks source link

Deal with heterogeneous nodes #7

Open ickc opened 1 year ago

ickc commented 1 year ago

A meta-issue collecting issues with heterogeneous nodes first.

software deployment: AMD vs Intel, AVX-512 availability, no-mkl. etc.
- possibly look for ways to continue to use MKL while disguising AMD CPU as Intel's.
- dispatch software environment per node by detecting architecture on-the-fly. (Possibly make debugging harder.)
  - dispatch over x86-64-v3, x86-64-v4?
load-balancing per job. C.f. #6

ickc commented 9 months ago

As mentioned in #17, currently provided environment is compiled with AVX-512 and some newly added nodes from September is not capable of that. We need to update the documentation to add a constraint there for now.

ickc commented 7 months ago

See #36, see also commit 13e441e about upcoming plan to auto-dispatch. Push back to next release.