Basic documentation on architecture of Blackett

[ickc]

Running multi-nodes applications at Blackett would benefits from some knowledge on the architecture of Blackett. E.g.

1. (compute capability) load-balancing w.r.t. heterogeneous nodes
2. (network capability) how the nodes are connected and if there are bottle necks, say, communications between multiple nodes will be choking between 2 network switches

I think something like this would be a example to follow: Architecture - NERSC Documentation (but obviously don't need to be as detailed.)

I think we can help with documentations. If information is passed to us then we can compile a documentation like that. Information needed are probably something like:

1. configuration of each node, e.g. N nodes has n-socket "MODEL" CPU with X amount of RAM. A raw table of hardware information is probably enough for us to compile a summary table.
2. network topology, e.g. how they are connected, bandwidth, etc. This would sheds some light on how many nodes we can launch an MPI application until the network cannot cope up.

@rwf14f, what do you think about this? Thanks.

[ickc]

• Node availability on the current testbed according to @rwf14f:

  The nodes in the testbed are unused highmem nodes (bought for SKA) which we put into the testbed because our normal testbed nodes are down due to an aircon failure

  • 1 node with 40 SMT-cores
  • dedicated to the parallel universe
  • exclusive to us
  • 5 nodes with 16 SMT-cores
  • 1 out of 5 is exclusive to us
  • Max RAM to request: 33000 MB

We'll come back to this issue only after we mature away from testbed. Will mark as pending.

[ickc]

Our current testbed has access to:

hostname	CPU	sockets	cores	exclusive to parallel universe
wn1905340.in.tier2.hep.manchester.ac.uk	Intel(R) Xeon(R) Gold 5122 CPU @ 3.60GHz	2	4
wn5914090.in.tier2.hep.manchester.ac.uk	Intel(R) Xeon(R) Gold 5222 CPU @ 3.80GHz	2	4
wn5916090.in.tier2.hep.manchester.ac.uk	Intel(R) Xeon(R) Gold 5222 CPU @ 3.80GHz	2	4
wn5914340.in.tier2.hep.manchester.ac.uk	Intel(R) Xeon(R) Gold 5215L CPU @ 2.50GHz	2	10	True
wn5916340.in.tier2.hep.manchester.ac.uk	Intel(R) Xeon(R) Gold 5222 CPU @ 3.80GHz	2	4
wn5917090.in.tier2.hep.manchester.ac.uk	Intel(R) Xeon(R) Gold 5222 CPU @ 3.80GHz	2	4	True

[ickc]

More notes after today's meeting between SO:UK DC and Blackett people:

• The network topology is complicated, we hope to get the info available (e.g. how they are connected, link capacity, etc.) some time in the future.
• Due to power limitation of the building, the machines we buy would not land within the same network switch. This will present challenges in MPI jobs, partly also due to the limitation of how HTCondor can be used to constrain jobs.
• The testbed has some nodes in a far-away building (the production machines will not have this problem) so it would have worst latency.
• CVMFS is cached locally (i.e. on SSD), while CephFS (that the facility will soon be migrating to) is mounted over the network, with HDDs. I expect CVMFS (i.e. more like NERSC's global common with more limitations) is better for "massively" parallel applications while CephFS more flexible (i.e. more like NERSC's scratch space). See shifter-vs-filesystems · main · NERSC / python-benchmark · GitLab for example (NERSC used to have a documentation on the implications related to this but I can't find it right now.)

[ickc]

Closes via 2280193