aprokop
commented
7 years ago Per @jjellio :
My experience with Cray machines has been:
1) Restrict jobs so that they span as few switches as possible. For slurm, this is via the --switches=N@
To figure out the number of switches (N), you need to understand the machine's layout. For Cori, two cabinets share a switch, so you need to compute the number of nodes per cabinet, times 2, and the minimum number of switches is then the ceiling of your desired node count by that magic number. E.g., on Cori this is 1 switch per 392 nodes.
2) Compile with hugepage support. You can use which ever pagesize you like, and unless your code depends in some way on the page size (hopefully it doesn't), you can then use whichever pagesize you want at runtime. (But you must compile with Hugepages, this modifies the link line, as well as sets important system variables that define the page size)
2.b) When you run codes compiled with a hugepage support, and you wish to use hugepages, you need to load that module before running. (This is when you can choose a pagesize as well). On Cray systems, the module system sets up environment variables that control the hugepage library that was linked. This is odd, since you now need to load a module from build time at runtime.
3) Make sure the KNL system admins have installed Intel's zonesort kernel module. If they have not (or will not), you should reboot the node before use. KNL performance in Cache mode can degrade over time due to ordering (and availability) of free memory pages. Zonesort compacts and sorts the list of free pages so that future memory allocations will not cause unnecessary HBM cache conflicts.
Rebooting may be the preferred ticket though. If Hugepages are used, then available memory can become fragmented to the point that the OS is unable to obtain a Hugepage, in this case the OS defaults to providing 4KB pages again. When this happens, you then observe performance variations between nodes that are use all hugepages and those that are using on a few hugepages + 4KB pages. This is easy to observe by using the largest hugepage sizes (e.g., 512MB)
4) Check the uptime of the system. If the node has been up for days, I would reboot it. It is infuriating that KNL is so fragile w.r.t. the system state, but my best runtimes are usually right after the system was down for maintenance (I believe that is due to all nodes in my allocation being freshly booted)
That said, rebooting is expensive. Most systems charge your allocation for reboot time, and it can take around 20mins for a node to fully power cycle. This is also risky, because the node may not come back. (power cycling seems to be hard the systems)
5) Pin your threads to cores or to hardware threads. I use OMP_PLACES=threads
I am actively looking into ways to control MPI progress threads. You really do not want any thread active waiting on a KNL core that has other threads bound to it. You are pretty much guaranteed to observe around a 50% performance hit, because the active wait will force the core to share those resources with all threads contending.
6) Don't run on all 68 cores (if on the 68 core variant), leave a few cores free for the OS. (slurm provides this via --core_spec=N, where N is the number of cores to reserve). There is atleast one paper that explored this. The recommendation was leave 2 or 4 free. (leaving 4 free makes the node much easier to use, since you will have 64 cores)
7) (Trilinos specific), I've seen very nice results with 1 task on a KNL node. This results in a SerialComm, and that seems to be a very good thing. E.g., try to use repartitioning that will ultimately pack a level unto a single node, and MueLu's level logic should be using SerialComms if only one task is around. My data to support this is from Tpetra's SpMV. Regardless this seems like a good design decision anyway.
8) You need to do all of the above, all the time. Zonesort, hugepages, and node uptime all work together to make the system more stable.
sayerhs
jjellio
jhux2
ibaned
github-actions[bot]
I am opening this issue to record best practices for achieving good performance on the Cori KNL partition. @jjellio Could I ask you to document your recommendations in this ticket? I've had a number of questions regarding this. Thanks!
@sayerhs @aprokop @alanw0 @spdomin