GPU-Aware MPI on OLCF Frontier and Combined weak- & strong-scaling case

henryleberre commented 1 month ago

Description

Adds support for GPU-Aware MPI on OLCF Frontier. Closes #405.
Adds a combined and configurable weak- & strong-scaling case.
Adds scripts used to submit scaling runs on Frontier and export results.
Modifies case argument parsing to support non-positional arguments, too.
- Required the removal of the -f (--flags) option that became obsolete with the addition of template files for running cases and submitting jobs.
- The new format is `./mfc.sh run <path/to/case.py> --
- To be clear, "non-positional arguments" are arguments that start with a "-". They are called non-positional because they are identified by name, not by which argument number they are (their position in the list).

Type of change

[x] New feature (non-breaking change which adds functionality)
[x] Modifies existing behavior

Scope

[x] This PR comprises a set of related changes with a common goal

How Has This Been Tested?

Please describe the tests that you ran to verify your changes. Provide instructions so we can reproduce. Please also list any relevant details for your test configuration

[x] Ran strong scaling experiments on OLCF Frontier with and without cu_mpi.
[x] GitHub CI

sbryngelson commented 1 month ago

Very good. I think we need not hold .csv data in version control, though everything else looks great on a quick look. Several runners are failing but I have not investigated that either.

sbryngelson commented 1 month ago

Also, I think we need to rename cu_mpi $\to$ rdma_mpi or gpu_mpi or gpudirect or something (since cu was for cuda)

henryleberre commented 1 month ago

Thanks @sbryngelson. The Frontier runner is failing because of a fix for batch execution. I'll get it working with interactive execution, too.

henryleberre commented 1 month ago

I think I fixed all the issues.

sbryngelson commented 1 month ago

Seems like something isn't quite right. Also you can use the existing plots if you like they are fine, I am just tweaking them a bit for other purposes.

sbryngelson commented 1 month ago

I made this figure but you don't have to use it for anything strong_scaling_frontier.pdf strong_scaling_frontier.pdf

MFlowCode / MFC