Add command: `benchpark system init`

scheibelp commented 2 months ago

Closes https://github.com/LLNL/benchpark/pull/110

(note: ~500 of the +/- of this diff is from moving bin/benchpark to lib/main.py - I recommend collapsing these to make the diff more-readable)

Add a new command benchpark system init, that can generate a system config (a collection of files in a directory) that you can then pass to benchpark setup like:

benchpark setup saxpy/openmp <the-directory-created-by-benchpark-system-init> workspace/

Example 1:

./bin/benchpark system init --basedir=test-system-create aws instance_type=c4.xlarge (old) ~./bin/benchpark system create --basedir=test-system-create --type=aws --set instance-type=c4.xlarge~

Creates a directory in test-system-create, and writes a variables.yaml file to it:

variables:
  scheduler: "mpi"
  sys_cores_per_node: "4"
  sys_mem_per_node: 7.5
  ...

This doesn't tie in with boto yet, so doesn't provide a means to generate a description for all instance types. IMO that would be better to handle in a separate PR.

Example 2:

./bin/benchpark system init --basedir=test-system-create tioga rocm=551 compiler=cce ~gtl (old) ~./bin/benchpark system create --basedir=test-system-create --type=tioga~

will assemble a packages.yaml config such that:

there is only one set of ROCm packages (either all 5.4.3, or all 5.5.1, but not a mix of them)
the MPI external chosen matches the compiler choice

(Update August 3rd: done) ~This doesn't yet generate a variables.yaml or a compiler config, so the result is incomplete.~

TODOs

[x] --basedir argument can generate an entirely new system config directory, but benchpark setup can't actually use it yet
[x] This should accommodate the possibility of packages/compilersconfig
- As of 296e0e1, this provides a means to assemble a set of individual package configs into a single config
[x] ~This has a --from option to allow taking an existing config, but doesn't support it yet. I think it would be useful to combine this option with --set to override settings from that file~
- (update August 6th: this is no longer relevant: you can now create system classes that fully replicate an existing directory in configs/)

scheibelp commented 1 week ago

@becker33 I have renamed benchpark system create to benchpark system init and

all comments that resolved trivially I took the liberty of collapsing
If I responded but made no change (e.g. I liked your suggestion but wanted to punt it) I put "not addressed yet" at the front
Other comments that I think I resolved but involved some discretion, I responded to but did not collapse

Also I don't think I mentioned it yet, but the style checker was giving me trouble trying to block this on style issues in the system/experiment repos. I had some trouble getting it to ignore those, so I just fixed those issues for now. IMO we shouldn't apply the same stringency to system/experiment definitions as we do to the core (but if you disagree, then this PR is in the right place).

alecbcs commented 1 week ago

@scheibelp can we move lib/benchpark/system_cmd.py -> lib/benchpark/cmd/system.py ?

scheibelp commented 1 week ago

@becker33 SpecTemplate is renamed @alecbcs system_cmd was moved

The only remaining comment is https://github.com/LLNL/benchpark/pull/298#discussion_r1722342188, which I'd like to address in a different PR.

LLNL / benchpark