Fix for issue #395

[okBrian]

Description

Fixed the clarity issue for the CI benchmark comparison message when comparing a PR and master.

Example:

• Before

  Comparing Bencharks: master/bench-cpu.yaml is x times slower than pr/bench-cpu.yaml.

• Expected After

  Comparing Benchmarks:
  1.5x indicates pr/bench-cpu.yaml is 1.5-times as fast as master/bench-cpu.yaml (so pr/bench-cpu.yaml is faster than master/bench-cpu.yaml).
  0.5x indicates pr/bench-cpu.yaml is 0.5-times as fast as master/bench-cpu.yaml (so pr/bench-cpu.yaml is slower than master/bench-cpu.yaml).

Fixes #395

Type of change

• [x] Bug fix (non-breaking change which fixes an issue)

Scope

• [x] This PR comprises a set of related changes with a common goal

How Has This Been Tested?

Not sure how to run the diff benchmark comparison without the github runner

Test Configuration:

Compiled on a

Ubuntu 22.04.4 using Docker Intel i7-1165G7

Checklist

• [ ] I have added comments for the new code
• [ ] I added Doxygen docstrings to the new code
• [ ] I have made corresponding changes to the documentation (docs/)
• [ ] I have added regression tests to the test suite so that people can verify in the future that the feature is behaving as expected
• [ ] I have added example cases in examples/ that demonstrate my new feature performing as expected. They run to completion and demonstrate "interesting physics"
• [x] I ran ./mfc.sh format before committing my code
• [ ] New and existing tests pass locally with my changes, including with GPU capability enabled (both NVIDIA hardware with NVHPC compilers and AMD hardware with CRAY compilers) and disabled
• [x] This PR does not introduce any repeated code (it follows the DRY principle)
• [x] I cannot think of a way to condense this code and reduce any introduced additional line count

If your code changes any code source files (anything in src/simulation)

To make sure the code is performing as expected on GPU devices, I have:

• [ ] Checked that the code compiles using NVHPC compilers
• [ ] Checked that the code compiles using CRAY compilers
• [ ] Ran the code on either V100, A100, or H100 GPUs and ensured the new feature performed as expected (the GPU results match the CPU results)
• [ ] Ran the code on MI200+ GPUs and ensure the new features performed as expected (the GPU results match the CPU results)
• [ ] Enclosed the new feature via nvtx ranges so that they can be identified in profiles
• [ ] Ran a Nsight Systems profile using ./mfc.sh run XXXX --gpu -t simulation --nsys, and have attached the output file (.nsys-rep) and plain text results to this PR
• [ ] Ran an Omniperf profile using ./mfc.sh run XXXX --gpu -t simulation --omniperf, and have attached the output file and plain text results to this PR.
• [ ] Ran my code using various numbers of different GPUs (1, 2, and 8, for example) in parallel and made sure that the results scale similarly to what happens if you run without the new code/feature

[sbryngelson]

@henryleberre can confirm but I think this might be backwards. lhs -> master and rhs -> PR.

If the variable speedup below is > 1 then master is slower than then PR.

If the variable speedup below is < 1 then PR is slower than master.

At least that's my read of the code...

Code:

https://github.com/MFlowCode/MFC/blob/f23fceb103412864a9846fd75fdc13261699e147/toolchain/mfc/args.py#L135-L138

https://github.com/MFlowCode/MFC/blob/f23fceb103412864a9846fd75fdc13261699e147/.github/workflows/bench.yml#L38

https://github.com/MFlowCode/MFC/blob/64d9677e58f26a157e207afbdcc705e5cab90c7f/toolchain/mfc/bench.py#L108-L122

and your code says:

cons.print(f"[bold]\t1.5x indicates [magenta]{os.path.relpath(ARG('rhs'))}[/magenta] is 1.5-times as fast as [magenta]{os.path.relpath(ARG('lhs'))}[/magenta] (so [magenta]{os.path.relpath(ARG('rhs'))}[/magenta] is faster than [magenta]{os.path.relpath(ARG('lhs'))}[/magenta]).[/bold]")

[henryleberre]

Let's consider an example:

$ ./mfc.sh bench_diff lhs rhs

where lhs and rhs are YAML files. By the way, "lhs" and "rhs" just stand for "left-hand side" and "right-hand side". We could maybe opt for better names.

Say that for a given example case, you have:

     simulation
lhs:    1.0s
rhs:    0.5s 

So, $\frac{\text{lhs}}{\text{rhs}}=\frac{1.0}{0.5}=2$. So $\frac{\text{lhs}}{\text{rhs}}$ is the speedup going from lhs to rhs. In other words, lhs is what you are comparing rhs to. This implies that the text I had written and the text @okBrian wrote are both correct, although mine was worded in a way that might be confusing. It was like "LHS is X times SLOWER than RHS" rather than "RHS is X times FASTER than LHS" (@okBrian's version).

As a sidenote, there are a lot of cons.write calls that follow each other. These can be joined to be more readable like:

cons.write(f"""\
Line 1
Line 2
""")

[sbryngelson]

There should just be a line of text above the results that says "numbers < 1 indicate the PR is faster than master, > 1 indicate PR is slower" (if that is actually the case)

[sbryngelson]

PR merges are on hold until benchmark CI is working (issue #419)

[henryleberre]

@okBrian Why did you revert with 1e30c18 ?

[okBrian]

The past few tests have either failed or have the wrong output so I'm just making sure that's not the issue.

[sbryngelson]

@okBrian please sync your branch with master, which I just merged to (that's why your current CI tests are failing)

[sbryngelson]

CI benchmarking is failing and it seems related to this PR

[sbryngelson]

This might need another merge with master