Following up on 385, 411 -- computes time per grid_point per equation per

AiredaleDev commented 1 month ago

Description

Fixes #299 #385, follows up on #411

Type of change

[x] Something else

Scope

[x] This PR comprises a set of related changes with a common goal

How Has This Been Tested?

[] Local run of ./mfc.sh test

Test Configuration:

What computers and compilers did you use to test this:

System76 Pangolin (2024 Model, CPU only)

AMD Ryzen 7840U: 8 Cores, 16 Threads 32 GB of LPDDR5 @ 6400 MHz

Checklist

[x] I have added comments for the new code
[x] I have made corresponding changes to the documentation (docs/)
[x] I ran ./mfc.sh format before committing my code
[ ] New and existing tests pass locally with my changes, including with GPU capability enabled (both NVIDIA hardware with NVHPC compilers and AMD hardware with CRAY compilers) and disabled
[x] This PR does not introduce any repeated code (it follows the DRY principle)
[x] I cannot think of a way to condense this code and reduce any introduced additional line count

If your code changes any code source files (anything in `src/simulation`)

To make sure the code is performing as expected on GPU devices, I have:

[ ] Checked that the code compiles using NVHPC compilers
[ ] Checked that the code compiles using CRAY compilers
[ ] Ran the code on either V100, A100, or H100 GPUs and ensured the new feature performed as expected (the GPU results match the CPU results)
[ ] Ran the code on MI200+ GPUs and ensure the new features performed as expected (the GPU results match the CPU results)
[ ] Enclosed the new feature via nvtx ranges so that they can be identified in profiles
[ ] Ran a Nsight Systems profile using ./mfc.sh run XXXX --gpu -t simulation --nsys, and have attached the output file (.nsys-rep) and plain text results to this PR
[ ] Ran an Omniperf profile using ./mfc.sh run XXXX --gpu -t simulation --omniperf, and have attached the output file and plain text results to this PR.
[ ] Ran my code using various numbers of different GPUs (1, 2, and 8, for example) in parallel and made sure that the results scale similarly to what happens if you run without the new code/feature

NOTE: While I changed the caption in the docs, I have yet to collect and document new performance metrics! Please let me know if anything else looks amiss.

sbryngelson commented 1 month ago

Did you try this locally? For me I get the following on simulation (running on 4 ranks, though 1 rank has same problem)

 [100%]  Time step      992 of 1001 @ t_step = 991
 [100%]  Time step      993 of 1001 @ t_step = 992
 [100%]  Time step      994 of 1001 @ t_step = 993
 [100%]  Time step      995 of 1001 @ t_step = 994
 [100%]  Time step      996 of 1001 @ t_step = 995
 [100%]  Time step      997 of 1001 @ t_step = 996
 [100%]  Time step      998 of 1001 @ t_step = 997
 [100%]  Time step      999 of 1001 @ t_step = 998
 [100%]  Time step     1000 of 1001 @ t_step = 999
 Final Time                       NaN ns/gp/eqn/rhs

and on preprocess I actually still get the old metric

+ mpirun -np 4 /Users/spencer/Downloads/MFC/build/install/b61184720f/bin/pre_process
 Pre-processing a 200x0x0 case on 4 rank(s)
 Processing patch           1
 Processing patch           2
 Final Time   2.0140000000000019E-003

sbryngelson commented 1 month ago

There are now two separate people contributing to two PRs on the same topic, can we please converge?

MFlowCode / MFC