HPMG: Compute finest level directly on the field

AlexanderSinn commented 1 month ago

Quite a bit of time in HPMG is spent copying phi between the HPMG array and field array, as the coarsest level has the largest resolution. This PR changes the vcycle so that the finest level is partly directly calculated on the input field array, instead of computing the residual first and then starting at zero. The gsrb_4_residual function is now also used to compute the residual to determine if there should be another vcycle instead of compute_residual, this way gsrb_4_residual can be omitted at the start of the vcycle. This will result in four extra gsrb iterations being done on the finest level, slightly changing CI test results. Additionally, this PR includes some minor cleaning and other smaller HPMG performance improvements.

When using tighter tolerances, the PR and development MG solvers are closer to each other than to the checksum with the lower tolerance (the changes made by this PR are within the tolerance of the MG solver). See https://github.com/Hi-PACE/hipace/actions/runs/10078089809/job/27862184203?pr=1137 https://github.com/Hi-PACE/hipace/actions/runs/10077289597/job/27859470194?pr=1139

[ ] Small enough (< few 100s of lines), otherwise it should probably be split into smaller PRs
[ ] Tested (describe the tests in the PR description)
[ ] Runs on GPU (basic: the code compiles and run well with the new module)
[ ] Contains an automated test (checksum and/or comparison with theory)
[ ] Documented: all elements (classes and their members, functions, namespaces, etc.) are documented
[ ] Constified (All that can be const is const)
[ ] Code is clean (no unwanted comments, )
[ ] Style and code conventions are respected at the bottom of https://github.com/Hi-PACE/hipace
[ ] Proper label and GitHub project, if applicable

AlexanderSinn commented 1 month ago

On A100 with 4095^2 cells, this provides a 22% performance improvement. @WeiqunZhang could there be accuracy issues from doing gsrb directly on the sol array instead of taking the residual first?

WeiqunZhang commented 1 month ago

If it converges, I guess it's fine.

AlexanderSinn commented 1 month ago

I still need to check if setting the Bx By boundary condition with MR still works the same or if it has to be different now. ... works now.

Hi-PACE / hipace

HPMG: Compute finest level directly on the field #1137