Fat/long force optimizations

[weinbe2]

This PR constitutes a large refactor and optimization of the implementation of the fat/long force computation in QUDA. It introduces multiple different groups of optimizations:

Five- and seven-link terms

• Fully fuses the forward and backward seven-link term in the "rho" direction into the subsequent middle five-link term in the "nu" direction
• Fuses the middle five-link "nu" direction contribution into the side-link contribution in the next "nu" direction

Three-link and Lepage terms

• Fully fuses the middle three-link term and the middle and side Lepage terms (which enter in the force from the first part of the HISQ force chain rule)
• Fuses the middle three-link "mu" direction into the side-link contribution in the next "mu" direction

Ancillary detriments and benefits from fusion

• The above-noted fusions of middle-link force terms with the next side-link force terms requires additional "double buffer" fields
• However, this PR also removes the temporary Qmu field, which was ultimately a shifted version of the original U gauge link field, so the memory bloat from this PR is relatively minimal.
• The above fusion incidentally eliminates a few routines which required looping over every site in the depth-two extended field
• ...though it's worth noting that depth-two extended fields are still needed as intermediates in many cases, so while this reduces the total number of launched threads it does not lead to a commensurate reduction in the depth of the temporary fields

Gauge field compression

• Enables recon-13 for the input gauge field, which is applicable for both the "U" field in the first level of ASQTAD smearing and the "W" field in the second level of ASQTAD smearing (as part of the overall HISQ force)
• Extends the usage of recon-13 to the MILC interface, based on the existing environment variables for enabling recon-13 for outer solves

Testing and timing

• This PR does non-trivial cleanup of unit testing and timing for the HISQ force
• In particular, it splits the test into two separate components which emulate the first step (Lepage and long link) and second step (traditional ASQTAD smearing only) of the HISQ force

General clean-up of FLOPS/bytes counts

• This PR both streamlines and fixes some bugs in the FLOPS/bytes counts in the sub-components of the HISQ force

There is still some remaining clean-up to be done in this PR, none of which block opening this PR sooner as opposed to later:

• [x] Updating the FLOPS/bytes counts in the CPU unit tests (straightforward, just needs to be copied from the GPU counts)
• [x] Cleaning up (or eliminating?) the "low memory" code path, which removes some of the intermediate fields at the expense of removing some kernel fusion
• [x] clang-format

With regards to gauge compression, there is still outstanding work to enable recon-12 for the second step of the HISQ force chain rule because the U field is an SU(3) field (as opposed to the first step featuring the U(3) W field). Since this corresponds to a marginal gain in memory savings (relative to enabling recon-13) at the expense of a decent amount of coding headaches, we're going to punt this to a subsequent PR.

As a function of the problem size and the geometry of the decomposition, this can lead to a ~30+% performance boost in the computation of the HISQ force. This is due to the large amount of kernel fusion (and the corresponding cache reuse from it) as well as the introduction of gauge reconstruction for the U/W field and a reduction of the depth of the halo that needs to be explicitly looped over as part of the computation.

[maddyscientist]

~When building, I'm getting the following warning (CUDA 12.0 / GCC 11)~

/home/kate/github/quda-develop-old/tests/host_reference/hisq_force_reference.cpp: In instantiation of ‘su3_matrix* get_su3_matrix(int, su3_matrix*, int, int) [with su3_matrix = fsu3_matrix]’:
/home/kate/github/quda-develop-old/tests/host_reference/hisq_force_reference.cpp:109:37:   required from ‘void computeLinkOrderedOuterProduct(su3_vector*, su3_matrix*, size_t, int) [with su3_vector = fsu3_vector; su3_matrix = fsu3_matrix; size_t = long unsigned int]’
/home/kate/github/quda-develop-old/tests/host_reference/hisq_force_reference.cpp:118:35:   required from here
/home/kate/github/quda-develop-old/tests/host_reference/hisq_force_reference.cpp:87:63: warning: unused parameter ‘gauge_order’ [-Wunused-parameter]
   87 | template <typename su3_matrix> su3_matrix *get_su3_matrix(int gauge_order, su3_matrix *p, int idx, int dir)

Edit: this was caused by me using a stale local copy of the branch. Error is not present in HEAD.

[maddyscientist]

Something I just noticed from testing, is that the hisq_paths_force_test test will wrongly complain of failing if --verify false is pass. The correctness check should not be applied in this case.

[weinbe2]

Something I just noticed from testing, is that the hisq_paths_force_test test will wrongly complain of failing if --verify false is pass. The correctness check should not be applied in this case.

https://github.com/lattice/quda/pull/1367/commits/9e9845e5827a35ac61349c89a718f0ad192014e7

[maddyscientist]

@mathiaswagner are you wanting to review this before we merge?

[mathiaswagner]

I guess this has seen enough testing and I am not sure I'll have cycles this week so feel free to go ahead with the merge.