Improvements to HQ residual solves in CG

[weinbe2]

This PR provides an algorithmic solution to convergence issues with HQ residual solves noted in #1376 . In brief, instead of having one unified code path for "regular" L2-norm solves and HQ residual solves, this PR creates separate code path for solves requesting a HQ residual.

This new code path contained in a function hqsolve includes improved logic for tracking reliable updates based on the L2 residual as well as the HQ residual. In the interest of simplicity, this separate code path does not feature all of the optimizations in the traditional CG code. Since HQ solves typically take O(100) iterations, as opposed to thousands to tens of thousands of iterations for light quark solves, this seemed like an acceptable decision. Of note, this new code path still supports mixed precision.

The two key improvements in hqsolve are as follows:

• Updating the trigger for a HQ reliable update to be based on a drop in the HQ residual relative to the previous update as opposed to drops in the L2 residual
• Instead of always waiting for "L2 breakdown" before switching to tracking HQ reliable updates, the hqsolve logic now only tracks HQ reliable update conditions if there is no L2 convergence requirement (i.e., tol == 0).

In the case of my own reproducers, the first improvement addresses the issues @detar noted in #1376 . The second improvement leads to a great reduction in the number of iterations required for a HQ solve (much more in line with what is seen in MILC's CPU implementation) with the benefit of retaining mixed precision. This is due to removing the requirement of L2 breakdown before switching to HQ reliable updates.

Outstanding tasks:

• [x] Verification by @detar
• [x] Improved documentation
• [x] Possibly removing the HQ logic from the regular CG solve?
• [x] clang-format

[weinbe2]

Update: @detar is doing a set of thorough correctness checks, thank you!!

[weinbe2]

@detar confirmed offline that this PR addresses the issues he reported in #1376 , so I've removed the "draft" status of this PR and done some last bits of tidy-up.

@maddyscientist and/or @mathiaswagner can you give this PR a review/test run and let me know what you think? Barring any showstoppers, I'd like to get this in as-is sooner as opposed to later.

[weinbe2]

Follow-up work is being tracked here: https://github.com/lattice/quda/issues/1389

[weinbe2]

This passed a ctest -V, so once the latest round of Jenkins passes I'll hit merge