Feature/multi rhs

[maddyscientist]

This PR is focussed on framework evolution to get us ready for multi-RHS workflows

• Creation new vector_ref class which replaces the prior use of std::vector<ColorSpinorField_ref>
• Added multi-RHS interface to Dirac and DiracMatrix classes
• Move all regular Krylov solvers to use std::vector<ColorSpinorField> instead of std::vector<ColorSpinorField*>
• Significant cleanup of eigensolvers to use new multi-RHS framework
  • Block Lanczos eigen solver is now multi-RHS ready
• Created new class FieldTmp which provides a cache for temporary objects. Deployed in the Dirac operators to negate the need to allocate and maintain explicit temporary vectors
• Add max and max_deviation reduce kernels, useful for correctness testing
• Added new test, io_test, for testing both gauge and color-spinor storage when QIO is enabled
• Cleanup of the blas functions: blas functions are now const correct, and doxygen is now added for all functions.
• Add deflation to BiCGStab
• Fix PCG solver which was broken when no preconditioner is present
• Fix various issues with ARPACK (lack of SVD support, correct e-vector extraction with non-Hermitian matrices)
• Various other misc fixes and cleanup

Update

Some additional fixes added during review:

• Clean up PCG and fix some corner cases
• Tweak max_res_increase default parameter to ensure more reliable convergence in ctest
• Add PCG and Schwarz preconditioned solvers to gtest list
• Fix compiler warnings when QUDA_INVERFACE_NVTX=ON
• Fix initialization error when using split grid
• Fix issue with BiCGStab

[mathiaswagner]

There are still a couple of saveTuneCache in the interface. I guess they can all be deleted?

[maddyscientist]

There are still a couple of saveTuneCache in the interface. I guess they can all be deleted?

I left those intentionally, but happy to discuss their removal. Those were left to ensure that when the solver finishes, the tunecache is dumped which I naively thought was desirable. Given it would be dumped anyway when endQuda is called, I'm happy to delete for code cleanup.

Thoughts?

[maddyscientist]

Merging this now. I've verified manually:

• split grid solvers are working as expected
• multigrid file i/o

I've updated the description with a few late additions to this PR that were added during review.