Feature/gaugefield unity

[maddyscientist]

This PR's primary purpose is to unify the cpuGaugeField and cudaGaugeField classes, along with an assortment of other cleanups.

• Removed cpuGaugeField and cudaGaugeField classes, we have now only GaugeField
  • Similar to ColorSpinorField the communiciation routines for host and device are still separate within the GaugeField class, unification of these is deferred until later
  • Added move and copy constructors and assignment operators for GaugeField to facilitate significant cleanup
  • Raw field pointer is now obtained using the data() method. This is to align with STL design. Though here we also have an optional template parameter that can be used to perform a cast to the desired type. (Also applied to ColorSpinorField and CloverField)
• Introduced a smart pointer class quda_ptr, which provides heterogenous memory allocation
  • Deployed in GaugeField, ColorSpinorField and CloverField
• Introduced qudaMemcpy and qudaMemset functions that accept a quda_ptr
  • This allows for unified calls to memcpy and memset without needing to worry about the location of the field
• Introduced the concept of a profile stack, with pushProfile and popProfile functions, together with getProfile for getting a reference to the current top of stack profile.
  • This facilitates a significant cleanup of the profiling, we can now profile in arbitrary functions without needing to pass the profile to it explicitly
• Profiles now no longer complain about being double started: we now have a ref counter to allow things to just work (as long as the number of starts and stops match)
• Significant cleanup of the interface for the non-solver functions to utilize the technology introduced as part of this PR
• Jitify
  • Updated the version of Jitify included with QUDA
  • Some fixes for issues that creeped in (e.g., moved some function definitions out of headers to cpp files to avoid the run-time compiler from parsing them)

Outstanding items

• [x] Add quda_ptr for ROCm
• [x] Apply clang-format

[weinbe2]

I've reviewed every file not called interface_quda.cpp and I'm a big fan of all of the cleanup. I'll need to go through interface_quda.cpp with a fine-tipped comb before I feel good about it!

[weinbe2]

Why was timing QUDA_PROFILE_INIT removed in a handful of interface functions? (I know I mentioned this in one review comment, but it seems to appear in multiple places.)

[weinbe2]

I've completed my visual review of interface_quda.cpp and left a handful of comments/questions, most of which probably only beg clarification. I'll start doing some explicit tests soon.

[maddyscientist]

@weinbe2 Ok, all your comments should be addressed now.

[maddyscientist]

This has now been tested with MILC RHMC, and things are looking good.

[weinbe2]

Tentative approve, I noticed a few cosmetic things, and this needs a fresh merge of develop + clang-format. After that it's good to go!

[maddyscientist]

@SaltyChiang I have merged in your changes into this branch, that will shortly be merged into develop.

One line I have not included is this one:

gauge[d] = (param.gauge) ? ((void **)param.gauge)[d] : nullptr;

as this line doesn't make sense to me. If it's a reference to a pre-existing field, why should the value be a nullptr? Can you explain why this change is needed?

Thanks

[SaltyChiang]

Hi @maddyscientist, there are some if statements about the input gauge link, like https://github.com/lattice/quda/blob/6e90b49a2d12891f2f68e35b7cfa7defbd506526/lib/interface_quda.cpp#L3873 and https://github.com/lattice/quda/blob/6e90b49a2d12891f2f68e35b7cfa7defbd506526/lib/interface_quda.cpp#L3903. I believe the logic here is originally used for QUDA_MILC_GAUGE_ORDER, but this will fail if QUDA_QDP_GAUGE_ORDER is used here. For example, if the ulink is nullptr here, the unitarized fat7 link will not be calculated, which is right; but cpuUnitarizedLink defined in https://github.com/lattice/quda/blob/6e90b49a2d12891f2f68e35b7cfa7defbd506526/lib/interface_quda.cpp#L3847 fails now as you cannot get ((void **)nullptr)[d]. A null QUDA_QDP_GAUGE_ORDER field should be something like void *ulink[4] = {nullptr, nullptr, nullptr, nullptr} but now the if (ulink) will return true which we do not expect.

So I made all four reference fields nullptr when the input value itself is nullptr and I think this will not affect the existing logic.

[maddyscientist]

@SaltyChiang thanks for the explanation. I have fixed my branch accordingly (https://github.com/lattice/quda/pull/1384/commits/4c308f6f72a2239e97f3423137459e049dc56b03), so it should work for your usecase. If you could test this branch, that would be appreciated, thanks.

[SaltyChiang]

Hi @maddyscientist, very sorry for the late feedback. I think it should be !unitarizedLink.StaggeredPhaseApplied() && param->staggered_phase_applied here. https://github.com/lattice/quda/blob/0c016af709ad6a3c249540fdad809c8d18abc226/lib/interface_quda.cpp#L3748-L3749

Edit: I believe gauge_param.location should be changed to QUDA_CUDA_FIELD_LOCATION here. https://github.com/lattice/quda/blob/0c016af709ad6a3c249540fdad809c8d18abc226/lib/interface_quda.cpp#L765-L774

[maddyscientist]

@SaltyChiang thanks for pointing put my bug. I've pushed a fix to my next PR that will be merged (#1393). This is a fairly simple PR, so will likely be merged soon.

[SaltyChiang]

@maddyscientist thanks for the fix. You might miss another issue that will happen while saving the smeared gauge through saveGaugeQuda(). I updated the comment above and I repeat it here.

Edit: I believe gauge_param.location should be changed to QUDA_CUDA_FIELD_LOCATION here.

https://github.com/lattice/quda/blob/0c016af709ad6a3c249540fdad809c8d18abc226/lib/interface_quda.cpp#L765-L774

[maddyscientist]

Thanks again @SaltyChiang. Fix is here 44c41006158e493517452210a7a1d1e55412d63f (in #1393).