tm/tmclover mu=0.0 coarse grid deflation -> reuse for any mu value?

[kostrzewa]

For twisted mass (or tmclover), would it be possible to generate the vectors for coarse grid deflation once with mu=0.0 (or the smallest mass to be used in a particular run) and then re-use these for any mu value? I understand that the coarse operator must (should?) be updated, but why exactly should the unphysical singular eigenvectors change (substantially) as mu is increased? The true eigenvalues of the operator change, of course, but those are only marginally relevant, right?

The reason for my question is that I'm attempting to run a set of combined inversions and contractions which involve sign changes ("up-type" to "down-type" and back), as well as three different strange quark masses (with their own sign changes). In the tmLQCD QUDA interface, I trigger a setup update every time the mu value changes by some threshold, and each of the changes for this particular project triggers the update.

With the usual MG setup (using a coarse mu factor), these updates are a negligible contribution to the total time (the computation is also mostly organised in such a way as to reuse a given MG setup for the longest time possible). However, when coarse grid deflation is used, running the EigSolver at each call of updateMultigridQuda takes a while, destroying all of the speedup...

I tried simply avoiding updateMultigridQuda in the deflated run (by setting a very large threshold), but I get an obscure issue about mismatched precisions which, however, is probably due to some bug in our QUDA interface. (it happens whether coarse grid deflation is in use or not)

Long story short, could the eigenvectors from the "lightest" mu be used without rerunning the eigensolver?

[cpviolator]

@kostrzewa Absolutely. One can save and load eigenvectors any time the eigensolver is called. One simply needs to pass the correct string to the solver. For the regular eigensolver, one would use

--eig-save-vec <string>
--eig-load-vec <string>

For the MG routines, you use

--mg-load-vec <level> <string>
--mg-save-vec <level> <string>

where in your case you pass <level> as 2, the bottom level. This will load a vector space of size nConv, compute the eigenvalues for that space, and load them into the output arrays as if the eigenspace had been computed as usual. Let me know if you encounter any difficulty on this. I'm happy to polish the routines if they are difficult or cumbersome for production use.

[kostrzewa]

I see, but is there no way to simply avoid updateMultigridQuda performing the EigSolver step again?

Let me try to understand what you suggest: I start my inversions with the lightest mu that I have (or just set mu_factor = 0.0 on the coarsest level) and set mg_eig_param.vec_outfile, such that they are written to disk at the end.

When I switch quark masses, I need to run updateMultigridQuda, do I understand correctly that if I set mg_eig_param.vec_infile, the "EigSolver step" will just be a loading of the previously saved eigenvectors?

While this would be of help, of course, the difficulty (for us) with setting vec_infile and vec_outfile in our production code is that QUDA relies on QIO for this, which in turn relies on QMP. The way we use QUDA from tmLQCD relies on QUDA_MPI being set (fundamentally of course we could also compile against QMP, we do this for interfacing with QPhiX, but it would require some work to get everything up and running).

Can't I just tell the EigSolver to keep the vectors that it currently has in memory? The MG setup is retained through the lifetime of the application that I described in https://github.com/lattice/quda/issues/891#issue-489461499 above. Would it perhaps be sufficient to set mg_eig_param.require_convergence to QUDA_BOOLEAN_NO before calling updateMultigridQuda?

[cpviolator]

I think see the issue you're having. Let me reiterate to ensure we are both on the same page:

1. You construct an MG object that constructs a deflation space, (say DS)
2. Using the same gauge field, you switch quark mass (and presumably mu) and update the MG object so that it is properly configured for the new quark field.
3. You wish to use the same DS for each update, but updateMultigridQuda has no functionality to do this, outside of loading the vector space from IO.

If the above is correct, some sort of runtime option to transfer the DS from one MG object to another would fix this. Would that help with this issue?

[kostrzewa]

That's exactly right, yes. The MG object in question is not destroyed, simply updated using the C interface updateMultigridQuda function with suitably updated quark mass parameters.

[cpviolator]

OK, this seems fairly doable. I will write a subroutine that will detect if a deflation space is to be transferred, adjust the MG params to stop the new object from computing a new space, transfer the deflation space to the new object, then restore the MG params.

I'll ping you when its ready for testing.

[kostrzewa]

Awesome!

[cpviolator]

@maddyscientist WRT the above, should this go into develop or release 1.0?

[maddyscientist]

Definitely for develop.

[cpviolator]

Just inspecting the code path for this and making notes.

The deflation space is created and owned by the bottom level solver. So the level 2 (CA-)GCR is the instance that creates/loads the vectors, and they are discarded as soon as soon as the parent MG object is destroyed, as the parent MG destroys the bottom solver.

Provided the parent MG object persists, this feature can be implemented with boolean switches. One will need to recompute eigen/singular values, but that will be a minor overhead.

[kostrzewa]

Just inspecting the code path for this and making notes.

The deflation space is created and owned by the bottom level solver. So the level 2 (CA-)GCR is the instance that creates/loads the vectors, and they are discarded as soon as soon as the parent MG object is destroyed, as the parent MG destroys the bottom solver.

Provided the parent MG object persists, this feature can be implemented with boolean switches. One will need to recompute eigen/singular values, but that will be a minor overhead.

That sounds good. In our case, the MG object should be persistent (as we never destroy it explicitly). This will be of major help for future projects!

[cpviolator]

@kostrzewa Sorry for the delay on this, I hit a funny snag that soaked up a few days worth of bug hunting!

I've made an attempt to implement what you need in #900. If you can spin this up and test it, implement it in your production code and then give feed back, I'd be very appreciative.

Let me know if there is anything missing or not right.

[cpviolator]

@kostrzewa BTW, you can see how I've used this in multigrid_evolve_test. The first set of MG evolutions deal with an evolving gauge field. The second set fix the gauge field and augment the mass and mu parameters.

To see the changes I've made, take note of the MG member functions destroyCoarseSolver() and createCoarseSolver(), and also the part of inv_XXX_quda.cpp under the param.deflate boolean.

[kostrzewa]

This has been resolved by #900