Setting twist flavour when using MG

[cpviolator]

I noticed that in the setMultigridParams() function in the test function, the twist flavour is passed to the MG routines (default is MINUS.) Does this mean that in order to perform a calculation with both PLUS and MINUS twists, one needs two separate MG preconditioners for each type? In my code, I have a loop over spin and colour for a given source, and within that spin-colour loop I calculate both up and down type correlation functions. I have found that I get errors when I call the solver indicating that "Twist flavours do not match" if the MG preconditioner is created with MINUS and I then try to change to PLUS inside my calculations.

I can get around this my declaring two new preconditioner pointers in invert_quda.h and quda.h thus:

/**            
* Preconditioner instance, e.g., multigrid  
*/
    void *preconditioner;
    //QKXTM: DMH Exp addition
    void *preconditionerUP;
    void *preconditionerDN;

and then running the MG setup twice, once with QUDA_TWIST_PLUS passed to setMultigridParams() and once with QUDA_TWIST_MINUS:

// setup the multigridsolver 
//void *mg_preconditioner = newMultigridQuda(&mg_param); 
//inv_param.preconditioner = mg_preconditioner; 

//QKXTM: DMH EXP
// setup the multigrid solver for UP flavour
mg_param.invert_param->twist_flavor = QUDA_TWIST_PLUS;
void *mg_preconditionerUP = newMultigridQuda(&mg_param);
inv_param.preconditionerUP = mg_preconditionerUP;

// setup the multigrid solver for DN flavour
mg_param.invert_param->twist_flavor = QUDA_TWIST_MINUS;
void *mg_preconditionerDN = newMultigridQuda(&mg_param);
inv_param.preconditionerDN = mg_preconditionerDN;

Inside the spin color loop, when I wish to change flavour, I write

param->twist_flavor = QUDA_TWIST_PLUS;
...
dirac.prepare(in,out,*x,*b,param->solution_type);
//Set MG Preconditioner to UP
param->preconditioner = param->preconditionerUP;
SolverParam solverParamU(*param);
Solver *solveU = Solver::create(solverParamU, m, mSloppy, mPre, profileInvert);
...
(*solveU)(*out,*in);
...
delete solveU;

...

param->twist_flavor = QUDA_TWIST_MINUS;
...
dirac.prepare(in,out,*x,*b,param->solution_type);
//Set MG Preconditioner to DN
param->preconditioner = param->preconditionerDN;
SolverParam solverParamD(*param);
Solver *solveD = Solver::create(solverParamD, m, mSloppy, mPre, profileInvert);
...
(*solveD)(*out,*in);
...
delete solveD;

The creation of Dirac operators and such (d, dPre, dSloppy, m, mPre, mSloppy) are done before the iteration over spin/colour. I have some questions about this procedure:

1. Is it necessary to have two separate MG preconditioners when using twisted-mass or twisted-clover? Is there an easier way to change the twist of the preconditioner so that twist flavours match when doing PLUS and MINUS twists on the same source?
2. For each flavour, should one recreate the d, dPre, dSloppy, m, mPre, mSloppy objects, or is it sufficient to call: param->twist_flavor = QUDA_TWIST_MINUS; or param->twist_flavor = QUDA_TWIST_PLUS; as required before creating a SolverParam object and calling the solver?

[maddyscientist]

Apologies for the aeonic response time. To answer your questions

1. There is no need for this, and is this present functionality is rather accidental
2. I'm not sure what one has to do at present due to a little bit of ignorance on my behalf

Cleaning up this should be high in our priorities. Moreover, I'm not entirely sure why the ColorSpinorField class should know or care about what the twist_flavor is. This has some rather unintended side effects, such as linear algebra being retuned when changing flavour but having all other meta data constant.

@alexstrel can you remind me what the motivation is for each field knowing what twist_flavor it is be applied to?