progress for testing the PC ndeg twisted clover operator

[urbach]

soon to be used...

compiling and linking works right now with the Ndeg PC twisted clover operators.

[urbach]

warning: unused parameter ‘xpay’ [-Wunused-parameter]

[kostrzewa]

Note that this warning is for NdegTwistedPreconditionedCloverArg, where AFAICS, this should be used to set the a parameter of the Wilson Dslash:

 26   NdegTwistedCloverPreconditionedArg(ColorSpinorField &out, const ColorSpinorField &in,
 27                                      const GaugeField &U, const CloverField &A,
 28                                      double a, double b, double c, bool xpay,
 29                                      const ColorSpinorField &x, int parity, bool dagger,
 30                                      const int *comm_override) :
 31     WilsonArg<Float, nColor, nDim, reconstruct_>(out, in, U, xpay ? 1.0 : 0.0, x, parity, dagger, comm_override),                      
 32       A(A, false),
 33       A2inv(A, dynamic_clover ? false : true), // if dynamic clover we don't want the inverse field
 34       a(a),
 35       b(dagger ? -0.5 * b : 0.5 * b), // if dagger flip the chiral twist
 36       c(0.5*c)
 37       {
 38       }
 39   };

at least this is what other operators do. The weird thing is that, as far as I understand, we should only implement the xpay == true path anyway.

[kostrzewa]

I'm reasonably convinced that this test:

np=2                                                                                                                                                         
matpc_types=( 'odd-odd' 'even-even' )
for matpc_type in ${matpc_types[@]}; do
  QUDA_ENABLE_TUNING=0 \
  mpirun -np ${np} ${exe} \
    --gridsize 1 1 1 ${np} \
    --prec 'double' \
    --Lsdim 1 \ 
    --dagger \
    --matpc ${matpc_type} \
    --dslash-type 'twisted-clover' \
    --flavor 'nondeg-doublet' \
    --mu 0.2 \
    --epsilon 0.1 \
    --kappa 0.127 \
    --clover-coeff 0.5 \
    --test 'Dslash'
done

should pass, but it does not. There are three possibilities (which bifuricate into many potential failure points):

1. The tmc_ndeg_dslash has a bug which I simply can't spot because I've been staring at it for too long.
2. My modifications to the launch of the ApplyWilson kernel don't do what I think they should be doing.
3. My modifications to TwistClover don't work correctly.

Testing these components separately will requiring writing new independent tests, I think. The latter is a useful, if somewhat tedious, exercise. The benefit would be that one can hard-code various params, making it "easier" to see what is actually done.

[kostrzewa]

Okay, I'm getting somewhere. With https://github.com/qcdcode/quda/commit/045992381836aaef5f87340c1d4fcc07e2f75556, at least the pure Wilson Dslash for two flavours passes:

np=2
matpc_types=( 'odd-odd' 'even-even' )
for matpc_type in ${matpc_types[@]}; do
  QUDA_ENABLE_TUNING=0 \
  mpirun -np ${np} ${exe} \
    --gridsize 1 1 1 ${np} \
    --prec 'double' \
    --Lsdim 1 \
    --dagger \
    --matpc ${matpc_type} \
    --dslash-type 'twisted-clover' \
    --flavor 'nondeg-doublet' \
    --mu 0.0 \                                                                                                                                               
    --epsilon 0.0 \
    --kappa 0.127 \
    --clover-coeff 0.0 \
    --test 'Dslash'
done

[...]
running the following test:
prec    recon   dtest_type     matpc_type   dagger   S_dim         T_dimension   Ls_dimension dslash_type    niter
double   18       Dslash                odd-odd    1     24/ 24/ 24         24              1   twisted-clover   100
[...]
[==========] Running 1 test from 1 test suite.
[----------] Global test environment set-up.
[----------] 1 test from dslash
[ RUN      ] dslash.verify
0 fails = 0
[...]
1.000000e-15 Failures: 0 / 15925248  = 0.000000e+00
1.000000e-16 Failures: 2689809 / 15925248  = 1.689022e-01
[       OK ] dslash.verify (1977 ms)
[----------] 1 test from dslash (1977 ms total)

[----------] Global test environment tear-down
[==========] 1 test from 1 test suite ran. (1977 ms total)
[  PASSED  ] 1 test

and similarly, pure two-flavour Wilson clover works:

np=2
matpc_types=( 'odd-odd' 'even-even' )
for matpc_type in ${matpc_types[@]}; do
  QUDA_ENABLE_TUNING=0 \
  mpirun -np ${np} ${exe} \
    --gridsize 1 1 1 ${np} \
    --prec 'double' \
    --Lsdim 1 \
    --dagger \
    --matpc ${matpc_type} \
    --dslash-type 'twisted-clover' \
    --flavor 'nondeg-doublet' \
    --mu 0.0 \
    --epsilon 0.0 \                                                                                                                                          
    --kappa 0.127 \
    --clover-coeff 1.0 \
    --test 'Dslash'
done

[kostrzewa]

Had a bug (forgotten tau_3 sign -> 2dc7ffe8b44209b2162f5199ef7791a912b93ede) in ndegTwistCloverGamma5 for QUDA_TWIST_GAMMA5_INVERSE. Now also pure twisted mass passes:

np=2
matpc_types=( 'odd-odd' 'even-even' )
for matpc_type in ${matpc_types[@]}; do
  QUDA_ENABLE_TUNING=0 \
  mpirun -np ${np} ${exe} \
    --gridsize 1 1 1 ${np} \
    --prec 'double' \
    --Lsdim 1 \
    --dagger \
    --matpc ${matpc_type} \
    --dslash-type 'twisted-clover' \
    --flavor 'nondeg-doublet' \
    --mu 100000000 \                                                                                                                                         
    --epsilon 0.0 \
    --kappa 0.0000000001 \
    --clover-coeff 0.0 \
    --test 'Dslash'
done

as well as a more general:

np=2
matpc_types=( 'odd-odd' 'even-even' )
for matpc_type in ${matpc_types[@]}; do
  QUDA_ENABLE_TUNING=0 \
  mpirun -np ${np} ${exe} \
    --gridsize 1 1 1 ${np} \
    --prec 'double' \
    --Lsdim 1 \
    --dagger \
    --matpc ${matpc_type} \
    --dslash-type 'twisted-clover' \
    --flavor 'nondeg-doublet' \
    --mu 0.1 \
    --epsilon 0.0 \
    --kappa 0.127 \                                                                                                                                          
    --clover-coeff 0.0 \
    --test 'Dslash'
done

and non-deg twisted mass:

np=2
matpc_types=( 'odd-odd' 'even-even' )
for matpc_type in ${matpc_types[@]}; do
  QUDA_ENABLE_TUNING=0 \
  mpirun -np ${np} ${exe} \
    --gridsize 1 1 1 ${np} \
    --prec 'double' \
    --Lsdim 1 \
    --dagger \
    --matpc ${matpc_type} \
    --dslash-type 'twisted-clover' \
    --flavor 'nondeg-doublet' \
    --mu 0.2 \
    --epsilon 0.1 \                                                                                                                                          
    --kappa 0.127 \
    --clover-coeff 0.0 \
    --test 'Dslash'
done

and finally, non-deg twisted clover dslash:

np=2
matpc_types=( 'odd-odd' 'even-even' )
for matpc_type in ${matpc_types[@]}; do
  QUDA_ENABLE_TUNING=0 \
  mpirun -np ${np} ${exe} \
    --gridsize 1 1 1 ${np} \
    --prec 'double' \
    --Lsdim 1 \
    --dagger \
    --matpc ${matpc_type} \
    --dslash-type 'twisted-clover' \
    --flavor 'nondeg-doublet' \
    --mu 0.2 \
    --epsilon 0.1 \
    --kappa 0.127 \
    --clover-coeff 1.0 \                                                                                                                                     
    --test 'Dslash'
done

[kostrzewa]

So, PC non-deg twisted clover Dslash passes, now on to DslashXpay and M!

[kostrzewa]

Of course I was celebrating prematurely, as the asymmetric ops and the non-dagger version need to work properly ;)

[kostrzewa]

Alright, the non-dagger even-even and odd-odd work, now the dagger asymmetric version is still throwing differences. https://github.com/qcdcode/quda/pull/4/commits/fad04823999c8586173115813f22010992641ded

[urbach]

fantastic!

[kostrzewa]

fantastic!

I'm not so sure, I think I broke all other preconditioned Wilson-type operators in the process ...

[kostrzewa]

fantastic!

I'm not so sure, I think I broke all other preconditioned Wilson-type operators in the process ...

Yup, definitely messed something up, I think I know what though.

[kostrzewa]

Fixed. https://github.com/qcdcode/quda/pull/4/commits/b7b60f60dd900532ae0e5c7d436258fa5b6fa20b

[kostrzewa]

With #11, all the tests for PC::M and PC::MdagM pass.