search

etmc / tmLQCD

tmLQCD is a freely available software suite providing a set of tools to be used in lattice QCD simulations. This is mainly a HMC implementation (including PHMC and RHMC) for Wilson, Wilson Clover and Wilson twisted mass fermions and inverter for different versions of the Dirac operator. The code is fully parallelised and ships with optimisations for various modern architectures, such as commodity PC clusters and the Blue Gene family.
http://www.itkp.uni-bonn.de/~urbach/software.html
GNU General Public License v3.0
32 stars 47 forks source link

study behaviour of residuals in QUDA multishift CG (relevant for correction monomials) #496

Open kostrzewa opened 2 years ago

kostrzewa commented 2 years ago 
# solve_mms_nd residual check: shift 0 (9.932405e-07), res. 6.264323e-17
# solve_mms_nd residual check: shift 1 (1.214376e-05), res. 1.027747e-15
# solve_mms_nd residual check: shift 2 (5.934642e-05), res. 4.248070e-16
# solve_mms_nd residual check: shift 3 (2.480315e-04), res. 7.064471e-17
# solve_mms_nd residual check: shift 4 (9.998997e-04), res. 4.038716e-18
# solve_mms_nd residual check: shift 5 (3.999472e-03), res. 1.978892e-19
# solve_mms_nd residual check: shift 6 (1.603319e-02), res. 1.098602e-20
# solve_mms_nd residual check: shift 7 (6.540409e-02), res. 1.184556e-21
# solve_mms_nd residual check: shift 8 (2.873688e-01), res. 3.295545e-22
# solve_mms_nd residual check: shift 9 (1.791220e+00), res. 3.867626e-23
# NDRATCOR heatbath: c_1*(R * Z^1 * R) = -2.670276e-01
# NDRATCOR heatbath: c_2*(R * Z^2 * R) = -1.220353e-06

Some of the shifts are rather poorly converged here (and they fail the StrictResidualCheck).

Heavy quark residuals should help here, but I've never used them.

kostrzewa commented 2 years ago

QUDA itself also measures the mismatch in true and iterated residuals:

# QUDA:  shift=0, 2685 iterations, relative residual: iterated = 1.339998e-18, true = 3.072347e-13
# QUDA:  shift=1, 1988 iterations, relative residual: iterated = 2.190374e-16, true = 1.996018e-12
# QUDA:  shift=2, 1474 iterations, relative residual: iterated = 2.213764e-16, true = 1.258002e-12
# QUDA:  shift=3, 877 iterations, relative residual: iterated = 2.184230e-16, true = 5.164569e-13
# QUDA:  shift=4, 463 iterations, relative residual: iterated = 2.115163e-16, true = 1.213822e-13
# QUDA:  shift=5, 235 iterations, relative residual: iterated = 2.033043e-16, true = 2.587211e-14
# QUDA:  shift=6, 118 iterations, relative residual: iterated = 1.885423e-16, true = 5.621644e-15
# QUDA:  shift=7, 58 iterations, relative residual: iterated = 3.443531e-16, true = 1.431811e-15
# QUDA:  shift=8, 29 iterations, relative residual: iterated = 2.684125e-16, true = 5.400882e-16
# QUDA:  shift=9, 14 iterations, relative residual: iterated = 2.334832e-16, true = 3.442009e-16

but it does not refine the shifts that clearly have not reached the target precision

kostrzewa commented 2 years ago

The failing system is not unusual, but it's behaviour is. The residual of the unshifted system first increases and only at around iteration 50 does the residual being to decrease. (note that this is fully in double precision).

# QUDA: Source: CPU = 3.01998e+08, CUDA copy = 3.01998e+08
# QUDA: MultiShift CG: 0 iterations, <r,r> = 3.019984e+08, |r|/|b| = 1.000000e+00
# QUDA: MultiShift CG: 1 iterations, <r,r> = 4.124038e+08, |r|/|b| = 1.168581e+00
# QUDA: MultiShift CG: 2 iterations, <r,r> = 5.754957e+08, |r|/|b| = 1.380444e+00
# QUDA: MultiShift CG: 3 iterations, <r,r> = 7.783373e+08, |r|/|b| = 1.605394e+00
# QUDA: MultiShift CG: 4 iterations, <r,r> = 9.696961e+08, |r|/|b| = 1.791907e+00
# QUDA: MultiShift CG: 5 iterations, <r,r> = 1.147018e+09, |r|/|b| = 1.948870e+00
# QUDA: MultiShift CG: 6 iterations, <r,r> = 1.325877e+09, |r|/|b| = 2.095315e+00
# QUDA: MultiShift CG: 7 iterations, <r,r> = 1.514585e+09, |r|/|b| = 2.239466e+00
# QUDA: MultiShift CG: 8 iterations, <r,r> = 1.715470e+09, |r|/|b| = 2.383358e+00
# QUDA: MultiShift CG: 9 iterations, <r,r> = 1.890111e+09, |r|/|b| = 2.501735e+00
# QUDA: MultiShift CG: 10 iterations, <r,r> = 2.007104e+09, |r|/|b| = 2.577998e+00
# QUDA: MultiShift CG: 11 iterations, <r,r> = 2.102908e+09, |r|/|b| = 2.638808e+00
# QUDA: MultiShift CG: 12 iterations, <r,r> = 2.201186e+09, |r|/|b| = 2.699765e+00
# QUDA: MultiShift CG: 13 iterations, <r,r> = 2.281042e+09, |r|/|b| = 2.748301e+00
# QUDA: MultiShift CG: Shift 9 converged after 14 iterations
# QUDA: MultiShift CG: 14 iterations, <r,r> = 2.330030e+09, |r|/|b| = 2.777656e+00
# QUDA: MultiShift CG: 15 iterations, <r,r> = 2.348029e+09, |r|/|b| = 2.788363e+00
# QUDA: MultiShift CG: 16 iterations, <r,r> = 2.340090e+09, |r|/|b| = 2.783646e+00
# QUDA: MultiShift CG: 17 iterations, <r,r> = 2.319170e+09, |r|/|b| = 2.771175e+00
# QUDA: MultiShift CG: 18 iterations, <r,r> = 2.290146e+09, |r|/|b| = 2.753780e+00
# QUDA: MultiShift CG: 19 iterations, <r,r> = 2.249532e+09, |r|/|b| = 2.729253e+00
# QUDA: MultiShift CG: 20 iterations, <r,r> = 2.201072e+09, |r|/|b| = 2.699696e+00
# QUDA: MultiShift CG: 21 iterations, <r,r> = 2.142888e+09, |r|/|b| = 2.663774e+00
# QUDA: MultiShift CG: 22 iterations, <r,r> = 2.083006e+09, |r|/|b| = 2.626291e+00
# QUDA: MultiShift CG: 23 iterations, <r,r> = 2.027043e+09, |r|/|b| = 2.590772e+00
# QUDA: MultiShift CG: 24 iterations, <r,r> = 1.974508e+09, |r|/|b| = 2.556979e+00
# QUDA: MultiShift CG: 25 iterations, <r,r> = 1.921055e+09, |r|/|b| = 2.522130e+00
# QUDA: MultiShift CG: 26 iterations, <r,r> = 1.863668e+09, |r|/|b| = 2.484174e+00
# QUDA: MultiShift CG: 27 iterations, <r,r> = 1.807631e+09, |r|/|b| = 2.446541e+00
# QUDA: MultiShift CG: Shift 8 converged after 28 iterations
# QUDA: MultiShift CG: 28 iterations, <r,r> = 1.754635e+09, |r|/|b| = 2.410411e+00
# QUDA: MultiShift CG: 29 iterations, <r,r> = 1.699863e+09, |r|/|b| = 2.372491e+00
# QUDA: MultiShift CG: 30 iterations, <r,r> = 1.645655e+09, |r|/|b| = 2.334356e+00
# QUDA: MultiShift CG: 31 iterations, <r,r> = 1.595839e+09, |r|/|b| = 2.298752e+00
# QUDA: MultiShift CG: 32 iterations, <r,r> = 1.549435e+09, |r|/|b| = 2.265084e+00
# QUDA: MultiShift CG: 33 iterations, <r,r> = 1.502368e+09, |r|/|b| = 2.230416e+00
# QUDA: MultiShift CG: 34 iterations, <r,r> = 1.457592e+09, |r|/|b| = 2.196927e+00
# QUDA: MultiShift CG: 35 iterations, <r,r> = 1.414756e+09, |r|/|b| = 2.164404e+00
# QUDA: MultiShift CG: 36 iterations, <r,r> = 1.371005e+09, |r|/|b| = 2.130675e+00
# QUDA: MultiShift CG: 37 iterations, <r,r> = 1.326571e+09, |r|/|b| = 2.095863e+00
# QUDA: MultiShift CG: 38 iterations, <r,r> = 1.281006e+09, |r|/|b| = 2.059555e+00
# QUDA: MultiShift CG: 39 iterations, <r,r> = 1.236644e+09, |r|/|b| = 2.023579e+00
# QUDA: MultiShift CG: 40 iterations, <r,r> = 1.194798e+09, |r|/|b| = 1.989046e+00
# QUDA: MultiShift CG: 41 iterations, <r,r> = 1.151800e+09, |r|/|b| = 1.952928e+00
# QUDA: MultiShift CG: 42 iterations, <r,r> = 1.108440e+09, |r|/|b| = 1.915816e+00
# QUDA: MultiShift CG: 43 iterations, <r,r> = 1.068240e+09, |r|/|b| = 1.880755e+00
# QUDA: MultiShift CG: 44 iterations, <r,r> = 1.028136e+09, |r|/|b| = 1.845113e+00
# QUDA: MultiShift CG: 45 iterations, <r,r> = 9.902317e+08, |r|/|b| = 1.810782e+00
# QUDA: MultiShift CG: 46 iterations, <r,r> = 9.519337e+08, |r|/|b| = 1.775420e+00
# QUDA: MultiShift CG: 47 iterations, <r,r> = 9.114288e+08, |r|/|b| = 1.737237e+00
# QUDA: MultiShift CG: 48 iterations, <r,r> = 8.691801e+08, |r|/|b| = 1.696495e+00
# QUDA: MultiShift CG: 49 iterations, <r,r> = 8.326528e+08, |r|/|b| = 1.660465e+00
# QUDA: MultiShift CG: 50 iterations, <r,r> = 8.006601e+08, |r|/|b| = 1.628252e+00
# QUDA: MultiShift CG: 51 iterations, <r,r> = 7.703597e+08, |r|/|b| = 1.597145e+00
# QUDA: MultiShift CG: 52 iterations, <r,r> = 7.423859e+08, |r|/|b| = 1.567879e+00
# QUDA: MultiShift CG: 53 iterations, <r,r> = 7.174967e+08, |r|/|b| = 1.541373e+00
# QUDA: MultiShift CG: 54 iterations, <r,r> = 6.961897e+08, |r|/|b| = 1.518314e+00
# QUDA: MultiShift CG: 55 iterations, <r,r> = 6.755619e+08, |r|/|b| = 1.495651e+00
# QUDA: MultiShift CG: Shift 7 converged after 56 iterations
# QUDA: MultiShift CG: 56 iterations, <r,r> = 6.563961e+08, |r|/|b| = 1.474282e+00
# QUDA: MultiShift CG: 57 iterations, <r,r> = 6.369816e+08, |r|/|b| = 1.452316e+00
# QUDA: MultiShift CG: 58 iterations, <r,r> = 6.177638e+08, |r|/|b| = 1.430240e+00
# QUDA: MultiShift CG: 59 iterations, <r,r> = 6.009194e+08, |r|/|b| = 1.410606e+00
# QUDA: MultiShift CG: 60 iterations, <r,r> = 5.850583e+08, |r|/|b| = 1.391865e+00
# QUDA: MultiShift CG: 61 iterations, <r,r> = 5.678196e+08, |r|/|b| = 1.371206e+00
# QUDA: MultiShift CG: 62 iterations, <r,r> = 5.493354e+08, |r|/|b| = 1.348703e+00
# QUDA: MultiShift CG: 63 iterations, <r,r> = 5.305232e+08, |r|/|b| = 1.325409e+00
# QUDA: MultiShift CG: 64 iterations, <r,r> = 5.116405e+08, |r|/|b| = 1.301608e+00
# QUDA: MultiShift CG: 65 iterations, <r,r> = 4.956964e+08, |r|/|b| = 1.281166e+00
# QUDA: MultiShift CG: 66 iterations, <r,r> = 4.821926e+08, |r|/|b| = 1.263595e+00
# QUDA: MultiShift CG: 67 iterations, <r,r> = 4.702025e+08, |r|/|b| = 1.247786e+00
# QUDA: MultiShift CG: 68 iterations, <r,r> = 4.582030e+08, |r|/|b| = 1.231762e+00
# QUDA: MultiShift CG: 69 iterations, <r,r> = 4.450549e+08, |r|/|b| = 1.213960e+00
# QUDA: MultiShift CG: 70 iterations, <r,r> = 4.295934e+08, |r|/|b| = 1.192687e+00
# QUDA: MultiShift CG: 71 iterations, <r,r> = 4.143917e+08, |r|/|b| = 1.171394e+00
# QUDA: MultiShift CG: 72 iterations, <r,r> = 3.987838e+08, |r|/|b| = 1.149123e+00
# QUDA: MultiShift CG: 73 iterations, <r,r> = 3.836947e+08, |r|/|b| = 1.127173e+00
# QUDA: MultiShift CG: 74 iterations, <r,r> = 3.686857e+08, |r|/|b| = 1.104907e+00
# QUDA: MultiShift CG: 75 iterations, <r,r> = 3.542692e+08, |r|/|b| = 1.083090e+00
# QUDA: MultiShift CG: 76 iterations, <r,r> = 3.408402e+08, |r|/|b| = 1.062363e+00
# QUDA: MultiShift CG: 77 iterations, <r,r> = 3.279236e+08, |r|/|b| = 1.042039e+00
# QUDA: MultiShift CG: 78 iterations, <r,r> = 3.157982e+08, |r|/|b| = 1.022592e+00
# QUDA: MultiShift CG: 79 iterations, <r,r> = 3.051405e+08, |r|/|b| = 1.005189e+00
# QUDA: MultiShift CG: 80 iterations, <r,r> = 2.952715e+08, |r|/|b| = 9.887999e-01
# QUDA: MultiShift CG: 81 iterations, <r,r> = 2.860221e+08, |r|/|b| = 9.731896e-01
# QUDA: MultiShift CG: 82 iterations, <r,r> = 2.762835e+08, |r|/|b| = 9.564784e-01
# QUDA: MultiShift CG: 83 iterations, <r,r> = 2.668862e+08, |r|/|b| = 9.400712e-01
# QUDA: MultiShift CG: 84 iterations, <r,r> = 2.574764e+08, |r|/|b| = 9.233500e-01
# QUDA: MultiShift CG: 85 iterations, <r,r> = 2.487434e+08, |r|/|b| = 9.075560e-01
# QUDA: MultiShift CG: 86 iterations, <r,r> = 2.409832e+08, |r|/|b| = 8.932870e-01
# QUDA: MultiShift CG: 87 iterations, <r,r> = 2.337013e+08, |r|/|b| = 8.796870e-01
# QUDA: MultiShift CG: 88 iterations, <r,r> = 2.262689e+08, |r|/|b| = 8.655857e-01
# QUDA: MultiShift CG: 89 iterations, <r,r> = 2.188422e+08, |r|/|b| = 8.512618e-01
# QUDA: MultiShift CG: 90 iterations, <r,r> = 2.118671e+08, |r|/|b| = 8.375860e-01
# QUDA: MultiShift CG: 91 iterations, <r,r> = 2.050843e+08, |r|/|b| = 8.240696e-01
# QUDA: MultiShift CG: 92 iterations, <r,r> = 1.985795e+08, |r|/|b| = 8.108955e-01
# QUDA: MultiShift CG: 93 iterations, <r,r> = 1.928543e+08, |r|/|b| = 7.991206e-01
# QUDA: MultiShift CG: 94 iterations, <r,r> = 1.884237e+08, |r|/|b| = 7.898879e-01
# QUDA: MultiShift CG: 95 iterations, <r,r> = 1.860827e+08, |r|/|b| = 7.849658e-01
# QUDA: MultiShift CG: 96 iterations, <r,r> = 1.842173e+08, |r|/|b| = 7.810212e-01
# QUDA: MultiShift CG: 97 iterations, <r,r> = 1.796777e+08, |r|/|b| = 7.713379e-01
# QUDA: MultiShift CG: 98 iterations, <r,r> = 1.709753e+08, |r|/|b| = 7.524269e-01
# QUDA: MultiShift CG: 99 iterations, <r,r> = 1.618731e+08, |r|/|b| = 7.321247e-01
# QUDA: MultiShift CG: 100 iterations, <r,r> = 1.538236e+08, |r|/|b| = 7.136893e-01
# QUDA: MultiShift CG: 101 iterations, <r,r> = 1.464477e+08, |r|/|b| = 6.963682e-01
# QUDA: MultiShift CG: 102 iterations, <r,r> = 1.398137e+08, |r|/|b| = 6.804128e-01
# QUDA: MultiShift CG: 103 iterations, <r,r> = 1.340044e+08, |r|/|b| = 6.661271e-01
# QUDA: MultiShift CG: 104 iterations, <r,r> = 1.290325e+08, |r|/|b| = 6.536530e-01
# QUDA: MultiShift CG: 105 iterations, <r,r> = 1.246111e+08, |r|/|b| = 6.423563e-01
# QUDA: MultiShift CG: 106 iterations, <r,r> = 1.202584e+08, |r|/|b| = 6.310378e-01
# QUDA: MultiShift CG: 107 iterations, <r,r> = 1.160847e+08, |r|/|b| = 6.199907e-01
# QUDA: MultiShift CG: 108 iterations, <r,r> = 1.118958e+08, |r|/|b| = 6.087017e-01
# QUDA: MultiShift CG: 109 iterations, <r,r> = 1.077878e+08, |r|/|b| = 5.974238e-01
# QUDA: MultiShift CG: 110 iterations, <r,r> = 1.039634e+08, |r|/|b| = 5.867296e-01
# QUDA: MultiShift CG: Shift 6 converged after 111 iterations
# QUDA: MultiShift CG: 111 iterations, <r,r> = 1.002535e+08, |r|/|b| = 5.761656e-01
# QUDA: MultiShift CG: 112 iterations, <r,r> = 9.687114e+07, |r|/|b| = 5.663630e-01
# QUDA: MultiShift CG: 113 iterations, <r,r> = 9.367267e+07, |r|/|b| = 5.569345e-01
# QUDA: MultiShift CG: 114 iterations, <r,r> = 9.084937e+07, |r|/|b| = 5.484772e-01
# QUDA: MultiShift CG: 115 iterations, <r,r> = 8.812850e+07, |r|/|b| = 5.402016e-01
# QUDA: MultiShift CG: 116 iterations, <r,r> = 8.525218e+07, |r|/|b| = 5.313130e-01
# QUDA: MultiShift CG: 117 iterations, <r,r> = 8.229736e+07, |r|/|b| = 5.220242e-01
# QUDA: MultiShift CG: 118 iterations, <r,r> = 7.924090e+07, |r|/|b| = 5.122387e-01
# QUDA: MultiShift CG: 119 iterations, <r,r> = 7.624509e+07, |r|/|b| = 5.024624e-01
# QUDA: MultiShift CG: 120 iterations, <r,r> = 7.342360e+07, |r|/|b| = 4.930779e-01
# QUDA: MultiShift CG: 121 iterations, <r,r> = 7.071289e+07, |r|/|b| = 4.838903e-01
# QUDA: MultiShift CG: 122 iterations, <r,r> = 6.815394e+07, |r|/|b| = 4.750542e-01
# QUDA: MultiShift CG: 123 iterations, <r,r> = 6.567956e+07, |r|/|b| = 4.663508e-01
# QUDA: MultiShift CG: 124 iterations, <r,r> = 6.326844e+07, |r|/|b| = 4.577109e-01
# QUDA: MultiShift CG: 125 iterations, <r,r> = 6.091399e+07, |r|/|b| = 4.491136e-01
[...]
# QUDA: MultiShift CG: 2075 iterations, <r,r> = 1.930824e-21, |r|/|b| = 2.528535e-15
# QUDA: MultiShift CG: 2076 iterations, <r,r> = 1.831832e-21, |r|/|b| = 2.462864e-15
# QUDA: MultiShift CG: 2077 iterations, <r,r> = 1.706178e-21, |r|/|b| = 2.376894e-15
# QUDA: MultiShift CG: 2078 iterations, <r,r> = 1.614180e-21, |r|/|b| = 2.311924e-15
# QUDA: MultiShift CG: 2079 iterations, <r,r> = 1.550299e-21, |r|/|b| = 2.265715e-15
# QUDA: MultiShift CG: 2080 iterations, <r,r> = 1.503310e-21, |r|/|b| = 2.231115e-15
# QUDA: MultiShift CG: 2081 iterations, <r,r> = 1.464170e-21, |r|/|b| = 2.201878e-15
# QUDA: MultiShift CG: 2082 iterations, <r,r> = 1.426639e-21, |r|/|b| = 2.173475e-15
# QUDA: MultiShift CG: 2083 iterations, <r,r> = 1.387701e-21, |r|/|b| = 2.143609e-15
# QUDA: MultiShift CG: 2084 iterations, <r,r> = 1.346679e-21, |r|/|b| = 2.111688e-15
# QUDA: MultiShift CG: 2085 iterations, <r,r> = 1.309152e-21, |r|/|b| = 2.082058e-15
# QUDA: MultiShift CG: 2086 iterations, <r,r> = 1.278429e-21, |r|/|b| = 2.057482e-15
# QUDA: MultiShift CG: 2087 iterations, <r,r> = 1.251274e-21, |r|/|b| = 2.035513e-15
# QUDA: MultiShift CG: 2088 iterations, <r,r> = 1.223162e-21, |r|/|b| = 2.012517e-15
# QUDA: MultiShift CG: 2089 iterations, <r,r> = 1.190136e-21, |r|/|b| = 1.985162e-15
# QUDA: MultiShift CG: 2090 iterations, <r,r> = 1.152454e-21, |r|/|b| = 1.953482e-15
# QUDA: MultiShift CG: 2091 iterations, <r,r> = 1.114716e-21, |r|/|b| = 1.921232e-15
# QUDA: MultiShift CG: 2092 iterations, <r,r> = 1.083025e-21, |r|/|b| = 1.893725e-15
# QUDA: MultiShift CG: 2093 iterations, <r,r> = 1.056231e-21, |r|/|b| = 1.870153e-15
# QUDA: MultiShift CG: 2094 iterations, <r,r> = 1.041733e-21, |r|/|b| = 1.857273e-15
# QUDA: MultiShift CG: 2095 iterations, <r,r> = 1.038795e-21, |r|/|b| = 1.854653e-15
# QUDA: MultiShift CG: 2096 iterations, <r,r> = 1.028237e-21, |r|/|b| = 1.845204e-15
# QUDA: Convergence of unshifted system so trigger shiftUpdate
# QUDA: MultiShift CG: 2097 iterations, <r,r> = 9.870603e-22, |r|/|b| = 1.807880e-15
# QUDA: MultiShift CG: Reliable updates = 1
# QUDA: MultiShift CG: Converged after 2097 iterations
# QUDA:  shift=0, 2097 iterations, relative residual: iterated = 1.807880e-15, true = 3.551623e-13
# QUDA:  shift=1, 1871 iterations, relative residual: iterated = 1.810807e-15, true = 1.995843e-12
# QUDA:  shift=2, 1392 iterations, relative residual: iterated = 1.813952e-15, true = 1.262540e-12
# QUDA:  shift=3, 828 iterations, relative residual: iterated = 1.766019e-15, true = 5.075596e-13
# QUDA:  shift=4, 437 iterations, relative residual: iterated = 1.705232e-15, true = 1.181034e-13
# QUDA:  shift=5, 222 iterations, relative residual: iterated = 1.559026e-15, true = 2.524782e-14
# QUDA:  shift=6, 111 iterations, relative residual: iterated = 1.709715e-15, true = 5.793205e-15
# QUDA:  shift=7, 56 iterations, relative residual: iterated = 1.182585e-15, true = 1.816071e-15
# QUDA:  shift=8, 28 iterations, relative residual: iterated = 9.182625e-16, true = 1.030067e-15
# QUDA:  shift=9, 14 iterations, relative residual: iterated = 2.334832e-16, true = 3.087034e-16
# QUDA: Solution 0 = 3.59615e+14
# QUDA: Solution 1 = 2.63443e+14
# QUDA: Solution 2 = 1.066e+14
# QUDA: Solution 3 = 2.00113e+13
# QUDA: Solution 4 = 2.20153e+12
# QUDA: Solution 5 = 1.93229e+11
# QUDA: Solution 6 = 1.62965e+10
# QUDA: Solution 7 = 1.40688e+09
# QUDA: Solution 8 = 1.09456e+08
# QUDA: Solution 9 = 3.83703e+06
# TM_QUDA: time spent in reorder_spinor_eo_fromQuda: 0.004545 secs
# TM_QUDA: time spent in reorder_spinor_eo_fromQuda: 0.005458 secs
# TM_QUDA: time spent in reorder_spinor_eo_fromQuda: 0.004614 secs
# TM_QUDA: time spent in reorder_spinor_eo_fromQuda: 0.005233 secs
# TM_QUDA: time spent in reorder_spinor_eo_fromQuda: 0.004607 secs
# TM_QUDA: time spent in reorder_spinor_eo_fromQuda: 0.004674 secs
# TM_QUDA: time spent in reorder_spinor_eo_fromQuda: 0.004763 secs
# TM_QUDA: time spent in reorder_spinor_eo_fromQuda: 0.004765 secs
# TM_QUDA: time spent in reorder_spinor_eo_fromQuda: 0.004937 secs
# TM_QUDA: time spent in reorder_spinor_eo_fromQuda: 0.004929 secs
# TM_QUDA: Done: 2097 iter / 24.0167 secs = 95.0379 Gflops
# solve_mms_nd residual check: shift 0 (9.932405e-07), res. 7.549666e-17
# solve_mms_nd residual check: shift 1 (1.214376e-05), res. 1.229981e-15
# solve_mms_nd residual check: shift 2 (5.934642e-05), res. 4.921435e-16
# solve_mms_nd residual check: shift 3 (2.480315e-04), res. 7.978995e-17
# solve_mms_nd residual check: shift 4 (9.998997e-04), res. 4.440684e-18
# solve_mms_nd residual check: shift 5 (3.999472e-03), res. 2.120215e-19
# solve_mms_nd residual check: shift 6 (1.603319e-02), res. 1.183083e-20
# solve_mms_nd residual check: shift 7 (6.540409e-02), res. 1.169118e-21
# solve_mms_nd residual check: shift 8 (2.873688e-01), res. 3.290289e-22
# solve_mms_nd residual check: shift 9 (1.791220e+00), res. 4.086714e-23
kostrzewa commented 2 years ago

QPhiX behaves similarly, but the final residual does not increase from shift 0 to shift 1 (probably because QUDA refines the shift 0 system while QPhiX does not):

# solve_mms_nd residual check: 4.445527e-15
# solve_mms_nd residual check: 2.761395e-15
# solve_mms_nd residual check: 7.768592e-16
# solve_mms_nd residual check: 7.815831e-17
# solve_mms_nd residual check: 4.244760e-18
# solve_mms_nd residual check: 1.963739e-19
# solve_mms_nd residual check: 1.022310e-20
# solve_mms_nd residual check: 1.011501e-21
# solve_mms_nd residual check: 3.320794e-22
# solve_mms_nd residual check: 3.343652e-23
kostrzewa commented 2 years ago

tmLQCD itself behaves more or less like QPhiX:

# solve_mms_nd residual check: shift 0 (9.932405e-07), res. 2.573334e-15
# solve_mms_nd residual check: shift 1 (1.214376e-05), res. 1.757899e-15
# solve_mms_nd residual check: shift 2 (5.934642e-05), res. 5.304336e-16
# solve_mms_nd residual check: shift 3 (2.480315e-04), res. 6.130024e-17
# solve_mms_nd residual check: shift 4 (9.998997e-04), res. 3.680166e-18
# solve_mms_nd residual check: shift 5 (3.999472e-03), res. 1.743329e-19
# solve_mms_nd residual check: shift 6 (1.603319e-02), res. 8.464089e-21
# solve_mms_nd residual check: shift 7 (6.540409e-02), res. 5.363837e-22
# solve_mms_nd residual check: shift 8 (2.873688e-01), res. 6.173453e-23

In any case, neither solver manages to satisfy the criterion for this particular correction term. This might have to do with the gauge configuration used here, which is not fully thermalized and does not originate from the target action. To be studied further.

kostrzewa commented 2 years ago

@urbach I kind of understand why this is the case (the starting residual in the solve for the second term in the correction polynomial is very large and hence the target residual is beyond machine precision for the smallest shifts) but I don't quite see whether this affects the real precision of our correction monomial. This is also bound to get worse with increasing volume (this is on a 16c32 lattice...)

# solve_mms_nd residual check: shift 0 (9.932405e-07), res. 6.264323e-17
# solve_mms_nd residual check: shift 1 (1.214376e-05), res. 1.027747e-15
# solve_mms_nd residual check: shift 2 (5.934642e-05), res. 4.248070e-16
# solve_mms_nd residual check: shift 3 (2.480315e-04), res. 7.064471e-17
# solve_mms_nd residual check: shift 4 (9.998997e-04), res. 4.038716e-18
# solve_mms_nd residual check: shift 5 (3.999472e-03), res. 1.978892e-19
# solve_mms_nd residual check: shift 6 (1.603319e-02), res. 1.098602e-20
# solve_mms_nd residual check: shift 7 (6.540409e-02), res. 1.184556e-21
# solve_mms_nd residual check: shift 8 (2.873688e-01), res. 3.295545e-22
# solve_mms_nd residual check: shift 9 (1.791220e+00), res. 3.867626e-23
# NDRATCOR heatbath: c_1*(R * Z^1 * R) = -2.670276e-01
# NDRATCOR heatbath: c_2*(R * Z^2 * R) = -1.220353e-06

Usually a third solve is run to compute the c_3 term (which then of course has an even smaller contribution).

My hunch is that we can safely ignore this as the contribution of the second term is just 1e-6 but still...