`dslash_test` failing with `QUDA_REORDER_LOCATION=CPU`

[sbacchio]

In the current version of the develop, dslash_test is failing when QUDA_REORDER_LOCATION=CPU is set.

Reporducible:

export QUDA_REORDER_LOCATION=CPU
dslash_test --xdim 4 --ydim 4 --zdim 4 --tdim 4

Output:

Disabling GPU-Direct RDMA access Enabling peer-to-peer copy engine and direct load/store access Rank order is column major (t running fastest) [==========] Running 2 tests from 1 test suite. [----------] Global test environment set-up. [----------] 2 tests from DslashTest QUDA 1.1.0 (git 1.1.0-f8855bbec-sm_80) CUDA Driver version = 12010 CUDA Runtime version = 11080 Graphic driver version = 530.30.02 Found device 0: NVIDIA A100-SXM-64GB Using device 0: NVIDIA A100-SXM-64GB WARNING: Data reordering done on CPU (set with QUDA_REORDER_LOCATION=GPU/CPU) WARNING: Environment variable QUDA_RESOURCE_PATH is not set. WARNING: Caching of tuned parameters will be disabled. WARNING: Using device memory pool allocator WARNING: Using pinned memory pool allocator cublasCreated successfully [ RUN ] DslashTest.benchmark Randomizing fields... Sending gauge field to GPU Creating cudaSpinor with nParity = 1 Creating cudaSpinorOut with nParity = 1 Sending spinor field to GPU Source: CPU = 2.032196e+03, CUDA = 2.032196e+03 running the following test: prec recon dtest_type matpc_type dagger S_dim T_dimension Ls_dimension dslash_type niter single 18 Dslash even_even 0 4/ 4/ 4 8 16 wilson 100 Grid partition info: X Y Z T 0 0 0 0 Tuning... Executing 100 kernel loops... done. 10.659840us per kernel call 337920 flops per kernel call, 1320 flops per site 1344 bytes per site GFLOPS = 31.700288 GBYTES = 32.276656 Effective halo bi-directional bandwidth (GB/s) GPU = 0.000000 ( CPU = 0.000000, min = 0.000000 , max = 0.000000 ) for aggregate message size 0 bytes [ OK ] DslashTest.benchmark (128 ms) [ RUN ] DslashTest.verify Sending gauge field to GPU Creating cudaSpinor with nParity = 1 Creating cudaSpinorOut with nParity = 1 Sending spinor field to GPU Source: CPU = 2.032196e+03, CUDA = 2.032196e+03 running the following test: prec recon dtest_type matpc_type dagger S_dim T_dimension Ls_dimension dslash_type niter single 18 Dslash even_even 0 4/ 4/ 4 8 16 wilson 100 Grid partition info: X Y Z T 0 0 0 0 Calculating reference implementation...done. Tuning... Executing 2 kernel loops... done. 13.312000us per kernel call 337920 flops per kernel call, 1320 flops per site 1344 bytes per site GFLOPS = 25.384615 GBYTES = 25.846153 Effective halo bi-directional bandwidth (GB/s) GPU = 0.000000 ( CPU = 0.000000, min = 0.000000 , max = 0.000000 ) for aggregate message size 0 bytes Results: reference = 49469.949399, QUDA = 0.000000, L2 relative deviation = 1.000000e+00, max deviation = 9.964920e+00 0 fails = 255 1 fails = 254 2 fails = 256 3 fails = 256 4 fails = 256 5 fails = 255 6 fails = 256 7 fails = 256 8 fails = 254 9 fails = 255 10 fails = 256 11 fails = 255 12 fails = 255 13 fails = 256 14 fails = 253 15 fails = 256 16 fails = 255 17 fails = 255 18 fails = 254 19 fails = 254 20 fails = 255 21 fails = 256 22 fails = 255 23 fails = 254 1.000000e-01 Failures: 4494 / 6144 = 7.314453e-01 1.000000e-02 Failures: 5961 / 6144 = 9.702148e-01 1.000000e-03 Failures: 6122 / 6144 = 9.964193e-01 1.000000e-04 Failures: 6140 / 6144 = 9.993490e-01 1.000000e-05 Failures: 6144 / 6144 = 1.000000e+00 1.000000e-06 Failures: 6144 / 6144 = 1.000000e+00 1.000000e-07 Failures: 6144 / 6144 = 1.000000e+00 1.000000e-08 Failures: 6144 / 6144 = 1.000000e+00 1.000000e-09 Failures: 6144 / 6144 = 1.000000e+00 1.000000e-10 Failures: 6144 / 6144 = 1.000000e+00 1.000000e-11 Failures: 6144 / 6144 = 1.000000e+00 1.000000e-12 Failures: 6144 / 6144 = 1.000000e+00 1.000000e-13 Failures: 6144 / 6144 = 1.000000e+00 1.000000e-14 Failures: 6144 / 6144 = 1.000000e+00 1.000000e-15 Failures: 6144 / 6144 = 1.000000e+00 1.000000e-16 Failures: 6144 / 6144 = 1.000000e+00 /leonardo/home/userexternal/sbacchio/src/quda_new/tests/dslash_test.cpp:78: Failure Expected: (deviation) <= (tol), actual: 1 vs 0.0001 CPU and CUDA implementations do not agree [ FAILED ] DslashTest.verify (14 ms) initQuda Total time = 2.729 secs init = 2.729 secs (100.000%), with 2 calls at 1.364e+06 us per call total accounted = 2.729 secs (100.000%) total missing = 0.000 secs ( 0.000%) loadGaugeQuda Total time = 0.048 secs download = 0.045 secs ( 94.961%), with 2 calls at 2.264e+04 us per call init = 0.000 secs ( 0.369%), with 10 calls at 1.760e+01 us per call compute = 0.000 secs ( 0.002%), with 2 calls at 5.000e-01 us per call free = 0.000 secs ( 0.006%), with 73 calls at 4.110e-02 us per call total accounted = 0.045 secs ( 95.339%) total missing = 0.002 secs ( 4.661%) endQuda Total time = 0.002 secs free = 0.000 secs ( 0.120%), with 63 calls at 3.175e-02 us per call total accounted = 0.000 secs ( 0.120%) total missing = 0.002 secs ( 99.880%) initQuda-endQuda Total time = 2.874 secs QUDA Total time = 2.778 secs download = 0.045 secs ( 1.630%), with 2 calls at 2.264e+04 us per call init = 2.729 secs ( 98.230%), with 12 calls at 2.274e+05 us per call compute = 0.000 secs ( 0.000%), with 2 calls at 5.000e-01 us per call free = 0.000 secs ( 0.000%), with 136 calls at 4.412e-02 us per call total accounted = 2.774 secs ( 99.860%) total missing = 0.004 secs ( 0.140%) Device memory used = 0.4 MiB Pinned device memory used = 0.0 MiB Managed memory used = 0.0 MiB Shmem memory used = 0.0 MiB Page-locked host memory used = 0.3 MiB Total host memory used >= 1.1 MiB [----------] 2 tests from DslashTest (143 ms total) [----------] Global test environment tear-down [==========] 2 tests from 1 test suite ran. (2875 ms total) [ PASSED ] 1 test. [ FAILED ] 1 test, listed below: [ FAILED ] DslashTest.verify 1 FAILED TEST