Closed ybyang closed 1 year ago
Hi @ybyang
We should be able to make this work. Looking at the interface code I suspect the issue is centered around the use of checker boarding. Do you need apply the Wuppertal smearing to a full fermion field, or to a single parity field.
The rb
refers to red black, eg selects which parity of the field. By default chroma only passes the black parity to QUDA, eg the second half of the field whereas we need Ron make the interface is setup for copying an entire field into QUDA.
Hi @maddyscientist
I need to apply the Wuppertal smearing to a full fermion field. I tried to malloc the memory with the size V*12*8 (both spinorIn and spinorOut) in Chroma and then call the performWuppertalnStep, but it doesn’t work either. So it should be the problem of the inv_param or gauge_param I set?
Regards,
Yi-Bo
On Feb 23, 2018, at 8:47 AM, maddyscientist notifications@github.com wrote:
Hi @ybyang https://github.com/ybyang We should be able to make this work. Looking at the interface code I suspect the issue is centered around the use of checker boarding. Do you need apply the Wuppertal smearing to a full fermion field, or to a single parity field.
The rb`
— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/lattice/quda/issues/681#issuecomment-367873005, or mute the thread https://github.com/notifications/unsubscribe-auth/AES9KB8jRBsuFUjp2n9xGQzWL5ko2cTiks5tXgqqgaJpZM4SOawg.
Hi @maddyscientist,
I tried the following setup for the inv_param, but the problem is still the same. I think it should be an acceptable choice for me if we have to do the smearing for 4 Dirac indices individually..
quda_inv_param.dslash_type = QUDA_LAPLACE_DSLASH;
quda_inv_param.gamma_basis = QUDA_DEGRAND_ROSSI_GAMMA_BASIS;
quda_inv_param.Ls = 1;
quda_inv_param.solution_type = QUDA_MAT_SOLUTION ;
quda_inv_param.solve_type = QUDA_NORMOP_PC_SOLVE;
Regards,
Yi-Bo
On Feb 23, 2018, at 9:01 AM, yibo.yang yangyibo@pa.msu.edu wrote:
Hi @maddyscientist
I need to apply the Wuppertal smearing to a full fermion field. I tried to malloc the memory with the size V*12*8 (both spinorIn and spinorOut) in Chroma and then call the performWuppertalnStep, but it doesn’t work either. So it should be the problem of the inv_param or gauge_param I set?
Regards,
Yi-Bo
On Feb 23, 2018, at 8:47 AM, maddyscientist <notifications@github.com mailto:notifications@github.com> wrote:
Hi @ybyang https://github.com/ybyang We should be able to make this work. Looking at the interface code I suspect the issue is centered around the use of checker boarding. Do you need apply the Wuppertal smearing to a full fermion field, or to a single parity field.
The rb`
— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/lattice/quda/issues/681#issuecomment-367873005, or mute the thread https://github.com/notifications/unsubscribe-auth/AES9KB8jRBsuFUjp2n9xGQzWL5ko2cTiks5tXgqqgaJpZM4SOawg.
Hi Yi-Bo,
Sorry for the slow response time on this.
I think I may need to support to the Chroma-QUDA interface to allow for exchange of full quark fields as opposed to single parity fields. Perhaps the best way to proceed is for you to send me your code and I can work on this directly. Can you send me your modified chroma code that calls the QUDA Wuppertal smearing, together with instructions for testing and I will work on this directly.
Hi,
I found that the attached reply didn’t appear in the issue 681 on the github website. I guess I should reply the address @reply.github, not the others?
Besides, if I want to try the QUDA Wuppertal smearing with quda, should I switch to some special branch? It seems that no one called the function wuppertalStep or performWuppertalnStep from the tests, even though the staggered_invert_test support the dslash-type laplace.
Thanks,
Yi-Bo
On Mar 23, 2018, at 9:42 AM, yibo.yang yangyibo@pa.msu.edu wrote:
Hi @maddyscientist,
I tried to pull the newest version of QUDA but the problem is still the same:
———— ERROR: Failed to clear error state an illegal memory access was encountered (rank 0, host nid06592, /ccs/home/ybyang1/src/quda/lib/tune.cpp:619 in tuneLaunch()) last kernel called was (name=N4quda15CopyColorSpinorIffLi1ELi3ENS_18CopyColorSpinorArgINS_11colorspinor11FloatNOrderIfLi1ELi3ELi2ELb0EEENS2_21SpaceSpinorColorOrderIfLi1ELi3EEEEEEE,volume=8x8x8x8,aux=out_stride=2048,in_stride=2048) Saving 5 sets of cached parameters to /lustre/atlas1/nph122/scratch/ybyang1/QUDA_RESOURCE/tunecache_error.tsv ————
I used a 8^4 lattice so the memory should be enough. Should I send the code to the email address mclark@nvidia.com mailto:mclark@nvidia.com directly?
Thanks,
Yi-Bo
On Mar 22, 2018, at 8:41 PM, maddyscientist <notifications@github.com mailto:notifications@github.com> wrote:
Hi Yi-Bo,
Sorry for the slow response time on this.
I think I may need to support to the Chroma-QUDA interface to allow for exchange of full quark fields as opposed to single parity fields. Perhaps the best way to proceed is for you to send me your code and I can work on this directly. Can you send me your modified chroma code that calls the QUDA Wuppertal smearing, together with instructions for testing and I will work on this directly.
This email message is for the sole use of the intended recipient(s) and may contain confidential information. Any unauthorized review, use, disclosure or distribution is prohibited. If you are not the intended recipient, please contact the sender by reply email and destroy all copies of the original message.
— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/lattice/quda/issues/681#issuecomment-375502891, or mute the thread https://github.com/notifications/unsubscribe-auth/AES9KN3V08KhVnAvZ24QAG69aE8tbO08ks5thETUgaJpZM4SOawg.
@cpviolator this is probably relevant for your work
Hi, @maddyscientist @cpviolator
Ok, I figured out what the problem is.
First, the quda_inv_param.input_location and quda_inv_param.output_location set from chroma should be QUDA_CPU_FIELD_LOCATION, otherwise the function performWuppertalnStep will not copy the input/output data from/to CPU correctly. Such a feature is different from that used in the inverter, and then confused me.
Second, there are two bugs in the function performWuppertalnStep:
The following hack works fine and the result agrees with that on CPU:
int comm_dim[4] = {};
// only switch on comms needed for directions with a derivative
for (int i = 0; i < 4; i++) {
comm_dim[i] = comm_dim_partitioned(i);
if (i == 3) comm_dim[i] = 0;
}
for (unsigned int i=0; i<nSteps; i++) {
if (i) in = out;
ApplyLaplace(out, in, *precise, 3, a, b, in, parity, false, comm_dim, profileWuppertal);
// ApplyLaplace(out, in, *precise, 3, a, b, in, parity, false, nullptr, profileWuppertal);
Besides, use quda_inv_param.dslash_type = QUDA_WILSON_DSLASH is fine now.
Great to hear this makes it work ok 😄
@ybyang reopening this bug, since the bug isn't actually fixed in mainline QUDA yet. Can you file a pull request against the develop branch with the fix please? Thx.
Closed by #1381
Hi all,
I found that I just need use the performWuppertalnStep in the interface_quda.cpp from chroma to do the gaussian smearing I need. So the things I need to do are just passing the correct parameters to QUDA. With the present parameters, I met the following error after the first PreTune:
I tried 8^4 and 24^3x64 lattice so it should not be a problem about running out the memory.
Any suggestion?
Yi-Bo
PS: What I used in Chroma are listed below: