search

lattice / quda

QUDA is a library for performing calculations in lattice QCD on GPUs.
https://lattice.github.io/quda
Other
286 stars 94 forks source link

Milc-ordered gauge-field packing does not support half-precision fat links. #115

Closed jpfoley closed 11 years ago

jpfoley commented 11 years ago

I had been having problems with half-precision in the staggered DD solver. I now realise the problem arose because I had switched from QDP-ordered host gauge fields to MILC-ordered gauge fields in the interface. It turns out that the functions packMILCGaugeField and packQDPGaugeField defined in pack_gauge.h apply different operations to half-precision fields. packQDPGaugeField explicitly checks if the field is fattened. If so, it rescales the field elements before packing by dividing each element by the maximum real value contained in the field. I guess that this is done because the staggered fat link matrices are not unitary. packMILCGaugeField does not rescale. The half-precision staggered dslash kernels assume the fatlinks have been rescaled, so MILC packing does not work in that case. Note that single- and double-precision fields are not rescaled. I have added field rescaling to the MILC packing routines in the DD code, and will now add it to the master branch. I will also update the staggered_invert_test routine to use MILC-ordered host fields. Also, unpackQDPGaugeField currently does not work for HISQ fat links since it does not undo the rescaling.

Note that we can still use half-precision with the latest release if I switch back to QDP-ordered fields in milc_interface, although this seems like a retrograde step. Comments?

maddyscientist commented 11 years ago

I am glad you have finally worked out the source of your issues. Since no one actually uses half precision staggered in production, this isn't a really urgent fix. I would leave the milc interface using MILC ordering and fix master. I suspect we will release 0.5.1 soonish (month or two from now) so it's not a problem that will last long.

I guess that you will fix the unpackQDPGaugeField issue as well? Can you also update all other packing / unpacking routines to do this check for the field being field, so that say, the CPS would support half precision staggered if someone decided to use this?

jpfoley commented 11 years ago

Sure. I'll close this issue later this afternoon.

On 03/26/2013 12:58 PM, mikeaclark wrote:

I am glad you have finally worked out the source of your issues. Since no one actually uses half precision staggered in production, this isn't a really urgent fix. I would leave the milc interface using MILC ordering and fix master. I suspect we will release 0.5.1 soonish (month or two from now) so it's not a problem that will last long.

I guess that you will fix the unpackQDPGaugeField issue as well? Can you also update all other packing / unpacking routines to do this check for the field being field, so that say, the CPS would support half precision staggered if someone decided to use this?

— Reply to this email directly or view it on GitHub https://github.com/lattice/quda/issues/115#issuecomment-15480423.

jpfoley commented 11 years ago

Okay, I've modified the packing routines to handle half-precision fat links. It's a bit ugly since the code has an additional branch, but it should do the job. I have tested the MILC packing routines but not CPS or BQCD. However, they look okay to my eyes. Still, something to be conscious of. I have also switched to MILC-ordered host fields in staggered_invert_test.cpp. staggered_dslash_test.cpp still uses QDP ordering, but the code in these files needs to be refactored so that common functions are not repeated. This is not an immediate priority for me. I am closing this issue.