The copy_<to/from>_buffer methods of the various field types assumes the buffer is on the host - this forbids one from doing split and join fields from device buffers when GPU aware MPI can be used to avoid the additional host-device memory copies.
One could modify the copy_<to/from>_buffer methods to allow copying to/from device buffers when QUDA_ENABLE_GDR=1 and modify the usage above to use device_malloc instead of pinned_malloc when possible.
The
copy_<to/from>_buffer
methods of the various field types assumes the buffer is on the host - this forbids one from doing split and join fields from device buffers when GPU aware MPI can be used to avoid the additional host-device memory copies.CloverField::copy_<to/from>_buffer
in lib/clover_field.cpp.GaugeField::copy_<to/from>_buffer
in lib/gauge_field.cpp.ColorSpinorField::copy_<to/from>_buffer
in lib/color_spinor_field.cpp.Currently the usage is something like (in include/split_grid.h and lib/gauge_polyakov_loop.cu)
One could modify the
copy_<to/from>_buffer
methods to allow copying to/from device buffers whenQUDA_ENABLE_GDR=1
and modify the usage above to usedevice_malloc
instead ofpinned_malloc
when possible.