problem with staggered_dslash_test

[stevengottlieb]

Rich has requested a weak scaling study, so I am trying staggered_dslash_test. I am having trouble running multiple gpu jobs. I compiled QUDA for multi-gpu. Here is how I launch a job on a cray: aprun -n 4 -N 1 /N/u/sg/BigRed2/quda-0.7.0/tests//staggered_dslash_test --prec double --xdi m $nx --ydim $ny --zdim $nz --tdim $nt --xgridsize 1 --ygridsize 1 --zgridsize 2 --tgridsiz e 2 >>out_4procs.$ns

Here is what shows up in output: Tue Dec 16 09:11:07 EST 2014 -rwxr-xr-x 1 sg phys 192887697 Dec 4 02:25 /N/u/sg/BigRed2/quda-0.7.0/tests/staggered_dsla sh_test running the following test: prec recon test_type dagger S_dim T_dimension double 18 0 0 24/24/48 48 Grid partition info: X Y Z T 0 0 1 1 Found device 0: Tesla K20 Using device 0: Tesla K20 Setting NUMA affinity for device 0 to CPU core 0 WARNING: Environment variable QUDA_RESOURCE_PATH is not set. WARNING: Caching of tuned parameters will be disabled. Randomizing fields ...

[mathiaswagner]

Strange. It seems to work for me when using

mathwagn@aprun7:~/quda/tests> aprun -n4 -N1 ./staggered_dslash_test --prec double --xdim 24 --ydim 24 --zdim 48 --tdim 48 --xgridsize 1 --ygridsize 1 --zgridsize 2 --tgridsize 2

[AlexVaq]

You should set QUDA_RESOURCE_PATH to any path you wish (and that you have write permissions), otherwise QUDA will retune all the kernels every time you launch the job, and this can be extremely slow, so your results for the scaling test will be completely spoilt. Actually, if you don’t see anything for a while, it an mean that QUDA is tuning some kernels, but I would expect to see this a bit later, not right after “Randomizing fields…”.

El 16/12/2014, a las 18:51, stevengottlieb notifications@github.com escribió:

Rich has requested a weak scaling study, so I am trying staggered_dslash_test. I am having trouble running multiple gpu jobs. I compiled QUDA for multi-gpu. Here is how I launch a job on a cray: aprun -n 4 -N 1 /N/u/sg/BigRed2/quda-0.7.0/tests//staggered_dslash_test --prec double --xdi m $nx --ydim $ny --zdim $nz --tdim $nt --xgridsize 1 --ygridsize 1 --zgridsize 2 --tgridsiz e 2 >>out_4procs.$ns

Here is what shows up in output: Tue Dec 16 09:11:07 EST 2014 -rwxr-xr-x 1 sg phys 192887697 Dec 4 02:25 /N/u/sg/BigRed2/quda-0.7.0/tests/staggered_dsla sh_test running the following test: prec recon test_type dagger S_dim T_dimension double 18 0 0 24/24/48 48 Grid partition info: X Y Z T 0 0 1 1 Found device 0: Tesla K20 Using device 0: Tesla K20 Setting NUMA affinity for device 0 to CPU core 0 WARNING: Environment variable QUDA_RESOURCE_PATH is not set. WARNING: Caching of tuned parameters will be disabled. Randomizing fields ...

— Reply to this email directly or view it on GitHub https://github.com/lattice/quda/issues/200.

[maddyscientist]

So Steve, are you finding that it is hanging for you or is this the last thing it prints and then exits?

[stevengottlieb]

Hi Mike,

I have been traveling back from Mountain View. The job was not hanging, it just did not seem to use more than one GPU. Some of the larger volumes did fail when memory could not be allocated.

Here is a snippet from the compilation. If it looks like some flag is missing or if there are incompatible ones, please let me know.

CC -Wall -O3 -D__COMPUTE_CAPABILITY__=350 -DMULTI_GPU -DGPU_STAGGERED _DIRAC -DGPU_FATLINK -DGPU_UNITARIZE -DGPU_GAUGE_TOOLS -DGPU_GAUGE_FORCE -DGPU_GAUGE_TOOLS -DGPU_HISQ_FORCE -DGPU_STAGGERED_OPROD -DGPU_GAUGE_TOOLS -DGPU_DIRECT -DBUILD_QDP_INTERFACE -DBUILD_MILC_INTERFACE -DNUMA_AFFINITY -I/opt/nvidia/cudatoolkit/default/include -DMPI_COMMS -I/opt/cray/mpt/7.0.4/gni/mpich2-cray/83/include -I../include -Idslash_core -I. gauge_field.cpp -c -o gauge_field.o

Thanks, Steve

On Tue, 2014-12-16 at 10:56 -0800, mikeaclark wrote:

So Steve, are you finding that it is hanging for you or is this the last thing it prints and then exits?

— Reply to this email directly or view it on GitHub.

[maddyscientist]

Hi Steve,

I didn't see any reported dslash gflops which is why I asked if it was hanging. If you're telling me it is running to completion then I suspect all is running fine. One thing to note is that the test only prints out the performance for one gpu and not the aggregate performance. Could this be why you think it running on one gpu only? I guess we could update the tests to print aggregate performance as well, and also the number of GPUs it is running on.

This email message is for the sole use of the intended recipient(s) and may contain confidential information. Any unauthorized review, use, disclosure or distribution is prohibited. If you are not the intended recipient, please contact the sender by

reply email and destroy all copies of the original message.

[stevengottlieb]

Hi Mike,

Thanks for getting back to me. I was only showing a snippet of the output. I don't have a lot of experience running this test. What make me suspicious is this part of the output: running the following test: prec recon test_type dagger S_dim T_dimension double 18 0 0 24/24/48 48 Grid partition info: X Y Z T 0 0 1 1

I expected grid partition info to have 2 under Z and T, but now that I look and see that X and Y are zero, I was probably misinterpreting the meaning. Is the only possible value 0 or 1 depending on whether that dimension is cut?

The other issue that is causing me to worry is that several of the jobs are failing because they are running out of memory. On one node, I can run 40^4 with this information at the end of the run: Device memory used = 3155.7 MB Page-locked host memory used = 1591.2 MB Total host memory used >= 1986.8 MB

If I try to run 40 X 80^3 on 8 nodes, the job runs out of memory: running the following test: prec recon test_type dagger S_dim T_dimension double 18 0 0 40/80/80 80 Grid partition info: X Y Z T 0 1 1 1 Found device 0: Tesla K20 Using device 0: Tesla K20 Setting NUMA affinity for device 0 to CPU core 0 WARNING: Environment variable QUDA_RESOURCE_PATH is not set. WARNING: Caching of tuned parameters will be disabled. Randomizing fields ... Fat links sending...ERROR: Aborting (rank 6, host nid00193, malloc.cpp:156 in devicemalloc ()) last kernel called was (name=,volume=,aux=) ERROR: Failed to allocate device memory (cuda_gauge_field.cu:42 in cudaGaugeField())

In fact, on 8 nodes, only 24 X 48^3 runs. The larger volumes fail. Here is the report about memory usage for this run: Device memory used = 3488.3 MB Page-locked host memory used = 1974.7 MB Total host memory used >= 2384.7 MB

When I look at the 24^4 run on a single GPU, memory report is: Device memory used = 437.6 MB Page-locked host memory used = 231.3 MB Total host memory used >= 306.6 MB

This makes me wonder if the job is running only only 1 GPU or I don't understand the input parameters, i.e., are --xdim, -ydim, etc. the per GPU grid size or the total grid size?

I can make all the scripts and output available if we can find a convenient space. That might result in faster time to solution. I can put them on my webserver, dropbox (or IU's version of it) or Blue Waters. Any preference?

Thanks, Steve

On Wed, 2014-12-17 at 07:25 -0800, mikeaclark wrote:

Hi Steve,

I didn't see any reported dslash gflops which is why I asked if it was hanging. If you're telling me it is running to completion then I suspect all is running fine. One thing to note is that the test only prints out the performance for one gpu and not the aggregate performance. Could this be why you think it running on one gpu only? I guess we could update the tests to print aggregate performance as

well, and also the number of GPUs it is running on.

This email message is for the sole use of the intended recipient(s) and may contain confidential information. Any unauthorized review, use, disclosure or distribution is prohibited. If you are not the intended recipient, please contact the sender by

reply email and destroy all copies of the original message.

— Reply to this email directly or view it on GitHub.

[maddyscientist]

Is the only possible value 0 or 1 depending on whether that dimension is cut?

Yes.

With respect to memory, when running 40^4 on a single GPU, are you partitioning the dimensions (loop back communication)? You need to do this to have the same effective per GPU memory usage as the multi-GPU runs. The force routines in particular use so-called extended fields where we allocate a local field size that includes the halo regions. I suspect you are running afoul of these issues.

Having said that, I believe there's optimization that can be done there with respect to memory usage in particular in multi-gpu mode, in particular I believe some extended gauge fields allocate a non-zero halo region in dimensions that are not partitioned, this obviously leave scope for improvement.

We can open additional bugs where necessary to reduce memory consumption (post 0.7.0) if this is a priority for you.

[mathiaswagner]

I never happened used loopback communication. Could you give me some instructions to use it with QUDA?

[maddyscientist]

With the unit tests it's easy, you just use the --partition flag to force communication on. This is a 4-bit integer where each bit signifies communication in a different dimension, with X the least significant bit and T the most significant bit. E.g. --partition 11 has X, Y, T partitioning switched on.

[mathiaswagner]

Will try that. Thanks.

[maddyscientist]

Closing.