search

openucx / ucx

Unified Communication X (mailing list - https://elist.ornl.gov/mailman/listinfo/ucx-group)
http://www.openucx.org
Other
1.15k stars 426 forks source link

Odd Crashes in UCX on POWER8 Systems with CUDA 10.1 Enabled (OpenMPI 4.0.1) #3580

Open nmhamster opened 5 years ago

nmhamster commented 5 years ago

Hi UCX Developers,

We are building our Trilinos tests against UCX 1.5.1 with OpenMPI 4.0.1, also configured for CUDA10.1 on our POWER8 test systems. We have a new bug in one our tests that looks like it is UCX related. The code in this sequence, opens a file, reads small chunks of it in, broadcasts it to all MPI ranks who then process it locally to develop local partitions of a global (parallel) matrix. This code has been in our test suite for upwards of 11 years and has run well across all sorts of machines.

(See: https://github.com/trilinos/Trilinos/issues/5033)

When run with UCX enabled for OpenMPI the job crashes with the following:

/ascldap/users/projects/ppc64le-pwr8-nvidia/openmpi/4.0.1/gcc/7.2.0/cuda/10.1.105/bin/mpiexec -n 4 -map-by socket:PE=4 /ascldap/users/sdhammo/git/trilinos-github-repo/build-muelu-gcc-720-cuda-101105/packages/muelu/test/maxwell/MueLu_Maxwell3D.exe --linAlgebra=Epetra
[white30:107180:0:107180] Caught signal 11 (Segmentation fault: address not mapped to object at address (nil))
==== backtrace ====
[white30:107181:0:107181] Caught signal 11 (Segmentation fault: address not mapped to object at address (nil))
==== backtrace ====
[white30:107182:0:107182] Caught signal 11 (Segmentation fault: address not mapped to object at address (nil))
==== backtrace ====
    0  /home/projects/ppc64le-pwr8-nvidia/ucx/1.5.1/cuda/10.1.105/lib/libucs.so.0(+0x24b40) [0x3fff1a464b40]
    1  /home/projects/ppc64le-pwr8-nvidia/ucx/1.5.1/cuda/10.1.105/lib/libucs.so.0(+0x24e84) [0x3fff1a464e84]
    2  [0x3fffb7740478]
    3  [0x3fffec76e810]
    4  /lib64/libc.so.6(strtoll+0x2c) [0x3fffa4e3675c]
    5  /lib64/libc.so.6(atoi+0x20) [0x3fffa4e31c30]
    0  /home/projects/ppc64le-pwr8-nvidia/ucx/1.5.1/cuda/10.1.105/lib/libucs.so.0(+0x24b40) [0x3fff002f4b40]
    1  /home/projects/ppc64le-pwr8-nvidia/ucx/1.5.1/cuda/10.1.105/lib/libucs.so.0(+0x24e84) [0x3fff002f4e84]
    2  [0x3fff98610478]
    3  [0x3fffedbca6b0]
    4  /lib64/libc.so.6(strtoll+0x2c) [0x3fff85d0675c]
    5  /lib64/libc.so.6(atoi+0x20) [0x3fff85d01c30]
    0  /home/projects/ppc64le-pwr8-nvidia/ucx/1.5.1/cuda/10.1.105/lib/libucs.so.0(+0x24b40) [0x3ffef4384b40]
    1  /home/projects/ppc64le-pwr8-nvidia/ucx/1.5.1/cuda/10.1.105/lib/libucs.so.0(+0x24e84) [0x3ffef4384e84]
    2  [0x3fff8d660478]
    3  [0x3fffea690fc0]
    4  /lib64/libc.so.6(strtoll+0x2c) [0x3fff7ad5675c]
    5  /lib64/libc.so.6(atoi+0x20) [0x3fff7ad51c30]
    6  /ascldap/users/sdhammo/git/trilinos-github-repo/build-muelu-gcc-720-cuda-101105/packages/muelu/test/maxwell/MueLu_Maxwell3D.exe() [0x156bc38c]
    7  /ascldap/users/sdhammo/git/trilinos-github-repo/build-muelu-gcc-720-cuda-101105/packages/muelu/test/maxwell/MueLu_Maxwell3D.exe() [0x156b9f8c]
    8  /ascldap/users/sdhammo/git/trilinos-github-repo/build-muelu-gcc-720-cuda-101105/packages/muelu/test/maxwell/MueLu_Maxwell3D.exe() [0x101f55a0]
    6  /ascldap/users/sdhammo/git/trilinos-github-repo/build-muelu-gcc-720-cuda-101105/packages/muelu/test/maxwell/MueLu_Maxwell3D.exe() [0x156bc38c]
    7  /ascldap/users/sdhammo/git/trilinos-github-repo/build-muelu-gcc-720-cuda-101105/packages/muelu/test/maxwell/MueLu_Maxwell3D.exe() [0x156b9f8c]
    8  /ascldap/users/sdhammo/git/trilinos-github-repo/build-muelu-gcc-720-cuda-101105/packages/muelu/test/maxwell/MueLu_Maxwell3D.exe() [0x101f55a0]
    9  /ascldap/users/sdhammo/git/trilinos-github-repo/build-muelu-gcc-720-cuda-101105/packages/muelu/test/maxwell/MueLu_Maxwell3D.exe() [0x101ab5f4]
   10  /ascldap/users/sdhammo/git/trilinos-github-repo/build-muelu-gcc-720-cuda-101105/packages/muelu/test/maxwell/MueLu_Maxwell3D.exe() [0x10176fd4]
   11  /ascldap/users/sdhammo/git/trilinos-github-repo/build-muelu-gcc-720-cuda-101105/packages/muelu/test/maxwell/MueLu_Maxwell3D.exe() [0x100d3e88]
    6  /ascldap/users/sdhammo/git/trilinos-github-repo/build-muelu-gcc-720-cuda-101105/packages/muelu/test/maxwell/MueLu_Maxwell3D.exe() [0x156bc38c]
    7  /ascldap/users/sdhammo/git/trilinos-github-repo/build-muelu-gcc-720-cuda-101105/packages/muelu/test/maxwell/MueLu_Maxwell3D.exe() [0x156b9f8c]
    8  /ascldap/users/sdhammo/git/trilinos-github-repo/build-muelu-gcc-720-cuda-101105/packages/muelu/test/maxwell/MueLu_Maxwell3D.exe() [0x101f55a0]
    9  /ascldap/users/sdhammo/git/trilinos-github-repo/build-muelu-gcc-720-cuda-101105/packages/muelu/test/maxwell/MueLu_Maxwell3D.exe() [0x101ab5f4]
   10  /ascldap/users/sdhammo/git/trilinos-github-repo/build-muelu-gcc-720-cuda-101105/packages/muelu/test/maxwell/MueLu_Maxwell3D.exe() [0x10176fd4]
   11  /ascldap/users/sdhammo/git/trilinos-github-repo/build-muelu-gcc-720-cuda-101105/packages/muelu/test/maxwell/MueLu_Maxwell3D.exe() [0x100d3e88]
   12  /ascldap/users/sdhammo/git/trilinos-github-repo/build-muelu-gcc-720-cuda-101105/packages/muelu/test/maxwell/MueLu_Maxwell3D.exe() [0x100d4d88]
   13  /lib64/libc.so.6(+0x25100) [0x3fffa4e15100]
   14  /lib64/libc.so.6(__libc_start_main+0xc4) [0x3fffa4e152f4]
    9  /ascldap/users/sdhammo/git/trilinos-github-repo/build-muelu-gcc-720-cuda-101105/packages/muelu/test/maxwell/MueLu_Maxwell3D.exe() [0x101ab5f4]
   10  /ascldap/users/sdhammo/git/trilinos-github-repo/build-muelu-gcc-720-cuda-101105/packages/muelu/test/maxwell/MueLu_Maxwell3D.exe() [0x10176fd4]
   11  /ascldap/users/sdhammo/git/trilinos-github-repo/build-muelu-gcc-720-cuda-101105/packages/muelu/test/maxwell/MueLu_Maxwell3D.exe() [0x100d3e88]
   12  /ascldap/users/sdhammo/git/trilinos-github-repo/build-muelu-gcc-720-cuda-101105/packages/muelu/test/maxwell/MueLu_Maxwell3D.exe() [0x100d4d88]
   13  /lib64/libc.so.6(+0x25100) [0x3fff7ad35100]
   14  /lib64/libc.so.6(__libc_start_main+0xc4) [0x3fff7ad352f4]
===================
===================
   12  /ascldap/users/sdhammo/git/trilinos-github-repo/build-muelu-gcc-720-cuda-101105/packages/muelu/test/maxwell/MueLu_Maxwell3D.exe() [0x100d4d88]
   13  /lib64/libc.so.6(+0x25100) [0x3fff85ce5100]
   14  /lib64/libc.so.6(__libc_start_main+0xc4) [0x3fff85ce52f4]

If we completely disable UCX:

/ascldap/users/projects/ppc64le-pwr8-nvidia/openmpi/4.0.1/gcc/7.2.0/cuda/10.1.105/bin/mpiexec --mca pml ^ucx -n 4 -map-by socket:PE=4 /ascldap/users/sdhammo/git/trilinos-github-repo/build-muelu-gcc-720-cuda-101105/packages/muelu/test/maxwell/MueLu_Maxwell3D.exe --linAlgebra=Epetra
nnzPerRowInA = 27.0705, nnzPerRowInB = 27.0705
MatrixMatrix::TwoMatrixAdd : space allocated per row = 82, max possible nnz per row in sum = 66, using static profiling
RefMaxwell (1,1) A_nodal: Computing Ac (MueLu::RAPFactory)
RefMaxwell (1,1) A_nodal: Transpose P (MueLu::TransPFactory)
matrixmatrix: kernel params ->
 [empty list]

transpose: use implicit = 0
rap: triple product = 0
rap: fix zero diagonals = 1
CheckMainDiagonal = 0
RepairMainDiagonal = 0
matrixmatrix: kernel params ->
 [empty list]
...

We get correct behavior.

For reference our environment is configured to disable UCX memory caching (UCX_MEMTYPE_CACHE=n) because of the responses to UCX Issue #3550.

Checking the debug reference to libucs.so.0 I get the following:

$ addr2line -e /home/projects/ppc64le-pwr8-nvidia/ucx/1.5.1/cuda/10.1.105/lib/libucs.so.0
0x24b40
/ascldap/users/sdhammo/git/ucx-github-repo/src/ucs/debug/debug.c:934

Which is the segmentation-fault handler firing. I don't seem to get other references to UCX libraries in the stack trace for this particular crash. Are there other memory operations in UCX we should try to disable to see if we can track this down at all?

Thanks for your help.

nmhamster commented 5 years ago

I should have added that all the communications here are on-node and host-only, this specific code is not using CUDA or any kernel/code offloads.

yosefe commented 5 years ago

@nmhamster the segfault is originating from MueLu_Maxwell3D.exe (frame 6) calling 'atoi', probably a corrupted string pointer (maybe as a result of using UCX). Is it possible to track in the application why the pointer is corrupted?

    4  /lib64/libc.so.6(strtoll+0x2c) [0x3fff7ad5675c]
    5  /lib64/libc.so.6(atoi+0x20) [0x3fff7ad51c30]
    6  /ascldap/users/sdhammo/git/trilinos-github-repo/build-muelu-gcc-720-cuda-101105/packages/muelu/test/maxwell/MueLu_Maxwell3D.exe() [0x156bc38c]