search

GEOS-DEV / GEOS

GEOS Simulation Framework
GNU Lesser General Public License v2.1
207 stars 83 forks source link

Running GPU GEOS on one node with 8 X 80GB A100 GPUs generates Umpire failures #2867

Closed drmichaeltcvx closed 9 months ago

drmichaeltcvx commented 9 months ago

Describe the bug Running GPU GEOS with MAELSTROM/usecases/francois/SPE10/flow/ on a single 8XA100 80GB node crashes with the following Umpire messages:

...
terminate called after throwing an instance of 'umpire::runtime_error'
terminate called after throwing an instance of '  what():  ! Umpire runtime_error [/dev/shm/mtml/src/GEOS/thirdPartyLibs/build-GPU-Hypre-GCC-CUDA_11.8-ompi_hpcx-OMP-relwithdebinfo/chai/src/chai/src/tpl/umpire/src/umpire/alloc/CudaPinnedAllocator.hpp:43]: cudaFreeHost( ptr = 0x2b230e000000 ) failed with error: an illegal memory access was encountered
    Backtrace: 13 frames
    0 0x2b20e783ef55 No dladdr: /data/saet/mtml/software/x86_64/RHEL7/GEOS/0.2.0/install-GPU-Hypre-GCC-CUDA_11.8-ompi_hpcx-OMP-relwithdebinfo/lib/libgeosx_core.so(_ZN6umpire4util49_GLOBAL__N__25f8fd63_16_ArrayManager_cpp_ab41d17d15build_backtraceEv+0x35) [0x2b20e783ef55]
    1 0x2b20e7841514 No dladdr: /data/saet/mtml/software/x86_64/RHEL7/GEOS/0.2.0/install-GPU-Hypre-GCC-CUDA_11.8-ompi_hpcx-OMP-relwithdebinfo/lib/libgeosx_core.so(_ZNK6umpire13runtime_error7messageB5cxx11Ev+0x44) [0x2b20e7841514]
    2 0x2b20e786135f No dladdr: /data/saet/mtml/software/x86_64/RHEL7/GEOS/0.2.0/install-GPU-Hypre-GCC-CUDA_11.8-ompi_hpcx-OMP-relwithdebinfo/lib/libgeosx_core.so(_ZN6umpire13runtime_errorC2ERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEES8_i+0x8f) [0x2b20e786135f]
    3 0x2b20e78718e1 No dladdr: /data/saet/mtml/software/x86_64/RHEL7/GEOS/0.2.0/install-GPU-Hypre-GCC-CUDA_11.8-ompi_hpcx-OMP-relwithdebinfo/lib/libgeosx_core.so(_ZN6umpire5alloc19CudaPinnedAllocator10deallocateEPv+0x351) [0x2b20e78718e1]
    4 0x2b20e24a0f86 No dladdr: /data/saet/mtml/software/x86_64/RHEL7/GEOS/0.2.0/install-GPU-Hypre-GCC-CUDA_11.8-ompi_hpcx-OMP-relwithdebinfo/lib/libgeosx_core.so(_ZN6umpire9Allocator10deallocateEPv+0x226) [0x2b20e24a0f86]
    5 0x2b20e2496c21 No dladdr: /data/saet/mtml/software/x86_64/RHEL7/GEOS/0.2.0/install-GPU-Hypre-GCC-CUDA_11.8-ompi_hpcx-OMP-relwithdebinfo/lib/libgeosx_core.so(_ZN4geos15DomainPartitionD2Ev+0x61) [0x2b20e2496c21]
    6 0x2b20e2496dc9 No dladdr: /data/saet/mtml/software/x86_64/RHEL7/GEOS/0.2.0/install-GPU-Hypre-GCC-CUDA_11.8-ompi_hpcx-OMP-relwithdebinfo/lib/libgeosx_core.so(_ZN4geos15DomainPartitionD0Ev+0x9) [0x2b20e2496dc9]
    7 0x2b20e1b5a6a2 No dladdr: /data/saet/mtml/software/x86_64/RHEL7/GEOS/0.2.0/install-GPU-Hypre-GCC-CUDA_11.8-ompi_hpcx-OMP-relwithdebinfo/lib/libgeosx_core.so(_ZN4geos14dataRepository5GroupD1Ev+0x8f2) [0x2b20e1b5a6a2]
    8 0x2b20e7344bb9 No dladdr: /data/saet/mtml/software/x86_64/RHEL7/GEOS/0.2.0/install-GPU-Hypre-GCC-CUDA_11.8-ompi_hpcx-OMP-relwithdebinfo/lib/libgeosx_core.so(_ZN4geos14ProblemManagerD0Ev+0x9) [0x2b20e7344bb9]
    9 0x2b20e7341056 No dladdr: /data/saet/mtml/software/x86_64/RHEL7/GEOS/0.2.0/install-GPU-Hypre-GCC-CUDA_11.8-ompi_hpcx-OMP-relwithdebinfo/lib/libgeosx_core.so(_ZN4geos10GeosxStateD2Ev+0x3f6) [0x2b20e7341056]
    10 0x40d2dc No dladdr: /data/saet/mtml/software/x86_64/RHEL7/GEOS/0.2.0/install-GPU-Hypre-GCC-CUDA_11.8-ompi_hpcx-OMP-relwithdebinfo/bin/geosx() [0x40d2dc]
    11 0x2b214f201555 No dladdr: /lib64/libc.so.6(__libc_start_main+0xf5) [0x2b214f201555]
    12 0x40e25e No dladdr: /data/saet/mtml/software/x86_64/RHEL7/GEOS/0.2.0/install-GPU-Hypre-GCC-CUDA_11.8-ompi_hpcx-OMP-relwithdebinfo/bin/geosx() [0x40e25e]

umpire::runtime_error'
  what():  ! Umpire runtime_error [/dev/shm/mtml/src/GEOS/thirdPartyLibs/build-GPU-Hypre-GCC-CUDA_11.8-ompi_hpcx-OMP-relwithdebinfo/chai/src/chai/src/tpl/umpire/src/umpire/alloc/CudaPinnedAllocator.hpp:43]: cudaFreeHost( ptr = 0x2afef0000000 ) failed with error: an illegal memory access was encountered
    Backtrace: 13 frames
    0 0x2afcca62bf55 No dladdr: /data/saet/mtml/software/x86_64/RHEL7/GEOS/0.2.0/install-GPU-Hypre-GCC-CUDA_11.8-ompi_hpcx-OMP-relwithdebinfo/lib/libgeosx_core.so(_ZN6umpire4util49_GLOBAL__N__25f8fd63_16_ArrayManager_cpp_ab41d17d15build_backtraceEv+0x35) [0x2afcca62bf55]
    1 0x2afcca62e514 No dladdr: /data/saet/mtml/software/x86_64/RHEL7/GEOS/0.2.0/install-GPU-Hypre-GCC-CUDA_11.8-ompi_hpcx-OMP-relwithdebinfo/lib/libgeosx_core.so(_ZNK6umpire13runtime_error7messageB5cxx11Ev+0x44) [0x2afcca62e514]
    2 0x2afcca64e35f No dladdr: /data/saet/mtml/software/x86_64/RHEL7/GEOS/0.2.0/install-GPU-Hypre-GCC-CUDA_11.8-ompi_hpcx-OMP-relwithdebinfo/lib/libgeosx_core.so(_ZN6umpire13runtime_errorC2ERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEES8_i+0x8f) [0x2afcca64e35f]
    3 0x2afcca65e8e1 No dladdr: /data/saet/mtml/software/x86_64/RHEL7/GEOS/0.2.0/install-GPU-Hypre-GCC-CUDA_11.8-ompi_hpcx-OMP-relwithdebinfo/lib/libgeosx_core.so(_ZN6umpire5alloc19CudaPinnedAllocator10deallocateEPv+0x351) [0x2afcca65e8e1]
    4 0x2afcc528df86 No dladdr: /data/saet/mtml/software/x86_64/RHEL7/GEOS/0.2.0/install-GPU-Hypre-GCC-CUDA_11.8-ompi_hpcx-OMP-relwithdebinfo/lib/libgeosx_core.so(_ZN6umpire9Allocator10deallocateEPv+0x226) [0x2afcc528df86]
    5 0x2afcc5283c21 No dladdr: /data/saet/mtml/software/x86_64/RHEL7/GEOS/0.2.0/install-GPU-Hypre-GCC-CUDA_11.8-ompi_hpcx-OMP-relwithdebinfo/lib/libgeosx_core.so(_ZN4geos15DomainPartitionD2Ev+0x61) [0x2afcc5283c21]
    6 0x2afcc5283dc9 No dladdr: /data/saet/mtml/software/x86_64/RHEL7/GEOS/0.2.0/install-GPU-Hypre-GCC-CUDA_11.8-ompi_hpcx-OMP-relwithdebinfo/lib/libgeosx_core.so(_ZN4geos15DomainPartitionD0Ev+0x9) [0x2afcc5283dc9]
    7 0x2afcc49476a2 No dladdr: /data/saet/mtml/software/x86_64/RHEL7/GEOS/0.2.0/install-GPU-Hypre-GCC-CUDA_11.8-ompi_hpcx-OMP-relwithdebinfo/lib/libgeosx_core.so(_ZN4geos14dataRepository5GroupD1Ev+0x8f2) [0x2afcc49476a2]
    8 0x2afcca131bb9 No dladdr: /data/saet/mtml/software/x86_64/RHEL7/GEOS/0.2.0/install-GPU-Hypre-GCC-CUDA_11.8-ompi_hpcx-OMP-relwithdebinfo/lib/libgeosx_core.so(_ZN4geos14ProblemManagerD0Ev+0x9) [0x2afcca131bb9]
    9 0x2afcca12e056 No dladdr: /data/saet/mtml/software/x86_64/RHEL7/GEOS/0.2.0/install-GPU-Hypre-GCC-CUDA_11.8-ompi_hpcx-OMP-relwithdebinfo/lib/libgeosx_core.so(_ZN4geos10GeosxStateD2Ev+0x3f6) [0x2afcca12e056]
    10 0x40d2dc No dladdr: /data/saet/mtml/software/x86_64/RHEL7/GEOS/0.2.0/install-GPU-Hypre-GCC-CUDA_11.8-ompi_hpcx-OMP-relwithdebinfo/bin/geosx() [0x40d2dc]
    11 0x2afd31fee555 No dladdr: /lib64/libc.so.6(__libc_start_main+0xf5) [0x2afd31fee555]
    12 0x40e25e No dladdr: /data/saet/mtml/software/x86_64/RHEL7/GEOS/0.2.0/install-GPU-Hypre-GCC-CUDA_11.8-ompi_hpcx-OMP-relwithdebinfo/bin/geosx() [0x40e25e]

...

To Reproduce Steps to reproduce the behavior:

  1. Run GEOS on SPE10_refined.xml :

mpirun --hostfile ./gpnpusc600002f.hosttab -x LD_LIBRARY_PATH -x V -x GPUMPICLI -x MPI -x MPIVER -x N_gpus -x GPU_cpu_aff_path -x GPU_mem_aff_path --np 5 --map-by ppr:5:node:PE=19 /home/mtml/cs691/utils/bin/map_ranks_gpus.sh /data/saet/mtml/software/x86_64/RHEL7/GEOS/0.2.0/install-GPU-Hypre-GCC-CUDA_11.8-ompi_hpcx-OMP-relwithdebinfo/bin/geosx -i ./SPE10_refined.xml -t runtime-report,max_column_width=200,calc.inclusive,mpi-report -x 1 -y 5 -z 1

  1. Note map_ranks_gpus.sh just selects a GPU unit for the current rank
  2. Scroll down to the end after the Caliper output and you will see Umpire messages
  3. See error

Minimal case : run the SPE10_refined.xml case.

Expected behavior GPU GEOS is expected to run to completion.

Screenshots This is from a run with 4 ranks and 1 OMP thread and 1 GPU per rank.

Num ranks: 4
Max threads: 1
GEOSX version: 0.2.0 (develop, sha1: c60be255c)
  - c++ compiler: gcc 10.4.0
  - cuda compiler version: 11.8
  - openmp version: 201511
  - MPI version: Open MPI v4.1.5rc2, package: Open MPI root@hpc-kernel-03 Distribution, ident: 4.1.5rc2, repo rev: v4.1.5rc1-16-g5980bac633, Unreleased developer copy
  - HDF5 version: 1.12.1
  - Conduit version: 0.8.2
  - VTK version: 9.2.6
  - RAJA version: 2023.6.1
  - umpire version: 2023.6.0
  -  adiak version: ..
  - caliper version: 2.10.0
  - METIS version: 5.1.0
  - PARAMETIS version: 4.0.3
  - scotch version: 7.0.3
  - superlu_dist version: 6.3.0
  - suitesparse version: 5.7.9
  - Python3 version: 3.10.9
  - hypre release version: 2.29.0
Started at 2023-11-20 20:46:29.765941326
Adding Event: PeriodicEvent, solverApplications
Adding Mesh: InternalMesh, mesh
Adding Geometric Object: Box, source
Adding Geometric Object: Box, sink1
Adding Geometric Object: Box, sink2
Adding Geometric Object: Box, sink3
Adding Geometric Object: Box, sink4
Adding Solver of type CompositionalMultiphaseFVM, named compflow
   TableFunction: permxFunc
   TableFunction: permyFunc
   TableFunction: permzFunc
   TableFunction: poroFunc
   TableFunction: B_o_table
   TableFunction: visc_o_table
Adding Output: VTK, vtkOutput
Adding Object CellElementRegion named region from ObjectManager::Catalog.
mesh: total number of nodes = 8512992
mesh: total number of elems = 8370882
regionQuadrature: meshBodyName, meshLevelName, regionName, subRegionName = mesh, Level0, region, block
mesh/Level0/region/block/fluid allocated 1 quadrature points
mesh/Level0/region/block/rock allocated 1 quadrature points
mesh/Level0/region/block/relperm allocated 1 quadrature points
mesh: importing field data from mesh dataset
Time: 0.00e+00 s, dt: 10000 s, Cycle: 0
    Attempt:  0, ConfigurationIter:  0, NewtonIter:  0
        ( Rflow ) = ( 2.23e+03 )        ( R ) = ( 2.23e+03 )
        MGR preconditioner: numComponentsPerField = [3]
        Linear Solver | Success | Iterations: 73 | Final Rel Res: 0.00091774 | Make Restrictor Time: 0 | Compute Auu Time: 0 | SC Filter Time: 0 | Setup Time: 2.32312 s | Solve Time: 3.12936 s
        compflow: Max pressure change: 18348874.884 Pa (before scaling)
        compflow: Max component density change: 1165.235 kg/m3 (before scaling)
        compflow: Global solution scaling factor = 0.13558173460316655
Path                                                                                     Min time/rank Max time/rank Avg time/rank Time %    
geos::GeosxState::initializeDataRepository                                                   15.522277     15.525297     15.524500 49.934891 
  geos::ProblemManager::problemSetup                                                         15.489391     15.494013     15.492518 49.832020 
    geos::ProblemManager::generateMesh                                                       14.826989     14.940992     14.886811 47.883751 
      geos::InternalMeshGenerator::fillCellBlockManager                                       4.114430      4.133383      4.122174 13.259061 

...

terminate called after throwing an instance of 'umpire::runtime_error'
terminate called after throwing an instance of '  what():  ! Umpire runtime_error [/dev/shm/mtml/src/GEOS/thirdPartyLibs/build-GPU-Hypre-GCC-CUDA_11.8-ompi_hpcx-OMP-relwithdebinfo/chai/src/chai/src/tpl/umpire/src/umpire/alloc/CudaPinnedAllocator.hpp:43]: cudaFreeHost( ptr = 0x2b230e000000 ) failed with error: an illegal memory access was encountered
    Backtrace: 13 frames
    0 0x2b20e783ef55 No dladdr: /data/saet/mtml/software/x86_64/RHEL7/GEOS/0.2.0/install-GPU-Hypre-GCC-CUDA_11.8-ompi_hpcx-OMP-relwithdebinfo/lib/libgeosx_core.so(_ZN6umpire4util49_GLOBAL__N__25f8fd63_16_ArrayManager_cpp_ab41d17d15build_backtraceEv+0x35) [0x2b20e783ef55]
    1 0x2b20e7841514 No dladdr: /data/saet/mtml/software/x86_64/RHEL7/GEOS/0.2.0/install-GPU-Hypre-GCC-CUDA_11.8-ompi_hpcx-OMP-relwithdebinfo/lib/libgeosx_core.so(_ZNK6umpire13runtime_error7messageB5cxx11Ev+0x44) [0x2b20e7841514]
    2 0x2b20e786135f No dladdr: /data/saet/mtml/software/x86_64/RHEL7/GEOS/0.2.0/install-GPU-Hypre-GCC-CUDA_11.8-ompi_hpcx-OMP-relwithdebinfo/lib/libgeosx_core.so(_ZN6umpire13runtime_errorC2ERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEES8_i+0x8f) [0x2b20e786135f]
    3 0x2b20e78718e1 No dladdr: /data/saet/mtml/software/x86_64/RHEL7/GEOS/0.2.0/install-GPU-Hypre-GCC-CUDA_11.8-ompi_hpcx-OMP-relwithdebinfo/lib/libgeosx_core.so(_ZN6umpire5alloc19CudaPinnedAllocator10deallocateEPv+0x351) [0x2b20e78718e1]
    4 0x2b20e24a0f86 No dladdr: /data/saet/mtml/software/x86_64/RHEL7/GEOS/0.2.0/install-GPU-Hypre-GCC-CUDA_11.8-ompi_hpcx-OMP-relwithdebinfo/lib/libgeosx_core.so(_ZN6umpire9Allocator10deallocateEPv+0x226) [0x2b20e24a0f86]
    5 0x2b20e2496c21 No dladdr: /data/saet/mtml/software/x86_64/RHEL7/GEOS/0.2.0/install-GPU-Hypre-GCC-CUDA_11.8-ompi_hpcx-OMP-relwithdebinfo/lib/libgeosx_core.so(_ZN4geos15DomainPartitionD2Ev+0x61) [0x2b20e2496c21]
    6 0x2b20e2496dc9 No dladdr: /data/saet/mtml/software/x86_64/RHEL7/GEOS/0.2.0/install-GPU-Hypre-GCC-CUDA_11.8-ompi_hpcx-OMP-relwithdebinfo/lib/libgeosx_core.so(_ZN4geos15DomainPartitionD0Ev+0x9) [0x2b20e2496dc9]
    7 0x2b20e1b5a6a2 No dladdr: /data/saet/mtml/software/x86_64/RHEL7/GEOS/0.2.0/install-GPU-Hypre-GCC-CUDA_11.8-ompi_hpcx-OMP-relwithdebinfo/lib/libgeosx_core.so(_ZN4geos14dataRepository5GroupD1Ev+0x8f2) [0x2b20e1b5a6a2]
    8 0x2b20e7344bb9 No dladdr: /data/saet/mtml/software/x86_64/RHEL7/GEOS/0.2.0/install-GPU-Hypre-GCC-CUDA_11.8-ompi_hpcx-OMP-relwithdebinfo/lib/libgeosx_core.so(_ZN4geos14ProblemManagerD0Ev+0x9) [0x2b20e7344bb9]
    9 0x2b20e7341056 No dladdr: /data/saet/mtml/software/x86_64/RHEL7/GEOS/0.2.0/install-GPU-Hypre-GCC-CUDA_11.8-ompi_hpcx-OMP-relwithdebinfo/lib/libgeosx_core.so(_ZN4geos10GeosxStateD2Ev+0x3f6) [0x2b20e7341056]
    10 0x40d2dc No dladdr: /data/saet/mtml/software/x86_64/RHEL7/GEOS/0.2.0/install-GPU-Hypre-GCC-CUDA_11.8-ompi_hpcx-OMP-relwithdebinfo/bin/geosx() [0x40d2dc]
    11 0x2b214f201555 No dladdr: /lib64/libc.so.6(__libc_start_main+0xf5) [0x2b214f201555]
    12 0x40e25e No dladdr: /data/saet/mtml/software/x86_64/RHEL7/GEOS/0.2.0/install-GPU-Hypre-GCC-CUDA_11.8-ompi_hpcx-OMP-relwithdebinfo/bin/geosx() [0x40e25e]

umpire::runtime_error'
  what():  ! Umpire runtime_error [/dev/shm/mtml/src/GEOS/thirdPartyLibs/build-GPU-Hypre-GCC-CUDA_11.8-ompi_hpcx-OMP-relwithdebinfo/chai/src/chai/src/tpl/umpire/src/umpire/alloc/CudaPinnedAllocator.hpp:43]: cudaFreeHost( ptr = 0x2afef0000000 ) failed with error: an illegal memory access was encountered
    Backtrace: 13 frames
    0 0x2afcca62bf55 No dladdr: /data/saet/mtml/software/x86_64/RHEL7/GEOS/0.2.0/install-GPU-Hypre-GCC-CUDA_11.8-ompi_hpcx-OMP-relwithdebinfo/lib/libgeosx_core.so(_ZN6umpire4util49_GLOBAL__N__25f8fd63_16_ArrayManager_cpp_ab41d17d15build_backtraceEv+0x35) [0x2afcca62bf55]
    1 0x2afcca62e514 No dladdr: /data/saet/mtml/software/x86_64/RHEL7/GEOS/0.2.0/install-GPU-Hypre-GCC-CUDA_11.8-ompi_hpcx-OMP-relwithdebinfo/lib/libgeosx_core.so(_ZNK6umpire13runtime_error7messageB5cxx11Ev+0x44) [0x2afcca62e514]
    2 0x2afcca64e35f No dladdr: /data/saet/mtml/software/x86_64/RHEL7/GEOS/0.2.0/install-GPU-Hypre-GCC-CUDA_11.8-ompi_hpcx-OMP-relwithdebinfo/lib/libgeosx_core.so(_ZN6umpire13runtime_errorC2ERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEES8_i+0x8f) [0x2afcca64e35f]
    3 0x2afcca65e8e1 No dladdr: /data/saet/mtml/software/x86_64/RHEL7/GEOS/0.2.0/install-GPU-Hypre-GCC-CUDA_11.8-ompi_hpcx-OMP-relwithdebinfo/lib/libgeosx_core.so(_ZN6umpire5alloc19CudaPinnedAllocator10deallocateEPv+0x351) [0x2afcca65e8e1]
    4 0x2afcc528df86 No dladdr: /data/saet/mtml/software/x86_64/RHEL7/GEOS/0.2.0/install-GPU-Hypre-GCC-CUDA_11.8-ompi_hpcx-OMP-relwithdebinfo/lib/libgeosx_core.so(_ZN6umpire9Allocator10deallocateEPv+0x226) [0x2afcc528df86]
    5 0x2afcc5283c21 No dladdr: /data/saet/mtml/software/x86_64/RHEL7/GEOS/0.2.0/install-GPU-Hypre-GCC-CUDA_11.8-ompi_hpcx-OMP-relwithdebinfo/lib/libgeosx_core.so(_ZN4geos15DomainPartitionD2Ev+0x61) [0x2afcc5283c21]
    6 0x2afcc5283dc9 No dladdr: /data/saet/mtml/software/x86_64/RHEL7/GEOS/0.2.0/install-GPU-Hypre-GCC-CUDA_11.8-ompi_hpcx-OMP-relwithdebinfo/lib/libgeosx_core.so(_ZN4geos15DomainPartitionD0Ev+0x9) [0x2afcc5283dc9]
    7 0x2afcc49476a2 No dladdr: /data/saet/mtml/software/x86_64/RHEL7/GEOS/0.2.0/install-GPU-Hypre-GCC-CUDA_11.8-ompi_hpcx-OMP-relwithdebinfo/lib/libgeosx_core.so(_ZN4geos14dataRepository5GroupD1Ev+0x8f2) [0x2afcc49476a2]
    8 0x2afcca131bb9 No dladdr: /data/saet/mtml/software/x86_64/RHEL7/GEOS/0.2.0/install-GPU-Hypre-GCC-CUDA_11.8-ompi_hpcx-OMP-relwithdebinfo/lib/libgeosx_core.so(_ZN4geos14ProblemManagerD0Ev+0x9) [0x2afcca131bb9]
    9 0x2afcca12e056 No dladdr: /data/saet/mtml/software/x86_64/RHEL7/GEOS/0.2.0/install-GPU-Hypre-GCC-CUDA_11.8-ompi_hpcx-OMP-relwithdebinfo/lib/libgeosx_core.so(_ZN4geos10GeosxStateD2Ev+0x3f6) [0x2afcca12e056]
    10 0x40d2dc No dladdr: /data/saet/mtml/software/x86_64/RHEL7/GEOS/0.2.0/install-GPU-Hypre-GCC-CUDA_11.8-ompi_hpcx-OMP-relwithdebinfo/bin/geosx() [0x40d2dc]
    11 0x2afd31fee555 No dladdr: /lib64/libc.so.6(__libc_start_main+0xf5) [0x2afd31fee555]
    12 0x40e25e No dladdr: /data/saet/mtml/software/x86_64/RHEL7/GEOS/0.2.0/install-GPU-Hypre-GCC-CUDA_11.8-ompi_hpcx-OMP-relwithdebinfo/bin/geosx() [0x40e25e]

:

Platform (please complete the following information):

Additional context Add any other context about the problem here.

drmichaeltcvx commented 9 months ago

Running with the smaller MAELSTROM/usecases/francois/SPE10/flow case on an 8 X100 40GB GPU node as

mpirun --hostfile ./gpnpusc500000x.hosttab -x LD_LIBRARY_PATH -x V -x GPUMPICLI -x MPI -x MPIVER -x N_gpus -x GPU_cpu_aff_path -x GPU_mem_aff_path --np 8 --map-by ppr:8:node:PE=12 /home/mtml/cs691/utils/bin/map_ranks_gpus.sh /data/saet/mtml/software/x86_64/RHEL7/GEOS/0.2.0/install-GPU-Hypre-GCC-CUDA_11.8-ompi_hpcx-OMP-relwithdebinfo/bin/geosx -i ./SPE10_small.xml -t runtime-report,max_column_

still crashes Umpire:

...
terminate called after throwing an instance of 'umpire::runtime_error'
terminate called after throwing an instance of 'umpire::runtime_error'
  what():  ! Umpire runtime_error [/dev/shm/mtml/src/GEOS/thirdPartyLibs/build-GPU-Hypre-GCC-CUDA_11.8-ompi_hpcx-OMP-relwithdebinfo/chai/src/chai/src/tpl/umpire/src/umpire/alloc/CudaPinnedAllocator.hpp:43]: cudaFreeHost( ptr = 0x2b688e000000 ) failed with error: an illegal memory access was encountered
    Backtrace: 13 frames
    0 0x2b672e279f55 No dladdr: /data/saet/mtml/software/x86_64/RHEL7/GEOS/0.2.0/install-GPU-Hypre-GCC-CUDA_11.8-ompi_hpcx-OMP-relwithdebinfo/lib/libgeosx_core.so(_ZN6umpire4util49_GLOBAL__N__25f8fd63_16_ArrayManager_cpp_ab41d17d15build_backtraceEv+0x35) [0x2b672e279f55]
    1 0x2b672e27c514 No dladdr: /data/saet/mtml/software/x86_64/RHEL7/GEOS/0.2.0/install-GPU-Hypre-GCC-CUDA_11.8-ompi_hpcx-OMP-relwithdebinfo/lib/libgeosx_core.so(_ZNK6umpire13runtime_error7messageB5cxx11Ev+0x44) [0x2b672e27c514]
    2 0x2b672e29c35f No dladdr: /data/saet/mtml/software/x86_64/RHEL7/GEOS/0.2.0/install-GPU-Hypre-GCC-CUDA_11.8-ompi_hpcx-OMP-relwithdebinfo/lib/libgeosx_core.so(_ZN6umpire13runtime_errorC2ERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEES8_i+0x8f) [0x2b672e29c35f]
    3 0x2b672e2ac8e1 No dladdr: /data/saet/mtml/software/x86_64/RHEL7/GEOS/0.2.0/install-GPU-Hypre-GCC-CUDA_11.8-ompi_hpcx-OMP-relwithdebinfo/lib/libgeosx_core.so(_ZN6umpire5alloc19CudaPinnedAllocator10deallocateEPv+0x351) [0x2b672e2ac8e1]
    4 0x2b6728edbf86 No dladdr: /data/saet/mtml/software/x86_64/RHEL7/GEOS/0.2.0/install-GPU-Hypre-GCC-CUDA_11.8-ompi_hpcx-OMP-relwithdebinfo/lib/libgeosx_core.so(_ZN6umpire9Allocator10deallocateEPv+0x226) [0x2b6728edbf86]
    5 0x2b6728ed1c21 No dladdr: /data/saet/mtml/software/x86_64/RHEL7/GEOS/0.2.0/install-GPU-Hypre-GCC-CUDA_11.8-ompi_hpcx-OMP-relwithdebinfo/lib/libgeosx_core.so(_ZN4geos15DomainPartitionD2Ev+0x61) [0x2b6728ed1c21]
    6 0x2b6728ed1dc9 No dladdr: /data/saet/mtml/software/x86_64/RHEL7/GEOS/0.2.0/install-GPU-Hypre-GCC-CUDA_11.8-ompi_hpcx-OMP-relwithdebinfo/lib/libgeosx_core.so(_ZN4geos15DomainPartitionD0Ev+0x9) [0x2b6728ed1dc9]
    7 0x2b67285956a2 No dladdr: /data/saet/mtml/software/x86_64/RHEL7/GEOS/0.2.0/install-GPU-Hypre-GCC-CUDA_11.8-ompi_hpcx-OMP-relwithdebinfo/lib/libgeosx_core.so(_ZN4geos14dataRepository5GroupD1Ev+0x8f2) [0x2b67285956a2]
    8 0x2b672dd7fbb9 No dladdr: /data/saet/mtml/software/x86_64/RHEL7/GEOS/0.2.0/install-GPU-Hypre-GCC-CUDA_11.8-ompi_hpcx-OMP-relwithdebinfo/lib/libgeosx_core.so(_ZN4geos14ProblemManagerD0Ev+0x9) [0x2b672dd7fbb9]
    9 0x2b672dd7c056 No dladdr: /data/saet/mtml/software/x86_64/RHEL7/GEOS/0.2.0/install-GPU-Hypre-GCC-CUDA_11.8-ompi_hpcx-OMP-relwithdebinfo/lib/libgeosx_core.so(_ZN4geos10GeosxStateD2Ev+0x3f6) [0x2b672dd7c056]
    10 0x40d2dc No dladdr: /data/saet/mtml/software/x86_64/RHEL7/GEOS/0.2.0/install-GPU-Hypre-GCC-CUDA_11.8-ompi_hpcx-OMP-relwithdebinfo/bin/geosx() [0x40d2dc]
    11 0x2b6795c3c555 No dladdr: /lib64/libc.so.6(__libc_start_main+0xf5) [0x2b6795c3c555]
    12 0x40e25e No dladdr: /data/saet/mtml/software/x86_64/RHEL7/GEOS/0.2.0/install-GPU-Hypre-GCC-CUDA_11.8-ompi_hpcx-OMP-relwithdebinfo/bin/geosx() [0x40e25e]

  what():  ! Umpire runtime_error [/dev/shm/mtml/src/GEOS/thirdPartyLibs/build-GPU-Hypre-GCC-CUDA_11.8-ompi_hpcx-OMP-relwithdebinfo/chai/src/chai/src/tpl/umpire/src/umpire/alloc/CudaPinnedAllocator.hpp:43]: cudaFreeHost( ptr = 0x2b5534000000 ) failed with error: an illegal memory access was encountered
    Backtrace: 13 frames
    0 0x2b53d35c8f55 No dladdr: /data/saet/mtml/software/x86_64/RHEL7/GEOS/0.2.0/install-GPU-Hypre-GCC-CUDA_11.8-ompi_hpcx-OMP-relwithdebinfo/lib/libgeosx_core.so(_ZN6umpire4util49_GLOBAL__N__25f8fd63_16_ArrayManager_cpp_ab41d17d15build_backtraceEv+0x35) [0x2b53d35c8f55]
    1 0x2b53d35cb514 No dladdr: /data/saet/mtml/software/x86_64/RHEL7/GEOS/0.2.0/install-GPU-Hypre-GCC-CUDA_11.8-ompi_hpcx-OMP-relwithdebinfo/lib/libgeosx_core.so(_ZNK6umpire13runtime_error7messageB5cxx11Ev+0x44) [0x2b53d35cb514]
    2 0x2b53d35eb35f No dladdr: /data/saet/mtml/software/x86_64/RHEL7/GEOS/0.2.0/install-GPU-Hypre-GCC-CUDA_11.8-ompi_hpcx-OMP-relwithdebinfo/lib/libgeosx_core.so(_ZN6umpire13runtime_errorC2ERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEES8_i+0x8f) [0x2b53d35eb35f]
    3 0x2b53d35fb8e1 No dladdr: /data/saet/mtml/software/x86_64/RHEL7/GEOS/0.2.0/install-GPU-Hypre-GCC-CUDA_11.8-ompi_hpcx-OMP-relwithdebinfo/lib/libgeosx_core.so(_ZN6umpire5alloc19CudaPinnedAllocator10deallocateEPv+0x351) [0x2b53d35fb8e1]
    4 0x2b53ce22af86 No dladdr: /data/saet/mtml/software/x86_64/RHEL7/GEOS/0.2.0/install-GPU-Hypre-GCC-CUDA_11.8-ompi_hpcx-OMP-relwithdebinfo/lib/libgeosx_core.so(_ZN6umpire9Allocator10deallocateEPv+0x226) [0x2b53ce22af86]
    5 0x2b53ce220c21 No dladdr: /data/saet/mtml/software/x86_64/RHEL7/GEOS/0.2.0/install-GPU-Hypre-GCC-CUDA_11.8-ompi_hpcx-OMP-relwithdebinfo/lib/libgeosx_core.so(_ZN4geos15DomainPartitionD2Ev+0x61) [0x2b53ce220c21]
    6 0x2b53ce220dc9 No dladdr: /data/saet/mtml/software/x86_64/RHEL7/GEOS/0.2.0/install-GPU-Hypre-GCC-CUDA_11.8-ompi_hpcx-OMP-relwithdebinfo/lib/libgeosx_core.so(_ZN4geos15DomainPartitionD0Ev+0x9) [0x2b53ce220dc9]
    7 0x2b53cd8e46a2 No dladdr: /data/saet/mtml/software/x86_64/RHEL7/GEOS/0.2.0/install-GPU-Hypre-GCC-CUDA_11.8-ompi_hpcx-OMP-relwithdebinfo/lib/libgeosx_core.so(_ZN4geos14dataRepository5GroupD1Ev+0x8f2) [0x2b53cd8e46a2]
    8 0x2b53d30cebb9 No dladdr: /data/saet/mtml/software/x86_64/RHEL7/GEOS/0.2.0/install-GPU-Hypre-GCC-CUDA_11.8-ompi_hpcx-OMP-relwithdebinfo/lib/libgeosx_core.so(_ZN4geos14ProblemManagerD0Ev+0x9) [0x2b53d30cebb9]
    9 0x2b53d30cb056 No dladdr: /data/saet/mtml/software/x86_64/RHEL7/GEOS/0.2.0/install-GPU-Hypre-GCC-CUDA_11.8-ompi_hpcx-OMP-relwithdebinfo/lib/libgeosx_core.so(_ZN4geos10GeosxStateD2Ev+0x3f6) [0x2b53d30cb056]
    10 0x40d2dc No dladdr: /data/saet/mtml/software/x86_64/RHEL7/GEOS/0.2.0/install-GPU-Hypre-GCC-CUDA_11.8-ompi_hpcx-OMP-relwithdebinfo/bin/geosx() [0x40d2dc]
    11 0x2b543af8b555 No dladdr: /lib64/libc.so.6(__libc_start_main+0xf5) [0x2b543af8b555]
    12 0x40e25e No dladdr: /data/saet/mtml/software/x86_64/RHEL7/GEOS/0.2.0/install-GPU-Hypre-GCC-CUDA_11.8-ompi_hpcx-OMP-relwithdebinfo/bin/geosx() [0x40e25e]

terminate called after throwing an instance of 'umpire::runtime_error'
terminate called after throwing an instance of 'umpire::runtime_error'

terminate called after throwing an instance of '  what():  ! Umpire runtime_error [/dev/shm/mtml/src/GEOS/thirdPartyLibs/build-GPU-Hypre-GCC-CUDA_11.8-ompi_hpcx-OMP-relwithdebinfo/chai/src/chai/src/tpl/umpire/src/umpire/alloc/CudaPinnedAllocator.hpp:43]: cudaFreeHost( ptr = 0x2b57b0000000 ) failed with error: an illegal memory access was encountered
    Backtrace: 13 frames
    0 0x2b564b056f55 No dladdr: /data/saet/mtml/software/x86_64/RHEL7/GEOS/0.2.0/install-GPU-Hypre-GCC-CUDA_11.8-ompi_hpcx-OMP-relwithdebinfo/lib/libgeosx_core.so(_ZN6umpire4util49_GLOBAL__N__25f8fd63_16_ArrayManager_cpp_ab41d17d15build_backtraceEv+0x35) [0x2b564b056f55]
    1 0x2b564b059514 No dladdr: /data/saet/mtml/software/x86_64/RHEL7/GEOS/0.2.0/install-GPU-Hypre-GCC-CUDA_11.8-ompi_hpcx-OMP-relwithdebinfo/lib/libgeosx_core.so(_ZNK6umpire13runtime_error7messageB5cxx11Ev+0x44) [0x2b564b059514]
    2 0x2b564b07935f No dladdr: /data/saet/mtml/software/x86_64/RHEL7/GEOS/0.2.0/install-GPU-Hypre-GCC-CUDA_11.8-ompi_hpcx-OMP-relwithdebinfo/lib/libgeosx_core.so(_ZN6umpire13runtime_errorC2ERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEES8_i+0x8f) [0x2b564b07935f]
    3 0x2b564b0898e1 No dladdr: /data/saet/mtml/software/x86_64/RHEL7/GEOS/0.2.0/install-GPU-Hypre-GCC-CUDA_11.8-ompi_hpcx-OMP-relwithdebinfo/lib/libgeosx_core.so(_ZN6umpire5alloc19CudaPinnedAllocator10deallocateEPv+0x351) [0x2b564b0898e1]
    4 0x2b5645cb8f86 No dladdr: /data/saet/mtml/software/x86_64/RHEL7/GEOS/0.2.0/install-GPU-Hypre-GCC-CUDA_11.8-ompi_hpcx-OMP-relwithdebinfo/lib/libgeosx_core.so(_ZN6umpire9Allocator10deallocateEPv+0x226) [0x2b5645cb8f86]
    5 0x2b5645caec21 No dladdr: /data/saet/mtml/software/x86_64/RHEL7/GEOS/0.2.0/install-GPU-Hypre-GCC-CUDA_11.8-ompi_hpcx-OMP-relwithdebinfo/lib/libgeosx_core.so(_ZN4geos15DomainPartitionD2Ev+0x61) [0x2b5645caec21]
    6 0x2b5645caedc9 No dladdr: /data/saet/mtml/software/x86_64/RHEL7/GEOS/0.2.0/install-GPU-Hypre-GCC-CUDA_11.8-ompi_hpcx-OMP-relwithdebinfo/lib/libgeosx_core.so(_ZN4geos15DomainPartitionD0Ev+0x9) [0x2b5645caedc9]
    7 0x2b56453726a2 No dladdr: /data/saet/mtml/software/x86_64/RHEL7/GEOS/0.2.0/install-GPU-Hypre-GCC-CUDA_11.8-ompi_hpcx-OMP-relwithdebinfo/lib/libgeosx_core.so(_ZN4geos14dataRepository5GroupD1Ev+0x8f2) [0x2b56453726a2]
    8 0x2b564ab5cbb9 No dladdr: /data/saet/mtml/software/x86_64/RHEL7/GEOS/0.2.0/install-GPU-Hypre-GCC-CUDA_11.8-ompi_hpcx-OMP-relwithdebinfo/lib/libgeosx_core.so(_ZN4geos14ProblemManagerD0Ev+0x9) [0x2b564ab5cbb9]
    9 0x2b564ab59056 No dladdr: /data/saet/mtml/software/x86_64/RHEL7/GEOS/0.2.0/install-GPU-Hypre-GCC-CUDA_11.8-ompi_hpcx-OMP-relwithdebinfo/lib/libgeosx_core.so(_ZN4geos10GeosxStateD2Ev+0x3f6) [0x2b564ab59056]
    10 0x40d2dc No dladdr: /data/saet/mtml/software/x86_64/RHEL7/GEOS/0.2.0/install-GPU-Hypre-GCC-CUDA_11.8-ompi_hpcx-OMP-relwithdebinfo/bin/geosx() [0x40d2dc]
    11 0x2b56b2a19555 No dladdr: /lib64/libc.so.6(__libc_start_main+0xf5) [0x2b56b2a19555]
    12 0x40e25e No dladdr: /data/saet/mtml/software/x86_64/RHEL7/GEOS/0.2.0/install-GPU-Hypre-GCC-CUDA_11.8-ompi_hpcx-OMP-relwithdebinfo/bin/geosx() [0x40e25e]

  what():  ! Umpire runtime_error [/dev/shm/mtml/src/GEOS/thirdPartyLibs/build-GPU-Hypre-GCC-CUDA_11.8-ompi_hpcx-OMP-relwithdebinfo/chai/src/chai/src/tpl/umpire/src/umpire/alloc/CudaPinnedAllocator.hpp:43]: cudaFreeHost( ptr = 0x2aaddc000000 ) failed with error: an illegal memory access was encountered
    Backtrace: 13 frames
    0 0x2aac799a3f55 No dladdr: /data/saet/mtml/software/x86_64/RHEL7/GEOS/0.2.0/install-GPU-Hypre-GCC-CUDA_11.8-ompi_hpcx-OMP-relwithdebinfo/lib/libgeosx_core.so(_ZN6umpire4util49_GLOBAL__N__25f8fd63_16_ArrayManager_cpp_ab41d17d15build_backtraceEv+0x35) [0x2aac799a3f55]
    1 0x2aac799a6514 No dladdr: /data/saet/mtml/software/x86_64/RHEL7/GEOS/0.2.0/install-GPU-Hypre-GCC-CUDA_11.8-ompi_hpcx-OMP-relwithdebinfo/lib/libgeosx_core.so(_ZNK6umpire13runtime_error7messageB5cxx11Ev+0x44) [0x2aac799a6514]
    2 0x2aac799c635f No dladdr: /data/saet/mtml/software/x86_64/RHEL7/GEOS/0.2.0/install-GPU-Hypre-GCC-CUDA_11.8-ompi_hpcx-OMP-relwithdebinfo/lib/libgeosx_core.so(_ZN6umpire13runtime_errorC2ERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEES8_i+0x8f) [0x2aac799c635f]
    3 0x2aac799d68e1 No dladdr: /data/saet/mtml/software/x86_64/RHEL7/GEOS/0.2.0/install-GPU-Hypre-GCC-CUDA_11.8-ompi_hpcx-OMP-relwithdebinfo/lib/libgeosx_core.so(_ZN6umpire5alloc19CudaPinnedAllocator10deallocateEPv+0x351) [0x2aac799d68e1]
    4 0x2aac74605f86 No dladdr: /data/saet/mtml/software/x86_64/RHEL7/GEOS/0.2.0/install-GPU-Hypre-GCC-CUDA_11.8-ompi_hpcx-OMP-relwithdebinfo/lib/libgeosx_core.so(_ZN6umpire9Allocator10deallocateEPv+0x226) [0x2aac74605f86]
    5 0x2aac745fbc21 No dladdr: /data/saet/mtml/software/x86_64/RHEL7/GEOS/0.2.0/install-GPU-Hypre-GCC-CUDA_11.8-ompi_hpcx-OMP-relwithdebinfo/lib/libgeosx_core.so(_ZN4geos15DomainPartitionD2Ev+0x61) [0x2aac745fbc21]
    6 0x2aac745fbdc9 No dladdr: /data/saet/mtml/software/x86_64/RHEL7/GEOS/0.2.0/install-GPU-Hypre-GCC-CUDA_11.8-ompi_hpcx-OMP-relwithdebinfo/lib/libgeosx_core.so(_ZN4geos15DomainPartitionD0Ev+0x9) [0x2aac745fbdc9]
    7 0x2aac73cbf6a2 No dladdr: /data/saet/mtml/software/x86_64/RHEL7/GEOS/0.2.0/install-GPU-Hypre-GCC-CUDA_11.8-ompi_hpcx-OMP-relwithdebinfo/lib/libgeosx_core.so(_ZN4geos14dataRepository5GroupD1Ev+0x8f2) [0x2aac73cbf6a2]
    8 0x2aac794a9bb9 No dladdr: /data/saet/mtml/software/x86_64/RHEL7/GEOS/0.2.0/install-GPU-Hypre-GCC-CUDA_11.8-ompi_hpcx-OMP-relwithdebinfo/lib/libgeosx_core.so(_ZN4geos14ProblemManagerD0Ev+0x9) [0x2aac794a9bb9]
    9 0x2aac794a6056 No dladdr: /data/saet/mtml/software/x86_64/RHEL7/GEOS/0.2.0/install-GPU-Hypre-GCC-CUDA_11.8-ompi_hpcx-OMP-relwithdebinfo/lib/libgeosx_core.so(_ZN4geos10GeosxStateD2Ev+0x3f6) [0x2aac794a6056]
    10 0x40d2dc No dladdr: /data/saet/mtml/software/x86_64/RHEL7/GEOS/0.2.0/install-GPU-Hypre-GCC-CUDA_11.8-ompi_hpcx-OMP-relwithdebinfo/bin/geosx() [0x40d2dc]
    11 0x2aace1366555 No dladdr: /lib64/libc.so.6(__libc_start_main+0xf5) [0x2aace1366555]
    12 0x40e25e No dladdr: /data/saet/mtml/software/x86_64/RHEL7/GEOS/0.2.0/install-GPU-Hypre-GCC-CUDA_11.8-ompi_hpcx-OMP-relwithdebinfo/bin/geosx() [0x40e25e]

umpire::runtime_error'
jhuang2601 commented 9 months ago

I can reproduce the same issue by running SEAM CO2 case with GPU compilation on Cypress.

terminate called after throwing an instance of 'umpire::runtime_error'
  what():  ! Umpire runtime_error [/shared/data1/Users/j0551570/Compilation/Build_120523/thirdPartyLibs/build-cypress-GPU-gcc-std17-release/chai/src/chai/src/tpl/umpire/src/umpire/alloc/CudaPinnedAllocator.hpp:44]: cudaFreeHost( ptr = 0x7fef4b600000 ) failed with error: an illegal memory access was encountered
    Backtrace: 12 frames
    0 0x7feff05445c3 No dladdr: /shared/data1/Users/j0551570/Compilation/Build_120523/GEOS/build-cypress-GPU-gcc-std17-release/lib/libgeosx_core.so(_ZNK6umpire13runtime_error7messageB5cxx11Ev+0x43) [0x7feff05445c3]
    1 0x7feff054503e No dladdr: /shared/data1/Users/j0551570/Compilation/Build_120523/GEOS/build-cypress-GPU-gcc-std17-release/lib/libgeosx_core.so(_ZN6umpire13runtime_errorC2ERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEES8_i+0x8e) [0x7feff054503e]
    2 0x7feff05554a5 No dladdr: /shared/data1/Users/j0551570/Compilation/Build_120523/GEOS/build-cypress-GPU-gcc-std17-release/lib/libgeosx_core.so(_ZN6umpire8resource21DefaultMemoryResourceINS_5alloc19CudaPinnedAllocatorEE10deallocateEPvm+0x705) [0x7feff05554a5]
    3 0x7fefec2f82a3 No dladdr: /shared/data1/Users/j0551570/Compilation/Build_120523/GEOS/build-cypress-GPU-gcc-std17-release/lib/libgeosx_core.so(_ZN6umpire9Allocator10deallocateEPv+0x193) [0x7fefec2f82a3]
    4 0x7fefec2eee62 No dladdr: /shared/data1/Users/j0551570/Compilation/Build_120523/GEOS/build-cypress-GPU-gcc-std17-release/lib/libgeosx_core.so(_ZN4geos15DomainPartitionD1Ev+0x62) [0x7fefec2eee62]
    5 0x7fefec2eefc9 No dladdr: /shared/data1/Users/j0551570/Compilation/Build_120523/GEOS/build-cypress-GPU-gcc-std17-release/lib/libgeosx_core.so(_ZN4geos15DomainPartitionD0Ev+0x9) [0x7fefec2eefc9]
    6 0x7fefebba71e2 No dladdr: /shared/data1/Users/j0551570/Compilation/Build_120523/GEOS/build-cypress-GPU-gcc-std17-release/lib/libgeosx_core.so(_ZN4geos14dataRepository5GroupD1Ev+0x822) [0x7fefebba71e2]
    7 0x7feff01473f9 No dladdr: /shared/data1/Users/j0551570/Compilation/Build_120523/GEOS/build-cypress-GPU-gcc-std17-release/lib/libgeosx_core.so(_ZN4geos14ProblemManagerD0Ev+0x9) [0x7feff01473f9]
    8 0x7feff0145156 No dladdr: /shared/data1/Users/j0551570/Compilation/Build_120523/GEOS/build-cypress-GPU-gcc-std17-release/lib/libgeosx_core.so(_ZN4geos10GeosxStateD1Ev+0x1f6) [0x7feff0145156]
    9 0x40d003 No dladdr: /shared/data1/Users/j0551570/Compilation/Build_120523/GEOS/build-cypress-GPU-gcc-std17-release/bin/geosx() [0x40d003]
    10 0x7fefe1b467b3 No dladdr: /lib64/libc.so.6(__libc_start_main+0xf3) [0x7fefe1b467b3]
    11 0x40e1ae No dladdr: /shared/data1/Users/j0551570/Compilation/Build_120523/GEOS/build-cypress-GPU-gcc-std17-release/bin/geosx(_start+0x2e) [0x40e1ae]

terminate called after throwing an instance of 'umpire::runtime_error'
  what():  ! Umpire runtime_error [/shared/data1/Users/j0551570/Compilation/Build_120523/thirdPartyLibs/build-cypress-GPU-gcc-std17-release/chai/src/chai/src/tpl/umpire/src/umpire/alloc/CudaPinnedAllocator.hpp:44]: cudaFreeHost( ptr = 0x7f4e5ad03000 ) failed with error: an illegal memory access was encountered
    Backtrace: 12 frames
    0 0x7f4effc6c5c3 No dladdr: /shared/data1/Users/j0551570/Compilation/Build_120523/GEOS/build-cypress-GPU-gcc-std17-release/lib/libgeosx_core.so(_ZNK6umpire13runtime_error7messageB5cxx11Ev+0x43) [0x7f4effc6c5c3]
    1 0x7f4effc6d03e No dladdr: /shared/data1/Users/j0551570/Compilation/Build_120523/GEOS/build-cypress-GPU-gcc-std17-release/lib/libgeosx_core.so(_ZN6umpire13runtime_errorC2ERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEES8_i+0x8e) [0x7f4effc6d03e]
    2 0x7f4effc7d4a5 No dladdr: /shared/data1/Users/j0551570/Compilation/Build_120523/GEOS/build-cypress-GPU-gcc-std17-release/lib/libgeosx_core.so(_ZN6umpire8resource21DefaultMemoryResourceINS_5alloc19CudaPinnedAllocatorEE10deallocateEPvm+0x705) [0x7f4effc7d4a5]
    3 0x7f4efba202a3 No dladdr: /shared/data1/Users/j0551570/Compilation/Build_120523/GEOS/build-cypress-GPU-gcc-std17-release/lib/libgeosx_core.so(_ZN6umpire9Allocator10deallocateEPv+0x193) [0x7f4efba202a3]
    4 0x7f4efba16e62 No dladdr: /shared/data1/Users/j0551570/Compilation/Build_120523/GEOS/build-cypress-GPU-gcc-std17-release/lib/libgeosx_core.so(_ZN4geos15DomainPartitionD1Ev+0x62) [0x7f4efba16e62]
    5 0x7f4efba16fc9 No dladdr: /shared/data1/Users/j0551570/Compilation/Build_120523/GEOS/build-cypress-GPU-gcc-std17-release/lib/libgeosx_core.so(_ZN4geos15DomainPartitionD0Ev+0x9) [0x7f4efba16fc9]
    6 0x7f4efb2cf1e2 No dladdr: /shared/data1/Users/j0551570/Compilation/Build_120523/GEOS/build-cypress-GPU-gcc-std17-release/lib/libgeosx_core.so(_ZN4geos14dataRepository5GroupD1Ev+0x822) [0x7f4efb2cf1e2]
    7 0x7f4eff86f3f9 No dladdr: /shared/data1/Users/j0551570/Compilation/Build_120523/GEOS/build-cypress-GPU-gcc-std17-release/lib/libgeosx_core.so(_ZN4geos14ProblemManagerD0Ev+0x9) [0x7f4eff86f3f9]
    8 0x7f4eff86d156 No dladdr: /shared/data1/Users/j0551570/Compilation/Build_120523/GEOS/build-cypress-GPU-gcc-std17-release/lib/libgeosx_core.so(_ZN4geos10GeosxStateD1Ev+0x1f6) [0x7f4eff86d156]
    9 0x40d003 No dladdr: /shared/data1/Users/j0551570/Compilation/Build_120523/GEOS/build-cypress-GPU-gcc-std17-release/bin/geosx() [0x40d003]
    10 0x7f4ef126e7b3 No dladdr: /lib64/libc.so.6(__libc_start_main+0xf3) [0x7f4ef126e7b3]
    11 0x40e1ae No dladdr: /shared/data1/Users/j0551570/Compilation/Build_120523/GEOS/build-cypress-GPU-gcc-std17-release/bin/geosx(_start+0x2e) [0x40e1ae]