search

TEAR-ERC / tandem

A HPC DG method for 2D and 3D SEAS problems
BSD 3-Clause "New" or "Revised" License
16 stars 9 forks source link

tandem on GPUs #50

Closed Thomas-Ulrich closed 1 year ago

Thomas-Ulrich commented 1 year ago

I've installed a cuda-aware petsc and associated tandem with: spack install -j 50 tandem@main polynomial_degree=2 domain_dimension=3 ^petsc +cuda ^cuda@11 ^openmpi+cuda

This yields the following dependencies:

(base) ulrich@heisenbug:/import/exception-dump/ulrich/spack/var/spack/repos/builtin/packages/tandem$ spack spec tandem@main polynomial_degree=2 domain_dimension=3 ^petsc +cuda ^cuda@11 ^openmpi+cuda
Input spec
--------------------------------
 -   tandem@main domain_dimension=3 polynomial_degree=2
 -       ^cuda@11
 -       ^openmpi+cuda
 -       ^petsc+cuda

Concretized
--------------------------------
[+]  tandem@main%gcc@11.2.0~ipo~libxsmm build_system=cmake build_type=Release domain_dimension=3 generator=make min_quadrature_order=0 polynomial_degree=2 arch=linux-debian11-zen2
[+]      ^cmake@3.26.3%gcc@11.2.0~doc+ncurses+ownlibs~qt build_system=generic build_type=Release arch=linux-debian11-zen2
[+]          ^curl@8.1.2%gcc@11.2.0~gssapi~ldap~libidn2~librtmp~libssh~libssh2+nghttp2 build_system=autotools libs=shared,static tls=openssl arch=linux-debian11-zen2
[+]              ^nghttp2@1.52.0%gcc@11.2.0 build_system=autotools arch=linux-debian11-zen2
[+]              ^openssl@3.1.1%gcc@11.2.0~docs~shared build_system=generic certs=mozilla arch=linux-debian11-zen2
[+]                  ^ca-certificates-mozilla@2023-05-30%gcc@11.2.0 build_system=generic arch=linux-debian11-zen2
[+]          ^ncurses@6.4%gcc@11.2.0~symlinks+termlib abi=none build_system=autotools arch=linux-debian11-zen2
[+]      ^eigen@3.4.0%gcc@11.2.0~ipo build_system=cmake build_type=RelWithDebInfo generator=make arch=linux-debian11-zen2
[+]      ^gmake@4.4.1%gcc@11.2.0~guile build_system=autotools arch=linux-debian11-zen2
[+]      ^lua@5.4.4%gcc@11.2.0~pcfile+shared build_system=makefile fetcher=curl arch=linux-debian11-zen2
[+]          ^readline@8.2%gcc@11.2.0 build_system=autotools patches=bbf97f1 arch=linux-debian11-zen2
[+]          ^unzip@6.0%gcc@11.2.0 build_system=makefile arch=linux-debian11-zen2
[+]      ^metis@5.1.0%gcc@11.2.0~gdb+int64~ipo~real64+shared build_system=cmake build_type=Release generator=make patches=4991da9,93a7903,b1225da arch=linux-debian11-zen2
[+]      ^openmpi@4.1.5%gcc@11.2.0~atomics+cuda~cxx~cxx_exceptions~gpfs~internal-hwloc~internal-pmix~java~legacylaunchers~lustre~memchecker~openshmem~orterunprefix+romio+rsh~singularity+static+vt+wrapper-rpath build_system=autotools cuda_arch=none fabrics=none schedulers=none arch=linux-debian11-zen2
[+]          ^cuda@11.8.0%gcc@11.2.0~allow-unsupported-compilers~dev build_system=generic arch=linux-debian11-zen2
[+]              ^libxml2@2.10.3%gcc@11.2.0~python build_system=autotools arch=linux-debian11-zen2
[+]                  ^xz@5.4.1%gcc@11.2.0~pic build_system=autotools libs=shared,static arch=linux-debian11-zen2
[+]          ^hwloc@2.9.1%gcc@11.2.0~cairo+cuda~gl~libudev+libxml2~netloc~nvml~oneapi-level-zero~opencl+pci~rocm build_system=autotools cuda_arch=none libs=shared,static arch=linux-debian11-zen2
[+]              ^libpciaccess@0.17%gcc@11.2.0 build_system=autotools arch=linux-debian11-zen2
[+]                  ^util-macros@1.19.3%gcc@11.2.0 build_system=autotools arch=linux-debian11-zen2
[+]          ^numactl@2.0.14%gcc@11.2.0 build_system=autotools patches=4e1d78c,62fc8a8,ff37630 arch=linux-debian11-zen2
[+]              ^autoconf@2.69%gcc@11.2.0 build_system=autotools patches=35c4492,7793209,a49dd5b arch=linux-debian11-zen2
[+]              ^automake@1.16.5%gcc@11.2.0 build_system=autotools arch=linux-debian11-zen2
[+]              ^libtool@2.4.7%gcc@11.2.0 build_system=autotools arch=linux-debian11-zen2
[+]              ^m4@1.4.19%gcc@11.2.0+sigsegv build_system=autotools patches=9dc5fbd,bfdffa7 arch=linux-debian11-zen2
[+]                  ^libsigsegv@2.14%gcc@11.2.0 build_system=autotools arch=linux-debian11-zen2
[+]          ^openssh@9.3p1%gcc@11.2.0+gssapi build_system=autotools arch=linux-debian11-zen2
[+]              ^krb5@1.20.1%gcc@11.2.0+shared build_system=autotools arch=linux-debian11-zen2
[+]                  ^bison@3.8.2%gcc@11.2.0 build_system=autotools arch=linux-debian11-zen2
[+]                  ^findutils@4.9.0%gcc@11.2.0 build_system=autotools patches=440b954 arch=linux-debian11-zen2
[+]                  ^gettext@0.21.1%gcc@11.2.0+bzip2+curses+git~libunistring+libxml2+tar+xz build_system=autotools arch=linux-debian11-zen2
[+]                      ^tar@1.34%gcc@11.2.0 build_system=autotools zip=pigz arch=linux-debian11-zen2
[+]                          ^pigz@2.7%gcc@11.2.0 build_system=makefile arch=linux-debian11-zen2
[+]                          ^zstd@1.5.5%gcc@11.2.0+programs build_system=makefile compression=none libs=shared,static arch=linux-debian11-zen2
[+]              ^libedit@3.1-20210216%gcc@11.2.0 build_system=autotools arch=linux-debian11-zen2
[+]              ^libxcrypt@4.4.35%gcc@11.2.0~obsolete_api build_system=autotools patches=4885da3 arch=linux-debian11-zen2
[+]          ^perl@5.38.0%gcc@11.2.0+cpanm+opcode+open+shared+threads build_system=generic arch=linux-debian11-zen2
[+]              ^berkeley-db@18.1.40%gcc@11.2.0+cxx~docs+stl build_system=autotools patches=26090f4,b231fcc arch=linux-debian11-zen2
[+]              ^bzip2@1.0.8%gcc@11.2.0~debug~pic+shared build_system=generic arch=linux-debian11-zen2
[+]              ^gdbm@1.23%gcc@11.2.0 build_system=autotools arch=linux-debian11-zen2
[+]          ^pkgconf@1.9.5%gcc@11.2.0 build_system=autotools arch=linux-debian11-zen2
[+]          ^pmix@4.2.4%gcc@11.2.0~docs+pmi_backwards_compatibility~python~restful build_system=autotools arch=linux-debian11-zen2
[+]              ^libevent@2.1.12%gcc@11.2.0+openssl build_system=autotools arch=linux-debian11-zen2
[+]      ^parmetis@4.0.3%gcc@11.2.0~gdb+int64~ipo+shared build_system=cmake build_type=Release generator=make patches=4f89253,50ed208,704b84f arch=linux-debian11-zen2
[+]      ^petsc@3.18.5%gcc@11.2.0~X~batch~cgns~complex+cuda~debug+double~exodusii~fftw+fortran~giflib+hdf5~hpddm~hwloc+hypre+int64~jpeg~knl~kokkos~libpng~libyaml~memkind+metis~mkl-pardiso~mmg~moab~mpfr+mpi+mumps~openmp~p4est~parmmg~ptscotch~random123~rocm~saws+scalapack+shared~strumpack~suite-sparse+superlu-dist~tetgen~trilinos~valgrind build_system=generic clanguage=C cuda_arch=none memalign=32 arch=linux-debian11-zen2
[+]          ^diffutils@3.9%gcc@11.2.0 build_system=autotools arch=linux-debian11-zen2
[+]              ^libiconv@1.17%gcc@11.2.0 build_system=autotools libs=shared,static arch=linux-debian11-zen2
[+]          ^hdf5@1.14.2%gcc@11.2.0~cxx~fortran~hl~ipo~java~map+mpi+shared~szip~threadsafe+tools api=default build_system=cmake build_type=Release generator=make arch=linux-debian11-zen2
[+]          ^hypre@2.28.0%gcc@11.2.0~caliper~complex~cuda~debug+fortran~gptune+int64~internal-superlu~mixedint+mpi~openmp~rocm+shared~superlu-dist~sycl~umpire~unified-memory build_system=autotools arch=linux-debian11-zen2
[+]          ^mumps@5.5.1%gcc@11.2.0~blr_mt+complex+double+float~incfort~int64+metis+mpi~openmp+parmetis~ptscotch~scotch+shared build_system=generic patches=373d736 arch=linux-debian11-zen2
[+]          ^netlib-scalapack@2.2.0%gcc@11.2.0~ipo~pic+shared build_system=cmake build_type=Release generator=make patches=072b006,1c9ce5f,244a9aa arch=linux-debian11-zen2
[+]          ^openblas@0.3.23%gcc@11.2.0~bignuma~consistent_fpcsr+fortran~ilp64+locking+pic+shared build_system=makefile symbol_suffix=none threads=none arch=linux-debian11-zen2
[e]          ^python@3.9.2%gcc@11.2.0+bz2+crypt+ctypes+dbm~debug+libxml2+lzma+nis~optimizations+pic+pyexpat~pythoncmd+readline+shared+sqlite3+ssl+tix+tkinter+uuid+zlib build_system=generic patches=0d98e93,f2fd060 arch=linux-debian11-zen2
[+]          ^superlu-dist@8.1.2%gcc@11.2.0~cuda+int64~ipo~openmp~rocm+shared build_system=cmake build_type=Release generator=make arch=linux-debian11-zen2
[+]      ^zlib@1.2.13%gcc@11.2.0+optimize+pic+shared build_system=makefile arch=linux-debian11-zen2

I've tested it on the Ridgecrest example from the paper (note that I am able to run it full CPUs). One node (note the added Petsc arguments -mg_levels_mat_type aijcusparse -vec_type cuda -mat_type aijcusparse):

(base) ulrich@heisenbug:/export/dump/ulrich/section_8_2/scenario-rc$ static ridge.toml  --output ridgecrest  --mg_strategy twolevel --mg_coarse_level 1  --petsc -ksp_view -ksp_monitor -ksp_converged_reason -ksp_max_it 400 -pc_type mg -mg_levels_ksp_max_it 4 -mg_levels_ksp_type cg -mg_levels_pc_type bjacobi -options_left -ksp_rtol 1.0e-6 -mg_coarse_pc_type gamg -mg_coarse_ksp_type cg -mg_coarse_ksp_rtol 1.0e-1 -mg_coarse_ksp_converged_reason -ksp_type gcr -mg_levels_mat_type aijcusparse -vec_type cuda -mat_type aijcusparse

               ___          ___         _____         ___          ___
      ___     /  /\        /__/\       /  /::\       /  /\        /__/\
     /  /\   /  /::\       \  \:\     /  /:/\:\     /  /:/_      |  |::\
    /  /:/  /  /:/\:\       \  \:\   /  /:/  \:\   /  /:/ /\     |  |:|:\
   /  /:/  /  /:/~/::\  _____\__\:\ /__/:/ \__\:| /  /:/ /:/_  __|__|:|\:\
  /  /::\ /__/:/ /:/\:\/__/::::::::\\  \:\ /  /://__/:/ /:/ /\/__/::::| \:\
 /__/:/\:\\  \:\/:/__\/\  \:\~~\~~\/ \  \:\  /:/ \  \:\/:/ /:/\  \:\~~\__\/
 \__\/  \:\\  \::/      \  \:\  ~~~   \  \:\/:/   \  \::/ /:/  \  \:\
      \  \:\\  \:\       \  \:\        \  \::/     \  \:\/:/    \  \:\
       \__\/ \  \:\       \  \:\        \__\/       \  \::/      \  \:\
              \__\/        \__\/                     \__\/        \__\/

                          tandem version 9e964fe

                       stack size limit = unlimited

                              Worker affinity
    0123456789|0123456789|0123456789|0123456789|0123456789|0123456789|
    0123456789|0123456789|0123456789|0123456789|0123456789|0123456789|
    0123456789|0123456789|0123456789|0123456789|0123456789|0123456789|
    0123456789|0123456789|0123456789|0123456789|0123456789|0123456789|
    0123456789|012345

Ridge:create()
parsing fault_many_trace.csv
DOFs: 12634590
Mesh size: 36.7959
[0]PETSC ERROR: ------------------------------------------------------------------------
[0]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range
[0]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger
[0]PETSC ERROR: or see https://petsc.org/release/faq/#valgrind and https://petsc.org/release/faq/
[0]PETSC ERROR: or try https://docs.nvidia.com/cuda/cuda-memcheck/index.html on NVIDIA CUDA systems to find memory corruption errors
[0]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run 
[0]PETSC ERROR: to get more information on the crash.
[0]PETSC ERROR: Run with -malloc_debug to check if memory corruption is causing the crash.

Same error on 2 nodes:

(base) ulrich@heisenbug:/export/dump/ulrich/section_8_2/scenario-rc$ sh run2.sh 

               ___          ___         _____         ___          ___
      ___     /  /\        /__/\       /  /::\       /  /\        /__/\
     /  /\   /  /::\       \  \:\     /  /:/\:\     /  /:/_      |  |::\
    /  /:/  /  /:/\:\       \  \:\   /  /:/  \:\   /  /:/ /\     |  |:|:\
   /  /:/  /  /:/~/::\  _____\__\:\ /__/:/ \__\:| /  /:/ /:/_  __|__|:|\:\
  /  /::\ /__/:/ /:/\:\/__/::::::::\\  \:\ /  /://__/:/ /:/ /\/__/::::| \:\
 /__/:/\:\\  \:\/:/__\/\  \:\~~\~~\/ \  \:\  /:/ \  \:\/:/ /:/\  \:\~~\__\/
 \__\/  \:\\  \::/      \  \:\  ~~~   \  \:\/:/   \  \::/ /:/  \  \:\
      \  \:\\  \:\       \  \:\        \  \::/     \  \:\/:/    \  \:\
       \__\/ \  \:\       \  \:\        \__\/       \  \::/      \  \:\
              \__\/        \__\/                     \__\/        \__\/

                          tandem version 9e964fe

                       stack size limit = unlimited

                              Worker affinity
    0---------|----------|----------|----------|----------|----------|
    ----------|----------|----------|----------|----------|----------|
    --------8-|----------|----------|----------|----------|----------|
    ----------|----------|----------|----------|----------|----------|
    ----------|------

Ridge:create()
parsing fault_many_trace.csv
Ridge:create()
parsing fault_many_trace.csv
DOFs: 12634590
Mesh size: 36.7959
[0]PETSC ERROR: ------------------------------------------------------------------------
[0]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range
[0]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger
[0]PETSC ERROR: or see https://petsc.org/release/faq/#valgrind and https://petsc.org/release/faq/
[0]PETSC ERROR: or try https://docs.nvidia.com/cuda/cuda-memcheck/index.html on NVIDIA CUDA systems to find memory corruption errors
[0]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run 
[0]PETSC ERROR: to get more information on the crash.
[0]PETSC ERROR: Run with -malloc_debug to check if memory corruption is causing the crash.

launch script:

(base) ulrich@heisenbug:/export/dump/ulrich/section_8_2/scenario-rc$ cat run2.sh 
#!/bin/bash
mpiexec -n 2  static ridge.toml  --output ridgecrest  --mg_strategy twolevel --mg_coarse_level 1  --petsc -ksp_view -ksp_monitor -ksp_converged_reason -ksp_max_it 400 -pc_type mg -mg_levels_ksp_max_it 4 -mg_levels_ksp_type cg -mg_levels_pc_type bjacobi -options_left -ksp_rtol 1.0e-6 -mg_coarse_pc_type gamg -mg_coarse_ksp_type cg -mg_coarse_ksp_rtol 1.0e-1 -mg_coarse_ksp_converged_reason -ksp_type gcr -mg_levels_mat_type aijcusparse -vec_type cuda -mat_type aijcusparse
hpc4geo commented 1 year ago

Okay. Couple of things.

[1] Lets first establish that the PETSc + GPU build is working with a very simple example. Let's start by not using multigrid at all. Please try this and upload what is sent to stdout.

mpiexec -n 2 static ridge.toml --output ridgecrest --matrix_free false --petsc -ksp_max_it 400 -pc_type jacobi -ksp_rtol 1.0e-6 -ksp_type gcr -vec_type cuda -mat_type aijcusparse -ksp_view -ksp_monitor -ksp_converged_reason -options_left -log_view

[2] In general, debugging this will be impossible with an optimized build of PETSc. For testing you absolutely must use a debug build of PETSc. Does spack support this?

Thomas-Ulrich commented 1 year ago

I get the following error right at the start:

(base) ulrich@heisenbug:/export/dump/ulrich/section_8_2/scenario-rc$ mpiexec -n 2 static ridge.toml --output ridgecrest --matrix_free false --petsc -ksp_max_it 400 -pc_type jacobi -ksp_rtol 1.0e-6 -ksp_type gcr -vec_type cuda -mat_type aijcusparse -ksp_view -ksp_monitor -ksp_converged_reason -options_left -log_view
[1]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
[1]PETSC ERROR: Petsc has generated inconsistent data
[1]PETSC ERROR: Invalid stack size 0, pop register_PCs /tmp/ulrich/spack-stage/spack-stage-tandem-main-wnfinsgrh3swjkyofitwrffzldwourlj/spack-src/app/pc/register.cpp:18.

[1]PETSC ERROR: WARNING! There are option(s) set that were not used! Could be the program crashed before they were used or a spelling mistake, etc!
[1]PETSC ERROR: Option left: name:-ksp_converged_reason (no value)
[1]PETSC ERROR: Option left: name:-ksp_max_it value: 400
[1]PETSC ERROR: Option left: name:-ksp_monitor (no value)
[1]PETSC ERROR: Option left: name:-ksp_rtol value: 1.0e-6
[1]PETSC ERROR: Option left: name:-ksp_type value: gcr
[1]PETSC ERROR: Option left: name:-ksp_view (no value)
[1]PETSC ERROR: Option left: name:-mat_type value: aijcusparse
[1]PETSC ERROR: Option left: name:-options_left (no value)
[1]PETSC ERROR: Option left: name:-pc_type value: jacobi
[1]PETSC ERROR: Option left: name:-vec_type value: cuda
[1]PETSC ERROR: See https://petsc.org/release/faq/ for trouble shooting.
[1]PETSC ERROR: Petsc Release Version 3.18.5, Feb 27, 2023 
[1]PETSC ERROR: --petsc on a  named heisenbug by ulrich Thu Aug 31 18:31:30 2023
[1]PETSC ERROR: Configure options --prefix=/import/exception-dump/ulrich/myLibs/spack-packages/linux-debian11-zen2/gcc-11.2.0/petsc-3.18.5-pppgzrc7ykm26yk4dmy7nzuf2f6jopah --with-ssl=0 --download-c2html=0 --download-sowing=0 --download-hwloc=0 --with-make-exec=make --with-cc=/import/exception-dump/ulrich/myLibs/spack-packages/linux-debian11-zen2/gcc-11.2.0/openmpi-4.1.5-sf7iuo3braeberbmnnyt4emapkxz4jtw/bin/mpicc --with-cxx=/import/exception-dump/ulrich/myLibs/spack-packages/linux-debian11-zen2/gcc-11.2.0/openmpi-4.1.5-sf7iuo3braeberbmnnyt4emapkxz4jtw/bin/mpic++ --with-fc=/import/exception-dump/ulrich/myLibs/spack-packages/linux-debian11-zen2/gcc-11.2.0/openmpi-4.1.5-sf7iuo3braeberbmnnyt4emapkxz4jtw/bin/mpif90 --with-precision=double --with-scalar-type=real --with-shared-libraries=1 --with-debugging=1 --with-openmp=0 --with-64-bit-indices=1 --with-blaslapack-lib=/import/exception-dump/ulrich/myLibs/spack-packages/linux-debian11-zen2/gcc-11.2.0/openblas-0.3.23-kxy7bxom4lam3wx44st22oytplyuisby/lib/libopenblas.so --with-memalign=32 --with-x=0 --with-clanguage=C --with-cuda=1 --with-cuda-dir=/import/exception-dump/ulrich/myLibs/spack-packages/linux-debian11-zen2/gcc-11.2.0/cuda-11.8.0-uivzrqah34rk27dutzqehubyc57zi46u --with-hip=0 --with-metis=1 --with-metis-include=/import/exception-dump/ulrich/myLibs/spack-packages/linux-debian11-zen2/gcc-11.2.0/metis-5.1.0-6zrgzhlze5ujsajtktgwf73dlpdjtdxk/include --with-metis-lib=/import/exception-dump/ulrich/myLibs/spack-packages/linux-debian11-zen2/gcc-11.2.0/metis-5.1.0-6zrgzhlze5ujsajtktgwf73dlpdjtdxk/lib/libmetis.so --with-hypre=1 --with-hypre-include=/import/exception-dump/ulrich/myLibs/spack-packages/linux-debian11-zen2/gcc-11.2.0/hypre-2.28.0-73iiib7speczh6bpszpkvbixgpkwyt7z/include --with-hypre-lib=/import/exception-dump/ulrich/myLibs/spack-packages/linux-debian11-zen2/gcc-11.2.0/hypre-2.28.0-73iiib7speczh6bpszpkvbixgpkwyt7z/lib/libHYPRE.so --with-parmetis=1 --with-parmetis-include=/import/exception-dump/ulrich/myLibs/spack-packages/linux-debian11-zen2/gcc-11.2.0/parmetis-4.0.3-gy6mgt3a4qt2ppnsorma4tabjw3f55fw/include --with-parmetis-lib=/import/exception-dump/ulrich/myLibs/spack-packages/linux-debian11-zen2/gcc-11.2.0/parmetis-4.0.3-gy6mgt3a4qt2ppnsorma4tabjw3f55fw/lib/libparmetis.so --with-kokkos=0 --with-kokkos-kernels=0 --with-superlu_dist=1 --with-superlu_dist-include=/import/exception-dump/ulrich/myLibs/spack-packages/linux-debian11-zen2/gcc-11.2.0/superlu-dist-8.1.2-7mrjdpdpu5l2g7x546r4kcssv2minjsa/include --with-superlu_dist-lib=/import/exception-dump/ulrich/myLibs/spack-packages/linux-debian11-zen2/gcc-11.2.0/superlu-dist-8.1.2-7mrjdpdpu5l2g7x546r4kcssv2minjsa/lib/libsuperlu_dist.so --with-ptscotch=0 --with-suitesparse=0 --with-hdf5=1 --with-hdf5-include=/import/exception-dump/ulrich/myLibs/spack-packages/linux-debian11-zen2/gcc-11.2.0/hdf5-1.14.2-dstkldnhuuoqsxniwh7piimjbyva4b2c/include --with-hdf5-lib=/import/exception-dump/ulrich/myLibs/spack-packages/linux-debian11-zen2/gcc-11.2.0/hdf5-1.14.2-dstkldnhuuoqsxniwh7piimjbyva4b2c/lib/libhdf5.so --with-zlib=1 --with-zlib-include=/import/exception-dump/ulrich/myLibs/spack-packages/linux-debian11-zen2/gcc-11.2.0/zlib-1.2.13-ufda23qzqsm5z2eqwsne76yoazck6fsz/include --with-zlib-lib=/import/exception-dump/ulrich/myLibs/spack-packages/linux-debian11-zen2/gcc-11.2.0/zlib-1.2.13-ufda23qzqsm5z2eqwsne76yoazck6fsz/lib/libz.so --with-mumps=1 --with-mumps-include=/import/exception-dump/ulrich/myLibs/spack-packages/linux-debian11-zen2/gcc-11.2.0/mumps-5.5.1-umzads25ggryhklmvvto4a5i6ue74fbk/include --with-mumps-lib="/import/exception-dump/ulrich/myLibs/spack-packages/linux-debian11-zen2/gcc-11.2.0/mumps-5.5.1-umzads25ggryhklmvvto4a5i6ue74fbk/lib/libsmumps.so /import/exception-dump/ulrich/myLibs/spack-packages/linux-debian11-zen2/gcc-11.2.0/mumps-5.5.1-umzads25ggryhklmvvto4a5i6ue74fbk/lib/libzmumps.so /import/exception-dump/ulrich/myLibs/spack-packages/linux-debian11-zen2/gcc-11.2.0/mumps-5.5.1-umzads25ggryhklmvvto4a5i6ue74fbk/lib/libcmumps.so /import/exception-dump/ulrich/myLibs/spack-packages/linux-debian11-zen2/gcc-11.2.0/mumps-5.5.1-umzads25ggryhklmvvto4a5i6ue74fbk/lib/libdmumps.so /import/exception-dump/ulrich/myLibs/spack-packages/linux-debian11-zen2/gcc-11.2.0/mumps-5.5.1-umzads25ggryhklmvvto4a5i6ue74fbk/lib/libmumps_common.so /import/exception-dump/ulrich/myLibs/spack-packages/linux-debian11-zen2/gcc-11.2.0/mumps-5.5.1-umzads25ggryhklmvvto4a5i6ue74fbk/lib/libpord.so" --with-trilinos=0 --with-fftw=0 --with-valgrind=0 --with-gmp=0 --with-libpng=0 --with-giflib=0 --with-mpfr=0 --with-netcdf=0 --with-pnetcdf=0 --with-moab=0 --with-random123=0 --with-exodusii=0 --with-cgns=0 --with-memkind=0 --with-p4est=0 --with-saws=0 --with-yaml=0 --with-hwloc=0 --with-libjpeg=0 --with-scalapack=1 --with-scalapack-lib=/import/exception-dump/ulrich/myLibs/spack-packages/linux-debian11-zen2/gcc-11.2.0/netlib-scalapack-2.2.0-vs6d5fhfa4i77gfo5iurpifbgxwfrmbp/lib/libscalapack.so --with-strumpack=0 --with-mmg=0 --with-parmmg=0 --with-tetgen=0
[1]PETSC ERROR: #1 register_PCs() at /tmp/ulrich/spack-stage/spack-stage-tandem-main-wnfinsgrh3swjkyofitwrffzldwourlj/spack-src/app/pc/register.cpp:18
[0]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
[0]PETSC ERROR: Petsc has generated inconsistent data
[0]PETSC ERROR: Invalid stack size 0, pop register_PCs /tmp/ulrich/spack-stage/spack-stage-tandem-main-wnfinsgrh3swjkyofitwrffzldwourlj/spack-src/app/pc/register.cpp:18.

[0]PETSC ERROR: WARNING! There are option(s) set that were not used! Could be the program crashed before they were used or a spelling mistake, etc!
[0]PETSC ERROR: Option left: name:-ksp_converged_reason (no value)
[0]PETSC ERROR: Option left: name:-ksp_max_it value: 400
[0]PETSC ERROR: Option left: name:-ksp_monitor (no value)
[0]PETSC ERROR: Option left: name:-ksp_rtol value: 1.0e-6
[0]PETSC ERROR: Option left: name:-ksp_type value: gcr
[0]PETSC ERROR: Option left: name:-ksp_view (no value)
[0]PETSC ERROR: Option left: name:-mat_type value: aijcusparse
[0]PETSC ERROR: Option left: name:-options_left (no value)
[0]PETSC ERROR: Option left: name:-pc_type value: jacobi
[0]PETSC ERROR: Option left: name:-vec_type value: cuda
[0]PETSC ERROR: See https://petsc.org/release/faq/ for trouble shooting.
[0]PETSC ERROR: Petsc Release Version 3.18.5, Feb 27, 2023 
[0]PETSC ERROR: --petsc on a  named heisenbug by ulrich Thu Aug 31 18:31:30 2023
[0]PETSC ERROR: Configure options --prefix=/import/exception-dump/ulrich/myLibs/spack-packages/linux-debian11-zen2/gcc-11.2.0/petsc-3.18.5-pppgzrc7ykm26yk4dmy7nzuf2f6jopah --with-ssl=0 --download-c2html=0 --download-sowing=0 --download-hwloc=0 --with-make-exec=make --with-cc=/import/exception-dump/ulrich/myLibs/spack-packages/linux-debian11-zen2/gcc-11.2.0/openmpi-4.1.5-sf7iuo3braeberbmnnyt4emapkxz4jtw/bin/mpicc --with-cxx=/import/exception-dump/ulrich/myLibs/spack-packages/linux-debian11-zen2/gcc-11.2.0/openmpi-4.1.5-sf7iuo3braeberbmnnyt4emapkxz4jtw/bin/mpic++ --with-fc=/import/exception-dump/ulrich/myLibs/spack-packages/linux-debian11-zen2/gcc-11.2.0/openmpi-4.1.5-sf7iuo3braeberbmnnyt4emapkxz4jtw/bin/mpif90 --with-precision=double --with-scalar-type=real --with-shared-libraries=1 --with-debugging=1 --with-openmp=0 --with-64-bit-indices=1 --with-blaslapack-lib=/import/exception-dump/ulrich/myLibs/spack-packages/linux-debian11-zen2/gcc-11.2.0/openblas-0.3.23-kxy7bxom4lam3wx44st22oytplyuisby/lib/libopenblas.so --with-memalign=32 --with-x=0 --with-clanguage=C --with-cuda=1 --with-cuda-dir=/import/exception-dump/ulrich/myLibs/spack-packages/linux-debian11-zen2/gcc-11.2.0/cuda-11.8.0-uivzrqah34rk27dutzqehubyc57zi46u --with-hip=0 --with-metis=1 --with-metis-include=/import/exception-dump/ulrich/myLibs/spack-packages/linux-debian11-zen2/gcc-11.2.0/metis-5.1.0-6zrgzhlze5ujsajtktgwf73dlpdjtdxk/include --with-metis-lib=/import/exception-dump/ulrich/myLibs/spack-packages/linux-debian11-zen2/gcc-11.2.0/metis-5.1.0-6zrgzhlze5ujsajtktgwf73dlpdjtdxk/lib/libmetis.so --with-hypre=1 --with-hypre-include=/import/exception-dump/ulrich/myLibs/spack-packages/linux-debian11-zen2/gcc-11.2.0/hypre-2.28.0-73iiib7speczh6bpszpkvbixgpkwyt7z/include --with-hypre-lib=/import/exception-dump/ulrich/myLibs/spack-packages/linux-debian11-zen2/gcc-11.2.0/hypre-2.28.0-73iiib7speczh6bpszpkvbixgpkwyt7z/lib/libHYPRE.so --with-parmetis=1 --with-parmetis-include=/import/exception-dump/ulrich/myLibs/spack-packages/linux-debian11-zen2/gcc-11.2.0/parmetis-4.0.3-gy6mgt3a4qt2ppnsorma4tabjw3f55fw/include --with-parmetis-lib=/import/exception-dump/ulrich/myLibs/spack-packages/linux-debian11-zen2/gcc-11.2.0/parmetis-4.0.3-gy6mgt3a4qt2ppnsorma4tabjw3f55fw/lib/libparmetis.so --with-kokkos=0 --with-kokkos-kernels=0 --with-superlu_dist=1 --with-superlu_dist-include=/import/exception-dump/ulrich/myLibs/spack-packages/linux-debian11-zen2/gcc-11.2.0/superlu-dist-8.1.2-7mrjdpdpu5l2g7x546r4kcssv2minjsa/include --with-superlu_dist-lib=/import/exception-dump/ulrich/myLibs/spack-packages/linux-debian11-zen2/gcc-11.2.0/superlu-dist-8.1.2-7mrjdpdpu5l2g7x546r4kcssv2minjsa/lib/libsuperlu_dist.so --with-ptscotch=0 --with-suitesparse=0 --with-hdf5=1 --with-hdf5-include=/import/exception-dump/ulrich/myLibs/spack-packages/linux-debian11-zen2/gcc-11.2.0/hdf5-1.14.2-dstkldnhuuoqsxniwh7piimjbyva4b2c/include --with-hdf5-lib=/import/exception-dump/ulrich/myLibs/spack-packages/linux-debian11-zen2/gcc-11.2.0/hdf5-1.14.2-dstkldnhuuoqsxniwh7piimjbyva4b2c/lib/libhdf5.so --with-zlib=1 --with-zlib-include=/import/exception-dump/ulrich/myLibs/spack-packages/linux-debian11-zen2/gcc-11.2.0/zlib-1.2.13-ufda23qzqsm5z2eqwsne76yoazck6fsz/include --with-zlib-lib=/import/exception-dump/ulrich/myLibs/spack-packages/linux-debian11-zen2/gcc-11.2.0/zlib-1.2.13-ufda23qzqsm5z2eqwsne76yoazck6fsz/lib/libz.so --with-mumps=1 --with-mumps-include=/import/exception-dump/ulrich/myLibs/spack-packages/linux-debian11-zen2/gcc-11.2.0/mumps-5.5.1-umzads25ggryhklmvvto4a5i6ue74fbk/include --with-mumps-lib="/import/exception-dump/ulrich/myLibs/spack-packages/linux-debian11-zen2/gcc-11.2.0/mumps-5.5.1-umzads25ggryhklmvvto4a5i6ue74fbk/lib/libsmumps.so /import/exception-dump/ulrich/myLibs/spack-packages/linux-debian11-zen2/gcc-11.2.0/mumps-5.5.1-umzads25ggryhklmvvto4a5i6ue74fbk/lib/libzmumps.so /import/exception-dump/ulrich/myLibs/spack-packages/linux-debian11-zen2/gcc-11.2.0/mumps-5.5.1-umzads25ggryhklmvvto4a5i6ue74fbk/lib/libcmumps.so /import/exception-dump/ulrich/myLibs/spack-packages/linux-debian11-zen2/gcc-11.2.0/mumps-5.5.1-umzads25ggryhklmvvto4a5i6ue74fbk/lib/libdmumps.so /import/exception-dump/ulrich/myLibs/spack-packages/linux-debian11-zen2/gcc-11.2.0/mumps-5.5.1-umzads25ggryhklmvvto4a5i6ue74fbk/lib/libmumps_common.so /import/exception-dump/ulrich/myLibs/spack-packages/linux-debian11-zen2/gcc-11.2.0/mumps-5.5.1-umzads25ggryhklmvvto4a5i6ue74fbk/lib/libpord.so" --with-trilinos=0 --with-fftw=0 --with-valgrind=0 --with-gmp=0 --with-libpng=0 --with-giflib=0 --with-mpfr=0 --with-netcdf=0 --with-pnetcdf=0 --with-moab=0 --with-random123=0 --with-exodusii=0 --with-cgns=0 --with-memkind=0 --with-p4est=0 --with-saws=0 --with-yaml=0 --with-hwloc=0 --with-libjpeg=0 --with-scalapack=1 --with-scalapack-lib=/import/exception-dump/ulrich/myLibs/spack-packages/linux-debian11-zen2/gcc-11.2.0/netlib-scalapack-2.2.0-vs6d5fhfa4i77gfo5iurpifbgxwfrmbp/lib/libscalapack.so --with-strumpack=0 --with-mmg=0 --with-parmmg=0 --with-tetgen=0
[0]PETSC ERROR: #1 register_PCs() at /tmp/ulrich/spack-stage/spack-stage-tandem-main-wnfinsgrh3swjkyofitwrffzldwourlj/spack-src/app/pc/register.cpp:18
hpc4geo commented 1 year ago

@Thomas-Ulrich Thomas - I believe be52dc6 (now in main) should resolve the error you encountered. The error stems from changes in PETSc and its error handler. Can you please try your test again?

Thomas-Ulrich commented 1 year ago

Hi Dave, It indeed saves the first problem.

(base) ulrich@heisenbug:/export/dump/ulrich/section_8_2/scenario-rc$ mpiexec -n 2 static ridge.toml --output ridgecrest --matrix_free false --petsc -ksp_max_it 400 -pc_type jacobi -ksp_rtol 1.0e-6 -ksp_type gcr -vec_type cuda -mat_type aijcusparse -ksp_view -ksp_monitor -ksp_converged_reason -options_left -log_view

               ___          ___         _____         ___          ___
      ___     /  /\        /__/\       /  /::\       /  /\        /__/\
     /  /\   /  /::\       \  \:\     /  /:/\:\     /  /:/_      |  |::\
    /  /:/  /  /:/\:\       \  \:\   /  /:/  \:\   /  /:/ /\     |  |:|:\
   /  /:/  /  /:/~/::\  _____\__\:\ /__/:/ \__\:| /  /:/ /:/_  __|__|:|\:\
  /  /::\ /__/:/ /:/\:\/__/::::::::\\  \:\ /  /://__/:/ /:/ /\/__/::::| \:\
 /__/:/\:\\  \:\/:/__\/\  \:\~~\~~\/ \  \:\  /:/ \  \:\/:/ /:/\  \:\~~\__\/
 \__\/  \:\\  \::/      \  \:\  ~~~   \  \:\/:/   \  \::/ /:/  \  \:\
      \  \:\\  \:\       \  \:\        \  \::/     \  \:\/:/    \  \:\
       \__\/ \  \:\       \  \:\        \__\/       \  \::/      \  \:\
              \__\/        \__\/                     \__\/        \__\/

                          tandem version be52dc6

                       stack size limit = unlimited

                              Worker affinity
    0---------|----------|----------|----------|----------|----------|
    ----------|----------|----------|----------|----------|----------|
    --------8-|----------|----------|----------|----------|----------|
    ----------|----------|----------|----------|----------|----------|
    ----------|------

Ridge:create()
parsing fault_many_trace.csv
Ridge:create()
parsing fault_many_trace.csv
DOFs: 12634590
Mesh size: 36.7959
[0]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
[0]PETSC ERROR: Object is in wrong state
[0]PETSC ERROR: Must call MatXXXSetPreallocation(), MatSetUp() or the matrix has not yet been factored on argument 1 "A" before MatSetOption_MPIAIJ()
[0]PETSC ERROR: WARNING! There are option(s) set that were not used! Could be the program crashed before they were used or a spelling mistake, etc!
[0]PETSC ERROR: Option left: name:-ksp_converged_reason (no value)
[0]PETSC ERROR: Option left: name:-ksp_max_it value: 400
[0]PETSC ERROR: Option left: name:-ksp_monitor (no value)
[0]PETSC ERROR: Option left: name:-ksp_rtol value: 1.0e-6
[0]PETSC ERROR: Option left: name:-ksp_type value: gcr
[0]PETSC ERROR: Option left: name:-ksp_view (no value)
[0]PETSC ERROR: Option left: name:-options_left (no value)
[0]PETSC ERROR: Option left: name:-pc_type value: jacobi
[0]PETSC ERROR: Option left: name:-vec_type value: cuda
[0]PETSC ERROR: See https://petsc.org/release/faq/ for trouble shooting.
[0]PETSC ERROR: Petsc Release Version 3.18.5, Feb 27, 2023 
[0]PETSC ERROR: [1]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
[1]PETSC ERROR: Object is in wrong state
[1]PETSC ERROR: Must call MatXXXSetPreallocation(), MatSetUp() or the matrix has not yet been factored on argument 1 "A" before MatSetOption_MPIAIJ()
[1]PETSC ERROR: WARNING! There are option(s) set that were not used! Could be the program crashed before they were used or a spelling mistake, etc!
[1]PETSC ERROR: Option left: name:-ksp_converged_reason (no value)
[1]PETSC ERROR: Option left: name:-ksp_max_it value: 400
[1]PETSC ERROR: Option left: name:-ksp_monitor (no value)
[1]PETSC ERROR: Option left: name:-ksp_rtol value: 1.0e-6
[1]PETSC ERROR: Option left: name:-ksp_type value: gcr
[1]PETSC ERROR: Option left: name:-ksp_view (no value)
[1]PETSC ERROR: Option left: name:-options_left (no value)
[1]PETSC ERROR: Option left: name:-pc_type value: jacobi
[1]PETSC ERROR: Option left: name:-vec_type value: cuda
[1]PETSC ERROR: See https://petsc.org/release/faq/ for trouble shooting.
[1]PETSC ERROR: Petsc Release Version 3.18.5, Feb 27, 2023 
[1]PETSC ERROR: --petsc on a  named heisenbug by ulrich Mon Sep 11 09:50:07 2023
[1]PETSC ERROR: Configure options --prefix=/import/exception-dump/ulrich/myLibs/spack-packages/linux-debian11-zen2/gcc-11.2.0/petsc-3.18.5-pppgzrc7ykm26yk4dmy7nzuf2f6jopah --with-ssl=0 --download-c2html=0 --download-sowing=0 --download-hwloc=0 --with-make-exec=make --with-cc=/import/exception-dump/ulrich/myLibs/spack-packages/linux-debian11-zen2/gcc-11.2.0/openmpi-4.1.5-sf7iuo3braeberbmnnyt4emapkxz4jtw/bin/mpicc --with-cxx=/import/exception-dump/ulrich/myLibs/spack-packages/linux-debian11-zen2/gcc-11.2.0/openmpi-4.1.5-sf7iuo3braeberbmnnyt4emapkxz4jtw/bin/mpic++ --with-fc=/import/exception-dump/ulrich/myLibs/spack-packages/linux-debian11-zen2/gcc-11.2.0/openmpi-4.1.5-sf7iuo3braeberbmnnyt4emapkxz4jtw/bin/mpif90 --with-precision=double --with-scalar-type=real --with-shared-libraries=1 --with-debugging=1 --with-openmp=0 --with-64-bit-indices=1 --with-blaslapack-lib=/import/exception-dump/ulrich/myLibs/spack-packages/linux-debian11-zen2/gcc-11.2.0/openblas-0.3.23-kxy7bxom4lam3wx44st22oytplyuisby/lib/libopenblas.so --with-memalign=32 --with-x=0 --with-clanguage=C --with-cuda=1 --with-cuda-dir=/import/exception-dump/ulrich/myLibs/spack-packages/linux-debian11-zen2/gcc-11.2.0/cuda-11.8.0-uivzrqah34rk27dutzqehubyc57zi46u --with-hip=0 --with-metis=1 --with-metis-include=/import/exception-dump/ulrich/myLibs/spack-packages/linux-debian11-zen2/gcc-11.2.0/metis-5.1.0-6zrgzhlze5ujsajtktgwf73dlpdjtdxk/include --with-metis-lib=/import/exception-dump/ulrich/myLibs/spack-packages/linux-debian11-zen2/gcc-11.2.0/metis-5.1.0-6zrgzhlze5ujsajtktgwf73dlpdjtdxk/lib/libmetis.so --with-hypre=1 --with-hypre-include=/import/exception-dump/ulrich/myLibs/spack-packages/linux-debian11-zen2/gcc-11.2.0/hypre-2.28.0-73iiib7speczh6bpszpkvbixgpkwyt7z/include --with-hypre-lib=/import/exception-dump/ulrich/myLibs/spack-packages/linux-debian11-zen2/gcc-11.2.0/hypre-2.28.0-73iiib7speczh6bpszpkvbixgpkwyt7z/lib/libHYPRE.so --with-parmetis=1 --with-parmetis-include=/import/exception-dump/ulrich/myLibs/spack-packages/linux-debian11-zen2/gcc-11.2.0/parmetis-4.0.3-gy6mgt3a4qt2ppnsorma4tabjw3f55fw/include --with-parmetis-lib=/import/exception-dump/ulrich/myLibs/spack-packages/linux-debian11-zen2/gcc-11.2.0/parmetis-4.0.3-gy6mgt3a4qt2ppnsorma4tabjw3f55fw/lib/libparmetis.so --with-kokkos=0 --with-kokkos-kernels=0 --with-superlu_dist=1 --with-superlu_dist-include=/import/exception-dump/ulrich/myLibs/spack-packages/linux-debian11-zen2/gcc-11.2.0/superlu-dist-8.1.2-7mrjdpdpu5l2g7x546r4kcssv2minjsa/include --with-superlu_dist-lib=/import/exception-dump/ulrich/myLibs/spack-packages/linux-debian11-zen2/gcc-11.2.0/superlu-dist-8.1.2-7mrjdpdpu5l2g7x546r4kcssv2minjsa/lib/libsuperlu_dist.so --with-ptscotch=0 --with-suitesparse=0 --with-hdf5=1 --with-hdf5-include=/import/exception-dump/ulrich/myLibs/spack-packages/linux-debian11-zen2/gcc-11.2.0/hdf5-1.14.2-dstkldnhuuoqsxniwh7piimjbyva4b2c/include --with-hdf5-lib=/import/exception-dump/ulrich/myLibs/spack-packages/linux-debian11-zen2/gcc-11.2.0/hdf5-1.14.2-dstkldnhuuoqsxniwh7piimjbyva4b2c/lib/libhdf5.so --with-zlib=1 --with-zlib-include=/import/exception-dump/ulrich/myLibs/spack-packages/linux-debian11-zen2/gcc-11.2.0/zlib-1.2.13-ufda23qzqsm5z2eqwsne76yoazck6fsz/include --with-zlib-lib=/import/exception-dump/ulrich/myLibs/spack-packages/linux-debian11-zen2/gcc-11.2.0/zlib-1.2.13-ufda23qzqsm5z2eqwsne76yoazck6fsz/lib/libz.so --with-mumps=1 --with-mumps-include=/import/exception-dump/ulrich/myLibs/spack-packages/linux-debian11-zen2/gcc-11.2.0/mumps-5.5.1-umzads25ggryhklmvvto4a5i6ue74fbk/include --with-mumps-lib="/import/exception-dump/ulrich/myLibs/spack-packages/linux-debian11-zen2/gcc-11.2.0/mumps-5.5.1-umzads25ggryhklmvvto4a5i6ue74fbk/lib/libsmumps.so /i--petsc on a  named heisenbug by ulrich Mon Sep 11 09:50:07 2023
[0]PETSC ERROR: Configure options --prefix=/import/exception-dump/ulrich/myLibs/spack-packages/linux-debian11-zen2/gcc-11.2.0/petsc-3.18.5-pppgzrc7ykm26yk4dmy7nzuf2f6jopah --with-ssl=0 --download-c2html=0 --download-sowing=0 --download-hwloc=0 --with-make-exec=make --with-cc=/import/exception-dump/ulrich/myLibs/spack-packages/linux-debian11-zen2/gcc-11.2.0/openmpi-4.1.5-sf7iuo3braeberbmnnyt4emapkxz4jtw/bin/mpicc --with-cxx=/import/exception-dump/ulrich/myLibs/spack-packages/linux-debian11-zen2/gcc-11.2.0/openmpi-4.1.5-sf7iuo3braeberbmnnyt4emapkxz4jtw/bin/mpic++ --with-fc=/import/exception-dump/ulrich/myLibs/spack-packages/linux-debian11-zen2/gcc-11.2.0/openmpi-4.1.5-sf7iuo3braeberbmnnyt4emapkxz4jtw/bin/mpif90 --with-precision=double --with-scalar-type=real --with-shared-libraries=1 --with-debugging=1 --with-openmp=0 --with-64-bit-indices=1 --with-blaslapack-lib=/import/exception-dump/ulrich/myLibs/spack-packages/linux-debian11-zen2/gcc-11.2.0/openblas-0.3.23-kxy7bxom4lam3wx44st22oytplyuisby/lib/libopenblas.so --with-memalign=32 --with-x=0 --with-clanguage=C --with-cuda=1 --with-cuda-dir=/import/exception-dump/ulrich/myLibs/spack-packages/linux-debian11-zen2/gcc-11.2.0/cuda-11.8.0-uivzrqah34rk27dutzqehubyc57zi46u --with-hip=0 --with-metis=1 --with-metis-include=/import/exception-dump/ulrich/myLibs/spack-packages/linux-debian11-zen2/gcc-11.2.0/metis-5.1.0-6zrgzhlze5ujsajtktgwf73dlpdjtdxk/include --with-metis-lib=/import/exception-dump/ulrich/myLibs/spack-packages/linux-debian11-zen2/gcc-11.2.0/metis-5.1.0-6zrgzhlze5ujsajtktgwf73dlpdjtdxk/lib/libmetis.so --with-hypre=1 --with-hypre-include=/import/exception-dump/ulrich/myLibs/spack-packages/linux-debian11-zen2/gcc-11.2.0/hypre-2.28.0-73iiib7speczh6bpszpkvbixgpkwyt7z/include --with-hypre-lib=/import/exception-dump/ulrich/myLibs/spack-packages/linux-debian11-zen2/gcc-11.2.0/hypre-2.28.0-73iiib7speczh6bpszpkvbixgpkwyt7z/lib/libHYPRE.so --with-parmetis=1 --with-parmetis-include=/import/exception-dump/ulrich/myLibs/spack-packages/linux-debian11-zen2/gcc-11.2.0/parmetis-4.0.3-gy6mgt3a4qt2ppnsorma4tabjw3f55fw/include --with-parmetis-lib=/import/exception-dump/ulrich/myLibs/spack-packages/linux-debian11-zen2/gcc-11.2.0/parmetis-4.0.3-gy6mgt3a4qt2ppnsorma4tabjw3f55fw/lib/libparmetis.so --with-kokkos=0 --with-kokkos-kernels=0 --with-superlu_dist=1 --with-superlu_dist-include=/import/exception-dump/ulrich/myLibs/spack-packages/linux-debian11-zen2/gcc-11.2.0/superlu-dist-8.1.2-7mrjdpdpu5l2g7x546r4kcssv2minjsa/include --with-superlu_dist-lib=/import/exception-dump/ulrich/myLibs/spack-packages/linux-debian11-zen2/gcc-11.2.0/superlu-dist-8.1.2-7mrjdpdpu5l2g7x546r4kcssv2minjsa/lib/libsuperlu_dist.so --with-ptscotch=0 --with-suitesparse=0 --with-hdf5=1 --with-hdf5-include=/import/exception-dump/ulrich/myLibs/spack-packages/linux-debian11-zen2/gcc-11.2.0/hdf5-1.14.2-dstkldnhuuoqsxniwh7piimjbyva4b2c/include --with-hdf5-lib=/import/exception-dump/ulrich/myLibs/spack-packages/linux-debian11-zen2/gcc-11.2.0/hdf5-1.14.2-dstkldnhuuoqsxniwh7piimjbyva4b2c/lib/libhdf5.so --with-zlib=1 --with-zlib-include=/import/exception-dump/ulrich/myLibs/spack-packages/linux-debian11-zen2/gcc-11.2.0/zlib-1.2.13-ufda23qzqsm5z2eqwsne76yoazck6fsz/include --with-zlib-lib=/import/exception-dump/ulrich/myLibs/spack-packages/linux-debian11-zen2/gcc-11.2.0/zlib-1.2.13-ufda23qzqsm5z2eqwsne76yoazck6fsz/lib/libz.so --with-mumps=1 --with-mumps-include=/import/exception-dump/ulrich/myLibs/spack-packages/linux-debian11-zen2/gcc-11.2.0/mumps-5.5.1-umzads25ggryhklmvvto4a5i6ue74fbk/include --with-mumps-lib="/import/exception-dump/ulrich/myLibs/spack-packages/linux-debian11-zen2/gcc-11.2.0/mumps-5.5.1-umzads25ggryhklmvvto4a5i6ue74fbk/lib/libsmumps.so /import/exception-dump/ulrich/myLibs/spack-packages/linux-debian11-zen2/gcc-11.2.0/mumps-5.5.1-umzads25ggryhklmvvto4a5i6ue74fbk/lib/libzmumps.so /import/exception-dump/ulrich/myLibs/spack-packages/linux-debian11-zen2/gcc-11.2.0/mumps-5.5.1-umzads25ggryhklmvvto4a5i6ue74fbk/lib/libcmumps.so /import/exception-dump/ulrich/myLibs/spack-packages/linux-debian11-zen2/gcc-11.2.0/mumps-5.5.1-umzads25ggryhklmvvto4a5i6ue74fbk/lib/libdmumps.so /import/exception-dump/ulrich/myLibs/spack-packages/linux-debian11-zen2/gcc-11.2.0/mumps-5.5.1-umzads25ggryhklmvvto4a5i6ue74fbk/lib/libmumps_common.so /import/exception-dump/ulrich/myLibs/spack-packages/linux-debian11-zen2/gcc-11.2.0/mumps-5.5.1-umzads25ggryhklmvvto4a5i6ue74fbk/lib/libpord.so" --with-trilinos=0 --with-fftw=0 --with-valgrind=0 --with-gmp=0 --with-libpng=0 --with-giflib=0 --with-mpfr=0 --with-netcdf=0 --with-pnetcdf=0 --with-moab=0 --with-random123=0 --with-exodusii=0 --with-cgns=0 --with-memkind=0 --with-p4est=0 --with-saws=0 --with-yaml=0 --with-hwloc=0 --with-libjpeg=0 --with-scalapack=1 --with-scalapack-lib=/import/exception-dump/ulrich/myLibs/spack-packages/linux-debian11-zen2/gcc-11.2.0/netlib-scalapack-2.2.0-vs6d5fhfa4i77gfo5iurpifbgxwfrmbp/lib/libscalapack.so --with-strumpack=0 --with-mmg=0 --with-parmmg=0 --with-tetgen=0
[0]PETSC ERROR: #1 MatSetOption_MPIAIJ() at /tmp/ulrich/spack-stage/spack-stage-petsc-3.18.5-pppgzrc7ykm26yk4dmy7nzuf2f6jopah/spack-src/src/mat/impls/aij/mpi/mpiaij.c:1695
[0]PETSC ERROR: #2 MatSetOption() at /tmp/ulrich/spack-stage/spack-stage-petsc-3.18.5-pppgzrc7ykm26yk4dmy7nzuf2f6jopah/spack-src/src/mat/interface/matrix.c:5965
[0]PETSC ERROR: #3 PetscDGMatrix() at /tmp/ulrich/spack-stage/spack-stage-tandem-main-wnfinsgrh3swjkyofitwrffzldwourlj/spack-src/app/common/PetscDGMatrix.cpp:63
terminate called after throwing an instance of 'tndm::petsc_error'
mport/exception-dump/ulrich/myLibs/spack-packages/linux-debian11-zen2/gcc-11.2.0/mumps-5.5.1-umzads25ggryhklmvvto4a5i6ue74fbk/lib/libzmumps.so /import/exception-dump/ulrich/myLibs/spack-packages/linux-debian11-zen2/gcc-11.2.0/mumps-5.5.1-umzads25ggryhklmvvto4a5i6ue74fbk/lib/libcmumps.so /import/exception-dump/ulrich/myLibs/spack-packages/linux-debian11-zen2/gcc-11.2.0/mumps-5.5.1-umzads25ggryhklmvvto4a5i6ue74fbk/lib/libdmumps.so /import/exception-dump/ulrich/myLibs/spack-packages/linux-debian11-zen2/gcc-11.2.0/mumps-5.5.1-umzads25ggryhklmvvto4a5i6ue74fbk/lib/libmumps_common.so /import/exception-dump/ulrich/myLibs/spack-packages/linux-debian11-zen2/gcc-11.2.0/mumps-5.5.1-umzads25ggryhklmvvto4a5i6ue74fbk/lib/libpord.so" --with-trilinos=0 --with-fftw=0 --with-valgrind=0 --with-gmp=0 --with-libpng=0 --with-giflib=0 --with-mpfr=0 --with-netcdf=0 --with-pnetcdf=0 --with-moab=0 --with-random123=0 --with-exodusii=0 --with-cgns=0 --with-memkind=0 --with-p4est=0 --with-saws=0 --with-yaml=0 --with-hwloc=0 --with-libjpeg=0 --with-scalapack=1 --with-scalapack-lib=/import/exception-dump/ulrich/myLibs/spack-packages/linux-debian11-zen2/gcc-11.2.0/netlib-scalapack-2.2.0-vs6d5fhfa4i77gfo5iurpifbgxwfrmbp/lib/libscalapack.so --with-strumpack=0 --with-mmg=0 --with-parmmg=0 --with-tetgen=0
[1]PETSC ERROR: #1 MatSetOption_MPIAIJ() at /tmp/ulrich/spack-stage/spack-stage-petsc-3.18.5-pppgzrc7ykm26yk4dmy7nzuf2f6jopah/spack-src/src/mat/impls/aij/mpi/mpiaij.c:1695
[1]PETSC ERROR: #2 MatSetOption() at /tmp/ulrich/spack-stage/spack-stage-petsc-3.18.5-pppgzrc7ykm26yk4dmy7nzuf2f6jopah/spack-src/src/mat/interface/matrix.c:5965
[1]PETSC ERROR: #3 PetscDGMatrix() at /tmp/ulrich/spack-stage/spack-stage-tandem-main-wnfinsgrh3swjkyofitwrffzldwourlj/spack-src/app/common/PetscDGMatrix.cpp:63
terminate called after throwing an instance of 'tndm::petsc_error'
  what():  Object is in wrong state
[heisenbug:1769346] *** Process received signal ***
[heisenbug:1769346] Signal: Aborted (6)
[heisenbug:1769346] Associated errno: Unknown error -1 (-1)
[heisenbug:1769346] Signal code:  (1)
  what():  Object is in wrong state
[heisenbug:1769347] *** Process received signal ***
[heisenbug:1769347] Signal: Aborted (6)
[heisenbug:1769347] Signal code: User function (kill, sigsend, abort, etc.) (0)
[heisenbug:1769347] [ 0] /lib/x86_64-linux-gnu/libc.so.6(+0x38d60)[0x1531cc9bfd60]
[heisenbug:1769347] [ 1] [heisenbug:1769346] [ 0] /lib/x86_64-linux-gnu/libc.so.6(gsignal+0x141)[0x1531cc9bfce1]
[heisenbug:1769347] [ 2] /lib/x86_64-linux-gnu/libc.so.6(+0x38d60)[0x15190e181d60]
[heisenbug:1769346] [ 1] /lib/x86_64-linux-gnu/libc.so.6(abort+0x123)[0x1531cc9a9537]
[heisenbug:1769347] [ 3] /lib/x86_64-linux-gnu/libc.so.6(gsignal+0x141)[0x15190e181ce1]
[heisenbug:1769346] [ 2] /lib/x86_64-linux-gnu/libc.so.6(abort+0x123)[0x15190e16b537]
[heisenbug:1769346] [ 3] /import/exception-dump/ulrich/myLibs/spack-packages/linux-debian11-zen2/gcc-12.2.0/gcc-11.2.0-iokm55jhztsmmlbgy3mr5rvu3y43qxyz/lib64/libstdc++.so.6(+0xa58e3)[0x1531ccc1d8e3]
[heisenbug:1769347] [ 4] /import/exception-dump/ulrich/myLibs/spack-packages/linux-debian11-zen2/gcc-12.2.0/gcc-11.2.0-iokm55jhztsmmlbgy3mr5rvu3y43qxyz/lib64/libstdc++.so.6(+0xa58e3)[0x15190e3df8e3]
[heisenbug:1769346] [ 4] /import/exception-dump/ulrich/myLibs/spack-packages/linux-debian11-zen2/gcc-12.2.0/gcc-11.2.0-iokm55jhztsmmlbgy3mr5rvu3y43qxyz/lib64/libstdc++.so.6(+0xb1cda)[0x1531ccc29cda]
[heisenbug:1769347] [ 5] /import/exception-dump/ulrich/myLibs/spack-packages/linux-debian11-zen2/gcc-12.2.0/gcc-11.2.0-iokm55jhztsmmlbgy3mr5rvu3y43qxyz/lib64/libstdc++.so.6(+0xb1d45)[0x1531ccc29d45]
[heisenbug:1769347] [ 6] /import/exception-dump/ulrich/myLibs/spack-packages/linux-debian11-zen2/gcc-12.2.0/gcc-11.2.0-iokm55jhztsmmlbgy3mr5rvu3y43qxyz/lib64/libstdc++.so.6(+0xb1cda)[0x15190e3ebcda]
[heisenbug:1769346] [ 5] /import/exception-dump/ulrich/myLibs/spack-packages/linux-debian11-zen2/gcc-12.2.0/gcc-11.2.0-iokm55jhztsmmlbgy3mr5rvu3y43qxyz/lib64/libstdc++.so.6(+0xb1ff7)[0x1531ccc29ff7]
[heisenbug:1769347] [ 7] static[0x41732b]
[heisenbug:1769347] [ 8] static[0x46f426]
[heisenbug:1769347] [ 9] static[0x445145]
[heisenbug:1769347] [10] static[0x41b9a3]
[heisenbug:1769347] [11] /import/exception-dump/ulrich/myLibs/spack-packages/linux-debian11-zen2/gcc-12.2.0/gcc-11.2.0-iokm55jhztsmmlbgy3mr5rvu3y43qxyz/lib64/libstdc++.so.6(+0xb1d45)[0x15190e3ebd45]
[heisenbug:1769346] [ 6] /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xea)[0x1531cc9aad0a]
[heisenbug:1769347] [12] static[0x41f3ea]
[heisenbug:1769347] *** End of error message ***
/import/exception-dump/ulrich/myLibs/spack-packages/linux-debian11-zen2/gcc-12.2.0/gcc-11.2.0-iokm55jhztsmmlbgy3mr5rvu3y43qxyz/lib64/libstdc++.so.6(+0xb1ff7)[0x15190e3ebff7]
[heisenbug:1769346] [ 7] static[0x41732b]
[heisenbug:1769346] [ 8] static[0x46f426]
[heisenbug:1769346] [ 9] static[0x445145]
[heisenbug:1769346] [10] static[0x41b9a3]
[heisenbug:1769346] [11] /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xea)[0x15190e16cd0a]
[heisenbug:1769346] [12] static[0x41f3ea]
[heisenbug:1769346] *** End of error message ***
--------------------------------------------------------------------------
Primary job  terminated normally, but 1 process returned
a non-zero exit code. Per user-direction, the job has been aborted.
--------------------------------------------------------------------------
--------------------------------------------------------------------------
mpiexec noticed that process rank 0 with PID 0 on node heisenbug exited on signal 6 (Aborted).
--------------------------------------------------------------------------
hpc4geo commented 1 year ago

@Thomas-Ulrich Great you ran that test and got past the immediate error. The new error is also resolved on branch dmay/petsc-gpu. Could you please re-run your test again using this branch?

Thomas-Ulrich commented 1 year ago

It worked!

(base) ulrich@heisenbug:/export/dump/ulrich/section_8_2/scenario-rc$ mpiexec -n 2 static ridge.toml --output ridgecrest --matrix_free false --petsc -ksp_max_it 400 -pc_type jacobi -ksp_rtol 1.0e-6 -ksp_type gcr -vec_type cuda -mat_type aijcusparse -ksp_view -ksp_monitor -ksp_converged_reason -options_left -log_view

               ___          ___         _____         ___          ___
      ___     /  /\        /__/\       /  /::\       /  /\        /__/\
     /  /\   /  /::\       \  \:\     /  /:/\:\     /  /:/_      |  |::\
    /  /:/  /  /:/\:\       \  \:\   /  /:/  \:\   /  /:/ /\     |  |:|:\
   /  /:/  /  /:/~/::\  _____\__\:\ /__/:/ \__\:| /  /:/ /:/_  __|__|:|\:\
  /  /::\ /__/:/ /:/\:\/__/::::::::\\  \:\ /  /://__/:/ /:/ /\/__/::::| \:\
 /__/:/\:\\  \:\/:/__\/\  \:\~~\~~\/ \  \:\  /:/ \  \:\/:/ /:/\  \:\~~\__\/
 \__\/  \:\\  \::/      \  \:\  ~~~   \  \:\/:/   \  \::/ /:/  \  \:\
      \  \:\\  \:\       \  \:\        \  \::/     \  \:\/:/    \  \:\
       \__\/ \  \:\       \  \:\        \__\/       \  \::/      \  \:\
              \__\/        \__\/                     \__\/        \__\/

                          tandem version 4b6c45e

                       stack size limit = unlimited

                              Worker affinity
    0---------|----------|----------|----------|----------|----------|
    ----------|----------|----------|----------|----------|----------|
    --------8-|----------|----------|----------|----------|----------|
    ----------|----------|----------|----------|----------|----------|
    ----------|------

Ridge:create()
parsing fault_many_trace.csv
Ridge:create()
parsing fault_many_trace.csv
DOFs: 12634590
Mesh size: 36.7959
Assembly: 236.758 s
Solver warmup: 3.39949 s
  0 KSP Residual norm 1.043596231397e+04 
  1 KSP Residual norm 5.057045424157e+03 
  2 KSP Residual norm 2.501262926538e+03 
  3 KSP Residual norm 1.404021748561e+03 
  4 KSP Residual norm 9.405595891919e+02 
  5 KSP Residual norm 6.255057299835e+02 
  6 KSP Residual norm 4.470847627376e+02 
  7 KSP Residual norm 3.490249256949e+02 
  8 KSP Residual norm 2.809121341066e+02 
  9 KSP Residual norm 2.221057145177e+02 
 10 KSP Residual norm 1.748130469702e+02 
 11 KSP Residual norm 1.401902300234e+02 
 12 KSP Residual norm 1.164758176460e+02 
 13 KSP Residual norm 9.867590915172e+01 
 14 KSP Residual norm 8.539765892122e+01 
 15 KSP Residual norm 7.650900132972e+01 
 16 KSP Residual norm 7.000515989883e+01 
 17 KSP Residual norm 6.498017121417e+01 
 18 KSP Residual norm 6.110665939969e+01 
 19 KSP Residual norm 5.791056463201e+01 
 20 KSP Residual norm 5.505070821390e+01 
 21 KSP Residual norm 5.226404388711e+01 
 22 KSP Residual norm 4.943990647117e+01 
 23 KSP Residual norm 4.665549043271e+01 
 24 KSP Residual norm 4.399544536644e+01 
 25 KSP Residual norm 4.154164239417e+01 
 26 KSP Residual norm 3.939470141669e+01 
 27 KSP Residual norm 3.756470281537e+01 
 28 KSP Residual norm 3.598582603206e+01 
 29 KSP Residual norm 3.464281664502e+01 
 30 KSP Residual norm 3.347478047406e+01 
 31 KSP Residual norm 3.297684428650e+01 
 32 KSP Residual norm 3.258314495897e+01 
 33 KSP Residual norm 3.198439669474e+01 
 34 KSP Residual norm 3.125069693726e+01 
 35 KSP Residual norm 3.050668986486e+01 
 36 KSP Residual norm 2.968574246339e+01 
 37 KSP Residual norm 2.880077966912e+01 
 38 KSP Residual norm 2.804585632615e+01 
 39 KSP Residual norm 2.729940854350e+01 
 40 KSP Residual norm 2.643360874144e+01 
 41 KSP Residual norm 2.565851385890e+01 
 42 KSP Residual norm 2.505383388313e+01 
 43 KSP Residual norm 2.445484086684e+01 
 44 KSP Residual norm 2.353615935008e+01 
 45 KSP Residual norm 2.275423567003e+01 
 46 KSP Residual norm 2.176545206889e+01 
 47 KSP Residual norm 2.080768417124e+01 
 48 KSP Residual norm 2.001738877707e+01 
 49 KSP Residual norm 1.913086340748e+01 
 50 KSP Residual norm 1.827692355724e+01 
 51 KSP Residual norm 1.768092737476e+01 
 52 KSP Residual norm 1.717450697766e+01 
 53 KSP Residual norm 1.668094961826e+01 
 54 KSP Residual norm 1.621022411943e+01 
 55 KSP Residual norm 1.575309194028e+01 
 56 KSP Residual norm 1.539103535047e+01 
 57 KSP Residual norm 1.512056339802e+01 
 58 KSP Residual norm 1.481764991587e+01 
 59 KSP Residual norm 1.458155833150e+01 
 60 KSP Residual norm 1.429919323552e+01 
 61 KSP Residual norm 1.415517037049e+01 
 62 KSP Residual norm 1.399234017229e+01 
 63 KSP Residual norm 1.378311668948e+01 
 64 KSP Residual norm 1.361947706540e+01 
 65 KSP Residual norm 1.344335545506e+01 
 66 KSP Residual norm 1.326532197942e+01 
 67 KSP Residual norm 1.309807648346e+01 
 68 KSP Residual norm 1.296657427835e+01 
 69 KSP Residual norm 1.283695108698e+01 
 70 KSP Residual norm 1.267007359103e+01 
 71 KSP Residual norm 1.244710456209e+01 
 72 KSP Residual norm 1.220370234429e+01 
 73 KSP Residual norm 1.203832481144e+01 
 74 KSP Residual norm 1.187122962688e+01 
 75 KSP Residual norm 1.170152324190e+01 
 76 KSP Residual norm 1.153462959335e+01 
 77 KSP Residual norm 1.134697897958e+01 
 78 KSP Residual norm 1.117215824238e+01 
 79 KSP Residual norm 1.101994065827e+01 
 80 KSP Residual norm 1.086476019357e+01 
 81 KSP Residual norm 1.070942975098e+01 
 82 KSP Residual norm 1.052267996890e+01 
 83 KSP Residual norm 1.034052442724e+01 
 84 KSP Residual norm 1.021009119411e+01 
 85 KSP Residual norm 1.010111395955e+01 
 86 KSP Residual norm 9.999392332292e+00 
 87 KSP Residual norm 9.905773804438e+00 
 88 KSP Residual norm 9.801442737859e+00 
 89 KSP Residual norm 9.677046033847e+00 
 90 KSP Residual norm 9.556033128703e+00 
 91 KSP Residual norm 9.478547429629e+00 
 92 KSP Residual norm 9.407015314605e+00 
 93 KSP Residual norm 9.333198724367e+00 
 94 KSP Residual norm 9.256767561751e+00 
 95 KSP Residual norm 9.190387305344e+00 
 96 KSP Residual norm 9.126105862277e+00 
 97 KSP Residual norm 9.065686270195e+00 
 98 KSP Residual norm 8.999945759040e+00 
 99 KSP Residual norm 8.926154965140e+00 
100 KSP Residual norm 8.849610367048e+00 
101 KSP Residual norm 8.789846778250e+00 
102 KSP Residual norm 8.741873464051e+00 
103 KSP Residual norm 8.682880521869e+00 
104 KSP Residual norm 8.608840997956e+00 
105 KSP Residual norm 8.538616256590e+00 
106 KSP Residual norm 8.447613366794e+00 
107 KSP Residual norm 8.369318452434e+00 
108 KSP Residual norm 8.293197836615e+00 
109 KSP Residual norm 8.186303860330e+00 
110 KSP Residual norm 8.080866637053e+00 
111 KSP Residual norm 8.009963000677e+00 
112 KSP Residual norm 7.936414858358e+00 
113 KSP Residual norm 7.858744503449e+00 
114 KSP Residual norm 7.785325044790e+00 
115 KSP Residual norm 7.718060802479e+00 
116 KSP Residual norm 7.652779344931e+00 
117 KSP Residual norm 7.597380224417e+00 
118 KSP Residual norm 7.537225446873e+00 
119 KSP Residual norm 7.483421623934e+00 
120 KSP Residual norm 7.423911804535e+00 
121 KSP Residual norm 7.379583814341e+00 
122 KSP Residual norm 7.331324777632e+00 
123 KSP Residual norm 7.272492000865e+00 
124 KSP Residual norm 7.222459053248e+00 
125 KSP Residual norm 7.172445329407e+00 
126 KSP Residual norm 7.131041426961e+00 
127 KSP Residual norm 7.096845311950e+00 
128 KSP Residual norm 7.068263965019e+00 
129 KSP Residual norm 7.041589260996e+00 
130 KSP Residual norm 7.007516785897e+00 
131 KSP Residual norm 6.953625587450e+00 
132 KSP Residual norm 6.898105685477e+00 
133 KSP Residual norm 6.861636296388e+00 
134 KSP Residual norm 6.825263405788e+00 
135 KSP Residual norm 6.782291761785e+00 
136 KSP Residual norm 6.741969965335e+00 
137 KSP Residual norm 6.692448746921e+00 
138 KSP Residual norm 6.638196566239e+00 
139 KSP Residual norm 6.593128517003e+00 
140 KSP Residual norm 6.550147749365e+00 
141 KSP Residual norm 6.493865509721e+00 
142 KSP Residual norm 6.425750033650e+00 
143 KSP Residual norm 6.361853225145e+00 
144 KSP Residual norm 6.315089040734e+00 
145 KSP Residual norm 6.279686250644e+00 
146 KSP Residual norm 6.250052873209e+00 
147 KSP Residual norm 6.222320326304e+00 
148 KSP Residual norm 6.188458221744e+00 
149 KSP Residual norm 6.141144743962e+00 
150 KSP Residual norm 6.096183275765e+00 
151 KSP Residual norm 6.063027506216e+00 
152 KSP Residual norm 6.032543296161e+00 
153 KSP Residual norm 6.005497369794e+00 
154 KSP Residual norm 5.978094977646e+00 
155 KSP Residual norm 5.954260349365e+00 
156 KSP Residual norm 5.929271521578e+00 
157 KSP Residual norm 5.905501967380e+00 
158 KSP Residual norm 5.878640338841e+00 
159 KSP Residual norm 5.845898054536e+00 
160 KSP Residual norm 5.814149282054e+00 
161 KSP Residual norm 5.793082821883e+00 
162 KSP Residual norm 5.775315899151e+00 
163 KSP Residual norm 5.750861401791e+00 
164 KSP Residual norm 5.721078852141e+00 
165 KSP Residual norm 5.696176634816e+00 
166 KSP Residual norm 5.662368413063e+00 
167 KSP Residual norm 5.633670555778e+00 
168 KSP Residual norm 5.606703346582e+00 
169 KSP Residual norm 5.562363039072e+00 
170 KSP Residual norm 5.519887153291e+00 
171 KSP Residual norm 5.497221286684e+00 
172 KSP Residual norm 5.474307358870e+00 
173 KSP Residual norm 5.442147697828e+00 
174 KSP Residual norm 5.414935999541e+00 
175 KSP Residual norm 5.389603003780e+00 
176 KSP Residual norm 5.359257652197e+00 
177 KSP Residual norm 5.332741202265e+00 
178 KSP Residual norm 5.306101555765e+00 
179 KSP Residual norm 5.279035487319e+00 
180 KSP Residual norm 5.246320139231e+00 
181 KSP Residual norm 5.220859567258e+00 
182 KSP Residual norm 5.197090958094e+00 
183 KSP Residual norm 5.170997760256e+00 
184 KSP Residual norm 5.147418048527e+00 
185 KSP Residual norm 5.122829552154e+00 
186 KSP Residual norm 5.104238656476e+00 
187 KSP Residual norm 5.088648018557e+00 
188 KSP Residual norm 5.074672467950e+00 
189 KSP Residual norm 5.064078259071e+00 
190 KSP Residual norm 5.051134957237e+00 
191 KSP Residual norm 5.026408331782e+00 
192 KSP Residual norm 5.001387802286e+00 
193 KSP Residual norm 4.987260410521e+00 
194 KSP Residual norm 4.972466261626e+00 
195 KSP Residual norm 4.952317170538e+00 
196 KSP Residual norm 4.933161328568e+00 
197 KSP Residual norm 4.908199410388e+00 
198 KSP Residual norm 4.881668844609e+00 
199 KSP Residual norm 4.859451470171e+00 
200 KSP Residual norm 4.837492073040e+00 
201 KSP Residual norm 4.802548131604e+00 
202 KSP Residual norm 4.763116268308e+00 
203 KSP Residual norm 4.729593735448e+00 
204 KSP Residual norm 4.702323341804e+00 
205 KSP Residual norm 4.682236020272e+00 
206 KSP Residual norm 4.667127557911e+00 
207 KSP Residual norm 4.653891837228e+00 
208 KSP Residual norm 4.637050314339e+00 
209 KSP Residual norm 4.611100460951e+00 
210 KSP Residual norm 4.586613371416e+00 
211 KSP Residual norm 4.567567622865e+00 
212 KSP Residual norm 4.550257380318e+00 
213 KSP Residual norm 4.535396817447e+00 
214 KSP Residual norm 4.520708652979e+00 
215 KSP Residual norm 4.506936326780e+00 
216 KSP Residual norm 4.491583422005e+00 
217 KSP Residual norm 4.476983996457e+00 
218 KSP Residual norm 4.462243678662e+00 
219 KSP Residual norm 4.442670098889e+00 
220 KSP Residual norm 4.421820770732e+00 
221 KSP Residual norm 4.409589159754e+00 
222 KSP Residual norm 4.399408053243e+00 
223 KSP Residual norm 4.385784857156e+00 
224 KSP Residual norm 4.367992808661e+00 
225 KSP Residual norm 4.354092605304e+00 
226 KSP Residual norm 4.335595592299e+00 
227 KSP Residual norm 4.320547006262e+00 
228 KSP Residual norm 4.307546157350e+00 
229 KSP Residual norm 4.284966186583e+00 
230 KSP Residual norm 4.265524634571e+00 
231 KSP Residual norm 4.254219021495e+00 
232 KSP Residual norm 4.241769749282e+00 
233 KSP Residual norm 4.224541635452e+00 
234 KSP Residual norm 4.212076794694e+00 
235 KSP Residual norm 4.199364068099e+00 
236 KSP Residual norm 4.182420122043e+00 
237 KSP Residual norm 4.167392512806e+00 
238 KSP Residual norm 4.149258317056e+00 
239 KSP Residual norm 4.134252028716e+00 
240 KSP Residual norm 4.114890232435e+00 
241 KSP Residual norm 4.099629659110e+00 
242 KSP Residual norm 4.086309475062e+00 
243 KSP Residual norm 4.068164064749e+00 
244 KSP Residual norm 4.053303344208e+00 
245 KSP Residual norm 4.038519239985e+00 
246 KSP Residual norm 4.029588912426e+00 
247 KSP Residual norm 4.022566445018e+00 
248 KSP Residual norm 4.015229952986e+00 
249 KSP Residual norm 4.008435017122e+00 
250 KSP Residual norm 4.001202566444e+00 
251 KSP Residual norm 3.989624635887e+00 
252 KSP Residual norm 3.976891273831e+00 
253 KSP Residual norm 3.968458076782e+00 
254 KSP Residual norm 3.959786812362e+00 
255 KSP Residual norm 3.948659177770e+00 
256 KSP Residual norm 3.937139609303e+00 
257 KSP Residual norm 3.923237527944e+00 
258 KSP Residual norm 3.909421728740e+00 
259 KSP Residual norm 3.895462326455e+00 
260 KSP Residual norm 3.879782111884e+00 
261 KSP Residual norm 3.857084779396e+00 
262 KSP Residual norm 3.834156877016e+00 
263 KSP Residual norm 3.812373918671e+00 
264 KSP Residual norm 3.791430954743e+00 
265 KSP Residual norm 3.776323581593e+00 
266 KSP Residual norm 3.764956376798e+00 
267 KSP Residual norm 3.755674485260e+00 
268 KSP Residual norm 3.746805121124e+00 
269 KSP Residual norm 3.732410498860e+00 
270 KSP Residual norm 3.716334071608e+00 
271 KSP Residual norm 3.704603927799e+00 
272 KSP Residual norm 3.693963613260e+00 
273 KSP Residual norm 3.685913887226e+00 
274 KSP Residual norm 3.675983155710e+00 
275 KSP Residual norm 3.666747603813e+00 
276 KSP Residual norm 3.655282991832e+00 
277 KSP Residual norm 3.643299078303e+00 
278 KSP Residual norm 3.632794313968e+00 
279 KSP Residual norm 3.620066540668e+00 
280 KSP Residual norm 3.604988051275e+00 
281 KSP Residual norm 3.595573325810e+00 
282 KSP Residual norm 3.589074695633e+00 
283 KSP Residual norm 3.580612407923e+00 
284 KSP Residual norm 3.569280200629e+00 
285 KSP Residual norm 3.559112180224e+00 
286 KSP Residual norm 3.547885161717e+00 
287 KSP Residual norm 3.538750379302e+00 
288 KSP Residual norm 3.529913444267e+00 
289 KSP Residual norm 3.517916190869e+00 
290 KSP Residual norm 3.506831629967e+00 
291 KSP Residual norm 3.499628471422e+00 
292 KSP Residual norm 3.491947274112e+00 
293 KSP Residual norm 3.483867470911e+00 
294 KSP Residual norm 3.477023496516e+00 
295 KSP Residual norm 3.469299394561e+00 
296 KSP Residual norm 3.458304509888e+00 
297 KSP Residual norm 3.448524871219e+00 
298 KSP Residual norm 3.435541423956e+00 
299 KSP Residual norm 3.426325791695e+00 
300 KSP Residual norm 3.415704180855e+00 
301 KSP Residual norm 3.406956378580e+00 
302 KSP Residual norm 3.398157213870e+00 
303 KSP Residual norm 3.385176233993e+00 
304 KSP Residual norm 3.376231299875e+00 
305 KSP Residual norm 3.367346162844e+00 
306 KSP Residual norm 3.361408504911e+00 
307 KSP Residual norm 3.357002963659e+00 
308 KSP Residual norm 3.353052451083e+00 
309 KSP Residual norm 3.348588875229e+00 
310 KSP Residual norm 3.343814464155e+00 
311 KSP Residual norm 3.336636510074e+00 
312 KSP Residual norm 3.329340373091e+00 
313 KSP Residual norm 3.323882913378e+00 
314 KSP Residual norm 3.318333421839e+00 
315 KSP Residual norm 3.311670969810e+00 
316 KSP Residual norm 3.303866081683e+00 
317 KSP Residual norm 3.293903325053e+00 
318 KSP Residual norm 3.284415370212e+00 
319 KSP Residual norm 3.276445522193e+00 
320 KSP Residual norm 3.265088857299e+00 
321 KSP Residual norm 3.246017818685e+00 
322 KSP Residual norm 3.228557958164e+00 
323 KSP Residual norm 3.213326763456e+00 
324 KSP Residual norm 3.197662546060e+00 
325 KSP Residual norm 3.186347625444e+00 
326 KSP Residual norm 3.178750308424e+00 
327 KSP Residual norm 3.172216453971e+00 
328 KSP Residual norm 3.165281508617e+00 
329 KSP Residual norm 3.154641178174e+00 
330 KSP Residual norm 3.144982566200e+00 
331 KSP Residual norm 3.137119791603e+00 
332 KSP Residual norm 3.129246608571e+00 
333 KSP Residual norm 3.122992707017e+00 
334 KSP Residual norm 3.115608713024e+00 
335 KSP Residual norm 3.109089926533e+00 
336 KSP Residual norm 3.100374442263e+00 
337 KSP Residual norm 3.090489356646e+00 
338 KSP Residual norm 3.082110045784e+00 
339 KSP Residual norm 3.071728903912e+00 
340 KSP Residual norm 3.059906974716e+00 
341 KSP Residual norm 3.052784156457e+00 
342 KSP Residual norm 3.048278027309e+00 
343 KSP Residual norm 3.041593216215e+00 
344 KSP Residual norm 3.032786260029e+00 
345 KSP Residual norm 3.025747098684e+00 
346 KSP Residual norm 3.018450997932e+00 
347 KSP Residual norm 3.012318935933e+00 
348 KSP Residual norm 3.006302956491e+00 
349 KSP Residual norm 2.998708362493e+00 
350 KSP Residual norm 2.991714324831e+00 
351 KSP Residual norm 2.986781453606e+00 
352 KSP Residual norm 2.982059693684e+00 
353 KSP Residual norm 2.976779119791e+00 
354 KSP Residual norm 2.972556574587e+00 
355 KSP Residual norm 2.967460106369e+00 
356 KSP Residual norm 2.959862792046e+00 
357 KSP Residual norm 2.952554334154e+00 
358 KSP Residual norm 2.944615992893e+00 
359 KSP Residual norm 2.937923253544e+00 
360 KSP Residual norm 2.930246358250e+00 
361 KSP Residual norm 2.923790136805e+00 
362 KSP Residual norm 2.917473890658e+00 
363 KSP Residual norm 2.909686398104e+00 
364 KSP Residual norm 2.902549932091e+00 
365 KSP Residual norm 2.895930952786e+00 
366 KSP Residual norm 2.891807691300e+00 
367 KSP Residual norm 2.889196349786e+00 
368 KSP Residual norm 2.886384509164e+00 
369 KSP Residual norm 2.883553930330e+00 
370 KSP Residual norm 2.880239285194e+00 
371 KSP Residual norm 2.875691771212e+00 
372 KSP Residual norm 2.870633791727e+00 
373 KSP Residual norm 2.866884219455e+00 
374 KSP Residual norm 2.862822558442e+00 
375 KSP Residual norm 2.858089817529e+00 
376 KSP Residual norm 2.853032859178e+00 
377 KSP Residual norm 2.845858835949e+00 
378 KSP Residual norm 2.838521568686e+00 
379 KSP Residual norm 2.832934327027e+00 
380 KSP Residual norm 2.824901796916e+00 
381 KSP Residual norm 2.811610040249e+00 
382 KSP Residual norm 2.796171157330e+00 
383 KSP Residual norm 2.783463007661e+00 
384 KSP Residual norm 2.770689943351e+00 
385 KSP Residual norm 2.761879288812e+00 
386 KSP Residual norm 2.756281179635e+00 
387 KSP Residual norm 2.751514098498e+00 
388 KSP Residual norm 2.745807044433e+00 
389 KSP Residual norm 2.737994732321e+00 
390 KSP Residual norm 2.730991568069e+00 
391 KSP Residual norm 2.725330722286e+00 
392 KSP Residual norm 2.719405115155e+00 
393 KSP Residual norm 2.714264846840e+00 
394 KSP Residual norm 2.708919391119e+00 
395 KSP Residual norm 2.703866634930e+00 
396 KSP Residual norm 2.696704213343e+00 
397 KSP Residual norm 2.688371849182e+00 
398 KSP Residual norm 2.680824401227e+00 
399 KSP Residual norm 2.671782610951e+00 
400 KSP Residual norm 2.662524934212e+00 
Linear solve converged due to CONVERGED_ITS iterations 400
KSP Object: 2 MPI processes
  type: gcr
    restart = 30 
    restarts performed = 14 
  maximum iterations=400, initial guess is zero
  tolerances:  relative=1e-06, absolute=1e-50, divergence=10000.
  right preconditioning
  using UNPRECONDITIONED norm type for convergence test
PC Object: 2 MPI processes
  type: jacobi
    type DIAGONAL
  linear system matrix = precond matrix:
  Mat Object: 2 MPI processes
    type: mpiaijcusparse
    rows=12634590, cols=12634590, bs=30
    total: nonzeros=1886343300, allocated nonzeros=1886343300
    total number of mallocs used during MatSetValues calls=0
      not using I-node (on process 0) routines
Solve: 65.892 s
Residual norm: 2.66252
Iterations: 400
****************************************************************************************************************************************************************
***                                WIDEN YOUR WINDOW TO 160 CHARACTERS.  Use 'enscript -r -fCourier9' to print this document                                 ***
****************************************************************************************************************************************************************

------------------------------------------------------------------ PETSc Performance Summary: ------------------------------------------------------------------

      ##########################################################
      #                                                        #
      #                       WARNING!!!                       #
      #                                                        #
      #   This code was compiled with a debugging option.      #
      #   To get timing results run ./configure                #
      #   using --with-debugging=no, the performance will      #
      #   be generally two or three times faster.              #
      #                                                        #
      ##########################################################

--petsc on a  named heisenbug with 2 processors, by ulrich Wed Sep 13 21:43:59 2023
Using Petsc Release Version 3.18.5, Feb 27, 2023 

                         Max       Max/Min     Avg       Total
Time (sec):           3.338e+02     1.000   3.338e+02
Objects:              8.200e+01     1.000   8.200e+01
Flops:                1.028e+12     1.051   1.004e+12  2.007e+12
Flops/sec:            3.081e+09     1.051   3.007e+09  6.013e+09
Memory (bytes):       1.608e+10     1.050   1.570e+10  3.139e+10
MPI Msg Count:        4.030e+02     1.000   4.030e+02  8.060e+02
MPI Msg Len (bytes):  1.868e+08     1.000   4.635e+05  3.736e+08
MPI Reductions:       3.103e+04     1.000

Flop counting convention: 1 flop = 1 real number operation of type (multiply/divide/add/subtract)
                            e.g., VecAXPY() for real vectors of length N --> 2N flops
                            and VecAXPY() for complex vectors of length N --> 8N flops

Summary of Stages:   ----- Time ------  ----- Flop ------  --- Messages ---  -- Message Lengths --  -- Reductions --
                        Avg     %Total     Avg     %Total    Count   %Total     Avg         %Total    Count   %Total
 0:      Main Stage: 2.6791e+02  80.3%  1.5075e+09   0.1%  0.000e+00   0.0%  0.000e+00        0.0%  2.100e+02   0.7%
 2:           solve: 6.5892e+01  19.7%  2.0056e+12  99.9%  8.060e+02 100.0%  4.635e+05      100.0%  3.080e+04  99.3%

------------------------------------------------------------------------------------------------------------------------
See the 'Profiling' chapter of the users' manual for details on interpreting output.
Phase summary info:
   Count: number of times phase was executed
   Time and Flop: Max - maximum over all processors
                  Ratio - ratio of maximum to minimum over all processors
   Mess: number of messages sent
   AvgLen: average message length (bytes)
   Reduct: number of global reductions
   Global: entire computation
   Stage: stages of a computation. Set stages with PetscLogStagePush() and PetscLogStagePop().
      %T - percent time in this phase         %F - percent flop in this phase
      %M - percent messages in this phase     %L - percent message lengths in this phase
      %R - percent reductions in this phase
   Total Mflop/s: 10e-6 * (sum of flop over all processors)/(max time over all processors)
   GPU Mflop/s: 10e-6 * (sum of flop on GPU over all processors)/(max GPU time over all processors)
   CpuToGpu Count: total number of CPU to GPU copies per processor
   CpuToGpu Size (Mbytes): 10e-6 * (total size of CPU to GPU copies per processor)
   GpuToCpu Count: total number of GPU to CPU copies per processor
   GpuToCpu Size (Mbytes): 10e-6 * (total size of GPU to CPU copies per processor)
   GPU %F: percent flops on GPU in this event
------------------------------------------------------------------------------------------------------------------------

      ##########################################################
      #                                                        #
      #                       WARNING!!!                       #
      #                                                        #
      #   This code was compiled with a debugging option.      #
      #   To get timing results run ./configure                #
      #   using --with-debugging=no, the performance will      #
      #   be generally two or three times faster.              #
      #                                                        #
      ##########################################################

Event                Count      Time (sec)     Flop                              --- Global ---  --- Stage ----  Total   GPU    - CpuToGpu -   - GpuToCpu - GPU
                   Max Ratio  Max     Ratio   Max  Ratio  Mess   AvgLen  Reduct  %T %F %M %L %R  %T %F %M %L %R Mflop/s Mflop/s Count   Size   Count   Size  %F
---------------------------------------------------------------------------------------------------------------------------------------------------------------

--- Event Stage 0: Main Stage

BuildTwoSided          1 1.0   nan nan 0.00e+00 0.0 0.0e+00 0.0e+00 2.0e+00  0  0  0  0  0   0  0  0  0  1  -nan    -nan      0 0.00e+00    0 0.00e+00  0
BuildTwoSidedF         1 1.0   nan nan 0.00e+00 0.0 0.0e+00 0.0e+00 2.0e+00  0  0  0  0  0   0  0  0  0  1  -nan    -nan      0 0.00e+00    0 0.00e+00  0
PCSetUp                1 1.0   nan nan 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0  -nan    -nan      0 0.00e+00    0 0.00e+00  0
MatAssemblyBegin       1 1.0   nan nan 0.00e+00 0.0 0.0e+00 0.0e+00 4.0e+00  1  0  0  0  0   1  0  0  0  2  -nan    -nan      0 0.00e+00    0 0.00e+00  0
MatAssemblyEnd         1 1.0   nan nan 0.00e+00 0.0 0.0e+00 0.0e+00 1.9e+01  1  0  0  0  0   1  0  0  0  9  -nan    -nan      0 0.00e+00    0 0.00e+00  0
VecSet                63 1.0   nan nan 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0  -nan    -nan      0 0.00e+00    0 0.00e+00  0
VecCUDACopyFrom        1 1.0   nan nan 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0  -nan    -nan      0 0.00e+00    1 5.05e+01  0
SFSetGraph             1 1.0   nan nan 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0  -nan    -nan      0 0.00e+00    0 0.00e+00  0
KSPSetUp               1 1.0   nan nan 0.00e+00 0.0 0.0e+00 0.0e+00 1.3e+02  1  0  0  0  0   1  0  0  0 61  -nan    -nan      0 0.00e+00    0 0.00e+00  0

--- Event Stage 1: Unknown

--- Event Stage 2: solve

BuildTwoSided          1 1.0   nan nan 0.00e+00 0.0 2.0e+00 8.0e+00 2.0e+00  0  0  0  0  0   0  0  0  0  0  -nan    -nan      0 0.00e+00    0 0.00e+00  0
PCApply              400 1.0   nan nan 2.59e+09 1.1 0.0e+00 0.0e+00 9.0e+00  1  0  0  0  0   3  0  0  0  0  -nan    -nan      2 1.01e+02    1 5.05e+01  100
MatMult              401 1.0   nan nan 7.73e+11 1.1 8.1e+02 4.6e+05 2.0e+00 17 75 100 100  0  87 75 100 100  0  -nan    -nan      2 1.13e+04    0 0.00e+00  100
MatView                1 1.0   nan nan 0.00e+00 0.0 0.0e+00 0.0e+00 2.0e+00  0  0  0  0  0   0  0  0  0  0  -nan    -nan      0 0.00e+00    0 0.00e+00  0
MatCUSPARSCopyTo       2 1.0   nan nan 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 14  0  0  0  0  71  0  0  0  0  -nan    -nan      2 1.13e+04    0 0.00e+00  0
cuBLAS Init            1 1.0   nan nan 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  1  0  0  0  0   5  0  0  0  0  -nan    -nan      0 0.00e+00    0 0.00e+00  0
DCtxCreate             1 1.0   nan nan 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0  -nan    -nan      0 0.00e+00    0 0.00e+00  0
DCtxSetUp              1 1.0   nan nan 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0  -nan    -nan      0 0.00e+00    0 0.00e+00  0
DCtxSetDevice          1 1.0   nan nan 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0  -nan    -nan      0 0.00e+00    0 0.00e+00  0
VecDotNorm2          400 1.0   nan nan 1.04e+10 1.1 0.0e+00 0.0e+00 8.0e+02  1  1  0  0  3   3  1  0  0  3  -nan    -nan      0 0.00e+00    0 0.00e+00  100
VecMDot              386 1.0   nan nan 7.38e+10 1.1 0.0e+00 0.0e+00 7.7e+02  0  7  0  0  2   1  7  0  0  3  -nan    -nan      0 0.00e+00    0 0.00e+00  100
VecNorm              401 1.0   nan nan 5.19e+09 1.1 0.0e+00 0.0e+00 8.0e+02  0  1  0  0  3   0  1  0  0  3  -nan    -nan      0 0.00e+00    0 0.00e+00  100
VecScale             800 1.0   nan nan 5.18e+09 1.1 0.0e+00 0.0e+00 0.0e+00  0  1  0  0  0   0  1  0  0  0  -nan    -nan      0 0.00e+00    0 0.00e+00  100
VecSet                 2 1.0   nan nan 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0  -nan    -nan      0 0.00e+00    0 0.00e+00  0
VecAXPY              800 1.0   nan nan 1.04e+10 1.1 0.0e+00 0.0e+00 0.0e+00  0  1  0  0  0   0  1  0  0  0  -nan    -nan      0 0.00e+00    0 0.00e+00  100
VecAYPX                1 1.0   nan nan 1.30e+07 1.1 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0  -nan    -nan      1 5.05e+01    0 0.00e+00  100
VecMAXPY             772 1.0   nan nan 1.48e+11 1.1 0.0e+00 0.0e+00 0.0e+00  0 14  0  0  0   0 14  0  0  0  -nan    -nan      0 0.00e+00    0 0.00e+00  100
VecPointwiseMult     400 1.0   nan nan 2.59e+09 1.1 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0  -nan    -nan      1 5.05e+01    0 0.00e+00  100
VecScatterBegin      401 1.0   nan nan 0.00e+00 0.0 8.1e+02 4.6e+05 2.0e+00  0  0 100 100  0   0  0 100 100  0  -nan    -nan      0 0.00e+00    0 0.00e+00  0
VecScatterEnd        401 1.0   nan nan 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  3  0  0  0  0  16  0  0  0  0  -nan    -nan      0 0.00e+00    0 0.00e+00  0
VecCUDACopyTo          3 1.0   nan nan 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0  -nan    -nan      3 1.52e+02    0 0.00e+00  0
VecCUDACopyFrom        1 1.0   nan nan 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0  -nan    -nan      0 0.00e+00    1 5.05e+01  0
SFSetUp                1 1.0   nan nan 0.00e+00 0.0 4.0e+00 2.3e+05 2.0e+00  0  0  0  0  0   0  0  0  0  0  -nan    -nan      0 0.00e+00    0 0.00e+00  0
SFPack               401 1.0   nan nan 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0  -nan    -nan      0 0.00e+00    0 0.00e+00  0
SFUnpack             401 1.0   nan nan 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0  -nan    -nan      0 0.00e+00    0 0.00e+00  0
KSPSolve               1 1.0 6.5892e+01 1.0 1.03e+12 1.1 8.1e+02 4.6e+05 3.1e+04 20 100 100 100 99 100 100 100 100 100 30438    -nan      5 1.15e+04    1 5.05e+01  100
---------------------------------------------------------------------------------------------------------------------------------------------------------------

Object Type          Creations   Destructions. Reports information only for process 0.

--- Event Stage 0: Main Stage

           Container     0              3
      Preconditioner     1              1
              Matrix     3              3
           Index Set     2              2
   IS L to G Mapping     2              2
              Vector    65             66
   Star Forest Graph     1              1
       Krylov Solver     1              1
              Viewer     2              1

--- Event Stage 1: Unknown

--- Event Stage 2: solve

           Container     3              0
  PetscDeviceContext     1              0
              Vector     1              0
========================================================================================================================
Average time to get PetscTime(): 3.7e-08
Average time for MPI_Barrier(): 1.09678e-05
Average time for zero size MPI_Send(): 1.772e-05
#PETSc Option Table entries:
-ksp_converged_reason
-ksp_max_it 400
-ksp_monitor
-ksp_rtol 1.0e-6
-ksp_type gcr
-ksp_view
-log_view
-mat_type aijcusparse
-options_left
-pc_type jacobi
-vec_type cuda
#End of PETSc Option Table entries
Compiled without FORTRAN kernels
Compiled with 64 bit PetscInt
Compiled with full precision matrices (default)
sizeof(short) 2 sizeof(int) 4 sizeof(long) 8 sizeof(void*) 8 sizeof(PetscScalar) 8 sizeof(PetscInt) 8
Configure options: --prefix=/import/exception-dump/ulrich/myLibs/spack-packages/linux-debian11-zen2/gcc-11.2.0/petsc-3.18.5-pppgzrc7ykm26yk4dmy7nzuf2f6jopah --with-ssl=0 --download-c2html=0 --download-sowing=0 --download-hwloc=0 --with-make-exec=make --with-cc=/import/exception-dump/ulrich/myLibs/spack-packages/linux-debian11-zen2/gcc-11.2.0/openmpi-4.1.5-sf7iuo3braeberbmnnyt4emapkxz4jtw/bin/mpicc --with-cxx=/import/exception-dump/ulrich/myLibs/spack-packages/linux-debian11-zen2/gcc-11.2.0/openmpi-4.1.5-sf7iuo3braeberbmnnyt4emapkxz4jtw/bin/mpic++ --with-fc=/import/exception-dump/ulrich/myLibs/spack-packages/linux-debian11-zen2/gcc-11.2.0/openmpi-4.1.5-sf7iuo3braeberbmnnyt4emapkxz4jtw/bin/mpif90 --with-precision=double --with-scalar-type=real --with-shared-libraries=1 --with-debugging=1 --with-openmp=0 --with-64-bit-indices=1 --with-blaslapack-lib=/import/exception-dump/ulrich/myLibs/spack-packages/linux-debian11-zen2/gcc-11.2.0/openblas-0.3.23-kxy7bxom4lam3wx44st22oytplyuisby/lib/libopenblas.so --with-memalign=32 --with-x=0 --with-clanguage=C --with-cuda=1 --with-cuda-dir=/import/exception-dump/ulrich/myLibs/spack-packages/linux-debian11-zen2/gcc-11.2.0/cuda-11.8.0-uivzrqah34rk27dutzqehubyc57zi46u --with-hip=0 --with-metis=1 --with-metis-include=/import/exception-dump/ulrich/myLibs/spack-packages/linux-debian11-zen2/gcc-11.2.0/metis-5.1.0-6zrgzhlze5ujsajtktgwf73dlpdjtdxk/include --with-metis-lib=/import/exception-dump/ulrich/myLibs/spack-packages/linux-debian11-zen2/gcc-11.2.0/metis-5.1.0-6zrgzhlze5ujsajtktgwf73dlpdjtdxk/lib/libmetis.so --with-hypre=1 --with-hypre-include=/import/exception-dump/ulrich/myLibs/spack-packages/linux-debian11-zen2/gcc-11.2.0/hypre-2.28.0-73iiib7speczh6bpszpkvbixgpkwyt7z/include --with-hypre-lib=/import/exception-dump/ulrich/myLibs/spack-packages/linux-debian11-zen2/gcc-11.2.0/hypre-2.28.0-73iiib7speczh6bpszpkvbixgpkwyt7z/lib/libHYPRE.so --with-parmetis=1 --with-parmetis-include=/import/exception-dump/ulrich/myLibs/spack-packages/linux-debian11-zen2/gcc-11.2.0/parmetis-4.0.3-gy6mgt3a4qt2ppnsorma4tabjw3f55fw/include --with-parmetis-lib=/import/exception-dump/ulrich/myLibs/spack-packages/linux-debian11-zen2/gcc-11.2.0/parmetis-4.0.3-gy6mgt3a4qt2ppnsorma4tabjw3f55fw/lib/libparmetis.so --with-kokkos=0 --with-kokkos-kernels=0 --with-superlu_dist=1 --with-superlu_dist-include=/import/exception-dump/ulrich/myLibs/spack-packages/linux-debian11-zen2/gcc-11.2.0/superlu-dist-8.1.2-7mrjdpdpu5l2g7x546r4kcssv2minjsa/include --with-superlu_dist-lib=/import/exception-dump/ulrich/myLibs/spack-packages/linux-debian11-zen2/gcc-11.2.0/superlu-dist-8.1.2-7mrjdpdpu5l2g7x546r4kcssv2minjsa/lib/libsuperlu_dist.so --with-ptscotch=0 --with-suitesparse=0 --with-hdf5=1 --with-hdf5-include=/import/exception-dump/ulrich/myLibs/spack-packages/linux-debian11-zen2/gcc-11.2.0/hdf5-1.14.2-dstkldnhuuoqsxniwh7piimjbyva4b2c/include --with-hdf5-lib=/import/exception-dump/ulrich/myLibs/spack-packages/linux-debian11-zen2/gcc-11.2.0/hdf5-1.14.2-dstkldnhuuoqsxniwh7piimjbyva4b2c/lib/libhdf5.so --with-zlib=1 --with-zlib-include=/import/exception-dump/ulrich/myLibs/spack-packages/linux-debian11-zen2/gcc-11.2.0/zlib-1.2.13-ufda23qzqsm5z2eqwsne76yoazck6fsz/include --with-zlib-lib=/import/exception-dump/ulrich/myLibs/spack-packages/linux-debian11-zen2/gcc-11.2.0/zlib-1.2.13-ufda23qzqsm5z2eqwsne76yoazck6fsz/lib/libz.so --with-mumps=1 --with-mumps-include=/import/exception-dump/ulrich/myLibs/spack-packages/linux-debian11-zen2/gcc-11.2.0/mumps-5.5.1-umzads25ggryhklmvvto4a5i6ue74fbk/include --with-mumps-lib="/import/exception-dump/ulrich/myLibs/spack-packages/linux-debian11-zen2/gcc-11.2.0/mumps-5.5.1-umzads25ggryhklmvvto4a5i6ue74fbk/lib/libsmumps.so /import/exception-dump/ulrich/myLibs/spack-packages/linux-debian11-zen2/gcc-11.2.0/mumps-5.5.1-umzads25ggryhklmvvto4a5i6ue74fbk/lib/libzmumps.so /import/exception-dump/ulrich/myLibs/spack-packages/linux-debian11-zen2/gcc-11.2.0/mumps-5.5.1-umzads25ggryhklmvvto4a5i6ue74fbk/lib/libcmumps.so /import/exception-dump/ulrich/myLibs/spack-packages/linux-debian11-zen2/gcc-11.2.0/mumps-5.5.1-umzads25ggryhklmvvto4a5i6ue74fbk/lib/libdmumps.so /import/exception-dump/ulrich/myLibs/spack-packages/linux-debian11-zen2/gcc-11.2.0/mumps-5.5.1-umzads25ggryhklmvvto4a5i6ue74fbk/lib/libmumps_common.so /import/exception-dump/ulrich/myLibs/spack-packages/linux-debian11-zen2/gcc-11.2.0/mumps-5.5.1-umzads25ggryhklmvvto4a5i6ue74fbk/lib/libpord.so" --with-trilinos=0 --with-fftw=0 --with-valgrind=0 --with-gmp=0 --with-libpng=0 --with-giflib=0 --with-mpfr=0 --with-netcdf=0 --with-pnetcdf=0 --with-moab=0 --with-random123=0 --with-exodusii=0 --with-cgns=0 --with-memkind=0 --with-p4est=0 --with-saws=0 --with-yaml=0 --with-hwloc=0 --with-libjpeg=0 --with-scalapack=1 --with-scalapack-lib=/import/exception-dump/ulrich/myLibs/spack-packages/linux-debian11-zen2/gcc-11.2.0/netlib-scalapack-2.2.0-vs6d5fhfa4i77gfo5iurpifbgxwfrmbp/lib/libscalapack.so --with-strumpack=0 --with-mmg=0 --with-parmmg=0 --with-tetgen=0
-----------------------------------------
Libraries compiled on 2023-08-31 16:19:23 on heisenbug 
Machine characteristics: Linux-6.1.0-0.deb11.7-amd64-x86_64-with-glibc2.31
Using PETSc directory: /import/exception-dump/ulrich/myLibs/spack-packages/linux-debian11-zen2/gcc-11.2.0/petsc-3.18.5-pppgzrc7ykm26yk4dmy7nzuf2f6jopah
Using PETSc arch: 
-----------------------------------------

Using C compiler: /import/exception-dump/ulrich/myLibs/spack-packages/linux-debian11-zen2/gcc-11.2.0/openmpi-4.1.5-sf7iuo3braeberbmnnyt4emapkxz4jtw/bin/mpicc  -fPIC -Wall -Wwrite-strings -Wno-unknown-pragmas -Wno-lto-type-mismatch -Wno-stringop-overflow -fstack-protector -fvisibility=hidden -g3 -O0   
Using Fortran compiler: /import/exception-dump/ulrich/myLibs/spack-packages/linux-debian11-zen2/gcc-11.2.0/openmpi-4.1.5-sf7iuo3braeberbmnnyt4emapkxz4jtw/bin/mpif90  -fPIC -Wall -ffree-line-length-none -ffree-line-length-0 -Wno-lto-type-mismatch -Wno-unused-dummy-argument -g -O0     
-----------------------------------------

Using include paths: -I/import/exception-dump/ulrich/myLibs/spack-packages/linux-debian11-zen2/gcc-11.2.0/petsc-3.18.5-pppgzrc7ykm26yk4dmy7nzuf2f6jopah/include -I/import/exception-dump/ulrich/myLibs/spack-packages/linux-debian11-zen2/gcc-11.2.0/hypre-2.28.0-73iiib7speczh6bpszpkvbixgpkwyt7z/include -I/import/exception-dump/ulrich/myLibs/spack-packages/linux-debian11-zen2/gcc-11.2.0/mumps-5.5.1-umzads25ggryhklmvvto4a5i6ue74fbk/include -I/import/exception-dump/ulrich/myLibs/spack-packages/linux-debian11-zen2/gcc-11.2.0/superlu-dist-8.1.2-7mrjdpdpu5l2g7x546r4kcssv2minjsa/include -I/import/exception-dump/ulrich/myLibs/spack-packages/linux-debian11-zen2/gcc-11.2.0/hdf5-1.14.2-dstkldnhuuoqsxniwh7piimjbyva4b2c/include -I/import/exception-dump/ulrich/myLibs/spack-packages/linux-debian11-zen2/gcc-11.2.0/parmetis-4.0.3-gy6mgt3a4qt2ppnsorma4tabjw3f55fw/include -I/import/exception-dump/ulrich/myLibs/spack-packages/linux-debian11-zen2/gcc-11.2.0/metis-5.1.0-6zrgzhlze5ujsajtktgwf73dlpdjtdxk/include -I/import/exception-dump/ulrich/myLibs/spack-packages/linux-debian11-zen2/gcc-11.2.0/zlib-1.2.13-ufda23qzqsm5z2eqwsne76yoazck6fsz/include -I/import/exception-dump/ulrich/myLibs/spack-packages/linux-debian11-zen2/gcc-11.2.0/cuda-11.8.0-uivzrqah34rk27dutzqehubyc57zi46u/include
-----------------------------------------

Using C linker: /import/exception-dump/ulrich/myLibs/spack-packages/linux-debian11-zen2/gcc-11.2.0/openmpi-4.1.5-sf7iuo3braeberbmnnyt4emapkxz4jtw/bin/mpicc
Using Fortran linker: /import/exception-dump/ulrich/myLibs/spack-packages/linux-debian11-zen2/gcc-11.2.0/openmpi-4.1.5-sf7iuo3braeberbmnnyt4emapkxz4jtw/bin/mpif90
Using libraries: -Wl,-rpath,/import/exception-dump/ulrich/myLibs/spack-packages/linux-debian11-zen2/gcc-11.2.0/petsc-3.18.5-pppgzrc7ykm26yk4dmy7nzuf2f6jopah/lib -L/import/exception-dump/ulrich/myLibs/spack-packages/linux-debian11-zen2/gcc-11.2.0/petsc-3.18.5-pppgzrc7ykm26yk4dmy7nzuf2f6jopah/lib -lpetsc -Wl,-rpath,/import/exception-dump/ulrich/myLibs/spack-packages/linux-debian11-zen2/gcc-11.2.0/hypre-2.28.0-73iiib7speczh6bpszpkvbixgpkwyt7z/lib -L/import/exception-dump/ulrich/myLibs/spack-packages/linux-debian11-zen2/gcc-11.2.0/hypre-2.28.0-73iiib7speczh6bpszpkvbixgpkwyt7z/lib -Wl,-rpath,/import/exception-dump/ulrich/myLibs/spack-packages/linux-debian11-zen2/gcc-11.2.0/mumps-5.5.1-umzads25ggryhklmvvto4a5i6ue74fbk/lib -L/import/exception-dump/ulrich/myLibs/spack-packages/linux-debian11-zen2/gcc-11.2.0/mumps-5.5.1-umzads25ggryhklmvvto4a5i6ue74fbk/lib -Wl,-rpath,/import/exception-dump/ulrich/myLibs/spack-packages/linux-debian11-zen2/gcc-11.2.0/netlib-scalapack-2.2.0-vs6d5fhfa4i77gfo5iurpifbgxwfrmbp/lib -L/import/exception-dump/ulrich/myLibs/spack-packages/linux-debian11-zen2/gcc-11.2.0/netlib-scalapack-2.2.0-vs6d5fhfa4i77gfo5iurpifbgxwfrmbp/lib -Wl,-rpath,/import/exception-dump/ulrich/myLibs/spack-packages/linux-debian11-zen2/gcc-11.2.0/superlu-dist-8.1.2-7mrjdpdpu5l2g7x546r4kcssv2minjsa/lib -L/import/exception-dump/ulrich/myLibs/spack-packages/linux-debian11-zen2/gcc-11.2.0/superlu-dist-8.1.2-7mrjdpdpu5l2g7x546r4kcssv2minjsa/lib -Wl,-rpath,/import/exception-dump/ulrich/myLibs/spack-packages/linux-debian11-zen2/gcc-11.2.0/openblas-0.3.23-kxy7bxom4lam3wx44st22oytplyuisby/lib -L/import/exception-dump/ulrich/myLibs/spack-packages/linux-debian11-zen2/gcc-11.2.0/openblas-0.3.23-kxy7bxom4lam3wx44st22oytplyuisby/lib -Wl,-rpath,/import/exception-dump/ulrich/myLibs/spack-packages/linux-debian11-zen2/gcc-11.2.0/hdf5-1.14.2-dstkldnhuuoqsxniwh7piimjbyva4b2c/lib -L/import/exception-dump/ulrich/myLibs/spack-packages/linux-debian11-zen2/gcc-11.2.0/hdf5-1.14.2-dstkldnhuuoqsxniwh7piimjbyva4b2c/lib -Wl,-rpath,/import/exception-dump/ulrich/myLibs/spack-packages/linux-debian11-zen2/gcc-11.2.0/parmetis-4.0.3-gy6mgt3a4qt2ppnsorma4tabjw3f55fw/lib -L/import/exception-dump/ulrich/myLibs/spack-packages/linux-debian11-zen2/gcc-11.2.0/parmetis-4.0.3-gy6mgt3a4qt2ppnsorma4tabjw3f55fw/lib -Wl,-rpath,/import/exception-dump/ulrich/myLibs/spack-packages/linux-debian11-zen2/gcc-11.2.0/metis-5.1.0-6zrgzhlze5ujsajtktgwf73dlpdjtdxk/lib -L/import/exception-dump/ulrich/myLibs/spack-packages/linux-debian11-zen2/gcc-11.2.0/metis-5.1.0-6zrgzhlze5ujsajtktgwf73dlpdjtdxk/lib -Wl,-rpath,/import/exception-dump/ulrich/myLibs/spack-packages/linux-debian11-zen2/gcc-11.2.0/zlib-1.2.13-ufda23qzqsm5z2eqwsne76yoazck6fsz/lib -L/import/exception-dump/ulrich/myLibs/spack-packages/linux-debian11-zen2/gcc-11.2.0/zlib-1.2.13-ufda23qzqsm5z2eqwsne76yoazck6fsz/lib -Wl,-rpath,/import/exception-dump/ulrich/myLibs/spack-packages/linux-debian11-zen2/gcc-11.2.0/cuda-11.8.0-uivzrqah34rk27dutzqehubyc57zi46u/lib64 -L/import/exception-dump/ulrich/myLibs/spack-packages/linux-debian11-zen2/gcc-11.2.0/cuda-11.8.0-uivzrqah34rk27dutzqehubyc57zi46u/lib64 -L/import/exception-dump/ulrich/myLibs/spack-packages/linux-debian11-zen2/gcc-11.2.0/cuda-11.8.0-uivzrqah34rk27dutzqehubyc57zi46u/lib64/stubs -Wl,-rpath,/import/exception-dump/ulrich/myLibs/spack-packages/linux-debian11-zen2/gcc-11.2.0/openmpi-4.1.5-sf7iuo3braeberbmnnyt4emapkxz4jtw/lib -L/import/exception-dump/ulrich/myLibs/spack-packages/linux-debian11-zen2/gcc-11.2.0/openmpi-4.1.5-sf7iuo3braeberbmnnyt4emapkxz4jtw/lib -Wl,-rpath,/import/exception-dump/ulrich/myLibs/spack-packages/linux-debian11-zen2/gcc-11.2.0/hwloc-2.9.1-sm5zkzc5cbvuw6us4oahy5bg6ygp5goi/lib -L/import/exception-dump/ulrich/myLibs/spack-packages/linux-debian11-zen2/gcc-11.2.0/hwloc-2.9.1-sm5zkzc5cbvuw6us4oahy5bg6ygp5goi/lib -Wl,-rpath,/import/exception-dump/ulrich/myLibs/spack-packages/linux-debian11-zen2/gcc-11.2.0/libevent-2.1.12-7xvtpwwyxrysr7sq5atthy7hekevfifs/lib -L/import/exception-dump/ulrich/myLibs/spack-packages/linux-debian11-zen2/gcc-11.2.0/libevent-2.1.12-7xvtpwwyxrysr7sq5atthy7hekevfifs/lib -Wl,-rpath,/import/exception-dump/ulrich/myLibs/spack-packages/linux-debian11-zen2/gcc-11.2.0/pmix-4.2.4-l3hzxh27fe2svyq3tkwykk744kvdunps/lib -L/import/exception-dump/ulrich/myLibs/spack-packages/linux-debian11-zen2/gcc-11.2.0/pmix-4.2.4-l3hzxh27fe2svyq3tkwykk744kvdunps/lib -Wl,-rpath,/import/exception-dump/ulrich/myLibs/spack-packages/linux-debian11-zen2/gcc-12.2.0/gcc-11.2.0-iokm55jhztsmmlbgy3mr5rvu3y43qxyz/lib64 -Wl,-rpath,/import/exception-dump/ulrich/myLibs/spack-packages/linux-debian11-zen2/gcc-12.2.0/gcc-11.2.0-iokm55jhztsmmlbgy3mr5rvu3y43qxyz/lib/gcc/x86_64-pc-linux-gnu/11.2.0 -L/import/exception-dump/ulrich/myLibs/spack-packages/linux-debian11-zen2/gcc-12.2.0/gcc-11.2.0-iokm55jhztsmmlbgy3mr5rvu3y43qxyz/lib/gcc/x86_64-pc-linux-gnu/11.2.0 -L/import/exception-dump/ulrich/myLibs/spack-packages/linux-debian11-zen2/gcc-12.2.0/gcc-11.2.0-iokm55jhztsmmlbgy3mr5rvu3y43qxyz/lib64 -Wl,-rpath,/import/exception-dump/ulrich/myLibs/spack-packages/linux-debian11-zen2/gcc-12.2.0/gcc-11.2.0-iokm55jhztsmmlbgy3mr5rvu3y43qxyz/lib -L/import/exception-dump/ulrich/myLibs/spack-packages/linux-debian11-zen2/gcc-12.2.0/gcc-11.2.0-iokm55jhztsmmlbgy3mr5rvu3y43qxyz/lib -lHYPRE -lsmumps -lzmumps -lcmumps -ldmumps -lmumps_common -lpord -lscalapack -lsuperlu_dist -lopenblas -lhdf5 -lparmetis -lmetis -lm -lz -lcudart -lnvToolsExt -lcufft -lcublas -lcusparse -lcusolver -lcurand -lcuda -lstdc++ -ldl -lmpi_usempif08 -lmpi_usempi_ignore_tkr -lmpi_mpifh -lmpi -lgfortran -lm -lgfortran -lm -lgcc_s -lquadmath -lpthread -lquadmath -lstdc++ -ldl
-----------------------------------------

      ##########################################################
      #                                                        #
      #                       WARNING!!!                       #
      #                                                        #
      #   This code was compiled with a debugging option.      #
      #   To get timing results run ./configure                #
      #   using --with-debugging=no, the performance will      #
      #   be generally two or three times faster.              #
      #                                                        #
      ##########################################################

#PETSc Option Table entries:
-ksp_converged_reason
-ksp_max_it 400
-ksp_monitor
-ksp_rtol 1.0e-6
-ksp_type gcr
-ksp_view
-log_view
-mat_type aijcusparse
-options_left
-pc_type jacobi
-vec_type cuda
#End of PETSc Option Table entries
There are no unused options.
Thomas-Ulrich commented 1 year ago

Note that I now get with multigrid and no debug:

(base) ulrich@heisenbug:/export/dump/ulrich/section_8_2/scenario-rc$ mpiexec -n 2  static ridge.toml  --output ridgecrest  --mg_strategy twolevel --mg_coarse_level 1  --petsc -ksp_view -ksp_monitor -ksp_converged_reason -ksp_max_it 400 -pc_type mg -mg_levels_ksp_max_it 4 -mg_levels_ksp_type cg -mg_levels_pc_type bjacobi -options_left -ksp_rtol 1.0e-6 -mg_coarse_pc_type gamg -mg_coarse_ksp_type cg -mg_coarse_ksp_rtol 1.0e-1 -mg_coarse_ksp_converged_reason -ksp_type gcr -mg_levels_mat_type aijcusparse -vec_type cuda -mat_type aijcusparse

               ___          ___         _____         ___          ___
      ___     /  /\        /__/\       /  /::\       /  /\        /__/\
     /  /\   /  /::\       \  \:\     /  /:/\:\     /  /:/_      |  |::\
    /  /:/  /  /:/\:\       \  \:\   /  /:/  \:\   /  /:/ /\     |  |:|:\
   /  /:/  /  /:/~/::\  _____\__\:\ /__/:/ \__\:| /  /:/ /:/_  __|__|:|\:\
  /  /::\ /__/:/ /:/\:\/__/::::::::\\  \:\ /  /://__/:/ /:/ /\/__/::::| \:\
 /__/:/\:\\  \:\/:/__\/\  \:\~~\~~\/ \  \:\  /:/ \  \:\/:/ /:/\  \:\~~\__\/
 \__\/  \:\\  \::/      \  \:\  ~~~   \  \:\/:/   \  \::/ /:/  \  \:\
      \  \:\\  \:\       \  \:\        \  \::/     \  \:\/:/    \  \:\
       \__\/ \  \:\       \  \:\        \__\/       \  \::/      \  \:\
              \__\/        \__\/                     \__\/        \__\/

                          tandem version 4b6c45e

                       stack size limit = unlimited

                              Worker affinity
    0---------|----------|----------|----------|----------|----------|
    ----------|----------|----------|----------|----------|----------|
    --------8-|----------|----------|----------|----------|----------|
    ----------|----------|----------|----------|----------|----------|
    ----------|------

Ridge:create()
parsing fault_many_trace.csv
Ridge:create()
parsing fault_many_trace.csv
DOFs: 12634590
Mesh size: 36.7959
Multigrid P-levels: 1 2 
[1]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
[1]PETSC ERROR: GPU error
[1]PETSC ERROR: cuSPARSE errorcode 11 (CUSPARSE_STATUS_INSUFFICIENT_RESOURCES) : insufficient resources
hpc4geo commented 1 year ago

Nice. It is getting closer.

CUSPARSE_STATUS_INSUFFICIENT_RESOURCES refers to all conditions that prevent computing the results. In your case here, I think the error indicates the GPU ran out of memory.

  1. Could you replace -mg_levels_pc_type bjacobi with -mg_levels_pc_type jacobi and re-run?
  2. If the test above also fails, I suggest reducing the memory usage by coarsening the mesh.

I am very puzzled why the log generated contains lots of nan's where there should be run-times. I've never seen that before.

Thomas-Ulrich commented 1 year ago
  1. did not help so that's definitively a out of memory (see also on one GPU clearer error below).

               ___          ___         _____         ___          ___
      ___     /  /\        /__/\       /  /::\       /  /\        /__/\
     /  /\   /  /::\       \  \:\     /  /:/\:\     /  /:/_      |  |::\
    /  /:/  /  /:/\:\       \  \:\   /  /:/  \:\   /  /:/ /\     |  |:|:\
   /  /:/  /  /:/~/::\  _____\__\:\ /__/:/ \__\:| /  /:/ /:/_  __|__|:|\:\
  /  /::\ /__/:/ /:/\:\/__/::::::::\\  \:\ /  /://__/:/ /:/ /\/__/::::| \:\
 /__/:/\:\\  \:\/:/__\/\  \:\~~\~~\/ \  \:\  /:/ \  \:\/:/ /:/\  \:\~~\__\/
 \__\/  \:\\  \::/      \  \:\  ~~~   \  \:\/:/   \  \::/ /:/  \  \:\
      \  \:\\  \:\       \  \:\        \  \::/     \  \:\/:/    \  \:\
       \__\/ \  \:\       \  \:\        \__\/       \  \::/      \  \:\
              \__\/        \__\/                     \__\/        \__\/

                          tandem version 4b6c45e

                       stack size limit = unlimited

                              Worker affinity
    0123456789|0123456789|0123456789|0123456789|0123456789|0123456789|
    0123456789|0123456789|0123456789|0123456789|0123456789|0123456789|
    0123456789|0123456789|0123456789|0123456789|0123456789|0123456789|
    0123456789|0123456789|0123456789|0123456789|0123456789|0123456789|
    0123456789|012345

Ridge:create()
parsing fault_many_trace.csv
DOFs: 12634590
Mesh size: 36.7959
Multigrid P-levels: 1 2 
[0]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
[0]PETSC ERROR: GPU error
[0]PETSC ERROR: cuda error 2 (cudaErrorMemoryAllocation) : out of memory

Can be also see when running nvidia-smi -l,

Thu Sep 14 08:25:19 2023       
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 470.182.03   Driver Version: 470.182.03   CUDA Version: 11.4     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|                               |                      |               MIG M. |
|===============================+======================+======================|
|   0  NVIDIA GeForce ...  Off  | 00000000:01:00.0 Off |                  N/A |
| 30%   22C    P8     9W / 350W |    567MiB / 24268MiB |      0%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+
|   1  NVIDIA GeForce ...  Off  | 00000000:C1:00.0 Off |                  N/A |
| 30%   18C    P8     7W / 350W |      2MiB / 24268MiB |      0%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Processes:                                                                  |
|  GPU   GI   CI        PID   Type   Process name                  GPU Memory |
|        ID   ID                                                   Usage      |
|=============================================================================|
|    0   N/A  N/A    176938      C   static                            565MiB |
+-----------------------------------------------------------------------------+
Thu Sep 14 08:25:24 2023       
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 470.182.03   Driver Version: 470.182.03   CUDA Version: 11.4     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|                               |                      |               MIG M. |
|===============================+======================+======================|
|   0  NVIDIA GeForce ...  Off  | 00000000:01:00.0 Off |                  N/A |
| 30%   31C    P2   118W / 350W |  22205MiB / 24268MiB |     69%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+
|   1  NVIDIA GeForce ...  Off  | 00000000:C1:00.0 Off |                  N/A |
| 30%   18C    P8     7W / 350W |      2MiB / 24268MiB |      0%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+

Indeed when reducing the mesh size to 3914 nodes 25174 elements (h_fault=1) (DOFs: 608700), I can run the setup on the 2 GPUs.

Thomas-Ulrich commented 1 year ago

Overall, the GPU code converges much slowly to solution. Here is the log running on CPUs as mpiexec -n 2 static ridge.toml --output ridgecrest --mg_strategy twolevel --mg_coarse_level 1 --petsc -ksp_view -ksp_monitor -ksp_converged_reason -ksp_max_it 400 -pc_type mg -mg_levels_ksp_max_it 4 -mg_levels_ksp_type cg -mg_levels_pc_type bjacobi -options_left -ksp_rtol 1.0e-6 -mg_coarse_pc_type gamg -mg_coarse_ksp_type cg -mg_coarse_ksp_rtol 1.0e-1 -mg_coarse_ksp_converged_reason -ksp_type gcr

  0 KSP Residual norm 1.734965842704e+04 
    Linear mg_coarse_ solve converged due to CONVERGED_RTOL iterations 18
  1 KSP Residual norm 1.031526607053e+02 
    Linear mg_coarse_ solve converged due to CONVERGED_RTOL iterations 21
  2 KSP Residual norm 3.572058785172e+01 
    Linear mg_coarse_ solve converged due to CONVERGED_RTOL iterations 15
  3 KSP Residual norm 1.140666097752e+01 
    Linear mg_coarse_ solve converged due to CONVERGED_RTOL iterations 18
  4 KSP Residual norm 4.219243222064e+00 
    Linear mg_coarse_ solve converged due to CONVERGED_RTOL iterations 16
  5 KSP Residual norm 1.726757190108e+00 
    Linear mg_coarse_ solve converged due to CONVERGED_RTOL iterations 19
  6 KSP Residual norm 7.684999488409e-01 
    Linear mg_coarse_ solve converged due to CONVERGED_RTOL iterations 16
  7 KSP Residual norm 3.047540104055e-01 
    Linear mg_coarse_ solve converged due to CONVERGED_RTOL iterations 16
  8 KSP Residual norm 1.073314964141e-01 
    Linear mg_coarse_ solve converged due to CONVERGED_RTOL iterations 16
  9 KSP Residual norm 3.571480635536e-02 
    Linear mg_coarse_ solve converged due to CONVERGED_RTOL iterations 15
 10 KSP Residual norm 1.270504655145e-02 
Linear solve converged due to CONVERGED_RTOL iterations 10

The same setup converges much more slowly when running with the additional options: -mg_levels_mat_type aijcusparse -vec_type cuda -mat_type aijcusparse

  0 KSP Residual norm 1.734965842704e+04 
    Linear mg_coarse_ solve converged due to CONVERGED_RTOL iterations 37
  1 KSP Residual norm 1.112811862053e+02 
    Linear mg_coarse_ solve converged due to CONVERGED_RTOL iterations 37
  2 KSP Residual norm 4.532554406057e+01 
    Linear mg_coarse_ solve converged due to CONVERGED_RTOL iterations 37
  3 KSP Residual norm 2.375197340723e+01 
    Linear mg_coarse_ solve converged due to CONVERGED_RTOL iterations 37
  4 KSP Residual norm 1.672539760740e+01 
    Linear mg_coarse_ solve converged due to CONVERGED_RTOL iterations 37
  5 KSP Residual norm 1.414756771772e+01 
    Linear mg_coarse_ solve converged due to CONVERGED_RTOL iterations 37
  6 KSP Residual norm 1.121542586970e+01 
    Linear mg_coarse_ solve converged due to CONVERGED_RTOL iterations 37
  7 KSP Residual norm 8.941253504767e+00 
    Linear mg_coarse_ solve converged due to CONVERGED_RTOL iterations 37
  8 KSP Residual norm 7.158088359090e+00 
    Linear mg_coarse_ solve converged due to CONVERGED_RTOL iterations 37
  9 KSP Residual norm 5.939360463184e+00 
    Linear mg_coarse_ solve converged due to CONVERGED_RTOL iterations 37
 10 KSP Residual norm 5.106714061854e+00 
    Linear mg_coarse_ solve converged due to CONVERGED_RTOL iterations 37
 11 KSP Residual norm 4.616036762828e+00 
    Linear mg_coarse_ solve converged due to CONVERGED_RTOL iterations 37
 12 KSP Residual norm 4.151022296180e+00 
    Linear mg_coarse_ solve converged due to CONVERGED_RTOL iterations 37
 13 KSP Residual norm 3.745663945543e+00 
    Linear mg_coarse_ solve converged due to CONVERGED_RTOL iterations 37
 14 KSP Residual norm 3.342640042622e+00 
    Linear mg_coarse_ solve converged due to CONVERGED_RTOL iterations 37
 15 KSP Residual norm 2.870591058752e+00 
    Linear mg_coarse_ solve converged due to CONVERGED_RTOL iterations 37
 16 KSP Residual norm 2.577647604225e+00 
    Linear mg_coarse_ solve converged due to CONVERGED_RTOL iterations 37
 17 KSP Residual norm 2.329289494495e+00 
    Linear mg_coarse_ solve converged due to CONVERGED_RTOL iterations 37
 18 KSP Residual norm 1.958163659908e+00 
    Linear mg_coarse_ solve converged due to CONVERGED_RTOL iterations 37
 19 KSP Residual norm 1.648044186263e+00 
    Linear mg_coarse_ solve converged due to CONVERGED_RTOL iterations 37
 20 KSP Residual norm 1.417638055974e+00 
    Linear mg_coarse_ solve converged due to CONVERGED_RTOL iterations 37
 21 KSP Residual norm 1.280970685123e+00 
    Linear mg_coarse_ solve converged due to CONVERGED_RTOL iterations 37
 22 KSP Residual norm 1.101320821088e+00 
    Linear mg_coarse_ solve converged due to CONVERGED_RTOL iterations 37
 23 KSP Residual norm 9.074620117579e-01 
    Linear mg_coarse_ solve converged due to CONVERGED_RTOL iterations 37
 24 KSP Residual norm 7.159338239786e-01 
    Linear mg_coarse_ solve converged due to CONVERGED_RTOL iterations 37
 25 KSP Residual norm 6.068813693601e-01 
    Linear mg_coarse_ solve converged due to CONVERGED_RTOL iterations 37
 26 KSP Residual norm 5.600414430325e-01 
    Linear mg_coarse_ solve converged due to CONVERGED_RTOL iterations 37
 27 KSP Residual norm 5.085937212294e-01 
    Linear mg_coarse_ solve converged due to CONVERGED_RTOL iterations 37
 28 KSP Residual norm 4.576119448521e-01 
    Linear mg_coarse_ solve converged due to CONVERGED_RTOL iterations 37
 29 KSP Residual norm 3.928048766129e-01 
    Linear mg_coarse_ solve converged due to CONVERGED_RTOL iterations 37
 30 KSP Residual norm 3.234931639246e-01 
    Linear mg_coarse_ solve converged due to CONVERGED_RTOL iterations 37
 31 KSP Residual norm 2.947903349793e-01 
    Linear mg_coarse_ solve converged due to CONVERGED_RTOL iterations 37
 32 KSP Residual norm 2.695541476443e-01 
    Linear mg_coarse_ solve converged due to CONVERGED_RTOL iterations 37
 33 KSP Residual norm 2.434954480914e-01 
 (...)
     Linear mg_coarse_ solve converged due to CONVERGED_RTOL iterations 37
103 KSP Residual norm 1.668932459434e-02

Note also

There is one unused database option. It is:
Option left: name:-mg_levels_mat_type value: aijcusparse
Thomas-Ulrich commented 1 year ago

(for tracking the cause of the difference in the residual decrease) Here are the two logs, on 2 nodes: tandem_CPU.log tandem_GPU.log

Thomas-Ulrich commented 1 year ago

Testing the matrix free I get:

base) ulrich@heisenbug:/export/dump/ulrich/section_8_2/scenario-rc$ mpiexec -n 2 static ridge.toml  --output ridgecrest  --matrix_free --mg_strategy twolevel --mg_coarse_level 1 --petsc -ksp_view -ksp_monitor -ksp_converged_reason -ksp_max_it 40 -pc_type mg -mg_levels_ksp_max_it 4 -mg_levels_ksp_type cg -mg_levels_pc_type bjacobi -options_left -ksp_rtol 1.0e-6 -mg_coarse_pc_type gamg -mg_coarse_ksp_type cg -mg_coarse_ksp_rtol 1.0e-1 -mg_coarse_ksp_converged_reason -ksp_type gcr  -vec_type cuda -mat_type aijcusparse

optional argument in parameter sequence
Usage: static [options] config 

Positional arguments:
config                  Configuration file (.toml)

Optional arguments:
-h --help               shows help message and exits
-v --version            prints version information and exits
--petsc                 PETSc options, must be passed last!
--resolution            Non-negative resolution parameter
--method           
--type             
--lib              
--scenario         
--matrix_free      
--test_matrix_free 
--mg_coarse_level  
--mg_strategy      
--profile               Run static in profile mode. The parameter controls the amount of repetitions.
--output                Output file name
--mesh_file        
--------------------------------------------------------------------------
Primary job  terminated normally, but 1 process returned
a non-zero exit code. Per user-direction, the job has been aborted.
--------------------------------------------------------------------------
hpc4geo commented 1 year ago

Thanks for the log files. I will take a look. To use matrix-free you need --matrix_free true. However don't waste your time with this option. Despite what you might think it will NOT reduce the memory foot print. When I was referring to 'matrix-free' and reducing the memory usage I was explicitly making reference to branch dmay/solver-dev . This branch is not yet GPU compatible.

hpc4geo commented 1 year ago

The name of the issue is too vague. The current issues are not related to getting something running on a GPU. Closing this and make a new issue.