search

LLNL / Caliper

Caliper is an instrumentation and performance profiling library
http://software.llnl.gov/Caliper/
BSD 3-Clause "New" or "Revised" License
350 stars 66 forks source link

Segfault with caliper v2.10 and master (NEURON Program) #529

Open pramodk opened 9 months ago

pramodk commented 9 months ago

Dear David,

We have been using v2.8 for quite some time in our production deployment of the NEURON simulator and it was working fine. Last week we upgraded to latest v2.10.0 (via spack) and I am seeing below segfault with one of our test:

$ gdb /gpfs/bbp.cscs.ch/home/kumbhar/tmp/caliper/build_caliper_210/bin/nrniv
(gdb) set environment HOC_LIBRARY_PATH /gpfs/bbp.cscs.ch/home/kumbhar/tmp/caliper/spack-src/test/ringtes
(gdb) set environment CALI_CONFIG runtime-report,calc.inclusive

(gdb) r /gpfs/bbp.cscs.ch/home/kumbhar/tmp/caliper/spack-src/test/ringtest/ring.hoc

...
NEURON -- VERSION 9.0a-143-g94ce2eb+ HEAD (94ce2eb+) 2024-02-02
Duke, Yale, and the BlueBrain Project -- Copyright 1984-2022
See http://neuron.yale.edu/neuron/credits

Program received signal SIGSEGV, Segmentation fault.
0x00007fffed8ec730 in do_lookup_x () from /lib64/ld-linux-x86-64.so.2

and relevant backtrace + code section is:

(gdb) bt
#0  0x00007fffed8ec730 in do_lookup_x () from /lib64/ld-linux-x86-64.so.2
#1  0x00007fffed8ed09f in _dl_lookup_symbol_x () from /lib64/ld-linux-x86-64.so.2
#2  0x00007fffe9055d09 in do_sym () from /lib64/libc.so.6
#3  0x00007fffeb9280c4 in dlsym_doit () from /lib64/libdl.so.2
#4  0x00007fffed8f27d4 in _dl_catch_error () from /lib64/ld-linux-x86-64.so.2
#5  0x00007fffeb9285ad in _dlerror_run () from /lib64/libdl.so.2
#6  0x00007fffeb928118 in dlsym () from /lib64/libdl.so.2
#7  0x00007fffeb8b0938 in dlsym_wrapper () from /gpfs/bbp.cscs.ch/ssd/apps/bsd/2024-02-01/stage_applications/install_gcc-12.3.0-skylake/caliper-2.10.0-btrc4s/lib64/libcaliper.so.2
#8  0x00007fffeabb3dff in munmap (addr=0x7fffedad3000, len=16384) at reg_cache.c:723
#9  0x00007fffeb36d48f in tstate_delete_common (tstate=tstate@entry=0x7fffeb64cd78 <_PyRuntime+166328>, gilstate=<optimized out>) at Python/pystate.c:1105
#10 0x00007fffeb36e6f0 in _PyThreadState_Delete (check_current=0, tstate=0x7fffeb64cd78 <_PyRuntime+166328>) at Python/pystate.c:1119
#11 zapthreads (check_current=0, interp=0x7fffeb6329f8 <_PyRuntime+58936>) at Python/pystate.c:483
#12 PyInterpreterState_Delete (interp=0x7fffeb6329f8 <_PyRuntime+58936>) at Python/pystate.c:493
#13 0x00007fffeb36c455 in finalize_interp_delete (interp=<optimized out>) at Python/pylifecycle.c:1746
#14 Py_FinalizeEx () at Python/pylifecycle.c:1900
#15 Py_FinalizeEx () at Python/pylifecycle.c:1751
#16 0x00007fffed0623cc in nrnpython_start (b=0) at /gpfs/bbp.cscs.ch/home/kumbhar/tmp/caliper/spack-src/src/nrnpython/nrnpython.cpp:260
#17 0x00007fffec5208c9 in ivocmain_session (argc=2, argv=0x7ffffffef6f8, env=0x7ffffffef710, start_session=1) at /gpfs/bbp.cscs.ch/home/kumbhar/tmp/caliper/spack-src/src/ivoc/ivocmain.cpp:772
#18 0x00007fffec5203a8 in ivocmain (argc=2, argv=0x7ffffffef6f8, env=0x7ffffffef710) at /gpfs/bbp.cscs.ch/home/kumbhar/tmp/caliper/spack-src/src/ivoc/ivocmain.cpp:349
#19 0x0000000000402258 in main (argc=2, argv=0x7ffffffef6f8, env=0x7ffffffef710) at /gpfs/bbp.cscs.ch/home/kumbhar/tmp/caliper/spack-src/src/ivoc/nrnmain.cpp:71

(gdb) f 16
#16 0x00007fffed0623cc in nrnpython_start (b=0) at /gpfs/bbp.cscs.ch/home/kumbhar/tmp/caliper/spack-src/src/nrnpython/nrnpython.cpp:260
260             Py_Finalize();

(gdb) list
255         if (b == 0 && started) {
256             PyGILState_STATE gilsav = PyGILState_Ensure();
257             assert(basic_sys_path);
258             Py_DECREF(basic_sys_path);
259             basic_sys_path = nullptr;
260             Py_Finalize();
261             // because of finalize, no PyGILState_Release(gilsav);
262             started = 0;
263         }
264         if (b == 2 && started) {
(gdb)

Maybe this is sufficient for you to see what could have caused a regression? Happy to provide additional info!

Thanks in advance!

Notes:

  1. the same version of NEURON works fine with 2.8.0
  2. if I don't set CALI_CONFIG then program runs fine as well
  3. spack spec for caliper installation is:
$ spack spec -I caliper@2.10.0
==> Warning: duplicate found for clang@=3.4.2 on rhel7/x86_64. Edit your compilers.yaml configuration to remove it.
==> Warning: duplicate found for gcc@=4.8.5 on rhel7/x86_64. Edit your compilers.yaml configuration to remove it.
Input spec
--------------------------------
 -   caliper@2.10.0

Concretized
--------------------------------
[^]  caliper@2.10.0%gcc@12.3.0+adiak~cuda~fortran+gotcha~ipo+kokkos+libdw~libpfm+libunwind+mpi+papi~rocm+sampler+shared~sosflow~variorum build_system=cmake build_type=Release generator=make arch=linux-rhel7-skylake
[^]      ^adiak@0.2.2%gcc@12.3.0~ipo+mpi+shared build_system=cmake build_type=Release generator=make arch=linux-rhel7-skylake
[e]      ^cmake@3.27.7%gcc@12.3.0~doc+ncurses+ownlibs build_system=generic build_type=Release arch=linux-rhel7-skylake
[^]      ^elfutils@0.189%gcc@12.3.0~debuginfod+exeprefix+nls build_system=autotools arch=linux-rhel7-skylake
[^]          ^bzip2@1.0.8%gcc@12.3.0~debug~pic+shared build_system=generic arch=linux-rhel7-skylake
[^]              ^diffutils@3.9%gcc@12.3.0 build_system=autotools arch=linux-rhel7-skylake
[^]          ^gettext@0.22.3%gcc@12.3.0+bzip2+curses+git~libunistring+libxml2+pic+shared+tar+xz build_system=autotools arch=linux-rhel7-skylake
[^]              ^libxml2@2.10.3%gcc@12.3.0+pic~python+shared build_system=autotools arch=linux-rhel7-skylake
[^]              ^ncurses@6.4%gcc@12.3.0~symlinks+termlib abi=none build_system=autotools arch=linux-rhel7-skylake
[^]              ^tar@1.34%gcc@12.3.0 build_system=autotools zip=pigz arch=linux-rhel7-skylake
[^]                  ^pigz@2.7%gcc@12.3.0 build_system=makefile arch=linux-rhel7-skylake
[^]          ^libiconv@1.17%gcc@12.3.0 build_system=autotools libs=shared,static arch=linux-rhel7-skylake
[^]          ^m4@1.4.19%gcc@12.3.0+sigsegv build_system=autotools patches=9dc5fbd,bfdffa7 arch=linux-rhel7-skylake
[^]              ^libsigsegv@2.14%gcc@12.3.0 build_system=autotools arch=linux-rhel7-skylake
[^]          ^pkgconf@1.9.5%gcc@12.3.0 build_system=autotools arch=linux-rhel7-skylake
[^]          ^xz@5.4.1%gcc@12.3.0~pic build_system=autotools libs=shared,static arch=linux-rhel7-skylake
[^]          ^zlib-ng@2.1.4%gcc@12.3.0+compat+opt build_system=autotools arch=linux-rhel7-skylake
[^]          ^zstd@1.5.5%gcc@12.3.0+programs build_system=makefile compression=none libs=shared,static arch=linux-rhel7-skylake
[^]      ^gmake@4.4.1%gcc@12.3.0~guile build_system=generic arch=linux-rhel7-skylake
[e]      ^hpe-mpi@2.27.p1.hmpt%gcc@12.3.0 build_system=generic arch=linux-rhel7-skylake
[^]      ^libunwind@1.6.2%gcc@12.3.0~block_signals~conservative_checks~cxx_exceptions~debug~debug_frame+docs~pic+tests+weak_backtrace~xz~zlib build_system=autotools components=none libs=shared,static arch=linux-rhel7-skylake
[^]      ^papi@6.0.0.1%gcc@12.3.0~cuda~debug+example~infiniband~lmsensors~nvml~powercap~rapl~rocm~rocm_smi~sde+shared~static_tools build_system=autotools arch=linux-rhel7-skylake
[e]      ^python@3.11.6%gcc@12.3.0+bz2+crypt+ctypes+dbm~debug+libxml2+lzma~nis~optimizations+pic+pyexpat+pythoncmd+readline+shared+sqlite3+ssl~tkinter+uuid+zlib build_system=generic patches=13fa8bf,b0615b2,ebdca64,f2fd060 arch=linux-rhel7-skylake
pramodk commented 9 months ago

By the way, I just checked v2.9.1 and its working as expected. So it's only 2.10.0 that has this issue.

daboehme commented 9 months ago

Hi @pramodk, thanks for reporting this. Looks like this is an issue in gotcha, our function interception library. I'll bring it up over there. In the meantime you can try building Caliper without gotcha support as a workaround (use spack install caliper~gotcha). Let me know if that helps!

If possible it would be great if you can run the code again with GOTCHA_DEBUG=3 set as an environment variable and send us the debug output, that may help us figure out what's going on here.

pramodk commented 9 months ago

If possible it would be great if you can run the code again with GOTCHA_DEBUG=3 set as an environment variable and send us the debug output

Certainly! Here is the debug log:

GOTCHA_DEBUG=3 CALI_CONFIG=runtime-report,calc.inclusive HOC_LIBRARY_PATH=/gpfs/bbp.cscs.ch/home/kumbhar/tmp/caliper/spack-src/test/ringtest/ /gpfs/bbp.cscs.ch/home/kumbhar/tmp/caliper/build_caliper_210/bin/nrniv /gpfs/bbp.cscs.ch/home/kumbhar/tmp/caliper/spack-src/test/ringtest/ring.hoc
NEURON -- VERSION 9.0a-143-g94ce2eb+ HEAD (94ce2eb+) 2024-02-02
Duke, Yale, and the BlueBrain Project -- Copyright 1984-2022
See http://neuron.yale.edu/neuron/credits

[31328/31328][gotcha_utils.c:45] - Gotcha debug initialized at level 3
[31328/31328][gotcha.c:282] - User called gotcha_wrap for tool gotcha with 2 bindings
        0: dlopen will map to 0x7fffeb8b09f0
        1: dlsym will map to 0x7fffeb8b0770
[31328/31328][gotcha.c:290] - Initializing 2 user binding entries to NULL
[31328/31328][gotcha.c:36] - Updating binding address pointer at 0x7fffeb926de8 to (nil)
[31328/31328][gotcha.c:36] - Updating binding address pointer at 0x7fffeb926de0 to (nil)
[31328/31328][tool.c:66] - Found no existing tool with name gotcha
[31328/31328][tool.c:82] - Created new tool gotcha
[31328/31328][gotcha.c:306] - Moved current_generation to 1 in gotcha_wrap
[31328/31328][gotcha.c:308] - Creating internal binding data structures and adding binding to tool
[31328/31328][tool.c:135] - Created new binding table of size 2 for tool gotcha
[31328/31328][gotcha.c:315] - Processing 2 bindings
[31328/31328][gotcha.c:149] - gotcha_rewrite_wrapper_orders for binding dlopen in tool gotcha of priority -1
[31328/31328][gotcha.c:102] - Creating new library object for /gpfs/bbp.cscs.ch/home/kumbhar/tmp/caliper/build_caliper_210/lib/libnrniv.so
[31328/31328][gotcha.c:57] - Searching for exported symbols in /gpfs/bbp.cscs.ch/home/kumbhar/tmp/caliper/build_caliper_210/lib/libnrniv.so
[31328/31328][gotcha.c:66] - Checking GNU hash for dlopen in /gpfs/bbp.cscs.ch/home/kumbhar/tmp/caliper/build_caliper_210/lib/libnrniv.so
[31328/31328][gotcha.c:71] - Checking ELF hash for dlopen in /gpfs/bbp.cscs.ch/home/kumbhar/tmp/caliper/build_caliper_210/lib/libnrniv.so
[31328/31328][gotcha.c:80] - Symbol dlopen found but not exported in /gpfs/bbp.cscs.ch/home/kumbhar/tmp/caliper/build_caliper_210/lib/libnrniv.so
[31328/31328][gotcha.c:102] - Creating new library object for /lib64/libdl.so.2
[31328/31328][gotcha.c:57] - Searching for exported symbols in /lib64/libdl.so.2
[31328/31328][gotcha.c:66] - Checking GNU hash for dlopen in /lib64/libdl.so.2
[31328/31328][gotcha.c:85] - Symbol dlopen found in /lib64/libdl.so.2 at 0x7fffeb928010
[31328/31328][gotcha.c:46] - Updating binding address pointer at 0x7d5138 to 0x7fffeb928010
[31328/31328][gotcha.c:334] - Symbol dlopen needs binding from application
[31328/31328][gotcha.c:149] - gotcha_rewrite_wrapper_orders for binding dlsym in tool gotcha of priority -1
[31328/31328][gotcha.c:156] - Adding new entry for dlsym to hash table
[31328/31328][gotcha.c:324] - Symbol dlsym needs lookup operation
[31328/31328][gotcha.c:98] - Looking up exported symbols for dlsym
[31328/31328][gotcha.c:57] - Searching for exported symbols in [EMPTY]
[31328/31328][gotcha.c:66] - Checking GNU hash for dlsym in [EMPTY]
[31328/31328][gotcha.c:71] - Checking ELF hash for dlsym in [EMPTY]
[31328/31328][gotcha.c:76] - dlsym not found in [EMPTY]
[31328/31328][gotcha.c:53] - Skipping VDSO library at 0x7fffee402000 with name [EMPTY]
[31328/31328][gotcha.c:57] - Searching for exported symbols in /gpfs/bbp.cscs.ch/home/kumbhar/tmp/caliper/build_caliper_210/lib/libnrniv.so
[31328/31328][gotcha.c:66] - Checking GNU hash for dlsym in /gpfs/bbp.cscs.ch/home/kumbhar/tmp/caliper/build_caliper_210/lib/libnrniv.so
[31328/31328][gotcha.c:71] - Checking ELF hash for dlsym in /gpfs/bbp.cscs.ch/home/kumbhar/tmp/caliper/build_caliper_210/lib/libnrniv.so
[31328/31328][gotcha.c:80] - Symbol dlsym found but not exported in /gpfs/bbp.cscs.ch/home/kumbhar/tmp/caliper/build_caliper_210/lib/libnrniv.so
[31328/31328][gotcha.c:57] - Searching for exported symbols in /lib64/libdl.so.2
[31328/31328][gotcha.c:66] - Checking GNU hash for dlsym in /lib64/libdl.so.2
[31328/31328][gotcha.c:85] - Symbol dlsym found in /lib64/libdl.so.2 at 0x7fffeb9280d0
[31328/31328][gotcha.c:46] - Updating binding address pointer at 0x7d5158 to 0x7fffeb9280d0
[31328/31328][gotcha.c:334] - Symbol dlsym needs binding from application
[31328/31328][gotcha.c:268] - Searching all callsites for 2 bindings
[31328/31328][gotcha.c:225] - Setting library [EMPTY] GOT table from 0x404000 to +4096 to writeable
[31328/31328][gotcha.c:225] - Setting library /gpfs/bbp.cscs.ch/home/kumbhar/tmp/caliper/build_caliper_210/lib/libnrniv.so GOT table from 0x7fffed638000 to +983040 to writeable
[31328/31328][gotcha.c:200] - Remapped call to dlsym at 0x7fffed639d68 in /gpfs/bbp.cscs.ch/home/kumbhar/tmp/caliper/build_caliper_210/lib/libnrniv.so to wrapper at 0x0x7fffeb8b0770
[31328/31328][gotcha.c:200] - Remapped call to dlopen at 0x7fffed67b4d8 in /gpfs/bbp.cscs.ch/home/kumbhar/tmp/caliper/build_caliper_210/lib/libnrniv.so to wrapper at 0x0x7fffeb8b09f0
[31328/31328][gotcha.c:225] - Setting library /lib64/libdl.so.2 GOT table from 0x7fffebb2a000 to +4096 to writeable
[31328/31328][gotcha.c:240] - Creating new library object for /gpfs/bbp.cscs.ch/ssd/apps/bsd/2024-02-01/stage_applications/install_gcc-12.3.0-skylake/readline-8.2-pdugg7/lib/libreadline.so.8
[31328/31328][gotcha.c:225] - Setting library /gpfs/bbp.cscs.ch/ssd/apps/bsd/2024-02-01/stage_applications/install_gcc-12.3.0-skylake/readline-8.2-pdugg7/lib/libreadline.so.8 GOT table from 0x7fffedabf000 to +12288 to writeable
[31328/31328][gotcha.c:240] - Creating new library object for /gpfs/bbp.cscs.ch/ssd/apps/bsd/2024-02-01/stage_applications/install_gcc-12.3.0-skylake/ncurses-6.4-6kdbhv/lib/libncurses.so.6
[31328/31328][gotcha.c:225] - Setting library /gpfs/bbp.cscs.ch/ssd/apps/bsd/2024-02-01/stage_applications/install_gcc-12.3.0-skylake/ncurses-6.4-6kdbhv/lib/libncurses.so.6 GOT table from 0x7fffeda6e000 to +8192 to writeable
[31328/31328][gotcha.c:240] - Creating new library object for /gpfs/bbp.cscs.ch/ssd/apps/bsd/2024-02-01/stage_applications/install_gcc-12.3.0-skylake/ncurses-6.4-6kdbhv/lib/libform.so.6
[31328/31328][gotcha.c:225] - Setting library /gpfs/bbp.cscs.ch/ssd/apps/bsd/2024-02-01/stage_applications/install_gcc-12.3.0-skylake/ncurses-6.4-6kdbhv/lib/libform.so.6 GOT table from 0x7fffeda45000 to +4096 to writeable
[31328/31328][gotcha.c:240] - Creating new library object for /gpfs/bbp.cscs.ch/ssd/apps/bsd/2024-02-01/stage_applications/install_gcc-12.3.0-skylake/caliper-2.10.0-btrc4s/lib64/libcaliper.so.2
[31328/31328][gotcha.c:225] - Setting library /gpfs/bbp.cscs.ch/ssd/apps/bsd/2024-02-01/stage_applications/install_gcc-12.3.0-skylake/caliper-2.10.0-btrc4s/lib64/libcaliper.so.2 GOT table from 0x7fffeb916000 to +28672 to writeable
[31328/31328][gotcha.c:240] - Creating new library object for /gpfs/bbp.cscs.ch/ssd/apps/bsd/2024-02-01/stage_externals/install_gcc-12.3.0-skylake/python-3.11.6-bj4i6m/lib/libpython3.11.so.1.0
[31328/31328][gotcha.c:225] - Setting library /gpfs/bbp.cscs.ch/ssd/apps/bsd/2024-02-01/stage_externals/install_gcc-12.3.0-skylake/python-3.11.6-bj4i6m/lib/libpython3.11.so.1.0 GOT table from 0x7fffeb519000 to +36864 to writeable
[31328/31328][gotcha.c:200] - Remapped call to dlopen at 0x7fffeb51b8b8 in /gpfs/bbp.cscs.ch/ssd/apps/bsd/2024-02-01/stage_externals/install_gcc-12.3.0-skylake/python-3.11.6-bj4i6m/lib/libpython3.11.so.1.0 to wrapper at 0x0x7fffeb8b09f0
[31328/31328][gotcha.c:200] - Remapped call to dlsym at 0x7fffeb51c7f8 in /gpfs/bbp.cscs.ch/ssd/apps/bsd/2024-02-01/stage_externals/install_gcc-12.3.0-skylake/python-3.11.6-bj4i6m/lib/libpython3.11.so.1.0 to wrapper at 0x0x7fffeb8b0770
[31328/31328][gotcha.c:240] - Creating new library object for /lib64/libpthread.so.0
[31328/31328][gotcha.c:225] - Setting library /lib64/libpthread.so.0 GOT table from 0x7fffeb0cd000 to +4096 to writeable
[31328/31328][gotcha.c:240] - Creating new library object for /gpfs/bbp.cscs.ch/ssd/apps/bsd/2024-02-01/stage_externals/install_gcc-12.3.0-skylake/hpe-mpi-2.27.p1.hmpt-6rr256/lib/libmpi.so
[31328/31328][gotcha.c:225] - Setting library /gpfs/bbp.cscs.ch/ssd/apps/bsd/2024-02-01/stage_externals/install_gcc-12.3.0-skylake/hpe-mpi-2.27.p1.hmpt-6rr256/lib/libmpi.so GOT table from 0x7fffeae78000 to +20480 to writeable
[31328/31328][gotcha.c:200] - Remapped call to dlopen at 0x7fffeae78c68 in /gpfs/bbp.cscs.ch/ssd/apps/bsd/2024-02-01/stage_externals/install_gcc-12.3.0-skylake/hpe-mpi-2.27.p1.hmpt-6rr256/lib/libmpi.so to wrapper at 0x0x7fffeb8b09f0
[31328/31328][gotcha.c:200] - Remapped call to dlsym at 0x7fffeae79440 in /gpfs/bbp.cscs.ch/ssd/apps/bsd/2024-02-01/stage_externals/install_gcc-12.3.0-skylake/hpe-mpi-2.27.p1.hmpt-6rr256/lib/libmpi.so to wrapper at 0x0x7fffeb8b0770
[31328/31328][gotcha.c:240] - Creating new library object for /gpfs/bbp.cscs.ch/ssd/apps/bsd/2024-02-01/stage_compilers/install_gcc-4.8.5-haswell/gcc-12.3.0-lvcks3/lib/gcc/x86_64-pc-linux-gnu/12.3.0/../../../../lib64/libatomic.so.1
[31328/31328][gotcha.c:225] - Setting library /gpfs/bbp.cscs.ch/ssd/apps/bsd/2024-02-01/stage_compilers/install_gcc-4.8.5-haswell/gcc-12.3.0-lvcks3/lib/gcc/x86_64-pc-linux-gnu/12.3.0/../../../../lib64/libatomic.so.1 GOT table from 0x7fffeda30000 to +4096 to writeable
[31328/31328][gotcha.c:240] - Creating new library object for /gpfs/bbp.cscs.ch/ssd/apps/bsd/2024-02-01/stage_externals/install_gcc-12.3.0-skylake/nvhpc-23.1-vficpy/Linux_x86_64/23.1/compilers/lib/libnvhpcatm.so
[31328/31328][gotcha.c:225] - Setting library /gpfs/bbp.cscs.ch/ssd/apps/bsd/2024-02-01/stage_externals/install_gcc-12.3.0-skylake/nvhpc-23.1-vficpy/Linux_x86_64/23.1/compilers/lib/libnvhpcatm.so GOT table from 0x7fffeaa82000 to +4096 to writeable
[31328/31328][gotcha.c:240] - Creating new library object for /gpfs/bbp.cscs.ch/ssd/apps/bsd/2024-02-01/stage_compilers/install_gcc-4.8.5-haswell/gcc-12.3.0-lvcks3/lib/gcc/x86_64-pc-linux-gnu/12.3.0/../../../../lib64/libstdc++.so.6
[31328/31328][gotcha.c:225] - Setting library /gpfs/bbp.cscs.ch/ssd/apps/bsd/2024-02-01/stage_compilers/install_gcc-4.8.5-haswell/gcc-12.3.0-lvcks3/lib/gcc/x86_64-pc-linux-gnu/12.3.0/../../../../lib64/libstdc++.so.6 GOT table from 0x7fffea873000 to +28672 to writeable
[31328/31328][gotcha.c:240] - Creating new library object for /gpfs/bbp.cscs.ch/ssd/apps/bsd/2024-02-01/stage_externals/install_gcc-12.3.0-skylake/nvhpc-23.1-vficpy/Linux_x86_64/23.1/compilers/lib/libnvomp.so
[31328/31328][gotcha.c:225] - Setting library /gpfs/bbp.cscs.ch/ssd/apps/bsd/2024-02-01/stage_externals/install_gcc-12.3.0-skylake/nvhpc-23.1-vficpy/Linux_x86_64/23.1/compilers/lib/libnvomp.so GOT table from 0x7fffe9c0f000 to +4096 to writeable
[31328/31328][gotcha.c:200] - Remapped call to dlopen at 0x7fffe9c0f378 in /gpfs/bbp.cscs.ch/ssd/apps/bsd/2024-02-01/stage_externals/install_gcc-12.3.0-skylake/nvhpc-23.1-vficpy/Linux_x86_64/23.1/compilers/lib/libnvomp.so to wrapper at 0x0x7fffeb8b09f0
[31328/31328][gotcha.c:200] - Remapped call to dlsym at 0x7fffe9c0f500 in /gpfs/bbp.cscs.ch/ssd/apps/bsd/2024-02-01/stage_externals/install_gcc-12.3.0-skylake/nvhpc-23.1-vficpy/Linux_x86_64/23.1/compilers/lib/libnvomp.so to wrapper at 0x0x7fffeb8b0770
[31328/31328][gotcha.c:240] - Creating new library object for /gpfs/bbp.cscs.ch/ssd/apps/bsd/2024-02-01/stage_externals/install_gcc-12.3.0-skylake/nvhpc-23.1-vficpy/Linux_x86_64/23.1/compilers/lib/libnvcpumath-avx2.so
[31328/31328][gotcha.c:225] - Setting library /gpfs/bbp.cscs.ch/ssd/apps/bsd/2024-02-01/stage_externals/install_gcc-12.3.0-skylake/nvhpc-23.1-vficpy/Linux_x86_64/23.1/compilers/lib/libnvcpumath-avx2.so GOT table from 0x7fffe992b000 to +12288 to writeable
[31328/31328][gotcha.c:240] - Creating new library object for /gpfs/bbp.cscs.ch/ssd/apps/bsd/2024-02-01/stage_externals/install_gcc-12.3.0-skylake/nvhpc-23.1-vficpy/Linux_x86_64/23.1/compilers/lib/libnvc.so
[31328/31328][gotcha.c:225] - Setting library /gpfs/bbp.cscs.ch/ssd/apps/bsd/2024-02-01/stage_externals/install_gcc-12.3.0-skylake/nvhpc-23.1-vficpy/Linux_x86_64/23.1/compilers/lib/libnvc.so GOT table from 0x7fffe9547000 to +8192 to writeable
[31328/31328][gotcha.c:240] - Creating new library object for /lib64/libc.so.6
[31328/31328][gotcha.c:225] - Setting library /lib64/libc.so.6 GOT table from 0x7fffe92df000 to +4096 to writeable
[31328/31328][gotcha.c:240] - Creating new library object for /gpfs/bbp.cscs.ch/ssd/apps/bsd/2024-02-01/stage_compilers/install_gcc-4.8.5-haswell/gcc-12.3.0-lvcks3/lib/gcc/x86_64-pc-linux-gnu/12.3.0/../../../../lib64/libgcc_s.so.1
[31328/31328][gotcha.c:225] - Setting library /gpfs/bbp.cscs.ch/ssd/apps/bsd/2024-02-01/stage_compilers/install_gcc-4.8.5-haswell/gcc-12.3.0-lvcks3/lib/gcc/x86_64-pc-linux-gnu/12.3.0/../../../../lib64/libgcc_s.so.1 GOT table from 0x7fffeda25000 to +4096 to writeable
[31328/31328][gotcha.c:240] - Creating new library object for /lib64/libm.so.6
[31328/31328][gotcha.c:225] - Setting library /lib64/libm.so.6 GOT table from 0x7fffe8f17000 to +4096 to writeable
[31328/31328][gotcha.c:240] - Creating new library object for /gpfs/bbp.cscs.ch/ssd/apps/bsd/2024-02-01/stage_externals/install_gcc-12.3.0-skylake/nvhpc-23.1-vficpy/Linux_x86_64/23.1/compilers/lib/libnvcpumath.so
[31328/31328][gotcha.c:225] - Setting library /gpfs/bbp.cscs.ch/ssd/apps/bsd/2024-02-01/stage_externals/install_gcc-12.3.0-skylake/nvhpc-23.1-vficpy/Linux_x86_64/23.1/compilers/lib/libnvcpumath.so GOT table from 0x7fffe8bde000 to +12288 to writeable
[31328/31328][gotcha.c:240] - Creating new library object for /lib64/ld-linux-x86-64.so.2
[31328/31328][gotcha.c:225] - Setting library /lib64/ld-linux-x86-64.so.2 GOT table from 0x7fffedb04000 to +4096 to writeable
[31328/31328][gotcha.c:240] - Creating new library object for /gpfs/bbp.cscs.ch/ssd/apps/bsd/2024-02-01/stage_applications/install_gcc-12.3.0-skylake/ncurses-6.4-6kdbhv/lib/libncursesw.so.6
[31328/31328][gotcha.c:225] - Setting library /gpfs/bbp.cscs.ch/ssd/apps/bsd/2024-02-01/stage_applications/install_gcc-12.3.0-skylake/ncurses-6.4-6kdbhv/lib/libncursesw.so.6 GOT table from 0x7fffeda02000 to +8192 to writeable
[31328/31328][gotcha.c:240] - Creating new library object for /gpfs/bbp.cscs.ch/ssd/apps/bsd/2024-02-01/stage_applications/install_gcc-12.3.0-skylake/ncurses-6.4-6kdbhv/lib/libtinfow.so.6
[31328/31328][gotcha.c:225] - Setting library /gpfs/bbp.cscs.ch/ssd/apps/bsd/2024-02-01/stage_applications/install_gcc-12.3.0-skylake/ncurses-6.4-6kdbhv/lib/libtinfow.so.6 GOT table from 0x7fffed9c8000 to +8192 to writeable
[31328/31328][gotcha.c:240] - Creating new library object for /gpfs/bbp.cscs.ch/ssd/apps/bsd/2024-02-01/stage_applications/install_gcc-12.3.0-skylake/ncurses-6.4-6kdbhv/lib/libtinfo.so.6
[31328/31328][gotcha.c:225] - Setting library /gpfs/bbp.cscs.ch/ssd/apps/bsd/2024-02-01/stage_applications/install_gcc-12.3.0-skylake/ncurses-6.4-6kdbhv/lib/libtinfo.so.6 GOT table from 0x7fffed985000 to +8192 to writeable
[31328/31328][gotcha.c:240] - Creating new library object for /gpfs/bbp.cscs.ch/ssd/apps/bsd/2024-02-01/stage_applications/install_gcc-12.3.0-skylake/papi-6.0.0.1-vol6np/lib/libpapi.so.6.0
[31328/31328][gotcha.c:225] - Setting library /gpfs/bbp.cscs.ch/ssd/apps/bsd/2024-02-01/stage_applications/install_gcc-12.3.0-skylake/papi-6.0.0.1-vol6np/lib/libpapi.so.6.0 GOT table from 0x7fffe87ee000 to +8192 to writeable
[31328/31328][gotcha.c:240] - Creating new library object for /gpfs/bbp.cscs.ch/ssd/apps/bsd/2024-02-01/stage_applications/install_gcc-12.3.0-skylake/elfutils-0.189-a46bi6/lib/libdw.so.1
[31328/31328][gotcha.c:225] - Setting library /gpfs/bbp.cscs.ch/ssd/apps/bsd/2024-02-01/stage_applications/install_gcc-12.3.0-skylake/elfutils-0.189-a46bi6/lib/libdw.so.1 GOT table from 0x7fffe879c000 to +8192 to writeable
[31328/31328][gotcha.c:240] - Creating new library object for /gpfs/bbp.cscs.ch/ssd/apps/bsd/2024-02-01/stage_applications/install_gcc-12.3.0-skylake/libunwind-1.6.2-7xnekx/lib/libunwind.so.8
[31328/31328][gotcha.c:225] - Setting library /gpfs/bbp.cscs.ch/ssd/apps/bsd/2024-02-01/stage_applications/install_gcc-12.3.0-skylake/libunwind-1.6.2-7xnekx/lib/libunwind.so.8 GOT table from 0x7fffed936000 to +4096 to writeable
[31328/31328][gotcha.c:240] - Creating new library object for /gpfs/bbp.cscs.ch/ssd/apps/bsd/2024-02-01/stage_applications/install_gcc-12.3.0-skylake/adiak-0.2.2-inlzf5/lib/libadiak.so
[31328/31328][gotcha.c:225] - Setting library /gpfs/bbp.cscs.ch/ssd/apps/bsd/2024-02-01/stage_applications/install_gcc-12.3.0-skylake/adiak-0.2.2-inlzf5/lib/libadiak.so GOT table from 0x7fffed927000 to +4096 to writeable
[31328/31328][gotcha.c:200] - Remapped call to dlsym at 0x7fffed928138 in /gpfs/bbp.cscs.ch/ssd/apps/bsd/2024-02-01/stage_applications/install_gcc-12.3.0-skylake/adiak-0.2.2-inlzf5/lib/libadiak.so to wrapper at 0x0x7fffeb8b0770
[31328/31328][gotcha.c:240] - Creating new library object for /lib64/librt.so.1
[31328/31328][gotcha.c:225] - Setting library /lib64/librt.so.1 GOT table from 0x7fffe8700000 to +4096 to writeable
[31328/31328][gotcha.c:240] - Creating new library object for /gpfs/bbp.cscs.ch/ssd/apps/bsd/2024-02-01/stage_externals/install_gcc-12.3.0-skylake/gettext-0.22.3-nbi5w5/lib/libintl.so.8
[31328/31328][gotcha.c:225] - Setting library /gpfs/bbp.cscs.ch/ssd/apps/bsd/2024-02-01/stage_externals/install_gcc-12.3.0-skylake/gettext-0.22.3-nbi5w5/lib/libintl.so.8 GOT table from 0x7fffe84f7000 to +4096 to writeable
[31328/31328][gotcha.c:240] - Creating new library object for /lib64/libutil.so.1
[31328/31328][gotcha.c:225] - Setting library /lib64/libutil.so.1 GOT table from 0x7fffe84de000 to +4096 to writeable
[31328/31328][gotcha.c:240] - Creating new library object for /gpfs/bbp.cscs.ch/ssd/apps/bsd/2024-02-01/stage_applications/install_gcc-12.3.0-skylake/papi-6.0.0.1-vol6np/lib/libpfm.so.4
[31328/31328][gotcha.c:225] - Setting library /gpfs/bbp.cscs.ch/ssd/apps/bsd/2024-02-01/stage_applications/install_gcc-12.3.0-skylake/papi-6.0.0.1-vol6np/lib/libpfm.so.4 GOT table from 0x7fffe8278000 to +4096 to writeable
[31328/31328][gotcha.c:240] - Creating new library object for /gpfs/bbp.cscs.ch/ssd/apps/bsd/2024-02-01/stage_applications/install_gcc-12.3.0-skylake/elfutils-0.189-a46bi6/lib/libelf.so.1
[31328/31328][gotcha.c:225] - Setting library /gpfs/bbp.cscs.ch/ssd/apps/bsd/2024-02-01/stage_applications/install_gcc-12.3.0-skylake/elfutils-0.189-a46bi6/lib/libelf.so.1 GOT table from 0x7fffe8006000 to +4096 to writeable
[31328/31328][gotcha.c:240] - Creating new library object for /gpfs/bbp.cscs.ch/ssd/apps/bsd/2024-02-01/stage_applications/install_gcc-12.3.0-skylake/zlib-ng-2.1.4-z7jpg7/lib/libz.so.1
[31328/31328][gotcha.c:225] - Setting library /gpfs/bbp.cscs.ch/ssd/apps/bsd/2024-02-01/stage_applications/install_gcc-12.3.0-skylake/zlib-ng-2.1.4-z7jpg7/lib/libz.so.1 GOT table from 0x7fffe7feb000 to +4096 to writeable
[31328/31328][gotcha.c:240] - Creating new library object for /gpfs/bbp.cscs.ch/ssd/apps/bsd/2024-02-01/stage_applications/install_gcc-12.3.0-skylake/zstd-1.5.5-aivjoo/lib/libzstd.so.1
[31328/31328][gotcha.c:225] - Setting library /gpfs/bbp.cscs.ch/ssd/apps/bsd/2024-02-01/stage_applications/install_gcc-12.3.0-skylake/zstd-1.5.5-aivjoo/lib/libzstd.so.1 GOT table from 0x7fffe7fc6000 to +4096 to writeable
[31328/31328][gotcha.c:240] - Creating new library object for /gpfs/bbp.cscs.ch/ssd/apps/bsd/2024-02-01/stage_applications/install_gcc-12.3.0-skylake/xz-5.4.1-wwn5en/lib/liblzma.so.5
[31328/31328][gotcha.c:225] - Setting library /gpfs/bbp.cscs.ch/ssd/apps/bsd/2024-02-01/stage_applications/install_gcc-12.3.0-skylake/xz-5.4.1-wwn5en/lib/liblzma.so.5 GOT table from 0x7fffe7edd000 to +4096 to writeable
[31328/31328][gotcha.c:240] - Creating new library object for /gpfs/bbp.cscs.ch/ssd/apps/bsd/2024-02-01/stage_applications/install_gcc-12.3.0-skylake/bzip2-1.0.8-vdqwmg/lib/libbz2.so.1.0
[31328/31328][gotcha.c:225] - Setting library /gpfs/bbp.cscs.ch/ssd/apps/bsd/2024-02-01/stage_applications/install_gcc-12.3.0-skylake/bzip2-1.0.8-vdqwmg/lib/libbz2.so.1.0 GOT table from 0x7fffe7eaf000 to +4096 to writeable
[31328/31328][gotcha.c:240] - Creating new library object for /gpfs/bbp.cscs.ch/ssd/apps/bsd/2024-02-01/stage_externals/install_gcc-12.3.0-skylake/libiconv-1.17-nv4xh4/lib/libiconv.so.2
[31328/31328][gotcha.c:225] - Setting library /gpfs/bbp.cscs.ch/ssd/apps/bsd/2024-02-01/stage_externals/install_gcc-12.3.0-skylake/libiconv-1.17-nv4xh4/lib/libiconv.so.2 GOT table from 0x7fffe7e9c000 to +4096 to writeable
[31328/31328][gotcha.c:240] - Creating new library object for /gpfs/bbp.cscs.ch/ssd/apps/bsd/2024-02-01/stage_externals/install_gcc-12.3.0-skylake/python-3.11.6-bj4i6m/lib/python3.11/lib-dynload/zlib.cpython-311-x86_64-linux-gnu.so
[31328/31328][gotcha.c:225] - Setting library /gpfs/bbp.cscs.ch/ssd/apps/bsd/2024-02-01/stage_externals/install_gcc-12.3.0-skylake/python-3.11.6-bj4i6m/lib/python3.11/lib-dynload/zlib.cpython-311-x86_64-linux-gnu.so GOT table from 0x7fffe1418000 to +4096 to writeable
[31328/31328][gotcha.c:240] - Creating new library object for /gpfs/bbp.cscs.ch/ssd/apps/bsd/2024-02-01/stage_externals/install_gcc-12.3.0-skylake/python-3.11.6-bj4i6m/lib/python3.11/lib-dynload/_bz2.cpython-311-x86_64-linux-gnu.so
[31328/31328][gotcha.c:225] - Setting library /gpfs/bbp.cscs.ch/ssd/apps/bsd/2024-02-01/stage_externals/install_gcc-12.3.0-skylake/python-3.11.6-bj4i6m/lib/python3.11/lib-dynload/_bz2.cpython-311-x86_64-linux-gnu.so GOT table from 0x7fffed90b000 to +4096 to writeable
[31328/31328][gotcha.c:240] - Creating new library object for /gpfs/bbp.cscs.ch/ssd/apps/bsd/2024-02-01/stage_externals/install_gcc-12.3.0-skylake/python-3.11.6-bj4i6m/lib/python3.11/lib-dynload/_lzma.cpython-311-x86_64-linux-gnu.so
[31328/31328][gotcha.c:225] - Setting library /gpfs/bbp.cscs.ch/ssd/apps/bsd/2024-02-01/stage_externals/install_gcc-12.3.0-skylake/python-3.11.6-bj4i6m/lib/python3.11/lib-dynload/_lzma.cpython-311-x86_64-linux-gnu.so GOT table from 0x7fffe140b000 to +4096 to writeable
[31328/31328][gotcha.c:240] - Creating new library object for /gpfs/bbp.cscs.ch/ssd/apps/bsd/2024-02-01/stage_externals/install_gcc-12.3.0-skylake/python-3.11.6-bj4i6m/lib/python3.11/lib-dynload/fcntl.cpython-311-x86_64-linux-gnu.so
[31328/31328][gotcha.c:225] - Setting library /gpfs/bbp.cscs.ch/ssd/apps/bsd/2024-02-01/stage_externals/install_gcc-12.3.0-skylake/python-3.11.6-bj4i6m/lib/python3.11/lib-dynload/fcntl.cpython-311-x86_64-linux-gnu.so GOT table from 0x7fffe13ff000 to +4096 to writeable
[31328/31328][gotcha.c:240] - Creating new library object for /gpfs/bbp.cscs.ch/ssd/apps/bsd/2024-02-01/stage_externals/install_gcc-12.3.0-skylake/python-3.11.6-bj4i6m/lib/python3.11/lib-dynload/_posixsubprocess.cpython-311-x86_64-linux-gnu.so
[31328/31328][gotcha.c:225] - Setting library /gpfs/bbp.cscs.ch/ssd/apps/bsd/2024-02-01/stage_externals/install_gcc-12.3.0-skylake/python-3.11.6-bj4i6m/lib/python3.11/lib-dynload/_posixsubprocess.cpython-311-x86_64-linux-gnu.so GOT table from 0x7fffe13f7000 to +4096 to writeable
[31328/31328][gotcha.c:240] - Creating new library object for /gpfs/bbp.cscs.ch/ssd/apps/bsd/2024-02-01/stage_externals/install_gcc-12.3.0-skylake/python-3.11.6-bj4i6m/lib/python3.11/lib-dynload/select.cpython-311-x86_64-linux-gnu.so
[31328/31328][gotcha.c:225] - Setting library /gpfs/bbp.cscs.ch/ssd/apps/bsd/2024-02-01/stage_externals/install_gcc-12.3.0-skylake/python-3.11.6-bj4i6m/lib/python3.11/lib-dynload/select.cpython-311-x86_64-linux-gnu.so GOT table from 0x7fffe13ef000 to +4096 to wri[31328/31328][gotcha.c:71] - Checking ELF hash for MPI_Init_thread in /gpfs/bbp.cscs.ch/ssd/apps/bsd/2024-02-01/stage_applications/install_gcc-12.3.0-skylake/ncurses-6.4-6kdbhv/lib/libform.so.6
[31328/31328][gotcha.c:76] - MPI_Init_thread not found in /gpfs/bbp.cscs.ch/ssd/apps/bsd/2024-02-01/stage_applications/install_gcc-12.3.0-skylake/ncurses-6.4-6kdbhv/lib/libform.so.6
[31328/31328][gotcha.c:57] - Searching for exported symbols in /gpfs/bbp.cscs.ch/ssd/apps/bsd/2024-02-01/stage_applications/install_gcc-12.3.0-skylake/caliper-2.10.0-btrc4s/lib64/libcaliper.so.2
[31328/31328][gotcha.c:66] - Checking GNU hash for MPI_Init_thread in /gpfs/bbp.cscs.ch/ssd/apps/bsd/2024-02-01/stage_applications/install_gcc-12.3.0-skylake/caliper-2.10.0-btrc4s/lib64/libcaliper.so.2
[31328/31328][gotcha.c:71] - Checking ELF hash for MPI_Init_thread in /gpfs/bbp.cscs.ch/ssd/apps/bsd/2024-02-01/stage_applications/install_gcc-12.3.0-skylake/caliper-2.10.0-btrc4s/lib64/libcaliper.so.2
[31328/31328][gotcha.c:76] - MPI_Init_thread not found in /gpfs/bbp.cscs.ch/ssd/apps/bsd/2024-02-01/stage_applications/install_gcc-12.3.0-skylake/caliper-2.10.0-btrc4s/lib64/libcaliper.so.2
[31328/31328][gotcha.c:57] - Searching for exported symbols in /gpfs/bbp.cscs.ch/ssd/apps/bsd/2024-02-01/stage_externals/install_gcc-12.3.0-skylake/python-3.11.6-bj4i6m/lib/libpython3.11.so.1.0
[31328/31328][gotcha.c:66] - Checking GNU hash for MPI_Init_thread in /gpfs/bbp.cscs.ch/ssd/apps/bsd/2024-02-01/stage_externals/install_gcc-12.3.0-skylake/python-3.11.6-bj4i6m/lib/libpython3.11.so.1.0
[31328/31328][gotcha.c:71] - Checking ELF hash for MPI_Init_thread in /gpfs/bbp.cscs.ch/ssd/apps/bsd/2024-02-01/stage_externals/install_gcc-12.3.0-skylake/python-3.11.6-bj4i6m/lib/libpython3.11.so.1.0
[31328/31328][gotcha.c:76] - MPI_Init_thread not found in /gpfs/bbp.cscs.ch/ssd/apps/bsd/2024-02-01/stage_externals/install_gcc-12.3.0-skylake/python-3.11.6-bj4i6m/lib/libpython3.11.so.1.0
[31328/31328][gotcha.c:57] - Searching for exported symbols in /lib64/libpthread.so.0
[31328/31328][gotcha.c:66] - Checking GNU hash for MPI_Init_thread in /lib64/libpthread.so.0
[31328/31328][gotcha.c:71] - Checking ELF hash for MPI_Init_thread in /lib64/libpthread.so.0
[31328/31328][gotcha.c:76] - MPI_Init_thread not found in /lib64/libpthread.so.0
[31328/31328][gotcha.c:57] - Searching for exported symbols in /gpfs/bbp.cscs.ch/ssd/apps/bsd/2024-02-01/stage_externals/install_gcc-12.3.0-skylake/hpe-mpi-2.27.p1.hmpt-6rr256/lib/libmpi.so
[31328/31328][gotcha.c:66] - Checking GNU hash for MPI_Init_thread in /gpfs/bbp.cscs.ch/ssd/apps/bsd/2024-02-01/stage_externals/install_gcc-12.3.0-skylake/hpe-mpi-2.27.p1.hmpt-6rr256/lib/libmpi.so
[31328/31328][gotcha.c:85] - Symbol MPI_Init_thread found in /gpfs/bbp.cscs.ch/ssd/apps/bsd/2024-02-01/stage_externals/install_gcc-12.3.0-skylake/hpe-mpi-2.27.p1.hmpt-6rr256/lib/libmpi.so at 0x7fffeab6b0d0
[31328/31328][gotcha.c:46] - Updating binding address pointer at 0x7d7b08 to 0x7fffeab6b0d0
[31328/31328][gotcha.c:334] - Symbol MPI_Init_thread needs binding from application
[31328/31328][gotcha.c:149] - gotcha_rewrite_wrapper_orders for binding MPI_Finalize in tool caliper/mpi of priority -1
[31328/31328][gotcha.c:156] - Adding new entry for MPI_Finalize to hash table
[31328/31328][gotcha.c:324] - Symbol MPI_Finalize needs lookup operation
[31328/31328][gotcha.c:98] - Looking up exported symbols for MPI_Finalize
[31328/31328][gotcha.c:57] - Searching for exported symbols in [EMPTY]
[31328/31328][gotcha.c:66] - Checking GNU hash for MPI_Finalize in [EMPTY]
[31328/31328][gotcha.c:71] - Checking ELF hash for MPI_Finalize in [EMPTY]
[31328/31328][gotcha.c:76] - MPI_Finalize not found in [EMPTY]
[31328/31328][gotcha.c:53] - Skipping VDSO library at 0x7fffee402000 with name [EMPTY]
[31328/31328][gotcha.c:57] - Searching for exported symbols in /gpfs/bbp.cscs.ch/home/kumbhar/tmp/caliper/build_caliper_210/lib/libnrniv.so
[31328/31328][gotcha.c:66] - Checking GNU hash for MPI_Finalize in /gpfs/bbp.cscs.ch/home/kumbhar/tmp/caliper/build_caliper_210/lib/libnrniv.so
[31328/31328][gotcha.c:71] - Checking ELF hash for MPI_Finalize in /gpfs/bbp.cscs.ch/home/kumbhar/tmp/caliper/build_caliper_210/lib/libnrniv.so
[31328/31328][gotcha.c:80] - Symbol MPI_Finalize found but not exported in /gpfs/bbp.cscs.ch/home/kumbhar/tmp/caliper/build_caliper_210/lib/libnrniv.so
[31328/31328][gotcha.c:57] - Searching for exported symbols in /lib64/libdl.so.2
[31328/31328][gotcha.c:66] - Checking GNU hash for MPI_Finalize in /lib64/libdl.so.2
[31328/31328][gotcha.c:71] - Checking ELF hash for MPI_Finalize in /lib64/libdl.so.2
[31328/31328][gotcha.c:76] - MPI_Finalize not found in /lib64/libdl.so.2
[31328/31328][gotcha.c:57] - Searching for exported symbols in /gpfs/bbp.cscs.ch/ssd/apps/bsd/2024-02-01/stage_applications/install_gcc-12.3.0-skylake/readline-8.2-pdugg7/lib/libreadline.so.8
[31328/31328][gotcha.c:66] - Checking GNU hash for MPI_Finalize in /gpfs/bbp.cscs.ch/ssd/apps/bsd/2024-02-01/stage_applications/install_gcc-12.3.0-skylake/readline-8.2-pdugg7/lib/libreadline.so.8
[31328/31328][gotcha.c:71] - Checking ELF hash for MPI_Finalize in /gpfs/bbp.cscs.ch/ssd/apps/bsd/2024-02-01/stage_applications/install_gcc-12.3.0-skylake/readline-8.2-pdugg7/lib/libreadline.so.8
[31328/31328][gotcha.c:76] - MPI_Finalize not found in /gpfs/bbp.cscs.ch/ssd/apps/bsd/2024-02-01/stage_applications/install_gcc-12.3.0-skylake/readline-8.2-pdugg7/lib/libreadline.so.8
[31328/31328][gotcha.c:57] - Searching for exported symbols in /gpfs/bbp.cscs.ch/ssd/apps/bsd/2024-02-01/stage_applications/install_gcc-12.3.0-skylake/ncurses-6.4-6kdbhv/lib/libncurses.so.6
[31328/31328][gotcha.c:66] - Checking GNU hash for MPI_Finalize in /gpfs/bbp.cscs.ch/ssd/apps/bsd/2024-02-01/stage_applications/install_gcc-12.3.0-skylake/ncurses-6.4-6kdbhv/lib/libncurses.so.6
[31328/31328][gotcha.c:71] - Checking ELF hash for MPI_Finalize in /gpfs/bbp.cscs.ch/ssd/apps/bsd/2024-02-01/stage_applications/install_gcc-12.3.0-skylake/ncurses-6.4-6kdbhv/lib/libncurses.so.6
[31328/31328][gotcha.c:76] - MPI_Finalize not found in /gpfs/bbp.cscs.ch/ssd/apps/bsd/2024-02-01/stage_applications/install_gcc-12.3.0-skylake/ncurses-6.4-6kdbhv/lib/libncurses.so.6
[31328/31328][gotcha.c:57] - Searching for exported symbols in /gpfs/bbp.cscs.ch/ssd/apps/bsd/2024-02-01/stage_applications/install_gcc-12.3.0-skylake/ncurses-6.4-6kdbhv/lib/libform.so.6
[31328/31328][gotcha.c:66] - Checking GNU hash for MPI_Finalize in /gpfs/bbp.cscs.ch/ssd/apps/bsd/2024-02-01/stage_applications/install_gcc-12.3.0-skylake/ncurses-6.4-6kdbhv/lib/libform.so.6
[31328/31328][gotcha.c:71] - Checking ELF hash for MPI_Finalize in /gpfs/bbp.cscs.ch/ssd/apps/bsd/2024-02-01/stage_applications/install_gcc-12.3.0-skylake/ncurses-6.4-6kdbhv/lib/libform.so.6
[31328/31328][gotcha.c:76] - MPI_Finalize not found in /gpfs/bbp.cscs.ch/ssd/apps/bsd/2024-02-01/stage_applications/install_gcc-12.3.0-skylake/ncurses-6.4-6kdbhv/lib/libform.so.6
[31328/31328][gotcha.c:57] - Searching for exported symbols in /gpfs/bbp.cscs.ch/ssd/apps/bsd/2024-02-01/stage_applications/install_gcc-12.3.0-skylake/caliper-2.10.0-btrc4s/lib64/libcaliper.so.2
[31328/31328][gotcha.c:66] - Checking GNU hash for MPI_Finalize in /gpfs/bbp.cscs.ch/ssd/apps/bsd/2024-02-01/stage_applications/install_gcc-12.3.0-skylake/caliper-2.10.0-btrc4s/lib64/libcaliper.so.2
[31328/31328][gotcha.c:71] - Checking ELF hash for MPI_Finalize in /gpfs/bbp.cscs.ch/ssd/apps/bsd/2024-02-01/stage_applications/install_gcc-12.3.0-skylake/caliper-2.10.0-btrc4s/lib64/libcaliper.so.2
[31328/31328][gotcha.c:76] - MPI_Finalize not found in /gpfs/bbp.cscs.ch/ssd/apps/bsd/2024-02-01/stage_applications/install_gcc-12.3.0-skylake/caliper-2.10.0-btrc4s/lib64/libcaliper.so.2
[31328/31328][gotcha.c:57] - Searching for exported symbols in /gpfs/bbp.cscs.ch/ssd/apps/bsd/2024-02-01/stage_externals/install_gcc-12.3.0-skylake/python-3.11.6-bj4i6m/lib/libpython3.11.so.1.0
[31328/31328][gotcha.c:66] - Checking GNU hash for MPI_Finalize in /gpfs/bbp.cscs.ch/ssd/apps/bsd/2024-02-01/stage_externals/install_gcc-12.3.0-skylake/python-3.11.6-bj4i6m/lib/libpython3.11.so.1.0
[31328/31328][gotcha.c:71] - Checking ELF hash for MPI_Finalize in /gpfs/bbp.cscs.ch/ssd/apps/bsd/2024-02-01/stage_externals/install_gcc-12.3.0-skylake/python-3.11.6-bj4i6m/lib/libpython3.11.so.1.0
[31328/31328][gotcha.c:76] - MPI_Finalize not found in /gpfs/bbp.cscs.ch/ssd/apps/bsd/2024-02-01/stage_externals/install_gcc-12.3.0-skylake/python-3.11.6-bj4i6m/lib/libpython3.11.so.1.0
[31328/31328][gotcha.c:57] - Searching for exported symbols in /lib64/libpthread.so.0
[31328/31328][gotcha.c:66] - Checking GNU hash for MPI_Finalize in /lib64/libpthread.so.0
[31328/31328][gotcha.c:71] - Checking ELF hash for MPI_Finalize in /lib64/libpthread.so.0
[31328/31328][gotcha.c:76] - MPI_Finalize not found in /lib64/libpthread.so.0
[31328/31328][gotcha.c:57] - Searching for exported symbols in /gpfs/bbp.cscs.ch/ssd/apps/bsd/2024-02-01/stage_externals/install_gcc-12.3.0-skylake/hpe-mpi-2.27.p1.hmpt-6rr256/lib/libmpi.so
[31328/31328][gotcha.c:66] - Checking GNU hash for MPI_Finalize in /gpfs/bbp.cscs.ch/ssd/apps/bsd/2024-02-01/stage_externals/install_gcc-12.3.0-skylake/hpe-mpi-2.27.p1.hmpt-6rr256/lib/libmpi.so
[31328/31328][gotcha.c:85] - Symbol MPI_Finalize found in /gpfs/bbp.cscs.ch/ssd/apps/bsd/2024-02-01/stage_externals/install_gcc-12.3.0-skylake/hpe-mpi-2.27.p1.hmpt-6rr256/lib/libmpi.so at 0x7fffeab39f90
[31328/31328][gotcha.c:46] - Updating binding address pointer at 0x7d7b28 to 0x7fffeab39f90
[31328/31328][gotcha.c:334] - Symbol MPI_Finalize needs binding from application
[31328/31328][gotcha.c:268] - Searching all callsites for 3 bindings
[31328/31328][gotcha.c:200] - Remapped call to MPI_Init_thread at 0x7fffed67bdd0 in /gpfs/bbp.cscs.ch/home/kumbhar/tmp/caliper/build_caliper_210/lib/libnrniv.so to wrapper at 0x0x7fffeb735980
[31328/31328][gotcha.c:200] - Remapped call to MPI_Finalize at 0x7fffed684430 in /gpfs/bbp.cscs.ch/home/kumbhar/tmp/caliper/build_caliper_210/lib/libnrniv.so to wrapper at 0x0x7fffeb734ec0
[31328/31328][gotcha.c:352] - Gotcha wrap completed successfully
[31328/31328][gotcha.c:282] - User called gotcha_wrap for tool caliper/mpi with 0 bindings
[31328/31328][gotcha.c:290] - Initializing 0 user binding entries to NULL
[31328/31328][gotcha.c:306] - Moved current_generation to 3 in gotcha_wrap
[31328/31328][gotcha.c:308] - Creating internal binding data structures and adding binding to tool
[31328/31328][tool.c:135] - Created new binding table of size 0 for tool caliper/mpi
[31328/31328][gotcha.c:315] - Processing 0 bindings
[31328/31328][gotcha.c:352] - Gotcha wrap completed successfully
[31328/31328][gotcha_dl.c:143] - User called dlsym(0xffffffffffffffff, munmap)
[31328/31328][gotcha.c:57] - Searching for exported symbols in /gpfs/bbp.cscs.ch/ssd/apps/bsd/2024-02-01/stage_compilers/install_gcc-4.8.5-haswell/gcc-12.3.0-lvcks3/lib/gcc/x86_64-pc-linux-gnu/12.3.0/../../../../lib64/libatomic.so.1
[31328/31328][gotcha.c:66] - Checking GNU hash for munmap in /gpfs/bbp.cscs.ch/ssd/apps/bsd/2024-02-01/stage_compilers/install_gcc-4.8.5-haswell/gcc-12.3.0-lvcks3/lib/gcc/x86_64-pc-linux-gnu/12.3.0/../../../../lib64/libatomic.so.1
[31328/31328][gotcha.c:71] - Checking ELF hash for munmap in /gpfs/bbp.cscs.ch/ssd/apps/bsd/2024-02-01/stage_compilers/install_gcc-4.8.5-haswell/gcc-12.3.0-lvcks3/lib/gcc/x86_64-pc-linux-gnu/12.3.0/../../../../lib64/libatomic.so.1
[31328/31328][gotcha.c:76] - munmap not found in /gpfs/bbp.cscs.ch/ssd/apps/bsd/2024-02-01/stage_compilers/install_gcc-4.8.5-haswell/gcc-12.3.0-lvcks3/lib/gcc/x86_64-pc-linux-gnu/12.3.0/../../../../lib64/libatomic.so.1
[31328/31328][gotcha_dl.c:90] - Symbol munmap not found in the library /gpfs/bbp.cscs.ch/ssd/apps/bsd/2024-02-01/stage_compilers/install_gcc-4.8.5-haswell/gcc-12.3.0-lvcks3/lib/gcc/x86_64-pc-linux-gnu/12.3.0/../../../../lib64/libatomic.so.1
[31328/31328][gotcha.c:57] - Searching for exported symbols in /gpfs/bbp.cscs.ch/ssd/apps/bsd/2024-02-01/stage_externals/install_gcc-12.3.0-skylake/nvhpc-23.1-vficpy/Linux_x86_64/23.1/compilers/lib/libnvhpcatm.so
[31328/31328][gotcha.c:66] - Checking GNU hash for munmap in /gpfs/bbp.cscs.ch/ssd/apps/bsd/2024-02-01/stage_externals/install_gcc-12.3.0-skylake/nvhpc-23.1-vficpy/Linux_x86_64/23.1/compilers/lib/libnvhpcatm.so
[31328/31328][gotcha.c:76] - munmap not found in /gpfs/bbp.cscs.ch/ssd/apps/bsd/2024-02-01/stage_externals/install_gcc-12.3.0-skylake/nvhpc-23.1-vficpy/Linux_x86_64/23.1/compilers/lib/libnvhpcatm.so
[31328/31328][gotcha_dl.c:90] - Symbol munmap not found in the library /gpfs/bbp.cscs.ch/ssd/apps/bsd/2024-02-01/stage_externals/install_gcc-12.3.0-skylake/nvhpc-23.1-vficpy/Linux_x86_64/23.1/compilers/lib/libnvhpcatm.so
[31328/31328][gotcha.c:57] - Searching for exported symbols in /gpfs/bbp.cscs.ch/ssd/apps/bsd/2024-02-01/stage_compilers/install_gcc-4.8.5-haswell/gcc-12.3.0-lvcks3/lib/gcc/x86_64-pc-linux-gnu/12.3.0/../../../../lib64/libstdc++.so.6
[31328/31328][gotcha.c:66] - Checking GNU hash for munmap in /gpfs/bbp.cscs.ch/ssd/apps/bsd/2024-02-01/stage_compilers/install_gcc-4.8.5-haswell/gcc-12.3.0-lvcks3/lib/gcc/x86_64-pc-linux-gnu/12.3.0/../../../../lib64/libstdc++.so.6
[31328/31328][gotcha.c:71] - Checking ELF hash for munmap in /gpfs/bbp.cscs.ch/ssd/apps/bsd/2024-02-01/stage_compilers/install_gcc-4.8.5-haswell/gcc-12.3.0-lvcks3/lib/gcc/x86_64-pc-linux-gnu/12.3.0/../../../../lib64/libstdc++.so.6
[31328/31328][gotcha.c:76] - munmap not found in /gpfs/bbp.cscs.ch/ssd/apps/bsd/2024-02-01/stage_compilers/install_gcc-4.8.5-haswell/gcc-12.3.0-lvcks3/lib/gcc/x86_64-pc-linux-gnu/12.3.0/../../../../lib64/libstdc++.so.6
[31328/31328][gotcha_dl.c:90] - Symbol munmap not found in the library /gpfs/bbp.cscs.ch/ssd/apps/bsd/2024-02-01/stage_compilers/install_gcc-4.8.5-haswell/gcc-12.3.0-lvcks3/lib/gcc/x86_64-pc-linux-gnu/12.3.0/../../../../lib64/libstdc++.so.6
[31328/31328][gotcha.c:57] - Searching for exported symbols in /gpfs/bbp.cscs.ch/ssd/apps/bsd/2024-02-01/stage_externals/install_gcc-12.3.0-skylake/nvhpc-23.1-vficpy/Linux_x86_64/23.1/compilers/lib/libnvomp.so
[31328/31328][gotcha.c:66] - Checking GNU hash for munmap in /gpfs/bbp.cscs.ch/ssd/apps/bsd/2024-02-01/stage_externals/install_gcc-12.3.0-skylake/nvhpc-23.1-vficpy/Linux_x86_64/23.1/compilers/lib/libnvomp.so
[31328/31328][gotcha.c:76] - munmap not found in /gpfs/bbp.cscs.ch/ssd/apps/bsd/2024-02-01/stage_externals/install_gcc-12.3.0-skylake/nvhpc-23.1-vficpy/Linux_x86_64/23.1/compilers/lib/libnvomp.so
[31328/31328][gotcha_dl.c:90] - Symbol munmap not found in the library /gpfs/bbp.cscs.ch/ssd/apps/bsd/2024-02-01/stage_externals/install_gcc-12.3.0-skylake/nvhpc-23.1-vficpy/Linux_x86_64/23.1/compilers/lib/libnvomp.so
[31328/31328][gotcha.c:57] - Searching for exported symbols in /gpfs/bbp.cscs.ch/ssd/apps/bsd/2024-02-01/stage_externals/install_gcc-12.3.0-skylake/nvhpc-23.1-vficpy/Linux_x86_64/23.1/compilers/lib/libnvcpumath-avx2.so
[31328/31328][gotcha.c:66] - Checking GNU hash for munmap in /gpfs/bbp.cscs.ch/ssd/apps/bsd/2024-02-01/stage_externals/install_gcc-12.3.0-skylake/nvhpc-23.1-vficpy/Linux_x86_64/23.1/compilers/lib/libnvcpumath-avx2.so
[31328/31328][gotcha.c:76] - munmap not found in /gpfs/bbp.cscs.ch/ssd/apps/bsd/2024-02-01/stage_externals/install_gcc-12.3.0-skylake/nvhpc-23.1-vficpy/Linux_x86_64/23.1/compilers/lib/libnvcpumath-avx2.so
[31328/31328][gotcha_dl.c:90] - Symbol munmap not found in the library /gpfs/bbp.cscs.ch/ssd/apps/bsd/2024-02-01/stage_externals/install_gcc-12.3.0-skylake/nvhpc-23.1-vficpy/Linux_x86_64/23.1/compilers/lib/libnvcpumath-avx2.so
[31328/31328][gotcha.c:57] - Searching for exported symbols in /gpfs/bbp.cscs.ch/ssd/apps/bsd/2024-02-01/stage_externals/install_gcc-12.3.0-skylake/nvhpc-23.1-vficpy/Linux_x86_64/23.1/compilers/lib/libnvc.so
[31328/31328][gotcha.c:66] - Checking GNU hash for munmap in /gpfs/bbp.cscs.ch/ssd/apps/bsd/2024-02-01/stage_externals/install_gcc-12.3.0-skylake/nvhpc-23.1-vficpy/Linux_x86_64/23.1/compilers/lib/libnvc.so
[31328/31328][gotcha.c:76] - munmap not found in /gpfs/bbp.cscs.ch/ssd/apps/bsd/2024-02-01/stage_externals/install_gcc-12.3.0-skylake/nvhpc-23.1-vficpy/Linux_x86_64/23.1/compilers/lib/libnvc.so
[31328/31328][gotcha_dl.c:90] - Symbol munmap not found in the library /gpfs/bbp.cscs.ch/ssd/apps/bsd/2024-02-01/stage_externals/install_gcc-12.3.0-skylake/nvhpc-23.1-vficpy/Linux_x86_64/23.1/compilers/lib/libnvc.so
[31328/31328][gotcha.c:57] - Searching for exported symbols in /lib64/libc.so.6
[31328/31328][gotcha.c:66] - Checking GNU hash for munmap in /lib64/libc.so.6
[31328/31328][gotcha.c:85] - Symbol munmap found in /lib64/libc.so.6 at 0x7fffe9011050
Segmentation fault
daboehme commented 8 months ago

Hi @pramodk , thanks again for helping us debug this issue. Is there any chance you could give us a small reproducer? I imagine the whole NEURON code is quite complex, but I'm curious how you are using Python in there. Maybe it's possible to isolate the Python initialization and finalization into a small example and see if that also triggers the problem?

pramodk commented 8 months ago

I have to see if it’s possible/easy to extract a small reproducer.

A certainly doable/straightforward thing would be to provide a docker container as a reproducer. Would that help?

daboehme commented 8 months ago

We'd need something that makes it easy to try out a possible fix. Not sure about the docker container, I think it's a bit tricky to run those on our HPC systems.

daboehme commented 8 months ago

Hi @pramodk, looks like a docker container as a reproducer would help actually. Would be great if you could provide one. Thanks!

pramodk commented 8 months ago

Sorry for delay. We will try to create one and share with you.

pramodk commented 7 months ago

I think we can close this issue. @alkino provided details and helped to fix LLNL/GOTCHA/issues/142.