search

dmtcp / dmtcp

DMTCP: Distributed MultiThreaded CheckPointing
http://dmtcp.sourceforge.net/
Other
381 stars 134 forks source link

Hang when running DMTCP in docker and importing scipy.spatial #1114

Open mikeawalker opened 1 month ago

mikeawalker commented 1 month ago

I'm running into a weird-ish issue when using DMTCP in Docker. Overall, it works great and has been extremely useful to our team

However, we have a few corner cases where things are locking up.

Consider the following python script which I run in the container via dmtcp_launch --new-coordinator python3 counter.py 

# counter.py 
import time 
print("A")
from scipy import spatial # remove me and all is good ! ??? 
print("B")

def main( ):
    k = 0

    print("started")
    while( True ):
        print( f"Step {k}" )
        time.sleep( 2 )  
        k += 1   

if __name__ == "__main__":
    main()

If I run this script using dmtcp_launch in the following container

# This dockerfile will hang 
FROM python:3.12-slim
ARG DEBIAN_FRONTEND=noninteractive

ARG TZ=Etc/UTC

RUN apt update && \ 
    apt install -y git gcc g++ make gdb

WORKDIR /dmtcp

RUN git clone --depth 1 --branch 3.0.0 https://github.com/dmtcp/dmtcp.git
RUN cd dmtcp && \
    ./configure --prefix=/usr --enable-unique-checkpoint-filenames && \
    make && \ 
    make install
RUN pip3 install scipy 
ENTRYPOINT ["/bin/bash"]

I get a hang that looks like this 

docker run -e ASSET=RADIO0 -e COMPONENT=radio-0 -v ./unittest:/config/ -it test/lifecycle
root@3e41ab678ed8:/dmtcp# dmtcp_launch --new-coordinator python3 counter.py 
A

If I change the docker container to start with a clean version of debian 12 and bring in scipy as part of an apt command I get 

FROM debian:12
ARG DEBIAN_FRONTEND=noninteractive

ARG TZ=Etc/UTC

RUN apt update && \ 
    apt install -y git gcc g++ make gdb python3 python3-scipy

WORKDIR /dmtcp

RUN git clone --depth 1 --branch 3.0.0 https://github.com/dmtcp/dmtcp.git
RUN cd dmtcp && \
    ./configure --prefix=/usr --enable-unique-checkpoint-filenames && \
    make && \ 
    make install
ENTRYPOINT ["/bin/bash"]

Results in: 

root@15fd9fa67151:/dmtcp# dmtcp_launch --new-coordinator python3 counter.py 
A
B
started
Step 0
Step 1

counter.py will run in a docker container by itself regardless of which way I make my container and bring in scipy.

I could use some guidance here. This definitely seems like an issue where I have mixed dependencies on a python package that is pre-compiled, but it definitely has something to do with running it under dmtcp.

mikeawalker commented 1 month ago

I also built and ran this with gdb so I could provide the stack traces - (this is going to be long) 

root@57ec358b1541:/dmtcp# gdb --args dmtcp_launch --new-coordinator python3 counter.py
GNU gdb (Debian 13.1-3) 13.1
Copyright (C) 2023 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Type "show copying" and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<https://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
    <http://www.gnu.org/software/gdb/documentation/>.

For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from dmtcp_launch...
(gdb) r
Starting program: /usr/bin/dmtcp_launch --new-coordinator python3 counter.py
warning: Error disabling address space randomization: Operation not permitted
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
[Detaching after fork from child process 28]
[Detaching after fork from child process 30]
[Detaching after vfork from child process 31]
process 25 is executing new program: /usr/local/bin/python3.12
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
[New Thread 0x7f9b8230f6c0 (LWP 33)]
A
[New Thread 0x7f9b7e3ff6c0 (LWP 34)]
[New Thread 0x7f9b7dbfe6c0 (LWP 35)]
[New Thread 0x7f9b7b3fd6c0 (LWP 36)]
[New Thread 0x7f9b76bfc6c0 (LWP 37)]
[New Thread 0x7f9b743fb6c0 (LWP 38)]
[New Thread 0x7f9b71bfa6c0 (LWP 39)]
[New Thread 0x7f9b6f3f96c0 (LWP 40)]
[New Thread 0x7f9b6cbf86c0 (LWP 41)]
[New Thread 0x7f9b6a3f76c0 (LWP 42)]
[New Thread 0x7f9b67bf66c0 (LWP 43)]
[New Thread 0x7f9b653f56c0 (LWP 44)]
[New Thread 0x7f9b62bf46c0 (LWP 45)]
[New Thread 0x7f9b623f36c0 (LWP 46)]
[New Thread 0x7f9b5dbf26c0 (LWP 47)]
[New Thread 0x7f9b5d3f16c0 (LWP 48)]
^C
Thread 3 "python3" received signal SIGINT, Interrupt.
[Switching to Thread 0x7f9b7e3ff6c0 (LWP 34)]
syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38
38      ../sysdeps/unix/sysv/linux/x86_64/syscall.S: No such file or directory.
(gdb) thread apply all bt

Thread 17 (Thread 0x7f9b5d3f16c0 (LWP 48) "python3"):
#0  syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38
#1  0x00007f9b850c4441 in _real_syscall (sys_num=202) at pid/pid_syscallsreal.c:346
#2  0x00007f9b850c1a78 in syscall (sys_num=202) at pid/pid_miscwrappers.cpp:507
#3  0x00007f9b851799e0 in _real_syscall (sys_num=202) at syscallsreal.c:872
#4  0x00007f9b85132794 in syscall (sys_num=202) at miscwrappers.cpp:611
#5  0x00007f9b8514d37a in futex (uaddr=0x7f9b851aa078 <_wrapperExecutionLock+24>, futex_op=0, val=2, timeout=0x0, uaddr2=0x0, val3=0) at ../include/futex.h:14
#6  0x00007f9b8514d3b4 in futex_wait (uaddr=0x7f9b851aa078 <_wrapperExecutionLock+24>, old_val=2) at ../include/futex.h:21
#7  0x00007f9b8514d48f in DmtcpMutexLock (mutex=0x7f9b851aa078 <_wrapperExecutionLock+24>) at mutex.cpp:59
#8  0x00007f9b85159292 in DmtcpRWLockRdLock (rwlock=0x7f9b851aa060 <_wrapperExecutionLock>) at rwlock.cpp:50
#9  0x00007f9b8513eb78 in dmtcp::ThreadSync::wrapperExecutionLockLock () at threadsync.cpp:182
#10 0x00007f9b85125d24 in dmtcp::WrapperLock::WrapperLock (this=0x7f9b5d3f0cd0, _exclusiveLock=false) at /dmtcp/dmtcp/src/threadsync.h:62
#11 0x00007f9b8513fa31 in thread_start (arg=0x7f9b7ecd4008) at threadwrappers.cpp:82
#12 0x00007f9b84920134 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442
#13 0x00007f9b8499fa40 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:100

Thread 16 (Thread 0x7f9b5dbf26c0 (LWP 47) "python3"):
#0  syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38
#1  0x00007f9b850c4441 in _real_syscall (sys_num=202) at pid/pid_syscallsreal.c:346
#2  0x00007f9b850c1a78 in syscall (sys_num=202) at pid/pid_miscwrappers.cpp:507
#3  0x00007f9b851799e0 in _real_syscall (sys_num=202) at syscallsreal.c:872
--Type <RET> for more, q to quit, c to continue without paging--c
#4  0x00007f9b85132794 in syscall (sys_num=202) at miscwrappers.cpp:611
#5  0x00007f9b8514d37a in futex (uaddr=0x7f9b851aa078 <_wrapperExecutionLock+24>, futex_op=0, val=2, timeout=0x0, uaddr2=0x0, val3=0) at ../include/futex.h:14
#6  0x00007f9b8514d3b4 in futex_wait (uaddr=0x7f9b851aa078 <_wrapperExecutionLock+24>, old_val=2) at ../include/futex.h:21
#7  0x00007f9b8514d48f in DmtcpMutexLock (mutex=0x7f9b851aa078 <_wrapperExecutionLock+24>) at mutex.cpp:59
#8  0x00007f9b85159292 in DmtcpRWLockRdLock (rwlock=0x7f9b851aa060 <_wrapperExecutionLock>) at rwlock.cpp:50
#9  0x00007f9b8513eb78 in dmtcp::ThreadSync::wrapperExecutionLockLock () at threadsync.cpp:182
#10 0x00007f9b85125d24 in dmtcp::WrapperLock::WrapperLock (this=0x7f9b5dbf1cd0, _exclusiveLock=false) at /dmtcp/dmtcp/src/threadsync.h:62
#11 0x00007f9b8513fa31 in thread_start (arg=0x7f9b7ecd3008) at threadwrappers.cpp:82
#12 0x00007f9b84920134 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442
#13 0x00007f9b8499fa40 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:100

Thread 15 (Thread 0x7f9b623f36c0 (LWP 46) "python3"):
#0  syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38
#1  0x00007f9b850c4441 in _real_syscall (sys_num=202) at pid/pid_syscallsreal.c:346
#2  0x00007f9b850c1a78 in syscall (sys_num=202) at pid/pid_miscwrappers.cpp:507
#3  0x00007f9b851799e0 in _real_syscall (sys_num=202) at syscallsreal.c:872
#4  0x00007f9b85132794 in syscall (sys_num=202) at miscwrappers.cpp:611
#5  0x00007f9b8514d37a in futex (uaddr=0x7f9b851aa078 <_wrapperExecutionLock+24>, futex_op=0, val=2, timeout=0x0, uaddr2=0x0, val3=0) at ../include/futex.h:14
#6  0x00007f9b8514d3b4 in futex_wait (uaddr=0x7f9b851aa078 <_wrapperExecutionLock+24>, old_val=2) at ../include/futex.h:21
#7  0x00007f9b8514d48f in DmtcpMutexLock (mutex=0x7f9b851aa078 <_wrapperExecutionLock+24>) at mutex.cpp:59
#8  0x00007f9b85159292 in DmtcpRWLockRdLock (rwlock=0x7f9b851aa060 <_wrapperExecutionLock>) at rwlock.cpp:50
#9  0x00007f9b8513eb78 in dmtcp::ThreadSync::wrapperExecutionLockLock () at threadsync.cpp:182
#10 0x00007f9b85125d24 in dmtcp::WrapperLock::WrapperLock (this=0x7f9b623f2cd0, _exclusiveLock=false) at /dmtcp/dmtcp/src/threadsync.h:62
#11 0x00007f9b8513fa31 in thread_start (arg=0x7f9b7ecd2008) at threadwrappers.cpp:82
#12 0x00007f9b84920134 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442
#13 0x00007f9b8499fa40 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:100

Thread 14 (Thread 0x7f9b62bf46c0 (LWP 45) "python3"):
#0  syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38
#1  0x00007f9b850c4441 in _real_syscall (sys_num=202) at pid/pid_syscallsreal.c:346
#2  0x00007f9b850c1a78 in syscall (sys_num=202) at pid/pid_miscwrappers.cpp:507
#3  0x00007f9b851799e0 in _real_syscall (sys_num=202) at syscallsreal.c:872
#4  0x00007f9b85132794 in syscall (sys_num=202) at miscwrappers.cpp:611
#5  0x00007f9b8514d37a in futex (uaddr=0x7f9b851aa078 <_wrapperExecutionLock+24>, futex_op=0, val=2, timeout=0x0, uaddr2=0x0, val3=0) at ../include/futex.h:14
#6  0x00007f9b8514d3b4 in futex_wait (uaddr=0x7f9b851aa078 <_wrapperExecutionLock+24>, old_val=2) at ../include/futex.h:21
#7  0x00007f9b8514d48f in DmtcpMutexLock (mutex=0x7f9b851aa078 <_wrapperExecutionLock+24>) at mutex.cpp:59
#8  0x00007f9b85159292 in DmtcpRWLockRdLock (rwlock=0x7f9b851aa060 <_wrapperExecutionLock>) at rwlock.cpp:50
#9  0x00007f9b8513eb78 in dmtcp::ThreadSync::wrapperExecutionLockLock () at threadsync.cpp:182
#10 0x00007f9b85125d24 in dmtcp::WrapperLock::WrapperLock (this=0x7f9b62bf3cd0, _exclusiveLock=false) at /dmtcp/dmtcp/src/threadsync.h:62
#11 0x00007f9b8513fa31 in thread_start (arg=0x7f9b7ecd9008) at threadwrappers.cpp:82
#12 0x00007f9b84920134 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442
#13 0x00007f9b8499fa40 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:100

Thread 13 (Thread 0x7f9b653f56c0 (LWP 44) "python3"):
#0  syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38
#1  0x00007f9b850c4441 in _real_syscall (sys_num=202) at pid/pid_syscallsreal.c:346
#2  0x00007f9b850c1a78 in syscall (sys_num=202) at pid/pid_miscwrappers.cpp:507
#3  0x00007f9b851799e0 in _real_syscall (sys_num=202) at syscallsreal.c:872
#4  0x00007f9b85132794 in syscall (sys_num=202) at miscwrappers.cpp:611
#5  0x00007f9b8514d37a in futex (uaddr=0x7f9b851aa078 <_wrapperExecutionLock+24>, futex_op=0, val=2, timeout=0x0, uaddr2=0x0, val3=0) at ../include/futex.h:14
#6  0x00007f9b8514d3b4 in futex_wait (uaddr=0x7f9b851aa078 <_wrapperExecutionLock+24>, old_val=2) at ../include/futex.h:21
#7  0x00007f9b8514d48f in DmtcpMutexLock (mutex=0x7f9b851aa078 <_wrapperExecutionLock+24>) at mutex.cpp:59
#8  0x00007f9b85159292 in DmtcpRWLockRdLock (rwlock=0x7f9b851aa060 <_wrapperExecutionLock>) at rwlock.cpp:50
#9  0x00007f9b8513eb78 in dmtcp::ThreadSync::wrapperExecutionLockLock () at threadsync.cpp:182
#10 0x00007f9b85125d24 in dmtcp::WrapperLock::WrapperLock (this=0x7f9b653f4cd0, _exclusiveLock=false) at /dmtcp/dmtcp/src/threadsync.h:62
#11 0x00007f9b8513fa31 in thread_start (arg=0x7f9b7ecd8008) at threadwrappers.cpp:82
#12 0x00007f9b84920134 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442
#13 0x00007f9b8499fa40 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:100

Thread 12 (Thread 0x7f9b67bf66c0 (LWP 43) "python3"):
#0  syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38
#1  0x00007f9b850c4441 in _real_syscall (sys_num=202) at pid/pid_syscallsreal.c:346
#2  0x00007f9b850c1a78 in syscall (sys_num=202) at pid/pid_miscwrappers.cpp:507
#3  0x00007f9b851799e0 in _real_syscall (sys_num=202) at syscallsreal.c:872
#4  0x00007f9b85132794 in syscall (sys_num=202) at miscwrappers.cpp:611
#5  0x00007f9b8514d37a in futex (uaddr=0x7f9b851aa078 <_wrapperExecutionLock+24>, futex_op=0, val=2, timeout=0x0, uaddr2=0x0, val3=0) at ../include/futex.h:14
#6  0x00007f9b8514d3b4 in futex_wait (uaddr=0x7f9b851aa078 <_wrapperExecutionLock+24>, old_val=2) at ../include/futex.h:21
#7  0x00007f9b8514d48f in DmtcpMutexLock (mutex=0x7f9b851aa078 <_wrapperExecutionLock+24>) at mutex.cpp:59
#8  0x00007f9b85159292 in DmtcpRWLockRdLock (rwlock=0x7f9b851aa060 <_wrapperExecutionLock>) at rwlock.cpp:50
#9  0x00007f9b8513eb78 in dmtcp::ThreadSync::wrapperExecutionLockLock () at threadsync.cpp:182
#10 0x00007f9b85125d24 in dmtcp::WrapperLock::WrapperLock (this=0x7f9b67bf5cd0, _exclusiveLock=false) at /dmtcp/dmtcp/src/threadsync.h:62
#11 0x00007f9b8513fa31 in thread_start (arg=0x7f9b7ecd7008) at threadwrappers.cpp:82
#12 0x00007f9b84920134 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442
#13 0x00007f9b8499fa40 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:100

Thread 11 (Thread 0x7f9b6a3f76c0 (LWP 42) "python3"):
#0  syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38
#1  0x00007f9b850c4441 in _real_syscall (sys_num=202) at pid/pid_syscallsreal.c:346
#2  0x00007f9b850c1a78 in syscall (sys_num=202) at pid/pid_miscwrappers.cpp:507
#3  0x00007f9b851799e0 in _real_syscall (sys_num=202) at syscallsreal.c:872
#4  0x00007f9b85132794 in syscall (sys_num=202) at miscwrappers.cpp:611
#5  0x00007f9b8514d37a in futex (uaddr=0x7f9b851aa078 <_wrapperExecutionLock+24>, futex_op=0, val=2, timeout=0x0, uaddr2=0x0, val3=0) at ../include/futex.h:14
#6  0x00007f9b8514d3b4 in futex_wait (uaddr=0x7f9b851aa078 <_wrapperExecutionLock+24>, old_val=2) at ../include/futex.h:21
#7  0x00007f9b8514d48f in DmtcpMutexLock (mutex=0x7f9b851aa078 <_wrapperExecutionLock+24>) at mutex.cpp:59
#8  0x00007f9b85159292 in DmtcpRWLockRdLock (rwlock=0x7f9b851aa060 <_wrapperExecutionLock>) at rwlock.cpp:50
#9  0x00007f9b8513eb78 in dmtcp::ThreadSync::wrapperExecutionLockLock () at threadsync.cpp:182
#10 0x00007f9b85125d24 in dmtcp::WrapperLock::WrapperLock (this=0x7f9b6a3f6cd0, _exclusiveLock=false) at /dmtcp/dmtcp/src/threadsync.h:62
#11 0x00007f9b8513fa31 in thread_start (arg=0x7f9b7ecd6008) at threadwrappers.cpp:82
#12 0x00007f9b84920134 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442
#13 0x00007f9b8499fa40 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:100

Thread 10 (Thread 0x7f9b6cbf86c0 (LWP 41) "python3"):
#0  syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38
#1  0x00007f9b850c4441 in _real_syscall (sys_num=202) at pid/pid_syscallsreal.c:346
#2  0x00007f9b850c1a78 in syscall (sys_num=202) at pid/pid_miscwrappers.cpp:507
#3  0x00007f9b851799e0 in _real_syscall (sys_num=202) at syscallsreal.c:872
#4  0x00007f9b85132794 in syscall (sys_num=202) at miscwrappers.cpp:611
#5  0x00007f9b8514d37a in futex (uaddr=0x7f9b851aa078 <_wrapperExecutionLock+24>, futex_op=0, val=2, timeout=0x0, uaddr2=0x0, val3=0) at ../include/futex.h:14
#6  0x00007f9b8514d3b4 in futex_wait (uaddr=0x7f9b851aa078 <_wrapperExecutionLock+24>, old_val=2) at ../include/futex.h:21
#7  0x00007f9b8514d48f in DmtcpMutexLock (mutex=0x7f9b851aa078 <_wrapperExecutionLock+24>) at mutex.cpp:59
#8  0x00007f9b85159292 in DmtcpRWLockRdLock (rwlock=0x7f9b851aa060 <_wrapperExecutionLock>) at rwlock.cpp:50
#9  0x00007f9b8513eb78 in dmtcp::ThreadSync::wrapperExecutionLockLock () at threadsync.cpp:182
#10 0x00007f9b85125d24 in dmtcp::WrapperLock::WrapperLock (this=0x7f9b6cbf7cd0, _exclusiveLock=false) at /dmtcp/dmtcp/src/threadsync.h:62
#11 0x00007f9b8513fa31 in thread_start (arg=0x7f9b7ecdd008) at threadwrappers.cpp:82
#12 0x00007f9b84920134 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442
#13 0x00007f9b8499fa40 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:100

Thread 9 (Thread 0x7f9b6f3f96c0 (LWP 40) "python3"):
#0  syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38
#1  0x00007f9b850c4441 in _real_syscall (sys_num=202) at pid/pid_syscallsreal.c:346
#2  0x00007f9b850c1a78 in syscall (sys_num=202) at pid/pid_miscwrappers.cpp:507
#3  0x00007f9b851799e0 in _real_syscall (sys_num=202) at syscallsreal.c:872
#4  0x00007f9b85132794 in syscall (sys_num=202) at miscwrappers.cpp:611
#5  0x00007f9b8514d37a in futex (uaddr=0x7f9b851aa078 <_wrapperExecutionLock+24>, futex_op=0, val=2, timeout=0x0, uaddr2=0x0, val3=0) at ../include/futex.h:14
#6  0x00007f9b8514d3b4 in futex_wait (uaddr=0x7f9b851aa078 <_wrapperExecutionLock+24>, old_val=2) at ../include/futex.h:21
#7  0x00007f9b8514d48f in DmtcpMutexLock (mutex=0x7f9b851aa078 <_wrapperExecutionLock+24>) at mutex.cpp:59
#8  0x00007f9b85159292 in DmtcpRWLockRdLock (rwlock=0x7f9b851aa060 <_wrapperExecutionLock>) at rwlock.cpp:50
#9  0x00007f9b8513eb78 in dmtcp::ThreadSync::wrapperExecutionLockLock () at threadsync.cpp:182
#10 0x00007f9b85125d24 in dmtcp::WrapperLock::WrapperLock (this=0x7f9b6f3f8cd0, _exclusiveLock=false) at /dmtcp/dmtcp/src/threadsync.h:62
#11 0x00007f9b8513fa31 in thread_start (arg=0x7f9b7ecdc008) at threadwrappers.cpp:82
#12 0x00007f9b84920134 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442
#13 0x00007f9b8499fa40 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:100

Thread 8 (Thread 0x7f9b71bfa6c0 (LWP 39) "python3"):
#0  syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38
#1  0x00007f9b850c4441 in _real_syscall (sys_num=202) at pid/pid_syscallsreal.c:346
#2  0x00007f9b850c1a78 in syscall (sys_num=202) at pid/pid_miscwrappers.cpp:507
#3  0x00007f9b851799e0 in _real_syscall (sys_num=202) at syscallsreal.c:872
#4  0x00007f9b85132794 in syscall (sys_num=202) at miscwrappers.cpp:611
#5  0x00007f9b8514d37a in futex (uaddr=0x7f9b851aa078 <_wrapperExecutionLock+24>, futex_op=0, val=2, timeout=0x0, uaddr2=0x0, val3=0) at ../include/futex.h:14
#6  0x00007f9b8514d3b4 in futex_wait (uaddr=0x7f9b851aa078 <_wrapperExecutionLock+24>, old_val=2) at ../include/futex.h:21
#7  0x00007f9b8514d48f in DmtcpMutexLock (mutex=0x7f9b851aa078 <_wrapperExecutionLock+24>) at mutex.cpp:59
#8  0x00007f9b85159292 in DmtcpRWLockRdLock (rwlock=0x7f9b851aa060 <_wrapperExecutionLock>) at rwlock.cpp:50
#9  0x00007f9b8513eb78 in dmtcp::ThreadSync::wrapperExecutionLockLock () at threadsync.cpp:182
#10 0x00007f9b85125d24 in dmtcp::WrapperLock::WrapperLock (this=0x7f9b71bf9cd0, _exclusiveLock=false) at /dmtcp/dmtcp/src/threadsync.h:62
#11 0x00007f9b8513fa31 in thread_start (arg=0x7f9b7ecdb008) at threadwrappers.cpp:82
#12 0x00007f9b84920134 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442
#13 0x00007f9b8499fa40 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:100

Thread 7 (Thread 0x7f9b743fb6c0 (LWP 38) "python3"):
#0  syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38
#1  0x00007f9b850c4441 in _real_syscall (sys_num=202) at pid/pid_syscallsreal.c:346
#2  0x00007f9b850c1a78 in syscall (sys_num=202) at pid/pid_miscwrappers.cpp:507
#3  0x00007f9b851799e0 in _real_syscall (sys_num=202) at syscallsreal.c:872
#4  0x00007f9b85132794 in syscall (sys_num=202) at miscwrappers.cpp:611
#5  0x00007f9b8514d37a in futex (uaddr=0x7f9b851aa078 <_wrapperExecutionLock+24>, futex_op=0, val=2, timeout=0x0, uaddr2=0x0, val3=0) at ../include/futex.h:14
#6  0x00007f9b8514d3b4 in futex_wait (uaddr=0x7f9b851aa078 <_wrapperExecutionLock+24>, old_val=2) at ../include/futex.h:21
#7  0x00007f9b8514d48f in DmtcpMutexLock (mutex=0x7f9b851aa078 <_wrapperExecutionLock+24>) at mutex.cpp:59
#8  0x00007f9b85159292 in DmtcpRWLockRdLock (rwlock=0x7f9b851aa060 <_wrapperExecutionLock>) at rwlock.cpp:50
#9  0x00007f9b8513eb78 in dmtcp::ThreadSync::wrapperExecutionLockLock () at threadsync.cpp:182
#10 0x00007f9b85125d24 in dmtcp::WrapperLock::WrapperLock (this=0x7f9b743facd0, _exclusiveLock=false) at /dmtcp/dmtcp/src/threadsync.h:62
#11 0x00007f9b8513fa31 in thread_start (arg=0x7f9b7ecda008) at threadwrappers.cpp:82
#12 0x00007f9b84920134 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442
#13 0x00007f9b8499fa40 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:100

Thread 6 (Thread 0x7f9b76bfc6c0 (LWP 37) "python3"):
#0  syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38
#1  0x00007f9b850c4441 in _real_syscall (sys_num=202) at pid/pid_syscallsreal.c:346
#2  0x00007f9b850c1a78 in syscall (sys_num=202) at pid/pid_miscwrappers.cpp:507
#3  0x00007f9b851799e0 in _real_syscall (sys_num=202) at syscallsreal.c:872
#4  0x00007f9b85132794 in syscall (sys_num=202) at miscwrappers.cpp:611
#5  0x00007f9b8514d37a in futex (uaddr=0x7f9b851aa078 <_wrapperExecutionLock+24>, futex_op=0, val=2, timeout=0x0, uaddr2=0x0, val3=0) at ../include/futex.h:14
#6  0x00007f9b8514d3b4 in futex_wait (uaddr=0x7f9b851aa078 <_wrapperExecutionLock+24>, old_val=2) at ../include/futex.h:21
#7  0x00007f9b8514d48f in DmtcpMutexLock (mutex=0x7f9b851aa078 <_wrapperExecutionLock+24>) at mutex.cpp:59
#8  0x00007f9b85159292 in DmtcpRWLockRdLock (rwlock=0x7f9b851aa060 <_wrapperExecutionLock>) at rwlock.cpp:50
#9  0x00007f9b8513eb78 in dmtcp::ThreadSync::wrapperExecutionLockLock () at threadsync.cpp:182
#10 0x00007f9b85125d24 in dmtcp::WrapperLock::WrapperLock (this=0x7f9b76bfbcd0, _exclusiveLock=false) at /dmtcp/dmtcp/src/threadsync.h:62
#11 0x00007f9b8513fa31 in thread_start (arg=0x7f9b7ece1008) at threadwrappers.cpp:82
#12 0x00007f9b84920134 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442
#13 0x00007f9b8499fa40 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:100

Thread 5 (Thread 0x7f9b7b3fd6c0 (LWP 36) "python3"):
#0  syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38
#1  0x00007f9b850c4441 in _real_syscall (sys_num=202) at pid/pid_syscallsreal.c:346
#2  0x00007f9b850c1a78 in syscall (sys_num=202) at pid/pid_miscwrappers.cpp:507
#3  0x00007f9b851799e0 in _real_syscall (sys_num=202) at syscallsreal.c:872
#4  0x00007f9b85132794 in syscall (sys_num=202) at miscwrappers.cpp:611
#5  0x00007f9b8514d37a in futex (uaddr=0x7f9b851aa078 <_wrapperExecutionLock+24>, futex_op=0, val=2, timeout=0x0, uaddr2=0x0, val3=0) at ../include/futex.h:14
#6  0x00007f9b8514d3b4 in futex_wait (uaddr=0x7f9b851aa078 <_wrapperExecutionLock+24>, old_val=2) at ../include/futex.h:21
#7  0x00007f9b8514d48f in DmtcpMutexLock (mutex=0x7f9b851aa078 <_wrapperExecutionLock+24>) at mutex.cpp:59
#8  0x00007f9b85159292 in DmtcpRWLockRdLock (rwlock=0x7f9b851aa060 <_wrapperExecutionLock>) at rwlock.cpp:50
#9  0x00007f9b8513eb78 in dmtcp::ThreadSync::wrapperExecutionLockLock () at threadsync.cpp:182
#10 0x00007f9b85125d24 in dmtcp::WrapperLock::WrapperLock (this=0x7f9b7b3fccd0, _exclusiveLock=false) at /dmtcp/dmtcp/src/threadsync.h:62
#11 0x00007f9b8513fa31 in thread_start (arg=0x7f9b7ece0008) at threadwrappers.cpp:82
#12 0x00007f9b84920134 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442
#13 0x00007f9b8499fa40 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:100

Thread 4 (Thread 0x7f9b7dbfe6c0 (LWP 35) "python3"):
#0  syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38
#1  0x00007f9b850c4441 in _real_syscall (sys_num=202) at pid/pid_syscallsreal.c:346
#2  0x00007f9b850c1a78 in syscall (sys_num=202) at pid/pid_miscwrappers.cpp:507
#3  0x00007f9b851799e0 in _real_syscall (sys_num=202) at syscallsreal.c:872
#4  0x00007f9b85132794 in syscall (sys_num=202) at miscwrappers.cpp:611
#5  0x00007f9b8514d37a in futex (uaddr=0x7f9b851aa078 <_wrapperExecutionLock+24>, futex_op=0, val=2, timeout=0x0, uaddr2=0x0, val3=0) at ../include/futex.h:14
#6  0x00007f9b8514d3b4 in futex_wait (uaddr=0x7f9b851aa078 <_wrapperExecutionLock+24>, old_val=2) at ../include/futex.h:21
#7  0x00007f9b8514d48f in DmtcpMutexLock (mutex=0x7f9b851aa078 <_wrapperExecutionLock+24>) at mutex.cpp:59
#8  0x00007f9b85159292 in DmtcpRWLockRdLock (rwlock=0x7f9b851aa060 <_wrapperExecutionLock>) at rwlock.cpp:50
#9  0x00007f9b8513eb78 in dmtcp::ThreadSync::wrapperExecutionLockLock () at threadsync.cpp:182
#10 0x00007f9b85125d24 in dmtcp::WrapperLock::WrapperLock (this=0x7f9b7dbfdcd0, _exclusiveLock=false) at /dmtcp/dmtcp/src/threadsync.h:62
#11 0x00007f9b8513fa31 in thread_start (arg=0x7f9b7ecdf008) at threadwrappers.cpp:82
#12 0x00007f9b84920134 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442
#13 0x00007f9b8499fa40 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:100

Thread 3 (Thread 0x7f9b7e3ff6c0 (LWP 34) "python3"):
#0  syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38
#1  0x00007f9b850c4441 in _real_syscall (sys_num=202) at pid/pid_syscallsreal.c:346
#2  0x00007f9b850c1a78 in syscall (sys_num=202) at pid/pid_miscwrappers.cpp:507
#3  0x00007f9b851799e0 in _real_syscall (sys_num=202) at syscallsreal.c:872
#4  0x00007f9b85132794 in syscall (sys_num=202) at miscwrappers.cpp:611
#5  0x00007f9b8514d37a in futex (uaddr=0x7f9b851aa078 <_wrapperExecutionLock+24>, futex_op=0, val=2, timeout=0x0, uaddr2=0x0, val3=0) at ../include/futex.h:14
#6  0x00007f9b8514d3b4 in futex_wait (uaddr=0x7f9b851aa078 <_wrapperExecutionLock+24>, old_val=2) at ../include/futex.h:21
#7  0x00007f9b8514d48f in DmtcpMutexLock (mutex=0x7f9b851aa078 <_wrapperExecutionLock+24>) at mutex.cpp:59
#8  0x00007f9b85159292 in DmtcpRWLockRdLock (rwlock=0x7f9b851aa060 <_wrapperExecutionLock>) at rwlock.cpp:50
#9  0x00007f9b8513eb78 in dmtcp::ThreadSync::wrapperExecutionLockLock () at threadsync.cpp:182
#10 0x00007f9b85125d24 in dmtcp::WrapperLock::WrapperLock (this=0x7f9b7e3fecd0, _exclusiveLock=false) at /dmtcp/dmtcp/src/threadsync.h:62
#11 0x00007f9b8513fa31 in thread_start (arg=0x7f9b7ecde008) at threadwrappers.cpp:82
#12 0x00007f9b84920134 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442
#13 0x00007f9b8499fa40 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:100

Thread 2 (Thread 0x7f9b8230f6c0 (LWP 33) "python3"):
#0  __GI___libc_read (nbytes=208, buf=0x7f9b8230ccd0, fd=821) at ../sysdeps/unix/sysv/linux/read.c:26
#1  __GI___libc_read (fd=821, buf=0x7f9b8230ccd0, nbytes=208) at ../sysdeps/unix/sysv/linux/read.c:24
#2  0x00007f9b85164868 in dmtcp::Util::readAll (fd=821, buf=0x7f9b8230ccd0, count=208) at util_misc.cpp:191
#3  0x00007f9b85146ced in dmtcp::CoordinatorAPI::recvMsgFromCoordinatorRaw (fd=821, msg=0x7f9b8230d110, extraData=0x0) at coordinatorapi.cpp:379
#4  0x00007f9b851472a7 in dmtcp::CoordinatorAPI::recvMsgFromCoordinator (msg=0x7f9b8230d110, extraData=0x0) at coordinatorapi.cpp:425
#5  0x00007f9b85126f2d in dmtcp::DmtcpWorker::waitForPreSuspendMessage () at dmtcpworker.cpp:337
#6  0x00007f9b851272d4 in dmtcp::DmtcpWorker::waitForCheckpointRequest () at dmtcpworker.cpp:367
#7  0x00007f9b8513bb4f in checkpointhread (dummy=0x0) at threadlist.cpp:391
#8  0x00007f9b8513fa19 in thread_start (arg=0x7f9b84314008) at threadwrappers.cpp:79
#9  0x00007f9b84920134 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442
#10 0x00007f9b8499fa40 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:100

Thread 1 (Thread 0x7f9b84577780 (LWP 25) "python3"):
#0  __futex_abstimed_wait_common64 (private=128, cancel=true, abstime=0x7ffc49fd1040, op=265, expected=34, futex_word=0x7f9b7e3ff990) at ./nptl/futex-internal.c:57
#1  __futex_abstimed_wait_common (futex_word=futex_word@entry=0x7f9b7e3ff990, expected=34, clockid=clockid@entry=0, abstime=abstime@entry=0x7ffc49fd1040, private=private@entry=128, cancel=cancel@entry=true) at ./nptl/futex-internal.c:87
#2  0x00007f9b8491cefb in __GI___futex_abstimed_wait_cancelable64 (futex_word=futex_word@entry=0x7f9b7e3ff990, expected=<optimized out>, clockid=clockid@entry=0, abstime=abstime@entry=0x7ffc49fd1040, private=private@entry=128) at ./nptl/futex-internal.c:139
#3  0x00007f9b84921c23 in __pthread_clockjoin_ex (threadid=140305814779584, thread_return=0x0, clockid=0, abstime=0x7ffc49fd1040, block=<optimized out>) at ./nptl/pthread_join_common.c:102
#4  0x00007f9b85179e18 in _real_pthread_timedjoin_np (thread=140305814779584, retval=0x0, abstime=0x7ffc49fd1040) at syscallsreal.c:954
#5  0x00007f9b8514033b in pthread_join (thread=140305814779584, retval=0x0) at threadwrappers.cpp:218
#6  0x00007f9b7f067a47 in blas_thread_shutdown_ () from /usr/local/lib/python3.12/site-packages/numpy/_core/../../numpy.libs/libscipy_openblas64_-99b71e71.so
#7  0x00007f9b8496b6da in __run_prefork_handlers (do_locking=do_locking@entry=true) at ./posix/register-atfork.c:141
#8  0x00007f9b8496adfa in __libc_fork () at ./posix/fork.c:51
#9  0x00007f9b850c44cb in _real_fork () at pid/pid_syscallsreal.c:355
#10 0x00007f9b850bffb6 in fork () at pid/pid_miscwrappers.cpp:65
#11 0x00007f9b8517793a in _real_fork () at syscallsreal.c:454
#12 0x00007f9b85128eee in fork () at execwrappers.cpp:188
#13 0x00007f9b85128f87 in vfork () at execwrappers.cpp:234
#14 0x00007f9b58655c92 in ?? () from /usr/local/lib/python3.12/lib-dynload/_posixsubprocess.cpython-312-x86_64-linux-gnu.so
#15 0x00007f9b586564a4 in ?? () from /usr/local/lib/python3.12/lib-dynload/_posixsubprocess.cpython-312-x86_64-linux-gnu.so
#16 0x00007f9b586568d8 in ?? () from /usr/local/lib/python3.12/lib-dynload/_posixsubprocess.cpython-312-x86_64-linux-gnu.so
#17 0x00007f9b84d101dd in ?? () from /usr/local/bin/../lib/libpython3.12.so.1.0
#18 0x00007f9b84ccbfaf in PyObject_Vectorcall () from /usr/local/bin/../lib/libpython3.12.so.1.0
#19 0x00007f9b84cb9c2f in _PyEval_EvalFrameDefault () from /usr/local/bin/../lib/libpython3.12.so.1.0
#20 0x00007f9b84cb1f7d in _PyObject_FastCallDictTstate () from /usr/local/bin/../lib/libpython3.12.so.1.0
#21 0x00007f9b84cd55cc in ?? () from /usr/local/bin/../lib/libpython3.12.so.1.0
#22 0x00007f9b84cb0d36 in ?? () from /usr/local/bin/../lib/libpython3.12.so.1.0
#23 0x00007f9b84d1a351 in _PyObject_Call () from /usr/local/bin/../lib/libpython3.12.so.1.0
#24 0x00007f9b84cc1430 in _PyEval_EvalFrameDefault () from /usr/local/bin/../lib/libpython3.12.so.1.0
#25 0x00007f9b84d0ccc2 in PyEval_EvalCode () from /usr/local/bin/../lib/libpython3.12.so.1.0
#26 0x00007f9b84bba7e5 in ?? () from /usr/local/bin/../lib/libpython3.12.so.1.0
#27 0x00007f9b84d0d54d in ?? () from /usr/local/bin/../lib/libpython3.12.so.1.0
#28 0x00007f9b84cc1430 in _PyEval_EvalFrameDefault () from /usr/local/bin/../lib/libpython3.12.so.1.0
#29 0x00007f9b84ccf275 in ?? () from /usr/local/bin/../lib/libpython3.12.so.1.0
#30 0x00007f9b84ce5a40 in PyObject_CallMethodObjArgs () from /usr/local/bin/../lib/libpython3.12.so.1.0
#31 0x00007f9b84d25963 in PyImport_ImportModuleLevelObject () from /usr/local/bin/../lib/libpython3.12.so.1.0
#32 0x00007f9b84cbe337 in _PyEval_EvalFrameDefault () from /usr/local/bin/../lib/libpython3.12.so.1.0
#33 0x00007f9b84d0ccc2 in PyEval_EvalCode () from /usr/local/bin/../lib/libpython3.12.so.1.0
#34 0x00007f9b84bba7e5 in ?? () from /usr/local/bin/../lib/libpython3.12.so.1.0
#35 0x00007f9b84d0d54d in ?? () from /usr/local/bin/../lib/libpython3.12.so.1.0
#36 0x00007f9b84cc1430 in _PyEval_EvalFrameDefault () from /usr/local/bin/../lib/libpython3.12.so.1.0
#37 0x00007f9b84ccf275 in ?? () from /usr/local/bin/../lib/libpython3.12.so.1.0
#38 0x00007f9b84ce5a40 in PyObject_CallMethodObjArgs () from /usr/local/bin/../lib/libpython3.12.so.1.0
#39 0x00007f9b84d25963 in PyImport_ImportModuleLevelObject () from /usr/local/bin/../lib/libpython3.12.so.1.0
#40 0x00007f9b84cbe337 in _PyEval_EvalFrameDefault () from /usr/local/bin/../lib/libpython3.12.so.1.0
#41 0x00007f9b84cd4751 in PyObject_CallOneArg () from /usr/local/bin/../lib/libpython3.12.so.1.0
#42 0x00007f9b84d15189 in ?? () from /usr/local/bin/../lib/libpython3.12.so.1.0
#43 0x00007f9b84ccc7f0 in _PyObject_LookupAttr () from /usr/local/bin/../lib/libpython3.12.so.1.0
#44 0x00007f9b84d104a1 in ?? () from /usr/local/bin/../lib/libpython3.12.so.1.0
#45 0x00007f9b84cbb5e7 in _PyEval_EvalFrameDefault () from /usr/local/bin/../lib/libpython3.12.so.1.0
#46 0x00007f9b84ccf275 in ?? () from /usr/local/bin/../lib/libpython3.12.so.1.0
#47 0x00007f9b84ce5a40 in PyObject_CallMethodObjArgs () from /usr/local/bin/../lib/libpython3.12.so.1.0
#48 0x00007f9b84d25aec in PyImport_ImportModuleLevelObject () from /usr/local/bin/../lib/libpython3.12.so.1.0
#49 0x00007f9b84cbe337 in _PyEval_EvalFrameDefault () from /usr/local/bin/../lib/libpython3.12.so.1.0
#50 0x00007f9b84d0ccc2 in PyEval_EvalCode () from /usr/local/bin/../lib/libpython3.12.so.1.0
#51 0x00007f9b84bba7e5 in ?? () from /usr/local/bin/../lib/libpython3.12.so.1.0
#52 0x00007f9b84d0d54d in ?? () from /usr/local/bin/../lib/libpython3.12.so.1.0
#53 0x00007f9b84cc1430 in _PyEval_EvalFrameDefault () from /usr/local/bin/../lib/libpython3.12.so.1.0
#54 0x00007f9b84ccf275 in ?? () from /usr/local/bin/../lib/libpython3.12.so.1.0
#55 0x00007f9b84ce5a40 in PyObject_CallMethodObjArgs () from /usr/local/bin/../lib/libpython3.12.so.1.0
#56 0x00007f9b84d25963 in PyImport_ImportModuleLevelObject () from /usr/local/bin/../lib/libpython3.12.so.1.0
#57 0x00007f9b84d2e658 in ?? () from /usr/local/bin/../lib/libpython3.12.so.1.0
#58 0x00007f9b84d0d54d in ?? () from /usr/local/bin/../lib/libpython3.12.so.1.0
#59 0x00007f9b84cc1430 in _PyEval_EvalFrameDefault () from /usr/local/bin/../lib/libpython3.12.so.1.0
#60 0x00007f9b84ccf275 in ?? () from /usr/local/bin/../lib/libpython3.12.so.1.0
#61 0x00007f9b84ce5a40 in PyObject_CallMethodObjArgs () from /usr/local/bin/../lib/libpython3.12.so.1.0
#62 0x00007f9b84d25aec in PyImport_ImportModuleLevelObject () from /usr/local/bin/../lib/libpython3.12.so.1.0
#63 0x00007f9b84cbe337 in _PyEval_EvalFrameDefault () from /usr/local/bin/../lib/libpython3.12.so.1.0
#64 0x00007f9b84d0ccc2 in PyEval_EvalCode () from /usr/local/bin/../lib/libpython3.12.so.1.0
#65 0x00007f9b84bba7e5 in ?? () from /usr/local/bin/../lib/libpython3.12.so.1.0
#66 0x00007f9b84d0d54d in ?? () from /usr/local/bin/../lib/libpython3.12.so.1.0
#67 0x00007f9b84cc1430 in _PyEval_EvalFrameDefault () from /usr/local/bin/../lib/libpython3.12.so.1.0
#68 0x00007f9b84ccf275 in ?? () from /usr/local/bin/../lib/libpython3.12.so.1.0
#69 0x00007f9b84ce5a40 in PyObject_CallMethodObjArgs () from /usr/local/bin/../lib/libpython3.12.so.1.0
#70 0x00007f9b84d25963 in PyImport_ImportModuleLevelObject () from /usr/local/bin/../lib/libpython3.12.so.1.0
#71 0x00007f9b84cbe337 in _PyEval_EvalFrameDefault () from /usr/local/bin/../lib/libpython3.12.so.1.0
#72 0x00007f9b84d0ccc2 in PyEval_EvalCode () from /usr/local/bin/../lib/libpython3.12.so.1.0
#73 0x00007f9b84bba7e5 in ?? () from /usr/local/bin/../lib/libpython3.12.so.1.0
#74 0x00007f9b84d0d54d in ?? () from /usr/local/bin/../lib/libpython3.12.so.1.0
#75 0x00007f9b84cc1430 in _PyEval_EvalFrameDefault () from /usr/local/bin/../lib/libpython3.12.so.1.0
#76 0x00007f9b84ccf275 in ?? () from /usr/local/bin/../lib/libpython3.12.so.1.0
#77 0x00007f9b84ce5a40 in PyObject_CallMethodObjArgs () from /usr/local/bin/../lib/libpython3.12.so.1.0
#78 0x00007f9b84d25963 in PyImport_ImportModuleLevelObject () from /usr/local/bin/../lib/libpython3.12.so.1.0
#79 0x00007f9b84cbe337 in _PyEval_EvalFrameDefault () from /usr/local/bin/../lib/libpython3.12.so.1.0
#80 0x00007f9b84d0ccc2 in PyEval_EvalCode () from /usr/local/bin/../lib/libpython3.12.so.1.0
#81 0x00007f9b84bba7e5 in ?? () from /usr/local/bin/../lib/libpython3.12.so.1.0
#82 0x00007f9b84d0d54d in ?? () from /usr/local/bin/../lib/libpython3.12.so.1.0
#83 0x00007f9b84cc1430 in _PyEval_EvalFrameDefault () from /usr/local/bin/../lib/libpython3.12.so.1.0
#84 0x00007f9b84ccf275 in ?? () from /usr/local/bin/../lib/libpython3.12.so.1.0
#85 0x00007f9b84ce5a40 in PyObject_CallMethodObjArgs () from /usr/local/bin/../lib/libpython3.12.so.1.0
#86 0x00007f9b84d25963 in PyImport_ImportModuleLevelObject () from /usr/local/bin/../lib/libpython3.12.so.1.0
#87 0x00007f9b84cbe337 in _PyEval_EvalFrameDefault () from /usr/local/bin/../lib/libpython3.12.so.1.0
#88 0x00007f9b84d0ccc2 in PyEval_EvalCode () from /usr/local/bin/../lib/libpython3.12.so.1.0
#89 0x00007f9b84bba7e5 in ?? () from /usr/local/bin/../lib/libpython3.12.so.1.0
#90 0x00007f9b84d0d54d in ?? () from /usr/local/bin/../lib/libpython3.12.so.1.0
#91 0x00007f9b84cc1430 in _PyEval_EvalFrameDefault () from /usr/local/bin/../lib/libpython3.12.so.1.0
#92 0x00007f9b84ccf275 in ?? () from /usr/local/bin/../lib/libpython3.12.so.1.0
#93 0x00007f9b84ce5a40 in PyObject_CallMethodObjArgs () from /usr/local/bin/../lib/libpython3.12.so.1.0
#94 0x00007f9b84d25963 in PyImport_ImportModuleLevelObject () from /usr/local/bin/../lib/libpython3.12.so.1.0
#95 0x00007f9b84cbe337 in _PyEval_EvalFrameDefault () from /usr/local/bin/../lib/libpython3.12.so.1.0
#96 0x00007f9b84d0ccc2 in PyEval_EvalCode () from /usr/local/bin/../lib/libpython3.12.so.1.0
#97 0x00007f9b84bba7e5 in ?? () from /usr/local/bin/../lib/libpython3.12.so.1.0
#98 0x00007f9b84d0d54d in ?? () from /usr/local/bin/../lib/libpython3.12.so.1.0
#99 0x00007f9b84cc1430 in _PyEval_EvalFrameDefault () from /usr/local/bin/../lib/libpython3.12.so.1.0
#100 0x00007f9b84ccf275 in ?? () from /usr/local/bin/../lib/libpython3.12.so.1.0
#101 0x00007f9b84ce5a40 in PyObject_CallMethodObjArgs () from /usr/local/bin/../lib/libpython3.12.so.1.0
#102 0x00007f9b84d25963 in PyImport_ImportModuleLevelObject () from /usr/local/bin/../lib/libpython3.12.so.1.0
#103 0x00007f9b5852bd90 in __Pyx_Import(_object*, _object*, int) [clone .constprop.0] () from /usr/local/lib/python3.12/site-packages/scipy/spatial/_ckdtree.cpython-312-x86_64-linux-gnu.so
#104 0x00007f9b58521804 in __pyx_pymod_exec__ckdtree(_object*) () from /usr/local/lib/python3.12/site-packages/scipy/spatial/_ckdtree.cpython-312-x86_64-linux-gnu.so
#105 0x00007f9b84d148d5 in PyModule_ExecDef () from /usr/local/bin/../lib/libpython3.12.so.1.0
#106 0x00007f9b84d1a71c in ?? () from /usr/local/bin/../lib/libpython3.12.so.1.0
#107 0x00007f9b84ccee23 in ?? () from /usr/local/bin/../lib/libpython3.12.so.1.0
#108 0x00007f9b84cc1430 in _PyEval_EvalFrameDefault () from /usr/local/bin/../lib/libpython3.12.so.1.0
#109 0x00007f9b84ccf275 in ?? () from /usr/local/bin/../lib/libpython3.12.so.1.0
#110 0x00007f9b84ce5a40 in PyObject_CallMethodObjArgs () from /usr/local/bin/../lib/libpython3.12.so.1.0
#111 0x00007f9b84d25963 in PyImport_ImportModuleLevelObject () from /usr/local/bin/../lib/libpython3.12.so.1.0
#112 0x00007f9b84cbe337 in _PyEval_EvalFrameDefault () from /usr/local/bin/../lib/libpython3.12.so.1.0
#113 0x00007f9b84d0ccc2 in PyEval_EvalCode () from /usr/local/bin/../lib/libpython3.12.so.1.0
#114 0x00007f9b84bba7e5 in ?? () from /usr/local/bin/../lib/libpython3.12.so.1.0
#115 0x00007f9b84d0d54d in ?? () from /usr/local/bin/../lib/libpython3.12.so.1.0
#116 0x00007f9b84cc1430 in _PyEval_EvalFrameDefault () from /usr/local/bin/../lib/libpython3.12.so.1.0
#117 0x00007f9b84ccf275 in ?? () from /usr/local/bin/../lib/libpython3.12.so.1.0
#118 0x00007f9b84ce5a40 in PyObject_CallMethodObjArgs () from /usr/local/bin/../lib/libpython3.12.so.1.0
#119 0x00007f9b84d25963 in PyImport_ImportModuleLevelObject () from /usr/local/bin/../lib/libpython3.12.so.1.0
#120 0x00007f9b84cbe337 in _PyEval_EvalFrameDefault () from /usr/local/bin/../lib/libpython3.12.so.1.0
#121 0x00007f9b84d0ccc2 in PyEval_EvalCode () from /usr/local/bin/../lib/libpython3.12.so.1.0
#122 0x00007f9b84bba7e5 in ?? () from /usr/local/bin/../lib/libpython3.12.so.1.0
#123 0x00007f9b84d0d54d in ?? () from /usr/local/bin/../lib/libpython3.12.so.1.0
#124 0x00007f9b84cc1430 in _PyEval_EvalFrameDefault () from /usr/local/bin/../lib/libpython3.12.so.1.0
#125 0x00007f9b84cd4751 in PyObject_CallOneArg () from /usr/local/bin/../lib/libpython3.12.so.1.0
#126 0x00007f9b84d15189 in ?? () from /usr/local/bin/../lib/libpython3.12.so.1.0
#127 0x00007f9b84ccc7f0 in _PyObject_LookupAttr () from /usr/local/bin/../lib/libpython3.12.so.1.0
#128 0x00007f9b84d104a1 in ?? () from /usr/local/bin/../lib/libpython3.12.so.1.0
#129 0x00007f9b84cbb5e7 in _PyEval_EvalFrameDefault () from /usr/local/bin/../lib/libpython3.12.so.1.0
#130 0x00007f9b84ccf275 in ?? () from /usr/local/bin/../lib/libpython3.12.so.1.0
#131 0x00007f9b84ce5a40 in PyObject_CallMethodObjArgs () from /usr/local/bin/../lib/libpython3.12.so.1.0
#132 0x00007f9b84d25aec in PyImport_ImportModuleLevelObject () from /usr/local/bin/../lib/libpython3.12.so.1.0
#133 0x00007f9b84cbe337 in _PyEval_EvalFrameDefault () from /usr/local/bin/../lib/libpython3.12.so.1.0
#134 0x00007f9b84d0ccc2 in PyEval_EvalCode () from /usr/local/bin/../lib/libpython3.12.so.1.0
#135 0x00007f9b84d41e17 in ?? () from /usr/local/bin/../lib/libpython3.12.so.1.0
#136 0x00007f9b84d41beb in ?? () from /usr/local/bin/../lib/libpython3.12.so.1.0
#137 0x00007f9b84d41802 in ?? () from /usr/local/bin/../lib/libpython3.12.so.1.0
#138 0x00007f9b84d4156b in _PyRun_SimpleFileObject () from /usr/local/bin/../lib/libpython3.12.so.1.0
#139 0x00007f9b84d41344 in _PyRun_AnyFileObject () from /usr/local/bin/../lib/libpython3.12.so.1.0
#140 0x00007f9b84d3ceb7 in Py_RunMain () from /usr/local/bin/../lib/libpython3.12.so.1.0
#141 0x00007f9b84cf6d47 in Py_BytesMain () from /usr/local/bin/../lib/libpython3.12.so.1.0
#142 0x00007f9b848be24a in __libc_start_call_main (main=main@entry=0x55d314073140, argc=argc@entry=2, argv=argv@entry=0x7ffc49fd88d8) at ../sysdeps/nptl/libc_start_call_main.h:58
#143 0x00007f9b848be305 in __libc_start_main_impl (main=0x55d314073140, argc=2, argv=0x7ffc49fd88d8, init=<optimized out>, fini=<optimized out>, rtld_fini=<optimized out>, stack_end=0x7ffc49fd88c8) at ../csu/libc-start.c:360
#144 0x000055d314073071 in _start ()
(gdb)