bloomberg / memray

Memray is a memory profiler for Python
https://bloomberg.github.io/memray/
Apache License 2.0
13.17k stars 392 forks source link

Using `memray attach` on a suspended process leads to a segfault #450

Closed godlygeek closed 2 months ago

godlygeek commented 1 year ago

Is there an existing issue for this?

Current Behavior

Running a process, then suspend it with Ctrl+Z, then trying to attach to it with memray attach leads to a segfault:

$ python memray-attach-issue.py
^Z
zsh: suspended  python memray-attach-issue.py
$ memray attach 6349 --method gdb --verbose
Debugger command line: gdb -batch -p 6349 -nx -nw '-iex=set auto-solib-add off' '-ex=set $rtld_now=2' '-ex=set $libpath="/src/memray/_inject.abi3.so"' '-ex=set $port=57823' -x=/src/memray/commands/_attach.gdb
debugger return code: 1
debugger output:
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib64/libthread_db.so.1".

Program received signal SIGTSTP, Stopped (user).
0x00007f6a0cd08afa in clock_nanosleep () from /lib64/libc.so.6
$1 = "MEMRAY: Attached to process."
From                To                  Syms Read   Shared Object Library
0x00007f6a0e0fc6e0  0x00007f6a0e34fdd4  Yes         /usr/local/lib64/libpython3.11.so.1.0
0x00007f6a0ddf68f0  0x00007f6a0de01db1  Yes (*)     /lib64/libpthread.so.0
0x00007f6a0dbede50  0x00007f6a0dbee94e  Yes (*)     /lib64/libdl.so.2
0x00007f6a0d9eae90  0x00007f6a0d9eb7a4  No          /lib64/libutil.so.1
0x00007f6a0d7e4230  0x00007f6a0d7e71bc  No          /lib64/librt.so.1
0x00007f6a0d5354e0  0x00007f6a0d59c59a  No          /lib64/libstdc++.so.6
0x00007f6a0d1dd350  0x00007f6a0d248336  No          /lib64/libm.so.6
0x00007f6a0cfc4ad0  0x00007f6a0cfd4285  No          /lib64/libgcc_s.so.1
0x00007f6a0cc139f0  0x00007f6a0cd63b6f  Yes (*)     /lib64/libc.so.6
0x00007f6a0e5ddaf0  0x00007f6a0e5f9060  No          /lib64/ld-linux-x86-64.so.2
(*): Shared library is missing debugging information.
$2 = "MEMRAY: Checking if process is Python 3.7+."
$3 = {void *(size_t)} 0x7f6a0e1d9250 <PyMem_Malloc>
$4 = {void *(size_t, size_t)} 0x7f6a0e1d97d0 <PyMem_Calloc>
$5 = {void *(void *, size_t)} 0x7f6a0e1e34e0 <PyMem_Realloc>
$6 = {void (void *)} 0x7f6a0e1d9b30 <PyMem_Free>
$7 = "MEMRAY: Process is Python 3.7+."
Breakpoint 1 at 0x7f6a0cc79740
Breakpoint 2 at 0x7f6a0cc7a160
Breakpoint 3 at 0x7f6a0cc79c40
Breakpoint 4 at 0x7f6a0cc79b60
Breakpoint 5 at 0x7f6a0e1d9250: PyMem_Malloc. (17 locations)
Breakpoint 6 at 0x7f6a0e1d97d0: PyMem_Calloc. (2 locations)
Breakpoint 7 at 0x7f6a0e1e3283: PyMem_Realloc. (5 locations)
Breakpoint 8 at 0x7f6a0e1d9b30: PyMem_Free. (13 locations)

Breakpoint 1, 0x00007f6a0cc79740 in malloc () from /lib64/libc.so.6
$8 = (void *) 0x1ee9dc0
$9 = 0x0
[New Thread 0x7f6a0bd76700 (LWP 6652)]

Thread 2 "python" received signal SIGTSTP, Stopped (user).
[Switching to Thread 0x7f6a0bd76700 (LWP 6652)]
0x00007f6a0ccf2ad1 in clone () from /lib64/libc.so.6
/src/memray/commands/_attach.gdb:37: Error in sourced command file:
The program received a signal in another thread while
making a function call from GDB.
Evaluation of the expression containing the function
(memray_spawn_client(int)) will be abandoned.
When the function is done executing, GDB will silently stop.
[Inferior 1 (process 6349) detached]

An unexpected error occurred. Run with --verbose to debug the failure.
[1]  + segmentation fault  python memray-attach-issue.py
$

Expected Behavior

No segfault 🙃

Steps To Reproduce

Run a script, suspend it, try to attach to that pid with memray attach

Memray Version

1.9.1

Python Version

3.11

Operating System

Linux

Anything else?

No response

godlygeek commented 2 months ago

We're no longer able to reproduce this with the latest main.