Open venu-shapley opened 2 months ago
What is the pip package version? Does it crash without pyroscope?
Just to clarify, does it crash every time at the startup? or some times it does not crash?
Can you run the python app with env RUST_BACKTRACE=full
?
Another thing that could help in theory is full debug which you can enable like this
l = logging.getLogger()
l.setLevel(logging.DEBUG)
pyroscope.configure(
# ...
enable_logging = True,
)
Can you attach lldb to a running python process like this lldb -p $(echo $YOUR_PYTHON_PID) --one-line continue
and then when you get the Segmentation fault in the debugger you could run two commands bt
and reg read
- that could give some starting point for investigation. And then share the backtrace and regs dump.
Or if the crash happens too quickly before you can attach with the lldb debugger, you can start it from the debugger like this lldb /Users/korniltsev/.pyenv/versions/3.11.9/bin/python3 --one-line 'run your_script.py'
and again once you get the sgementation fault do bt
and reg read
and share here.
You may need adjust the commands little for your app.
The package version we are using is pyroscope-io = "0.8.7". It doesn't crash when I don't initialize the pyroscope profiler. It doesn't crash everytime, it does it sometimes, but it is not rare to crash. The crash is happening very early, even before the fast api server is ready to serve requests. I'll try to get the lldb debug info shortly.
I got the same issue with the 0.8.7 version.
As mentioned before it doesn't happen every time.
Here is the backtrace from gdb in case this helps. (I did not do more investigration into the issue, and downgraded to 0.8.5)
Facing similar issue RUST backtrace MacOS 14.5 (Sonoma) M1
2024-06-03T08:09:47.973Z WARN py_spy::sampler > Failed to open process 66517: Undefined error: 0 (os error 0)
2024-06-03T08:09:48.012Z WARN py_spy::sampler > Failed to create spy for 66517: Undefined error: 0 (os error 0)
2024-06-03T08:09:48.117Z WARN py_spy::sampler > Failed to create spy for 66517: Operation timed out (os error 60)
2024-06-03T08:09:48.223Z WARN py_spy::sampler > Failed to create spy for 66517: Operation timed out (os error 60)
2024-06-03T08:09:48.324Z WARN py_spy::sampler > Failed to create spy for 66517: Operation timed out (os error 60)
2024-06-03T08:09:48.425Z WARN py_spy::sampler > Failed to create spy for 66517: Operation timed out (os error 60)
2024-06-03T08:09:48.526Z WARN py_spy::sampler > Failed to create spy for 66517: Operation timed out (os error 60)
2024-06-03T08:09:48.628Z WARN py_spy::sampler > Failed to create spy for 66517: Operation timed out (os error 60)
2024-06-03T08:09:48.733Z WARN py_spy::sampler > Failed to create spy for 66517: Operation timed out (os error 60)
2024-06-03T08:09:48.838Z WARN py_spy::sampler > Failed to create spy for 66517: Operation timed out (os error 60)
2024-06-03T08:09:48.942Z WARN py_spy::sampler > Failed to create spy for 66517: Operation timed out (os error 60)
thread '<unnamed>' panicked at 'called `Result::unwrap()` on an `Err` value: SetLoggerError(())', /Users/runner/.cargo/registry/src/index.crates.io-6f17d22bba15001f/pretty_env_logger-0.4.0/src/lib.rs:74:22
stack backtrace:
0: 0x17c8bb580 - <std::sys_common::backtrace::_print::DisplayBacktrace as core::fmt::Display>::fmt::h819e9cbdf1a9e730
1: 0x17c78b548 - core::fmt::write::ha5e9bf3131ecb7c0
2: 0x17c8a10ac - std::io::Write::write_fmt::h414ce9994bf17404
3: 0x17c8be880 - std::sys_common::backtrace::print::h8072db0bbd5bcc3d
4: 0x17c8be4e0 - std::panicking::default_hook::{{closure}}::h2c85c5b0c2ede151
5: 0x17c8bf3cc - std::panicking::rust_panic_with_hook::h023af7f90b47eb8b
6: 0x17c8bef7c - std::panicking::begin_panic_handler::{{closure}}::h14283519edc1d634
7: 0x17c8beeec - std::sys_common::backtrace::__rust_end_short_backtrace::hc366c0b0cef5b747
8: 0x17c8beee0 - _rust_begin_unwind
9: 0x17c8f9128 - core::panicking::panic_fmt::h324f50b29db90195
10: 0x17c8f9374 - core::result::unwrap_failed::hf783e6a14bbaf60b
11: 0x17c77e380 - _initialize_logging
12: 0x1a4981050 - <unknown>
13: 0x1a4989ae0 - <unknown>
14: 0x17c189478 - _cdata_call
15: 0x102289020 - __PyObject_MakeTpCall
16: 0x10235ebdc - _call_function
17: 0x10235bed4 - __PyEval_EvalFrameDefault
18: 0x10235f898 - __PyEval_EvalCodeWithName
19: 0x102289ba4 - __PyFunction_Vectorcall
20: 0x10235eb3c - _call_function
21: 0x10235bfbc - __PyEval_EvalFrameDefault
22: 0x102289a04 - _function_code_fastcall
23: 0x10235eb3c - _call_function
24: 0x10235bed4 - __PyEval_EvalFrameDefault
25: 0x10235f898 - __PyEval_EvalCodeWithName
26: 0x102356150 - _PyEval_EvalCode
27: 0x102353420 - _builtin_exec
28: 0x1022cbab4 - _cfunction_vectorcall_FASTCALL
29: 0x102289384 - _PyVectorcall_Call
30: 0x10235c26c - __PyEval_EvalFrameDefault
31: 0x10235f898 - __PyEval_EvalCodeWithName
32: 0x102289ba4 - __PyFunction_Vectorcall
33: 0x10235eb3c - _call_function
34: 0x10235bed4 - __PyEval_EvalFrameDefault
35: 0x102289a04 - _function_code_fastcall
36: 0x10235eb3c - _call_function
37: 0x10235beb0 - __PyEval_EvalFrameDefault
38: 0x102289a04 - _function_code_fastcall
39: 0x10235eb3c - _call_function
40: 0x10235bf48 - __PyEval_EvalFrameDefault
41: 0x102289a04 - _function_code_fastcall
42: 0x10235eb3c - _call_function
43: 0x10235bf48 - __PyEval_EvalFrameDefault
44: 0x102289a04 - _function_code_fastcall
45: 0x10235eb3c - _call_function
46: 0x10235bf48 - __PyEval_EvalFrameDefault
47: 0x10235f898 - __PyEval_EvalCodeWithName
48: 0x102289ba4 - __PyFunction_Vectorcall
49: 0x10235eb3c - _call_function
50: 0x10235bed4 - __PyEval_EvalFrameDefault
51: 0x10235f898 - __PyEval_EvalCodeWithName
52: 0x102289ba4 - __PyFunction_Vectorcall
53: 0x10235eb3c - _call_function
54: 0x10235bed4 - __PyEval_EvalFrameDefault
55: 0x102289a04 - _function_code_fastcall
56: 0x10235eb3c - _call_function
57: 0x10235bf48 - __PyEval_EvalFrameDefault
58: 0x102289a04 - _function_code_fastcall
59: 0x10235eb3c - _call_function
60: 0x10235beb0 - __PyEval_EvalFrameDefault
61: 0x10229b8f4 - _gen_send_ex
62: 0x102daf01c - _task_step
63: 0x102289020 - __PyObject_MakeTpCall
64: 0x102378394 - _context_run
65: 0x1022cbbdc - _cfunction_vectorcall_FASTCALL_KEYWORDS
66: 0x102289384 - _PyVectorcall_Call
67: 0x10235c26c - __PyEval_EvalFrameDefault
68: 0x102289a04 - _function_code_fastcall
69: 0x10235eb3c - _call_function
70: 0x10235beb0 - __PyEval_EvalFrameDefault
71: 0x102289a04 - _function_code_fastcall
72: 0x10235eb3c - _call_function
73: 0x10235beb0 - __PyEval_EvalFrameDefault
74: 0x102289a04 - _function_code_fastcall
75: 0x10235eb3c - _call_function
76: 0x10235beb0 - __PyEval_EvalFrameDefault
77: 0x102289a04 - _function_code_fastcall
78: 0x10235eb3c - _call_function
79: 0x10235beb0 - __PyEval_EvalFrameDefault
80: 0x10235f898 - __PyEval_EvalCodeWithName
81: 0x102289ba4 - __PyFunction_Vectorcall
82: 0x10235eb3c - _call_function
83: 0x10235beb0 - __PyEval_EvalFrameDefault
84: 0x10235f898 - __PyEval_EvalCodeWithName
85: 0x102289ba4 - __PyFunction_Vectorcall
86: 0x10235eb3c - _call_function
87: 0x10235bfbc - __PyEval_EvalFrameDefault
88: 0x10235f898 - __PyEval_EvalCodeWithName
89: 0x102356150 - _PyEval_EvalCode
90: 0x10239ca78 - _PyRun_SimpleFileExFlags
91: 0x1023bb078 - _Py_RunMain
92: 0x1023bb508 - _pymain_main
93: 0x1023bb5a8 - _Py_BytesMain
fatal runtime error: failed to initiate panic, error 5
[1] 66516 abort RUST_BACKTRACE=full python app_fastapi.py
We are using pyroscope-io package to profile a python application which is a fastAPI server.
During the initialization, I configure the pyroscope but I am seeing Segmentation faults and pointed to the following difference when the service starts up vs running into a segmentation fault.
Can you please help me understand what is going on?