search

benfred / py-spy

Sampling profiler for Python programs
MIT License
12.16k stars 401 forks source link

Incomplete profiling results #568

Open steve36871 opened 1 year ago

steve36871 commented 1 year ago

Hello! I noticed that ~98% of the samples are missing in my case. I cannot share the profile for reasons of confidentiality, but at the process level, I see ~600.000 samples, and one level below it, only ~11.000 samples are left. This happens with the following parameters: py-spy record -r 1 -s. I intentionally set the sampling rate low because the process runs for about 1 day. How do you think I should interpret this result? What is my application doing 98% of the time when it is inside the process but not within any function inside that process?

benfred commented 1 year ago

Do you have a reproducing script that you can share ?

Also what operating system, and python version are you using?

steve36871 commented 1 year ago

Apologies, I cannot share the script, since I developed it for my employer. I am using CentOS 9 stream within a Docker container and the host is using Debian. The Python version is 3.9.

Jongy commented 1 year ago

A possible cause, I think, is an app which runs many "active" threads which have no Python context. For example, fully native threads, or threads that run only Cython code. You can try taking a native profile with perf to see accurately what CPU is spending time on (you'll see if it's native code / Cython / Python interpreter/runtime code, etc).

EricCousineau-TRI commented 1 year ago

Do you still see missing samples when using py-spy record --native?