search

benfred / py-spy

Sampling profiler for Python programs
MIT License
12.53k stars 414 forks source link

Support subprocesses with the `dump` command #491

Open benfred opened 2 years ago

benfred commented 2 years ago

We can profile subprocesses with the record or top commands, but not yet with the dump command. We should add support for that as well (see https://twitter.com/StasBekman/status/1525636315936485376 )

stas00 commented 2 years ago

that would be very useful, thank you, @benfred!

and additionally controlling how many generations of ancestry to descend might be useful as well.

e.g. currently I have a need to py-spy the first generation of children and not the grandchildren (which are just helper processes). Here is a pstree output to show what I mean:

$ pstree -lT `whoami`
slurm_script─┬─srun───srun
             └─tee

python─┬─2*[python───60*[python]]
       └─6*[python]

sshd───bash───bash───pstree

So I want just the 2+6 python processes.

The 1 main python process is the pytorch launcher, the 60 grandchildren are helper dataloaders. the hanging usually happens in the 8 children.

Currently extracting this 2nd only generation with:

pgrep -P $(pgrep -o python) | xargs -I {} py-spy dump --pid {}

but, of course, I'm totally fine with the working pgrep solution.

benfred commented 2 years ago

I've got the first part of this done in #492 - this adds support for the subprocesses flag in dump (but not limiting the depth of the subprocesses). This will be in the next release, but in the meantime you can grab the prebuilt wheels from the CI artifacts https://github.com/benfred/py-spy/actions/runs/2333647967#artifacts