search

pyutils / line_profiler

Line-by-line profiling for Python
Other
2.56k stars 118 forks source link

Support for Multi-Processing #248

Closed EadCat closed 8 months ago

EadCat commented 8 months ago

Hello. Thank you for your efforts on line_profiler.

I have a question if line_profiler supports Python Multi-Processing (e.g. mp.spawn) because I'm going to use PyTorch DistributedDataParallel(DDP).

If possible, How can I use it?

Thank you.

Erotemic commented 8 months ago

I don't believe that we can track and unify profiling measurements across multiple processes.

I suggest looking into other profilers like scalene, which is better at holistic resource analysis. Once you have a rough idea of where slowness is happening, line-profiler is a great way to dig into performance when you can isolate an example in a single threaded process.

I'm not the original author of the library, so I'm not sure how / if it could be extended to support multiprocessing, but I did write a test to check if line-profiler works under various circumstances.

https://github.com/pyutils/line_profiler/blob/main/tests/complex_example.py

Running:

PROFILE_TYPE=explicit LINE_PROFILE=1  python complex_example.py --serial_size 0 --thread_size 0 --process_size 100
python -m line_profiler -rtmz profile_output.lprof

Shows that anything outside of the main process is ignored.

However: using thread-level parallelism does at least report results, although I can't make claims for their correctness. In this case it looks fine, but I'm not sure if we handle threads well or not. 

PROFILE_TYPE=explicit LINE_PROFILE=1  python complex_example.py --serial_size 0 --thread_size 1000 --process_size 0
python -m line_profiler -rtmz profile_output.lprof
Theelx commented 8 months ago

It's almost impossible as far as I know for a callback-based profiler to measure multiple processes. It's significantly easier for sampling profilers such as py-spy and scalene to do this, so I'd have to agree with Erotemic that this isn't the right project for that at this moment.

EadCat commented 8 months ago

I see. I'm not going to use thread-level, so adopting another profiler would be the better way. Thanks to @Erotemic and @Theelx for the recommending another profiler! I will investigate them.