Scalene does not seem to work with multiprocessing

[makmanalp]

Describe the bug

Scalene does not seem to collect profiling data from subprocesses forked by multiprocessing.

To Reproduce

Run this:

import multiprocessing as mp

def fun(args):
    return args

if __name__ == "__main__":
    p = mp.Pool(mp.cpu_count())
    r = p.map(fun, range(1000000))
    print(sum(x for x in r))
    print(((1000000-1) * 1000000) / 2)

in scalene via python3 -m scalene scalenetest.py

Expected behavior

Expect to see line profiler show stats from inside the function fun(), gathered from the subprocesses. What happens instead is that I get a profile that shows that the bulk of the time is spent in the main process waiting for the subprocesses to complete.

Screenshots

Desktop (please complete the following information):

• OS: OSX Ventura 13.6.6
• Python: Python 3.12.2
• Scalene: Scalene version 1.5.38 (2024.03.24) - also tried repository version

Additional context Add any other context about the problem here.

[ogencoglu]

Same issue here

[emeryberger]

Thanks for the note. I dug in and there are basically two problems with this particular example:

1. The function is empty, so it spends basically no time in the function. It's not a huge surprise that the sampling doesn't see execution in the function.
2. The pool is not properly being closed, so its statistics don't get properly propagated to the main Scalene profiler process. When using it correctly (e.g., in a context manager), the results look fine.

I've modified your code to spend some compute time in fun and also to use a context manager for Pool so it cleans up after itself.

import multiprocessing as mp

def fun(args):
    x = 0
    for i in range(100):
        x += 1
    return args

if __name__ == "__main__":
    with mp.Pool(mp.cpu_count()) as p:
        r = p.map(fun, range(1000000))
    print(sum(x for x in r))
    print(((1000000-1) * 1000000) / 2)

This is the result using the repository version (on a Mac running Sonoma, though I don't think it should matter). As you can see, most of the time is now correctly attributed to code in fun.