Multiprocessing Pool Crash

jadczak commented 3 months ago

Attempting to profile programs with multiprocessing yields the following error:

Error in program being profiled:
 Can't pickle <class 'scalene.replacement_sem_lock.replacement_semlock.ReplacementSemLock'>: attribute lookup replacement_semlock.ReplacementSemLock on scalene.replacement_sem_lock failed

I was able to reproduce the error using the example from https://github.com/plasma-umass/scalene/issues/809#issuecomment-2211984550

import multiprocessing as mp

def fun(args):
    x = 0
    for i in range(100):
        x += 1
    return args

if __name__ == "__main__":
    with mp.Pool(mp.cpu_count()) as p:
        r = p.map(fun, range(1000000))
    print(sum(x for x in r))
    print(((1000000-1) * 1000000) / 2)

Expected behavior No crash?

Screenshots

Desktop (please complete the following information):

OS: Windows 11
Browser Firefox
Version 129
Python: CPython 3.12.3
Scalene: Repository version

Error Text

Error in program being profiled:
 Can't pickle <class 'scalene.replacement_sem_lock.replacement_semlock.ReplacementSemLock'>: attribute lookup replacement_semlock.ReplacementSemLock on scalene.replacement_sem_lock failed
Traceback (most recent call last):
  File "C:\Users\||||||||\scratch\py312\Lib\site-packages\scalene\scalene_profiler.py", line 1791, in profile_code
    exec(code, the_globals, the_locals)
  File "C:\Users\||||||||\scratch\scalene_mp.py", line 10, in <module>
    with mp.Pool(mp.cpu_count()) as p:
         ^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\||||||||\AppData\Local\Programs\Python\Python312\Lib\multiprocessing\context.py", line 119, in Pool
    return Pool(processes, initializer, initargs, maxtasksperchild,
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\||||||||\AppData\Local\Programs\Python\Python312\Lib\multiprocessing\pool.py", line 215, in __init__
    self._repopulate_pool()
  File "C:\Users\||||||||\AppData\Local\Programs\Python\Python312\Lib\multiprocessing\pool.py", line 306, in _repopulate_pool
    return self._repopulate_pool_static(self._ctx, self.Process,
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\||||||||\AppData\Local\Programs\Python\Python312\Lib\multiprocessing\pool.py", line 329, in _repopulate_pool_static
    w.start()
  File "C:\Users\||||||||\AppData\Local\Programs\Python\Python312\Lib\multiprocessing\process.py", line 121, in start
    self._popen = self._Popen(self)
                  ^^^^^^^^^^^^^^^^^
  File "C:\Users\||||||||\AppData\Local\Programs\Python\Python312\Lib\multiprocessing\context.py", line 337, in _Popen
    return Popen(process_obj)
           ^^^^^^^^^^^^^^^^^^
  File "C:\Users\||||||||\AppData\Local\Programs\Python\Python312\Lib\multiprocessing\popen_spawn_win32.py", line 95, in __init__
    reduction.dump(process_obj, to_child)
  File "C:\Users\||||||||\AppData\Local\Programs\Python\Python312\Lib\multiprocessing\reduction.py", line 60, in dump
    ForkingPickler(file, protocol).dump(obj)
_pickle.PicklingError: Can't pickle <class 'scalene.replacement_sem_lock.replacement_semlock.ReplacementSemLock'>: attribute lookup replacement_semlock.ReplacementSemLock on scalene.replacement_sem_lock failed
Scalene: The specified code did not run for long enough to profile.
By default, Scalene only profiles code in the file executed and its subdirectories.
To track the time spent in all files, use the `--profile-all` option.
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "C:\Users\||||||||\AppData\Local\Programs\Python\Python312\Lib\multiprocessing\spawn.py", line 113, in spawn_main
    new_handle = reduction.duplicate(pipe_handle,
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\||||||||\AppData\Local\Programs\Python\Python312\Lib\multiprocessing\reduction.py", line 79, in duplicate
    return _winapi.DuplicateHandle(
           ^^^^^^^^^^^^^^^^^^^^^^^^
OSError: [WinError 6] The handle is invalid

emeryberger commented 3 months ago

Works fine on Mac OS X, repository version of Scalene.
Found an occasional race condition on exit on Ubuntu 20, but not the issue mentioned above (perhaps it's a WSL thing)

jadczak commented 2 months ago

This was on my windows machine not running in WSL. My wild shot in the dark is maybe a fork vs spawn thing since windows doesn't support fork :shrug:

Once I get back to my windows machine I'll give WSL a try and see if I run into similar problems. It doesn't look like python 3.12.4 fixed anything relevant to multiprocessing, but I'll give a few other versions a try to see if there is something funky going on with my particular version of python.

jadczak commented 2 months ago

Tested the following python versions on Windows: 3.12.5 - OSError every time 3.11.9 - OSError every time

Tested the following python version on Clear Linux 3.12.4 - Worked every time tested

Test the following python version WSL 3.11.2 - Frequently hangs. No error when it finishes.

hnguyenHWI commented 1 week ago

I'm seeing something similar. Windows 11 22H2, Python 3.10.2, Scalene version 1.5.45 (2024.10.01)

I was also able to reproduce from #809 (comment).

main.py (stripped down code from the app for a client)

``` import multiprocessing import sys import time import traceback class AppLog: """ Implements configuration and handling of logging for XXXXXX application. Provides the target to be invoked in a separate process.""" def __init__( self, log_queue: multiprocessing.Queue, ) -> None: self.log_queue = log_queue # Configure loggers, filters, handlers, then watches the queue and writes the # queued messages to different files self.__loop_handle_log_queue() def __loop_handle_log_queue(self) -> None: while True: try: # Loop forever, block until a message is put onto the queue by a # QueueHandler from a different process, handle the message and write # it out to the appropriate file time.sleep(1) continue except Exception: print("Logging exception: ", file=sys.stderr) traceback.print_exc(file=sys.stderr) def __log_proc( log_queue: multiprocessing.Queue, ): AppLog(log_queue=log_queue) def start_app_logging_process( log_queue: multiprocessing.Queue, ) -> multiprocessing.Process: """Target for logging process which will log to application-wide file.""" process = multiprocessing.Process( target=__log_proc, args=(log_queue,), ) process.start() return process def main_wrapper(): """Wraps the call to main so that we can use the Click library \ (e.g. for option flags).""" # Now launching the app, show splash screen until launcher is initialized log_queue = multiprocessing.Queue(-1) log_process = start_app_logging_process(log_queue=log_queue) # Have started logging queue, move on to the rest of the application: # open more processes, do other stuff def func_issue_809(): def fun(args): x = 0 for i in range(100): x += 1 return args with multiprocessing.Pool(multiprocessing.cpu_count()) as p: r = p.map(fun, range(1000000)) print(sum(x for x in r)) print(((1000000-1) * 1000000) / 2) if __name__ == "__main__": # disable pylint, need this wrapper so Click will work func_issue_809() main_wrapper() # pylint:disable=no-value-for-parameter ```

Error output from the code from 809

``` (venv) PS C:\Users\USER\ORGANIZATION\customer\CLIENT1\CLIENT_PROJECT\scalene\PROJECT_ROOT> scalene --version Scalene version 1.5.45 (2024.10.01) (venv) PS C:\Users\USER\ORGANIZATION\customer\CLIENT1\CLIENT_PROJECT\scalene\PROJECT_ROOT> scalene .\scalene_minimal\main.py Error in program being profiled: Can't pickle : attribute lookup replacement_semlock.ReplacementSemLock on scalene.replacement_sem_lock failed Traceback (most recent call last): File "C:\Users\USER\ORGANIZATION\customer\CLIENT1\CLIENT_PROJECT\scalene\PROJECT_ROOT\venv\lib\site-packages\scalene\scalene_profiler.py", line 1795, in profile_code exec(code, the_globals, the_locals) File "C:\Users\USER\ORGANIZATION\customer\CLIENT1\CLIENT_PROJECT\scalene\PROJECT_ROOT\scalene_minimal\main.py", line 79, in func_issue_809() File "C:\Users\USER\ORGANIZATION\customer\CLIENT1\CLIENT_PROJECT\scalene\PROJECT_ROOT\scalene_minimal\main.py", line 72, in func_issue_809 with multiprocessing.Pool(multiprocessing.cpu_count()) as p: File "C:\Users\USER\AppData\Local\Programs\Python\Python310\lib\multiprocessing\context.py", line 119, in Pool return Pool(processes, initializer, initargs, maxtasksperchild, File "C:\Users\USER\AppData\Local\Programs\Python\Python310\lib\multiprocessing\pool.py", line 212, in __init__ self._repopulate_pool() File "C:\Users\USER\AppData\Local\Programs\Python\Python310\lib\multiprocessing\pool.py", line 303, in _repopulate_pool return self._repopulate_pool_static(self._ctx, self.Process, File "C:\Users\USER\AppData\Local\Programs\Python\Python310\lib\multiprocessing\pool.py", line 326, in _repopulate_pool_static w.start() File "C:\Users\USER\AppData\Local\Programs\Python\Python310\lib\multiprocessing\process.py", line 121, in start self._popen = self._Popen(self) File "C:\Users\USER\AppData\Local\Programs\Python\Python310\lib\multiprocessing\context.py", line 327, in _Popen return Popen(process_obj) File "C:\Users\USER\AppData\Local\Programs\Python\Python310\lib\multiprocessing\popen_spawn_win32.py", line 93, in __init__ reduction.dump(process_obj, to_child) File "C:\Users\USER\AppData\Local\Programs\Python\Python310\lib\multiprocessing\reduction.py", line 60, in dump ForkingPickler(file, protocol).dump(obj) _pickle.PicklingError: Can't pickle : attribute lookup replacement_semlock.ReplacementSemLock on scalene.replacement_sem_lock failed Traceback (most recent call last): File "", line 1, in File "C:\Users\USER\AppData\Local\Programs\Python\Python310\Lib\multiprocessing\spawn.py", line 107, in spawn_main new_handle = reduction.duplicate(pipe_handle, File "C:\Users\USER\AppData\Local\Programs\Python\Python310\Lib\multiprocessing\reduction.py", line 79, in duplicate return _winapi.DuplicateHandle( OSError: [WinError 6] The handle is invalid ```

Error output from my minimal code, main_wrapper() function in main.py above

``` (venv) PS C:\Users\USER\ORGANIZATION\customer\CLIENT1\CLIENT_PROJECT\scalene\PROJECT_ROOT> scalene .\scalene_minimal\main.py Error in program being profiled: Can't pickle : attribute lookup replacement_semlock.ReplacementSemLock on scalene.replacement_sem_lock failed Traceback (most recent call last): File "C:\Users\USER\ORGANIZATION\customer\CLIENT1\CLIENT_PROJECT\scalene\PROJECT_ROOT\venv\lib\site-packages\scalene\scalene_profiler.py", line 1795, in profile_code exec(code, the_globals, the_locals) File "C:\Users\USER\ORGANIZATION\customer\CLIENT1\CLIENT_PROJECT\scalene\PROJECT_ROOT\scalene_minimal\main.py", line 67, in main_wrapper() # pylint:disable=no-value-for-parameter File "C:\Users\USER\ORGANIZATION\customer\CLIENT1\CLIENT_PROJECT\scalene\PROJECT_ROOT\scalene_minimal\main.py", line 60, in main_wrapper log_process = start_app_logging_process(log_queue=log_queue) File "C:\Users\USER\ORGANIZATION\customer\CLIENT1\CLIENT_PROJECT\scalene\PROJECT_ROOT\scalene_minimal\main.py", line 49, in start_app_logging_process process.start() File "C:\Users\USER\AppData\Local\Programs\Python\Python310\lib\multiprocessing\process.py", line 121, in start self._popen = self._Popen(self) File "C:\Users\USER\AppData\Local\Programs\Python\Python310\lib\multiprocessing\context.py", line 224, in _Popen return _default_context.get_context().Process._Popen(process_obj) File "C:\Users\USER\AppData\Local\Programs\Python\Python310\lib\multiprocessing\context.py", line 327, in _Popen return Popen(process_obj) File "C:\Users\USER\AppData\Local\Programs\Python\Python310\lib\multiprocessing\popen_spawn_win32.py", line 93, in __init__ reduction.dump(process_obj, to_child) File "C:\Users\USER\AppData\Local\Programs\Python\Python310\lib\multiprocessing\reduction.py", line 60, in dump ForkingPickler(file, protocol).dump(obj) _pickle.PicklingError: Can't pickle : attribute lookup replacement_semlock.ReplacementSemLock on scalene.replacement_sem_lock failed Traceback (most recent call last): File "", line 1, in File "C:\Users\USER\AppData\Local\Programs\Python\Python310\Lib\multiprocessing\spawn.py", line 107, in spawn_main new_handle = reduction.duplicate(pipe_handle, File "C:\Users\USER\AppData\Local\Programs\Python\Python310\Lib\multiprocessing\reduction.py", line 79, in duplicate return _winapi.DuplicateHandle( OSError: [WinError 6] The handle is invalid ```

plasma-umass / scalene

Multiprocessing Pool Crash #846