Multiprocessing error during import

SeanZhang99 commented 1 month ago

First of all, this error occurs during the import of a brain_pipe module "brain_pipe.preprocessing.brain.link". However, the script using this module is in your another repo. I'm not sure if it is appropriate to post the issue here, but I think the two repos are all maintained by you guys so I just add this issue in the repo where an error occur.

And here is the description:

Windows 10. Conda environment list results: $ conda list packages in environment at C:\Users\sean\miniconda3\envs\brain-pip: Name Version Build Channel audioread 3.0.1 pypi_0 pypi brain-pipe 0.0.4 pypi_0 pypi brian2 2.7.1 pypi_0 pypi brian2hears 0.9.2 pypi_0 pypi bzip2 1.0.8 h2bbff1b_6 ca-certificates 2024.7.2 haa95532_0 certifi 2024.7.4 pypi_0 pypi cffi 1.17.0 pypi_0 pypi charset-normalizer 3.3.2 pypi_0 pypi colorama 0.4.6 pypi_0 pypi contourpy 1.2.1 pypi_0 pypi cycler 0.12.1 pypi_0 pypi cython 3.0.11 pypi_0 pypi decorator 5.1.1 pypi_0 pypi expat 2.6.2 hd77b12b_0 fonttools 4.53.1 pypi_0 pypi idna 3.7 pypi_0 pypi jinja2 3.1.4 pypi_0 pypi joblib 1.4.2 pypi_0 pypi kiwisolver 1.4.5 pypi_0 pypi lazy-loader 0.4 pypi_0 pypi libffi 3.4.4 hd77b12b_1 librosa 0.10.2.post1 pypi_0 pypi llvmlite 0.43.0 pypi_0 pypi markupsafe 2.1.5 pypi_0 pypi matplotlib 3.9.1.post1 pypi_0 pypi mne 1.7.1 pypi_0 pypi mpmath 1.3.0 pypi_0 pypi msgpack 1.0.8 pypi_0 pypi numba 0.60.0 pypi_0 pypi numpy 2.0.1 pypi_0 pypi openssl 3.0.14 h827c3e9_0 packaging 24.1 pypi_0 pypi pillow 10.4.0 pypi_0 pypi pip 24.0 py312haa95532_0 platformdirs 4.2.2 pypi_0 pypi pooch 1.8.2 pypi_0 pypi py-cpuinfo 9.0.0 pypi_0 pypi pycparser 2.22 pypi_0 pypi pyparsing 3.1.2 pypi_0 pypi python 3.12.4 h14ffc60_1 python-dateutil 2.9.0.post0 pypi_0 pypi pyyaml 6.0.2 pypi_0 pypi requests 2.32.3 pypi_0 pypi scikit-learn 1.5.1 pypi_0 pypi scipy 1.14.0 pypi_0 pypi setuptools 72.1.0 py312haa95532_0 six 1.16.0 pypi_0 pypi soundfile 0.12.1 pypi_0 pypi soxr 0.4.0 pypi_0 pypi sqlite 3.45.3 h2bbff1b_0 sympy 1.13.1 pypi_0 pypi threadpoolctl 3.5.0 pypi_0 pypi tk 8.6.14 h0416ee5_0 tqdm 4.66.5 pypi_0 pypi typing-extensions 4.12.2 pypi_0 pypi tzdata 2024a h04d1e81_0 urllib3 2.2.2 pypi_0 pypi vc 14.2 h2eaa2aa_4 vs2015_runtime 14.29.30133 h43f2093_4 wheel 0.43.0 py312haa95532_0 xz 5.4.6 h8cc25b3_1 zlib 1.2.13 h8cc25b3_1

(VS Code) run python files in terminal

(brain-pip) parent_folder\training>C:/Users/sean/miniconda3/envs/brain-pip/python.exe c:/Users/sean/Downloads/auditory-eeg-challenge-2024-code-main/preprocessing_code/sparrKULee.py
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "C:\Users\sean\miniconda3\envs\brain-pip\Lib\multiprocessing\spawn.py", line 122, in spawn_main
    exitcode = _main(fd, parent_sentinel)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\sean\miniconda3\envs\brain-pip\Lib\multiprocessing\spawn.py", line 131, in _main
    prepare(preparation_data)
  File "C:\Users\sean\miniconda3\envs\brain-pip\Lib\multiprocessing\spawn.py", line 246, in prepare
    _fixup_main_from_path(data['init_main_from_path'])
  File "C:\Users\sean\miniconda3\envs\brain-pip\Lib\multiprocessing\spawn.py", line 297, in _fixup_main_from_path
    main_content = runpy.run_path(main_path,
                   ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "<frozen runpy>", line 286, in run_path
  File "<frozen runpy>", line 98, in _run_module_code
  File "<frozen runpy>", line 88, in _run_code
  File "c:\Users\sean\Downloads\auditory-eeg-challenge-2024-code-main\preprocessing_code\sparrKULee.py", line 26, in <module>
    from brain_pipe.preprocessing.brain.link import (
  File "C:\Users\sean\miniconda3\envs\brain-pip\Lib\site-packages\brain_pipe\preprocessing\brain\link.py", line 9, in <module>
    from brain_pipe.utils.multiprocess import MultiprocessingSingleton
  File "C:\Users\sean\miniconda3\envs\brain-pip\Lib\site-packages\brain_pipe\utils\multiprocess.py", line 45, in <module>
    class MultiprocessingSingleton:
  File "C:\Users\sean\miniconda3\envs\brain-pip\Lib\site-packages\brain_pipe\utils\multiprocess.py", line 48, in MultiprocessingSingleton
    manager = multiprocessing.Manager()
              ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\sean\miniconda3\envs\brain-pip\Lib\multiprocessing\context.py", line 57, in Manager
    m.start()
  File "C:\Users\sean\miniconda3\envs\brain-pip\Lib\multiprocessing\managers.py", line 562, in start
    self._process.start()
  File "C:\Users\sean\miniconda3\envs\brain-pip\Lib\multiprocessing\process.py", line 121, in start
    self._popen = self._Popen(self)
                  ^^^^^^^^^^^^^^^^^
  File "C:\Users\sean\miniconda3\envs\brain-pip\Lib\multiprocessing\context.py", line 337, in _Popen
    return Popen(process_obj)
           ^^^^^^^^^^^^^^^^^^
  File "C:\Users\sean\miniconda3\envs\brain-pip\Lib\multiprocessing\popen_spawn_win32.py", line 46, in __init__
    prep_data = spawn.get_preparation_data(process_obj._name)
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\sean\miniconda3\envs\brain-pip\Lib\multiprocessing\spawn.py", line 164, in get_preparation_data
    _check_not_importing_main()
  File "C:\Users\sean\miniconda3\envs\brain-pip\Lib\multiprocessing\spawn.py", line 140, in _check_not_importing_main
    raise RuntimeError('''
RuntimeError:
        An attempt has been made to start a new process before the
        current process has finished its bootstrapping phase.

        This probably means that you are not using fork to start your
        child processes and you have forgotten to use the proper idiom
        in the main module:

            if __name__ == '__main__':
                freeze_support()
                ...

        The "freeze_support()" line can be omitted if the program
        is not going to be frozen to produce an executable.

        To fix this issue, refer to the "Safe importing of main module"
        section in https://docs.python.org/3/library/multiprocessing.html

And here are all things I've met. The preprocessing codes are provided as in the 85 subject dataset. I just download the repo and the dataset and directly run the script, and the error occurs.

My analysis: it seems that the error occurs during the import of necessary modules, and the main code is not executed yet. The code that direcly raise the error is

class MultiprocessingSingleton:
    """Singleton class for multiprocessing."""

    manager = multiprocessing.Manager()

And the RuntimeError said the error occurs if the start mp method is not fork (but I can only use spawn in Windows, it does not support fork.) I wonder if you had complete the code writing and testing in a Unix or MacOS machine? And is there any way to skip this error to normally execute the preprocessing pipeline on a windows PC?

Let me know if you wish to find out more informations.

berndie commented 3 weeks ago

Hi, This seems to be a Windows-related problem with the multiprocessing package. All tests and code writing were done on Unix-like machines, so I'm unsure how the package performs on Windows.

I will try to implement a quick fix and adjust our CI pipeline to include Windows machines.

SeanZhang99 commented 3 weeks ago

Dear Berndie, Thanks for your rapid reply. I've tried the codes on a WSL 1 Ubuntu and they work well. It should be the 'spawn' method (the default and only supported way of generating new process on Windows 10/11) of multiprocess which causes the problem. BTW I found an unlocated memory leakage when mp on AMD 7900 and it quickly drawn out my RAM (96GB). Still I can run the code on single process mode and it does not take much time for me waiting the preprocess finished. Ag I would like to express my appreciate for your quick response. If you need further information about the issue or the memory leakage problem, plz feel free to contact me and I'm glad to provide you with additional logs or debug info. Yours, Yuanming Zhang.

@.***

From: berndie Date: 2024-08-29 20:15 To: exporl/brain_pipe CC: SeanZhang99; Author Subject: Re: [exporl/brain_pipe] Multiprocessing error during import (Issue #7) Hi, This seems to be a Windows-related problem with the multiprocessing package. All tests and code writing were done on Unix-like machines, so I'm unsure how the package performs on Windows. I will try to implement a quick fix and adjust our CI pipeline to include Windows machines. — Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you authored the thread.Message ID: @.***>

exporl / brain_pipe

Multiprocessing error during import #7