MacOS multiprocess issues

cburgess commented 2 years ago

If you attempt to use soufi on MacOS you will hit multiprocess issues. As an example I created the following simply test script.

#!/usr/bin/env python

import pylru
import soufi

LRU_CACHE = pylru.lrucache(size=1024)
finder = soufi.finder.factory('centos',
                              'vim-minimal',
                              '7.4.629-8.el7_9',
                              soufi.finder.SourceType.os,
                              cache_backend='dogpile.cache.memory',
                              cache_args=dict(cache_dict=LRU_CACHE))

print(finder.find())

On my Mac I have installed python 3.10.3 via pyenv. If I setup a venv with the latest soufi and then try and run the above script (and the fix for #29) I get an error about calling freeze_support() which is a windows only thing, but the error is coming from multiprocessing/spawn.py. MacOS and Windows share the same start method (spawn, see https://docs.python.org/3/library/multiprocessing.html) so I suspect the issue is actually related to the use of spawn vs fork. Note this script works fine on Linux.

cfb@sandman:ccs/> python -m venv venv
cfb@sandman:ccs/> venv/bin/python --version
Python 3.10.3
cfb@sandman:ccs/> venv/bin/pip install -U pip setuptools wheel
Requirement already satisfied: pip in ./venv/lib/python3.10/site-packages (22.0.4)
Collecting pip
  Downloading pip-22.1.2-py3-none-any.whl (2.1 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 2.1/2.1 MB 18.2 MB/s eta 0:00:00
Requirement already satisfied: setuptools in ./venv/lib/python3.10/site-packages (58.1.0)
Collecting setuptools
  Downloading setuptools-62.3.2-py3-none-any.whl (1.2 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 1.2/1.2 MB 15.2 MB/s eta 0:00:00
Collecting wheel
  Using cached wheel-0.37.1-py2.py3-none-any.whl (35 kB)
Installing collected packages: wheel, setuptools, pip
  Attempting uninstall: setuptools
    Found existing installation: setuptools 58.1.0
    Uninstalling setuptools-58.1.0:
      Successfully uninstalled setuptools-58.1.0
  Attempting uninstall: pip
    Found existing installation: pip 22.0.4
    Uninstalling pip-22.0.4:
      Successfully uninstalled pip-22.0.4
Successfully installed pip-22.1.2 setuptools-62.3.2 wheel-0.37.1
cfb@sandman:ccs/> venv/bin/pip install soufi
Collecting soufi
  Using cached soufi-2022.3.14-py2.py3-none-any.whl (37 kB)
Collecting lxml>=4.6.3
  Using cached lxml-4.9.0-cp310-cp310-macosx_10_15_x86_64.whl (4.6 MB)
Collecting click>=7.1.2
  Using cached click-8.1.3-py3-none-any.whl (96 kB)
Collecting dogpile.cache>=1.1.5
  Using cached dogpile.cache-1.1.5-py3-none-any.whl (51 kB)
Collecting requests>=2.25.1
  Using cached requests-2.27.1-py2.py3-none-any.whl (63 kB)
Collecting cryptography>=3.3.1
  Using cached cryptography-37.0.2-cp36-abi3-macosx_10_10_x86_64.whl (2.8 MB)
Collecting pylru>=1.2.0
  Using cached pylru-1.2.1-py3-none-any.whl (16 kB)
Collecting requests-oauthlib>=1.3.0
  Using cached requests_oauthlib-1.3.1-py2.py3-none-any.whl (23 kB)
Collecting launchpadlib>=1.10.13
  Using cached launchpadlib-1.10.16-py2.py3-none-any.whl (216 kB)
Collecting repomd>=0.2.1
  Using cached repomd-0.2.1-py3-none-any.whl (3.9 kB)
Collecting cffi>=1.12
  Using cached cffi-1.15.0-cp310-cp310-macosx_10_9_x86_64.whl (178 kB)
Collecting decorator>=4.0.0
  Using cached decorator-5.1.1-py3-none-any.whl (9.1 kB)
Collecting stevedore>=3.0.0
  Using cached stevedore-3.5.0-py3-none-any.whl (49 kB)
Collecting httplib2
  Using cached httplib2-0.20.4-py3-none-any.whl (96 kB)
Collecting keyring
  Using cached keyring-23.5.1-py3-none-any.whl (33 kB)
Collecting six
  Using cached six-1.16.0-py2.py3-none-any.whl (11 kB)
Collecting lazr.restfulclient>=0.9.19
  Using cached lazr.restfulclient-0.14.4.tar.gz (62 kB)
  Preparing metadata (setup.py) ... done
Collecting lazr.uri
  Using cached lazr.uri-1.0.6.tar.gz (18 kB)
  Preparing metadata (setup.py) ... done
Collecting defusedxml
  Using cached defusedxml-0.7.1-py2.py3-none-any.whl (25 kB)
Collecting charset-normalizer~=2.0.0
  Using cached charset_normalizer-2.0.12-py3-none-any.whl (39 kB)
Collecting idna<4,>=2.5
  Using cached idna-3.3-py3-none-any.whl (61 kB)
Collecting certifi>=2017.4.17
  Using cached certifi-2022.5.18.1-py3-none-any.whl (155 kB)
Collecting urllib3<1.27,>=1.21.1
  Using cached urllib3-1.26.9-py2.py3-none-any.whl (138 kB)
Collecting oauthlib>=3.0.0
  Using cached oauthlib-3.2.0-py3-none-any.whl (151 kB)
Collecting pycparser
  Using cached pycparser-2.21-py2.py3-none-any.whl (118 kB)
Collecting distro
  Using cached distro-1.7.0-py3-none-any.whl (20 kB)
Requirement already satisfied: setuptools in ./venv/lib/python3.10/site-packages (from lazr.restfulclient>=0.9.19->launchpadlib>=1.10.13->soufi) (62.3.2)
Collecting wadllib>=1.1.4
  Using cached wadllib-1.3.6.tar.gz (62 kB)
  Preparing metadata (setup.py) ... done
Collecting pyparsing!=3.0.0,!=3.0.1,!=3.0.2,!=3.0.3,<4,>=2.4.2
  Using cached pyparsing-3.0.9-py3-none-any.whl (98 kB)
Collecting pbr!=2.1.0,>=2.0.0
  Using cached pbr-5.9.0-py2.py3-none-any.whl (112 kB)
Collecting importlib-metadata>=3.6
  Using cached importlib_metadata-4.11.4-py3-none-any.whl (18 kB)
Collecting zipp>=0.5
  Using cached zipp-3.8.0-py3-none-any.whl (5.4 kB)
Building wheels for collected packages: lazr.restfulclient, lazr.uri, wadllib
  Building wheel for lazr.restfulclient (setup.py) ... done
  Created wheel for lazr.restfulclient: filename=lazr.restfulclient-0.14.4-py3-none-any.whl size=70795 sha256=05572584d048c23747a4c6fa01343e8de7e8b8f8e158e73bffd04fde053a5480
  Stored in directory: /Users/cfb/Library/Caches/pip/wheels/bb/9c/51/fcd209a75c812670370bd6013e0746b947f4baa0242c342029
  Building wheel for lazr.uri (setup.py) ... done
  Created wheel for lazr.uri: filename=lazr.uri-1.0.6-py3-none-any.whl size=20362 sha256=cf33027a6d722b77c556af313a1a000291350c1a7b3bbae110ed5b10f7657f16
  Stored in directory: /Users/cfb/Library/Caches/pip/wheels/cb/27/9a/232fcb7a33bce3386e70db9f87f9cbe67d4dded43faff30129
  Building wheel for wadllib (setup.py) ... done
  Created wheel for wadllib: filename=wadllib-1.3.6-py3-none-any.whl size=58280 sha256=96c0e83bf3f065f9216f00ec3a388b904d4ea7bfee895fa975f3ede779f98258
  Stored in directory: /Users/cfb/Library/Caches/pip/wheels/70/14/54/863a40409358975621e53fee1fe000478845aff98c1a9b8e4f
Successfully built lazr.restfulclient lazr.uri wadllib
Installing collected packages: pylru, zipp, urllib3, six, pyparsing, pycparser, pbr, oauthlib, lxml, lazr.uri, idna, distro, defusedxml, decorator, click, charset-normalizer, certifi, wadllib, stevedore, requests, repomd, importlib-metadata, httplib2, cffi, requests-oauthlib, lazr.restfulclient, keyring, dogpile.cache, cryptography, launchpadlib, soufi
Successfully installed certifi-2022.5.18.1 cffi-1.15.0 charset-normalizer-2.0.12 click-8.1.3 cryptography-37.0.2 decorator-5.1.1 defusedxml-0.7.1 distro-1.7.0 dogpile.cache-1.1.5 httplib2-0.20.4 idna-3.3 importlib-metadata-4.11.4 keyring-23.5.1 launchpadlib-1.10.16 lazr.restfulclient-0.14.4 lazr.uri-1.0.6 lxml-4.9.0 oauthlib-3.2.0 pbr-5.9.0 pycparser-2.21 pylru-1.2.1 pyparsing-3.0.9 repomd-0.2.1 requests-2.27.1 requests-oauthlib-1.3.1 six-1.16.0 soufi-2022.3.14 stevedore-3.5.0 urllib3-1.26.9 wadllib-1.3.6 zipp-3.8.0
cfb@sandman:ccs/> venv/bin/python find_packages.py
Traceback (most recent call last):
  File "/Users/cfb/cisco/scratch/ccs/venv/lib/python3.10/site-packages/soufi/finders/yum.py", line 92, in get_source_url
    source_name, source_ver = self._walk_binary_repos(self.name)
  File "/Users/cfb/cisco/scratch/ccs/venv/lib/python3.10/site-packages/soufi/finders/yum.py", line 144, in _walk_binary_repos
    baseurl, repo_xml = self._cache.get_or_create(
  File "/Users/cfb/cisco/scratch/ccs/venv/lib/python3.10/site-packages/dogpile/cache/region.py", line 1042, in get_or_create
    with Lock(
  File "/Users/cfb/cisco/scratch/ccs/venv/lib/python3.10/site-packages/dogpile/lock.py", line 185, in __enter__
    return self._enter()
  File "/Users/cfb/cisco/scratch/ccs/venv/lib/python3.10/site-packages/dogpile/lock.py", line 94, in _enter
    generated = self._enter_create(value, createdtime)
  File "/Users/cfb/cisco/scratch/ccs/venv/lib/python3.10/site-packages/dogpile/lock.py", line 178, in _enter_create
    return self.creator()
  File "/Users/cfb/cisco/scratch/ccs/venv/lib/python3.10/site-packages/dogpile/cache/region.py", line 995, in gen_value
    created_value = creator(
  File "/Users/cfb/cisco/scratch/ccs/venv/lib/python3.10/site-packages/soufi/finders/yum.py", line 233, in do_task
    process.start()
  File "/Users/cfb/.pyenv/versions/3.10.3/lib/python3.10/multiprocessing/process.py", line 121, in start
    self._popen = self._Popen(self)
  File "/Users/cfb/.pyenv/versions/3.10.3/lib/python3.10/multiprocessing/context.py", line 224, in _Popen
    return _default_context.get_context().Process._Popen(process_obj)
  File "/Users/cfb/.pyenv/versions/3.10.3/lib/python3.10/multiprocessing/context.py", line 284, in _Popen
    return Popen(process_obj)
  File "/Users/cfb/.pyenv/versions/3.10.3/lib/python3.10/multiprocessing/popen_spawn_posix.py", line 32, in __init__
    super().__init__(process_obj)
  File "/Users/cfb/.pyenv/versions/3.10.3/lib/python3.10/multiprocessing/popen_fork.py", line 19, in __init__
    self._launch(process_obj)
  File "/Users/cfb/.pyenv/versions/3.10.3/lib/python3.10/multiprocessing/popen_spawn_posix.py", line 42, in _launch
    prep_data = spawn.get_preparation_data(process_obj._name)
  File "/Users/cfb/.pyenv/versions/3.10.3/lib/python3.10/multiprocessing/spawn.py", line 154, in get_preparation_data
    _check_not_importing_main()
  File "/Users/cfb/.pyenv/versions/3.10.3/lib/python3.10/multiprocessing/spawn.py", line 134, in _check_not_importing_main
    raise RuntimeError('''
RuntimeError:
        An attempt has been made to start a new process before the
        current process has finished its bootstrapping phase.

        This probably means that you are not using fork to start your
        child processes and you have forgotten to use the proper idiom
        in the main module:

            if __name__ == '__main__':
                freeze_support()
                ...

        The "freeze_support()" line can be omitted if the program
        is not going to be frozen to produce an executable.

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/Users/cfb/.pyenv/versions/3.10.3/lib/python3.10/multiprocessing/spawn.py", line 116, in spawn_main
    exitcode = _main(fd, parent_sentinel)
  File "/Users/cfb/.pyenv/versions/3.10.3/lib/python3.10/multiprocessing/spawn.py", line 125, in _main
    prepare(preparation_data)
  File "/Users/cfb/.pyenv/versions/3.10.3/lib/python3.10/multiprocessing/spawn.py", line 236, in prepare
    _fixup_main_from_path(data['init_main_from_path'])
  File "/Users/cfb/.pyenv/versions/3.10.3/lib/python3.10/multiprocessing/spawn.py", line 287, in _fixup_main_from_path
    main_content = runpy.run_path(main_path,
  File "/Users/cfb/.pyenv/versions/3.10.3/lib/python3.10/runpy.py", line 269, in run_path
    return _run_module_code(code, init_globals, run_name,
  File "/Users/cfb/.pyenv/versions/3.10.3/lib/python3.10/runpy.py", line 96, in _run_module_code
    _run_code(code, mod_globals, init_globals,
  File "/Users/cfb/.pyenv/versions/3.10.3/lib/python3.10/runpy.py", line 86, in _run_code
    exec(code, run_globals)
  File "/Users/cfb/cisco/scratch/ccs/find_packages.py", line 14, in <module>
    print(finder.find())
  File "/Users/cfb/cisco/scratch/ccs/venv/lib/python3.10/site-packages/soufi/finder.py", line 228, in find
    return self._find()
  File "/Users/cfb/cisco/scratch/ccs/venv/lib/python3.10/site-packages/soufi/finders/yum.py", line 74, in _find
    source_url = self.get_source_url()
  File "/Users/cfb/cisco/scratch/ccs/venv/lib/python3.10/site-packages/soufi/finders/yum.py", line 94, in get_source_url
    raise exceptions.DownloadError from e
soufi.exceptions.DownloadError
^CTraceback (most recent call last):
  File "/Users/cfb/cisco/scratch/ccs/find_packages.py", line 14, in <module>
    print(finder.find())
  File "/Users/cfb/cisco/scratch/ccs/venv/lib/python3.10/site-packages/soufi/finder.py", line 228, in find
    return self._find()
  File "/Users/cfb/cisco/scratch/ccs/venv/lib/python3.10/site-packages/soufi/finders/yum.py", line 74, in _find
    source_url = self.get_source_url()
  File "/Users/cfb/cisco/scratch/ccs/venv/lib/python3.10/site-packages/soufi/finders/yum.py", line 92, in get_source_url
    source_name, source_ver = self._walk_binary_repos(self.name)
  File "/Users/cfb/cisco/scratch/ccs/venv/lib/python3.10/site-packages/soufi/finders/yum.py", line 144, in _walk_binary_repos
    baseurl, repo_xml = self._cache.get_or_create(
  File "/Users/cfb/cisco/scratch/ccs/venv/lib/python3.10/site-packages/dogpile/cache/region.py", line 1042, in get_or_create
    with Lock(
  File "/Users/cfb/cisco/scratch/ccs/venv/lib/python3.10/site-packages/dogpile/lock.py", line 185, in __enter__
    return self._enter()
  File "/Users/cfb/cisco/scratch/ccs/venv/lib/python3.10/site-packages/dogpile/lock.py", line 94, in _enter
    generated = self._enter_create(value, createdtime)
  File "/Users/cfb/cisco/scratch/ccs/venv/lib/python3.10/site-packages/dogpile/lock.py", line 178, in _enter_create
    return self.creator()
  File "/Users/cfb/cisco/scratch/ccs/venv/lib/python3.10/site-packages/dogpile/cache/region.py", line 995, in gen_value
    created_value = creator(
  File "/Users/cfb/cisco/scratch/ccs/venv/lib/python3.10/site-packages/soufi/finders/yum.py", line 236, in do_task
    response = queue.get(timeout=600)
  File "/Users/cfb/.pyenv/versions/3.10.3/lib/python3.10/multiprocessing/queues.py", line 113, in get
    if not self._poll(timeout):
  File "/Users/cfb/.pyenv/versions/3.10.3/lib/python3.10/multiprocessing/connection.py", line 262, in poll
    return self._poll(timeout)
  File "/Users/cfb/.pyenv/versions/3.10.3/lib/python3.10/multiprocessing/connection.py", line 429, in _poll
    r = wait([self], timeout)
  File "/Users/cfb/.pyenv/versions/3.10.3/lib/python3.10/multiprocessing/connection.py", line 936, in wait
    ready = selector.select(timeout)
  File "/Users/cfb/.pyenv/versions/3.10.3/lib/python3.10/selectors.py", line 416, in select
    fd_event_list = self._selector.poll(timeout)
KeyboardInterrupt

juledwar commented 2 years ago

Confirming that this works fine on Linux with Python 3.10, so it does appear to be a MacOS thing.

juledwar commented 2 years ago

Given that I don't have a Mac, I'll have to defer to @0xDEC0DE on this one.

0xDEC0DE commented 2 years ago

I am able to reproduce the error with both Python 3.8.3 and 3.10.4 on MacOS.

A cursory web search infers that multiprocessing has some explicit protection against being run in a global scope, and indeed, this version of the script runs without complaint:

#!/usr/bin/env python

import pylru
import soufi

LRU_CACHE = pylru.lrucache(size=1024)

def main():
    finder = soufi.finder.factory('centos',
                                  'vim-minimal',
                                  '7.4.629-8.el7_9',
                                  soufi.finder.SourceType.os,
                                  cache_backend='dogpile.cache.memory',
                                  cache_args=dict(cache_dict=LRU_CACHE))
    print(finder.find())

if __name__ == '__main__':
    main()

Soufi should probably make a good-faith effort to catch the RuntimeError and present a useful error message.

bigjools commented 2 years ago

Bizarre. Ok then.

juledwar commented 1 year ago

Did you want to submit a fix for this Nic?

juledwar / soufi

MacOS multiprocess issues #31