search

JohannesBuchner / UltraNest

Fit and compare complex models reliably and rapidly. Advanced nested sampling.
https://johannesbuchner.github.io/UltraNest/
Other
142 stars 30 forks source link

Ultranest attempts a requirement import during installation #56

Closed jpl-jengelke closed 2 years ago

jpl-jengelke commented 2 years ago

Description

During installation Ultranest tries to import Cython. Pip installations bundle gathering dependencies and then at a later stage they install all the dependencies. The bundles aren't actually installed yet. So when Ultranest tries to reference Cython during the installation process in setup.py, it's unavailable. 

someorg:EXOTIC jpl-jengelke$ pip install -r requirements.txt 
Ignoring importlib-metadata: markers 'python_version <= "3.7"' don't match your environment
Ignoring dynesty: markers 'platform_system == "Windows"' don't match your environment
Collecting wheel
  Downloading wheel-0.37.1-py2.py3-none-any.whl (35 kB)
Requirement already satisfied: setuptools in /Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages (from -r requirements.txt (line 3)) (60.9.3)
Collecting python_version
  Downloading python_version-0.0.2-py2.py3-none-any.whl (3.4 kB)
Collecting astroalign~=2.4
  Downloading astroalign-2.4.1.tar.gz (1.6 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 1.6/1.6 MB 10.8 MB/s eta 0:00:00
  Preparing metadata (setup.py) ... done
Collecting astropy>=4.3
  Downloading astropy-5.0.2-cp39-cp39-macosx_10_9_x86_64.whl (6.9 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 6.9/6.9 MB 12.4 MB/s eta 0:00:00
Collecting astroquery~=0.4
  Downloading astroquery-0.4.5-py3-none-any.whl (4.5 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 4.5/4.5 MB 15.3 MB/s eta 0:00:00
Collecting barycorrpy~=0.4
  Downloading barycorrpy-0.4.4-py3-none-any.whl (9.6 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 9.6/9.6 MB 14.5 MB/s eta 0:00:00
Collecting cython~=0.29.26
  Downloading Cython-0.29.28-py2.py3-none-any.whl (983 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 983.8/983.8 KB 8.5 MB/s eta 0:00:00
Collecting holoviews~=1.14
  Downloading holoviews-1.14.8-py2.py3-none-any.whl (4.3 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 4.3/4.3 MB 15.2 MB/s eta 0:00:00
Collecting LDTk~=1.7
  Downloading LDTk-1.7.0-py3-none-any.whl (37 kB)
Collecting lmfit~=1.0
  Downloading lmfit-1.0.3.tar.gz (292 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 292.5/292.5 KB 6.4 MB/s eta 0:00:00
  Preparing metadata (setup.py) ... done
Collecting matplotlib>=3.4
  Downloading matplotlib-3.5.1-cp39-cp39-macosx_10_9_x86_64.whl (7.3 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 7.3/7.3 MB 3.7 MB/s eta 0:00:00
Collecting numpy~=1.21
  Downloading numpy-1.22.3-cp39-cp39-macosx_10_14_x86_64.whl (17.6 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 17.6/17.6 MB 14.3 MB/s eta 0:00:00
Collecting pandas~=1.3
  Downloading pandas-1.4.1-cp39-cp39-macosx_10_9_x86_64.whl (11.5 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 11.5/11.5 MB 15.2 MB/s eta 0:00:00
Collecting panel~=0.12
  Downloading panel-0.12.6-py2.py3-none-any.whl (12.9 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 12.9/12.9 MB 15.6 MB/s eta 0:00:00
Collecting photutils>=0.7
  Downloading photutils-1.3.0-cp39-cp39-macosx_10_9_x86_64.whl (559 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 559.1/559.1 KB 8.2 MB/s eta 0:00:00
Collecting python_dateutil~=2.8
  Downloading python_dateutil-2.8.2-py2.py3-none-any.whl (247 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 247.7/247.7 KB 5.6 MB/s eta 0:00:00
Collecting pyvo~=1.2
  Downloading pyvo-1.3-py3-none-any.whl (829 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 829.6/829.6 KB 8.5 MB/s eta 0:00:00
Collecting requests~=2.27
  Downloading requests-2.27.1-py2.py3-none-any.whl (63 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 63.1/63.1 KB 2.0 MB/s eta 0:00:00
Collecting scipy~=1.7
  Downloading scipy-1.8.0-cp39-cp39-macosx_12_0_universal2.macosx_10_9_x86_64.whl (55.6 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 55.6/55.6 MB 13.2 MB/s eta 0:00:00
Collecting scikit-image~=0.18
  Downloading scikit_image-0.19.2-cp39-cp39-macosx_10_13_x86_64.whl (13.3 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 13.3/13.3 MB 15.7 MB/s eta 0:00:00
Collecting tenacity~=8.0
  Downloading tenacity-8.0.1-py3-none-any.whl (24 kB)
Collecting ultranest~=3.3.3
  Downloading ultranest-3.3.3.tar.gz (21.5 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 21.5/21.5 MB 14.3 MB/s eta 0:00:00
  Preparing metadata (setup.py) ... error
  error: subprocess-exited-with-error

  × python setup.py egg_info did not run successfully.
  │ exit code: 1
  ╰─> [6 lines of output]
      Traceback (most recent call last):
        File "<string>", line 2, in <module>
        File "<pip-setuptools-caller>", line 34, in <module>
        File "/private/var/folders/0_/0mssd1656q78c47k0_yk7sn00000gp/T/pip-install-zzyhdkbo/ultranest_45cde6e84d9d4627ab12dbddb6315fea/setup.py", line 9, in <module>
          from Cython.Build import cythonize
      ModuleNotFoundError: No module named 'Cython'
      [end of output]

  note: This error originates from a subprocess, and is likely not a problem with pip.
error: metadata-generation-failed

× Encountered error while generating package metadata.
╰─> See above for output.

note: This is an issue with the package mentioned above, not pip.
hint: See above for details.
WARNING: You are using pip version 22.0.3; however, version 22.0.4 is available.
You should consider upgrading via the '/Library/Frameworks/Python.framework/Versions/3.9/bin/python3 -m pip install --upgrade pip' command.

What I Did

Workaround is to install Cython first, so Ultranest can use it on a second install pass. We made two requirements files to call.
JohannesBuchner commented 2 years ago

Yes, this is a known limitation of pip.

See https://github.com/JohannesBuchner/UltraNest/issues/26. Could potentially be resolved with pyproject.toml.

I'd be interested in seeing it resolved if you have the time to look into creating such a pyproject.toml file for ultranest.

jpl-jengelke commented 2 years ago

Not sure if there is a way to solve this. I found: https://sadafnoor.me/2015/01/07/how-to-automate-numpy-installation-in-your-project-using-setuptool/ but it's unclear if this invokes the limitation.

Thank you. I will see if I have time to implement the TOML solution.

ahnitz commented 2 years ago

The last round of releases seems to have triggered this issue for us downstream as we use Ultranest within our testsuite. I'm not sure why it would be a problem with the April 6th release and not earlier though. Perhaps pip just decided to install stuff in a different order.

ahnitz commented 2 years ago

I don't see a PR ready to go atm, so I'll look into one for Ultrnest. @JohannesBuchner Would it be possible to do a small point release after a proposed PR is merged to make sure this works as intended (and to resolve downstream issues)?

ahnitz commented 2 years ago

Ok, I believe this is resolved by https://github.com/JohannesBuchner/UltraNest/pull/59

ahnitz commented 2 years ago

The error we see with 3.4.2 of ultranest is as follows 

Collecting ultranest
  Downloading ultranest-3.4.2.tar.gz (21.8 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 21.8/21.8 MB 76.6 MB/s eta 0:00:00
  Installing build dependencies: started
  Installing build dependencies: finished with status 'done'
  WARNING: Missing build requirements in pyproject.toml for ultranest from https://files.pythonhosted.org/packages/12/50/71eb4b4f259c524fc5783300db4ffc2447573874d18015deef95bced891a/ultranest-3.4.2.tar.gz#sha256=b3872217c648a4e98932b94935a554f9b55a5fe225709f60336677a887ecb1ab (from -r companion.txt (line 17)).
  WARNING: The project does not specify a build backend, and pip cannot fall back to setuptools without 'setuptools>=40.8.0' and 'wheel'.
  Getting requirements to build wheel: started
  Getting requirements to build wheel: finished with status 'done'
ERROR: Exception:
Traceback (most recent call last):
  File "/home/runner/work/pycbc/pycbc/.tox/py-inference/lib/python3.9/site-packages/pip/_internal/cli/base_command.py", line 167, in exc_logging_wrapper
    status = run_func(*args)
  File "/home/runner/work/pycbc/pycbc/.tox/py-inference/lib/python3.9/site-packages/pip/_internal/cli/req_command.py", line 205, in wrapper
    return func(self, options, args)
  File "/home/runner/work/pycbc/pycbc/.tox/py-inference/lib/python3.9/site-packages/pip/_internal/commands/install.py", line 339, in run
    requirement_set = resolver.resolve(
  File "/home/runner/work/pycbc/pycbc/.tox/py-inference/lib/python3.9/site-packages/pip/_internal/resolution/resolvelib/resolver.py", line 94, in resolve
    result = self._result = resolver.resolve(
  File "/home/runner/work/pycbc/pycbc/.tox/py-inference/lib/python3.9/site-packages/pip/_vendor/resolvelib/resolvers.py", line 481, in resolve
    state = resolution.resolve(requirements, max_rounds=max_rounds)
  File "/home/runner/work/pycbc/pycbc/.tox/py-inference/lib/python3.9/site-packages/pip/_vendor/resolvelib/resolvers.py", line 348, in resolve
    self._add_to_criteria(self.state.criteria, r, parent=None)
  File "/home/runner/work/pycbc/pycbc/.tox/py-inference/lib/python3.9/site-packages/pip/_vendor/resolvelib/resolvers.py", line 172, in _add_to_criteria
    if not criterion.candidates:
  File "/home/runner/work/pycbc/pycbc/.tox/py-inference/lib/python3.9/site-packages/pip/_vendor/resolvelib/structs.py", line 151, in __bool__
    return bool(self._sequence)
  File "/home/runner/work/pycbc/pycbc/.tox/py-inference/lib/python3.9/site-packages/pip/_internal/resolution/resolvelib/found_candidates.py", line 155, in __bool__
    return any(self)
  File "/home/runner/work/pycbc/pycbc/.tox/py-inference/lib/python3.9/site-packages/pip/_internal/resolution/resolvelib/found_candidates.py", line 1[43](https://github.com/ahnitz/pycbc/runs/5868694991?check_suite_focus=true#step:6:43), in <genexpr>
    return (c for c in iterator if id(c) not in self._incompatible_ids)
  File "/home/runner/work/pycbc/pycbc/.tox/py-inference/lib/python3.9/site-packages/pip/_internal/resolution/resolvelib/found_candidates.py", line 47, in _iter_built
    candidate = func()
  File "/home/runner/work/pycbc/pycbc/.tox/py-inference/lib/python3.9/site-packages/pip/_internal/resolution/resolvelib/factory.py", line 215, in _make_candidate_from_link
    self._link_candidate_cache[link] = LinkCandidate(
  File "/home/runner/work/pycbc/pycbc/.tox/py-inference/lib/python3.9/site-packages/pip/_internal/resolution/resolvelib/candidates.py", line 288, in __init__
    super().__init__(
  File "/home/runner/work/pycbc/pycbc/.tox/py-inference/lib/python3.9/site-packages/pip/_internal/resolution/resolvelib/candidates.py", line 158, in __init__
    self.dist = self._prepare()
  File "/home/runner/work/pycbc/pycbc/.tox/py-inference/lib/python3.9/site-packages/pip/_internal/resolution/resolvelib/candidates.py", line 227, in _prepare
    dist = self._prepare_distribution()
  File "/home/runner/work/pycbc/pycbc/.tox/py-inference/lib/python3.9/site-packages/pip/_internal/resolution/resolvelib/candidates.py", line 299, in _prepare_distribution
    return preparer.prepare_linked_requirement(self._ireq, parallel_builds=True)
  File "/home/runner/work/pycbc/pycbc/.tox/py-inference/lib/python3.9/site-packages/pip/_internal/operations/prepare.py", line 487, in prepare_linked_requirement
    return self._prepare_linked_requirement(req, parallel_builds)
  File "/home/runner/work/pycbc/pycbc/.tox/py-inference/lib/python3.9/site-packages/pip/_internal/operations/prepare.py", line 556, in _prepare_linked_requirement
    dist = _get_prepared_distribution(
  File "/home/runner/work/pycbc/pycbc/.tox/py-inference/lib/python3.9/site-packages/pip/_internal/operations/prepare.py", line 58, in _get_prepared_distribution
    abstract_dist.prepare_distribution_metadata(finder, build_isolation)
  File "/home/runner/work/pycbc/pycbc/.tox/py-inference/lib/python3.9/site-packages/pip/_internal/distributions/sdist.py", line [45](https://github.com/ahnitz/pycbc/runs/5868694991?check_suite_focus=true#step:6:45), in prepare_distribution_metadata
    self._install_build_reqs(finder)
  File "/home/runner/work/pycbc/pycbc/.tox/py-inference/lib/python3.9/site-packages/pip/_internal/distributions/sdist.py", line 104, in _install_build_reqs
    build_reqs = self._get_build_requires_wheel()
  File "/home/runner/work/pycbc/pycbc/.tox/py-inference/lib/python3.9/site-packages/pip/_internal/distributions/sdist.py", line 81, in _get_build_requires_wheel
    return backend.get_requires_for_build_wheel()
  File "/home/runner/work/pycbc/pycbc/.tox/py-inference/lib/python3.9/site-packages/pip/_vendor/pep[51](https://github.com/ahnitz/pycbc/runs/5868694991?check_suite_focus=true#step:6:51)7/wrappers.py", line 1[72](https://github.com/ahnitz/pycbc/runs/5868694991?check_suite_focus=true#step:6:72), in get_requires_for_build_wheel
    return self._call_hook('get_requires_for_build_wheel', {
  File "/home/runner/work/pycbc/pycbc/.tox/py-inference/lib/python3.9/site-packages/pip/_vendor/pep517/wrappers.py", line 332, in _call_hook
    raise BackendUnavailable(data.get('traceback', ''))
pip._vendor.pep517.wrappers.BackendUnavailable: Traceback (most recent call last):
  File "/home/runner/work/pycbc/pycbc/.tox/py-inference/lib/python3.9/site-packages/pip/_vendor/pep517/in_process/_in_process.py", line 89, in _build_backend
    obj = import_module(mod_path)
  File "/home/runner/work/pycbc/pycbc/.tox/py-inference/lib/python3.9/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1030, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1007, in _find_and_load
  File "<frozen importlib._bootstrap>", line 972, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 228, in _call_with_frames_removed
  File "<frozen importlib._bootstrap>", line 1030, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1007, in _find_and_load
  File "<frozen importlib._bootstrap>", line 9[84](https://github.com/ahnitz/pycbc/runs/5868694991?check_suite_focus=true#step:6:84), in _find_and_load_unlocked
ModuleNotFoundError: No module named 'setuptools'
JohannesBuchner commented 2 years ago

OK, I added commit 46abf66, and should release a new version to pypi.

jpl-jengelke commented 2 years ago

Thanks everyone!

JohannesBuchner commented 2 years ago

ultranest 3.4.3 is on pypi

The conda-forge issues also seems to be solved soon.

JohannesBuchner commented 2 years ago

Closing this, please reopen if it is still an issue