desihub / desiutil

General DESI utilities, shell scripts, desiInstall, etc.
BSD 3-Clause "New" or "Revised" License
3 stars 9 forks source link

desiutil.setup may be incompatible with recent versions of setuptools #198

Open weaverba137 opened 1 year ago

weaverba137 commented 1 year ago

During recent documentation tests, imports of desiutil.setup triggered errors related to importing setuptools.

Since this is likely related to the deprecation of setup.py, we may need to abandon backward-compatible support for desiutil enhancements to setup.py sooner rather than later.

We already have replacements for the functionality. The only thing needed here is a decision to abandon backward-compatibility.

Full traceback:

WARNING: autodoc: failed to import module 'setup' from module 'desiutil'; the following exception was raised:
Traceback (most recent call last):
  File "/Users/benjamin.weaver/Documents/local/products/venv/desiconda/lib/python3.10/site-packages/sphinx/ext/autodoc/importer.py", line 64, in import_module
    return importlib.import_module(modname)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/importlib/__init__.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1050, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1027, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1006, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 688, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 883, in exec_module
  File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
  File "/Users/benjamin.weaver/Documents/Code/git/desihub/desiutil/py/desiutil/setup.py", line 20, in <module>
    from setuptools import Command
  File "/Users/benjamin.weaver/Documents/local/products/venv/desiconda/lib/python3.10/site-packages/setuptools/__init__.py", line 18, in <module>
    from setuptools.dist import Distribution
  File "/Users/benjamin.weaver/Documents/local/products/venv/desiconda/lib/python3.10/site-packages/setuptools/dist.py", line 41, in <module>
    from setuptools.config import setupcfg, pyprojecttoml
  File "/Users/benjamin.weaver/Documents/local/products/venv/desiconda/lib/python3.10/site-packages/setuptools/config/__init__.py", line 10, in <module>
    from . import setupcfg
  File "/Users/benjamin.weaver/Documents/local/products/venv/desiconda/lib/python3.10/site-packages/setuptools/config/setupcfg.py", line 23, in <module>
    from . import expand
  File "/Users/benjamin.weaver/Documents/local/products/venv/desiconda/lib/python3.10/site-packages/setuptools/config/expand.py", line 53, in <module>
    from setuptools.dist import Distribution  # noqa
ImportError: cannot import name 'Distribution' from partially initialized module 'setuptools.dist' (most likely due to a circular import) (/Users/benjamin.weaver/Documents/local/products/venv/desiconda/lib/python3.10/site-packages/setuptools/dist.py)

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/Users/benjamin.weaver/Documents/local/products/venv/desiconda/lib/python3.10/site-packages/sphinx/ext/autodoc/importer.py", line 88, in import_object
    module = import_module(modname, warningiserror=warningiserror)
  File "/Users/benjamin.weaver/Documents/local/products/venv/desiconda/lib/python3.10/site-packages/sphinx/ext/autodoc/importer.py", line 68, in import_module
    raise ImportError(exc, traceback.format_exc()) from exc
ImportError: (ImportError("cannot import name 'Distribution' from partially initialized module 'setuptools.dist' (most likely due to a circular import) (/Users/benjamin.weaver/Documents/local/products/venv/desiconda/lib/python3.10/site-packages/setuptools/dist.py)"), 'Traceback (most recent call last):\n  File "/Users/benjamin.weaver/Documents/local/products/venv/desiconda/lib/python3.10/site-packages/sphinx/ext/autodoc/importer.py", line 64, in import_module\n    return importlib.import_module(modname)\n  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/importlib/__init__.py", line 126, in import_module\n    return _bootstrap._gcd_import(name[level:], package, level)\n  File "<frozen importlib._bootstrap>", line 1050, in _gcd_import\n  File "<frozen importlib._bootstrap>", line 1027, in _find_and_load\n  File "<frozen importlib._bootstrap>", line 1006, in _find_and_load_unlocked\n  File "<frozen importlib._bootstrap>", line 688, in _load_unlocked\n  File "<frozen importlib._bootstrap_external>", line 883, in exec_module\n  File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed\n  File "/Users/benjamin.weaver/Documents/Code/git/desihub/desiutil/py/desiutil/setup.py", line 20, in <module>\n    from setuptools import Command\n  File "/Users/benjamin.weaver/Documents/local/products/venv/desiconda/lib/python3.10/site-packages/setuptools/__init__.py", line 18, in <module>\n    from setuptools.dist import Distribution\n  File "/Users/benjamin.weaver/Documents/local/products/venv/desiconda/lib/python3.10/site-packages/setuptools/dist.py", line 41, in <module>\n    from setuptools.config import setupcfg, pyprojecttoml\n  File "/Users/benjamin.weaver/Documents/local/products/venv/desiconda/lib/python3.10/site-packages/setuptools/config/__init__.py", line 10, in <module>\n    from . import setupcfg\n  File "/Users/benjamin.weaver/Documents/local/products/venv/desiconda/lib/python3.10/site-packages/setuptools/config/setupcfg.py", line 23, in <module>\n    from . import expand\n  File "/Users/benjamin.weaver/Documents/local/products/venv/desiconda/lib/python3.10/site-packages/setuptools/config/expand.py", line 53, in <module>\n    from setuptools.dist import Distribution  # noqa\nImportError: cannot import name \'Distribution\' from partially initialized module \'setuptools.dist\' (most likely due to a circular import) (/Users/benjamin.weaver/Documents/local/products/venv/desiconda/lib/python3.10/site-packages/setuptools/dist.py)\n')

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/benjamin.weaver/Documents/local/products/venv/desiconda/lib/python3.10/site-packages/sphinx/ext/autodoc/importer.py", line 64, in import_module
    return importlib.import_module(modname)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/importlib/__init__.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1050, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1027, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1006, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 688, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 883, in exec_module
  File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
  File "/Users/benjamin.weaver/Documents/Code/git/desihub/desiutil/py/desiutil/setup.py", line 20, in <module>
    from setuptools import Command
  File "/Users/benjamin.weaver/Documents/local/products/venv/desiconda/lib/python3.10/site-packages/setuptools/__init__.py", line 36, in <module>
    __version__ = setuptools.version.__version__
AttributeError: partially initialized module 'setuptools' has no attribute 'version' (most likely due to a circular import)
sbailey commented 1 year ago

To abandon backwards compatibility, we need:

After those have been completed, then we can remove support for setup.py itself for all packages en-masse, but let's not remove setup.py package-by-package until the new commands work everywhere so that we don't have to keep track of which packages require the old vs. new commands.

@weaverba137 please edit this comment to correct/expand this table as needed:

old command new command
python setup.py test pytest
python setup.py version desi_update_version
python setup.py version --tag X.Y desi_update_version -t X.Y
python setup.py build_sphinx sphinx-build doc/ build/sphinx/html (?)
python setup.py install pip install . (?)
python setup.py develop pip install --editable . (?)
python setup.py build_ext --inplace ??? (used by fiberassign and specex)
weaverba137 commented 1 year ago

The table already exists: https://desiutil.readthedocs.io/en/latest/helpers.html. Is that sufficient?

weaverba137 commented 1 year ago

To be clear, we will still need python setup.py build_ext --inplace for fiberassign and specex. I'm only proposing abandoning the enhancements provided by desiutil.setup, not necessarily the top-level setup.py file in packages.

weaverba137 commented 1 year ago

The exception above only appeared when compiling documentation with Sphinx. Pinning Sphinx<7 in the test suite eliminates this error.

We still need to do the task of eliminating desiutil from various setup.py files, but for the purposes of desiutil, there is a workaround that will allow documentation builds to succeed.