sphinx-doc / sphinx

The Sphinx documentation generator
https://www.sphinx-doc.org/
Other
6.6k stars 2.12k forks source link

Sphinx still imports from "distutils", which is going away in Python ≥ 3.12 #9820

Closed leycec closed 2 years ago

leycec commented 3 years ago

Describe the bug

Python 3.10 officially deprecated the standard distutils package after standardization of PEP 632 -- Deprecate distutils module. Python 3.12 is slated to entirely remove that package – and justifiably so. It's an ongoing dumpster fire that can't be put out soon enough. :fire:

Sadly, Sphinx still widely imports from distutils. GitHub shows at least twelve pending references. As a very temporary circumvention, globally replacing distutils with setuptools._distutils should suffice to resolve this. Of course, that's also guaranteed to blow up. Like CPython, setuptools intends to disembowel and eventually remove setuptools._distutils. Ain't nobody got volunteer time to maintain cruft in perpetuity.

Does This Really Matter Now?

Yup. I author @beartype, which doesn't necessarily play well with Sphinx's autodoc extension (as detailed here). I've kludged around that incompatibility on my end with ad-hoc and inexplicably dangerous heuristics that conditionally disable @beartype when we vaguely think autodoc might be active. Nobody actually knows when autodoc is active, of course. There's currently no public Sphinx API to externally detect that. Cue #9805. </ahem>

Because that's dangerous, we exercise that with a non-trivial functional test programmatically running sphinx-build from within the active pytest process by calling the sphinx.cmd.build.main() entry point. Works great – under Python < 3.10, that is. Under Python ≥ 3.10, Sphinx emits deprecation warnings all over the place like a flailing water buffalo besieged by leeches. For safety, our test suite treats any warning as a test failure. Chaos ensues with extreme pytest tracebacks resembling:

>       from sphinx.cmd.build import main as sphinx_build

get_test_func_data_lib_sphinx_dir = <function get_test_func_data_lib_sphinx_dir at 0x7f9579c875f0>
is_success = <function is_success at 0x7f9579c86db0>
tmp_path   = PosixPath('/tmp/pytest-of-leycec/pytest-29/test_sphinx0')

../../../beartype_test/a90_func/lib/test_sphinx.py:43: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
../lib/python3.10/site-packages/sphinx/cmd/build.py:25: in <module>
    from sphinx.application import Sphinx
        Any        = typing.Any
        IO         = <class 'typing.IO'>
        List       = typing.List
        SystemMessage = <class 'docutils.utils.SystemMessage'>
        __builtins__ = <builtins>
        __cached__ = '/home/leycec/py/beartype/.tox/py310/lib/python3.10/site-packages/sphinx/cmd/__pycache__/build.cpython-310.pyc'
        __display_version__ = '4.2.0'
        __doc__    = '\n    sphinx.cmd.build\n    ~~~~~~~~~~~~~~~~\n\n    Build documentation from a provided source.\n\n    :copyright: Copyright 2007-2021 by the Sphinx team, see AUTHORS.\n    :license: BSD, see LICENSE for details.\n'
        __file__   = '/home/leycec/py/beartype/.tox/py310/lib/python3.10/site-packages/sphinx/cmd/build.py'
        __loader__ = <_frozen_importlib_external.SourceFileLoader object at 0x7f9579a118b0>
        __name__   = 'sphinx.cmd.build'
        __package__ = 'sphinx.cmd'
        __spec__   = ModuleSpec(name='sphinx.cmd.build', loader=<_frozen_importlib_external.SourceFileLoader object at 0x7f9579a118b0>, origin='/home/leycec/py/beartype/.tox/py310/lib/python3.10/site-packages/sphinx/cmd/build.py')
        argparse   = <module 'argparse' from '/usr/lib/python3.10/argparse.py'>
        bdb        = <module 'bdb' from '/usr/lib/python3.10/bdb.py'>
        locale     = <module 'locale' from '/usr/lib/python3.10/locale.py'>
        multiprocessing = <module 'multiprocessing' from '/usr/lib/python3.10/multiprocessing/__init__.py'>
        os         = <module 'os' from '/usr/lib/python3.10/os.py'>
        package_dir = '/home/leycec/py/beartype/.tox/py310/lib/python3.10/site-packages/sphinx'
        pdb        = <module 'pdb' from '/usr/lib/python3.10/pdb.py'>
        sphinx     = <module 'sphinx' from '/home/leycec/py/beartype/.tox/py310/lib/python3.10/site-packages/sphinx/__init__.py'>
        sys        = <module 'sys' (built-in)>
        traceback  = <module 'traceback' from '/usr/lib/python3.10/traceback.py'>
../lib/python3.10/site-packages/sphinx/application.py:34: in <module>
    from sphinx.domains import Domain, Index
        Any        = typing.Any
        Callable   = typing.Callable
        Config     = <class 'sphinx.config.Config'>
        Dict       = typing.Dict
        Directive  = <class 'docutils.parsers.rst.Directive'>
        Element    = <class 'docutils.nodes.Element'>
        IO         = <class 'typing.IO'>
        Lexer      = <class 'pygments.lexer.Lexer'>
        List       = typing.List
        Optional   = typing.Optional
        Parser     = <class 'docutils.parsers.Parser'>
        RemovedInSphinx60Warning = <class 'sphinx.deprecation.RemovedInSphinx60Warning'>
        StringIO   = <class '_io.StringIO'>
        TYPE_CHECKING = False
        TextElement = <class 'docutils.nodes.TextElement'>
        Transform  = <class 'docutils.transforms.Transform'>
        Tuple      = typing.Tuple
        Type       = typing.Type
        Union      = typing.Union
        __builtins__ = <builtins>
        __cached__ = '/home/leycec/py/beartype/.tox/py310/lib/python3.10/site-packages/sphinx/__pycache__/application.cpython-310.pyc'
        __doc__    = '\n    sphinx.application\n    ~~~~~~~~~~~~~~~~~~\n\n    Sphinx application class and extensibility interface.\n\n    ...n\n    :copyright: Copyright 2007-2021 by the Sphinx team, see AUTHORS.\n    :license: BSD, see LICENSE for details.\n'
        __file__   = '/home/leycec/py/beartype/.tox/py310/lib/python3.10/site-packages/sphinx/application.py'
        __loader__ = <_frozen_importlib_external.SourceFileLoader object at 0x7f9579a3e8f0>
        __name__   = 'sphinx.application'
        __package__ = 'sphinx'
        __spec__   = ModuleSpec(name='sphinx.application', loader=<_frozen_importlib_external.SourceFileLoader object at 0x7f9579a3e8f0>, origin='/home/leycec/py/beartype/.tox/py310/lib/python3.10/site-packages/sphinx/application.py')
        deque      = <class 'collections.deque'>
        locale     = <module 'sphinx.locale' from '/home/leycec/py/beartype/.tox/py310/lib/python3.10/site-packages/sphinx/locale/__init__.py'>
        nodes      = <module 'docutils.nodes' from '/home/leycec/py/beartype/.tox/py310/lib/python3.10/site-packages/docutils/nodes.py'>
        os         = <module 'os' from '/usr/lib/python3.10/os.py'>
        package_dir = '/home/leycec/py/beartype/.tox/py310/lib/python3.10/site-packages/sphinx'
        path       = <module 'posixpath' from '/usr/lib/python3.10/posixpath.py'>
        pickle     = <module 'pickle' from '/usr/lib/python3.10/pickle.py'>
        platform   = <module 'platform' from '/usr/lib/python3.10/platform.py'>
        roles      = <module 'docutils.parsers.rst.roles' from '/home/leycec/py/beartype/.tox/py310/lib/python3.10/site-packages/docutils/parsers/rst/roles.py'>
        sphinx     = <module 'sphinx' from '/home/leycec/py/beartype/.tox/py310/lib/python3.10/site-packages/sphinx/__init__.py'>
        sys        = <module 'sys' (built-in)>
        warnings   = <module 'warnings' from '/usr/lib/python3.10/warnings.py'>
../lib/python3.10/site-packages/sphinx/domains/__init__.py:24: in <module>
    from sphinx.roles import XRefRole
        ABC        = <class 'abc.ABC'>
        Any        = typing.Any
        Callable   = typing.Callable
        Dict       = typing.Dict
        Element    = <class 'docutils.nodes.Element'>
        Inliner    = <class 'docutils.parsers.rst.states.Inliner'>
        Iterable   = typing.Iterable
        List       = typing.List
        NamedTuple = <function NamedTuple at 0x7f9582bbda70>
        Node       = <class 'docutils.nodes.Node'>
        Optional   = typing.Optional
        SphinxError = <class 'sphinx.errors.SphinxError'>
        TYPE_CHECKING = False
        Tuple      = typing.Tuple
        Type       = typing.Type
        Union      = typing.Union
        _          = <function get_translation.<locals>.gettext at 0x7f9579a49180>
        __builtins__ = <builtins>
        __cached__ = '/home/leycec/py/beartype/.tox/py310/lib/python3.10/site-packages/sphinx/domains/__pycache__/__init__.cpython-310.pyc'
        __doc__    = '\n    sphinx.domains\n    ~~~~~~~~~~~~~~\n\n    Support for domains, which are groupings of description directives\n ...n\n    :copyright: Copyright 2007-2021 by the Sphinx team, see AUTHORS.\n    :license: BSD, see LICENSE for details.\n'
        __file__   = '/home/leycec/py/beartype/.tox/py310/lib/python3.10/site-packages/sphinx/domains/__init__.py'
        __loader__ = <_frozen_importlib_external.SourceFileLoader object at 0x7f95790f8370>
        __name__   = 'sphinx.domains'
        __package__ = 'sphinx.domains'
        __path__   = ['/home/leycec/py/beartype/.tox/py310/lib/python3.10/site-packages/sphinx/domains']
        __spec__   = ModuleSpec(name='sphinx.domains', loader=<_frozen_importlib_external.SourceFileLoader object at 0x7f95790f8370>, origi...__.py', submodule_search_locations=['/home/leycec/py/beartype/.tox/py310/lib/python3.10/site-packages/sphinx/domains'])
        abstractmethod = <function abstractmethod at 0x7f9582e0b3e0>
        cast       = <function cast at 0x7f9582bbc7e0>
        copy       = <module 'copy' from '/usr/lib/python3.10/copy.py'>
        nodes      = <module 'docutils.nodes' from '/home/leycec/py/beartype/.tox/py310/lib/python3.10/site-packages/docutils/nodes.py'>
        pending_xref = <class 'sphinx.addnodes.pending_xref'>
        system_message = <class 'docutils.nodes.system_message'>
../lib/python3.10/site-packages/sphinx/roles.py:20: in <module>
    from sphinx.util.docutils import ReferenceRole, SphinxRole
        Any        = typing.Any
        Dict       = typing.Dict
        Element    = <class 'docutils.nodes.Element'>
        List       = typing.List
        Node       = <class 'docutils.nodes.Node'>
        TYPE_CHECKING = False
        TextElement = <class 'docutils.nodes.TextElement'>
        Tuple      = typing.Tuple
        Type       = typing.Type
        _          = <function get_translation.<locals>.gettext at 0x7f9579a49180>
        __annotations__ = {}
        __builtins__ = <builtins>
        __cached__ = '/home/leycec/py/beartype/.tox/py310/lib/python3.10/site-packages/sphinx/__pycache__/roles.cpython-310.pyc'
        __doc__    = '\n    sphinx.roles\n    ~~~~~~~~~~~~\n\n    Handlers for additional ReST roles.\n\n    :copyright: Copyright 2007-2021 by the Sphinx team, see AUTHORS.\n    :license: BSD, see LICENSE for details.\n'
        __file__   = '/home/leycec/py/beartype/.tox/py310/lib/python3.10/site-packages/sphinx/roles.py'
        __loader__ = <_frozen_importlib_external.SourceFileLoader object at 0x7f95790f8960>
        __name__   = 'sphinx.roles'
        __package__ = 'sphinx'
        __spec__   = ModuleSpec(name='sphinx.roles', loader=<_frozen_importlib_external.SourceFileLoader object at 0x7f95790f8960>, origin='/home/leycec/py/beartype/.tox/py310/lib/python3.10/site-packages/sphinx/roles.py')
        addnodes   = <module 'sphinx.addnodes' from '/home/leycec/py/beartype/.tox/py310/lib/python3.10/site-packages/sphinx/addnodes.py'>
        nodes      = <module 'docutils.nodes' from '/home/leycec/py/beartype/.tox/py310/lib/python3.10/site-packages/docutils/nodes.py'>
        re         = <module 're' from '/usr/lib/python3.10/re.py'>
        system_message = <class 'docutils.nodes.system_message'>
        utils      = <module 'docutils.utils' from '/home/leycec/py/beartype/.tox/py310/lib/python3.10/site-packages/docutils/utils/__init__.py'>
        ws_re      = re.compile('\\s+')
../lib/python3.10/site-packages/sphinx/util/docutils.py:15: in <module>
    from distutils.version import LooseVersion
        __annotations__ = {}
        __builtins__ = <builtins>
        __cached__ = '/home/leycec/py/beartype/.tox/py310/lib/python3.10/site-packages/sphinx/util/__pycache__/docutils.cpython-310.pyc'
        __doc__    = '\n    sphinx.util.docutils\n    ~~~~~~~~~~~~~~~~~~~~\n\n    Utility functions for docutils.\n\n    :copyright: Copyright 2007-2021 by the Sphinx team, see AUTHORS.\n    :license: BSD, see LICENSE for details.\n'
        __file__   = '/home/leycec/py/beartype/.tox/py310/lib/python3.10/site-packages/sphinx/util/docutils.py'
        __loader__ = <_frozen_importlib_external.SourceFileLoader object at 0x7f95790f97c0>
        __name__   = 'sphinx.util.docutils'
        __package__ = 'sphinx.util'
        __spec__   = ModuleSpec(name='sphinx.util.docutils', loader=<_frozen_importlib_external.SourceFileLoader object at 0x7f95790f97c0>, origin='/home/leycec/py/beartype/.tox/py310/lib/python3.10/site-packages/sphinx/util/docutils.py')
        __warningregistry__ = {'version': 678}
        contextmanager = <function contextmanager at 0x7f9582d98470>
        copy       = <function copy at 0x7f9582c855a0>
        os         = <module 'os' from '/usr/lib/python3.10/os.py'>
        re         = <module 're' from '/usr/lib/python3.10/re.py'>
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

    """distutils

    The main package for the Python Module Distribution Utilities.  Normally
    used from a setup script as

       from distutils.core import setup

       setup (...)
    """

    import sys
    import warnings

    __version__ = sys.version[:sys.version.index(' ')]

    _DEPRECATION_MESSAGE = ("The distutils package is deprecated and slated for "
                            "removal in Python 3.12. Use setuptools or check "
                            "PEP 632 for potential alternatives")
>   warnings.warn(_DEPRECATION_MESSAGE,
                  DeprecationWarning, 2)
E   DeprecationWarning: The distutils package is deprecated and slated for removal in Python 3.12. Use setuptools or check PEP 632 for potential alternatives

_DEPRECATION_MESSAGE = 'The distutils package is deprecated and slated for removal in Python 3.12. Use setuptools or check PEP 632 for potential alternatives'
__builtins__ = <builtins>
__cached__ = '/usr/lib/python3.10/distutils/__pycache__/__init__.cpython-310.pyc'
__doc__    = 'distutils\n\nThe main package for the Python Module Distribution Utilities.  Normally\nused from a setup script as\n\n   from distutils.core import setup\n\n   setup (...)\n'
__file__   = '/usr/lib/python3.10/distutils/__init__.py'
__loader__ = <_frozen_importlib_external.SourceFileLoader object at 0x7f95790fa5d0>
__name__   = 'distutils'
__package__ = 'distutils'
__path__   = ['/usr/lib/python3.10/distutils']
__spec__   = ModuleSpec(name='distutils', loader=<_frozen_importlib_external.SourceFileLoader object at 0x7f95790fa5d0>, origin='/usr/lib/python3.10/distutils/__init__.py', submodule_search_locations=['/usr/lib/python3.10/distutils'])
__version__ = '3.10.0'
sys        = <module 'sys' (built-in)>
warnings   = <module 'warnings' from '/usr/lib/python3.10/warnings.py'>

/usr/lib/python3.10/distutils/__init__.py:19: DeprecationWarning
=================================================== short test summary info ===================================================
SKIPPED [1] ../../../beartype_test/a00_unit/a10_pep/test_pep563.py:227: Python 3.10.0 >= 3.10.0.
FAILED ../../../beartype_test/a90_func/lib/test_sphinx.py::test_sphinx - DeprecationWarning: The distutils package is deprec...
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! stopping after 1 failures !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
========================================== 1 failed, 183 passed, 1 skipped in 7.03s ===========================================
ERROR: InvocationError for command /home/leycec/py/beartype/.tox/py310/bin/python -W 'ignore:The distutils package is deprecated:DeprecationWarning' -m pytest --maxfail=1 /home/leycec/py/beartype (exited with code 1)
___________________________________________________________ summary ___________________________________________________________
ERROR:   py310: commands failed

To preserve our sanity, we're currently just ignoring that test under Python ≥ 3.10. We're not proud – but sometimes you just gotta get to sleep.

Thanks for all the Structured Docos

As always, thanks a well-documented ton for all the amazing volunteer work everyone does here. Sphinx rocks! May this issue find the final reST it so deserves. :wink:

How to Reproduce

Just look at the Sphinx codebase, maybe? Not sure what to say here. Badness is bad. The bitrot has gotta be removed – ideally sooner than later.

Expected behavior

...that Sphinx not import from distutils. Work with me here, people.

Your project

The problem is in Sphinx itself. Ain't no small project gonna help.

Screenshots

Let's be honest: nobody wants another screenshot of my puce monochrome CLI ViM setup. Nobody.

OS

Gentoo Linux, of course! O_o

Python version

3.10

Sphinx version

4.2.0

Sphinx extensions

Irrelevant.

Extra tools

Irrelevant.

Additional context

Irrelevant.

tk0miya commented 3 years ago

Reasonable. Now I posted #9826 to reduce the dependency to the distutils module. I hope it helps you.

After merging it, we still have 4 references in our code. We need to investigate their successor.

$ grep -r distutils sphinx tests setup.py | grep import
sphinx/setup_command.py:from distutils.cmd import Command
sphinx/setup_command.py:from distutils.errors import DistutilsExecError
tests/roots/test-setup/setup.py:from distutils.core import setup
setup.py:from distutils import log
leycec commented 3 years ago

ありがとうございます。Thanks so much, Takeshi! Your volunteer time is scarce, precious, and very much appreciated here.

I fully agree that and install- and test-time imports of distutils are much less mission-critical than runtime imports of distutils. Thankfully, you've already resolved all of the latter – which is as unexpected as it is impressive. You're the heroic Godzilla of the Sphinx world. Please take this warm hug for your continual good deeds. :hugs:

tk0miya commented 3 years ago

Now I merged #9826 to 4.x branch. So it will be released as 4.3.0 soon (maybe in this week). But I'll keep this open until removing other usages.

tk0miya commented 2 years ago

Note: I posted #10042 to reduce the usage of docutils. After merging it, we'll still have two implementations.

pradyunsg commented 2 years ago

FWIW, you can replace the import distutils with import setuptools and things should work the exact same; both in your setup.py as well as sphinx.setuptools.

The sneaky thing is the pip already does this for you, by importing setuptools before running the setup.py file; which means you shouldn't see any change in behaviours by replacing distutils with setuptools.

tk0miya commented 2 years ago

Now we only have two importings:

$ grep -r distutils sphinx tests setup.py | grep import
sphinx/setup_command.py:from distutils.cmd import Command
sphinx/setup_command.py:from distutils.errors import DistutilsExecError

Is it possible to replace them by import setuptools? I tried to replace them by from setuptools._distutils... import ..., but it failed. Either way, we need to support old packages also. So it's not in a hurry replacing, IMO.

pradyunsg commented 2 years ago

from setuptools import Command

And

from setuptools.errors import ExecError

tk0miya commented 2 years ago

It seems setuptools.errors has been available since v60.2.0. To replace it, we need to update our dependencies. It's unacceptable. https://github.com/pypa/setuptools/commit/974dbb03769a500c5077d37291817f30359a0d7b#diff-d46ab9d69835e789642a521f470657990ff684ea71c8df818328c54bf19f3cd5

tk0miya commented 2 years ago

Note: Ubuntu-18.04 ships setuptools-39.0.1. https://packages.ubuntu.com/ja/bionic/python-setuptools

pradyunsg commented 2 years ago

v60.2.0.

59.0.0, but your point still stands.

To replace it, we need to update our dependencies.

Could you clarify this further? What would you need here, other than a setuptools > 59.0.0 constraint? Or, is that the constraint that you consider unacceptable?

packages.ubuntu.com/ja/bionic/python-setuptools

I'm not sure how you model the relationship with redistributors like Ubuntu, but they're not going to be adding the latest Sphinx release without adding a newer version of setuptools. Users should generally use virtual environments and be isolated from their redistributor-provided setuptools anyway. If they can't do that, then they'd be restricted to whatever their redistributor (i.e. Ubuntu maintainers) provides; so it's not like they should be using a newer version of Sphinx anyway.

pradyunsg commented 2 years ago

I'll also note that you can still do something like: https://github.com/scikit-image/scikit-image/pull/6044/files#diff-60f61ab7a8d1910d86d9fda2261620314edcae5894d5aaa236b821c7256badd7R14

try:
   from setuptools.errors import ... as ...
except ImportError:
   from distutils.errors import ... as ...

That would fix this issue and still be equally backwards compatible.

AA-Turner commented 2 years ago

At the current release schedule, Sphinx 7 will be released in 2024, and Python 3.12 in October 2023, so we either need to close this as the setuptools integration is deprecated and we won't support it, or replace the imports with imports from setuptools.

A

AA-Turner commented 2 years ago

Clicked the wrong button.

A

nanonyme commented 2 years ago

Btw, if you distribute pyproject.toml, you can requests pip to install into isolated sandbox a version of setuptools that you need as far as I understand. That should help with end-users installing this project assuming they have new enough pip to understand PEP517. As for distros like Ubuntu 18.04 LTS, I doubt they will ever update Sphinx. They will for remainder of their existence have Sphinx 1.6.7.

AA-Turner commented 2 years ago

Sphinx doesn't use pyproject.toml (I have a proposal for that at #10356). We only support the latest released Sphinx, so 1.6.7 is not relevant here.

The real issue is do we care about supporting the intersection of "Python >= 3.12" and "using deprecated setup.py commands". If we do, the solution is pretty simple, as @pradyunsg pointed out (s/distutils/setuptools/).

A

pradyunsg commented 2 years ago

I'll repeat something I've said before in various places: just because a redistributor ships an old version of a software does not mean that the upstream developers/maintainers are responsible for extending support for that version -- the redistributor is on the hook for that.

nanonyme commented 2 years ago

@pradyunsg kind of. But with PEP517 you can have best out of both worlds as said so you can define which version of setuptools you need and pip will automatically obtain it. My comment was as @AA-Turner said in the end offtopic as there is a second issue for that.

tk0miya commented 2 years ago

Unfortunately, our software is not perfect. It would contain a lot of bugs. Actually, we've fixed these bugs on each release. I think it's a good way to support old platforms and libraries to help real users.

Additionally, our software is not only for python developers. They might not know about python at all. So it's difficult to force to use virtualenv to install or upgrade Sphinx.

I think try-expect fallback is a good approach.

nanonyme commented 2 years ago

Note that pip will transparently create build virtualenvs with select build deps installed in its current versions. You actually have to have an expert running through things to avoid this.