Intersphinx: py:class reference target not found: _io.BytesIO

radarhere commented 3 weeks ago

Describe the bug

It appears to me that intersphinx is unable to create a link to BytesIO when a class inherits from it.

from io import BytesIO

class AppendingTiffWriter(BytesIO):
    pass

.. autoclass:: PIL.AppendingTiffWriter
  :show-inheritance:

/opt/hostedtoolcache/Python/3.12.5/x64/lib/python3.12/site-packages/PIL/init.py:docstring of PIL.AppendingTiffWriter:1: WARNING: py:class reference target not found: _io.BytesIO [ref.class]

How to Reproduce

I've created a minimal reproduction at https://github.com/radarhere/sphinx_demo. The build can be triggered using GitHub Actions. See https://github.com/radarhere/sphinx_demo/actions/runs/10748344237 for full output.

Environment Information

Platform:              linux; (Linux-6.5.0-1025-azure-x86_64-with-glibc2.35)
Python version:        3.12.5 (main, Aug 13 2024, 19:25:41) [GCC 11.4.0])
Python implementation: CPython
Sphinx version:        8.0.2
Docutils version:      0.21.2
Jinja2 version:        3.1.4
Pygments version:      2.18.0

Sphinx extensions

["sphinx.ext.autodoc", "sphinx.ext.intersphinx"]

Additional context

No response

electric-coder commented 2 weeks ago

I'm making an educated guess that something goes wrong when you try to display the inheritance because that implies linking to the Python standard library and the error message seems to suggest there's some magic under the hood (as is hinted by the unexpected change from a public io.BytesIO to a private _io.BytesIO):

WARNING: py:class reference target not found: _io.BytesIO [ref.class]

Did you try getting rid of :show-inheritance: to see if it builds?

.. autoclass:: PIL.AppendingTiffWriter
  :show-inheritance:

another alternative would be hardcoding the fully qualified name in autodoc, like:

.. autoclass:: PIL.AppendingTiffWriter(io.BytesIO)

Anyway, I checked your intersphinx and it seems alright. When one of these problems happens the surest workaround (generally happens to me with ABCMeta stuff) is to add the inheritance in conf.py circumventing the resolution step by using autodoc-process-bases , something like:

def add_superclass(app, name, obj, options, bases):
    """Used to add a superclass (or metaclass) to the bases of a class."""
    from your_module import your_super_class

    if name == "your_module.your_subclass":
        bases.append(your_super_class)

def setup(app):
    app.connect("autodoc-process-bases", add_superclass)

As to what causes these things? Who's to say...?! Your MRE seems alright to me.

radarhere commented 2 weeks ago

Removing :show-inheritance: does work, but that's not an ideal solution.

io.BytesIO doesn't actually make a difference.

electric-coder commented 2 weeks ago

but that's not an ideal solution.

I agree. I don't contribute fixes for Sphinx's internals and this issue is beyond my knowledge of the internals. But of the 3 workaround alternatives using the conf.py autodoc event is generally considered the most reliable - it's what I use when the easier options don't work out.

electric-coder commented 2 weeks ago

io.BytesIO

I didn't mean change the Python code. I meant write the fully qualified name directly into the autodoc declaration parenthesis so that the cross-reference gets resolved without checking the run-time imported AppendingTiffWriter - exactly like I showed in the 2nd alternative (again these changes are in the .rst not the .py).

.. autoclass:: PIL.AppendingTiffWriter(io.BytesIO)
   :show-inheritance:

I checked the uncompiled objects.inv and for BytesIO and it's:

py:class
    io.BytesIO                               library/io.html#io.BytesIO

So all 3 methods should resolve it.

radarhere commented 2 weeks ago

I didn't mean change the Python code. I meant write the fully qualified name directly into the autodoc declaration parenthesis

You're right, sorry, I misread that.

My own workaround was simply nitpick_ignore = [("py:class", "_io.BytesIO")].

But I had imagined that either a) there was something wrong in my code, or b) the situation implied a bug in sphinx or a sphinx-related extension, or perhaps that a workaround should be hardcoded into sphinx-related code itself somewhere to account for a Python oddity.

If the considered opinion of the sphinx community is that this is just how things are though, that's... unexpected, but thanks for your time.

AA-Turner commented 2 weeks ago

Hi Andrew,

Thank you for writing this up, I agree it is a bug, and I imagine probably because Sphinx is resolving the type at runtime to the builtin _io extension module rather than the public io module. I haven't tested this, though.

The simplest way to resolve this is probably a map of overrides, though ugly. It is interesting that it only manifests with :show-inheritance:, though.

A

electric-coder commented 2 weeks ago

@radarhere

to account for a Python oddity.

I guess it's something non standard in Python's io.BytesIO that autodoc then fails to handle properly.

this is just how things are though, that's... unexpected

I personally prefer the alternatives I've shown but using nitpick_ignore might be the better choice depending on project/doc layout (for some users changing .rst might be less intrusive, while others may prefer adapting their conf.py).

There's an additional problem to using nitpick_ignore = [("py:class", "_io.BytesIO")] because you're silencing other potential problems with _io.Bytes while adapting an individual autodoc directive solves the linking issue locally.

@AA-Turner

The simplest way to resolve this is probably a map of overrides, though ugly.

If it gets the job done it's better to get an easy interim solution until something more durable is thought of.

picnixz commented 2 weeks ago

I guess it's something non standard in Python's io.BytesIO that autodoc then fails to handle properly.

Actually, _io.BytesIO is not documented as such but as io.BytesIO. So intersphinx is trying to find _io.BytesIO. It could be solved if CPython included an index for _io.BytesIO but that's not currently the case. The reason why we actually try to find _io.BytesIO is because the fully-qualified name of BytesIO is _io.BytesIO even though the latter is being imported from io (to be precise, there is an io.py module which itself imports _io and re-exports some members. So the declaring module for BytesIO is _io.BytesIO but exposed and documented as io.BytesIO).

I have less time to work on autodoc. I don't know when I'll have time. I had in mind some improvements for autodoc_type_aliases but it wouldn't be helpful in the context of inheritance.

If I recall correctly, here's the actual flow:

The classes are looked up. The class _io.BytesIO appears.
Autodoc restifies it, making it appear in the list of bases. It's roughly equivalent to have written directly :class:`_io.BytesIO` in the RST.
The references resolver processes the node and tries to lookup _io.BytesIO. This reference does not exist.
Warnings are emitted by another post-transformation.

One way to fix it is to pre-process nodes before calling the reference resolver. This can be done by remapping fully-qualified names into names that are known to exist. I had something locally setup for that but I'm not sure it's universal enough. In addition the implementation depends on various other utilities that I implemented so I'm not even sure I can bundle it as a standalone fix.

electric-coder commented 2 weeks ago

It could be solved if CPython included an index for _io.BytesIO but that's not currently the case.

This might actually be a CPython problem, googling for the generic error message generally leads to https://bugs.python.org/issue11975 so maybe they should adjust their index for _io.BytesIO?

sphinx-doc / sphinx