autodoc can't detect overloads for functions defined in other file

[akpircher]

Describe the bug

Related to #7786, overloads for functions defined in a "private" (or other) module with overloads but exposed in a public-facing __init__.py are not detected

How to Reproduce

# demo/__init__.py
from ._private import foobar

__all__ = ['foobar']

# demo/_private.py
from __future__ import annotations

from typing import overload

from typing_extensions import Literal

@overload
def foobar(a: Literal[True]) -> Literal['a']:
    ...

@overload
def foobar(a: Literal[False]) -> Literal['b']:
    ...

@overload
def foobar(a: bool) -> Literal['a', 'b']:
    ...

def foobar(a: bool) -> Literal['a', 'b']:
    """return either a or b"""
    if a:
        return 'a'
    return 'b'

# index.rst

.. automodule:: demo
   :members:
   :undoc-members:

Expected output:

demo.foobar(a: Literal[True]) -> Literal['a']
demo.foobar(a: Literal[False]) -> Literal['b']
demo.foobar(a: bool) -> Literal['a', 'b']

   return either a or b

Environment Information

Platform:              darwin; (macOS-13.3.1-x86_64-i386-64bit)
Python version:        3.10.9 (main, Mar 30 2023, 13:44:33) [Clang 14.0.0 (clang-1400.0.29.202)])
Python implementation: CPython
Sphinx version:        6.1.3
Docutils version:      0.19
Jinja2 version:        3.1.2
Pygments version:      2.14.0

Sphinx extensions

["sphinx.ext.autodoc"]

Additional context

No response

[picnixz]

• Does the issue persist with the latest version? (I think it does but let's check).
• Does the issue occur when it's not the __init__.py file?

I know that imports in __init__.py files lead to some bad surprises since I also faced them and I know that there are other similar issues so I want to know if your problem also impacts other files, e.g., demo/public.py instead of demo/__init__.py.

[akpircher]

@picnixz yes, I just confirmed that it's still an issue.

It's been a minute and I needed a solution, so I created a monkeypatch locally that I'm using, which solves the problem for me.

I basically have an overload_fix.py module in my docs directory that's a copy-paste of sphinx.ext.autodoc.Documenter.generate, except that I've replaced the if self.real_modname != guess_modname section with

    if self.real_modname != guess_modname:
        # Add module to dependency list if target object is defined in other module.
        try:
            analyzer = ModuleAnalyzer.for_module(guess_modname)
            self.directive.record_dependencies.add(analyzer.srcname)
            if self.analyzer:
                analyzer.find_attr_docs()
                for name, signatures in analyzer.overloads.items():
                    if name not in self.analyzer.overloads:
                        self.analyzer.overloads[name] = signatures
        except PycodeError:
            pass

It looks like if the object is a function, instead of a method, the analyzer needs to run twice in order for the overload to be found. I think I took this from the solution for the other issue.

But I'm not super familiar with this code base, so I'm not sure if that's the right solution to go with.

[akpircher]

I guess for consistency, my conf.py file now has the following:

from docs.overload_fix import generate

sphinx.ext.autodoc.Documenter.generate = generate

[akpircher]

I didn't answer both your questions, sorry. @picnixz, it is an issue with imported overloaded functions, such as in the __init__.py.

If I were to run autodoc on demo/public.py (and it defines the overloaded function), then the overloads show. It's only hidden when an overloaded function is imported, and autodoc is run on the file with the import (and having the documentation in the module with the import is desired).

(Edited to fix my mis-wording/contradiction)

Adding demo/public.py with:

# demo/public.py
from ._private import foobar

__all__ = ['foobar']

# index.rst

.. automodule:: demo
   :members:
   :undoc-members:

.. automodule:: demo.public
   :members:
   :undoc-members:

Then make -C docs/ text does not generate the overloads:

# index.rst

demo.foobar(a: bool) -> Literal['a', 'b']

   return either a or b

demo.public.foobar(a: bool) -> Literal['a', 'b']

   return either a or b

[picnixz]

What I meant is actually: importing the overloaded function in public.py and not in __init__.py and re-exporting it in public.py as well (not defining it in public.py).

But I think that when we document the members of __init__.py, we actually assume that its defining module actually __init__.py and not _private hence we don't actually parse _private.

Long story short: should work with public, fails with the init file.

[akpircher]

It does not work either in public or init. Sorry, I probably should've added the samples as a separate commant, but I didn't want to bombard with emails, so I edited my previous comment.

Here is the current running environment for the edited answer above:

Platform:              darwin; (macOS-13.5.1-arm64-arm-64bit)
Python version:        3.10.12 (main, Aug 14 2023, 18:10:04) [Clang 14.0.3 (clang-1403.0.22.14.1)])
Python implementation: CPython
Sphinx version:        7.2.5
Docutils version:      0.19
Jinja2 version:        3.1.2
Pygments version:      2.14.0

(Does the previous edit answer your question?)

[picnixz]

Thank you. I think the patch in the PR of the issue you linked only concerns methods and not functions so that's why it won't work in the public module (I thought it was actually due to the __init__.py but it was a more general issue).

I will work on this issue next month as I don't have access to a computer (currently I am doing everything I can on my phone with termux but there are limitations to what I can do).

[picnixz]

Sorry for the very late reply. I didn't have any time for this one. Since I don't have a straightforward answer to your issue, I won't work now on this. As such, if anyone wants to help, that would be good !

[akpircher]

I have a local workaround, I'll create a PR based on what I implemented locally in a fork.