`python-apigen` doesn't document type alias imported from another module

[mhostetter]

python-apigen doesn't detect / document type aliases when they have been imported from another module. Here's a simple example foo-153.zip that works when the alias is defined in __init__.py but doesn't when it is defined in another module and imported in __init__.py.

Works

# __init__.py
from typing import Union

IntOrStr = Union[int, str]
"""
A common type hint for an integer or string.
"""

Build output

``` $ rm -rf docs/build && rm -rf docs/api && sphinx-build -b dirhtml -v docs/ docs/build/ Running Sphinx v5.0.0 /usr/lib/python3/dist-packages/requests/__init__.py:89: RequestsDependencyWarning: urllib3 (1.26.11) or chardet (3.0.4) doesn't match a supported version! warnings.warn("urllib3 ({}) or chardet ({}) doesn't match a supported " making output directory... done locale_dir /mnt/c/Users/matth/repos/foo-153/docs/locales/en/LC_MESSAGES does not exists loading intersphinx inventory from https://docs.python.org/3/objects.inv... loading intersphinx inventory from https://numpy.org/doc/stable/objects.inv... locale_dir /mnt/c/Users/matth/repos/foo-153/docs/locales/en/LC_MESSAGES does not exists building [mo]: targets for 0 po files that are out of date building [dirhtml]: targets for 2 source files that are out of date updating environment: locale_dir /mnt/c/Users/matth/repos/foo-153/docs/locales/en/LC_MESSAGES does not exists [new config] 3 added, 0 changed, 0 removed reading sources... [ 33%] api reading sources... [ 66%] api/foo.IntOrStr reading sources... [100%] index /mnt/c/Users/matth/repos/foo-153/docs/api.rst.rst:7: WARNING: No top-level Python API group named: 'classes', valid groups are: ['public-members'] looking for now-outdated files... none found pickling environment... done checking consistency... done preparing documents... WARNING: unsupported theme option 'social' given done writing output... [ 33%] api writing output... [ 66%] api/foo.IntOrStr writing output... [100%] index generating indices... genindex done writing additional pages... done copying static files... done copying extra files... done dumping search index in English (code: en)... done dumping object inventory... done Generating sitemap for 4 pages in /mnt/c/Users/matth/repos/foo-153/docs/build/sitemap.xml build succeeded, 2 warnings. The HTML pages are in docs/build. ```

Doesn't work

# __init__.py
from ._typing import IntOrStr

# foo/_typing.py
from typing import Union

IntOrStr = Union[int, str]
"""
A common type hint for an integer or string.
"""

Build output

``` $ rm -rf docs/build && rm -rf docs/api && sphinx-build -b dirhtml -v docs/ docs/build/ Running Sphinx v5.0.0 /usr/lib/python3/dist-packages/requests/__init__.py:89: RequestsDependencyWarning: urllib3 (1.26.11) or chardet (3.0.4) doesn't match a supported version! warnings.warn("urllib3 ({}) or chardet ({}) doesn't match a supported " making output directory... done locale_dir /mnt/c/Users/matth/repos/foo-153/docs/locales/en/LC_MESSAGES does not exists loading intersphinx inventory from https://docs.python.org/3/objects.inv... loading intersphinx inventory from https://numpy.org/doc/stable/objects.inv... locale_dir /mnt/c/Users/matth/repos/foo-153/docs/locales/en/LC_MESSAGES does not exists building [mo]: targets for 0 po files that are out of date building [dirhtml]: targets for 2 source files that are out of date updating environment: locale_dir /mnt/c/Users/matth/repos/foo-153/docs/locales/en/LC_MESSAGES does not exists [new config] 2 added, 0 changed, 0 removed reading sources... [ 50%] api reading sources... [100%] index /mnt/c/Users/matth/repos/foo-153/docs/api.rst.rst:7: WARNING: No top-level Python API group named: 'classes', valid groups are: [] /mnt/c/Users/matth/repos/foo-153/docs/api.rst.rst:12: WARNING: No top-level Python API group named: 'public-members', valid groups are: [] looking for now-outdated files... none found pickling environment... done checking consistency... done preparing documents... WARNING: unsupported theme option 'social' given done writing output... [ 50%] api writing output... [100%] index generating indices... genindex done writing additional pages... done copying static files... done copying extra files... done dumping search index in English (code: en)... done dumping object inventory... done Generating sitemap for 3 pages in /mnt/c/Users/matth/repos/foo-153/docs/build/sitemap.xml build succeeded, 3 warnings. The HTML pages are in docs/build. ```

[jbms]

Currently apigen uses autodoc to decide which members of the module are documented.

In general this is a bit tricky, because sometimes symbols are imported into a module just because they are used by the module, while in other cases they are imported in order to be publicly exported.

There is __all__, but that isn't used by all modules.

Do you have a suggestion for how to decide which names are documented?

[2bndy5]

Could rst templates be used like auto-summary does?

[jbms]

I think rST templates are not ideal, because we need to know the complete list of entities to document before reading any of the rST, so we would need multiple passes. I think various inclusion/exclusion lists/patterns might fit better with how the extension currently works, e.g. similar to what is done for C++ in the work-in-progress cpp apigen branch.

[2bndy5]

similar to what is done for C++ in the work-in-progress cpp apigen branch.

This was my second idea. I think that would also solve #152

[jbms]

The case of attributes (as opposed to functions or classes) is actually particularly difficult to support, because the docstring is only found via Sphinx's ModuleAnalyzer that parses the source code of the module. Since the docstring is in _typing.py rather than __init__.py, it won't be found by ModuleAnalyzer. Currently ModuleAnalyzer is not sophisticated enough to resolve import statements, and then try to parse the source code of the other module to find attribute docs. Probably ModuleAnalyzer could be improved to do that, but if it happened by default, that might lead to a bunch of extra entities being documented.

[mhostetter]

Ahh... I didn't realize this was an autodoc issue. I confirmed this same behavior exists with other themes.

The case of attributes (as opposed to functions or classes) is actually particularly difficult to support, because the docstring is only found via Sphinx's ModuleAnalyzer that parses the source code of the module. Since the docstring is in _typing.py rather than init.py, it won't be found by ModuleAnalyzer. Currently ModuleAnalyzer is not sophisticated enough to resolve import statements, and then try to parse the source code of the other module to find attribute docs. Probably ModuleAnalyzer could be improved to do that, but if it happened by default, that might lead to a bunch of extra entities being documented.

I see the difficulty you're referring to...

There is __all__, but that isn't used by all modules. Do you have a suggestion for how to decide which names are documented?

I do use __all__ to indicate all public objects defined in a module, but I realize that may not be standard / not everyone uses it.

I'm not really sure what to suggest to do here. Perhaps I should just open an issue in Sphinx itself (if one doesn't already exist)?