Stub files with absolute imports

[nstarman]

This might be a deeper issue with AST, but the attach_stubs doesn't work in namespace packages that import from other modules in the namespace package.

An example structure that fails:

libs/namespace/
  core/
    pyproject.toml
    src/namespace/core/
      __init__.py
      thing.py
  plugin/
    pyproject.toml
    src/namespace/plugin/
      __init__.py  # uses `attach_stubs`
      __init__.pyi  # has `from namespace.core import thing as thing`

[stefanv]

@nstarman Can you make a minimal example and push it to a repo I can play with?

[nstarman]

I took a closer look and the underlying issue is actually that the AST parser can't handle absolute imports. I was encountering within a namespace package, but it's more generic. I think this will trigger the issue.

package/
  pyproject.toml
  src/package/
    __init__.py  # with `attach_stubs`
    __init__.pyi
    foo.py  # has class `Foo`
    bar.py  # imports `from package.foo import Foo`

Absolute imports are preferred over relative imports, but I think this might be fundamentally incompatible with lazy_loader. Hopefully I'm wrong!

[stefanv]

I don't think we'll be able to do anything about that, unfortunately. Feel free to re-open if you think that's incorrect.

[nstarman]

In a different discussion I suggested that lazy_loader package a mypy extension to allow mypy to understand lazy_loader's dynamic __all__, without the need for the .pyi stub file, which led to this Issue. Since this Issue is really upstream in the AST, maybe the mypy extension is the better solution. I made some progress on the extension in https://github.com/nstarman/lazy_loader/tree/mypy_plugin, but I ran into some difficulties I didn't understand since I'm not overly familiar with the internals of mypy.

[stefanv]

We have no guarantee that the generated __all__ will be used, of course. I thought the stub files provide enough guidance for mypi to function correctly?

[stefanv]

Thanks, I just saw your issue https://github.com/scientific-python/lazy_loader/issues/46 which explains the deficiency. I'll re-post my concerns there and we can clarify.