Pyright emits meaningless `reportMissingModuleSource` warnings for submodules of native extension

[apparebit]

The Bug

Pyright emits reportMissingModuleSource warnings for submodules implemented by the same native extension, even though:

• The Python sources for prettypretty include typing stubs for prettypretty.color (the primary native module) and submodules such as prettypretty.color.gamut and prettypretty.color.style

• They also include symbolic links for each of the submodules, which follow native library naming convention and point to the actual native library. E.g., prettypretty/color/gamut.abi3.so points to prettypretty/color.abi3.so.

  • That technique is blessed by PEP 489 for just this use case.
  • It seems to work otherwise: I can successfully import prettypretty.color or import prettypretty.color.style as the first statement in a fresh interactive session.

Typing stubs, native library, and symbolic library links really should be enough to appease the type checker. If we really want two independent sources of truth for accepting native code, I suggest loading the native library and inspecting that. I must admit I'm not a fan of the symbolic links. I also don't know what to do about Windows. My understanding is that Windows finally does support symbolic links even for files (yay!), but that most Windows users leave that feature disabled because symbolic links might turn into a vector for exploits (oops!).

The Details

• Prettypretty uses PyO3 for FFI. AFAIK, it doesn't currently use the multiphase extension initialization of PEP 489 but simply instantiates all module objects during initial loading.

  • PyO3's level of integration between Rust and Python is impressive. But its module wrangling is disappointing:

    • The module objects for submodules have incorrect values for basic module attributes such as __name__ (which is the simple name of the submodule, not the fully qualified name it should be) and __package__ (which is empty).
    • Submodules are not added to Python's sys.modules registry and are only accessible through attributes of the main module.
    • Prettypretty's module initialization function cleans up much of the mess left by PyO3, patching __name__ and __package__ attributes as well as registering submodules in sys.modules.
    • I don't think that PyO3's dynamic short-comings should nor do impact type checking. The typing stubs and symbolic links should be sufficient for establishing the modules' existence. But since importing is a related topic, I figured I'd mention this.

• VS Code or the command line as well as Pyright versions seem to make no difference. The warnings happen in:

  • up-to-date VS Code (1.92.1) with Pylance 2024.8.1
  • up-to-date VS Code (1.92.1) with Pylance 2024.8.1, Remote, SSH flavor
  • a somewhat older NPM package for Pyright (1.1.361)
  • the latest version of the package (1.1.375)

• All other software is of similarly recent yet stable vintage:

  • CPython 3.12.2
  • stable Rust toolchain, 1.80.0
  • PyO3 0.22.0
  • macOS 14.5 and 14.6.1, Intel only, but can test Apple Silicon if necessary

• The one thing I noticed in Pyright's verbose output is that it repeatedly tries to find the same source files. I assume it doesn't cache failures then. Otherwise, I couldn't glean anything else from the trace. It is attached below.

• If you want to try for yourself, you can just:

  • clone the repository
  • in the root, run ./rr.sh install to install necessary tools
  • once everything is installed, activate the just created virtual env
  • run ./rr.sh to build the Python extension module
  • run ./rr.sh check to check project, including with Pyright

While prettypretty does build and run on Windows for CI, the above steps only work with (macOS + Homebrew) or (Linux + APT) for now.

The Gory Details

Verbose command line output

Prettypretty uses `# pyright: ignore [reportMissingModuleSource]` comments to avoid useless warnings. I did enable one of them for this run. ``` (.venv) rgrimm@Precious prettypretty % npm run pyright -- --verbose --pythonpath ./.venv/bin/python > pyright > pyright --verbose --pythonpath ./.venv/bin/python Setting pythonPath for service "": "/Users/rgrimm/Work/prettypretty/.venv/bin/python" Loading pyproject.toml file at /Users/rgrimm/Work/prettypretty/pyproject.toml Auto-excluding **/node_modules Auto-excluding **/__pycache__ Auto-excluding **/.* Search paths for file:///Users/rgrimm/Work/prettypretty /Users/rgrimm/Work/prettypretty/node_modules/pyright/dist/typeshed-fallback/stdlib /Users/rgrimm/Work/prettypretty /Users/rgrimm/Work/prettypretty/src /Users/rgrimm/Work/prettypretty/typings /Users/rgrimm/Work/prettypretty/node_modules/pyright/dist/typeshed-fallback/stubs/... /Users/rgrimm/.pyenv/versions/3.12.2/lib/python3.12 /Users/rgrimm/.pyenv/versions/3.12.2/lib/python3.12/lib-dynload /Users/rgrimm/Work/prettypretty/.venv/lib/python3.12/site-packages /Users/rgrimm/Work/prettypretty Found 20 source files Could not resolve source for '.color.style' in file '/Users/rgrimm/Work/prettypretty/prettypretty/color_types.py' Attempting to resolve relative import Attempting to resolve using root path 'file:///Users/rgrimm/Work/prettypretty/prettypretty' Resolved import with file 'file:///Users/rgrimm/Work/prettypretty/prettypretty/color/__init__.pyi' Resolved import with file 'file:///Users/rgrimm/Work/prettypretty/prettypretty/color/style.pyi' Attempting to resolve using root path 'file:///Users/rgrimm/Work/prettypretty/prettypretty' Resolved import with file 'file:///Users/rgrimm/Work/prettypretty/prettypretty/color.abi3.so' Resolved with native lib '/Users/rgrimm/Work/prettypretty/prettypretty/color.abi3.so' Did not find file 'file:///Users/rgrimm/Work/prettypretty/prettypretty/color.pyi' or 'file:///Users/rgrimm/Work/prettypretty/prettypretty/color.py' Could not resolve source for '.color.style' in file '/Users/rgrimm/Work/prettypretty/prettypretty/grid.py' Attempting to resolve relative import Attempting to resolve using root path 'file:///Users/rgrimm/Work/prettypretty/prettypretty' Resolved import with file 'file:///Users/rgrimm/Work/prettypretty/prettypretty/color/__init__.pyi' Resolved import with file 'file:///Users/rgrimm/Work/prettypretty/prettypretty/color/style.pyi' Attempting to resolve using root path 'file:///Users/rgrimm/Work/prettypretty/prettypretty' Resolved import with file 'file:///Users/rgrimm/Work/prettypretty/prettypretty/color.abi3.so' Resolved with native lib '/Users/rgrimm/Work/prettypretty/prettypretty/color.abi3.so' Did not find file 'file:///Users/rgrimm/Work/prettypretty/prettypretty/color.pyi' or 'file:///Users/rgrimm/Work/prettypretty/prettypretty/color.py' Could not resolve source for '.color.trans' in file '/Users/rgrimm/Work/prettypretty/prettypretty/theme.py' Attempting to resolve relative import Attempting to resolve using root path 'file:///Users/rgrimm/Work/prettypretty/prettypretty' Resolved import with file 'file:///Users/rgrimm/Work/prettypretty/prettypretty/color/__init__.pyi' Resolved import with file 'file:///Users/rgrimm/Work/prettypretty/prettypretty/color/trans.pyi' Attempting to resolve using root path 'file:///Users/rgrimm/Work/prettypretty/prettypretty' Resolved import with file 'file:///Users/rgrimm/Work/prettypretty/prettypretty/color.abi3.so' Resolved with native lib '/Users/rgrimm/Work/prettypretty/prettypretty/color.abi3.so' Did not find file 'file:///Users/rgrimm/Work/prettypretty/prettypretty/color.pyi' or 'file:///Users/rgrimm/Work/prettypretty/prettypretty/color.py' Could not resolve source for '.color.style' in file '/Users/rgrimm/Work/prettypretty/prettypretty/terminal.py' Attempting to resolve relative import Attempting to resolve using root path 'file:///Users/rgrimm/Work/prettypretty/prettypretty' Resolved import with file 'file:///Users/rgrimm/Work/prettypretty/prettypretty/color/__init__.pyi' Resolved import with file 'file:///Users/rgrimm/Work/prettypretty/prettypretty/color/style.pyi' Attempting to resolve using root path 'file:///Users/rgrimm/Work/prettypretty/prettypretty' Resolved import with file 'file:///Users/rgrimm/Work/prettypretty/prettypretty/color.abi3.so' Resolved with native lib '/Users/rgrimm/Work/prettypretty/prettypretty/color.abi3.so' Did not find file 'file:///Users/rgrimm/Work/prettypretty/prettypretty/color.pyi' or 'file:///Users/rgrimm/Work/prettypretty/prettypretty/color.py' Heap stats: total_memory_size=131072MB, total_free_size=72256MB, total_heap_size=131MB, used_heap_size=106MB, cross_worker_used_heap_size=106MB, total_physical_size=131MB, total_available_size=4036MB, heap_size_limit=4144MB Could not resolve source for '.color.style' in file '/Users/rgrimm/Work/prettypretty/prettypretty/style_extras.py' Attempting to resolve relative import Attempting to resolve using root path 'file:///Users/rgrimm/Work/prettypretty/prettypretty' Resolved import with file 'file:///Users/rgrimm/Work/prettypretty/prettypretty/color/__init__.pyi' Resolved import with file 'file:///Users/rgrimm/Work/prettypretty/prettypretty/color/style.pyi' Attempting to resolve using root path 'file:///Users/rgrimm/Work/prettypretty/prettypretty' Resolved import with file 'file:///Users/rgrimm/Work/prettypretty/prettypretty/color.abi3.so' Resolved with native lib '/Users/rgrimm/Work/prettypretty/prettypretty/color.abi3.so' Did not find file 'file:///Users/rgrimm/Work/prettypretty/prettypretty/color.pyi' or 'file:///Users/rgrimm/Work/prettypretty/prettypretty/color.py' pyright 1.1.375 /Users/rgrimm/Work/prettypretty/prettypretty/plot.py /Users/rgrimm/Work/prettypretty/prettypretty/plot.py:26:5 - warning: Import ".color.gamut" could not be resolved from source (reportMissingModuleSource) 0 errors, 1 warning, 0 informations Completed in 1.495sec ```

[erictraut]

The reportMissingModuleSource check arguably doesn't belong in a type checker. It was added by the pylance team for the benefit of their users. It makes more sense when pyright is being used to power a language server.

That said, I agree that this is a bogus error. The check for reportMissingModuleSource should be smart enough to ignore cases where the import statement is referencing a submodule of a native module.

@debonte, @rchiodo could someone from the pylance team please take ownership of this issue since you're the most familiar with the code that implements the reportMissingModuleSource check?

BTW, I was initially unable to repro this bug because I thought it was talking about code that imports the prettypretty library. The bug report is actually talking about type checking the prettypretty code itself. Here are the repro steps:

1. Clone the prettypretty repo.
2. Open the style_extras.py file and delete the # pyright: ignore [reportMissingModuleSource] comment.

[apparebit]

BTW, I was initially unable to repro this bug because I thought it was talking about code that imports the prettypretty library. The bug report is actually talking about type checking the prettypretty code itself.

Oops, good point. Thanks for the new and improved repro steps!