mypy fails importing my package with --silent-imports and -p

[The-Compiler]

When I run mypy over qutebrowser, things look good initially (apart from some use of type annotations for my own purposes):

$ mypy -m qutebrowser
qutebrowser/qutebrowser.py:27: note: In module imported here,
qutebrowser.py:24: note: ... from here:
qutebrowser/misc/checkpyver.py:29: error: No library stub file for standard library module 'tkinter'
qutebrowser/misc/checkpyver.py:29: note: (Stub files are from https://github.com/python/typeshed)
qutebrowser/misc/checkpyver.py:33: error: Cannot find module named 'Tkinter'
qutebrowser/misc/checkpyver.py:33: note: (Perhaps setting MYPYPATH or using the "--silent-imports" flag would help)
qutebrowser/misc/checkpyver.py:34: error: Cannot find module named 'tkMessageBox'
qutebrowser/qutebrowser.py:42: note: In module imported here,
qutebrowser.py:24: note: ... from here:
qutebrowser/misc/earlyinit.py:28: error: Cannot find module named 'hunter'
qutebrowser/misc/earlyinit.py:34: error: Cannot find module named 'faulthandler'
qutebrowser/misc/earlyinit.py:40: error: No library stub file for standard library module 'tkinter'
qutebrowser/misc/earlyinit.py:83: error: No library stub file for module 'PyQt5.QtWidgets'
qutebrowser/misc/earlyinit.py:84: error: No library stub file for module 'PyQt5.QtCore'
qutebrowser/misc/earlyinit.py:222: error: No library stub file for module 'PyQt5.QtNetwork'
qutebrowser/qutebrowser.py:156: note: In module imported here,
qutebrowser.py:24: note: ... from here:
qutebrowser/app.py:34: error: No library stub file for module 'PyQt5.QtWidgets'
qutebrowser/app.py:35: error: No library stub file for module 'PyQt5.QtWebKit'
qutebrowser/app.py:36: error: No library stub file for module 'PyQt5.QtGui'
qutebrowser/app.py:37: error: No library stub file for module 'PyQt5.QtCore'
qutebrowser/app.py:40: error: Cannot find module named 'hunter'
qutebrowser/app.py:52: note: In module imported here,
qutebrowser/qutebrowser.py:156: note: ... from here,
qutebrowser.py:24: note: ... from here:
qutebrowser/misc/sessions.py: note: In member "session_save" of class "SessionManager":
qutebrowser/misc/sessions.py:390: error: Parse error before {
qutebrowser/misc/sessions.py:391: error: Parse error before )
qutebrowser/misc/sessions.py: note: In class "SessionManager":
qutebrowser/misc/sessions.py:391: error: Parse error before :
qutebrowser/misc/sessions.py:392: error: Inconsistent indentation
qutebrowser/misc/sessions.py: note: At top level:
qutebrowser/misc/sessions.py:443: error: Inconsistent indentation

I then tried using --silent-imports as I don't care about the missing stubs for now:

$ mypy --silent-imports -m qutebrowser
qutebrowser.py:29: error: "module" has no attribute "main"

--almost-silent reveals a bit more:

$ mypy --silent-imports --almost-silent -m qutebrowser
qutebrowser.py:24: note: Import of 'qutebrowser.qutebrowser' silently ignored
qutebrowser.py:24: note: (Using --silent-imports, module not passed on command line)
qutebrowser.py:24: note: (This note courtesy of --almost-silent)
qutebrowser.py:29: error: "module" has no attribute "main"

Note my directory structure looks like this:

├── qutebrowser
│   ├── app.py
│   ├── [...]
│   ├── qutebrowser.py
│   ├── resources.py
│   └── utils
│       [...]
├── [...]
├── qutebrowser.py

That is, there's a toplevel qutebrowser.py which is a simple launcher script:

import qutebrowser.qutebrowser
import sys

if __name__ == '__main__':
    sys.exit(qutebrowser.qutebrowser.main())

Then there's the qutebrowser package, and a qutebrowser.py in it which starts the actual application (in main()).

I should probably rename at least qutebrowser/qutebrowser.py and maybe the toplevel qutebrowser.py, but I think there's still no reason mypy should get confused by it with --silent-imports.

[JukkaL]

-m if used together with --silent-imports only type checks the target module (in this case, qutebrowser/__init__.py). If you want to recursively type check submodules, you can use qutebrowser/ (without -m) or -p qutebrowser.

[The-Compiler]

-p doesn't seem to change anything:

$ mypy --silent-imports --almost-silent -p qutebrowser
qutebrowser.py:24: note: Import of 'qutebrowser.qutebrowser' silently ignored
qutebrowser.py:24: note: (Using --silent-imports, module not passed on command line)
qutebrowser.py:24: note: (This note courtesy of --almost-silent)
qutebrowser.py:29: error: "module" has no attribute "main"

Using qutebrowser/ seems to work, i.e. I get some more parse errors which will go away when I fix my annotations :wink:

This whole behaviour strikes me as unintuitive though. Why does --silent-imports change the behaviour of -m and -p? What's the difference between the two? Why couldn't -m check the whole package if a package is passed, similarily to what python does? It seems to do exactly that without --silent-imports?

[gvanrossum]

The problem seems to be confusing between your toplevel qutebrowser.py and toplevel package qutebrowser. That looks like a genuine mypy bug, though I'm not sure exactly where. It seems the -p flag somehow manages to lie about the name of the file it actually reads vs. the file it claims to read; the -v flag shows no signs that the toplevel qutebrowser.py is ever read. How -s affects this is also not immediately clear (but there are definitely dragons there).

One work-around would be to skip the -p flag altogether and just say mypy -s qutebrowser (as of the 0.4 release the trailing slash is no longer needed). That seems to work. (It reveals problems with qutebrowser/browser/commands.py which you can silence using @no_type_check.)

[The-Compiler]

Thanks! I'm actually working on getting rid of my own weird dict type annotations so I can consistently use PEP484 annotations.

(I remember we talked about this a while back and you recommended to use decorators, while I insisted a dict annotation wasn't as ugly as you feeled it was. Now that I switched to decorators I realized you were right. :laughing: )

[gvanrossum]

I was wrong about one thing: mypy doesn't lie about which module it's reading (I was confused because there are so many things named qutebrowser here :-).

However the genuine bug is that when choosing between module x.py and package x/ (containing x/__init__.py), mypy incorrectly prefers the module x.py, whereas the official algorithm prefers the package x/. I find this pretty serious.