Error: There are no .py[i] files in directory '.'

[insanitybit]

I've just installed the plugin and I get this error: There are no .py[i] files in directory '.'

There are certainly many python files in my directory structure, and I'm hitting 'run' while I have a Python file open. There's even an init.py file in the directory I'm running in.

I'm unsure what '.' is referring to, but if I specify the full path to the directory I'm trying to scan it does get the files (though it is giving incorrect errors about imports that it can't find, I suspect it can't understand that it should follow the venv/../site-packages/ ?)

I suspect that there are two issues here:

1) The '.' is referring to the root of my project, but the Python code is nested multiple folders deep before I get to the init.py

2) For some reason mypy needs an explicit MYPYPATH for the intellij configured venv

Regardless, the default installation of this plugin is broken for what I imagine is not an uncommon use case.

It seems like the plugin should

• Understand the file being viewed, and override the '.' to be a fully qualified path to that file
• Set MYPYPATH to the venv automatically

Assuming I understand the issues correctly.

[insanitybit]

For reference, this is the repo that I'm working on: https://github.com/insanitybit/grapl-analyzers

You can see that there are many directories, each with their own 'main.py', but in intellij I use a single project at the root of the git directory. I hope this provides some more context.

[ilevkivskyi]

Yes, there are two issues:

1. For the first issue using current file is not a good option. Mypy is not a linter, it can't type check a single file it needs to know the build scope, which is most typically the whole project. You can configure the plugin by replacing . with a relative path to root of your Python code (IOW use the same command you would use on command line from the project directory).
2. Setting MYPYPATH to site-packages is not recommended (and is actively discouraged) by mypy, you can find some better options in https://mypy.readthedocs.io/en/latest/running_mypy.html depending on your situation.

[ilevkivskyi]

Oh I just noticed your second comment. That directory structure may be tricky for mypy (not specific to the plugin). You can try --namespace-packages, or maybe just add an __init__.py to the root if possible.

[insanitybit]

Yeah, I'd tried --namespace-packages and the same error occurred. I could add an init.py to each directory, I suppose, it's just a bit odd. (edit: Adding init.py to every directory up to one of the analyzers did not change the error)

I've been reading that imports doc and I'm not sure what the intended solution for this is. Should I really add a new path to every submodule that I want checked in site-packages? It seems so strange to me that mypy doesn't discover these - is that just a current limitation, or am I missing something?

[ilevkivskyi]

is that just a current limitation, or am I missing something?

It may be both. One problem is that there are couple hard to fix bugs in --namespace-packages in mypy, another thing is static typing comes with all the burden of dependency/scope management from static languages. In particular mypy needs to know the boundaries of build units, if in your case each directory contains an independent self-contained program (that should not import each other), then I would instead recommend having multiple mypy builds. You can write a tiny script that runs mypy (or mypy daemon) on each directory, and then configure plugin to run this script.

[insanitybit]

Ah, yes I can work with the idea of just offloading this to a small script, that seems reasonable. Though I'm still unsure about how to solve the dependency issue.

As an example, I have the grapl_analyzerlib import, and it's in the venv site-packages. Should I just specify that directly in the MYPYPATH for each project's venv?

And thanks for the assistance so far.

[ilevkivskyi]

IIUC the best option may be to declare grapl_analyzerlib a PEP 561 package (probably by adding py.typed that specifies inline) and then just install it in the same venv where mypy is installed.

[insanitybit]

I see, I'll give that a try. Thanks.

[insanitybit]

https://github.com/insanitybit/grapl_analyzerlib/tree/master/grapl_analyzerlib

I have the py.typed files in every module. It is updated and in my virtual environemnt. I have sourced ./venv/bin/activate

MYPYPATH=./venv/lib/python3.7/site-packages/grapl_analyzerlib  dmypy run -- --follow-imports=skip "./analyzers/ssh_agent_ipc/main.py"
analyzers/ssh_agent_ipc/main.py:3: error: Cannot find module named 'grapl_analyzerlib.analyzer'
analyzers/ssh_agent_ipc/main.py:3: note: See https://mypy.readthedocs.io/en/latest/running_mypy.html#missing-imports
analyzers/ssh_agent_ipc/main.py:4: error: Cannot find module named 'grapl_analyzerlib.execution'
analyzers/ssh_agent_ipc/main.py:5: error: Cannot find module named 'grapl_analyzerlib.nodes.comparators'
analyzers/ssh_agent_ipc/main.py:6: error: Cannot find module named 'grapl_analyzerlib.prelude'
analyzers/ssh_agent_ipc/main.py:7: error: Cannot find module named 'grapl_ipc_analyzer_plugin.ipc_node'

I don't really understand why this isn't working. Perhaps I should open a mypy issue?

[ilevkivskyi]

No, there is a big blue box in the docs I sent before, around here https://mypy.readthedocs.io/en/latest/running_mypy.html#how-imports-are-found

Did you try following exactly what I proposed above?

[ilevkivskyi]

Also please read carefully https://mypy.readthedocs.io/en/latest/installed_packages.html#making-pep-561-compatible-packages.

Btw, since you develop both packages, you can also point MYPYPATH to the sources of grapl_analyzerlib.

[insanitybit]

I've read the document multiple times. As the author that works for me, but anyone using the grapl_analyzerlib will find that a bit complex. (edit: Actually, upon trying this, I still get the same exact errors).

Yeah, I tried with and without the path, but good to know that it explicitly will not work with the path I guess.

Every package in grapl_analyzerlib has a py.typed file. The py.typed file does not appear to be in the install, however, perhaps due to my setup.py being incorrect in some way. I can 'ls' the directory in the venv and see that it is not there.

When I 'python setup.py' I can see lines such as this: adding 'grapl_analyzerlib/py.typed'

My setup.py explicitly sets the package_data, as the document suggests. zip_safe=False, as the document suggests. I've tried multiple iterations of setting the package_data to no avail.

https://github.com/insanitybit/grapl_analyzerlib/blob/master/setup.py

[ilevkivskyi]

It's hard to tell what exactly is wrong with grapl_analyzerlib, I tried comparing with aiohttp (they also have https://github.com/aio-libs/aiohttp/blob/master/aiohttp/py.typed and I know it always worked well), but I didn't find anything.

Also, I just tried pip install -U git+https://github.com/insanitybit/grapl_analyzerlib on my laptop with a file that contains:

from grapl_analyzerlib.analyzer import Analyzer, OneOrMany
from grapl_analyzerlib.counters import SubgraphCounter
from grapl_analyzerlib.execution import ExecutionHit
from grapl_analyzerlib.prelude import ProcessQuery, FileQuery, ProcessView

And mypy returned no errors from the first time! So, sorry, can't help you here.

[ilevkivskyi]

Actually, I have one guess, you might need to restart the daemon forcefully. It doesn't pick up newly installed packages while daemon is up. Use dmypy stop.

[insanitybit]

Your guess was correct. dmypy stop did it. So I guess after package upgrades I'll have to run that.

Thanks.