Open gdetrez opened 5 years ago
Hi @gdetrez and thank you for the bugreport.
modulefinder
can't find pkg1 and reports it as "Missing" (internally it calls it a bad module ;-)
c:\srv\tmp\pep420>\Python34\python.exe -m modulefinder main.py
Name File
---- ----
m __main__ main.py
P pkg2 pkg2\__init__.py
m pkg2.bar pkg2\bar.py
Missing modules:
? pkg1.foo imported from __main__
pydeps --externals
will list "bad modules":
c:\srv\tmp\pep420>pydeps --externals main.py
[
"pkg1",
"pkg2"
]
I suppose we could include all the "bad" modules in the graph as well, but with the current code they would be empty.
The best alternative would be to fix pydeps' copy of modulefinder (mf27.py
) to also find PEP420 modules, but at this point I'm not sure if that's even possible...?
Hi @thebjorn. Sorry for the radio silence.
I see that if I use the --include-missing
flag, I get a bit more complete picture. But it seems that I don't get the transitive dependencies of the namespaced package (i.e if main
imports ns1.foo
and ns1.foo
import ns1.bar
, the latter is not included in the graph).
While researching the same issue for mypy, I found python/mypy#5691 which implement PEP 420 support (it's behind a --namespace-packages
flag). The module is also called modulefinder
but it looks rather different than the one here so I'm not sure how much is reusable.
Otherwise, if you don't mind the extra dependency, maybe it'd be worth taking a look at modulegraph
by Ronald Oussoren, which already supports namespaces:
❯❯❯ python3 -m modulegraph main.py
Class Name File
----- ---- ----
Script /home/gregoire/experiments/20190128-pydeps-namespace/test2/main.py /home/gregoire/experiments/20190128-pydeps-namespace/test2/main.py
NamespacePackage ns1 -
SourceModule ns1.bar /home/gregoire/experiments/20190128-pydeps-namespace/test2/ns1/bar.py
SourceModule ns1.foo /home/gregoire/experiments/20190128-pydeps-namespace/test2/ns1/foo.py
Just tried pydeps
after using it a long time ago on the project I work on. We recently moved to namespace packages too, and now instead of the nice graph, I get only three nodes in the graph, and only if I specify pydeps cylc/flow
. It fails when running pydeps cylc
(parent directory). Looks like it's an issue with modulefinder
, so hopefully that'll be fixed soon there, so that I can use pydeps
again 🤞
That is a huge bug and --include-missing
seems to avoid it but not without nasty side effects. Sadly I will not be able to adopt pydeps as part of the docs until is fixed.
@ssbarnea I'm wondering if you're using this feature to create namespace package, or if it's simply to avoid having an __init__.py
file? I've never found a reason to use namespace packages that outweigh the mess it causes. Regardless, pydeps
is dependent on modulefinder
(a python standard module - vendorized with some bug-fixes) supporting the feature first. I'll happily merge a PR if you or anyone else wants to work on it though...
modulegraph seems to be built on the same foundational technology and does some work to support namespace packages (https://github.com/ronaldoussoren/modulegraph/blob/master/modulegraph/modulegraph.py#L63)
Thanks, will keep it in mind. Decided to go with findimports
. It is fast and does the trick for the time being. Although it doesn't have good clustering / collapsing, my needs are still simple enough for it to not be a problem.
When I need something more extensive & customisable I will have a look at modulegraph
, pydeps
and findmodules
in more depth.
PEP 420 introduced implicit namespace packages, which are packages without
__init__.py
. Those packages, and the modules under them, don't seem to appear in the dependency graph generated by pydeps.Here's an example. Given the following directory structure:
And
main.py
:pydeps main.py
generates the following graph:I would have expected
pkg1.foo
to be included in the dependency graph. (On the other hand, I'm not sure there's a point in havingpkg1
there since it's going to be empty.)