search

thebjorn / pydeps

Python Module Dependency graphs
https://pydeps.readthedocs.io/en/latest/
BSD 2-Clause "Simplified" License
1.8k stars 114 forks source link

Resolving relative dependencies? #200

Open JohnPeng47 opened 1 year ago

JohnPeng47 commented 1 year ago

Does Pydeps not resolve relative dependencies? (Aside: are relative dependencies considered malpractise in python?)

Following command: python -m pydeps --show-deps --debug route.py

Does not show any of the relative dependencies imported like so from .schema import CreateGraphReq, ... from .service import GraphManager, ....

Cheers (also thanks for making this, especially looking forward to using the intermediate output for a project I'm working on for writing automatic test cases)

thebjorn commented 1 year ago

Does Pydeps not resolve relative dependencies?

It should (there are tests: https://github.com/thebjorn/pydeps/blob/master/tests/test_relative_imports.py). If you have a testcase I can look into it.

Aside: are relative dependencies considered malpractise in python?

Relative imports are kosher in Python.

JohnPeng47 commented 1 year ago

Hey thanks for the quick response, in the middle of restructuring my Python dir layout, was using really weird pathing with potentially some circular dependencies along the way, will rerun pydeps after and let you know

JohnPeng47 commented 1 year ago

Here, I ran my command from the root of this directory: python -m pydeps --show-deps --max-module-depth 2 src\server\routes\graph\route.py

Here is the file: https://github.com/JohnPeng47/KongMono/blob/master/KongServer/src/server/routes/graph/route.py

Here is a snippet of the generated deps file: { "fastapi": { "bacon": 1, "imported_by": [ "fastapi.applications", "fastapi.param_functions", "fastapi.routing", "route.py" ], "imports": [ "fastapi.applications", "fastapi.background", "fastapi.datastructures", "fastapi.exceptions",

The generated deps file only shows fastapi and networkx, but it misses both my relative imports and the src.KongBot imports

itaowei commented 11 months ago

I also have the similar problem and one test case for that issue is this file under the repository matplotlib.

The error message is shown below

Traceback (most recent call last):
..........
  File "/$HOME/anaconda3/envs/AA/lib/python3.11/site-packages/pydeps/py2depgraph.py", line 237, in py2dep
    mf.run_script(dummy.fname)
  File "/$HOME/anaconda3/envs/AA/lib/python3.11/site-packages/pydeps/py2depgraph.py", line 122, in run_script
    self.load_module('__main__', fp, pathname, stuff)
  File "/$HOME/anaconda3/envs/AA/lib/python3.11/site-packages/pydeps/py2depgraph.py", line 159, in load_module
    module = mf27.ModuleFinder.load_module(
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/$HOME/anaconda3/envs/AA/lib/python3.11/site-packages/pydeps/mf27.py", line 153, in load_module
    self.scan_code(co, m)
  File "/$HOME/anaconda3/envs/AA/lib/python3.11/site-packages/pydeps/mf27.py", line 206, in scan_code
    parent = self.determine_parent(m, level=level)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/$HOME/anaconda3/envs/AA/lib/python3.11/modulefinder.py", line 188, in determine_parent
    raise ImportError("relative importpath too deep")
ImportError: relative importpath too deep
thebjorn commented 11 months ago

@itaowei interesting. The error is coming from Python's modulefinder module, and indeed if you run in directly you'll see the same result: 

(pydeps200) go|C:\srv\venv\pydeps200\Lib\site-packages> python -m modulefinder matplotlib\__init__.py
Traceback (most recent call last):
  File "<frozen runpy>", line 198, in _run_module_as_main
  File "<frozen runpy>", line 88, in _run_code
  File "C:\Program Files\Python311\Lib\modulefinder.py", line 664, in <module>
    mf = test()
         ^^^^^^
  File "C:\Program Files\Python311\Lib\modulefinder.py", line 657, in test
    mf.run_script(script)
  File "C:\Program Files\Python311\Lib\modulefinder.py", line 153, in run_script
    self.load_module('__main__', fp, pathname, stuff)
  File "C:\Program Files\Python311\Lib\modulefinder.py", line 349, in load_module
    self.scan_code(co, m)
  File "C:\Program Files\Python311\Lib\modulefinder.py", line 436, in scan_code
    parent = self.determine_parent(m, level=level)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Program Files\Python311\Lib\modulefinder.py", line 188, in determine_parent
    raise ImportError("relative importpath too deep")
ImportError: relative importpath too deep

If I modify the modulefinder.py code to print out what is causing the problem:

    def determine_parent(self, caller, level=-1):
        self.msgin(4, "determine_parent", caller, level)
        if not caller or level == 0:
            self.msgout(4, "determine_parent -> None")
            return None
        pname = caller.__name__
        if level >= 1: # relative import
            if caller.__path__:
                level -= 1
            if level == 0:
                parent = self.modules[pname]
                assert parent is caller
                self.msgout(4, "determine_parent ->", parent)
                return parent
            if pname.count(".") < level:
                raise ImportError(f"relative importpath too deep for {pname}, level {level}")

I get the following output:

(pydeps200) go|C:\srv\venv\pydeps200\Lib\site-packages> python -m modulefinder matplotlib\__init__.py
Traceback (most recent call last):
  File "<frozen runpy>", line 198, in _run_module_as_main
  File "<frozen runpy>", line 88, in _run_code
  File "C:\Program Files\Python311\Lib\modulefinder.py", line 664, in <module>
    mf = test()
         ^^^^^^
  File "C:\Program Files\Python311\Lib\modulefinder.py", line 657, in test
    mf.run_script(script)
  File "C:\Program Files\Python311\Lib\modulefinder.py", line 153, in run_script
    self.load_module('__main__', fp, pathname, stuff)
  File "C:\Program Files\Python311\Lib\modulefinder.py", line 349, in load_module
    self.scan_code(co, m)
  File "C:\Program Files\Python311\Lib\modulefinder.py", line 436, in scan_code
    parent = self.determine_parent(m, level=level)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Program Files\Python311\Lib\modulefinder.py", line 188, in determine_parent
    raise ImportError(f"relative importpath too deep for {pname}, level {level}")
ImportError: relative importpath too deep for __main__, level 1

Which tells me that it isn't due to excessive directory depth in the source tree...

Looking at the matplotlib/__init__.py file I can see that they're doing some hacks to get around circular imports. I don't have the time to dig into it right now, but my assumption is that they're doing "things" with partially initialized modules that modulefinder can't follow...

ps: running modulefinder with maximum debug options doesn't seem to give much more clarity

[...]
                            load_module -> Module('numpy', 'C:\\srv\\venv\\pydeps200\\Lib\\site-packages\\numpy\\__init__.py', ['C:\\srv\\venv\\pydeps200\\Lib\\site-packages\\numpy'])
                        import_module -> Module('numpy', 'C:\\srv\\venv\\pydeps200\\Lib\\site-packages\\numpy\\__init__.py', ['C:\\srv\\venv\\pydeps200\\Lib\\site-packages\\numpy'])
                    find_head_package -> (Module('numpy', 'C:\\srv\\venv\\pydeps200\\Lib\\site-packages\\numpy\\__init__.py', ['C:\\srv\\venv\\pydeps200\\Lib\\site-packages\\numpy']), '')
                        load_tail Module('numpy', 'C:\\srv\\venv\\pydeps200\\Lib\\site-packages\\numpy\\__init__.py', ['C:\\srv\\venv\\pydeps200\\Lib\\site-packages\\numpy']) ''
                    load_tail -> Module('numpy', 'C:\\srv\\venv\\pydeps200\\Lib\\site-packages\\numpy\\__init__.py', ['C:\\srv\\venv\\pydeps200\\Lib\\site-packages\\numpy'])
                    import_hook 'packaging.version' Module('__main__', 'matplotlib\\__init__.py') None 0
                        determine_parent Module('__main__', 'matplotlib\\__init__.py') 0
                    determine_parent -> None
                        find_head_package None 'packaging.version'
                            import_module 'packaging' 'packaging' None
                        import_module -> Module('packaging', 'C:\\srv\\venv\\pydeps200\\Lib\\site-packages\\packaging\\__init__.py', ['C:\\srv\\venv\\pydeps200\\Lib\\site-packages\\packaging'])
                    find_head_package -> (Module('packaging', 'C:\\srv\\venv\\pydeps200\\Lib\\site-packages\\packaging\\__init__.py', ['C:\\srv\\venv\\pydeps200\\Lib\\site-packages\\packaging']), 'version')
                        load_tail Module('packaging', 'C:\\srv\\venv\\pydeps200\\Lib\\site-packages\\packaging\\__init__.py', ['C:\\srv\\venv\\pydeps200\\Lib\\site-packages\\packaging']) 'version'
                            import_module 'version' 'packaging.version' Module('packaging', 'C:\\srv\\venv\\pydeps200\\Lib\\site-packages\\packaging\\__init__.py', ['C:\\srv\\venv\\pydeps200\\Lib\\site-packages\\packaging'])
                        import_module -> Module('packaging.version', 'C:\\srv\\venv\\pydeps200\\Lib\\site-packages\\packaging\\version.py')
                    load_tail -> Module('packaging.version', 'C:\\srv\\venv\\pydeps200\\Lib\\site-packages\\packaging\\version.py')
                    import_hook 'packaging.version' Module('__main__', 'matplotlib\\__init__.py') ['parse'] 0
                        determine_parent Module('__main__', 'matplotlib\\__init__.py') 0
                    determine_parent -> None
                        find_head_package None 'packaging.version'
                            import_module 'packaging' 'packaging' None
                        import_module -> Module('packaging', 'C:\\srv\\venv\\pydeps200\\Lib\\site-packages\\packaging\\__init__.py', ['C:\\srv\\venv\\pydeps200\\Lib\\site-packages\\packaging'])
                    find_head_package -> (Module('packaging', 'C:\\srv\\venv\\pydeps200\\Lib\\site-packages\\packaging\\__init__.py', ['C:\\srv\\venv\\pydeps200\\Lib\\site-packages\\packaging']), 'version')
                        load_tail Module('packaging', 'C:\\srv\\venv\\pydeps200\\Lib\\site-packages\\packaging\\__init__.py', ['C:\\srv\\venv\\pydeps200\\Lib\\site-packages\\packaging']) 'version'
                            import_module 'version' 'packaging.version' Module('packaging', 'C:\\srv\\venv\\pydeps200\\Lib\\site-packages\\packaging\\__init__.py', ['C:\\srv\\venv\\pydeps200\\Lib\\site-packages\\packaging'])
                        import_module -> Module('packaging.version', 'C:\\srv\\venv\\pydeps200\\Lib\\site-packages\\packaging\\version.py')
                    load_tail -> Module('packaging.version', 'C:\\srv\\venv\\pydeps200\\Lib\\site-packages\\packaging\\version.py')
                        determine_parent Module('__main__', 'matplotlib\\__init__.py') 1
Traceback (most recent call last):
  File "<frozen runpy>", line 198, in _run_module_as_main
  File "<frozen runpy>", line 88, in _run_code
  File "C:\Program Files\Python311\Lib\modulefinder.py", line 664, in <module>
    mf = test()
         ^^^^^^
  File "C:\Program Files\Python311\Lib\modulefinder.py", line 657, in test
    mf.run_script(script)
  File "C:\Program Files\Python311\Lib\modulefinder.py", line 153, in run_script
    self.load_module('__main__', fp, pathname, stuff)
  File "C:\Program Files\Python311\Lib\modulefinder.py", line 349, in load_module
    self.scan_code(co, m)
  File "C:\Program Files\Python311\Lib\modulefinder.py", line 436, in scan_code
    parent = self.determine_parent(m, level=level)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Program Files\Python311\Lib\modulefinder.py", line 188, in determine_parent
    raise ImportError(f"relative importpath too deep for {pname}, level {level}")
ImportError: relative importpath too deep for __main__, level 1
thebjorn commented 11 months ago

Hmm... it might actually be numpy..?

(pydeps200) go|C:\srv\venv\pydeps200\Lib\site-packages> python -m modulefinder numpy\__init__.py
Traceback (most recent call last):
  File "<frozen runpy>", line 198, in _run_module_as_main
  File "<frozen runpy>", line 88, in _run_code
  File "C:\Program Files\Python311\Lib\modulefinder.py", line 664, in <module>
    mf = test()
         ^^^^^^
  File "C:\Program Files\Python311\Lib\modulefinder.py", line 657, in test
    mf.run_script(script)
  File "C:\Program Files\Python311\Lib\modulefinder.py", line 153, in run_script
    self.load_module('__main__', fp, pathname, stuff)
  File "C:\Program Files\Python311\Lib\modulefinder.py", line 349, in load_module
    self.scan_code(co, m)
  File "C:\Program Files\Python311\Lib\modulefinder.py", line 436, in scan_code
    parent = self.determine_parent(m, level=level)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Program Files\Python311\Lib\modulefinder.py", line 188, in determine_parent
    raise ImportError(f"relative importpath too deep for {pname}, level {level}")
ImportError: relative importpath too deep for __main__, level 1
ZiqianXu commented 7 months ago

Hey @thebjorn, were you able to root cause the issue? I'm having similar issue with relative path import via from . import <moduleName>. It's for a simple OSS module grpc python file _server.py

python3 -m pydeps grpc/aio/_server.py --show-deps --pylib --no-output --noshow --include-missing
Traceback (most recent call last):
  File "/usr/lib/python3.8/runpy.py", line 194, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/usr/lib/python3.8/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "/usr/local/lib/python3.8/dist-packages/pydeps/__main__.py", line 2, in <module>
    pydeps()
  File "/usr/local/lib/python3.8/dist-packages/pydeps/pydeps.py", line 175, in pydeps
    return _pydeps(inp, **_args)
  File "/usr/local/lib/python3.8/dist-packages/pydeps/pydeps.py", line 39, in _pydeps
    dep_graph = py2depgraph.py2dep(trgt, **kw)
  File "/usr/local/lib/python3.8/dist-packages/pydeps/py2depgraph.py", line 237, in py2dep
    mf.run_script(dummy.fname)
  File "/usr/local/lib/python3.8/dist-packages/pydeps/py2depgraph.py", line 122, in run_script
    self.load_module('__main__', fp, pathname, stuff)
  File "/usr/local/lib/python3.8/dist-packages/pydeps/py2depgraph.py", line 159, in load_module
    module = mf27.ModuleFinder.load_module(
  File "/usr/local/lib/python3.8/dist-packages/pydeps/mf27.py", line 153, in load_module
    self.scan_code(co, m)
  File "/usr/local/lib/python3.8/dist-packages/pydeps/mf27.py", line 206, in scan_code
    parent = self.determine_parent(m, level=level)
  File "/usr/lib/python3.8/modulefinder.py", line 198, in determine_parent
    raise ImportError("relative importpath too deep")
ImportError: relative importpath too deep

Checking my python3.8 source path directory, all file with from . import <moduleName> returns the same type of error from pydeps.

Thanks for your help on debugging this!