Can't build documentation after creating submodules

fabifont commented 3 years ago

Problem Description

I had a project with that structure:

buchino/
|-- init.py
|-- data.py
|-- controller.py
|-- documents/
|-- user.py
|-- place.py
|-- validation.py
|-- config.py
|-- bot.py
|-- scraper.py
|-- booker.py
|-- main.py

and I could build the documentation with: pdoc -d google *.py -o docs/ Then I decided to put some files into some folders and the last project structure is:

buchino/
|-- db/
|    |-- __init__.py
|    |-- init.py
|    |-- data.py
|    |-- controller.py
|    |-- user.py
|    |-- place.py
|-- utils/
|    |-- __init__.py
|    |-- validation.py
|    |-- config.py
|-- __init__.py
|-- bot.py
|-- scraper.py
|-- booker.py
|-- main.py

But now when I try to build the documentation with the same command I get that error:

Traceback (most recent call last):
  File "/usr/lib/python3.9/pathlib.py", line 736, in __str__
    return self._str
AttributeError: _str

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/bin/pdoc", line 33, in <module>
    sys.exit(load_entry_point('pdoc==7.1.1', 'console_scripts', 'pdoc')())
  File "/usr/lib/python3.9/site-packages/pdoc/__main__.py", line 159, in cli
    pdoc.pdoc(
  File "/usr/lib/python3.9/site-packages/pdoc/__init__.py", line 384, in pdoc
    all_modules = extract.walk_specs(modules)
  File "/usr/lib/python3.9/site-packages/pdoc/extract.py", line 49, in walk_specs
    modname = parse_spec(spec)
  File "/usr/lib/python3.9/site-packages/pdoc/extract.py", line 119, in parse_spec
    return parse_spec(spec.parent) + f".{spec.stem}"
  File "/usr/lib/python3.9/site-packages/pdoc/extract.py", line 119, in parse_spec
    return parse_spec(spec.parent) + f".{spec.stem}"
  File "/usr/lib/python3.9/site-packages/pdoc/extract.py", line 119, in parse_spec
    return parse_spec(spec.parent) + f".{spec.stem}"
  [Previous line repeated 984 more times]
  File "/usr/lib/python3.9/site-packages/pdoc/extract.py", line 118, in parse_spec
    if (spec.parent / "__init__.py").exists():
  File "/usr/lib/python3.9/pathlib.py", line 1414, in exists
    self.stat()
  File "/usr/lib/python3.9/pathlib.py", line 1222, in stat
    return self._accessor.stat(self)
  File "/usr/lib/python3.9/pathlib.py", line 743, in __fspath__
    return str(self)
  File "/usr/lib/python3.9/pathlib.py", line 738, in __str__
    self._str = self._format_parsed_parts(self._drv, self._root,
  File "/usr/lib/python3.9/pathlib.py", line 720, in _format_parsed_parts
    return cls._flavour.join(parts)
RecursionError: maximum recursion depth exceeded while calling a Python object

More info about the project structure: https://github.com/fabifont/buchino/pull/16#issuecomment-888171961

Steps to reproduce the behavior:

https://github.com/fabifont/buchino.git
cd buchino
git checkout dev
pdoc -d google *.py -o docs/

System Information

pdoc: 7.1.1
Python: 3.9.6
Platform: Linux-5.13.5-arch1-1-x86_64-with-glibc2.33

mhils commented 3 years ago

Thanks for the excellent repro steps! I'd generally recommend to use pdoc buchino over pdoc *.py, but we certainly should not crash.

I have just pushed a fix to main. Could you please update and verify that it works for you?

pip install -U https://github.com/mitmproxy/pdoc/archive/main.zip

fabifont commented 3 years ago

Thanks for the excellent repro steps! I'd generally recommend to use pdoc buchino over pdoc *.py, but we certainly should not crash.

I have just pushed a fix to main. Could you please update and verify that it works for you?
pip install -U https://github.com/mitmproxy/pdoc/archive/main.zip

Thanks for the reply. With your latest update now pdoc partially builds the documentation. I get that error:

/home/fabio/.local/lib/python3.9/site-packages/pdoc/doc.py:423: RuntimeWarning: Couldn't import buchino.booker:
Traceback (most recent call last):
  File "/home/fabio/.local/lib/python3.9/site-packages/pdoc/extract.py", line 173, in load_module
    return importlib.import_module(module)
  File "/usr/lib/python3.9/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1030, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1007, in _find_and_load
  File "<frozen importlib._bootstrap>", line 986, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 680, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 850, in exec_module
  File "<frozen importlib._bootstrap>", line 228, in _call_with_frames_removed
  File "/home/fabio/Projects/./buchino/booker.py", line 7, in <module>
    import scraper as scraper
ModuleNotFoundError: No module named 'scraper'

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/fabio/.local/lib/python3.9/site-packages/pdoc/doc.py", line 421, in submodules
    module = extract.load_module(mod.name)
  File "/usr/lib/python3.9/contextlib.py", line 79, in inner
    return func(*args, **kwds)
  File "/home/fabio/.local/lib/python3.9/site-packages/pdoc/extract.py", line 175, in load_module
    raise RuntimeError(f"Error importing {module}") from e
RuntimeError: Error importing buchino.booker

  warnings.warn(
/home/fabio/.local/lib/python3.9/site-packages/pdoc/doc.py:423: RuntimeWarning: Couldn't import buchino.bot:
Traceback (most recent call last):
  File "/home/fabio/.local/lib/python3.9/site-packages/pdoc/extract.py", line 173, in load_module
    return importlib.import_module(module)
  File "/usr/lib/python3.9/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1030, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1007, in _find_and_load
  File "<frozen importlib._bootstrap>", line 986, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 680, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 850, in exec_module
  File "<frozen importlib._bootstrap>", line 228, in _call_with_frames_removed
  File "/home/fabio/Projects/./buchino/bot.py", line 11, in <module>
    from db import controller as controller
ModuleNotFoundError: No module named 'db'

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/fabio/.local/lib/python3.9/site-packages/pdoc/doc.py", line 421, in submodules
    module = extract.load_module(mod.name)
  File "/usr/lib/python3.9/contextlib.py", line 79, in inner
    return func(*args, **kwds)
  File "/home/fabio/.local/lib/python3.9/site-packages/pdoc/extract.py", line 175, in load_module
    raise RuntimeError(f"Error importing {module}") from e
RuntimeError: Error importing buchino.bot

  warnings.warn(
/home/fabio/.local/lib/python3.9/site-packages/pdoc/doc.py:423: RuntimeWarning: Couldn't import buchino.main:
Traceback (most recent call last):
  File "/home/fabio/.local/lib/python3.9/site-packages/pdoc/extract.py", line 173, in load_module
    return importlib.import_module(module)
  File "/usr/lib/python3.9/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1030, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1007, in _find_and_load
  File "<frozen importlib._bootstrap>", line 986, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 680, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 850, in exec_module
  File "<frozen importlib._bootstrap>", line 228, in _call_with_frames_removed
  File "/home/fabio/Projects/./buchino/main.py", line 5, in <module>
    import bot as bot
ModuleNotFoundError: No module named 'bot'

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/fabio/.local/lib/python3.9/site-packages/pdoc/doc.py", line 421, in submodules
    module = extract.load_module(mod.name)
  File "/usr/lib/python3.9/contextlib.py", line 79, in inner
    return func(*args, **kwds)
  File "/home/fabio/.local/lib/python3.9/site-packages/pdoc/extract.py", line 175, in load_module
    raise RuntimeError(f"Error importing {module}") from e
RuntimeError: Error importing buchino.main

  warnings.warn(
/home/fabio/.local/lib/python3.9/site-packages/pdoc/doc.py:423: RuntimeWarning: Couldn't import buchino.scraper:
Traceback (most recent call last):
  File "/home/fabio/.local/lib/python3.9/site-packages/pdoc/extract.py", line 173, in load_module
    return importlib.import_module(module)
  File "/usr/lib/python3.9/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1030, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1007, in _find_and_load
  File "<frozen importlib._bootstrap>", line 986, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 680, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 850, in exec_module
  File "<frozen importlib._bootstrap>", line 228, in _call_with_frames_removed
  File "/home/fabio/Projects/./buchino/scraper.py", line 12, in <module>
    from db import controller as controller
ModuleNotFoundError: No module named 'db'

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/fabio/.local/lib/python3.9/site-packages/pdoc/doc.py", line 421, in submodules
    module = extract.load_module(mod.name)
  File "/usr/lib/python3.9/contextlib.py", line 79, in inner
    return func(*args, **kwds)
  File "/home/fabio/.local/lib/python3.9/site-packages/pdoc/extract.py", line 175, in load_module
    raise RuntimeError(f"Error importing {module}") from e
RuntimeError: Error importing buchino.scraper

  warnings.warn(
/home/fabio/.local/lib/python3.9/site-packages/pdoc/__init__.py:405: RuntimeWarning: Error importing buchino.booker:
Traceback (most recent call last):
  File "/home/fabio/.local/lib/python3.9/site-packages/pdoc/extract.py", line 173, in load_module
    return importlib.import_module(module)
  File "/usr/lib/python3.9/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1030, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1007, in _find_and_load
  File "<frozen importlib._bootstrap>", line 986, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 680, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 850, in exec_module
  File "<frozen importlib._bootstrap>", line 228, in _call_with_frames_removed
  File "/home/fabio/Projects/./buchino/booker.py", line 7, in <module>
    import scraper as scraper
ModuleNotFoundError: No module named 'scraper'

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/fabio/.local/lib/python3.9/site-packages/pdoc/__init__.py", line 403, in pdoc
    m = extract.load_module(module)
  File "/usr/lib/python3.9/contextlib.py", line 79, in inner
    return func(*args, **kwds)
  File "/home/fabio/.local/lib/python3.9/site-packages/pdoc/extract.py", line 175, in load_module
    raise RuntimeError(f"Error importing {module}") from e
RuntimeError: Error importing buchino.booker

  warnings.warn(
/home/fabio/.local/lib/python3.9/site-packages/pdoc/__init__.py:405: RuntimeWarning: Error importing buchino.bot:
Traceback (most recent call last):
  File "/home/fabio/.local/lib/python3.9/site-packages/pdoc/extract.py", line 173, in load_module
    return importlib.import_module(module)
  File "/usr/lib/python3.9/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1030, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1007, in _find_and_load
  File "<frozen importlib._bootstrap>", line 986, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 680, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 850, in exec_module
  File "<frozen importlib._bootstrap>", line 228, in _call_with_frames_removed
  File "/home/fabio/Projects/./buchino/bot.py", line 11, in <module>
    from db import controller as controller
ModuleNotFoundError: No module named 'db'

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/fabio/.local/lib/python3.9/site-packages/pdoc/__init__.py", line 403, in pdoc
    m = extract.load_module(module)
  File "/usr/lib/python3.9/contextlib.py", line 79, in inner
    return func(*args, **kwds)
  File "/home/fabio/.local/lib/python3.9/site-packages/pdoc/extract.py", line 175, in load_module
    raise RuntimeError(f"Error importing {module}") from e
RuntimeError: Error importing buchino.bot

  warnings.warn(
/home/fabio/.local/lib/python3.9/site-packages/pdoc/__init__.py:405: RuntimeWarning: Error importing buchino.main:
Traceback (most recent call last):
  File "/home/fabio/.local/lib/python3.9/site-packages/pdoc/extract.py", line 173, in load_module
    return importlib.import_module(module)
  File "/usr/lib/python3.9/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1030, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1007, in _find_and_load
  File "<frozen importlib._bootstrap>", line 986, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 680, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 850, in exec_module
  File "<frozen importlib._bootstrap>", line 228, in _call_with_frames_removed
  File "/home/fabio/Projects/./buchino/main.py", line 5, in <module>
    import bot as bot
ModuleNotFoundError: No module named 'bot'

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/fabio/.local/lib/python3.9/site-packages/pdoc/__init__.py", line 403, in pdoc
    m = extract.load_module(module)
  File "/usr/lib/python3.9/contextlib.py", line 79, in inner
    return func(*args, **kwds)
  File "/home/fabio/.local/lib/python3.9/site-packages/pdoc/extract.py", line 175, in load_module
    raise RuntimeError(f"Error importing {module}") from e
RuntimeError: Error importing buchino.main

  warnings.warn(
/home/fabio/.local/lib/python3.9/site-packages/pdoc/__init__.py:405: RuntimeWarning: Error importing buchino.scraper:
Traceback (most recent call last):
  File "/home/fabio/.local/lib/python3.9/site-packages/pdoc/extract.py", line 173, in load_module
    return importlib.import_module(module)
  File "/usr/lib/python3.9/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1030, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1007, in _find_and_load
  File "<frozen importlib._bootstrap>", line 986, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 680, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 850, in exec_module
  File "<frozen importlib._bootstrap>", line 228, in _call_with_frames_removed
  File "/home/fabio/Projects/./buchino/scraper.py", line 12, in <module>
    from db import controller as controller
ModuleNotFoundError: No module named 'db'

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/fabio/.local/lib/python3.9/site-packages/pdoc/__init__.py", line 403, in pdoc
    m = extract.load_module(module)
  File "/usr/lib/python3.9/contextlib.py", line 79, in inner
    return func(*args, **kwds)
  File "/home/fabio/.local/lib/python3.9/site-packages/pdoc/extract.py", line 175, in load_module
    raise RuntimeError(f"Error importing {module}") from e
RuntimeError: Error importing buchino.scraper

  warnings.warn(

Project structure is the same. I tried both http (pdoc -d google buchino) and html (pdoc -d google buchino -o buchino/docs/) mode but they give the same error.

mhils commented 3 years ago

I think the core problem is really is that you have a __init__.py, which signals that your project is a module. However your imports do not match that: from db import controller as controller states that you want to import the module db, but if buchino is a module that should either be buchino.db or .db. So you should kind of make up your mind what you want to do here. :)

fabifont commented 3 years ago

There is always a problem:

if I remove __init__.py it says module buchino not found

if I add buchino. in front of imports it builds better but:

I can't execute anymore my program with python3 main.py: ModuleNotFoundError: No module named 'buchino'

pdoc can't build doc for bot.py:


Traceback (most recent call last):
File "/home/fabio/.local/lib/python3.9/site-packages/pdoc/extract.py", line 173, in load_module
return importlib.import_module(module)
File "/usr/lib/python3.9/importlib/__init__.py", line 127, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
File "<frozen importlib._bootstrap>", line 1030, in _gcd_import
File "<frozen importlib._bootstrap>", line 1007, in _find_and_load
File "<frozen importlib._bootstrap>", line 986, in _find_and_load_unlocked
File "<frozen importlib._bootstrap>", line 680, in _load_unlocked
File "<frozen importlib._bootstrap_external>", line 850, in exec_module
File "<frozen importlib._bootstrap>", line 228, in _call_with_frames_removed
File "/home/fabio/Projects/./buchino/bot.py", line 51, in <module>
bot = Bot(get_value("token"))
File "/home/fabio/Projects/./buchino/utils/config.py", line 18, in get_value
return config.get("Telegram", key)
File "/usr/lib/python3.9/configparser.py", line 781, in get
d = self._unify_values(section, vars)
File "/usr/lib/python3.9/configparser.py", line 1152, in _unify_values
raise NoSectionError(section) from None
configparser.NoSectionError: No section: 'Telegram'

The above exception was the direct cause of the following exception:

Traceback (most recent call last): File "/home/fabio/.local/lib/python3.9/site-packages/pdoc/doc.py", line 421, in submodules module = extract.load_module(mod.name) File "/usr/lib/python3.9/contextlib.py", line 79, in inner return func(*args, **kwds) File "/home/fabio/.local/lib/python3.9/site-packages/pdoc/extract.py", line 175, in load_module raise RuntimeError(f"Error importing {module}") from e RuntimeError: Error importing buchino.bot

warnings.warn( /home/fabio/.local/lib/python3.9/site-packages/pdoc/doc.py:423: RuntimeWarning: Couldn't import buchino.main: Traceback (most recent call last): File "/home/fabio/.local/lib/python3.9/site-packages/pdoc/extract.py", line 173, in load_module return importlib.import_module(module) File "/usr/lib/python3.9/importlib/init.py", line 127, in import_module return _bootstrap._gcd_import(name[level:], package, level) File "", line 1030, in _gcd_import File "", line 1007, in _find_and_load File "", line 986, in _find_and_load_unlocked File "", line 680, in _load_unlocked File "", line 850, in exec_module File "", line 228, in _call_with_frames_removed File "/home/fabio/Projects/./buchino/main.py", line 5, in import buchino.bot as bot File "/home/fabio/Projects/./buchino/bot.py", line 51, in bot = Bot(get_value("token")) File "/home/fabio/Projects/./buchino/utils/config.py", line 18, in get_value return config.get("Telegram", key) File "/usr/lib/python3.9/configparser.py", line 781, in get d = self._unify_values(section, vars) File "/usr/lib/python3.9/configparser.py", line 1152, in _unify_values raise NoSectionError(section) from None configparser.NoSectionError: No section: 'Telegram'

The above exception was the direct cause of the following exception:

Traceback (most recent call last): File "/home/fabio/.local/lib/python3.9/site-packages/pdoc/doc.py", line 421, in submodules module = extract.load_module(mod.name) File "/usr/lib/python3.9/contextlib.py", line 79, in inner return func(*args, **kwds) File "/home/fabio/.local/lib/python3.9/site-packages/pdoc/extract.py", line 175, in load_module raise RuntimeError(f"Error importing {module}") from e RuntimeError: Error importing buchino.main

warnings.warn(

mitmproxy / pdoc