mitmproxy / pdoc

API Documentation for Python Projects
https://pdoc.dev
The Unlicense
1.86k stars 189 forks source link

Modify import behavior to support documentation for micropython #697

Open HackXIt opened 1 month ago

HackXIt commented 1 month ago

Problem Description

I am using micropython in my project and it is fairly easy and straightforward to add docstrings.

However, it is not possible to use pdoc in all cases, as sometimes the source code requires imports that pdoc cannot resolve, since they might be C modules compiled into the micropython binary.

Proposal

To circumvent the stated problem, it would be nice if one could modify the behavior of pdoc to ignore certain imports and unknown modules.

Possibly in a similar way, which linters use via comments:

import unknown_module # pdoc: ignore

Alternatives

If the behavior can't be circumvented, it would be nice if one could "feed" pdoc with the relevant docstrings extracted by micropython. This could simply be in the form of some JSON, YAML or similar file format. Such a file could easily be generated by micropython, in a desired format.

The information read from the file could then be parsed into pdoc to generate the relevant documentation.

Additional context

I was not able to use pdoc on my micropython project based on LVGL, due to the following errors:

Console output ```shell $ pdoc src pdoc server ready at http://localhost:8080 Warn: Couldn't import src.design_parser: Traceback (most recent call last): File "/home/rini-debian/git-stash/lvgl-ui-detector/lvgl_ui_generator_v2/.venv/lib/python3.11/site-packages/pdoc/extract.py", line 218, in load_module return importlib.import_module(module) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3.11/importlib/__init__.py", line 126, in import_module return _bootstrap._gcd_import(name[level:], package, level) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "", line 1206, in _gcd_import File "", line 1178, in _find_and_load File "", line 1149, in _find_and_load_unlocked File "", line 690, in _load_unlocked File "", line 940, in exec_module File "", line 241, in _call_with_frames_removed File "/home/rini-debian/git-stash/lvgl-ui-detector/lvgl_ui_generator_v2/src/design_parser.py", line 1, in from display_driver_utils import driver ModuleNotFoundError: No module named 'display_driver_utils' The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/home/rini-debian/git-stash/lvgl-ui-detector/lvgl_ui_generator_v2/.venv/lib/python3.11/site-packages/pdoc/doc.py", line 480, in submodules module = Module.from_name(mod.name) ^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/rini-debian/git-stash/lvgl-ui-detector/lvgl_ui_generator_v2/.venv/lib/python3.11/site-packages/pdoc/doc.py", line 404, in from_name return cls(extract.load_module(name)) ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3.11/contextlib.py", line 81, in inner return func(*args, **kwds) ^^^^^^^^^^^^^^^^^^^ File "/home/rini-debian/git-stash/lvgl-ui-detector/lvgl_ui_generator_v2/.venv/lib/python3.11/site-packages/pdoc/extract.py", line 220, in load_module raise RuntimeError(f"Error importing {module}") from e RuntimeError: Error importing src.design_parser (/home/rini-debian/git-stash/lvgl-ui-detector/lvgl_ui_generator_v2/.venv/lib/python3.11/site-packages/pdoc/doc.py:482) Warn: Couldn't import src.main: Traceback (most recent call last): File "/home/rini-debian/git-stash/lvgl-ui-detector/lvgl_ui_generator_v2/.venv/lib/python3.11/site-packages/pdoc/extract.py", line 218, in load_module return importlib.import_module(module) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3.11/importlib/__init__.py", line 126, in import_module return _bootstrap._gcd_import(name[level:], package, level) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "", line 1206, in _gcd_import File "", line 1178, in _find_and_load File "", line 1149, in _find_and_load_unlocked File "", line 690, in _load_unlocked File "", line 940, in exec_module File "", line 241, in _call_with_frames_removed File "/home/rini-debian/git-stash/lvgl-ui-detector/lvgl_ui_generator_v2/src/main.py", line 1, in import cli ModuleNotFoundError: No module named 'cli' The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/home/rini-debian/git-stash/lvgl-ui-detector/lvgl_ui_generator_v2/.venv/lib/python3.11/site-packages/pdoc/doc.py", line 480, in submodules module = Module.from_name(mod.name) ^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/rini-debian/git-stash/lvgl-ui-detector/lvgl_ui_generator_v2/.venv/lib/python3.11/site-packages/pdoc/doc.py", line 404, in from_name return cls(extract.load_module(name)) ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3.11/contextlib.py", line 81, in inner return func(*args, **kwds) ^^^^^^^^^^^^^^^^^^^ File "/home/rini-debian/git-stash/lvgl-ui-detector/lvgl_ui_generator_v2/.venv/lib/python3.11/site-packages/pdoc/extract.py", line 220, in load_module raise RuntimeError(f"Error importing {module}") from e RuntimeError: Error importing src.main (/home/rini-debian/git-stash/lvgl-ui-detector/lvgl_ui_generator_v2/.venv/lib/python3.11/site-packages/pdoc/doc.py:482) Warn: Couldn't import src.random_ui: Traceback (most recent call last): File "/home/rini-debian/git-stash/lvgl-ui-detector/lvgl_ui_generator_v2/.venv/lib/python3.11/site-packages/pdoc/extract.py", line 218, in load_module return importlib.import_module(module) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3.11/importlib/__init__.py", line 126, in import_module return _bootstrap._gcd_import(name[level:], package, level) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "", line 1206, in _gcd_import File "", line 1178, in _find_and_load File "", line 1149, in _find_and_load_unlocked File "", line 690, in _load_unlocked File "", line 940, in exec_module File "", line 241, in _call_with_frames_removed File "/home/rini-debian/git-stash/lvgl-ui-detector/lvgl_ui_generator_v2/src/random_ui.py", line 1, in from display_driver_utils import driver ModuleNotFoundError: No module named 'display_driver_utils' The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/home/rini-debian/git-stash/lvgl-ui-detector/lvgl_ui_generator_v2/.venv/lib/python3.11/site-packages/pdoc/doc.py", line 480, in submodules module = Module.from_name(mod.name) ^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/rini-debian/git-stash/lvgl-ui-detector/lvgl_ui_generator_v2/.venv/lib/python3.11/site-packages/pdoc/doc.py", line 404, in from_name return cls(extract.load_module(name)) ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3.11/contextlib.py", line 81, in inner return func(*args, **kwds) ^^^^^^^^^^^^^^^^^^^ File "/home/rini-debian/git-stash/lvgl-ui-detector/lvgl_ui_generator_v2/.venv/lib/python3.11/site-packages/pdoc/extract.py", line 220, in load_module raise RuntimeError(f"Error importing {module}") from e RuntimeError: Error importing src.random_ui (/home/rini-debian/git-stash/lvgl-ui-detector/lvgl_ui_generator_v2/.venv/lib/python3.11/site-packages/pdoc/doc.py:482) Warn: Couldn't import src.screenshot: Traceback (most recent call last): File "/home/rini-debian/git-stash/lvgl-ui-detector/lvgl_ui_generator_v2/.venv/lib/python3.11/site-packages/pdoc/extract.py", line 218, in load_module return importlib.import_module(module) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3.11/importlib/__init__.py", line 126, in import_module return _bootstrap._gcd_import(name[level:], package, level) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "", line 1206, in _gcd_import File "", line 1178, in _find_and_load File "", line 1149, in _find_and_load_unlocked File "", line 690, in _load_unlocked File "", line 940, in exec_module File "", line 241, in _call_with_frames_removed File "/home/rini-debian/git-stash/lvgl-ui-detector/lvgl_ui_generator_v2/src/screenshot.py", line 1, in import lvgl as lv ModuleNotFoundError: No module named 'lvgl' The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/home/rini-debian/git-stash/lvgl-ui-detector/lvgl_ui_generator_v2/.venv/lib/python3.11/site-packages/pdoc/doc.py", line 480, in submodules module = Module.from_name(mod.name) ^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/rini-debian/git-stash/lvgl-ui-detector/lvgl_ui_generator_v2/.venv/lib/python3.11/site-packages/pdoc/doc.py", line 404, in from_name return cls(extract.load_module(name)) ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3.11/contextlib.py", line 81, in inner return func(*args, **kwds) ^^^^^^^^^^^^^^^^^^^ File "/home/rini-debian/git-stash/lvgl-ui-detector/lvgl_ui_generator_v2/.venv/lib/python3.11/site-packages/pdoc/extract.py", line 220, in load_module raise RuntimeError(f"Error importing {module}") from e RuntimeError: Error importing src.screenshot (/home/rini-debian/git-stash/lvgl-ui-detector/lvgl_ui_generator_v2/.venv/lib/python3.11/site-packages/pdoc/doc.py:482) Warn: Couldn't import src.screenshot_v2: Traceback (most recent call last): File "/home/rini-debian/git-stash/lvgl-ui-detector/lvgl_ui_generator_v2/.venv/lib/python3.11/site-packages/pdoc/extract.py", line 218, in load_module return importlib.import_module(module) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3.11/importlib/__init__.py", line 126, in import_module return _bootstrap._gcd_import(name[level:], package, level) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "", line 1206, in _gcd_import File "", line 1178, in _find_and_load File "", line 1149, in _find_and_load_unlocked File "", line 690, in _load_unlocked File "", line 940, in exec_module File "", line 241, in _call_with_frames_removed File "/home/rini-debian/git-stash/lvgl-ui-detector/lvgl_ui_generator_v2/src/screenshot_v2.py", line 1, in import jpeg ModuleNotFoundError: No module named 'jpeg' The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/home/rini-debian/git-stash/lvgl-ui-detector/lvgl_ui_generator_v2/.venv/lib/python3.11/site-packages/pdoc/doc.py", line 480, in submodules module = Module.from_name(mod.name) ^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/rini-debian/git-stash/lvgl-ui-detector/lvgl_ui_generator_v2/.venv/lib/python3.11/site-packages/pdoc/doc.py", line 404, in from_name return cls(extract.load_module(name)) ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3.11/contextlib.py", line 81, in inner return func(*args, **kwds) ^^^^^^^^^^^^^^^^^^^ File "/home/rini-debian/git-stash/lvgl-ui-detector/lvgl_ui_generator_v2/.venv/lib/python3.11/site-packages/pdoc/extract.py", line 220, in load_module raise RuntimeError(f"Error importing {module}") from e RuntimeError: Error importing src.screenshot_v2 (/home/rini-debian/git-stash/lvgl-ui-detector/lvgl_ui_generator_v2/.venv/lib/python3.11/site-packages/pdoc/doc.py:482) Warn: Couldn't import src.widget: Traceback (most recent call last): File "/home/rini-debian/git-stash/lvgl-ui-detector/lvgl_ui_generator_v2/.venv/lib/python3.11/site-packages/pdoc/extract.py", line 218, in load_module return importlib.import_module(module) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3.11/importlib/__init__.py", line 126, in import_module return _bootstrap._gcd_import(name[level:], package, level) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "", line 1206, in _gcd_import File "", line 1178, in _find_and_load File "", line 1149, in _find_and_load_unlocked File "", line 690, in _load_unlocked File "", line 940, in exec_module File "", line 241, in _call_with_frames_removed File "/home/rini-debian/git-stash/lvgl-ui-detector/lvgl_ui_generator_v2/src/widget.py", line 1, in import lvgl as lv ModuleNotFoundError: No module named 'lvgl' The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/home/rini-debian/git-stash/lvgl-ui-detector/lvgl_ui_generator_v2/.venv/lib/python3.11/site-packages/pdoc/doc.py", line 480, in submodules module = Module.from_name(mod.name) ^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/rini-debian/git-stash/lvgl-ui-detector/lvgl_ui_generator_v2/.venv/lib/python3.11/site-packages/pdoc/doc.py", line 404, in from_name return cls(extract.load_module(name)) ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3.11/contextlib.py", line 81, in inner return func(*args, **kwds) ^^^^^^^^^^^^^^^^^^^ File "/home/rini-debian/git-stash/lvgl-ui-detector/lvgl_ui_generator_v2/.venv/lib/python3.11/site-packages/pdoc/extract.py", line 220, in load_module raise RuntimeError(f"Error importing {module}") from e RuntimeError: Error importing src.widget (/home/rini-debian/git-stash/lvgl-ui-detector/lvgl_ui_generator_v2/.venv/lib/python3.11/site-packages/pdoc/doc.py:482) Warn: Couldn't import src.yolo: Traceback (most recent call last): File "/home/rini-debian/git-stash/lvgl-ui-detector/lvgl_ui_generator_v2/.venv/lib/python3.11/site-packages/pdoc/extract.py", line 218, in load_module return importlib.import_module(module) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3.11/importlib/__init__.py", line 126, in import_module return _bootstrap._gcd_import(name[level:], package, level) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "", line 1206, in _gcd_import File "", line 1178, in _find_and_load File "", line 1149, in _find_and_load_unlocked File "", line 690, in _load_unlocked File "", line 940, in exec_module File "", line 241, in _call_with_frames_removed File "/home/rini-debian/git-stash/lvgl-ui-detector/lvgl_ui_generator_v2/src/yolo.py", line 1, in from ui import UI ModuleNotFoundError: No module named 'ui' The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/home/rini-debian/git-stash/lvgl-ui-detector/lvgl_ui_generator_v2/.venv/lib/python3.11/site-packages/pdoc/doc.py", line 480, in submodules module = Module.from_name(mod.name) ^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/rini-debian/git-stash/lvgl-ui-detector/lvgl_ui_generator_v2/.venv/lib/python3.11/site-packages/pdoc/doc.py", line 404, in from_name return cls(extract.load_module(name)) ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3.11/contextlib.py", line 81, in inner return func(*args, **kwds) ^^^^^^^^^^^^^^^^^^^ File "/home/rini-debian/git-stash/lvgl-ui-detector/lvgl_ui_generator_v2/.venv/lib/python3.11/site-packages/pdoc/extract.py", line 220, in load_module raise RuntimeError(f"Error importing {module}") from e RuntimeError: Error importing src.yolo (/home/rini-debian/git-stash/lvgl-ui-detector/lvgl_ui_generator_v2/.venv/lib/python3.11/site-packages/pdoc/doc.py:482) ```
HackXIt commented 1 month ago

Just for anyone else stumbling upon this, the import or module errors are actually easily dealt with via the usage of mock

import sys
if sys.implementation.name == "micropython": # Regular import when inside micropython
    import lvgl as lv
else: # Mocked import when run through pdoc
    import mock # Mock library
    from .mock.lvgl import lv # Fake module folder containing source files with necessary mocked defines
ChatGPT gave the final hints To mock the lvgl module and make your source loadable for pdoc, you can use the unittest.mock library to create a mock version of the lvgl module. Here's a step-by-step guide on how to achieve this: 1. Create a mock for lvgl: Create a file, say mock_lvgl.py, where you define a mock version of the lvgl module. ```python from unittest.mock import MagicMock # Mock the lvgl module lv = MagicMock() # You can define any specific behavior or attributes for the mock here # For example: lv.SomeClass = MagicMock() lv.some_function = MagicMock() ``` 2. Modify the import in your code: Adjust your conditional import to use this mock module when running in a regular Python environment. ```python import sys if sys.implementation.name == "micropython": import lvgl as lv import random from global_definitions import ascii_letters else: import mock_lvgl as lv # Import the mock_lvgl module instead of lvgl import random from .global_definitions import ascii_letters ``` 3. Ensure the mock is discoverable by pdoc: Make sure pdoc can find and use the mock_lvgl module. This might require adjusting your PYTHONPATH or ensuring that the mock_lvgl.py file is in the same directory as your main script or in a directory that is part of the Python path. Here's a more complete example to illustrate: **Project Structure** ``` your_project/ │ ├── your_script.py ├── mock_lvgl.py └── global_definitions.py ``` `mock_lvgl.py` ```python from unittest.mock import MagicMock # Mock the lvgl module lv = MagicMock() # Define mock attributes and methods lv.SomeClass = MagicMock() lv.some_function = MagicMock() # Optionally, you can add some documentation to the mocks lv.SomeClass.__doc__ = "Mocked SomeClass" lv.some_function.__doc__ = "Mocked some_function" ``` `your_script.py` ```python import sys if sys.implementation.name == "micropython": import lvgl as lv import random from global_definitions import ascii_letters else: import mock_lvgl as lv # Import the mock_lvgl module instead of lvgl import random from .global_definitions import ascii_letters # Your existing code here ``` **Running pdoc** When you run pdoc to generate the documentation, ensure that the current directory is in your Python path: ```bash PYTHONPATH=. pdoc your_script.py ``` This setup ensures that when pdoc processes your code, it will use the mock_lvgl module, which should resolve any import errors and allow pdoc to generate the documentation from the existing docstrings.