Unable to build documentation for Django project

[jaoxford]

Hi there. I am trying to build the documentation for our package. It fails with either opening the development server or by trying to build the documention.

I am doing pdoc --http : <package> --config show_source_code=False or pdoc --html --config show_source_code=False <package>/

Both result in this traceback. Does Pdoc not work with Django? I'm sure it did work last time I had tried to generate documentation for my Django project.

/home/jacob/.virtualenvs/venv/lib/python3.9/site-packages/pdoc/cli.py:532: UserWarning: Couldn't read PEP-224 variable docstrings from <Module 'module'>: could not get source code
  modules = [pdoc.Module(module, docfilter=docfilter,
Traceback (most recent call last):
  File "/home/jacob/.virtualenvs/venv/lib/python3.9/site-packages/pdoc/__init__.py", line 225, in import_module
    module = importlib.import_module(module_path)
  File "/usr/lib/python3.9/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1030, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1007, in _find_and_load
  File "<frozen importlib._bootstrap>", line 986, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 680, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 790, in exec_module
  File "<frozen importlib._bootstrap>", line 228, in _call_with_frames_removed
  File "/home/jacob/.virtualenvs/venv/lib/python3.9/site-packages/django/db/models/base.py", line 108, in __new__
    app_config = apps.get_containing_app_config(module)
  File "/home/jacob/.virtualenvs/venv/lib/python3.9/site-packages/django/apps/registry.py", line 252, in get_containing_app_config
    self.check_apps_ready()
  File "/home/jacob/.virtualenvs/venv/lib/python3.9/site-packages/django/apps/registry.py", line 134, in check_apps_ready
    settings.INSTALLED_APPS
  File "/home/jacob/.virtualenvs/venv/lib/python3.9/site-packages/django/conf/__init__.py", line 83, in __getattr__
    self._setup(name)
  File "/home/jacob/.virtualenvs/venv/lib/python3.9/site-packages/django/conf/__init__.py", line 64, in _setup
    raise ImproperlyConfigured(
django.core.exceptions.ImproperlyConfigured: Requested setting INSTALLED_APPS, but settings are not configured. You must either define the environment variable DJANGO_SETTINGS_MODULE or call settings.configure() before accessing settings.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/jacob/.virtualenvs/venv/bin/pdoc", line 33, in <module>
    sys.exit(load_entry_point('pdoc3==0.9.2', 'console_scripts', 'pdoc')())
  File "/home/jacob/.virtualenvs/venv/lib/python3.9/site-packages/pdoc/cli.py", line 532, in main
    modules = [pdoc.Module(module, docfilter=docfilter,
  File "/home/jacob/.virtualenvs/venv/lib/python3.9/site-packages/pdoc/cli.py", line 532, in <listcomp>
    modules = [pdoc.Module(module, docfilter=docfilter,
  File "/home/jacob/.virtualenvs/venv/lib/python3.9/site-packages/pdoc/__init__.py", line 708, in __init__
    m = Module(import_module(fullname),
  File "/home/jacob/.virtualenvs/venv/lib/python3.9/site-packages/pdoc/__init__.py", line 227, in import_module
    raise ImportError('Error importing {!r}: {}: {}'
ImportError: Error importing 'module': ImproperlyConfigured: Requested setting INSTALLED_APPS, but settings are not configured. You must either define the environment variable DJANGO_SETTINGS_MODULE or call settings.configure() before accessing settings.

[kernc]

It should work, but it seems to be failing while importing module "module", running into Django error:

Requested setting INSTALLED_APPS, but settings are not configured.
You must either define the environment variable DJANGO_SETTINGS_MODULE or 
call settings.configure() before accessing settings.

I'm not familiar with Django, but I guess you're trying to access something at a module top level when the Django app is not yet configured. Either move that module-global Django-touching code behind a main guard (as appropriate), or see if setting DJANGO_SETTINGS_MODULE env variable helps. :thinking:

[jaoxford]

Setting the env variable for the settings module with DJANGO_SETTINGS_MODULE=<settings_module> gives a different traceback.

I forgot to mention that last time we had run pdoc3 there were no errors and the documentation would be built successfully. (Last time I checked and it worked was about April last year).

Traceback (most recent call last):
  File "/home/jacob/.virtualenvs/venv/lib/python3.9/site-packages/pdoc/__init__.py", line 225, in import_module
    module = importlib.import_module(module_path)
  File "/usr/lib/python3.9/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1030, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1007, in _find_and_load
  File "<frozen importlib._bootstrap>", line 986, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 680, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 790, in exec_module
  File "<frozen importlib._bootstrap>", line 228, in _call_with_frames_removed
  File "/home/jacob/.virtualenvs/venv/lib/python3.9/site-packages/rest_framework_simplejwt/views.py", line 4, in <module>
    from . import serializers
  File "/home/jacob/.virtualenvs/venv/lib/python3.9/site-packages/rest_framework_simplejwt/serializers.py", line 4, in <module>
    from django.contrib.auth.models import update_last_login
  File "/home/jacob/.virtualenvs/venv/lib/python3.9/site-packages/django/contrib/auth/models.py", line 2, in <module>
    from django.contrib.auth.base_user import AbstractBaseUser, BaseUserManager
  File "/home/jacob/.virtualenvs/venv/lib/python3.9/site-packages/django/contrib/auth/base_user.py", line 48, in <module>
    class AbstractBaseUser(models.Model):
  File "/home/jacob/.virtualenvs/venv/lib/python3.9/site-packages/django/db/models/base.py", line 108, in __new__
    app_config = apps.get_containing_app_config(module)
  File "/home/jacob/.virtualenvs/venv/lib/python3.9/site-packages/django/apps/registry.py", line 252, in get_containing_app_config
    self.check_apps_ready()
  File "/home/jacob/.virtualenvs/venv/lib/python3.9/site-packages/django/apps/registry.py", line 135, in check_apps_ready
    raise AppRegistryNotReady("Apps aren't loaded yet.")
django.core.exceptions.AppRegistryNotReady: Apps aren't loaded yet.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/jacob/.virtualenvs/venv/lib/python3.9/site-packages/pdoc/cli.py", line 234, in do_GET
    out = self.html()
  File "/home/jacob/.virtualenvs/venv/lib/python3.9/site-packages/pdoc/cli.py", line 263, in html
    return pdoc.html(self.import_path_from_req_url,
  File "/home/jacob/.virtualenvs/venv/lib/python3.9/site-packages/pdoc/__init__.py", line 176, in html
    mod = Module(import_module(module_name, reload=reload),
  File "/home/jacob/.virtualenvs/venv/lib/python3.9/site-packages/pdoc/__init__.py", line 708, in __init__
    m = Module(import_module(fullname),
  File "/home/jacob/.virtualenvs/venv/lib/python3.9/site-packages/pdoc/__init__.py", line 227, in import_module
    raise ImportError('Error importing {!r}: {}: {}'
ImportError: Error importing 'module': AppRegistryNotReady: Apps aren't loaded yet.

[kernc]

Last time I checked and it worked was about April last year

Can you try, then, with pip install -U pdoc3==0.8.1, released about that time. I think (hope :crossed_fingers:), though, you should find the cause is actually in the your codebase, which has likely also changed since. :grinning:

Is you django app installable (installed)? Have you tried pdoc <package> instead of pdoc <package>/?

The base issue is that Django is complaining that some parts of its structure are not set/loaded when some other of its parts are touched. Do you have a function where you "init" your app that is called when you start-up your app/server service, but is not called when pdoc simply recursively imports your modules?

[sentouki]

pdoc CLI won't work with Django, you'd have to write a script. I use this one

import os
import sys

sys.path.append(os.path.abspath('../project_dir/'))  # path to your django project

# Specify settings module
os.environ.setdefault('DJANGO_SETTINGS_MODULE', 'project_name.settings')

# Setup Django
import django
django.setup()

import pdoc

modules = ['project_name', 'app_name']
context = pdoc.Context()

modules = [pdoc.Module(mod, context=context)
           for mod in modules]
pdoc.link_inheritance(context)

def recursive_htmls(mod):
    yield mod.name, mod.html()
    for submod in mod.submodules():
        yield from recursive_htmls(submod)

for mod in modules:
    for module_name, html in recursive_htmls(mod):
        with open(f"{module_name}.html", "w", encoding="utf8") as f:
            f.write(html)

[kernc]

@sentouki So, what differs from running the command line:

PYTHONPATH=../project_dir \
    DJANGO_SETTINGS_MODULE=project_name.settings \
    pdoc --html project_name app_name

is primarily:

# Setup Django
import django; django.setup()

and that can't be initialized rather in project files or config.mako, which is evaluated?

[sentouki]

@kernc sry, I've been busy last few days. I tried to initialize django in config.mako but, correct me if I'm wrong, it seems like the templates are loaded after pdoc has imported the modules, but that'd be exactly the problem since you can't import django-related modules without initializing django first. I also tried adding

# Setup Django
import django; django.setup()

to cli.py, something like this

sys.path.append(os.path.abspath(r"path/to/my/project"))

# Specify settings module
os.environ.setdefault('DJANGO_SETTINGS_MODULE', 'appName.settings')

# Setup Django
import django
django.setup()

import pdoc

parser = argparse.ArgumentParser(
    description="Automatically generate API docs for Python modules.",
    epilog="Further documentation is available at <https://pdoc3.github.io/pdoc/doc>.",
)
....

and it worked great (even better than my own script).

This problem could be solved by adding a template which is loaded before anything else, something like setup.mako

[kernc]

since you can't import django-related modules without initializing django first. I also tried adding

# Setup Django
import django; django.setup()

So why, again, not add above lines simply into your app/__init__.py?

[sentouki]

I've tried and it works, but Django doesn't start anymore because of it. I mean, you could add those lines to __init__.py everytime you want to generate docs but that's kinda annoying xP

[craigmbooth]

I ended up solving the problem in the comment directly above by moving the relevant lines in the __init__.py file into a conditional

if os.environ.get("CI_MAKING_DOCS") is not None:
    # pdoc will not work with django unless you first call django.setup().
    # Unfortunately, calling that function in an __init__ file means that
    # django will crash if actually run, so only execute the following if
    # we are generating docs
    # https://github.com/pdoc3/pdoc/issues/314

    # Ignore a few directories for doc generation
    __pdoc__ = {}
    __pdoc__["migrations"] = False

    import django
    django.setup()

Then in my CI script that makes docs:

CI_MAKING_DOCS=1 pdoc -o ./docs my_project_name --html

[sakinaf]

I ended up solving the problem in the comment directly above by moving the relevant lines in the __init__.py file into a conditional

if os.environ.get("CI_MAKING_DOCS") is not None:
    # pdoc will not work with django unless you first call django.setup().
    # Unfortunately, calling that function in an __init__ file means that
    # django will crash if actually run, so only execute the following if
    # we are generating docs
    # https://github.com/pdoc3/pdoc/issues/314

    # Ignore a few directories for doc generation
    __pdoc__ = {}
    __pdoc__["migrations"] = False

    import django
    django.setup()

Then in my CI script that makes docs:

CI_MAKING_DOCS=1 pdoc -o ./docs my_project_name --html

Which __init__.py are you talking about here?

[datenhahn]

For me it worked to just call the pdoc.cli main programmatically.

Put this file create_pdoc.py in the same folder where your manage.py is and change the MODULE variable to your main module.

from pdoc.cli import *

OUTPUT_DIR = "../pdoc3_output"

# Setup Django
import django
django.setup()

cmdline_args = ["--html", "-o" , OUTPUT_DIR, "."]

if __name__ == "__main__":
    main(parser.parse_args(cmdline_args))

Then run

export DJANGO_SETTINGS_MODULE=yourmodule.settings
python3 create_pdoc.py

You can also hardcode the django settings module to be able to just run the program without any other configuration.

from pdoc.cli import *

OUTPUT_DIR = "../pdoc3_output"

# Programmatically provide settings module
SETTINGS_MODULE = "yourmodule.settings"
os.environ.setdefault('DJANGO_SETTINGS_MODULE', SETTINGS_MODULE)

# Setup Django
import django
django.setup()

cmdline_args = ["--html", "-o" , OUTPUT_DIR, "."]

if __name__ == "__main__":
    main(parser.parse_args(cmdline_args))

[VaasuDevanS]

I kept this in __init__.py file and it worked for me

import os

if not os.environ.get('DJANGO_SETTINGS_MODULE'):
    import django
    os.environ['DJANGO_SETTINGS_MODULE'] = 'mysite.settings'
    django.setup()

[bryant-finney]

You can also use a custom management command to invoke pdoc:

from __future__ import annotations
from pathlib import Path
from django.core.management.base import BaseCommand
from pdoc.__main__ import cli, parser

class Command(BaseCommand):
    def run_from_argv(self, argv: list[str]) -> None:
        parser.prog = " ".join((Path(argv[0]).name, argv[1]))
        cli(argv[2:])

Place the above snippet into pydoc.py in the following directory within a Django app:

management
├── __init__.py
└── commands
   ├── __init__.py
   └── pdoc.py

Now, django-admin pdoc / python manage.py pdoc can be used to invoke pdoc:

❯ django-admin pdoc --help

usage: django-admin pdoc [-o DIR] [-d {markdown,google,numpy,restructuredtext}] [-e module=url] [--favicon URL] [--footer-text TEXT] [--logo URL] [--logo-link URL] [--math] [--search] [--show-source] [-t DIR] [-h HOST] [-p PORT] [-n] [--help] [--version] [module [module ...]]

Automatically generate API docs for Python modules.

Main Arguments:
  module                Python module names. These may be importable Python module names ("pdoc.doc") or file paths ("./pdoc/doc.py"). Exclude submodules by specifying a negative !regex pattern, e.g. "foo !foo.bar". (default: [])
  -o DIR, --output-directory DIR
                        Write rendered documentation to the specified directory, don't start a webserver. (default: None)

Customize Rendering:
  -d {markdown,google,numpy,restructuredtext}, --docformat {markdown,google,numpy,restructuredtext}
                        The default docstring format. For non-Markdown formats, pdoc will first convert matching syntax elements to Markdown and then process everything as Markdown. (default: restructuredtext)
  -e module=url, --edit-url module=url
                        A mapping between module names and URL prefixes, used to display an 'Edit' button. May be passed multiple times. Example: pdoc=https://github.com/mitmproxy/pdoc/blob/main/pdoc/ (default: [])
  --favicon URL         Specify a custom favicon URL. (default: None)
  --footer-text TEXT    Custom text for the page footer, for example the project name and current version number. (default: None)
  --logo URL            Add a project logo image. (default: None)
  --logo-link URL       Optional URL the logo should point to. (default: None)
  --math, --no-math     Include MathJax from a CDN to enable math formula rendering. (default: False)
  --search, --no-search
                        Enable search functionality. (default: True)
  --show-source, --no-show-source
                        Display "View Source" buttons. (default: True)
  -t DIR, --template-directory DIR
                        A directory containing Jinja2 templates to customize output. Alternatively, put your templates in $XDG_CONFIG_HOME/pdoc and pdoc will automatically find them. (default: None)

Miscellaneous Options:
  -h HOST, --host HOST  The host on which to run the HTTP server. (default: localhost)
  -p PORT, --port PORT  The port on which to run the HTTP server. (default: None)
  -n, --no-browser      Don't open a browser after the web server has started. (default: False)
  --help                Show this help message and exit.
  --version             Show version information and exit.