Question about Python dependency resolution when using the monrepo plugin with mkdocstrings plugin

[kirannhegde]

Hello there,

We make use of a plugin called mkdocstrings to generate documentation for the source code. This plugin loads the different Python libraries mentioned in the source code during the documentation generation process. As a result of this, there are Python dependency resolution issues that we encounter when using the monrepo plugin in a root repository . If the mkdocs.yaml file in the root repository references multiple mkdocs.yaml files from different repositories, we run into dependency resolution errors as there are conflicting dependency requirements in the different repositories.

The code in each of the different repositories uses different versions of the same Python module. Hence, the issue. I am wondering that this must be an issue at a lot of organizations. How is this overcome elsewhere?

To overcome this, we were thinking that the docs folder in each of the different repositories should be processed independently and the output of the mkdocs build command should be copied to some place in the root repository and then have mkdocs somehow assemble the documentation. I am not sure if this is even possible. If someone has already done this, please respond back.

Thanks, Kiran Hegde

[pawamoy]

You might find this comment interesting. mkdocstrings does not particularly support building docs from another directory. We can improve it on our side as well.

[lennizle]

@kirannhegde I'm curious: How do you make monorepo & mkdocstrings work? (related to my Issue here). Is mkdocstrings executed in the submodule or in monorepo?

[kirannhegde]

Hello @lennizle ,

Apologies for the delayed response. Let me try to explain my problem and how i ended up solving the problem at hand.

Problem: We have multiple git repositories. Each of these GIT repositories uses their own mkdocs.yaml file. Each of these git repositories have their own build of mkdocs.yaml. The output from each of these repositories is a Docker container containing the respective documentation(The Docker container is a nginx container). In other words, each of these individual git repositories generate a nginx container which is capable of serving documentation. Now there is a main documentation git repository which has it's own mkdocs.yaml file. In addition, this main doc Git repository has an index.md file which contains references to each of these other documentation sites., What we do is to deploy all of these nginx containers and serve the documentation from the individual Git repositories and access these individual sites under the main site(i.e the nginx container from the main site). The challenge with using mkdocstrings in a mono repo is that, the Python handler in mkdocstrings attempts to load Python modules when generating source code documentation.. This is problematic for us as each of the individual git repositories use different versions of the same Python modules. I created git submodules for each of these git repositories in the main documentation git repository and referenced their mkdocs.yaml in the mkdocs.yaml of the main documentation git repository. However, that errored out due to conflicting Python dependencies.

The way i overcame this issue was by concatenating the search indices from each of the mkdocs build and building one large search index. This way, each of the individual git repositories have their own documentation build but we are still able to serve the documentation in a main documentation site. The search index is a file named search_index.json and can be found in the output of mkdocs build.

Does this answer your question?

So in our case, we no longer have a monorepo. Instead, we split our monorepo into multiple smaller repos, build documentation separately but serve the documentation on a main docs site.

Please let me know if you have further questions.

Regards, Kiran Hegde

[pawamoy]

For reference: https://github.com/mkdocstrings/mkdocstrings/issues/311

[roshambo919]

The thread referenced above solved this for me. https://github.com/mkdocstrings/mkdocstrings/issues/311#issuecomment-1104466238

tldr; adding options for mkdocstrings to add additional paths

plugins:
- mkdocstrings:
    handlers:
      python:
        paths: [src1, src2]

[tboddyspargo]

I'm coming up against some related (I think) challenges, myself. If I have two mkdocs.yml files like this:

# /mkdocs.yml
plugins:
  - monorepo
nav:
  - Reference:
    - Package A: '!include ./package-a/mkdocs.yml

# /package-a/mkdocs.yml
site_name: package-a
docs_dir: docs
plugins:
  - mkdocstrings

...my hope was that I could build the site using the root mkdocs.yml, but have the mkdocstrings plugin run in the context of package-a (i.e. with package-a as it's current working directory and prepended to search_paths in Griffe). From there, I hoped that I could add package-b, package-c, etc. and the resolution of all the import paths would be scoped to each sub-mkdocs.yml file. I could, of course use the paths option, but that actually exposes all paths for the resolution of each individual package's documentation, so any ambiguity in common top-level sub-package names would cause Griffe to "see" the wrong import package, causing unexpected edge case behavior.

I'm curious to know how realistic/unrealistic it would be for the mkdocstrings plugin search paths to be based on the "child" mkdocs.yml location rather than the root one? or if you think there's something I'm doing wrong in the usage. Thanks in advance for any input!