Working with git submodules

[djhoese]

I'm attempting to switch to sphinx-multiversion, but I've run into an issue that I don't think there is an existing solution for. If there isn't, I'm curious if this is something you'd be willing to support and how you'd like it implemented.

I have a project where the code is in one repository and the sphinx docs source files are in another. The docs repository is included in the code repository as a git submodule. So you have something like:

myrepos/
    mypkg/
        __init__.py
    doc/  <-- git submodule
        conf.py

I'd like sphinx-multiversion to take the tags and branches it uses from the code repository (myrepos/), but because doc/ is also a git repository itself sphinx-multiversion is only using the ones defined there. Is it possible to make sphinx-multiversion look one level up for the main repository?

[djhoese]

If this:

https://github.com/Holzhaus/sphinx-multiversion/blob/76dba73d0d4bebee3906176ed3f1f7dc58baec81/sphinx_multiversion/git.py#L19-L26

Had an initial first try with the --show-superproject-working-tree flag instead of the --show-toplevel and if that returned nothing then did the above it would solve my problem. However, I could see how some people may not want this (tags and branches of the submodule and not the super project), but maybe it is less likely than my use case?

Or you could specify both flags at the same time and parse the output to see if there is one (no super project) or two lines (a super project):

$ git rev-parse --show-toplevel --show-superproject-working-tree

[taironemagalhaes]

I'm running into the same issue here. My docs depend on some audio files that are stored in a submodule to generate plots. A FileNotFoundError is being raised when I try to build the project using sphinx-multiversion.

[djhoese]

As you will find in #49 there are a lot of issues with trying to update this to work with git submodules. Some are limitations of git and the git commands used by sphinx-multiversion. By that I mean, the git commands chosen inside sphinx-multiversion work perfect for what it is trying to do without submodules, but those commands ignore or exclude submodules (ex. git archive). Some require changes to how sphinx-multiversion approaches these things altogether. I ended up giving up for a while and working on other parts of my documentation that needed updating.

I think I had some mild success using git worktree as a replacement for some of the stuff done by this project that I may not have committed to #49 as I ran out of time to get it to work completely. I don't remember being particularly happy with my solution though either. There are also third party packages/tools that try to make git archive work with submodules or other similar git commands, but they all seemed just a little wrong for this project. I'll see if I can get back to this sometime this month but pretty swamped so no promises (and this also isn't my project, just something I want to use).

[Holzhaus]

I think git worktree is not perfect either, because it modifies the original repo. I'll look into switching to a local git clone (yes, it works with local directories, too!), but the issues is that we still need access to the remotes. Cloning with --mirror keeps the remotes, but implies --bare. But I'm sure there will be some way to get this working.

I originally wrote this extension to get versioned builds for the Mixxx manual working (because all other extensions were broken), and it works fine for that use case, so this is not high-priority for me. If you want to contribute, feel free to do so. ;-)

[djhoese]

@Holzhaus Thanks for the info. My thought/idea was to use git worktree but put it in a context manager to make sure it always deleted the modified .git worktree. Plus, apparently, the worktree are automatically cleaned up over time if the directory they point to is deleted. At least that was my understanding.

[taironemagalhaes]

I also ran into another issue that is probably related to the git archive strategy. In my project I'm using the extension matplotlib.sphinxext.plot_directive to generate plots dynamically. The images generated using this extension are pointing to the build directory in the final html (e.g. /var/folders/0x/lypfmsxn567blxmckglkdk7w0000gn/T/tmpnnbp7yj1/95cac2b19a3156da5a6d46f657da237d0ad4f54d/docs), so when I deploy my docs to github pages the images are not found. I wasn't able to find any workaround to deal with this issue.

For now I guess I will deploy only the docs generated for the latest version of my project, but in the future I plan to dive further into this issue.

[Holzhaus]

@taironemagalhaes That sounds like a bug in the extension you're using. It should generate the images in the output directory, not the source directory. Or it should use relative paths and copy the generated images to the output dir.

[skabbit]

Faced a similar problem: git archive doesn't copy the content of submodules, which is required to build the docs properly.

[aserechenko]

Here PR to enable partial support for submodules in main repo: https://github.com/Holzhaus/sphinx-multiversion/pull/105

[BurningEnlightenment]

Duplicate of #17