Issues when conf.py invokes git

[ntolia]

I have been trying to use sphinx-multiversion but I am running into what seems like a blocking issue because my conf.py calls git internally. Basically, similar to https://protips.readthedocs.io/git-tag-version.html, we use git describe to get the version of the docs we should be building.

However, given what sphinx-multiversion does to copy a tree into /tmp, that fails as the tree copy isn't a real git repo and the conf.py git commands fail.

Any suggestions on how to address this?

[Holzhaus]

I don't think it's possible at the moment. Maybe it's feasible to make sphinx-multiversion set some environment variables with the actual git repository path and branch name when evaluating the config, and in your conf.py you check if these are set and use them instead. Would this solve your issue?

[ntolia]

@Holzhaus Any reason why we couldn't just do a full tree copy in the copy_tree function and use git checkout to switch between branches?

[Holzhaus]

I think for big repos this would be slow and very memory intensive (all repositories are checked out at once). It also could lead to all kinds of issues when the working tree is not clean. Originally I tried to use a local git clone but that also had problems with remote branches.

[ntolia]

@Holzhaus Is it possible to have this as an option then? Would be happy to test and contribute too.

[ntolia]

Using the below change gets me past the issue highlighted here. It also does a shallow clone to be efficient. However, I have now run into another problem. We use a plugin (sphinx_substitution_extensions) that recently had a breaking change. So, I can either get old versions to work or new versions. Will have to figure out what to do here.

diff --git a/sphinx_multiversion/git.py b/sphinx_multiversion/git.py
index 8a02f24..9f34fbb 100644
--- a/sphinx_multiversion/git.py
+++ b/sphinx_multiversion/git.py
@@ -146,3 +146,23 @@ def copy_tree(src, dst, reference, sourcepath="."):
         fp.seek(0)
         with tarfile.TarFile(fileobj=fp) as tarfp:
             tarfp.extractall(dst)
+
+
+def shallow_clone(src, dst, reference, sourcepath="."):
+    with tempfile.SpooledTemporaryFile() as fp:
+        cmd = (
+            "git",
+            "clone",
+            "--quiet",
+            "--depth",
+            "1",
+            "-c",
+            "advice.detachedHead=false",
+            "-b",
+            reference.name,
+            "--",
+            src,
+            dst
+        )
+        subprocess.check_call(cmd, stdout=fp)
+        fp.seek(0)
diff --git a/sphinx_multiversion/main.py b/sphinx_multiversion/main.py
index fb84714..a53a252 100644
--- a/sphinx_multiversion/main.py
+++ b/sphinx_multiversion/main.py
@@ -129,7 +129,7 @@ def main(argv=None):
             # Clone Git repo
             repopath = os.path.join(tmp, gitref.commit)
             try:
-                git.copy_tree(gitroot.as_uri(), repopath, gitref)
+                git.shallow_clone(gitroot.as_uri(), repopath, gitref)
             except (OSError, subprocess.CalledProcessError):
                 logger.error(
                     "Failed to copy git tree for %s to %s",

[dolfinus]

Hello.

I've just faced with the same issue while building docs within docker container.

There is a conf.py script which fetches list of tags and passes it to Read The Docs template. If I run sphinx-multiversion on locak machine, ir works perfectly, but in case of docker container it fail due to lack of .git folder in tmp dir.

My suggestion is to provide additional script arguments for managing the way of making repo copies:

1. If argument is not provided, behavior remains unchanged - script uses git archive. Large repositories are happy because .git dir is not copied.
2. If arguments is provided, git clone is used. If someone is facing just the same issue with accessing .git folder, he/she will just pass this arguments and conf.py will start working as expected.

[Holzhaus]

I'll try to come up with a PoC PR that will use git clone. By default it will use shallow clone, but that will be configurable.

[BurningEnlightenment]

I think this should be solvable with git worktree.

[Holzhaus]

I think this should be solvable with git worktree.

Hmm, git worktree modifies the original repo, and I think it's not possible to have the same branch checked out twice, so if the user already uses git worktree this will lead to errors. I think a shallow git clone from a local directory will be the more reliable solution.

[BurningEnlightenment]

Hmm, git worktree modifies the original repo, and I think it's not possible to have the same branch checked out twice, so if the user already uses git worktree this will lead to errors.

You can have multiple worktrees with the same commit / ref, but you do need to specify --force. The only issue would be that the user would have to do the same while sphinx-multiversion is working.

[BurningEnlightenment]

git worktree modifies the original repo

I mainly had CI workflows in mind where this usually isn't an issue. On a second thought it might be possible to combine the git worktree -f strategy with an opt-out git clone --mirror, i.e. add a worktree to the bare mirror repository. This way we wouldn't cause any overhead for CI scenarios while keeping dev repositories pristine. I might give that a shot.