Read the docs build failures

[jameshadfield]

Upstream issue: https://github.com/readthedocs/readthedocs.org/issues/11416

Progress

Repos affected were found using GitHub search for org:nextstrain python: "mambaforge-22.9".

• [x] https://github.com/nextstrain/ncov/pull/1122
• [x] https://github.com/nextstrain/auspice/pull/1798
• [x] https://github.com/nextstrain/nextclade/pull/1500
• [x] https://github.com/nextstrain/docs.nextstrain.org/pull/211

Solutions/workarounds

1. ⏳ Wait for upstream issue to be resolved
2. ⛔️ Use python instead of mambaforge
3. ⛔️ Pin to sphinx=7.2
4. ✅ Bump the mambaforge version to 23.11

---

Initial commentary

We recently had multiple failures of the Read-the-docs build in Auspice PRs - https://github.com/nextstrain/auspice/pull/1788, https://github.com/nextstrain/auspice/pull/1779. There were no salient changes to docs related code in those PRs (I believe), and in all cases our action which builds the docs succeeded (example action log). Error (full log here):

critical libmamba Pathname can't be converted from UTF-8 to current locale.
Preparing transaction: ...working... done
Verifying transaction: ...working... failed

CondaVerificationError: The package for sphinx located at /home/docs/.asdf/installs/python/mambaforge-22.9.0-3/pkgs/sphinx-7.3.7-py312h5eee18b_0
appears to be corrupted. The path 'bin/sphinx-apidoc'
specified in the package manifest cannot be found.

CondaVerificationError: The package for sphinx located at /home/docs/.asdf/installs/python/mambaforge-22.9.0-3/pkgs/sphinx-7.3.7-py312h5eee18b_0
appears to be corrupted. The path 'bin/sphinx-autogen'
specified in the package manifest cannot be found.

...

The same failure occured on the latest/stable version of Auspice's RTD

[jameshadfield]

The same failure observed in the latest Nextclade build so I'll transfer this issue...

[jameshadfield]

Nextstrain.org build just completed successfully however this uses python to build not mamba.

[tsibley]

My first question is: What's the current locale for the RTD build system?

Driven by this error:

critical libmamba Pathname can't be converted from UTF-8 to current locale.

which I think is likely the root cause of all the other errors.

This feels like an RTD build system bug.

[tsibley]

https://github.com/readthedocs/readthedocs.org/issues/11416

[victorlin]

I think these are the workarounds:

1. Use python instead of mambaforge
2. Pin to sphinx=7.2

I think (1) is too big and unnecessary of a change. (2) seems sensible with an inline comment.

[joverlee521]

Latest build of docs in augur for https://github.com/nextstrain/augur/pull/1493 completed successfully today.

[victorlin]

Augur is not affected since, similar to nextstrain.org, it uses Python to build instead of mambaforge.

[victorlin]

The upstream issue isn't getting any attention. I'll apply the workaround of pinning sphinx=7.2 to all affected repos.

[victorlin]

I just realized that docs.nextstrain.org is not actually affected as a side effect of the pin to docutils<0.18, which the broken Sphinx version is not compatible with. It'd still be good to pin sphinx separately.

Also, the pin shouldn't be sphinx=7.2 but a range with an upper cap. I pulled that directly from the upstream issue as a known workaround. Will do some testing to find the earliest version of Sphinx that breaks and cap it to lower than that.

[victorlin]

Update❗️I found a better solution: bump the mambaforge version.

I did some searching on the error message and came across https://github.com/conda-forge/python-libarchive-c-feedstock/issues/35#issuecomment-1623133587. This is a known bug in mamba versions <1.4.7. The mambaforge version that we use across affected repos, 22.9, bundles mamba 1.1.0. The next available mambaforge version, 23.11, bundles mamba 1.5.5.

[victorlin]

All RTD configs using mambaforge-22.9 have been updated to use mambaforge-23.11.