Pinning sphinx version sometimes fails

[zaneselvans]

Details

I am attempting to set up RTD to build documentation for a project using a bunch of the pydata tools. I ran into resource constraints early on, and the RTD docs recommended using the system versions of numpy, scipy, pandas, etc. rather than putting them all in the requirements.txt for the doc environment.

Thus, in my .readthedocs.yml I have:

python:
  version: 3.7
  install:
    - requirements: docs/requirements.txt
  system_packages: true

and in docs/requirements.txt I have:

dbfread
sqlalchemy
sphinx==2.1.1
# from read-the-docs system python:
# scipy
# numpy
# matplotlib
# pandas

However, my pinned sphinx version only gets used sometimes. Other times the system sphinx v1.8.5 ends up being used (and not finding my index.rst... causing an error). If I change something trivial and re-commit, triggering a new build, sometimes it works. Maybe this is another manifestation of #5350, but the fact that it only happens sometimes seems really strange.

Also, in the failed build below, the log indicates that the correct version of sphinx is being found in the build environment, but then it's not being used:

Requirement already satisfied: sphinx==2.1.1 in /home/docs/checkouts/readthedocs.org/user_builds/catalyst-cooperative-pudl/envs/doc-setup/lib/python3.7/site-packages (from -r docs/requirements.txt (line 3)) (2.1.1)

• Read the Docs project URL: https://catalyst-cooperative-pudl.readthedocs.io/
• Build URL (if applicable):
  • A build that used sphinx 1.8.5: https://readthedocs.org/projects/catalyst-cooperative-pudl/builds/9228210/
  • A build that used sphinx 2.1.1: https://readthedocs.org/projects/catalyst-cooperative-pudl/builds/9226025/
• Read the Docs username (if applicable): zaneselvans

Expected Result

Expected that the pinned version of sphinx in my docs/requirements.txt file would set the version of sphinx used to build my documentation.

Actual Result

The pre-installed system sphinx version 1.8.5 takes precedence... but only sometimes for reasons that I don't understand.

[stsewd]

This looks related to https://github.com/rtfd/readthedocs.org/pull/5107.

The builds have the same characteristics, the successful build don't use the cache https://readthedocs.org/projects/catalyst-cooperative-pudl/builds/9226025/

and the failed build use the cache https://readthedocs.org/projects/catalyst-cooperative-pudl/builds/9228210/

Till we have a solution, you can wipe the version before each build https://docs.readthedocs.io/en/stable/guides/wipe-environment.html

[stsewd]

I can confirm this is because of system_packages: true, I think we can fix this the same way the other ones.

[stsewd]

Using the method used to fix the similar problem doesn't work :/ this needs more research

[mik-laj]

I have also been affected by this problem. https://readthedocs.org/projects/airflow/builds/

[matfax]

I can confirm that this happens with setuptools as well.

Using /home/docs/checkouts/readthedocs.org/user_builds/project/envs/latest/lib/python3.7/site-packages
Searching for Sphinx==1.8.5
Best match: Sphinx 1.8.5
Adding Sphinx 1.8.5 to easy-install.pth file
Installing sphinx-apidoc script to /home/docs/checkouts/readthedocs.org/user_builds/project/envs/latest/bin
Installing sphinx-autogen script to /home/docs/checkouts/readthedocs.org/user_builds/project/envs/latest/bin
Installing sphinx-build script to /home/docs/checkouts/readthedocs.org/user_builds/project/envs/latest/bin
Installing sphinx-quickstart script to /home/docs/checkouts/readthedocs.org/user_builds/project/envs/latest/bin

However, when I use both methods, it works fine:

.readthedocs.yml

python:
  version: 3.7
  install:
    - requirements: docs/requirements.txt
    - method: setuptools
      path: docs

requirements.txt

sphinx==2.2.0

pip install log

Found existing installation: Sphinx 1.8.5
    Uninstalling Sphinx-1.8.5:
      Successfully uninstalled Sphinx-1.8.5
Successfully installed sphinx-2.2.0 sphinxcontrib-applehelp-1.0.1 sphinxcontrib-devhelp-1.0.1 sphinxcontrib-htmlhelp-1.0.2 sphinxcontrib-jsmath-1.0.1 sphinxcontrib-qthelp-1.0.2 sphinxcontrib-serializinghtml-1.1.3

setup.py install

Installing collected packages: sphinxcontrib-devhelp, sphinxcontrib-applehelp, sphinxcontrib-jsmath, sphinxcontrib-qthelp, sphinxcontrib-serializinghtml, sphinxcontrib-htmlhelp, sphinx
  Found existing installation: Sphinx 1.8.5
    Uninstalling Sphinx-1.8.5:
      Successfully uninstalled Sphinx-1.8.5
Successfully installed sphinx-2.2.0 sphinxcontrib-applehelp-1.0.1 sphinxcontrib-devhelp-1.0.1 sphinxcontrib-htmlhelp-1.0.2 sphinxcontrib-jsmath-1.0.1 sphinxcontrib-qthelp-1.0.2 sphinxcontrib-serializinghtml-1.1.3

[jkochNU]

I can confirm that this is still an active issue. I am constantly running into this with https://scqubits.readthedocs.io Employing manual wipes appears to help, but I would not call it an ideal solution. Are there any updates or more sustainable workarounds?

[stsewd]

This maybe has been fixed by https://github.com/readthedocs/readthedocs.org/pull/6598. Let us know if you don't see this problem anymore.

[no-response[bot]]

This issue has been automatically closed because there has been no response to our request for more information from the original author. With only the information that is currently in the issue, we don't have enough information to take action. Please reach out if you have or find the answers we need so that we can investigate further. Thanks!

[kiuru]

@stsewd I'm still getting a old version of sphinx in the rtd. I have tried wipe and build my latest version multiple times, but I always get sphinx-1.8.5. I haven't define a specific sphinx version in the requirements.txt file, so I should get a latest version.

[stsewd]

I haven't define a specific sphinx version in the requirements.txt file, so I should get a latest version.

If you don't specify a version, you get the default (1.8.5).

[kiuru]

I haven't define a specific sphinx version in the requirements.txt file, so I should get a latest version.

If you don't specify a version, you get the default (1.8.5).

Yes that was the problem. Now my sphinx's version has changed in the rtd! Thanks!