docfx build fails with "The sphinxcontrib.applehelp extension used by this project needs at least Sphinx v5.0"

[parthea]

See build log here which shows the error reported in the title.

Releases of the sphinxcontrib-* dependencies were recently published which remove the requirement for sphinx>5: https://github.com/sphinx-doc/sphinx/issues/11567#issuecomment-1890273833

This library pins to sphinx==4.5.0 https://github.com/googleapis/sphinx-docfx-yaml/blob/2cda2ab6f5ef8f8c39cbe5584dfc1e3ccc88587e/setup.py#L26

Since the requirement for sphinx>5 was removed from the sphinxcontrib-* dependencies, we're pulling in the latest versions of sphinxcontrib-* instead of the ones that are compatible with sphinx 4.x

As a workaround, pinning to older versions of sphinxcontrib-* packages seems to allow the build pass

(py31013) partheniou@partheniou-vm-3:~/git/google-cloud-python/packages/google-cloud-securitycentermanagement$ git diff
diff --git a/packages/google-cloud-securitycentermanagement/noxfile.py b/packages/google-cloud-securitycentermanagement/noxfile.py
index 7d3551347c..6ab2bae199 100644
--- a/packages/google-cloud-securitycentermanagement/noxfile.py
+++ b/packages/google-cloud-securitycentermanagement/noxfile.py
@@ -308,6 +308,11 @@ def docfx(session):

     session.install("-e", ".")
     session.install(
+        "sphinxcontrib-applehelp==1.0.4",
+        "sphinxcontrib-devhelp==1.0.2",
+        "sphinxcontrib-htmlhelp==2.0.1",
+        "sphinxcontrib-qthelp==1.0.3",
+        "sphinxcontrib-serializinghtml==1.1.5",
         "gcp-sphinx-docfx-yaml",
         "alabaster",
         "recommonmark",

To see which versions are compatible with sphinx 4.x, see the changelogs here: https://github.com/sphinx-doc/sphinxcontrib-applehelp/blob/master/CHANGES https://github.com/sphinx-doc/sphinxcontrib-devhelp/blob/master/CHANGES https://github.com/sphinx-doc/sphinxcontrib-htmlhelp/blob/master/CHANGES https://github.com/sphinx-doc/sphinxcontrib-qthelp/blob/master/CHANGES https://github.com/sphinx-doc/sphinxcontrib-serializinghtml/blob/master/CHANGES

Output from local run without the workaround:

(py31013) partheniou@partheniou-vm-3:~/git/google-cloud-python/packages/google-cloud-securitycentermanagement$ nox -s docfx
nox > Running session docfx
nox > Creating virtual environment (virtualenv) using python3.10 in .nox/docfx
nox > python -m pip install -e .
nox > python -m pip install gcp-sphinx-docfx-yaml alabaster recommonmark
nox > sphinx-build -T -N -D extensions=sphinx.ext.autodoc,sphinx.ext.autosummary,docfx_yaml.extension,sphinx.ext.intersphinx,sphinx.ext.coverage,sphinx.ext.napoleon,sphinx.ext.todo,sphinx.ext.viewcode,recommonmark -b html -d docs/_build/doctrees/ docs/ docs/_build/html/
Running Sphinx v4.5.0

Traceback (most recent call last):
  File "/usr/local/google/home/partheniou/git/google-cloud-python/packages/google-cloud-securitycentermanagement/.nox/docfx/lib/python3.10/site-packages/sphinx/registry.py", line 438, in load_extension
    metadata = setup(app)
  File "/usr/local/google/home/partheniou/git/google-cloud-python/packages/google-cloud-securitycentermanagement/.nox/docfx/lib/python3.10/site-packages/sphinxcontrib/applehelp/__init__.py", line 230, in setup
    app.require_sphinx('5.0')
  File "/usr/local/google/home/partheniou/git/google-cloud-python/packages/google-cloud-securitycentermanagement/.nox/docfx/lib/python3.10/site-packages/sphinx/application.py", line 393, in require_sphinx
    raise VersionRequirementError(version)
sphinx.errors.VersionRequirementError: 5.0

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/google/home/partheniou/git/google-cloud-python/packages/google-cloud-securitycentermanagement/.nox/docfx/lib/python3.10/site-packages/sphinx/cmd/build.py", line 272, in build_main
    app = Sphinx(args.sourcedir, args.confdir, args.outputdir,
  File "/usr/local/google/home/partheniou/git/google-cloud-python/packages/google-cloud-securitycentermanagement/.nox/docfx/lib/python3.10/site-packages/sphinx/application.py", line 219, in __init__
    self.setup_extension(extension)
  File "/usr/local/google/home/partheniou/git/google-cloud-python/packages/google-cloud-securitycentermanagement/.nox/docfx/lib/python3.10/site-packages/sphinx/application.py", line 380, in setup_extension
    self.registry.load_extension(self, extname)
  File "/usr/local/google/home/partheniou/git/google-cloud-python/packages/google-cloud-securitycentermanagement/.nox/docfx/lib/python3.10/site-packages/sphinx/registry.py", line 441, in load_extension
    raise VersionRequirementError(
sphinx.errors.VersionRequirementError: The sphinxcontrib.applehelp extension used by this project needs at least Sphinx v5.0; it therefore cannot be built with this version.

Sphinx version error:
The sphinxcontrib.applehelp extension used by this project needs at least Sphinx v5.0; it therefore cannot be built with this version.
nox > Command sphinx-build -T -N -D extensions=sphinx.ext.autodoc,sphinx.ext.autosummary,docfx_yaml.extension,sphinx.ext.intersphinx,sphinx.ext.coverage,sphinx.ext.napoleon,sphinx.ext.todo,sphinx.ext.viewcode,recommonmark -b html -d docs/_build/doctrees/ docs/ docs/_build/html/ failed with exit code 2
nox > Session docfx failed.

[dandhlee]

Will work on this ASAP - partners are running into this issue when they use the plugin directly and not through Kokoro jobs.

[Ark-kun]

Kokoro jobs are also broken: https://btx-internal.corp.google.com/invocations/96471bdb-5c6f-4949-a963-b42c4de6f084/targets/cloud-devrel%2Fclient-libraries%2Fpython%2Fgoogleapis%2Fpython-aiplatform%2Fdocs%2Fdocs/log

The Vertex AI documentation completely stopped updated in recent months.

[dandhlee]

Oped to disable the problematic golden tests to prioritize fixing this first instead. New release has been made, verified that changes work on older client libraries.

We could optionally remove the added installs in nox, but not a priority.