Building documentation fails with requests~=2.27.1 installed

[schnemensch]

Describe the bug

If I have requests 2.27.1 (or 2.27.0) installed, I am unable to build my documentation. During the build I get the following exception:

Exception occurred:
  File "/path/to/py_venvs/py3_tif/lib/python3.9/site-packages/requests/exceptions.py", line 34, in <module>
    class JSONDecodeError(InvalidJSONError, CompatJSONDecodeError):
TypeError: metaclass conflict: the metaclass of a derived class must be a (non-strict) subclass of the metaclasses of all its bases
The full traceback has been saved in /tmp/sphinx-err-w5gv6r0g.log, if you want to report the issue to the developers.
Please also report this if it was a user error, so that a better error message can be provided next time.
A bug report can be filed in the tracker at <https://github.com/sphinx-doc/sphinx/issues>. Thanks!

As explained here they have added the JSONDecodeError in with the 2.27.0 release.

I have no issues with older versions of the requests package and I will go back to an older version, but maybe your package requirements should be updated as currently a fresh install is failing for me.

I am not using the command line of sphinx, but I am using: sphinx.cmd.build.main(['-q', '-W', '--keep-going', '.', str(docs_dir)])

Full Traceback:

# Sphinx version: 4.4.0
# Python version: 3.9.1 (CPython)
# Docutils version: 0.17.1 release
# Jinja2 version: 3.0.3
# Last messages:

# Loaded extensions:
Traceback (most recent call last):
  File "/localdev/user/py_venvs/py3_tif/lib/python3.9/site-packages/sphinx/cmd/build.py", line 280, in build_main
    app = Sphinx(args.sourcedir, args.confdir, args.outputdir,
  File "/localdev/user/py_venvs/py3_tif/lib/python3.9/site-packages/sphinx/application.py", line 226, in __init__
    self.setup_extension(extension)
  File "/localdev/user/py_venvs/py3_tif/lib/python3.9/site-packages/sphinx/application.py", line 387, in setup_extension
    self.registry.load_extension(self, extname)
  File "/localdev/user/py_venvs/py3_tif/lib/python3.9/site-packages/sphinx/registry.py", line 433, in load_extension
    mod = import_module(extname)
  File "/u/ccmaster/SDR/main/linux64/python3/gcc-9.2/lib/python3.9/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1030, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1007, in _find_and_load
  File "<frozen importlib._bootstrap>", line 986, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 680, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 790, in exec_module
  File "<frozen importlib._bootstrap>", line 228, in _call_with_frames_removed
  File "/localdev/user/py_venvs/py3_tif/lib/python3.9/site-packages/sphinx/builders/linkcheck.py", line 28, in <module>
    from requests import Response
  File "/u/ccmaster/SDR/main/linux64/python3/gcc-9.2/lib/python3.9/unittest/mock.py", line 1093, in __call__
    return self._mock_call(*args, **kwargs)
  File "/u/ccmaster/SDR/main/linux64/python3/gcc-9.2/lib/python3.9/unittest/mock.py", line 1097, in _mock_call
    return self._execute_mock_call(*args, **kwargs)
  File "/u/ccmaster/SDR/main/linux64/python3/gcc-9.2/lib/python3.9/unittest/mock.py", line 1158, in _execute_mock_call
    result = effect(*args, **kwargs)
  File "/localdev/user/tif/repo/vp_tif_core/doc_gen/doc_gen.py", line 70, in import_mock
    return orig_import(name, *args, **kwargs)
  File "/localdev/user/py_venvs/py3_tif/lib/python3.9/site-packages/requests/__init__.py", line 45, in <module>
    from .exceptions import RequestsDependencyWarning
  File "/u/ccmaster/SDR/main/linux64/python3/gcc-9.2/lib/python3.9/unittest/mock.py", line 1093, in __call__
    return self._mock_call(*args, **kwargs)
  File "/u/ccmaster/SDR/main/linux64/python3/gcc-9.2/lib/python3.9/unittest/mock.py", line 1097, in _mock_call
    return self._execute_mock_call(*args, **kwargs)
  File "/u/ccmaster/SDR/main/linux64/python3/gcc-9.2/lib/python3.9/unittest/mock.py", line 1158, in _execute_mock_call
    result = effect(*args, **kwargs)
  File "/localdev/user/tif/repo/vp_tif_core/doc_gen/doc_gen.py", line 70, in import_mock
    return orig_import(name, *args, **kwargs)
  File "/localdev/user/py_venvs/py3_tif/lib/python3.9/site-packages/requests/exceptions.py", line 34, in <module>
    class JSONDecodeError(InvalidJSONError, CompatJSONDecodeError):
TypeError: metaclass conflict: the metaclass of a derived class must be a (non-strict) subclass of the metaclasses of all its bases

How to Reproduce

pip install sphinx

Expected behavior

No response

Your project

Not able to share as it is classified

Screenshots

No response

OS

CentOS7

Python version

3.9.1

Sphinx version

4.4.0

Sphinx extensions

No response

Extra tools

No response

Additional context

No response

[tk0miya]

The stacktrace you pasted says unittest.mock was called when linkcheck builder goes importing the requests library. I guess the requests library has been mocked on your project, right?

  File "/localdev/user/py_venvs/py3_tif/lib/python3.9/site-packages/sphinx/builders/linkcheck.py", line 28, in <module>
    from requests import Response
  File "/u/ccmaster/SDR/main/linux64/python3/gcc-9.2/lib/python3.9/unittest/mock.py", line 1093, in __call__
    return self._mock_call(*args, **kwargs)
  File "/u/ccmaster/SDR/main/linux64/python3/gcc-9.2/lib/python3.9/unittest/mock.py", line 1097, in _mock_call
    return self._execute_mock_call(*args, **kwargs)
  File "/u/ccmaster/SDR/main/linux64/python3/gcc-9.2/lib/python3.9/unittest/mock.py", line 1158, in _execute_mock_call
    result = effect(*args, **kwargs)

[schnemensch]

That is a good point. I am using unittest.mock to mock some private modules which I do not have access to while generating the documentation. The requests packages should not be part of this mocking and I do not use the requests package at all in my code. I just have it installed because it is a sphinx requirement.

[tk0miya]

Could you share your project or a minimal reproducible example? I can't reproduce the error on my local. At least, I need to know what objects are mocked.