make docs fails with recent sphinx-doc

On my macOS 12.5.1 (Intel) system:

make docs fails with

Exception occurred:
  File "/usr/local/lib/python3.10/site-packages/sphinx/highlighting.py", line 151, in get_lexer
    lexer = lexer_classes[lang](**opts)
TypeError: 'BNFLexer' object is not callable
...
make[1]: *** [latexpdf] Error 2

There is a second error

WARNING: sphinx_rtd_theme (< 0.3.0) found. It will not be available since Sphinx-6.0
Theme error: no theme named 'sphinx_rtd_theme' found (missing theme.conf?)
make: *** [singlehtml] Error 2

As a result, make docs currently builds neither the PDF nor the HTML documentation.

From https://www.sphinx-doc.org/en/master/extdev/appapi.html?highlight=lexer#sphinx.application.Sphinx.add_lexer

Sphinx.add_lexer(alias: str, lexer: Type[Lexer]) → None [source] Register a new lexer for source code.

Changed in version 2.1: Take a lexer class as an argument. An instance of lexers are still supported until Sphinx-3.x.

My version of sphinx-doc:

> brew info sphinx-doc
sphinx-doc: stable 5.1.1

As far as I can tell sphinx-doc version 2.x is being run by both the build validation system and for the https://conjure.readthedocs.io/en/latest/ site to build the documentation, which is perhaps why this hasn't been flagged before. While this continues the issue is probably of minor importance since I think most people use the online documentation, but this will become serious if/when these third parties upgrade to more recent versions of sphinx-doc.

If I modify the name of BNFLexer (to avoid potential conflicts with an existing lexer), the error switches to EssenceLexer; I am somewhat confused now. I wasn't seeing this when just changing the order of the calls to the two lexers, but perhaps Python is hashing them and changing the name has changed which hash gets to be considered first? If this is right, then neither lexer is currently working. I do not know how to proceed at this point, my Python knowledge currently doesn't extend to the internals of its object model.

diff --git a/docs/BNFLexer.py b/docs/BNFLexer.py
index 2b4cc2929..93053f909 100644
--- a/docs/BNFLexer.py
+++ b/docs/BNFLexer.py
@@ -3,9 +3,9 @@ from pygments.lexer import *
 from pygments.token import *
 import re

-class BNFLexer(RegexLexer):
+class ConjureBNFLexer(RegexLexer):

-    name = 'bnf'
+    name = 'conjurebnf'
     # flags = re.MULTILINE | re.DOTALL

     ascii = ('NUL', 'SOH', '[SE]TX', 'EOT', 'ENQ', 'ACK',
diff --git a/docs/conf.py b/docs/conf.py
index 4c64296a0..b789c8f09 100644
--- a/docs/conf.py
+++ b/docs/conf.py
@@ -383,7 +383,7 @@ inline_highlight_respect_highlight = True
 inline_highlight_literals = True

 def setup(sphinx):
-    from BNFLexer import BNFLexer
-    sphinx.add_lexer("bnf", BNFLexer())
+    from BNFLexer import ConjureBNFLexer
+    sphinx.add_lexer("conjurebnf", ConjureBNFLexer())
     from EssenceLexer import EssenceLexer
     sphinx.add_lexer("essence", EssenceLexer())

Error:

Exception occurred:
  File "/usr/local/lib/python3.10/site-packages/sphinx/highlighting.py", line 151, in get_lexer
    lexer = lexer_classes[lang](**opts)
TypeError: 'EssenceLexer' object is not callable

Documenting current progress. I tweaked the syntax of the calls to add_lexer in conf.py:

     from BNFLexer import BNFLexer
-    sphinx.add_lexer("bnf", BNFLexer())
+    sphinx.add_lexer("bnf", BNFLexer)
     from EssenceLexer import EssenceLexer
-    sphinx.add_lexer("essence", EssenceLexer())
+    sphinx.add_lexer("essence", EssenceLexer)

and this now seems to allow make latexpdf to complete with Sphinx-5. However, there seem to be some version glitches. My setup:

> pip3 list

Package                       Version
----------------------------- ---------
alabaster                     0.7.12
Babel                         2.10.3
certifi                       2022.6.15
charset-normalizer            2.1.1
clingo                        5.5.2
docutils                      0.17.1
idna                          3.3
imagesize                     1.4.1
Jinja2                        3.1.2
latexcodec                    2.0.1
MarkupSafe                    2.1.1
packaging                     21.3
pip                           22.2.2
pybtex                        0.24.0
pybtex-docutils               1.0.2
Pygments                      2.13.0
pyparsing                     3.0.9
pytz                          2022.2.1
PyYAML                        6.0
requests                      2.28.1
setuptools                    65.3.0
six                           1.16.0
snowballstemmer               2.2.0
Sphinx                        5.1.1
sphinxcontrib-applehelp       1.0.2
sphinxcontrib-bibtex          2.5.0
sphinxcontrib-devhelp         1.0.2
sphinxcontrib-htmlhelp        2.0.0
sphinxcontrib-jsmath          1.0.1
sphinxcontrib-qthelp          1.0.3
sphinxcontrib-serializinghtml 1.1.5
urllib3                       1.26.12
wheel                         0.37.1

I think the key difference to what one would expect is docutils 0.17.1 which is quite old; I got this by installing Sphinx-2.x instead of 5.x, which removes docutils-0.19.x and installs this older one, then upgrading Sphinx again. Some version dependencies are unresolved and the one-sided intervals specified appear to be incorrect: some of the packages seem to need upper bounds as well as lower bounds but the requirements files do not reflect these constraints (a package can't know which future versions of its dependencies will break things).

To install the dependencies from a new Python3 install, I think pip3 install 'docutils = 0.17.1' sphinxcontrib-bibtex should work, but pip3 install sphinxcontrib-bibtex might fail. This needs to be confirmed.

It looks like with the current versions of the various Python packages,

pip3 install sphinx-rtd-theme sphinxcontrib-bibtex

will install the right versions of all the Python packages needed for make docs to work once the lexer calls have been modified as above. I will add this to the documentation and open a merge request.

conjure-cp / conjure

make docs fails with recent sphinx-doc #523