spatialaudio / nbsphinx

:ledger: Sphinx source parser for Jupyter notebooks
https://nbsphinx.readthedocs.io/
MIT License
459 stars 132 forks source link

Assertion Error in nbsphinx.py during latex build #584

Closed AAndersn closed 3 years ago

AAndersn commented 3 years ago

When using nbsphinx to build on readthedocs.org, nbsphinx has started to crash today (07/12/21) when asked to build latex:

python -m sphinx -b latex -D language=en -d _build/doctrees . _build/latex

The error message in the terminal is

processing notebook_1 notebook_2 notebook_n
resolving references...
done
writing... failed

Exception occurred:
  File "/home/docs/checkouts/readthedocs.org/user_builds/gridappsd-hackathon/envs/develop/lib/python3.7/site-packages/nbsphinx.py", line 2151, in depart_codearea_latex
    assert 'Verbatim' in lines[0]
AssertionError
The full traceback has been saved in /tmp/sphinx-err-oj7amn5e.log, if you want to report the issue to the developers.
Please also report this if it was a user error, so that a better error message can be provided next time.
A bug report can be filed in the tracker at <https://github.com/sphinx-doc/sphinx/issues>. Thanks!

I have been able to reproduce this message on my local machine, with the following log message:

# Sphinx version: 4.1.0
# Python version: 3.8.5 (CPython)
# Docutils version: 0.16 release
# Jinja2 version: 2.11.0
# Last messages:
#   api_usage/3.3--Lesson-3.3--Using-the-PowerGrid-Models-API
#   api_usage/3.4--Lesson-3.4--Using-the-Configuration-File-API
#   api_usage/3.5--Lesson-3.5--Creating-Running-Simulation-API
#   api_usage/3.6--Lesson-3.6--Controlling-Simulation-API
#   api_usage/3.7--Lesson-3.7--Using-the-Timeseries-API
#   
#   resolving references...
#   done
#   writing...
#   failed
# Loaded extensions:
#   sphinx.ext.mathjax (4.1.0) from /home/ubuntu/venv/lib/python3.8/site-packages/sphinx/ext/mathjax.py
#   sphinxcontrib.applehelp (1.0.2) from /home/ubuntu/venv/lib/python3.8/site-packages/sphinxcontrib/applehelp/__init__.py
#   sphinxcontrib.devhelp (1.0.2) from /home/ubuntu/venv/lib/python3.8/site-packages/sphinxcontrib/devhelp/__init__.py
#   sphinxcontrib.htmlhelp (2.0.0) from /home/ubuntu/venv/lib/python3.8/site-packages/sphinxcontrib/htmlhelp/__init__.py
#   sphinxcontrib.serializinghtml (1.1.5) from /home/ubuntu/venv/lib/python3.8/site-packages/sphinxcontrib/serializinghtml/__init__.py
#   sphinxcontrib.qthelp (1.0.3) from /home/ubuntu/venv/lib/python3.8/site-packages/sphinxcontrib/qthelp/__init__.py
#   alabaster (0.7.12) from /home/ubuntu/venv/lib/python3.8/site-packages/alabaster/__init__.py
#   nbsphinx (0.8.6) from /home/ubuntu/venv/lib/python3.8/site-packages/nbsphinx.py
#   sphinx.ext.autodoc.preserve_defaults (1.0) from /home/ubuntu/venv/lib/python3.8/site-packages/sphinx/ext/autodoc/preserve_defaults.py
#   sphinx.ext.autodoc.type_comment (4.1.0) from /home/ubuntu/venv/lib/python3.8/site-packages/sphinx/ext/autodoc/type_comment.py
#   sphinx.ext.autodoc (4.1.0) from /home/ubuntu/venv/lib/python3.8/site-packages/sphinx/ext/autodoc/__init__.py
#   sphinx.ext.intersphinx (4.1.0) from /home/ubuntu/venv/lib/python3.8/site-packages/sphinx/ext/intersphinx.py
#   sphinx.ext.todo (4.1.0) from /home/ubuntu/venv/lib/python3.8/site-packages/sphinx/ext/todo.py
#   sphinx.ext.coverage (4.1.0) from /home/ubuntu/venv/lib/python3.8/site-packages/sphinx/ext/coverage.py
#   sphinx.ext.ifconfig (4.1.0) from /home/ubuntu/venv/lib/python3.8/site-packages/sphinx/ext/ifconfig.py
#   sphinx.ext.viewcode (4.1.0) from /home/ubuntu/venv/lib/python3.8/site-packages/sphinx/ext/viewcode.py
#   sphinx.ext.githubpages (4.1.0) from /home/ubuntu/venv/lib/python3.8/site-packages/sphinx/ext/githubpages.py
Traceback (most recent call last):
  File "/home/ubuntu/venv/lib/python3.8/site-packages/sphinx/cmd/build.py", line 280, in build_main
    app.build(args.force_all, filenames)
  File "/home/ubuntu/venv/lib/python3.8/site-packages/sphinx/application.py", line 343, in build
    self.builder.build_update()
  File "/home/ubuntu/venv/lib/python3.8/site-packages/sphinx/builders/__init__.py", line 290, in build_update
    self.build(['__all__'], to_build)
  File "/home/ubuntu/venv/lib/python3.8/site-packages/sphinx/builders/__init__.py", line 357, in build
    self.write(docnames, list(updated_docnames), method)
  File "/home/ubuntu/venv/lib/python3.8/site-packages/sphinx/builders/latex/__init__.py", line 308, in write
    docwriter.write(doctree, destination)
  File "/home/ubuntu/venv/lib/python3.8/site-packages/docutils/writers/__init__.py", line 78, in write
    self.translate()
  File "/home/ubuntu/venv/lib/python3.8/site-packages/sphinx/writers/latex.py", line 99, in translate
    self.document.walkabout(visitor)
  File "/home/ubuntu/venv/lib/python3.8/site-packages/docutils/nodes.py", line 214, in walkabout
    if child.walkabout(visitor):
  File "/home/ubuntu/venv/lib/python3.8/site-packages/docutils/nodes.py", line 214, in walkabout
    if child.walkabout(visitor):
  File "/home/ubuntu/venv/lib/python3.8/site-packages/docutils/nodes.py", line 214, in walkabout
    if child.walkabout(visitor):
  [Previous line repeated 5 more times]
  File "/home/ubuntu/venv/lib/python3.8/site-packages/docutils/nodes.py", line 227, in walkabout
    visitor.dispatch_departure(self)
  File "/home/ubuntu/venv/lib/python3.8/site-packages/sphinx/util/docutils.py", line 486, in dispatch_departure
    method(node)
  File "/home/ubuntu/venv/lib/python3.8/site-packages/nbsphinx.py", line 2151, in depart_codearea_latex
    assert 'Verbatim' in lines[0]
AssertionError

My repo and source notebooks have not changed (readthedocs is building from the same commit)

image

image

AAndersn commented 3 years ago

Update: It seems to be a compatibility issue with sphinx 4.1.0.

Pinning sphinx to version 4.0.2 using the requirements.txt file resolved the build error.

aida-ahmadi commented 3 years ago

Update: It seems to be a compatibility issue with sphinx 4.1.0.

Pinning sphinx to version 4.0.2 using the requirements.txt file resolved the build error.

@AAndersn Thanks for figuring this out. I was stuck on it for hours yesterday.

mgeier commented 3 years ago

Thanks for the report!

Apparently (as git bisect told me), this was caused by https://github.com/sphinx-doc/sphinx/pull/9166.

I'll have to look into it a bit more for how to fix this.

aiqc commented 3 years ago

thank you! build passes with these dependencies:

requiresments.txt

sphinx==4.0.2
ipython==7.19.0
nbsphinx>=0.8.6
sphinx_copybutton==0.3.1
sphinx_rtd_theme==0.5.0
sphinxext-opengraph==0.4.1
jinja2>=3.0.1
mgeier commented 3 years ago

See also https://github.com/sphinx-doc/sphinx/issues/9458.

jfbu commented 3 years ago

Commenting out the assert lines I observe that the latex build output of nbsphinx own documentation contains:

\begin{sphinxuseclass}{nbinput}
\begin{sphinxuseclass}{nblast}
{
\sphinxsetup{VerbatimColor={named}{nbsphinx-code-bg}}
\sphinxsetup{VerbatimBorderColor={named}{nbsphinx-code-border}}
\begin{sphinxVerbatim}[commandchars=\\\{\}]
\llap{\color{nbsphinxin}[ ]:\,\hspace{\fboxrule}\hspace{\fboxsep}}
\end{sphinxVerbatim}
}

\end{sphinxuseclass}
\end{sphinxuseclass}

The fix to the issue is to define latex environment sphinxnbinput and sphinxnblast which will do the

\sphinxsetup{VerbatimColor={named}{nbsphinx-code-bg}}
\sphinxsetup{VerbatimBorderColor={named}{nbsphinx-code-border}}

thing (I don't know to which class assign these tasks), rather than having theme explicitly show up in the latex code. Also they would not add the explicit { and } which are not needed as LaTeX environments limit scopes.

Same problem with the class nboutput. I observe this kind of code:

\begin{sphinxuseclass}{nboutput}
\begin{sphinxuseclass}{nblast}
{

\kern-\sphinxverbatimsmallskipamount\kern-\baselineskip
\kern+\FrameHeightAdjust\kern-\fboxrule
\vspace{\nbsphinxcodecellspacing}

\sphinxsetup{VerbatimColor={named}{white}}
\sphinxsetup{VerbatimBorderColor={named}{nbsphinx-code-border}}
\begin{sphinxuseclass}{output_area}
\llap{\color{nbsphinxout}[2]:\,\hspace{\fboxrule}\hspace{\fboxsep}}\begin{sphinxuseclass}{}

\begin{sphinxVerbatim}[commandchars=\\\{\}]
42
\end{sphinxVerbatim}

\end{sphinxuseclass}
\end{sphinxuseclass}
}

\end{sphinxuseclass}
\end{sphinxuseclass}

which has 4 nested classes, from outer to inner nboutput, nblast, output_area, and (I don't know how this comes about) the "empty" class.

I'll see if task does not appear too complicated in the time I dispose of and possibly make a PR. I don't see an easy replacement to the assert 'Verbatim' in lines[0] and perhaps simply removing those 2 lines in nbsphinx.py is enough of a temporary hotfix? Indeed, regarding building nbsphinx.pdf, I got no further issue apart from one warning about a missing reference

Latexmk: List of undefined refs and citations:
  Reference `markdown-cells:' on page 27 undefined on input line 2384
Latexmk: Summary of warnings from last run of *latex:
  Latex failed to resolve 1 reference(s)
Latexmk: All targets (nbsphinx.pdf) are up-to-date

So simply removing these assert lines (whose purpose I don't know) appears to fix the build problems.

jfbu commented 3 years ago

After having looked a bit more I realize I don't have a clear enough view of the nbsphinx proceedings to understand clearly who originates what, for example output_area class. I started doing things like this:

% Define an environment to support the nbinput class (Sphinx>=4.1.0)
\newenvironment{sphinxclassnbinput}{%
    \sphinxsetup{VerbatimColor={named}{nbsphinx-code-bg}}%
    \sphinxsetup{VerbatimBorderColor={named}{nbsphinx-code-border}}%
}{}

to be added to LATEX_PREAMBLE. This would go in pair with conditionally removing the

        out.append(r'\sphinxsetup{VerbatimColor={named}{nbsphinx-code-bg}}')

and

    out.append(
        r'\sphinxsetup{VerbatimBorderColor={named}{nbsphinx-code-border}}')

lines in depart_codearea_latex(self, node). But I see also stderr class.

In brief, Sphinx 4.1.0 allows to support via LaTeX environments what is implemented in HTML via CSS classes. But the current nbsphinx code is backwards compatible and needs no adaptation apart from removing the two "assert".

In future though, a transfer of the hacking via DocUtils nodes and added visit and depart handlers to simply adding to LATEX_PREAMBLE a whole bunch of environment definitions, one for each class, seems the way to refactor the nbsphinx LaTeX code into something simpler, and much more parallel to HTML output. I can not engage into it at current time as I don't know the nbsphinx codebase enough.

mgeier commented 3 years ago

Sorry for my late response!

So simply removing these assert lines (whose purpose I don't know) appears to fix the build problems.

It indeed allows the build to pass, but the resulting PDF doesn't look as expected.

nbsphinx needs to insert the output prompt at the correct line (right after the sphinxVerbatim environment is opened).

Less importantly (but still), there might be some cases where additional blank lines have to be inserted in the beginning or the end of the output (because reST syntax doesn't allow leading or trailing blank lines).

I checked the LaTeX lines in question (for an output cell only containing the number 42) produced by Sphinx 4.0.2 (prior to changes made by nbsphinx):

['\\begin{sphinxVerbatim}[commandchars=\\\\\\{\\}]', '42', '\\end{sphinxVerbatim}']

With the latest Sphinx version, those lines are changed to:

['\\begin{sphinxuseclass}{output_area}', '\\begin{sphinxuseclass}{}', '', '', '\\begin{sphinxVerbatim}[commandchars=\\\\\\{\\}]', '42', '\\end{sphinxVerbatim}', '', '\\end{sphinxuseclass}', '\\end{sphinxuseclass}']

It would be interesting to properly use the sphinxuseclass environments, but as @jfbu said, I'd like nbsphinx to be compatible with older Sphinx versions

mgeier commented 3 years ago

I've just created #591, which hopefully fixes this issue.

mgeier commented 3 years ago

I have just created a new nbsphinx release (version 0.8.7) which should fix this issue.