sage --ipynb2rst is broken due to changes in nbconvert

[seblabbe]

On Ubuntu 18.04 with 9.5.beta3, the command

$ sage -t --optional=sage,pandoc src/sage/tests/cmdline.py

gives

Using --optional=pandoc,sage
Doctesting 1 file.
sage -t --random-seed=0 src/sage/tests/cmdline.py
**********************************************************************
File "src/sage/tests/cmdline.py", line 732, in sage.tests.cmdline.test_executable
Failed example:
    print(open(output, 'r').read() == t)          # optional - pandoc
Expected:
    True
Got:
    False
**********************************************************************
1 item had failures:
   1 of 209 in sage.tests.cmdline.test_executable
    [208 tests, 1 failure, 31.05 s]
----------------------------------------------------------------------
sage -t --random-seed=0 src/sage/tests/cmdline.py  # 1 doctest failed
----------------------------------------------------------------------

Depends on #32174

CC: @sagetrac-tmonteil

Component: packages: optional

Author: Thierry Monteil

Branch/Commit: u/mkoeppe/sage___ipynb2rst_is_broken_due_to_changes_in_nbconvert @ ad47d1f

Issue created by migration from https://trac.sagemath.org/ticket/32697

[seblabbe]

comment:1

Let the following be the content of the file a.ipynb:

{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "2"
      ]
     },
     "execution_count": 1,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "1+1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "SageMath 8.3.beta4",
   "language": "",
   "name": "sagemath"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.15"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}

Also it seems that I have pandoc installed:

sage: from sage.doctest.external import has_pandoc
sage: has_pandoc()
FeatureTestResult('Pandoc', True)
sage: bool(_)
True

although:

$ sage --optional | grep pandoc
pandoc..................................none (not_installed)
pandoc_attributes.......................8bc82f6d (8bc82f6d)

Then, the command

$ sage --ipynb2rst a.ipynb

gives

[NbConvertApp] WARNING | Config option `template_path` not recognized by `RSTExporter`.  Did you mean one of: `extra_template_paths, template_name, template_paths`?
[NbConvertApp] Converting notebook a.ipynb to rst
Traceback (most recent call last):
  File "/home/slabbe/GitBox/sage/local/bin/jupyter-nbconvert", line 8, in <module>
    sys.exit(main())
  File "/home/slabbe/GitBox/sage/local/lib/python3.9/site-packages/jupyter_core/application.py", line 254, in launch_instance
    return super(JupyterApp, cls).launch_instance(argv=argv, **kwargs)
  File "/home/slabbe/GitBox/sage/local/lib/python3.9/site-packages/traitlets/config/application.py", line 846, in launch_instance
    app.start()
  File "/home/slabbe/GitBox/sage/local/lib/python3.9/site-packages/nbconvert/nbconvertapp.py", line 346, in start
    self.convert_notebooks()
  File "/home/slabbe/GitBox/sage/local/lib/python3.9/site-packages/nbconvert/nbconvertapp.py", line 518, in convert_notebooks
    self.convert_single_notebook(notebook_filename)
  File "/home/slabbe/GitBox/sage/local/lib/python3.9/site-packages/nbconvert/nbconvertapp.py", line 483, in convert_single_notebook
    output, resources = self.export_single_notebook(notebook_filename, resources, input_buffer=input_buffer)
  File "/home/slabbe/GitBox/sage/local/lib/python3.9/site-packages/nbconvert/nbconvertapp.py", line 412, in export_single_notebook
    output, resources = self.exporter.from_filename(notebook_filename, resources=resources)
  File "/home/slabbe/GitBox/sage/local/lib/python3.9/site-packages/nbconvert/exporters/exporter.py", line 181, in from_filename
    return self.from_file(f, resources=resources, **kw)
  File "/home/slabbe/GitBox/sage/local/lib/python3.9/site-packages/nbconvert/exporters/exporter.py", line 199, in from_file
    return self.from_notebook_node(nbformat.read(file_stream, as_version=4), resources=resources, **kw)
  File "/home/slabbe/GitBox/sage/local/lib/python3.9/site-packages/nbconvert/exporters/templateexporter.py", line 390, in from_notebook_node
    output = self.template.render(nb=nb_copy, resources=resources)
  File "/home/slabbe/GitBox/sage/local/lib/python3.9/site-packages/nbconvert/exporters/templateexporter.py", line 149, in template
    self._template_cached = self._load_template()
  File "/home/slabbe/GitBox/sage/local/lib/python3.9/site-packages/nbconvert/exporters/templateexporter.py", line 360, in _load_template
    return self.environment.get_template(template_file)
  File "/home/slabbe/GitBox/sage/local/lib/python3.9/site-packages/jinja2/environment.py", line 883, in get_template
    return self._load_template(name, self.make_globals(globals))
  File "/home/slabbe/GitBox/sage/local/lib/python3.9/site-packages/jinja2/environment.py", line 857, in _load_template
    template = self.loader.load(self, name, globals)
  File "/home/slabbe/GitBox/sage/local/lib/python3.9/site-packages/jinja2/loaders.py", line 429, in load
    raise TemplateNotFound(name)
jinja2.exceptions.TemplateNotFound: rst_sage.tpl

Usage:

    sage -ipynb2rst <source> [<destination>]

Generates reStructuredText source (.rst) from IPython worksheet (.ipynb).

If the destination is not specified, the reStructuredText source is written in
the standard output.

Examples:

    sage -ipynb2rst file.ipynb file.rst

    sage -ipynb2rst file.ipynb

Options:
  -h, --help            show this help message and exit
Traceback (most recent call last):
  File "/home/slabbe/GitBox/sage/local/lib/python3.9/site-packages/sage/ext_data/nbconvert/postprocess.py", line 20, in <module>
    with open(file_name, 'r') as f:
FileNotFoundError: [Errno 2] No such file or directory: 'a.rst'

[seblabbe]

comment:2

Salut Thierry, it seems ipynb2rst is currently broken... Can you take a look?

[edd8e884-f507-429a-b577-5d554626c0fe]

Author: Thierry Monteil

[edd8e884-f507-429a-b577-5d554626c0fe]

comment:3

This is not a pandoc issue. I see two problems due to changes in nbconvert:

• the syntax to define template path changed
• the templates we are inheriting from disapeared from upstream

Let me fix that.

[edd8e884-f507-429a-b577-5d554626c0fe]

Branch: u/tmonteil/sage___ipynb2rst_is_broken_due_to_changes_in_nbconvert

[edd8e884-f507-429a-b577-5d554626c0fe]

Commit: e50afe6

[edd8e884-f507-429a-b577-5d554626c0fe]

New commits:

36635dc	#32697 : syntax to define template paths changed in nbconvert
e50afe6	#32697 : null.tpl and display_priority.tpl templates are not provided by nbconvert anymore

[edd8e884-f507-429a-b577-5d554626c0fe]

comment:7

I forgot to package jupyter_contrib_nbextensions that is a new dependency of nbconvert. At least, if you need it right now, you can just do sage -pip install jupyter_contrib_nbextensions and the current branch should work.

[mkoeppe]

comment:8

I don't see this dependency in https://github.com/jupyter/nbconvert/blob/main/setup.py, can you clarify?

[mkoeppe]

Dependencies: #32174

[mkoeppe]

Changed branch from u/tmonteil/sage___ipynb2rst_is_broken_due_to_changes_in_nbconvert to u/mkoeppe/sage___ipynb2rst_is_broken_due_to_changes_in_nbconvert

[seblabbe]

Changed commit from e50afe6 to ad47d1f

[seblabbe]

comment:12

Bonne année Thierry! Ce serait pas mal de régler ce problème pour 9.5. Est-ce tu as une idée comment corriger ce problème et ajouter le package manquant ou je ne sais trop? J'agirai comme reviewer.

---

New commits:

95efa55	Merge #32174
ad47d1f	Revert "src/sage/tests/cmdline.py: Mark a doctest # known bug #32697"

[slel]

comment:13

Set milestone to sage-9.6 after Sage 9.5 release.