search

jupyter / nbconvert

Jupyter Notebook Conversion
https://nbconvert.readthedocs.io/
BSD 3-Clause "New" or "Revised" License
1.76k stars 569 forks source link

nbconvert --to pdf crashes in resolve_one_reference #669

Open ivanov opened 7 years ago

ivanov commented 7 years ago

This looks like the same error reported in #629.

This happend with nbconvert 5.2.1 and 5.3.1 - pandoc version 1.13.1

(jjcon) 18:35@IPython Kernel$ nbconvert --to pdf  Terminal\ Usage.ipynb 
[TerminalIPythonApp] WARNING | Subcommand `ipython nbconvert` is deprecated and will be removed in future versions.
[TerminalIPythonApp] WARNING | You likely want to use `jupyter nbconvert` in the future
[NbConvertApp] Converting notebook Terminal Usage.ipynb to pdf
Traceback (most recent call last):
  File "/Users/pi/.virtualenvs/jcon/bin/ipython", line 11, in <module>
    load_entry_point('ipython', 'console_scripts', 'ipython')()
  File "/Users/pi/code/jpy/ipython/IPython/__init__.py", line 125, in start_ipython
    return launch_new_instance(argv=argv, **kwargs)
  File "/Users/pi/.virtualenvs/jcon/lib/python3.6/site-packages/traitlets/config/application.py", line 658, in launch_instance
    app.start()
  File "/Users/pi/code/jpy/ipython/IPython/terminal/ipapp.py", line 352, in start
    return self.subapp.start()
  File "/Users/pi/.virtualenvs/jcon/lib/python3.6/site-packages/nbconvert/nbconvertapp.py", line 325, in start
    self.convert_notebooks()
  File "/Users/pi/.virtualenvs/jcon/lib/python3.6/site-packages/nbconvert/nbconvertapp.py", line 493, in convert_notebooks
    self.convert_single_notebook(notebook_filename)
  File "/Users/pi/.virtualenvs/jcon/lib/python3.6/site-packages/nbconvert/nbconvertapp.py", line 464, in convert_single_notebook
    output, resources = self.export_single_notebook(notebook_filename, resources, input_buffer=input_buffer)
  File "/Users/pi/.virtualenvs/jcon/lib/python3.6/site-packages/nbconvert/nbconvertapp.py", line 393, in export_single_notebook
    output, resources = self.exporter.from_filename(notebook_filename, resources=resources)
  File "/Users/pi/.virtualenvs/jcon/lib/python3.6/site-packages/nbconvert/exporters/exporter.py", line 174, in from_filename
    return self.from_file(f, resources=resources, **kw)
  File "/Users/pi/.virtualenvs/jcon/lib/python3.6/site-packages/nbconvert/exporters/exporter.py", line 192, in from_file
    return self.from_notebook_node(nbformat.read(file_stream, as_version=4), resources=resources, **kw)
  File "/Users/pi/.virtualenvs/jcon/lib/python3.6/site-packages/nbconvert/exporters/pdf.py", line 162, in from_notebook_node
    nb, resources=resources, **kw
  File "/Users/pi/.virtualenvs/jcon/lib/python3.6/site-packages/nbconvert/exporters/latex.py", line 82, in from_notebook_node
    return super(LatexExporter, self).from_notebook_node(nb, resources, **kw)
  File "/Users/pi/.virtualenvs/jcon/lib/python3.6/site-packages/nbconvert/exporters/templateexporter.py", line 295, in from_notebook_node
    output = self.template.render(nb=nb_copy, resources=resources)
  File "/Users/pi/.virtualenvs/jcon/lib/python3.6/site-packages/jinja2/asyncsupport.py", line 76, in render
    return original_render(self, *args, **kwargs)
  File "/Users/pi/.virtualenvs/jcon/lib/python3.6/site-packages/jinja2/environment.py", line 1008, in render
    return self.environment.handle_exception(exc_info, True)
  File "/Users/pi/.virtualenvs/jcon/lib/python3.6/site-packages/jinja2/environment.py", line 780, in handle_exception
    reraise(exc_type, exc_value, tb)
  File "/Users/pi/.virtualenvs/jcon/lib/python3.6/site-packages/jinja2/_compat.py", line 37, in reraise
    raise value.with_traceback(tb)
  File "/Users/pi/.virtualenvs/jcon/lib/python3.6/site-packages/nbconvert/exporters/../templates/latex/article.tplx", line 8, in top-level template code
    ((* extends cell_style *))
  File "/Users/pi/.virtualenvs/jcon/lib/python3.6/site-packages/nbconvert/exporters/../templates/latex/style_ipython.tplx", line 60, in top-level template code
    ((( text | add_prompts(first='{\color{' ~ prompt_color ~ '}' ~ prompt ~ '[{\\color{' ~ prompt_color ~ '}' ~ execution_count ~ '}]:} ', cont=indention) )))
  File "/Users/pi/.virtualenvs/jcon/lib/python3.6/site-packages/nbconvert/exporters/../templates/latex/base.tplx", line 6, in top-level template code
    ((*- extends 'document_contents.tplx' -*))
  File "/Users/pi/.virtualenvs/jcon/lib/python3.6/site-packages/nbconvert/exporters/../templates/latex/document_contents.tplx", line 50, in top-level template code
    ((*- block figure scoped -*))
  File "/Users/pi/.virtualenvs/jcon/lib/python3.6/site-packages/nbconvert/exporters/../templates/latex/skeleton/display_priority.tplx", line 5, in top-level template code
    ((*- extends 'null.tplx' -*))
  File "/Users/pi/.virtualenvs/jcon/lib/python3.6/site-packages/nbconvert/exporters/../templates/latex/skeleton/null.tplx", line 30, in top-level template code
    ((*- block body -*))
  File "/Users/pi/.virtualenvs/jcon/lib/python3.6/site-packages/nbconvert/exporters/../templates/latex/base.tplx", line 173, in block "body"
    ((( super() )))
  File "/Users/pi/.virtualenvs/jcon/lib/python3.6/site-packages/nbconvert/exporters/../templates/latex/skeleton/null.tplx", line 32, in block "body"
    ((*- block any_cell scoped -*))
  File "/Users/pi/.virtualenvs/jcon/lib/python3.6/site-packages/nbconvert/exporters/../templates/latex/skeleton/null.tplx", line 85, in block "any_cell"
    ((*- block markdowncell scoped-*)) ((*- endblock markdowncell -*))
  File "/Users/pi/.virtualenvs/jcon/lib/python3.6/site-packages/nbconvert/exporters/../templates/latex/document_contents.tplx", line 67, in block "markdowncell"
    ((( cell.source | citation2latex | strip_files_prefix | convert_pandoc('markdown+tex_math_double_backslash', 'json',extra_args=[]) | resolve_references | convert_pandoc('json','latex'))))
  File "/Users/pi/.virtualenvs/jcon/lib/python3.6/site-packages/nbconvert/filters/filter_links.py", line 16, in resolve_references
    return applyJSONFilters([resolve_one_reference], source)
  File "/Users/pi/.virtualenvs/jcon/lib/python3.6/site-packages/pandocfilters.py", line 197, in applyJSONFilters
    altered = walk(altered, action, format, meta)
  File "/Users/pi/.virtualenvs/jcon/lib/python3.6/site-packages/pandocfilters.py", line 120, in walk
    array.append(walk(item, action, format, meta))
  File "/Users/pi/.virtualenvs/jcon/lib/python3.6/site-packages/pandocfilters.py", line 113, in walk
    array.append(walk(item, action, format, meta))
  File "/Users/pi/.virtualenvs/jcon/lib/python3.6/site-packages/pandocfilters.py", line 124, in walk
    x[k] = walk(x[k], action, format, meta)
  File "/Users/pi/.virtualenvs/jcon/lib/python3.6/site-packages/pandocfilters.py", line 111, in walk
    item['c'] if 'c' in item else None, format, meta)
  File "/Users/pi/.virtualenvs/jcon/lib/python3.6/site-packages/nbconvert/filters/filter_links.py", line 29, in resolve_one_reference
    target = val[2][0]
IndexError: list index out of range

If you suspect this is an IPython bug, please report it at:
    https://github.com/ipython/ipython/issues
or send an email to the mailing list at ipython-dev@python.org

You can print a more detailed traceback right now with "%tb", or use "%debug"
to interactively debug it.

Extra-detailed tracebacks for bug-reporting purposes can be enabled via:
    c.Application.verbose_crash=True
ivanov commented 7 years ago

This may be due to a version mismatch of pandoc and pandocfilter?

pandocfilter docs say: 

Pandoc 1.16 introduced link and image attributes to the existing caption and target arguments, requiring a change in pandocfilters that breaks backwards compatibility. Consequently, you should use:

    pandocfilters version <= 1.2.4 for pandoc versions 1.12–1.15, and
    pandocfilters version >= 1.3.0 for pandoc versions >= 1.16.

Pandoc 1.17.3 (pandoc-types 1.17.*) introduced a new JSON format. pandocfilters 1.4.0 should work with both the old and the new format.

I have pandoc 1.13.1 and pandocfilter 1.4.2.

As a workaround, changes the line to read val[1][0] seems to not cause any issues.

mpacer commented 7 years ago

Is there a reason you don't want to upgrade your pandoc version?

The plan has been to not provide support for older versions of pandoc rather than to try to maintain compatibility with the older API.

rpgoldman commented 6 years ago

I have this problem as well, and as far as I can tell, it is not a version mismatch:

$ port installed pandoc py27-pandocfilters
The following ports are currently installed:
  pandoc @1.12.4.2_1 (active)
  py27-pandocfilters @1.4.2_0 (active)

I have nbconvert 5.3.1 (also from MacPorts).

Export to Markdown seems to work fine. But I get this error whenever I try to dump to Latex or PDF.

Here's part of the error message:

  File "/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/nbconvert/exporters/../templates/latex/document_contents.tplx", line 67, in block "markdowncell"
    ((( cell.source | citation2latex | strip_files_prefix | convert_pandoc('markdown+tex_math_double_backslash', 'json',extra_args=[]) | resolve_references | convert_pandoc('json','latex'))))
  File "/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/nbconvert/filters/filter_links.py", line 16, in resolve_references
    return applyJSONFilters([resolve_one_reference], source)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/pandocfilters.py", line 197, in applyJSONFilters
    altered = walk(altered, action, format, meta)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/pandocfilters.py", line 120, in walk
    array.append(walk(item, action, format, meta))
  File "/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/pandocfilters.py", line 113, in walk
    array.append(walk(item, action, format, meta))
  File "/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/pandocfilters.py", line 124, in walk
    x[k] = walk(x[k], action, format, meta)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/pandocfilters.py", line 111, in walk
    item['c'] if 'c' in item else None, format, meta)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/nbconvert/filters/filter_links.py", line 29, in resolve_one_reference
    target = val[2][0]
IndexError: list index out of range
mpacer commented 6 years ago

Ok, So we should catch this and then warn people to upgrade their version of pandoc explicitly. @pxhanus if you wanted this would be a straightforward issue to tackle.

rpgoldman commented 6 years ago

Sorry, can you clarify? What do I have that's out of date? Pandoc is 1.12.4.2 and the filters at 1.4.2 -- if anything it looks like the filters would be out of date, right? But I guess there's no reason to believe that the two version numbers correlate in any obvious way.

So what should I do to fix this? I got this stuff all from MacPorts, and these seem to be the latest versions. Thanks!

mpacer commented 6 years ago

pandoc needs to be greater than 1.16

for more info read the above comment: https://github.com/jupyter/nbconvert/issues/669#issuecomment-327355193

Pandoc 1.16 introduced link and image attributes to the existing caption and target arguments, requiring a change in pandocfilters that breaks backwards compatibility. Consequently, you should use:

pandocfilters version <= 1.2.4 for pandoc versions 1.12–1.15, and
pandocfilters version >= 1.3.0 for pandoc versions >= 1.16.

Pandoc 1.17.3 (pandoc-types 1.17.*) introduced a new JSON format. pandocfilters 1.4.0 should work with both the old and the new format.

mpacer commented 6 years ago

@pxhanus also, could you fix https://github.com/jupyter/nbconvert/blob/master/nbconvert/utils/pandoc.py to reflect the need to be using pandoc >= 1.16?

FCoulombeau commented 6 years ago

I've got the same issue with

pandoc                    1.19.2.1             hb2460c7_1

installed.

Here is the log :

[NbConvertApp] Converting notebook ./Exemple.ipynb to PDF
Traceback (most recent call last):
  File "C:\Anaconda\Scripts\Jupyter-nbconvert-script.py", line 10, in <module>
    sys.exit(main())
  File "C:\Anaconda\lib\site-packages\jupyter_core\application.py", line 266, in launch_instance
    return super(JupyterApp, cls).launch_instance(argv=argv, **kwargs)
  File "C:\Anaconda\lib\site-packages\traitlets\config\application.py", line 658, in launch_instance
    app.start()
  File "C:\Anaconda\lib\site-packages\nbconvert\nbconvertapp.py", line 325, in start
    self.convert_notebooks()
  File "C:\Anaconda\lib\site-packages\nbconvert\nbconvertapp.py", line 493, in convert_notebooks
    self.convert_single_notebook(notebook_filename)
  File "C:\Anaconda\lib\site-packages\nbconvert\nbconvertapp.py", line 464, in convert_single_notebook
    output, resources = self.export_single_notebook(notebook_filename, resources, input_buffer=input_buffer)
  File "C:\Anaconda\lib\site-packages\nbconvert\nbconvertapp.py", line 393, in export_single_notebook
    output, resources = self.exporter.from_filename(notebook_filename, resources=resources)
  File "C:\Anaconda\lib\site-packages\nbconvert\exporters\exporter.py", line 174, in from_filename
    return self.from_file(f, resources=resources, **kw)
  File "C:\Anaconda\lib\site-packages\nbconvert\exporters\exporter.py", line 192, in from_file
    return self.from_notebook_node(nbformat.read(file_stream, as_version=4), resources=resources, **kw)
  File "C:\Anaconda\lib\site-packages\nbconvert\exporters\pdf.py", line 162, in from_notebook_node
    nb, resources=resources, **kw
  File "C:\Anaconda\lib\site-packages\nbconvert\exporters\latex.py", line 82, in from_notebook_node
    return super(LatexExporter, self).from_notebook_node(nb, resources, **kw)
  File "C:\Anaconda\lib\site-packages\nbconvert\exporters\templateexporter.py", line 295, in from_notebook_node
    output = self.template.render(nb=nb_copy, resources=resources)
  File "C:\Anaconda\lib\site-packages\jinja2\asyncsupport.py", line 76, in render
    return original_render(self, *args, **kwargs)
  File "C:\Anaconda\lib\site-packages\jinja2\environment.py", line 1008, in render
    return self.environment.handle_exception(exc_info, True)
  File "C:\Anaconda\lib\site-packages\jinja2\environment.py", line 780, in handle_exception
    reraise(exc_type, exc_value, tb)
  File "C:\Anaconda\lib\site-packages\jinja2\_compat.py", line 37, in reraise
    raise value.with_traceback(tb)
  File "C:\Anaconda\lib\site-packages\nbconvert\exporters\..\templates\latex\article.tplx", line 8, in top-level template code
    ((* extends cell_style *))
  File "C:\Anaconda\lib\site-packages\nbconvert\exporters\..\templates\latex\style_ipython.tplx", line 60, in top-level template code
    ((( text | add_prompts(first='{\color{' ~ prompt_color ~ '}' ~ prompt ~ '[{\\color{' ~ prompt_color ~ '}' ~ execution_count ~ '}]:} ', cont=indention) )))
  File "C:\Anaconda\lib\site-packages\nbconvert\exporters\..\templates\latex\base.tplx", line 6, in top-level template code
    ((*- extends 'document_contents.tplx' -*))
  File "C:\Anaconda\lib\site-packages\nbconvert\exporters\..\templates\latex\document_contents.tplx", line 50, in top-level template code
    ((*- block figure scoped -*))
  File "C:\Anaconda\lib\site-packages\nbconvert\exporters\..\templates\latex\skeleton\display_priority.tplx", line 5, in top-level template code
    ((*- extends 'null.tplx' -*))
  File "C:\Anaconda\lib\site-packages\nbconvert\exporters\..\templates\latex\skeleton\null.tplx", line 30, in top-level template code
    ((*- block body -*))
  File "C:\Anaconda\lib\site-packages\nbconvert\exporters\..\templates\latex\base.tplx", line 173, in block "body"
    ((( super() )))
  File "C:\Anaconda\lib\site-packages\nbconvert\exporters\..\templates\latex\skeleton\null.tplx", line 32, in block "body"
    ((*- block any_cell scoped -*))
  File "C:\Anaconda\lib\site-packages\nbconvert\exporters\..\templates\latex\skeleton\null.tplx", line 85, in block "any_cell"
    ((*- block markdowncell scoped-*)) ((*- endblock markdowncell -*))
  File "C:\Anaconda\lib\site-packages\nbconvert\exporters\..\templates\latex\document_contents.tplx", line 67, in block "markdowncell"
    ((( cell.source | citation2latex | strip_files_prefix | convert_pandoc('markdown+tex_math_double_backslash', 'json',extra_args=[]) | resolve_references | convert_pandoc('json','latex'))))
  File "C:\Anaconda\lib\site-packages\nbconvert\filters\filter_links.py", line 16, in resolve_references
    return applyJSONFilters([resolve_one_reference], source)
  File "C:\Anaconda\lib\site-packages\pandocfilters.py", line 197, in applyJSONFilters
    altered = walk(altered, action, format, meta)
  File "C:\Anaconda\lib\site-packages\pandocfilters.py", line 120, in walk
    array.append(walk(item, action, format, meta))
  File "C:\Anaconda\lib\site-packages\pandocfilters.py", line 113, in walk
    array.append(walk(item, action, format, meta))
  File "C:\Anaconda\lib\site-packages\pandocfilters.py", line 124, in walk
    x[k] = walk(x[k], action, format, meta)
  File "C:\Anaconda\lib\site-packages\pandocfilters.py", line 111, in walk
    item['c'] if 'c' in item else None, format, meta)
  File "C:\Anaconda\lib\site-packages\nbconvert\filters\filter_links.py", line 29, in resolve_one_reference
    target = val[2][0]
IndexError: list index out of range
Tagar commented 5 years ago

Getting same errors ..

pandoc 1.0.2 is latest version ? https://pypi.org/project/pandoc/

cc @mpacer

thanks

piskvorky commented 5 years ago

Same here; pandoc-1.0.2 seems the latest version at PyPI. All dependencies installed automatically with pip via jupyter, no manual tinkering (and pandoc wasn't even among them).

@ivanov 's fix (?) above seems to resolve this (at least I don't see any adverse issues in my PDF).

MSeal commented 5 years ago

https://pypi.org/project/pandoc/ is not the supported pandoc wrapper that will install latest pandoc binaries (see the readme).

You need to follow http://pandoc.org/installing.html to install the pandoc version that pairs with the pandocfilters constraints.