Use of SVG generates TypeError: Object of type 'bytes' is not JSON serializable

[jmgc]

When using SVG plots

%config InlineBackend.figure_format = 'svg'

the conversion to latex generate the error:

TypeError: Object of type 'bytes' is not JSON serializable

If I use PDF or PNG

%config InlineBackend.figure_format = 'pdf'

or

%config InlineBackend.figure_format = 'png'

No error is detected. I understand that this is an issue associated with the svg to json conversion. The log is:

jupyter-nbconvert-3.6 --to pdf notebook.ipynb

[NbConvertApp] Converting notebook ProbeArmResults.ipynb to pdf
Traceback (most recent call last):
  File "/opt/local/bin/jupyter-nbconvert-3.6", line 11, in <module>
    load_entry_point('nbconvert==5.4.0', 'console_scripts', 'jupyter-nbconvert')()
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/jupyter_core/application.py", line 266, in launch_instance
    return super(JupyterApp, cls).launch_instance(argv=argv, **kwargs)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/traitlets/config/application.py", line 658, in launch_instance
    app.start()
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/nbconvert/nbconvertapp.py", line 337, in start
    self.convert_notebooks()
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/nbconvert/nbconvertapp.py", line 507, in convert_notebooks
    self.convert_single_notebook(notebook_filename)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/nbconvert/nbconvertapp.py", line 478, in convert_single_notebook
    output, resources = self.export_single_notebook(notebook_filename, resources, input_buffer=input_buffer)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/nbconvert/nbconvertapp.py", line 407, in export_single_notebook
    output, resources = self.exporter.from_filename(notebook_filename, resources=resources)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/nbconvert/exporters/exporter.py", line 178, in from_filename
    return self.from_file(f, resources=resources, **kw)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/nbconvert/exporters/exporter.py", line 196, in from_file
    return self.from_notebook_node(nbformat.read(file_stream, as_version=4), resources=resources, **kw)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/nbconvert/exporters/pdf.py", line 158, in from_notebook_node
    nb, resources=resources, **kw
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/nbconvert/exporters/latex.py", line 83, in from_notebook_node
    return super(LatexExporter, self).from_notebook_node(nb, resources, **kw)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/nbconvert/exporters/templateexporter.py", line 300, in from_notebook_node
    nb_copy, resources = super(TemplateExporter, self).from_notebook_node(nb, resources, **kw)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/nbconvert/exporters/exporter.py", line 138, in from_notebook_node
    nb_copy, resources = self._preprocess(nb_copy, resources)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/nbconvert/exporters/exporter.py", line 315, in _preprocess
    nbc, resc = preprocessor(nbc, resc)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/nbconvert/preprocessors/base.py", line 47, in __call__
    return self.preprocess(nb, resources)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/nbconvert/preprocessors/base.py", line 69, in preprocess
    nb.cells[index], resources = self.preprocess_cell(cell, resources, index)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/nbconvert/preprocessors/extractoutput.py", line 92, in preprocess_cell
    data = json.dumps(data)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/json/__init__.py", line 231, in dumps
    return _default_encoder.encode(obj)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/json/encoder.py", line 199, in encode
    chunks = self.iterencode(o, _one_shot=True)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/json/encoder.py", line 257, in iterencode
    return _iterencode(o, 0)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/json/encoder.py", line 180, in default
    o.__class__.__name__)
TypeError: Object of type 'bytes' is not JSON serializable

[sgsaenger]

There's something really strange going on.

847 introduced

if (                                                          
    not isinstance(data, text_type)                           
    or mime_type == 'application/json'                        
):                                                            
    # Data is either JSON-like and was parsed into a Python   
    # object according to the spec, or data is for sure       
    # JSON. In the latter case we want to go extra sure that  
    # we enclose a scalar string value into extra quotes by   
    # serializing it properly.                                
    data = json.dumps(data)                                   

assuming all non-string data to be otherwise valid json. I think this assumption is way too strong, as can be seen here:

• When using the pdf-backend, the resulting pdf will be stored as a string in the cell.outputs, and will thus pass the not isinstance(data, text_type) check
• When using the svg-backend, some other part of the code will convert the svg to pdf and store it as bytes, thus not passing the not isinstance(data, text_type) check and triggering the error

I don't know anything about json-data embedded in cells, but i don't think it's reasonable to assume all non-string data should be piped through json.dumps

[SpencerPark]

This does actually seem to be in svg2pdf. It is injecting invalid json into a notebook cell outputs. It just manifests itself as an issue with extractoutput.

It uses base64.encodestring which (despite the name) returns a bytes object. https://github.com/jupyter/nbconvert/blob/9a22426e8fb9e0027e0ce5532e3610fff9632721/nbconvert/preprocessors/svg2pdf.py#L100

[MSeal]

This should be fixed in the nbconvert 5.5 release already. I believe this is supposed to be returning bytes and not a unicode string.

[rpgoldman]

@MSeal

This should be fixed in the nbconvert 5.5 release already. I believe this is supposed to be returning bytes and not a unicode string.

FWIW, I am running with nbconvert 5.5.0 and I just got this error a few minutes ago trying to use the command to download as PDF through latex:

[E 12:18:44.434 NotebookApp] 500 GET /nbconvert/pdf/Continuous%20model.ipynb?download=true (::1) 1516.76ms referer=http://localhost:8888/notebooks/Continuous%20model.ipynb
[E 12:19:03.774 NotebookApp] nbconvert failed: Object of type bytes is not JSON serializable
    Traceback (most recent call last):
      File "/Users/rpg/Library/Python/3.7/lib/python/site-packages/notebook/nbconvert/handlers.py", line 130, in get
        resources=resource_dict
      File "/Users/rpg/Library/Python/3.7/lib/python/site-packages/nbconvert/exporters/pdf.py", line 158, in from_notebook_node
        nb, resources=resources, **kw
      File "/Users/rpg/Library/Python/3.7/lib/python/site-packages/nbconvert/exporters/latex.py", line 84, in from_notebook_node
        return super(LatexExporter, self).from_notebook_node(nb, resources, **kw)
      File "/Users/rpg/Library/Python/3.7/lib/python/site-packages/nbconvert/exporters/templateexporter.py", line 300, in from_notebook_node
        nb_copy, resources = super(TemplateExporter, self).from_notebook_node(nb, resources, **kw)
      File "/Users/rpg/Library/Python/3.7/lib/python/site-packages/nbconvert/exporters/exporter.py", line 138, in from_notebook_node
        nb_copy, resources = self._preprocess(nb_copy, resources)
      File "/Users/rpg/Library/Python/3.7/lib/python/site-packages/nbconvert/exporters/exporter.py", line 315, in _preprocess
        nbc, resc = preprocessor(nbc, resc)
      File "/Users/rpg/Library/Python/3.7/lib/python/site-packages/nbconvert/preprocessors/base.py", line 47, in __call__
        return self.preprocess(nb, resources)
      File "/Users/rpg/Library/Python/3.7/lib/python/site-packages/nbconvert/preprocessors/base.py", line 69, in preprocess
        nb.cells[index], resources = self.preprocess_cell(cell, resources, index)
      File "/Users/rpg/Library/Python/3.7/lib/python/site-packages/nbconvert/preprocessors/extractoutput.py", line 92, in preprocess_cell
        data = json.dumps(data)
      File "/usr/local/Cellar/python/3.7.3/Frameworks/Python.framework/Versions/3.7/lib/python3.7/json/__init__.py", line 231, in dumps
        return _default_encoder.encode(obj)
      File "/usr/local/Cellar/python/3.7.3/Frameworks/Python.framework/Versions/3.7/lib/python3.7/json/encoder.py", line 199, in encode
        chunks = self.iterencode(o, _one_shot=True)
      File "/usr/local/Cellar/python/3.7.3/Frameworks/Python.framework/Versions/3.7/lib/python3.7/json/encoder.py", line 257, in iterencode
        return _iterencode(o, 0)
      File "/usr/local/Cellar/python/3.7.3/Frameworks/Python.framework/Versions/3.7/lib/python3.7/json/encoder.py", line 179, in default
        raise TypeError(f'Object of type {o.__class__.__name__} '
    TypeError: Object of type bytes is not JSON serializable
[W 12:19:03.775 NotebookApp] 500 GET /nbconvert/pdf/Continuous%20model.ipynb?download=true (::1): nbconvert failed: Object of type bytes is not JSON serializable
[E 12:19:03.776 NotebookApp] {
      "Host": "localhost:8888",
      "Connection": "keep-alive",
      "Upgrade-Insecure-Requests": "1",
      "Dnt": "1",
      "User-Agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_4) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/74.0.3729.131 Safari/537.36",
      "Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3",
      "Referer": "http://localhost:8888/notebooks/Continuous%20model.ipynb",
      "Accept-Encoding": "gzip, deflate, br",
      "Accept-Language": "en-US,en;q=0.9",
      "Cookie": "username-localhost-8889=\"2|1:0|10:1557094549|23:username-localhost-8889|44:MTlkNjYxMmI5ZDdkNGJkYWFlMzJmYzcxMmM1YTUzNzI=|e07c4f8ee41d84dc25632f038c8dae0255876174ee09ae08e42cf547255e593d\"; username-localhost-8890=\"2|1:0|10:1557266878|23:username-localhost-8890|44:YjhkODllOWU2YzYwNGVlYzhmMzZjNjFiNzU1ZjRjMGQ=|b66a62bc2ccb2f3ee8535897c014aa6e4a9fa7d76b4070bcc70d60fd512593f0\"; username-localhost-8888=\"2|1:0|10:1558122572|23:username-localhost-8888|44:YjE2NTlhZDQ2NTJiNDc1MWJlMmI2ZmQ4MjY4NzM3OTI=|00edcbb85aacc7114ec06cb67cdba903843c640ad6e4831008eed32163e0c550\"; _xsrf=2|4b836cd5|6530c5b3448667e1abd65d4d9aa83125|1558716658"
    }

[rpgoldman]

Update: this occurred for a notebook with an embedded graphic generated by graphviz. When I cleared the cell containing that graphic, the error went away. I'm afraid I don't know what graphviz puts in an image in a notebook, so I don't have any idea how to repair this.

[MSeal]

Can you post the minimal notebook to reproduce? (might have to change the extension to post on the issue)

[rpgoldman]

@MSeal I'm attaching one that is minimal, but not runnable -- it has a huge dependency tail. But you should be able to open it and test the download as PDF, and if you see the same error as I do, should be able to see what it is in the file that triggers it (and reopen this issue?). If that isn't enough to diagnose the issue, let me know further, and I will try to get something more minimal. It's tricky, though, because I think it's a graphviz issue and I don't ever use graphviz directly, only through PyMC3. nbconvert-graphviz-out-error.ipynb.txt

[SpencerPark]

I was able to convert your example notebook @rpgoldman, maybe double check that you are indeed using nbconvert 5.5.0? I used

$ pip show nbconvert
Name: nbconvert
Version: 5.5.0
...

It's possible that you have 5.5.0 installed but the server is not picking that one up. Double check the nbconvert version in the shell you are using to run the jupyter server.

I ran jupyter nbconvert nbconvert-graphviz-out-error.ipynb --to latex and then pdflatex nbconvert-graphviz-out-error.tex -interaction=nonstopmode to get the pdf.

[rpgoldman]

@SpencerPark Sorry -- you were quite right: I was having a problem with my virtual environment so that the nbconvert that I was looking at, and the one that was being run by jupyter were two different things. Thanks for testing that.

I was able to convert your example notebook @rpgoldman, maybe double check that you are indeed using nbconvert 5.5.0? I used

$ pip show nbconvert
Name: nbconvert
Version: 5.5.0
...

It's possible that you have 5.5.0 installed but the server is not picking that one up. Double check the nbconvert version in the shell you are using to run the jupyter server.

I ran jupyter nbconvert nbconvert-graphviz-out-error.ipynb --to latex and then pdflatex nbconvert-graphviz-out-error.tex -interaction=nonstopmode to get the pdf.