"KeyError: 'state'" when trying to convert a notebook from Google Colab

[mrdbourke]

Hello,

I seem to every so often get the following error:

File "/opt/hostedtoolcache/Python/3.10.2/x64/lib/python3.10/site-packages/nbconvert/filters/widgetsdatatypefilter.py", line 56, in __call__
[95](https://github.com/mrdbourke/pytorch-deep-learning/runs/5597148101?check_suite_focus=true#step:8:95)
    metadata['widgets'][WIDGET_STATE_MIMETYPE]['state'] if
[96](https://github.com/mrdbourke/pytorch-deep-learning/runs/5597148101?check_suite_focus=true#step:8:96)
KeyError: 'state'
[97](https://github.com/mrdbourke/pytorch-deep-learning/runs/5597148101?check_suite_focus=true#step:8:97)
Error: Process completed with exit code 1.

It seems to be most often when I'm saving a notebook from Google Colab into GitHub and then running a GitHub Action to convert it directly into documentation.

The GitHub Action traceback can be seen here: https://github.com/mrdbourke/pytorch-deep-learning/runs/5597148101?check_suite_focus=true

The notebook in question can be found here: https://github.com/mrdbourke/pytorch-deep-learning/blob/main/03_pytorch_computer_vision.ipynb

The issue doesn't occur on other notebooks converted in the same manner.

Things I've tried:

• Make a copy of the notebook, resave under same name (in Google Colab)
• Clone the notebook locally, rerun, save, commit

Both of these result in the same result as above.

Any help would be greatly appreciated, I'm not quite sure where this error is coming from nor why it happens to some notebooks and not others.

To the best of my knowledge, the notebook it happens on "03_pytorch_computer_vision.ipynb" uses the exact same style as other notebooks (Python code and markdown).

[YIREN1]

seeing the same issue when saving using colab and then convert into html/pdf

[pplonski]

The same here! We are using nbconvert in our framework Mercury to add widgets to notebooks (to make notebooks as interactive web apps). What is more interesting the notebook is not using widgets at all!

Some workaround might be a removal of metadata/widgets from notebook JSON.

[scienception]

Same issue, did you manage to solve it?

[pplonski]

Remove metadata/widgets in notebook file.

[alimanfoo]

Just wondering if anyone has raised this with the colab devs? Looks like they need to update the way that widget state is stored in the notebook JSON.

[hs211216]

Just hit with https://colab.research.google.com/github/kstathou/vector_engine/blob/master/notebooks/001_vector_search.ipynb

The workaround recommended by @pplonski worked for me. Here is what I did

1. Download 001_vector_search.ipynb with Colab gui

2. Fail to convert it with nbconvert

% jupyter nbconvert --to html 001_vector_search.ipynb
  [NbConvertApp] Converting notebook 001_vector_search.ipynb to html
  Traceback (most recent call last):
    File "/usr/local/bin/jupyter-nbconvert", line 8, in <module>
      sys.exit(main())
    File "/usr/local/lib/python3.9/site-packages/jupyter_core/application.py", line 264, in launch_instance
      return super(JupyterApp, cls).launch_instance(argv=argv, **kwargs)
    ...
    File "/usr/local/share/jupyter/nbconvert/templates/base/display_priority.j2", line 7, in block 'data_priority'
      {%- for type in output.data | filter_data_type -%}
    File "/usr/local/lib/python3.9/site-packages/nbconvert/filters/widgetsdatatypefilter.py", line 57, in __call__
      metadata["widgets"][WIDGET_STATE_MIMETYPE]["state"]
  KeyError: 'state'

3. Remove .metadata.widgets with Jq

% jq -M 'del(.metadata.widgets)' 001_vector_search.ipynb > 001_vector_search.fixed.ipynb
% jupyter nbconvert --to html 001_vector_search.fixed.ipynb
  [NbConvertApp] Converting notebook 001_vector_search.fixed.ipynb to html
  [NbConvertApp] Writing 629781 bytes to 001_vector_search.fixed.html

% open 001_vector_search.fixed.html

  ... output looks good

For reference I'm on OSX 10.15.7 and here are my jupyter versions

% jupyter --version
  Selected Jupyter core packages...
  IPython          : 8.0.1
  ipykernel        : 6.9.0
  ipywidgets       : not installed
  jupyter_client   : 7.1.2
  jupyter_core     : 4.9.1
  Jupyter_server   : not installed
  jupyterlab       : not installed
  nbclient         : 0.6.4
  nbconvert        : 6.5.0
  nbformat         : 5.4.0
  notebook         : not installed
  qtconsole        : not installed
  traitlets        : 5.2.2

[Pqlet]

Remove metadata/widgets in notebook file.

Thanks

"Widgets" --> "Clear Notebook Widgets State" helped me in jupyter

[ndjhartman]

Dumb workaround for without access to CLI jq and without reuploading to Colab:

1. Open .ipynb file in notepad
2. Copy file JSON contents
3. Go to https://jqplay.org/
4. Paste contents into JSON field
5. Copy/paste del(.metadata.widgets) into filter pane (Thanks @hs211216)
6. Copy result from right Result field
7. Paste into original .ipynb file
8. Save file and exit

Works now.

[jameswcoquat]

Clear Notebook Widget State at the top of the notebook, under Widgets, FTW!

[Xu-Justin]

Notes: If you are unable to locate the Widget tab on your notebook, which happened to me as well, you will need to install ipywidgets by executing the following command and restart your notebook.

$ pip install ipywidgets

[tsoahii]

I first cleared the widgets section in the metadata fully (use e.g. Notepad as raw editor, the widget section is at the end of file): "widgets": {}

Then ran and saved the notebook from a jupyter server, which I normally don't do because I edit and run from vscode. After that the widget field looked like this: "widgets": { "application/vnd.jupyter.widget-state+json": { "state": {}, "version_major": 2, "version_minor": 0 } }

With this new notebook metadata, I was able to export without errors.

[mrdbourke]

Remove metadata/widgets in notebook file.

Thanks

"Widgets" --> "Clear Notebook Widgets State" helped me in jupyter

This helped me too!

--

Note: This requires a traditional notebook layout (e.g. Jupyter Notebook Classic).

See:

[hlin863]

Remove metadata/widgets in notebook file.

Thanks "Widgets" --> "Clear Notebook Widgets State" helped me in jupyter

This helped me too!

--

Note: This requires a traditional notebook layout (e.g. Jupyter Notebook Classic).

See:

I don't see that option for clearing widget stats on my Google collab. Would you like to clarify how to get that tab?

[mrdbourke]

Hi @hlin863, as far as I know the option isn't available in Google Colab.

I accessed it via using the original Jupyter Notebook.

For example, going to terminal and typing jupyter notebook.

[mrdbourke]

Remove metadata/widgets in notebook file.

Thanks "Widgets" --> "Clear Notebook Widgets State" helped me in jupyter

This helped me too!

--

Note: This requires a traditional notebook layout (e.g. Jupyter Notebook Classic).

See:

Update 2 Feb 2024:

You may need to install nbclassic to the get the widget item in the menu bar, see: https://github.com/jupyter/nbclassic

Install:

pip install nbclassic

Run:

jupyter nbclassic

Otherwise clearing the widget metadata as discussed above may work too, however, I'm not 100% sure.

[nfultz]

I hit this bug today, here was my workaround using jq to delete the widgets block from the shell, maybe someone else finds that helpful:

! jq  "del(.metadata.widgets)" "/content/drive/MyDrive/Colab Notebooks/etl-3.ipynb" > /tmp/test.ipynb

[mrdbourke]

Just hit with https://colab.research.google.com/github/kstathou/vector_engine/blob/master/notebooks/001_vector_search.ipynb

The workaround recommended by @pplonski worked for me. Here is what I did

1. Download 001_vector_search.ipynb with Colab gui
2. Fail to convert it with nbconvert

% jupyter nbconvert --to html 001_vector_search.ipynb
  [NbConvertApp] Converting notebook 001_vector_search.ipynb to html
  Traceback (most recent call last):
    File "/usr/local/bin/jupyter-nbconvert", line 8, in <module>
      sys.exit(main())
    File "/usr/local/lib/python3.9/site-packages/jupyter_core/application.py", line 264, in launch_instance
      return super(JupyterApp, cls).launch_instance(argv=argv, **kwargs)
    ...
    File "/usr/local/share/jupyter/nbconvert/templates/base/display_priority.j2", line 7, in block 'data_priority'
      {%- for type in output.data | filter_data_type -%}
    File "/usr/local/lib/python3.9/site-packages/nbconvert/filters/widgetsdatatypefilter.py", line 57, in __call__
      metadata["widgets"][WIDGET_STATE_MIMETYPE]["state"]
  KeyError: 'state'

3. Remove .metadata.widgets with Jq

% jq -M 'del(.metadata.widgets)' 001_vector_search.ipynb > 001_vector_search.fixed.ipynb
% jupyter nbconvert --to html 001_vector_search.fixed.ipynb
  [NbConvertApp] Converting notebook 001_vector_search.fixed.ipynb to html
  [NbConvertApp] Writing 629781 bytes to 001_vector_search.fixed.html

% open 001_vector_search.fixed.html

  ... output looks good

For reference I'm on OSX 10.15.7 and here are my jupyter versions

% jupyter --version
  Selected Jupyter core packages...
  IPython          : 8.0.1
  ipykernel        : 6.9.0
  ipywidgets       : not installed
  jupyter_client   : 7.1.2
  jupyter_core     : 4.9.1
  Jupyter_server   : not installed
  jupyterlab       : not installed
  nbclient         : 0.6.4
  nbconvert        : 6.5.0
  nbformat         : 5.4.0
  notebook         : not installed
  qtconsole        : not installed
  traitlets        : 5.2.2

Slight modification for this command to work in-place.

Because jq requires saving to another file, you can't just do notebook.ipynb > notebook.ipynb.

You can save to temp and then then mv to original filename.

jq -M 'del(.metadata.widgets)' NOTEBOOK_NAME.ipynb > NOTEBOOK_NAME.temp.ipynb && mv NOTEBOOK_NAME.temp.ipynb NOTEBOOK_NAME.ipynb

[ulisesrey]

If it helps anyone, I usually use Jupyterlab instead of Jupyternotebook. I did not find the widgets tab on Jupyterlab, even after installing

pip install ipywidgets

But I could find it in JupyterNotebook and then I did it from there.