search

drivendataorg / nbautoexport

Automatically export Jupyter notebooks to various file formats (.py, .html, and more) on save.
https://nbautoexport.drivendata.org/
MIT License
72 stars 9 forks source link

Export works with Jupyter Notebook but not JupyterLab #81

Open mjschlauch opened 3 years ago

mjschlauch commented 3 years ago

Description

I newly installed and configured nbautoexport. New/modified jupyter notebooks in the configured directory are not getting exported as scripts on save when using Jupyter Lab.

What I Did

I opened the same notebooks using Jupyter Notebook and they are exported as scripts on save. Interestingly, when I then go back to using Jupyter Lab, things are working as they should (new/modified notebooks are exported on save).

Interim workaround

As of now, nbautoexport does not work with jupyterlab when the version of nbclassic is 4.0 or greater. To get both jupyterlab and jupyter notebook to work, run pip install nbclassic=0.3.7 (earlier 0.3 versions also work)

Long term fix

Write out to ~/.jupyter/jupyter_server_config.py in addition to ~/.jupyter/jupyter_notebook_config.py. After running cp ~/.jupyter/jupyter_notebook_config.py ~/.jupyter/jupyter_server_config.py, both jupyter lab and jupyter notebook export scripts correctly regardless of the version of nbclassic

natl commented 2 years ago

In case anyone else finds there way here, it could be worth confirming that the notebook metadata contains the following fields under language_info in addition to anything else already there:

"metadata": {
  "language_info": {
   "file_extension": ".py",
   "nbconvert_exporter": "python"
  }

I found ensuring these elements were in the metadata helped nbautoexport to work as I expected - one of my environments wasn't creating these automatically.

klwetstone commented 2 years ago

@mjschlauch FYI I added some takeaways from a debug session to the issue description. The culprit was a new release of nbclassic, and the quick workaround is installing a version of nbclassic < 0.4

jayqi commented 2 years ago

@klwetstone's additional notes helped point me in a direction of research on the very confusing Jupyter ecosystem. I will try to summarize some key ideas here:

There are actually effectively three UI applications in the Jupyter ecosystem that let people run and interact with "Jupyter notebooks" (i.e., ipynb files).

  1. Jupyter Notebook—this is the original legacy application that predated JupyterLab
  2. JupyterLab
  3. Jupyter nbclassic—this is the original Jupyter Notebook UI but running on top of JupyterLab

One reason this is confusing is because there's been big decisions about the technology stack behind Jupyter applications that have been made over the years, that ended up getting reversed. The key ideas that I understand are:

It sounds like all three of these are expected to be normal entrypoints to Jupyter notebooks going forward, which means we should make an effort to explicitly support all of them.

With that in mind, it seems like here is why stuff hasn't been working.

Our code is written in a way (write to jupyter_notebook_config.py) that is the way things worked for Jupyter Notebook. This still works for Jupyter Notebook (currently v6, and earlier).

According to the jupyter-server docs, this should be instead jupyter_server_config.py for any app that uses jupyter-server as a backend. It seems like, though not totally certain, that some part of JupyterLab v2 -> v3 using jupyter-server is related to nbautoexport not working anymore, which would be explained by this fact.

Some additional complication though is that nbclassic has the functionality that it works with both jupyter_notebook_config.py and jupyter_server_config.py. Apparently, that functionality broke recently, which is the issue that @klwetstone found. But it sounds like the intent is that it will continue to support jupyter_notebook_config.py? There's also some complication that nbclassic contains some shim-like behavior that adapts JupyterLab to work with old Jupyter Notebook interfaces.

It looks like eventually even Jupyter Notebook v7 will migrate to jupyter_server_config.py, based on the documentation for it.

Here's a StackOverflow post that also covers some of the mechanics about the different backend servers.

Anyways

So I think the fix is definitely to start writing the nbautoexport code to jupyter_server_config.py.

There's a question about how we want to support current and older versions of Jupyter Notebook, which still only read jupyter_notebook_config.py and not jupyter_server_config.py (I just tested this). Jupyter Notebook v7 is in pre-release but is not out yet.

  1. Should we write out to both files on nbautoexport install?
    • This will work automagically for all users, but also everyone will end up with two files, which will be messy and potentially confusing for future debugging.
  2. Should we add a flag to write out to jupyter_notebook_config.py only when the flag is used when running nbautoexport install?
    • This is more explicit, less messy, and makes future debugging easier, but adds some friction for Jupyter Notebook users to get started.
pjbull commented 2 years ago

Fascinating read.

Personally, I'd vote for 1, since I use classic notebooks pretty often. May be worth it to switch this line to an info and print the path to the config as well for debugging ease: https://github.com/drivendataorg/nbautoexport/blob/eb5937d2559ad1ede6edbc6ba4d25ee673bf8237/nbautoexport/jupyter_config.py#L23

ejm714 commented 2 years ago

Looks like the PR to fix the nbclassic issue got merged in (https://github.com/jupyter/nbclassic/pull/137) which, if I'm understanding thing correctly, fixes the bug that @klwetstone originally found.