[Feature]: Convert sphinx gallery to jupyter notebooks

[rly]

What would you like to see added to PyNWB?

You can render notebooks in Sphinx docs and keep the sphinx gallery thumbnail feature: https://docs.readthedocs.io/en/stable/guides/jupyter.html

This makes it possible for users to download/copy a tutorial notebook, run it locally, and have the text formatting not be terrible.

However, you lose the :ref: directive and other automagic rst/sphinx features. So we would have to manually link to API doc pages. There's an upfront cost to that. But I think it's worth it. With another package, we can run notebooks in pytest as well.

Is your feature request related to a problem?

No response

What solution would you like?

Convert current sphinx gallery examples to jupyter notebooks.

Do you have any interest in helping implement the feature?

Yes.

Code of Conduct

• [X] I agree to follow this project's Code of Conduct
• [X] Have you checked the Contributing document?
• [X] Have you ensured this change was not already requested?

[bendichter]

I don't even think there would be that big of an up-front cost if we copy directly from the rendered HTML. As long as the API docs are pretty stable and we keep the link checkers active this approach should work. I guess the only remaining question is about the aesthetics of our current approach vs notebooks. I like the look of our current tutorials, but I also see the value in making them look like jupyter notebooks as that will be familiar to most users and will make it easier for them to transition from reading the tutorial to running the code themselves

[rly]

An example of how this would look: https://nbsphinx.readthedocs.io/en/0.9.2/code-cells.html

I wonder if we could even show interactive NWB widgets that way. That seems too good to be true.

I agree that the current sphinx gallery style is nice. We can make the nbsphinx output a little less bland with themes like this https://shibuya.lepture.com/extensions/nbsphinx/

[oruebel]

This makes it possible for users to download/copy a tutorial notebook, run it locally, and have the text formatting not be terrible.

Can you clarify what the issue is that you are trying to solve. We can already download the tutorials as notebooks. Is the main issue that the styling of the notebooks when converted from sphinx to notebooks is not "pretty"? Are there other issues that you are trying to address here? If styling of the notebooks is the main issue, then an alternate approach may be to enhance the conversion of sphinx gallery to notebooks.

I wonder if we could even show interactive NWB widgets that way

I doubt it, because those require python to run, and these seem to be still static pages. I.e., something like bokeh might work (because it renders HTML), but anything that requires Python to run, I don't think would work.

However, you lose the :ref: directive and other automagic rst/sphinx features. So we would have to manually link to API doc pages. There's an upfront cost to that. But I think it's worth it. With another package, we can run notebooks in pytest as well.

I think the issue here is less the upfront cost as it is a maintenance issue and additional burden on the tutorial creator to figure out the correct links to the docs. If references changes then those are automatically updated when we rebuild or sphinx will warn us, so we can keep them up-to-date. Also, I'm not sure if the linkchecker works for notebooks.

[rly]

Can you clarify what the issue is that you are trying to solve.

This is definitely a minor issue.

When we distributed jupyter notebooks as part of our tutorials, the main workflow of most new users who are learning NWB that I have seen is:

1. copy the notebook
2. follow along the live tutorial with their copied notebook
3. make changes to the notebook for their data

This has the small benefit of having all the explanations for their code next to their code.

When working off of readthedocs, they:

1. create a new notebook (or script)
2. copy individual code blocks from our tutorial to the new notebook while they watch/read through the tutorial
3. make changes to the notebook for their data

This is an OK workflow, but is more work and is harder to follow along with the live tutorial.

Rarely have I seen anyone download the jupyter notebook from the tutorial page. We could make that feature more prominent... But even if we do, the formatting of the jupyter notebook has a lot of broken sphinx links (see below), and I worry that they will not use it because the formatting looks broken and doesn't match what they see on screen as we go through the tutorial. I don't have any evidence that they won't use it though, so maybe this is a non-issue and I am being nitpicky...

In short, if the tutorial notebook is more pretty and matches our presentation, then when we encourage new users to use it, they might be more likely to use it as the basis of their conversion scripts / learning process.

an alternate approach may be to enhance the conversion of sphinx gallery to notebooks.

That would be great but looks difficult. We could manually replace internal links with links using absolute paths to the published readthedocs. That would solve most of the issues. The link checker would catch broken links.

[rly]

A related idea is to transition our docs to use MyST.

Myst-NB underlies Jupyter Executable Books, which the OpenScope team uses for their data book (notebook file, website). MyST seems mature and well-supported now, and it is easier to write markdown and jupyter notebooks as well as maintain them.

[bendichter]

+1 for MyST and MyST-NB. This looks like a great solution.