Support Jupytext (MyST-Markdown flavoured) notebooks in `NotebookLite`, `JupyterLite`, and `Voici` directives

[agriyakhetarpal]

Problem

jupyterlite-sphinx does not yet support Jupytext (Markdown) notebooks in its directives. Markdown notebooks are a wonderful solution to existing notebook tooling designed for IPyNB notebooks, which are hard to conquer for traditional version control systems like Git. One of the advantages of notebooks as text-based formats is that they pair up nicely with Markdown-based documentation tooling along with Sphinx and are supported for execution and for other standards in the ecosystem as well.

Proposed Solution

This feature request is copied over from the proposal added in https://github.com/jupyterlite/jupyterlite-sphinx/pull/180#issuecomment-2192628783 as a corollary to the functionality provided and now available since v0.16.0 (release):

The Jupyter extension, Jupytext, provides an API in addition to its CLI. Another worthwhile feature to add to jupyterlite-sphinx would be to allow the .. notebooklite::, ..jupyterlite::, and the ..voici:: directives to accept text-based Markdown notebooks – jupytext as a dependency can then be used with its API to convert the notebook from .md at jupyterlite-sphinx's runtime (maybe even by default, and not opt-in), and then pass/copy it as .ipynb to JupyterLite.

It is to be noted that the JupyterLite interface does have support for Jupytext, but I am not sure if the Markdown notebook is currently runnable – it is not supported right now, based on this discussion: https://github.com/jupyterlite/jupyterlite/discussions/731, so this conversion from .md to .ipynb at build time could be a reasonable temporary solution for notebook authors until JupyterLite itself can gain support. We can instruct users in the documentation and let them be aware through the means of an admonition that their notebooks are available in the JupyterLite interface as IPyNB, even if they pass them in their directives as Markdown. Some use cases are available in the "Additional context" section below.

Additional context

• jupyterlite-sphinx is increasingly used in Scientific Python documentation, as are Markdown notebooks. This feature can potentially resolve the problem of having extra targets/phonies in the docs Makefiles implemented in https://github.com/scipy/scipy/pull/20303, providing advantageous cleanup; and
• this also removes the addition of a custom Sphinx extension connecting to the config-inited event in https://github.com/PyWavelets/pywt/pull/741, which should also help make the docs directory cleaner at the time of rebuilds, since Markdown-based notebooks will subsequently not be required to be paired with their IPyNB counterparts (which may have been git-ignored or excluded from inclusion in Sphinx).
• Jupytext has an emphasis on round-trip conversion, which means that there is no potential loss of metadata (and data) when converting notebooks from one format to another (and back) – which can be used as a motivator towards this feature, even if it might be a temporary ask.

cc @melissawm and @steppi

[agriyakhetarpal]

Also, since jupyterlite-sphinx has not had a v1 release yet, we could make this an experimental feature by mentioning it as such in the documentation and enabling it only via a jupyterlite_experimental boolean config value in conf.py – similar to how scikit-build-core enables experimental features in pyproject.toml, therefore making this opt-in instead of opt-out in the meantime while Jupytext support matures.