Migration of Lectures to MyST / Jupyter-Book

[mmcky]

This Issue documents what needs to be done to migrate our lectures over to use the new jupyter-book platform. The current system builds myst well -- but we will need to add some site generation / assembly features before we can migrate our projects over to use the new platform.

• [ ] Build a sphinx extension for site assembly that coordinates various builders may be needed (i.e. sphinx-ebp-website) See jupyter-book issue

We will focus on lecture-python-programming

---

Missing Features:

• [x] Generation of downloadable jupyter notebooks with contextual link updates to a web site.

  Strategy: adapt sphinxcontrib-jupyter to build ipynb files, Alternative: Explore the use of jupytext for mirrored notebook sets if filters are available to update myst links. Discussion: The main issue with using jupytext is that it doesn't support contextual updating such as adding web links in place of direct notebook links when generating notebooks. This contextual substitution is needed when building our website so notebooks contain links back to the web location rather than a local file reference etc. Raise this as an issue on the jupyter-book repo to get feedback

• [x] Support for theme hooks :class: hide-output and :class: collapse for .. code-block::

  Strategy: Support in quantecon theme for lectures for html. Look at latex adjustments from the class attributes through LaTeX theme support (cc: @AakashGfude work on LaTeX theme) Discussion: This needs support in any theme that has been developed. Currently these are added as class attributes to the nodes in the AST which then get interpreted by the translator. In the case of html this is a simple class passthrough that theme can latch onto and use, In case case of latex supported class attributes need to be added to the writer/translator -- an alternative is to introduce a :latex: attribute to code-block nodes to support special latex features.

• [x] Support for coverage report building and statistics from jupyter-cache

  Stategy: Work with jupyter-cache to output execution statistics through an amendment process to a json file including date, execution time, and execution status Alternative: Build a jupyter-coverage package to provide execution statistics Discussion: Building as a separate package could allow the freedom to build a tool specialised on building execution statistics independent of the caching layer and include test runs in environments. Added a feature request on jupyter_cache. The information is now available in the sphinx.env so now we just need to write a report writer for myst_nb.

  • [ ] write report generator in myst_nb based on info available in sphinx.env

Setup support for chapter level pdf files:

• [x] Support for generating --singlepagepdf files for download pdf files. See jupyter-book PR
• [x] setup LaTeX theme to support features available in current latex writer in sphinxcontrib-jupyter such as In and Out block markers, fonts etc.

---

New Infrastructure:

1. Development of a new quantecon theme that is used with jupyter-book platform

[mmcky]

We have a number of avenues underway to address these in preparation for migration.

Trial Migration:

lecture-python-programming.myst has been set up to test a migration. Saya is helping with the manual components to see how we can improve sphinxcontrib-tomyst and what will need to be done manually.

Theme Development:

A theme repository has been setup with similar structure to sphinx-book-theme in preparation for quantecon-book-theme for jupyter-book.

• [ ] @DrDrij to populate with current draft styling elements.

Timeline: DRAFT due 14th September 2020

Coverage / Badge Support:

@AakashGfude is currently working with sphinx-book-theme to see how we can integrate status badges from the exeuction statistics being produced by jupyter-cache

Timeline: TBD

Fixes for Numbered HTML Headers:

Some reference code has been found to support part in html output. Work is underway to update it from sphinx=1.5 to sphinx=3+ to see if that will be a fruitful avenue to fix the current issues with numbering and get closer harmony between html and latex outputs.

Work is contained in sphinx-parttoc-numbering which will hopefully serve as a sphinx extension.

Planned Support Post Deployments:

• Contextually updated Notebooks to support Downloads (for image and links referencing the website)
• Single page LaTeX pdf files to support downloads (to replace the javascript approach to be used for initial migration)

[mmcky]

Update: 14-Sep-2020

Initial Migration does not require:

1. Status Badges and Execution Stats (but nice to develop for EBP)
2. Remove Colab support (focus on jupyter-book execution options) and perhaps reintroduce support later on.
3. PDF (individual files built by LaTeX) can be supported via Javascript scraping on launch and migrated to LaTeX files later on.
4. Support "code-only" downloadable notebooks for now (if no simple fix for contextual link updates)

[mmcky]

This issue is old and can be closed. Our operations have fully migrated to Jupyter-book and myst including download notebooks that are constructed using sphinx-tojupyter