DISC: Workflows using make jupyter (New make ipynb)

[mmcky]

Currently make jupyter will execute everything in the conf.py file which may include execution control, and html generation. But it would be nice to have the ability to just generate notebooks for editing and writing that doesn't have all the additional steps if the extension is used to generate websites.

So I think we need to introduce:

make ipynb or make notebook that specialises on generating a local collection of notebooks. So we will need to think about which options are parsed and which are ignored and make this VERY clear in the documentation. I think this is useful for editing and will be used heavily.

make jupyter will then respect the conf.py file as it does currently in line with typical sphinx behaviour.

I am reluctant to use make jupyter-notebook as it is long to type and it would be easy to just use make jupyter

@AakashGfude when you're free -- lets catchup sometime to discuss this.

[mmcky]

Summary of Entry Points to sphinxcontrib-jupyter:

1. make ipynb: generates notebook files only (ignoring all extended options)
2. make jupyter: respects the conf.py and is a fully configurable option with all support of execution, website generation (download notebooks sets and conversion to html), and coverage.

We could also add the following for running those stages of the pipeline (and any dependent ones before them) but I don't think they would be used a great deal except perhaps make jupyter-coverage for generating coverage reports as an independent process.

1. make jupyter-execute
2. make jupyter-html
3. make jupyter-coverage

[arnavs]

I think this is a great idea (and we had some initial .sh files to do this on the Julia side). Few thoughts:

• I think it would be great to have an execution option in between make ipynb (execute nothing) and make jupyter (execute everything.) Maybe attaching flags like -n name1 name2 -d dir1 dir2, where names are lecture names, and dirs are whole directories (e.g., tools_and_techniques/, or dynamic_programming_squared/.)

• While we're at it, a make rst-check or something which calls out to an RST linter would be very useful. I could do some playing around, but the important thing would be to make sure that links, tables, math, and code blocks are well-formatted. And enforce QuantEcon styling, like no periods at the ends of sentences, sentences followed by line-breaks, etc.

This was a big pain point while we were doing the Julia rewriting.

[arnavs]

• Regarding the above, we could eventually build language-specific linting rules into a make rst-check, depending on options in the conf.py. For example, regarding spaces in function calls, etc. More of a long-term goal, though.

Perhaps something like this already exists in the Python side? With the Julia, especially since we had a relatively large and rotating set of RAs, this would've also been a huge help.

[jstac]

@mmcky and I discussed this yesterday, but I have a different perspective to you guys. I'd like to keep the interface very simple and clear, so that it's easy for new users to understand.

For example, make ipynb and make jupyter might lead to confusion as to what their roles are. We should follow the philosophy that basic operation is clear to an intelligent user who meets the system for the first time and isn't trying to do something fancy, without having to read the manual.

How about, for the moment, we stick to

• make jupyter -- convert rsts to ipynbs but don't execute
• make website -- build the whole website
• make pdfs -- build the pdfs (this is in work on the Python side and we'll try to extend to all adopters)
• make all -- build everything

Then we can look at adding make rst-check later if we do add such functionality.

If we need more fine grained control, then we can add flags, as @arnavs suggested. But only when there's a clear use case and no other easy way to do it.

[arnavs]

Thanks @jstac, what you wrote makes sense to me.

Would we then have a make execute to do the ipynbs with the execution step (but no HTML, PDFs, etc.)?

[jstac]

Good question @arnavs

I guess my idea is to keep the number of make commands small and for the use of each one in standard cases to be immediately clear from the name. Anything else is done by adding flags to the basic commands --- such as the four I listed.

The make website command would, I suppose, regenerate ipynb files from those rst files that have changed, execute them and build html. We could add a -- refresh flag or something similar to override caching.

I'm thinking from the perspective of an average user who writes content, which is what I am. So I want a simple interface. But I don't know everything that you guys need to do. So I suggest we start with those four basic make commands I listed above and then you give me a scenario where they would be insufficient. Then we can have a debate as to whether the extra functionality is really worth cluttering the user interface. For example, do we need a make execute when we can just make website, which will also execute. Yes, it does a few other things, but they are almost instantaneous, so who cares? That's my thinking, but I might be wrong, so feel free to tell me. But let's start with a minimal and uncluttered user interface, and then build incrementally if we're all convinced it's necessary.

[jstac]

A small issue: due to an unfortunate bifurcation and my own mistakes, in the sandpit right now make notebooks is a synonym for make jupyter. Could we please preserve this, even thought it shouldn't have happened, so as not to break Tom's workflow?

[arnavs]

Roger that @jstac, thanks for clarifying. This makes sense.

(The main reason we were interested in make execute is because of difficulties assembling the site, e.g. with HTML templates, but it's likely these will evaporate as we iron out the build process.)

[jstac]

I understand, and I don't want to get in the way of you power users :-)

I just have to take the perspective of the average Joe, not the developers, when I think about the interface. So it's great to have this conversation, where we try to balance out competing needs.

[jstac]

I forgot that we need make preview and make setup, so the six basic ones I can think of now are

• make setup
• make jupyter / make notebooks
• make preview
• make website
• make pdfs
• make all

These can then be specialized with flags. How does that sound to everyone?

[mmcky]

thanks @jstac for summarizing the user interface (through the Makefile).

with regards to --refresh this is really catered for through make clean which is a standard approach to a new build in sphinx

[jstac]

Right, we should have make clean too. Thanks @mmcky, good call.

[mmcky]

This is implemented at the source repository level rather than here in the extension.

Updates to jupinx-quickstart:

• [x] @AakashGfude perhaps we should add website, preview, notebooks to the quickstart Makefile
• [x] we should add pdf only once v0.5 is released

[mmcky]

I think this issue can now be closed with the release and management of jupinx. Interfaces between jupinx and Makefile will need to be maintained (as per this issue) until jupinx interfaces directly with sphinx-build.