Example Dockerfiles for jupyter-book

[phaustin]

I'm wondering if Dockerfiles exist that would build a turn-key image containing a working jupyterhub with a sample executable book for demonstration/load testing etc. Specfically, we need to test and extend the climlab book by @brian-rose, and I'd like to do it on a macbook pro using docker run. How do other people develop and test the executable portion of an executable book?

[choldgraf]

Is there a reason that you'd need a JupyterHub inside the image? Or do you just want a notebook environment that could run the Jupyter Book's contents?

What I'd probably recommend is to use repo2docker to build the image, and structure your repository with the right configuration files that they'd result in the right environment for the book's content

[phaustin]

I think I know how to do that with a standard jupyterhub, but I'm less certain about how it would work with thebelab.

[choldgraf]

ohhh you want to test thebelab functionality for your book?

[phaustin]

well, more I'd like to see whether it added something significant as far as the students experience was concerned.

[choldgraf]

makes sense - I have less experience w/ Thebelab that isn't running off of Binder but would love to explore this more. Let me know what your experience there is like

[brian-rose]

Following!

I develop and test my executable content manually in a local conda environment. It's laborious though, and the book would definitely benefit from a streamlined CI service that would catch problems with upstream changes.

@phaustin I'd love to hear more about your plans

[chrisjsewell]

I'm very much a +1 on being able to run builds (+ notebook execution) in Docker containers 👍

[choldgraf]

can we tease apart some use-cases here? I can think of using this for the following purposes:

1. To self-contain the environment for the creation of books
2. To test out the book environment locally from a reader's perspective, that you'll then push to a docker registry
3. To integrate into your CI/CD to test your book's content as you make PRs or update things online

Am I missing something?

[phaustin]

I'd maybe add: "As an end-to-end demonstration of a jupyterhub application". We've got a local slack workspace among stats, cpsc and eosc profs trying to figure this out, and nothing is as eloquent as a complete working example.

[chrisjsewell]

Am I missing something?

sounds good to me, I guess (1) is my focus, but then I would think you would just use the same thing for (3)?

[choldgraf]

Yeah I agree (and good addition @phaustin)

I know that @betatim uses repo2docker quite a lot to develop locally and make sure that the development environment behaves as he'd expect it to. Maybe he's written up that workflow somewhere in a way that would be useful for @phaustin ?

[chrisjsewell]

its a shame RTD doesn't allow for arbitrary containers. Certainly for https://aiida-core.readthedocs.io/, if we wanted to execute notebooks, we require a container with postgres and rabbitmq also running inside (i.e. https://hub.docker.com/r/aiidateam/aiida-core)

[choldgraf]

yeah I agree - guess we'll need to keep running circleci/gha/travis for those use-cases

[phaustin]

here's the pangeo effort that prompted me to raise the issue: https://github.com/pangeo-data/pangeo-docker-images

[choldgraf]

I wonder if we could add a command to programmatically generate a docker file for a book repo or something. That'd be awesome (or maybe repo2docker would be enough? I dunno...)

[chrisjsewell]

It would be cool to get jupyter-book or another command to actually fire up the container, build, then teardown. This is what https://molecule.readthedocs.io does for testing ansible roles, using YAML files for the configuration

[chrisjsewell]

you can then run molecule locally (1) or it runs on CI (3): https://molecule.readthedocs.io/en/latest/ci.html

[choldgraf]

oh yeah that would be really cool...@emdupre did some work to facilitate this in Jupyter Book 1.0 and I think she found it quite useful (back then it was way more complicated to run jupyter book though).

[phaustin]

@phaustin I'd love to hear more about your plans

Rachel White (another UW alum -- we're legion) and I are planning to trial a version of your course in January, as a directed studies for graduate students with a parallel undergraduate version. This is part of a 3 year project that I gave a short presentation on at the Scipy Education BOF on Tuesday: https://eoas-ubc.github.io/scipy/scipy_bof_slides.slides.html#/ (note that climlab makes an appearance as my jupyter-book exemplar on slide 4).

If that Scipy talk had gone for another 15 minutes it would have been great to do a live demo with a docker container hosted securely on my laptop. A container would also come in handy for something like our UBC Jupyter Days this August: https://ubc-dsci.github.io/jupyter_days/content/introduction.html (@tomasbeuzen is presenting jupyter-book on day 1)

[brian-rose]

Rachel White (another UW alum -- we're legion) and I are planning to trial a version of your course in January

Excellent! @phaustin at some point, I invite you to contribute something to this thread: https://github.com/brian-rose/climlab/issues/68. Keeping track of these use cases helps when I report back to NSF.

[phaustin]

For those who might be interested in this, I've got the first step working now at: https://github.com/phaustin/Problem-Solving-with-Python-37-Edition/tree/jb As the Readme says, if you type docker-compose up you'll get two containers -- one is an apache webserver that's serving _build/html/index.html on port 8500 using a bind-mounted volume and the other is a jupyter notebook server on port 9500 with a bind mount to the notebooks folder. Building the book inside the container gives you the very nice python intro by @ProfessorKazarinoff

@choldgraf -- it looks like if I wanted to actually couple the jupyter-book to the notebooks running on localhost:9500 I would need to add a new context entry to html-page-context is that correct?

@brian-rose -- up next for me is https://github.com/brian-rose/ClimateLaboratoryBook -- am I correct that it still could use a port from jekyll to sphinx, or is that happening in another repo?

[brian-rose]

@brian-rose -- up next for me is https://github.com/brian-rose/ClimateLaboratoryBook -- am I correct that it still could use a port from jekyll to sphinx, or is that happening in another repo?

@phaustin correct, there is no other repo and I haven't yet done anything about updating my material for the changes in jupyterbook.

[choldgraf]

@phaustin hmmm I am not quite sure what you mean by the question. What do you mean by "couple to the notebooks"? You mean via thebe or something?

[phaustin]

for thebe, yes, that's on my todo list -- but for right now I'd just like a launchbutton that brings up the localhost jupyterhub pointed at notebooks bind-mounted inside the container -- i.e. since I've got the files I don't need nbgitpuller here: https://github.com/executablebooks/jupyter-book/blob/master/docs/_config.yml#L22-L31

[choldgraf]

ah, gotcha, and you couldn't just set jupyterhub_url to something like localhost:XXXX? (https://jupyterbook.org/interactive/launchbuttons.html#jupyterhub-buttons-for-your-pages)

[phaustin]

hmmm -- looks like I need to back up one step. Using the current jupyter-book master on macos catalina I get a clean build of the docs, but no launchbutton:

https://phaustin.github.io/jupyter-book/intro.html

Do I need to tweak something in _config.yml to get https://jupyterbook.org/interactive/launchbuttons.html# ?

[choldgraf]

They are there, just only on pages that have ipynb as the source file

[phaustin]

got it, there they are -- so for someone who uses a jupytext pairing of ipynb,md:myst and only commits the md files to github, is the expectation that a user would run a preprocess step first to produce the ipynb files, then build the interactive book?

[chrisjsewell]

No the markdown files created from jupytext already get parsed as notebooks (assuming they have the expected metadata)

[phaustin]

great, I'll close this and open a new issue if I hit any other snags.

[choldgraf]

No the markdown files created from jupytext already get parsed as notebooks (assuming they have the expected metadata)

But, I should note, I feel like we haven't quite nailed down this pattern yet. Maybe it's just a question of documenting things well, but I still think it's not clear to users what to expect here :-/

[phaustin]

I think a couple of small png screenshots of the icons for launchbuttons, collab, github etc. below the text at https://phaustin.github.io/jupyter-book/interactive/launchbuttons.html would have helped me make the connection -- but I definitely missed the crucial detail that the launch button was appearing only on some pages.

[phaustin]

back with an update:

1) for now, I've hijacked the jupyterhub button on my localhost fork: https://github.com/executablebooks/sphinx-book-theme/compare/master...phaustin:localhost

2) Here's an example of a book built with that change: https://github.com/phaustin/Problem-Solving-with-Python-37-Edition#problem-solving-with-python-37-edition (roll over the launch logo here: https://phaustin.github.io/Problem-Solving-with-Python-37-Edition/05-NumPy-and-Arrays/05_06-Array-Slicing.html to see what's changed).

3) I'd consider this setup the "developers" version. To get to something more like "binder@home", an approach woud be to create a third container that has just the notebooks, data folder, and rendered html as a docker volume, as suggested in this thread: https://forums.docker.com/t/we-cant-push-a-docker-volume-so-now-what/56160/3 and push that to dockerhub.

4) reading a book as a user would then involve a) installing docker b) downloading a docker-compose.yml file as a gist or from a website c) running docker-compose up

5) I'll be adding containers/tutorials etc. at https://github.com/eoas-ubc/eoas_hpc_edu#eoas-hpc-tutorials as we experiment with this.