search

mwouts / jupyter-book-experiments

Deploy pre-existing Markdown and R Markdown books with Jupyter Book
1 stars 0 forks source link

Render Elegant Scipy with Jupyter Book #1

Open mwouts opened 5 years ago

mwouts commented 5 years ago

I'd like to see how we can use Jupyter Book to render Elegant Scipy.

That would be a nice example for https://github.com/mwouts/jupytext/issues/247.

@choldgraf , do you think we could write together a short bash notebook that demoes this? I started drafting one for a Bookdown repo, but maybe this example will be simpler?

Also, we may need here an additional option to allow the execution of Python cells in Markdown documents, even if they don't have the Jupytext header.

choldgraf commented 5 years ago

Cool idea! What are the steps that would need to be taken (at a high level)? Are all the code cells in elegant scipy meant to be executable?

Maybe @jni or @stefanv would be interested in chiming in on this.

stefanv commented 5 years ago

If you succeed, I'd love to include that on our Makefile and put a hosted version on our website. I'm happy to help you understand any of the custom parts of our build, but mainly we just have a few special comments in the Markdown files that we use to identify things like examples.

mwouts commented 5 years ago

Thanks @stefanv for your interest. I think the chances that we get this to work are reasonable. And the example will tell how we can evolve both Jupyter Book and Jupytext to render real and challenging books like yours.

@choldgraf , the Elegant Scipy book is a great example as it is written entirely as Markdown documents which represent Jupyter notebooks using notedown, a predecessor of Jupytext. The repository seems to work very well with Jupytext, at least adding jupytext to the requirements was enough to render the .markdown files as notebooks on binder.

My initial attempt at turning Elegant Scipy into a Jupyter Book is at Elegant_Scipy_as_a_Jupyter_Book.md. A few steps there are not super user friendly yet (especially changing the kernel with jupytext seems buggy), but we will take care of that later on. @choldgraf , what do you think? It's not exactly working (I just get .md files in the _build directory), but maybe we're not that far away?

jni commented 5 years ago

@mwouts I don't understand from your comment whether it is working already?

mwouts commented 5 years ago

No, sorry it is not working yet. I need @choldgraf's feedback to see what is wrong, as I don't have much experience with Jupyter Book yet.

One thing that does work already, is to open the original Elegant Scipy repo with Jupytext on Binder.

Just add jupytext in the requirements, as here, and then you will be able to open the .markdown chapters as notebooks in Jupyter - give it a try here: https://mybinder.org/v2/gh/mwouts/elegant-scipy/binder_jupytext?filepath=markdown

jni commented 5 years ago

Oh wow, I get it now. That's super cool! Time to add a binder link and maybe deprecate the elegant-scipy/notebooks repository!

stefanv commented 5 years ago

The binder output renders the callouts correctly! I think @mwouts is on the right track here. @jni agree that we can use this as a default mechanism.

choldgraf commented 5 years ago

@mwouts - it should be finding the .markdown files in the master branch right now, is it still not working even though you are running from master?

mwouts commented 5 years ago

Thanks @stefanv , @jni. Please let me know if I can help with the binder setup. I'd recommend to check every chapter, as I tested only a few of them. Note that Jupytext expects that every ```python code chunk is executable, is that right in the case of your book? Also I believe that the compilation of the book itself could be done with jupytext --execute, tell me if anything does not go as expected.

@choldgraf , I do think I am using the master branch. Actually the code I used is in that (bash) notebook: https://github.com/mwouts/jupyter-book-experiments/blob/master/Elegant_Scipy_as_a_Jupyter_Book.md . In an ideal world I would have expected the notebook to be runnable on Binder with the bash kernel (there's a link to Binder at the top of the notebook), but apparently the bask kernel has an issue with my 'conda activate' command. I'll see if I can fix that. Unless you see a better way than a notebook to collaborate of the experiment? 😄

Also a specific question I had for you, Chris, is whether I could avoid creating a new folder for the book content. In the case of the Elegant Scipy book, I'd like to determine the minimal addition that can turn the repo into a Jupyter Book. Can I call jupyter-book and tell it that I don't want to copy the chapters, but leave them where they are found?

mwouts commented 5 years ago

@choldgraf , the binder now runs. It was just my conda create which missed a -y. So you can see my exact experiment at https://mybinder.org/v2/gh/mwouts/jupyter-book-experiments/master?filepath=Elegant_Scipy_as_a_Jupyter_Book.md.

At the current stage we have the following issues

Still I am confident that it may soon start working well!

The binder link is a great place to experiment. However, if you want to edit the notebook and contribute back your modifications, I recommend that you install jupytext and bash_kernel locally, execute python -m bash_kernel.install and then edit Elegant_Scipy_as_a_Jupyter_Book.md in Jupyter. Looking forward to seeing your fixes there!

mwouts commented 5 years ago

I just gave another try to this, and I've reach a point where jupyter-book seems to build the book. This is something already. Now I need to find out how to activate the book on github pages...

mwouts commented 4 years ago

@choldgraf , thanks for your input yesterday. I was able to build the book following your deploy demo.

The book is available here: https://mwouts.github.io/elegant-scipy-as-a-jp-book.

And the actions that led to that are documented here: https://github.com/mwouts/jupyter-book-experiments/blob/8a6491767801c77e0f473246c0bbcdef4fb81716/Elegant_Scipy_as_a_Jupyter_Book.md

The good news is that there are not too many steps there: it's only a matter of

I think that everything there (except the two config files) could go to a continuous integration file, as in Chris' deploy demo.

Now, there is probably much to do yet! Especially,

jni commented 4 years ago

@mwouts this is 😍😍😍😍! Thank you! I’m sorry that I haven’t had a chance to look at all the details, but at a first glance it just looks gorgeous! I’m happy to give the rest of the to-do items a whirl from here now that you’ve done the hard work! ;)

choldgraf commented 4 years ago

Beautiful! Really cool @mwouts :-)

A few things:

These could both be semi-automated with something like:

import jupytext as jpt
from glob import glob

path_files = glob('./content/ch*.md')
for ipath in path_files:
    ntbk = jpt.read(ipath)
    for cell in ntbk.cells:
        # Replace the callout cell with a popout tag
        if '{.callout}' in cell.source:
            cell.metadata.get('tags', []).append("popout")
            cell.source = cell.source.replace('{.callout}', '')

        # If we find what looks like a quotation syntax in the cell, make it an epigraph
        if "> —" in cell.source:
            if 'epigraph' not in cell.metadata.get('tags', []):
                cell.metadata.get('tags', []).append('epigraph')

    jpt.write(ntbk, ipath)

Although it'll probably miss a few things because there aren't explicit "end-cell" pieces in the page syntax.

What else is missing?! Let's make a wish-list!

jni commented 4 years ago

@choldgraf :wave:

I've now seen the words "callout", "popout", and "sidebar" all used for this kind of thing...I wonder what you all think would be the most intuitive word?

Software carpentry and I think O'Reilly both use callout. Who are in the other camp? =)

I'm worried that some of our callouts might be too big to fit nicely in a sidebar...

We could add an "epigraph" tag to the pages that begin with a quote to make them formatted a bit more nicely.

I think that's all of them? But, yes! =)

What else is missing?! Let's make a wish-list!

This one is kinda huge but it would be super-amazing if the functionality from @stefanv's exercise exciser was built into Jupyter Book. To summarise: exercise solutions need to be executed in the context of the original chapter because it uses a bunch of variables defined in the main text. However, we would like to copy the exercises and move the solutions to a new chapter at the end, after they've been executed, so that looking at the solution is "hard". Currently these are marked as <!-- exercise begin --> (exercise text) <!-- solution begin --> (solution text) <!-- solution end --> <!-- exercise end -->, but different tags/metadata could be used.

Also, can Jupyter Book do figure captions? If so, how? Again, for us, @stefanv rolled his own solution based on HTML comments, with possibly my favourite name for a script ever, caption-crunch.py. :joy:

choldgraf commented 4 years ago

@jni 👋 👋

Software carpentry and I think O'Reilly both use callout. Who are in the other camp? =)

Ummmm...maybe just me? :-)

That said, I wonder if "sidebar" and "callout" are two separate things. One is more like "I want to highlight this information" and the other is more "I want to highlight this information but in a way that it doesn't break the flow of the main text". Maybe "callout" and "sidebar" are two separate things?

This one is kinda huge but it would be super-amazing if the functionality from @stefanv's exercise exciser was built into Jupyter Book. To summarise: exercise solutions need to be executed in the context of the original chapter because it uses a bunch of variables defined in the main text. However, we would like to copy the exercises and move the solutions to a new chapter at the end, after they've been executed, so that looking at the solution is "hard". Currently these are marked as <!-- exercise begin --> (exercise text) <!-- solution begin --> (solution text) <!-- solution end --> <!-- exercise end -->, but different tags/metadata could be used.

Mmmm, the way I'd do this is probably similar to how you did it. We might be able to build in some (probably hacky) ways to improve the experience with this (e.g., maybe a dont_run tag that would prevent cells from being run?) but I bet it'd require some custom code regardless...

Also, can Jupyter Book do figure captions? If so, how? Again, for us, @stefanv rolled his own solution based on HTML comments, with possibly my favourite name for a script ever, caption-crunch.py. :joy:

Right now, no, but if we supported some extra cell metadata it'd be easy. Currently jupyter-book tries to only use tags for triggering certain behavior. You could imagine having some slightly more complex cell metadata (e.g. a "caption: 'mycaption'" key/value would assume that the cell outputs a PNG, and add the 'mycaption' as a div underneath it). This is the kinda stuff I'd want to run by folks in the community to figure out the right pattern etc, so this feedback is helpful!

mwouts commented 4 years ago

@mwouts this is 😍😍😍😍! Thank you!

Well, thank you yourself for sharing the beautiful book! And @choldgraf and the Jupyter Book team have done the hard work - As for myself I've not done much, I was just curious to see how these two wonderful projects could fit together!

Let's make a wish-list!

Great idea! What I am looking for is the following:

More generally, I think we could have a look at the Elegant Scipy Makefile and integrate into the Jupyter Book project the items that are generic enough.

Also, can Jupyter Book do figure captions?

Aren't captions supported by Jupyter? I think it's a natural request and I'd love to see this in Jupyter Notebook/Lab/nbconvert.

I don't find much material about that, yet the Making publication ready Python Notebooks post by Julius Schulz suggest that one can use the following cell metadata:

"caption": "somecaption",
"label": "fig:somelabel",
"widefigure": true

And I see that the same metadata is also used by ipypublish.

This one is kinda huge but it would be super-amazing if the functionality from @stefanv's exercise exciser was built into Jupyter Book. To summarise: exercise solutions need to be executed in the context of the original chapter because it uses a bunch of variables defined in the main text.

Interesting! Maybe I can suggest another approach here: why not moving the exercises to a separate notebook, that would source the corresponding chapter before executing the exercises? I mean, maybe we could do something like this

import jupytext as jpt
nb = jpt.read('ch1.markdown')
py = jpt.writes(nb, 'py')
exec(py)

and then continue with the exercises?

jni commented 4 years ago

why not moving the exercises to a separate notebook, that would source the corresponding chapter before executing the exercises?

Unfortunately, in some cases, the solution to the exercise is required for the rest of the notebook to execute. Perhaps this is a bad practice, but it's there now...

I don't find much material about that, yet the Making publication ready Python Notebooks post by Julius Schulz suggest that one can use the following cell metadata:

Well, this is not standard in any way currently, as far as I know, but I agree that following prior art is a very good idea. I liked that post back in the day and thought, brilliant, surely this will be stock standard jupyter very soon... :sweat_smile:

How does one define generic cell metadata in jupytext?

mwouts commented 4 years ago

How does one define generic cell metadata in jupytext?

Oh I see this is not documented, sorry. I'll make sure the documentation includes that in the near future... Anyway:

choldgraf commented 4 years ago

I like the idea of using ipypublish as inspiration here - I'll take a deeper dive into their API and syntax!