search

QuantEcon / sphinxcontrib-jupyter

A Sphinx Extension for Generating Jupyter Notebooks
BSD 3-Clause "New" or "Revised" License
77 stars 23 forks source link

Recycle stuff from rst2ipynb? #18

Closed nthiery closed 5 years ago

nthiery commented 6 years ago

Dear Sphinxcontrib-jupyter devs,

I just discovered this project. Having the same deep need for a rst to ipynb conversion, we have been dreaming for a while of exactly this: a jupyter export extension for Sphinx. Thank you!!! I am now impatient to try it out and see how it fits into our documents pipeline for SageMath. See e.g https://more-sagemath-tutorials.readthedocs.io/ .

The jupyter notebooks that are linked from each page there are produced using a hacky solution rst-to-ipynb that we implemented on top of pandoc and notedown. You may want to briefly explore it in case you find something to recycle there; e.g. the test files.

The tricky part of such a converter is how to flatten the tree structure of a RST document into the linear (code, text, code, text, ...) structure of a Jupyter notebook, without too much degradation. The above tests exercise this a bit.

Cheers, Nicolas cc: @slel

willingc commented 6 years ago

Just an FYI to @nthiery and the devs here, nbsphinx is another project that can be used to convert notebooks for Sphinx docs.

Very cool to see this project moving forward 👍

nthiery commented 6 years ago

Yes indeed, though nbsphinx goes the other way round: integrating existing notebooks into a sphinx collection of documents, rather than exporting sphinx documents as notebooks.

jstac commented 6 years ago

@nthiery Very glad it's useful for you!

We'll certainly explore rst-to-ipynb and see what we can combine.

I'm not one of the devs, I'm one of the authors of these lectures and, given that all source is RSTs, we had a pressing need to generate notebooks. For example, we use them to test the code in our RSTs every night. See the badges here:

https://lectures.quantecon.org/status.html

Anyway, I'm really pleased that some of what our project has put together is useful for others. I hope we can share more of our tools over time.

@willingc Thanks for comments and PR ;-)

The lead devs of this project are on holiday but will be back in action in a couple of weeks. (CC @NickSifniotis @myuuuuun @mmcky )

willingc commented 6 years ago

You are absolutely right @nthiery. Just giving a heads up...in case you hadn't seen. :sunny:

Very nice that folks are writing these tools. +1 for sharing. Happy holidays to all.

mmcky commented 6 years ago

Thanks @nthiery for the links. I will have a closer look at that project (rst-to-ipynb) over the next couple of weeks and I will divert a bit more attention to this project in February.

nthiery commented 6 years ago

Thanks [1]@nthiery for the links. I will have a closer look at that project (rst-to-ipynb) over the next couple of weeks and I will divert a bit more attention to this project in February.

Cool! Feel free to ping if you have questions.

mmcky commented 6 years ago

Thanks @nthiery I am working my way through the rst-to-ipynb project now - it will be an excellent resource. The collection of tests is great and I am currently trying to think of the best way to test the sphinx-extension. Would it be OK if we used many of those test cases? Also are those tests compared with known ipynb output or are they mainly for testing execution?

nthiery commented 6 years ago

Thanks [1]@nthiery I am working my way through the rst-to-ipynb project now - it will be an excellent resource. The [2]collection of tests is great and I am currently trying to think of the best way to [3]test the sphinx-extension. Would it be OK if we used many of those test cases?

Yes, please reuse whatever you can, in the tests or elsewhere.

Also are those tests compared with known ipynb output or are they mainly for testing execution?

Testing execution, plus occasionally visually inspecting the produced collated all.ipynb. Not great :-) Presumably, the right thing to do would be to store a copy of the produced notebooks (after visual inspection), and use nbdime [1] to automatically test later runs.

I can provide you with a copy of the currently produced ipynb if helpful.

Cheers, Nicolas

[1] https://github.com/jupyter/nbdime

willingc commented 6 years ago

fyi @vidartf re: nbdime

mmcky commented 6 years ago

Thanks @nthiery -- a reference copy of the notebooks would be great for testing against. I will take a closer look at nbdime to see how to setup some automated testing.

mmcky commented 6 years ago

I have setup some basic test infrastructure for testing a set of rst files against a known ipynb output as PR #34. Will populate with some test cases before merging.

nthiery commented 6 years ago

Thanks [1]@nthiery -- a reference copy of the notebooks would be great for testing against.

Please find it attached. Not implying it's necessarily the best possible output :-)

nthiery commented 6 years ago

Oops, mail attachments don't work. Second attempt :-) expected_outputs.zip

mmcky commented 6 years ago

thanks @nthiery. It is great to get a set of corresponding notebooks -- not necessarily to use as a golden master :) -- but to get another perspective on the translation between rst and ipynb. Thanks for sending them through -- it is greatly appreciated.

mmcky commented 6 years ago

Reviewing the rst2ipynb files has been really helpful. I have made a few changes so far to the extension as a result:

  1. use html based image inclusion in markdown cells within the notebook. This allows support for directive attributes such as scale etc. See PR #45
  2. updated display math to be simpler markdown output with default alignment properties as a result. Previously we had wrapped them in html tables.
mmcky commented 6 years ago

@nthiery when integrating tests for tables I noticed one of the test cases has no header.

In the PR https://github.com/QuantEcon/sphinxcontrib-jupyter/pull/43 I have some discussion about how the extension supports tables and we currently export to markdown for readability rather than html. However this would mean that the extension couldn't parse tables without headers based on a restriction in markdown.

Do you have any thoughts from a sage perspective? Are tables (without a header) important to support?

nthiery commented 6 years ago

[1]@nthiery when integrating tests for tables I noticed one of the test cases has not header.

In the PR [2]#43 I have some discussion about how the extension supports tables and we currently export to markdown for readability rather than html. However this would mean that the extension couldn't parse tables without headers based on a restriction in markdown.

Do you have any thoughts from a sage perspective? Are tables (without a header) important to support?

Sorry, this went under the radar. From the top of my head, I don't think this feature is used much, if at all, in Sage documents. Or worst comes to worst, we could avoid it.

Given the readability benefits of Markdown vs HTML tables, this sounds like a good compromise.

mmcky commented 6 years ago

thanks @nthiery. Great I will go ahead and merge the table support that we have. I think down the track it would be good to support both html and markdown output.

mmcky commented 5 years ago

thanks @nthiery for letting us use many of the tests cases from rst2ipynb. I think I will close this issue now that we have tests up and running with CI.