Create notebook with all possible markdown we support

[ellisonbg]

This is part of the larger work in this GH project:

https://github.com/jupyter/nbformat/projects/1

This particular task consists of creation a Jupyter notebook, using version 5.0 of the classic notebook, that contains examples (as markdown cells) of all the markdown we support. Here are some links to various other similar markdown syntaxes we should look at:

• CommonMark and its spec
• GitHub's variants is documented in this blog post and in their spec.

Finally, our own implementation uses marked along with customizations for embedded LaTeX math using MathJax.

The final result of this issue should be a PR to nbformat with the demo notebook.

Assigning this to @ashutoshbondre but @mpacer will help advise/guide this.

[ellisonbg]

Here is where we actually do the rendering:

https://github.com/jupyter/notebook/blob/master/notebook/static/notebook/js/textcell.js#L379

[ellisonbg]

A good starting point for this would be to write some python code that parses this file:

https://github.com/jgm/CommonMark/blob/master/spec.txt

And extracts the source of all the examples and puts them into Markdown cells.

Our library for creating notebooks from python is here:

http://nbformat.readthedocs.io/en/latest/

[ellisonbg]

Here is the GitHub version of spec.txt:

https://github.com/github/cmark/blob/master/test/spec.txt

[ellisonbg]

You can generate JSON data for all the examples using:

python3 test/spec_tests.py --dump-tests > spec.json

[mpacer]

Heads up, I'm on vacation for this next week so I won't be responsive.

[ashutoshbondre]

Hi, I am getting there,I have extracted the markdown cells from the json file, I have also tried and made a sample new notebook using python. But for the last 2 hours, I am stuck at figuring out, how do I transfer the extracted markdown code to the actual python notebook cells.

to be specific, I am not able to use this properly:

nbformat.v4.new_markdown_cell(source='', **kwargs)

I checked the documentation for nbformat which is really good, but still couldn't make it work.

this is some part of the code, the "new_markdown_cell" part is where I am stuck. How do I insert a new markdown cell into an empty ipynb I created?

import json
import nbformat
from pprint import pprint

with open('spec2.json') as data_file:    
    data = json.load(data_file)
x=nbformat.v4.new_notebook()
nbformat.v4.new_markdown_cell(source=(data[0]["markdown"]))

nbformat.write(x,"/Users/ashu/github/ans.ipynb",4.0)

FYI, in above code,

data[0]["markdown"]

is where, I have the actual markdown code.

would like some guidance please.

[mpacer]

Does your notebook also do the live scrape of the JSON? Do that and it'll make it easy for anyone to quickly debug this.

Link to a notebook in a gist that can download the data (using requests) and I'll prototype this.

If you want context: I don't want to have any chance that you and I are working off of fundamentally different files because we downloaded them slightly differently.

It's a generally good practice to make it as low friction as possible for people to help you (e.g., "download a notebook and then run it to get to exactly the place where you're at") when you ask for guidance. A hard-won lesson I've learned from many instances of asking for guidance. :D

On Fri, Jun 9, 2017 at 10:38 PM, Ashutosh Bondre notifications@github.com wrote:

Hi, I am getting there,I have extracted the markdown cells from the json file, I have also tried and made a sample new notebook using python. But for the last 2 hours, I am stuck at figuring out, how do I transfer the extracted markdown code to the actual python notebook cells.

to be specific, I am not able to use this properly:

nbformat.v4.new_markdown_cell(source='', **kwargs)

I checked the documentation for nbformat which is really good, but still couldn't make it work.

this is some part of the code, the "new_markdown_cell" part is where I am stuck. How do I insert a new markdown cell into an empty ipynb I created?

import json import nbformat from pprint import pprint

with open('spec2.json') as data_file: data = json.load(data_file) x=nbformat.v4.new_notebook() nbformat.v4.new_markdown_cell(source=(data[0]["markdown"]))

nbformat.write(x,"/Users/ashu/github/ans.ipynb",4.0)

FYI, in above code,

data[0]["markdown"]

is where, I have the actual markdown code.

would like some guidance please.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/jupyter/nbformat/issues/95#issuecomment-307544354, or mute the thread https://github.com/notifications/unsubscribe-auth/ACXg6D5hug5lhSJ8oQO1Ogl6YM6lUocrks5sCivxgaJpZM4N0ihs .

[ellisonbg]

I think the question is a more basic one about the nbformat API. To insert a cell you would do:

x.cells.append(cell)

Here are some examples of this:

http://programtalk.com/python-examples/nbformat.v4.new_markdown_cell/

On Fri, Jun 9, 2017 at 10:38 PM, Ashutosh Bondre notifications@github.com wrote:

Hi, I am getting there,I have extracted the markdown cells from the json file, I have also tried and made a sample new notebook using python. But for the last 2 hours, I am stuck at figuring out, how do I transfer the extracted markdown code to the actual python notebook cells.

to be specific, I am not able to use this properly:

nbformat.v4.new_markdown_cell(source='', **kwargs)

I checked the documentation for nbformat which is really good, but still couldn't make it work.

this is some part of the code, the "new_markdown_cell" part is where I am stuck. How do I insert a new markdown cell into an empty ipynb I created?

import json import nbformat from pprint import pprint

with open('spec2.json') as data_file: data = json.load(data_file) x=nbformat.v4.new_notebook() nbformat.v4.new_markdown_cell(source=(data[0]["markdown"]))

nbformat.write(x,"/Users/ashu/github/ans.ipynb",4.0)

FYI, in above code,

data[0]["markdown"]

is where, I have the actual markdown code.

would like some guidance please.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/jupyter/nbformat/issues/95#issuecomment-307544354, or mute the thread https://github.com/notifications/unsubscribe-auth/AABr0A7CIG-ukFxfTjxk24ANSGhAFQ9Yks5sCivxgaJpZM4N0ihs .

-- Brian E. Granger Associate Professor of Physics and Data Science Cal Poly State University, San Luis Obispo @ellisonbg on Twitter and GitHub bgranger@calpoly.edu and ellisonbg@gmail.com

[mpacer]

Yes, but if I didn't know that off the top of my head the best way for me to explore and figure it out with the documentation is to just try it out. I had thought it might have been something about how the json was being processed and transformed into text to go into a cell. Testing whether that was the case not easily possible without a reproducible example notebook.

On Fri, Jun 9, 2017 at 11:01 PM, Brian E. Granger notifications@github.com wrote:

I think the question is a more basic one about the nbformat API. To insert a cell you would do:

x.cells.append(cell)

Here are some examples of this:

http://programtalk.com/python-examples/nbformat.v4.new_markdown_cell/

On Fri, Jun 9, 2017 at 10:38 PM, Ashutosh Bondre <notifications@github.com

wrote:

Hi, I am getting there,I have extracted the markdown cells from the json file, I have also tried and made a sample new notebook using python. But for the last 2 hours, I am stuck at figuring out, how do I transfer the extracted markdown code to the actual python notebook cells.

to be specific, I am not able to use this properly:

nbformat.v4.new_markdown_cell(source='', **kwargs)

I checked the documentation for nbformat which is really good, but still couldn't make it work.

this is some part of the code, the "new_markdown_cell" part is where I am stuck. How do I insert a new markdown cell into an empty ipynb I created?

import json import nbformat from pprint import pprint

with open('spec2.json') as data_file: data = json.load(data_file) x=nbformat.v4.new_notebook() nbformat.v4.new_markdown_cell(source=(data[0]["markdown"]))

nbformat.write(x,"/Users/ashu/github/ans.ipynb",4.0)

FYI, in above code,

data[0]["markdown"]

is where, I have the actual markdown code.

would like some guidance please.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/jupyter/nbformat/issues/95#issuecomment-307544354, or mute the thread https://github.com/notifications/unsubscribe-auth/AABr0A7CIG- ukFxfTjxk24ANSGhAFQ9Yks5sCivxgaJpZM4N0ihs .

-- Brian E. Granger Associate Professor of Physics and Data Science Cal Poly State University, San Luis Obispo @ellisonbg on Twitter and GitHub bgranger@calpoly.edu and ellisonbg@gmail.com

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/jupyter/nbformat/issues/95#issuecomment-307545253, or mute the thread https://github.com/notifications/unsubscribe-auth/ACXg6NWnsmMj4BHgiYAHtiNIVbE3SJmsks5sCjFKgaJpZM4N0ihs .

[ellisonbg]

M - sorry I wasn't clear. Ashutosh began working at Cal Poly this week and is getting his bearings around the jupyter codebase and python. I had asked him to post questions to GitHub as he worked on this stuff.

As an aside the JSON data for this came from running the following command in the root of the CommonMark repo:

python3 test/spec_tests.py --dump-tests

On Fri, Jun 9, 2017 at 11:17 PM, M Pacer notifications@github.com wrote:

Yes, but if I didn't know that off the top of my head the best way for me to explore and figure it out with the documentation is to just try it out. I had thought it might have been something about how the json was being processed and transformed into text to go into a cell. Testing whether that was the case not easily possible without a reproducible example notebook.

On Fri, Jun 9, 2017 at 11:01 PM, Brian E. Granger < notifications@github.com> wrote:

I think the question is a more basic one about the nbformat API. To insert a cell you would do:

x.cells.append(cell)

Here are some examples of this:

http://programtalk.com/python-examples/nbformat.v4.new_markdown_cell/

On Fri, Jun 9, 2017 at 10:38 PM, Ashutosh Bondre < notifications@github.com

wrote:

Hi, I am getting there,I have extracted the markdown cells from the json file, I have also tried and made a sample new notebook using python. But for the last 2 hours, I am stuck at figuring out, how do I transfer the extracted markdown code to the actual python notebook cells.

to be specific, I am not able to use this properly:

nbformat.v4.new_markdown_cell(source='', **kwargs)

I checked the documentation for nbformat which is really good, but still couldn't make it work.

this is some part of the code, the "new_markdown_cell" part is where I am stuck. How do I insert a new markdown cell into an empty ipynb I created?

import json import nbformat from pprint import pprint

with open('spec2.json') as data_file: data = json.load(data_file) x=nbformat.v4.new_notebook() nbformat.v4.new_markdown_cell(source=(data[0]["markdown"]))

nbformat.write(x,"/Users/ashu/github/ans.ipynb",4.0)

FYI, in above code,

data[0]["markdown"]

is where, I have the actual markdown code.

would like some guidance please.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <https://github.com/jupyter/nbformat/issues/95#issuecomment-307544354 , or mute the thread https://github.com/notifications/unsubscribe-auth/AABr0A7CIG- ukFxfTjxk24ANSGhAFQ9Yks5sCivxgaJpZM4N0ihs .

-- Brian E. Granger Associate Professor of Physics and Data Science Cal Poly State University, San Luis Obispo @ellisonbg on Twitter and GitHub bgranger@calpoly.edu and ellisonbg@gmail.com

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/jupyter/nbformat/issues/95#issuecomment-307545253, or mute the thread https://github.com/notifications/unsubscribe-auth/ ACXg6NWnsmMj4BHgiYAHtiNIVbE3SJmsks5sCjFKgaJpZM4N0ihs .

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/jupyter/nbformat/issues/95#issuecomment-307545885, or mute the thread https://github.com/notifications/unsubscribe-auth/AABr0AxwmxCrpKQSX6-s2tBcRDidftM8ks5sCjUVgaJpZM4N0ihs .

-- Brian E. Granger Associate Professor of Physics and Data Science Cal Poly State University, San Luis Obispo @ellisonbg on Twitter and GitHub bgranger@calpoly.edu and ellisonbg@gmail.com

[mpacer]

I'm just encouraging the creation of a repo with periodic git commits so I can give feedback as it's being developed? Being able to only communicate via a single issue is an extremely limiting medium for providing guidance. I'm going to have to work with this code too, so I would like a hand in shaping it and it's APIs.

Why? Because I'd like to make it the basis of generating tests automatically too. I'll work on that once I have an idea of how this is going, because that should be part of our testing suites in the long run. And I don't want to have to rewrite tests if they release new versions of the spec.

So please @ashutoshbondre open a git repo and I'm happy to help you make it reproducible. I can even make PRs against your repo. I just need to see the current state of the problem & solutions to be able to provide much guidance. 'Tis how my brain works.

[ellisonbg]

Ahh, got it! I was thinking that we would use nbformat itself as that repo, but more than willing to create a new one for us to work with. But I agree a separate one would be very helpful to enable us to explore and move quickly with multiple PRs, etc. Should we create one as maybe jupyter/jupyter_markdown ?

On Sat, Jun 10, 2017 at 7:12 PM, M Pacer notifications@github.com wrote:

I'm just encouraging the creation of a repo with periodic git commits so I can give feedback as it's being developed? Being able to only communicate via a single issue is an extremely limiting medium of providing guidance. I'm going to have to work with this code too, so I would like a hand in shaping it and it's APIs.

Why? Because I'd like to make it the basis of generating tests automatically too. I'll work on that once I have an idea of how this is going, because that should be part of our testing suites in the long run. And I don't want to have to rewrite tests if they release new versions of the spec.

So please @ashutoshbondre https://github.com/ashutoshbondre open a git repo and I'm happy to help you make it reproducible. I can even make PRs against your repo. I just need to see the current state of the problem & solutions to be able to provide much guidance. 'Tis how my brain works.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/jupyter/nbformat/issues/95#issuecomment-307601473, or mute the thread https://github.com/notifications/unsubscribe-auth/AABr0DD12O2jnd6DFcFzjL8ohSzgvUoxks5sC00ogaJpZM4N0ihs .

-- Brian E. Granger Associate Professor of Physics and Data Science Cal Poly State University, San Luis Obispo @ellisonbg on Twitter and GitHub bgranger@calpoly.edu and ellisonbg@gmail.com

[mpacer]

Sounds perfect!

[mpacer]

Note: I'd have created it already but I don't think I actually have admin privileges to do so.

[ashutoshbondre]

I don't think, I have admin privileges either. Meanwhile, I have made a first draft of the notebook, after parsing the json data into separate markdown cells. Please take a look. Again, this is just a first draft & I will talk with Brian, on what else needs to be done here. https://github.com/ashutoshbondre/nbformat/blob/master/firstDraft.ipynb

[ashutoshbondre]

Not sure why clicking the link is not working, but if you copy the URL and paste it in a new tab its working.

[ellisonbg]

Hi all!

Ashutosh, glad you made progress! I just create this repo and added both of you as collaborators:

https://github.com/jupyter/jupyter_markdown

Ashutosh, can you submit a Pull Request to this repo with your notebook and maybe the JSON that generated it? Then we can discuss things there.

Cheers, Brian

On Mon, Jun 12, 2017 at 11:41 AM, Ashutosh Bondre notifications@github.com wrote:

Not sure why clicking the link is not working, but if you copy the URL and paste it in a new tab its working.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/jupyter/nbformat/issues/95#issuecomment-307881286, or mute the thread https://github.com/notifications/unsubscribe-auth/AABr0Hh0BbtOoWlN83GWjdeh1B2sgHKGks5sDYZMgaJpZM4N0ihs .

-- Brian E. Granger Associate Professor of Physics and Data Science Cal Poly State University, San Luis Obispo @ellisonbg on Twitter and GitHub bgranger@calpoly.edu and ellisonbg@gmail.com

[ashutoshbondre]

Done.

[ashutoshbondre]

Hey quick question, how can I run the code cell using python command? Like I know in the notebook you can do shift+enter, but if I don't want to manually do that, and just run the code cells using my script & get the output of the code cells in the notebook, what is the command / syntax for that? If at all that's possible. Thanks.