Load Magic Extensions

[ramnathv]

This is an awesome tool and exactly what I was looking for as I try to work with R and IPython. I was able to successfully convert a markdown file into an IPython notebook, but one glitch I noticed was that I had to manually load the rmagic extension before running my notebook. Since, your code automatically detects magics, I was wondering if it was possible to auto load these magics?

%load_ext rmagic

[ramnathv]

@aaren I put together a short proof-of-concept to convert R Markdown notebooks into IPython Notebooks using notedown. Thanks again for an awesome tool. I am new to Python, so it would take me some time to contribute back to this project!

[aaren]

I should get to this later in the week. There is some ipython documentation.

Would a command line switch provide what you need? Something like

notedown --load_ext rmagic your_markdown.md > your_notebook.ipynb

Do you mind linking to the proof of concept?

[ramnathv]

Here is a gist with my notes and a screencast on my proof-of-concept. I used a makefile to manually append %load_ext rmagic to my md file, which I then pass to notedown.

I like your idea of a commandline switch, but was wondering if it was possible to make it automatic, since you are already detecting what languages the user is using in their markdown file. I was thinking more in terms of auto loading magics, based on availability.

There is some work going on towards building an R Kernel for IPython. Given most R users work with R Markdown, I can see notedown being a very useful tool to handle conversions seamlessly. Thanks for all your hard work.

[aaren]

The way I understand this is: append %load_ext rmagic to the start of the file if there are R code blocks in the markdown.

The problem with doing this is that it would require restructuring the program. Currently we perform magic on code blocks on a block by block basis, with no scope for a block to cause modification of another block. This would be a more sensible way to structure the program so I might well do it.

In the meantime, I added code to solve one of the problems which is that we need %%R as the magic rather than %%r. This removes the need for sed 's/%%r/%%R/' in your linked Makefile.

[aaren]

@ramnathv are you ideally wanting notedown to be able to convert r-markdown into an ipython notebook?

If so, what do you think we should do with the attributes in the r-markdow (e.g. echo=FALSE)? Is it ok to just ignore them or do you want notedown to parse them somehow?

Or, are you just wanting notedown to create a notebook with all the R magic (%%R, %load_ext rmagic) enabled and ready to go?

See issue #3 for the wider problem of code attributes.

[ramnathv]

@aaren Thanks for your response. I am still pretty open to various possibilities for converting rmarkdown to ipynb. While it would be good to capture attributes, I am not sure how it will help in the ipython notebook. Moroever, I believe ipython notebooks will be doing away with cell level metadata, in which case it does not make sense to capture this at all. Let me know what you think.

As for Rmd to ipynb, I have an updated post here that makes use of the native R kernel for the ipython notebook, which is work-in-progress by @takluyver. The main difference is that the native kernel works without having to load cell magics. I would like to enable both types of conversion for the simple reason that sometimes I do write multi-language Rmd files, which work best with the magic driven notebooks. Right now, I am using makefiles to do these conversions, but it would be awesome if they can be enabled in notedown as commandline flags.

I read through your ideas on #3 and will add my comments this weekend.

[aaren]

I believe ipython notebooks will be doing away with cell level metadata

That is interesting, I did not know that. It would change a few other things I'm working on. source?

The problem with attributes is that IPython won't necessarily do anything with them. We could enhance notedown such that it does things differently depending on the attributes but I'm not sure I want to get into the complexity of supporting that.

For single language files, a native kernel makes a lot more sense, like engines in knitr.

[ramnathv]

I would agree with your observations. On a related notw, I was able to pair up notedown with runipy and my workflow to go from Rmd to a auto executed ipynb file in one go! I can see notedown being a critical tool in this toolchain.

[aaren]

If paulgb/runipy#25 is accepted, you'll be able to combine them in a single piped command like notedown file.md | runipy - notebook.ipynb :smile:

[ramnathv]

Awesome! This is going to be supercool.

[takluyver]

For the record, I don't know of any plan to get rid of cell level metadata. The language field, however, is moving from cell to notebook metadata - that may be what you're thinking of.

[aaren]

@takluyver will the 'language' field set the base language for the notebook then? is the intention to have it select a language kernel?

[ramnathv]

Ah. Okay. Thanks @takluyver. I might have misunderstood what you said at the hackathon. So to be clear, cell level metadata stays, just the language field is moving to notebook. Apologies for the confusion.

[takluyver]

@aaren : Yes, that's about right. There will also be a kernel field in the notebook metadata to select the kernel (because you may have more than one kernel with the same language).

@ramnathv : No problem!

[aaren]

@ramnathv have a go with the latest on the magic branch

You can get notedown running from this branch by doing python setup.py develop in the root of the branch.

Then try

notedown example.Rmd --knit --rmagic > example.ipynb

Also try

notedown example.Rmd --knit --rmagic --output

this should automatically generate the example.md and example.ipynb.

You need R with knitr installed as this is how we process the r-markdown.

[ramnathv]

@aaren This is really awesome! I will try and test this out sometime tomorrow. Great work!

[ramnathv]

@aaren I am getting an error when I tried this. Here is what I did

$ hub clone aaren/notedown
$ cd notedown
$ python setup.py develop
$ subl example.Rmd
$ notedown example.Rmd --knit --rmagic > example.ipynb

The error I am getting is the following

notedown example.Rmd --knit --rmagic > example.ipynb
usage: notedown [-h] [--output OUTPUT] [--code_block CODE_BLOCK] [input_file]
notedown: error: unrecognized arguments: --knit --rmagic

Any ideas why this might be happening?

[aaren]

The error I am getting is the following

notedown example.Rmd --knit --rmagic > example.ipynb
usage: notedown [-h] [--output OUTPUT] [--code_block CODE_BLOCK] [input_file]
notedown: error: unrecognized arguments: --knit --rmagic

what branch are you on? you need to be on magic, not master. Try this:

hub clone aaren/notedown
cd notedown
git checkout magic
python setup.py develop
notedown example.Rmd --knit --rmagic > example.ipynb

That should work. Have you used virtualenv? It makes developing python easier by creating a sandbox. This way you can play around with dev versions and leave your normal working install unaffected.

pip install virtualenv

hub clone aaren/notedown
cd notedown

# create and start a virtualenv called 'notedown-dev'
virtualenv notedown-dev
source notedown-dev/bin/activate 

git checkout magic
python setup.py develop

# this notedown is the magic version
echo `which notedown`
notedown example.Rmd --knit --rmagic > example.ipynb

# close the virtualenv
deactivate
# this notedown is the non-magic version
echo `which notedown`

When you close the virtualenv, notedown is restored to the non-magic version.

[ramnathv]

Perfect! It works now. One more question. What is the difference between these two lines?

notedown example.Rmd --knit --rmagic > example.ipynb
notedown example.Rmd --knit --rmagic --output

[aaren]

By default notedown outputs notebook to stdout.

If you use --output, it will output to filename.md and filename.ipynb, if 'filename.Rmd' is the source.

If you use --output filename2, it will output notebook to filename2, without any intermediate markdown.

I just did it like that to demonstrate - we can change it if you can think of a clearer way of doing it?

[aaren]

Would you mind me including your examples as test cases / examples in notedown?

[ramnathv]

Please go ahead and include them by all means. I hope the runipy project accepts your pull request so that one can automatically generate an ipynb with output from a Rmd document.

[aaren]

@ramnathv I pushed a load of this to master. Can you have a play and see if it is still working for you? Also have a look at the r-examples

[aaren]

@arne-cl, perhaps you're interested in this?