Add a notebook cell level magic system

[jankatins]

The IPython kernel has a magic system which can be used to do quite a lot of interesting things. It would be nice if that concept can be extended to the notebook.

Having the magics only in IPython has the problem that you can't use it to call things in the stack which are higher up, like creating new kernels and running code on them or building magics which work in julia and python.

I don't want line magics, only cell based ones. This could be realized by looking for "#!% " at the start of the cell and if found get the second word and look for a magic (=func) of that name and call it via func(<cell object>, <notebook session>) (notebook session would give access to the kernel manager and so on, cell would give access to both the content of the cell as to the representation to e.g. hide the inputs). It would need an extension point where extensions could add such functions (add_magic(name, func)).

An example magic would be #!% run_on_kernel ir\n<lots of R code>, which would open an R kernel (if not already existing in this session), run the code on that kernel and display the result.

Another example would be extending the display system https://github.com/jupyter/notebook/issues/652

[Carreau]

That can still be kernel only, the kernel protocol is open, so you likely can have a proxy-kernel that look at the first line of cell and do whatever, like dispatching to subkernel. @dsblank can likely show you how it does that in his many well alived kernels ! (right ?)

The kernel has no notion that it is ran from the notebook, and many project, now rely more and more to the fact that code might not be ran from a notebook: atom hydrogen, thebe, Microsoft PTVS, ...etc so we will likely not add a coupling between kernel and notebook/ cell document.

[jankatins]

@Carreau Actually I don't want to do any coupling (no kernel -> notebook link), the idea was, that the magics should live in the javascript part of the notebook.

[dsblank]

On Tue, Nov 24, 2015 at 6:19 AM, Matthias Bussonnier < notifications@github.com> wrote:

That can still be kernel only, the kernel protocol is open, so you likely can have a proxy-kernel that look at the first line of cell and do whatever, like dispatching to subkernel. @dsblank https://github.com/dsblank can likely show you how it does that in his many well alived kernels ! (right ?)

:)

The kernel has no notion that it is ran from the notebook, and many project, now rely more and more to the fact that code might not be ran from a notebook: atom hydrogen, thebe, Microsoft PTVS, ...etc so we will likely not add a coupling between kernel and notebook/ cell document.

— Reply to this email directly or view it on GitHub https://github.com/jupyter/notebook/issues/746#issuecomment-159234710.

[dartdog]

Similarly just noticed that access to shell commands ie !ls (along with no magic) don't work in the Scala Notebook provided by the Jupyter/allspark notebook notebook (docker stacks) so hopefully this approach would help that situation as well?

[dsblank]

Indeed, MetaKernel can handle magics and shell commands (% and !, respectively). If the allspark is using the wrapper-kernel API, then MetaKernel is a drop in replacement and will give you magics, shell, and more. If it is written in some other language than Python, then you either need to fire up a connection to Python (to re-use IPython's magics, or MetKernel's magics) or rewrite them in the host language.

[JamiesHQ]

@janschulz : As you know, we're doing a little housekeeping on our issue log and noticed this thread from 2015. Is this still an open request? thanks!

[jankatins]

It's still open, but given the state of the notebook I think it can be closed as unrealistic