Would it be possible to load Jupyter extensions with %load_ext?

[Naereen]

Hello there, I'm curious to know if it would be possible to load Jupyter extensions with %load_ext, like for Python/IPython?

I'm started to write teaching material using jupyter-ocaml, and I'm in love with the itikz Jupyter/IPython extension. It works fine for Python kernel, but of course it can't be loaded nor used from jupyter-ocaml:

In [1]: %load_ext itikz
File "[1]", line 1, characters 0-1:
Error: Syntax error
   1: %load_ext itikz

As these % lines are not supported by OCaml, and are naively passed to the OCaml kernel and underlying toplevel interpret. As the documentation of IPython magic states, "To Jupyter users: Magics are specific to and provided by the IPython kernel. Whether Magics are available on a kernel is a decision that is made by the kernel developer on a per-kernel basis. To work properly, Magics must use a syntax element which is not valid in the underlying language.".

But I wonder if we could add some partial support for Jupyter extensions to jupyter-ocaml.

I see different possibilities:

• interpret lines starting by % differently, and pass them to a second kernel, using IPython under the hood?
• or use lines like comments so they nicely degrade if the code is exported to .ml or other use, like

  (*#load_ext itikz *)

• or use OCaml commands that start with #?

What do you think?

Note: it is not a blocking bug for my workflow, as a very easy hack is to temporarily change the kernel from jupyter-ocaml to Python, use the IPython magic, then switch back. It's not clean, but it works.

• The advantages are that this "hack" works for any kernel, obviously, and requires no code change to any kernel,
• But the drawbacks is that it forbids integration between OCaml code and the IPython magic (like for instance generating a string which generates a figure), and any change of kernel restarts the OCaml kernel when coming back after the IPython kernel.

[akabe]

Related to #165, supporting jupyter extensions is probably possible, but I don't know detailed behavior of them. this kernel accepts code sent from jupyter at https://github.com/akabe/ocaml-jupyter/blob/master/jupyter/src/kernel/client.ml#L143 . I think the extensions can be implemented like:

if code.[0] = '%' then (* processing for extensions *)
else Repl.eval ~ctx:parent ~count client.repl code ...

I welcome your PR.

[Naereen]

Thanks for the direction, indeed it could be possible to hack this by checking if code[0] = '%' in src/kernel/client.ml. Now the question is what to do with this process...

The IPython documentation states

To Jupyter users: Magics are specific to and provided by the IPython kernel. Whether Magics are available on a kernel is a decision that is made by the kernel developer on a per-kernel basis. To work properly, Magics must use a syntax element which is not valid in the underlying language. For example, the IPython kernel uses the % syntax element for Magics as % is not a valid unary operator in Python. However, % might have meaning in other languages.

And I'm not surprised!

A very quick hack I just tried:

# let run_magic (code: string) : int = Sys.command (Format.sprintf "ipython -c '%s'" code);;
val run_magic : string -> int = <fun>
# run_magic "%timeit 2**1000" ;;
530 ns ± 7.03 ns per loop (mean ± std. dev. of 7 runs, 1000000 loops each)
- : int = 0
# 

It's not perfect, but it's a start!

[Naereen]

Plugging this in src/kernel/client.ml would simply to change code to "Sys.command \"ipython -c '%s'\" with %s interpolating code.

[akabe]

I think ## is appropriate because ocaml cannot define a operator ##. # is possibly misunderstood as a special command of ocaml like #load.

[Naereen]

Okay, thanks @akabe. I'll try this and see if its useful. But if IPython magics can only be sub-shells, I guess that a lot of their advantages will be lost: they will only be used on short Python snippets, not on OCaml code.

[akabe]

@Naereen Sorry for my long silence. I realized that my idea was wrong. IPython extension cannot be achieved as a simple hook because it is much flexible than I thought. However I have two ideas and plan to add the feature.

Now I have two ideas. First, the implementation as ocaml functions:

# sh {|curl -Lo "example.html" "https://example.com/"|}

A lot of IPython extensions are possibly ported by the above way. In this case, we add a function like sh to the library jupyter.notebook, and don't need to modify src/kernel/client.ml.

Second, the implementation by ocaml ppx like:

# [%timeit fib 12 - fact 12] ;;
- : (int * Timeit.result) = (xxx,  { mean = 1.23; std = 0.018; runs = 7; })

I consider to create a new library independent from ocaml-jupyter because [%time] and [%timeit] are generic.

Do you have any ideas?

[akabe]

Oh, I noticed ocaml ppx is unnecessary to time and timeit. They can be achieved as a function of type (unit -> 'a) -> ('a * Timeit.result) as follows:

# timeit (fun () -> fib 12 - fact 12) ;;
- : (int * Timeit.result) = (xxx,  { mean = 1.23; std = 0.018; runs = 7; })

I plan to add IPython extensions as utility functions into the library jupyter.notebook.

[akabe]

working in #187