Clojupyter support

[metasoarous]

@mikeyford has worked up some code for plotting from a Jupyter notebook using Clojupyter!

(require '[clojupyter.misc.display :as display])
(require '[clojupyter.misc.helper :as helper])
(helper/add-dependencies '[org.clojure/data.json "0.2.6"])
(require '[clojure.data.json :as json])
(helper/add-javascript "https://cdn.jsdelivr.net/npm//vega-embed@3")

(defn plot-vega [vega-json]
  (let [id (str (java.util.UUID/randomUUID))
        code (format "vegaEmbed('#%s', %s, {'mode': 'vega-lite'});" id, (json/write-str vega-json))]
      (display/hiccup-html 
        [:div [:div {:id id}]
                   [:script code]])))

(plot-vega some-vega)

I'd love to see this go into an oz.clojupyter namespace with functions view! and v! which more or less imitate the API of the oz.core namespace as much as possible (also, take note of #32).

Ideally, we'd do this so as to make clojupyter an optional dependency, wrapping all requires in a (try ... (catch ...)), but I'm actually having some trouble figuring out how to do this. So for now, let's just assume clojupyter will be a dependency and see if we can't relax this assumption down the road.

[metasoarous]

I'm not able to get the code above to work actually; I keep getting a vegaEmbed is not defined error. Am I missing something obvious to you @mikeyford?

[mikeyford]

Hmmm frustratingly I'm also having trouble once I've restarted everything today. I'm not clear on why this was working yesterday and not now, but presumably, there is some hidden state in jupyter that I'd loaded earlier in the session that enables this. (I had restarted the clojure kernel yesterday many times and it was still working, so it must be deeper. I was trying out some vega libs in other language kernels too earlier so perhaps something got loaded along the way).

I'll try and figure out what that was at some point over the coming days and let you know. I'll leave you with a tantalizing screen-shot I took yesterday to hopefully prove I'm not completely mad...

[metasoarous]

OK; Thanks for letting me know. I look forward to getting this working :-)

Thanks for your help!

[metasoarous]

@mikeyford Any luck figuring out how you got this to work?

[keesterbrugge]

Here the same issue is persued, but then for the iclojure kernel instead of clojupyter.

Partially it works, see screenshot below

[metasoarous]

@keesterbrugge Awesome! I'm adding another namespace at oz.notebook.iclojure with a function to wrap this. However, I'm not able to get this to work either! I'm beginning to suspect something is wrong with my environment. In any case, if you want to give it a whirl, I just pushed this namespace up to master, and released a snapshot build (1.5.0-SNAPSHOT) if you want to give it a whirl and see if it works for you. Thanks!

[keesterbrugge]

Hi,

Will try out your function as soon as I'm able to add oz as a dependency in iClojure. I'm having troubles with that too atm, see https://github.com/HCADatalab/IClojure/issues/4

Is that the same issue you experienced?

[metasoarous]

No, actually. I didn't get that far. I couldn't even get it to work with the bare #unrepl/mime {...} form. I basically get this

Any thoughts?

[keesterbrugge]

Sorry, no clue. Perhaps it's a good idea to use a reproducible test environment like a docker image.

[cgrand]

@metasoarous does it work in a python notebook? What happens here is that when you use unrepl/mime the execute_result message contains three representations of the result: text/plain text/iclojure-html (for our own plugin) and whatever/you-specified-in-unrepl-mime.

May it be a case of (lack of) priority between renderers? You are obviously being shown the text/plain representation.

[keesterbrugge]

@metasoarous oz.notebook.iclojure/view! works when a small change is made https://github.com/metasoarous/oz/pull/35

[metasoarous]

@keesterbrugge Amazing! Thanks so much for sorting that out. I've merged and will push out another snapshot release soon (and hope to have a non-snapshot release on Tues).

In the mean time, could you help me by telling me how to require oz.notebook.iclojure from an IClojure notebook? I couldn't immediately find this information, but would like to include it in the docs.

[metasoarous]

The [metasoarous/oz "1.5.0-SNAPSHOT"] release is now updated for anyone willing to give either the oz.notebook.iclojure or oz.notebook.clojupyter ns a spin. I'd be very appreciate of either confirmation that things are working or notification that there's an issue still.

Thanks in advance.

[keesterbrugge]

I can confirm that the view! function of oz.notebook.iclojure is working.

Requiring oz.notebook.clojupyter gives the following exception

CompilerException java.lang.ExceptionInInitializerError, compiling:(markdown_to_hiccup/core.cljc:1:1) 
               Var.java:    43 clojure.lang.Var$Unbound/throwArity                                  
               AFn.java:    32 clojure.lang.AFn/invoke                                              
.
.
.

I'm using the snapshot of this morning.

[keesterbrugge]

I've made a pull request to show how to require oz.notebook.iclojure from an IClojure notebook https://github.com/metasoarous/oz/pull/37

[metasoarous]

@keesterbrugge Thanks so much! The clojupyter message is awfully perplexing though. Can't imagine why markdown-to-hiccup would be posing an issue there.

[metasoarous]

Saw something about this here: https://stackoverflow.com/questions/7233334/exceptionininitializererror-when-using-leiningen

Is it possible you have a corrupt build target in your clojupyter project? Would you please try running lein clean &/or removing any related .lein directory and seeing if you still get the issue?

I have to head out for a bit today, but will try to check back in later.

Thanks again!

[jtcbrule]

I suspect there's a problem related to competing dependency versions. I got the following error:

(require '[oz.notebook.clojupyter :as oz])
CompilerException java.io.FileNotFoundException: Could not locate clojure/spec/alpha__init.class or clojure/spec/alpha.clj on classpath., compiling:(markdown_to_hiccup/core.cljc:1:1) 

I'm using lein jupyter, which depends on Clojupyter 0.1.0. There's also the potential for a conflict with the default Leiningen template, which uses Clojure 1.8.0; note that spec wasn't introduced until 1.9. However, even the latest Clojupyter depends on Clojure 1.8.0, so perhaps the problem lies elsewhere.

[keesterbrugge]

@metasoarous I tried your suggestion, I did lein clean and deleted the .lein dir in my home folder. Then I followed clojupyter's install instructions again. But I still get exactly the same exception.

[metasoarous]

Thank you both @jtcbrule & @keesterbrugge.

Based on @jtcbrule's comment I tried manually adding org.clojure/spec.alpha as a dependency, as I realized it was not listed as a dependency in markdown-to-hiccup. However, when I do that I get the following complete stack trace: https://gist.github.com/metasoarous/d4259e2d6f9cebfeea0f3134a5ffc557.

The important bits are here:

java.lang.ExceptionInInitializerError, compiling:(markdown_to_hiccup/core.cljc:1:1)
        at clojure.lang.Compiler.load(Compiler.java:7391)
        at clojure.lang.RT.loadResourceScript(RT.java:372)
        at clojure.lang.RT.loadResourceScript(RT.java:363)
        at clojure.lang.RT.load(RT.java:453)
        ...
        at clojure.core$require.invokeStatic(core.clj:5796)
        at clojure.core$require.doInvoke(core.clj:5796)
        at clojure.lang.RestFn.invoke(RestFn.java:1289)
        at oz.core$eval6158$loading__5569__auto____6159.invoke(core.clj:1)
        at oz.core$eval6158.invokeStatic(core.clj:1)
        at oz.core$eval6158.invoke(core.clj:1)
        at clojure.lang.Compiler.eval(Compiler.java:6927)
        at clojure.lang.Compiler.eval(Compiler.java:6916)
        at clojure.lang.Compiler.load(Compiler.java:7379)
        at clojure.lang.RT.loadResourceScript(RT.java:372)
        ...
        at clojure.core$require.doInvoke(core.clj:5796)
        at clojure.lang.RestFn.invoke(RestFn.java:482)
        at oz.notebook.clojupyter$eval6152$loading__5569__auto____6153.invoke(clojupyter.clj:1)
        at oz.notebook.clojupyter$eval6152.invokeStatic(clojupyter.clj:1)
        at oz.notebook.clojupyter$eval6152.invoke(clojupyter.clj:1)
        at clojure.lang.Compiler.eval(Compiler.java:6927)
        at clojure.lang.Compiler.eval(Compiler.java:6916)
        at clojure.lang.Compiler.load(Compiler.java:7379)
        ....
        at clojure.core$require.invokeStatic(core.clj:5796)
        at clojure.core$require.doInvoke(core.clj:5796)
        at clojure.lang.RestFn.invoke(RestFn.java:482)
        at markdown_to_hiccup.core$eval19343$loading__5569__auto____19344.invoke(core.cljc:1)
        at markdown_to_hiccup.core$eval19343.invokeStatic(core.cljc:1)
        at markdown_to_hiccup.core$eval19343.invoke(core.cljc:1)
        at clojure.lang.Compiler.eval(Compiler.java:6927)
        at clojure.lang.Compiler.eval(Compiler.java:6916)
        at clojure.lang.Compiler.load(Compiler.java:7379)
        ... 105 more
Caused by: java.lang.IllegalStateException: Attempting to call unbound fn: #'clojure.core/ident?
        at clojure.lang.Var$Unbound.throwArity(Var.java:43)
        at clojure.lang.AFn.invoke(AFn.java:32)
        at clojure.spec.alpha$spec_impl.invokeStatic(alpha.clj:915)
        at clojure.spec.alpha$spec_impl.invoke(alpha.clj:908)
        at clojure.spec.alpha__init.load(Unknown Source)
        at clojure.spec.alpha__init.<clinit>(Unknown Source)
... 136 more

This makes it look to me like markdown-to-hiccup's usage of spec will not work without having a more recent Clojure which comes with clojure.core/ident?. I've submitted an issue on Clojupyter to update the Clojure version (see clojupyter/clojupyter#77), so we'll see how that goes.

In the mean time, I think I may have an interim solution. Fundamentally the problem is that the oz.notebook.clojupyter namespace depends on oz.core which in turn depends on markdown-to-hiccup.core. It's possible that if I duplicate all of the logic in oz.core necessary for oz.notebook.clojupyter to work, I can avoid markdown-to-hiccup from getting loaded, and that we'll be able to get this working.

I'll try this out and report back.

Thanks again for everyone's work helping bag this feature!

[metasoarous]

We did it!

Thanks so much to everyone in this thread who has chipped in a piece of the puzzle! Please check out [metasoarous/oz "1.5.0"] to take advantage of the fruits of this labor.

[keesterbrugge]

@metasoarous Thanks for your work so far and great that it works for you, but I'm afraid I'm still having problems plotting inline.

I can successfully require oz.notebook.clojupyter

but when I try to plot the spec it doesn't show, and I get the following error in the console

I also tried version 1.5.1.

I also tried the clojupyter example notebook where I had the same problem

Do I need to update something?

Hope this is helpful.

[metasoarous]

Thanks for reporting @keesterbrugge.

It looks like you're using Chromium, which I had noticed some bugs with regarding loading of Clojure syntax highlighting code, and also has trouble loading the data at https://vega.github.io/vega-lite/data/cars.json. However, I tried the code you pasted (with data generated as in the README), and it worked just fine.

I think the specific issue you're experiencing could be related to the version of Clojupyter or Jupyter you're running. To isolate, would you please try running using the official docker container, as described in the README? This was how I got it running.

If you continue to have a problem with this, would you please open a separate issue?

Thanks again!

[keesterbrugge]

You're right, with the docker image it does work