How to generate graphs with common-lisp-jupyter?

[ryukinix]

Hi! Sorry for bothering you again. I'm just very happy to see this development. I see you add recently a collection of useful widgets. There is some way to to embedded graphs in this kernel?

It would be so awesome!

https://github.com/martinkersner/cl-plot Something like this or better.

[yitzchak]

No need to apologize.

Unless you want the graphs to have interactive sliders or such, you probably don't need widgets.

Media can be displayed with the functions defined in results.lisp The displayed media can come from a file or an inline value. In the case of something like a gnuplot wrapper you will probably need to write the result to a file then display the result in the notebook with jupyter:file. There are a few examples in the about notebook.

If I understand cl-plot correctly then you would do something like this.

(defparameter *dataframe*
  '((1 1)
    (2 2)
    (3 1.5)
    (4 3)
    (5 2.5)
    (6 4)))

(defparameter *fig* (make-instance 'figure))

(xlabel *fig* "x-label")
(ylabel *fig* "y-label")
(title *fig* "Basic example")
(scatter *fig* *dataframe*)

(save *fig* "fig.png")
(jupyter:file "fig.png")

If you are using JupyterLab there is also an extension for Vega-Lite that allows one to pass the JSON based Vega-Lite description right to the front end. Eventually I may make some wrapper code for this.

(jupyter:inline-result
  (jsown:new-js
    ("$schema" "https://vega.github.io/schema/vega-lite/v3.json")
    ("description" "A simple bar chart with embedded data.")
    ("data" (jsown:new-js
      ("values" (list
        (jsown:new-js ("a" "A") ("b"  28)) (jsown:new-js("a" "B") ("b" 55))  (jsown:new-js("a" "C") ("b" 43)) 
        (jsown:new-js("a" "D") ("b" 91)) (jsown:new-js("a" "E") ("b" 81))  (jsown:new-js("a" "F") ("b" 53)) 
        (jsown:new-js("a" "G") ("b" 19)) (jsown:new-js("a" "H") ("b" 87))  (jsown:new-js("a" "I") ("b" 52))))))
    ("mark" "bar")
    ("encoding" (jsown:new-js
      ("x" (jsown:new-js ("field" "a") ("type" "ordinal")))
      ("y" (jsown:new-js ("field" "b") ("type" "quantitative"))))))
  "application/vnd.vegalite.v2+json")

[ryukinix]

Thank you! I'll try that!

[Symbolics]

If you are using JupyterLab there is also an extension for Vega-Lite that allows one to pass the JSON based Vega-Lite description right to the front end. Eventually I may make some wrapper code for this.

What is the name of the recommended extension for Vega-Lite?

[yitzchak]

It's called @jupyterlab/vega5-extension. You can install from the extension manager in the lab interface or the following on the command line.

jupyter-labextension install @jupyterlab/vega5-extension 

[Symbolics]

Is the extension still required? I was able to generate some Vega-Lite graphs with JupyterLab 3.0.9 by creating a spec file and double clicking on it. See https://jupyterlab.readthedocs.io/en/stable/user/file_formats.html#vega-vega-lite. I admit though, it's a rather awkward way to create a plot.

You mentioned a wrapper. I tried wrapping a Vega Lite spec composed of alists, which seems a natural fit to me, and it worked well until the yason developer(s) reverted the recursive behaviour because it broke some existing functionality slightly, and were unwilling to address it properly because it would mean a breaking change to the API. This is unfortunate because developing a plot spec from lisp is more natural, and you can splice in data with the backquote mechanism. I suppose an alternative JSON library could be tried; I used yason because it's the library used by json-mop and it would be nice to go backwards and forwards between the two.

Did you give any thought to how you might wrap a Vega-Lite JSON spec?

[yitzchak]

The extension is for rendering the graph in a notebook output cell. I think it is still required if you want to do that.

I haven't thought about wrapping the spec much, but I if you are just trying to create inline graphs I would probably just write convenience functions that output the JSON in the above format for each kind of graph you want to do.

Right now common-lisp-jupyter is using jsown internally. That doesn't prevent you from using whatever JSON library you want though. Eventually I want to switch to my own library, shasht, which was designed to pass JSONTextSuite which contains over 300 different parsing tests. You can see differences between the various Common Lisp libraries here. If you are just generating JSON that this probably doesn't matter much, but common-lisp-jupyter is doing a lot of JSON parsing internally.

[Symbolics]

Interesting. I never heard of shasht before. I'll have a look. I might just try to work around any issues in yason with a macro, or maybe work with the yason guys to get those changes moved back in somehow.

[Symbolics]

Can shasht handle encoding of nested alists? For example in the above (though not an alist, imagine it is):

("encoding" (jsown:new-js
      ("x" (jsown:new-js ("field" "a") ("type" "ordinal")))
      ("y" (jsown:new-js ("field" "b") ("type" "quantitative"))))))

?

[yitzchak]

It can. It is in quicklisp, but probably not used by anyone yet. I haven't put much effort in to documentation yet, but you can see some of the configuration stuff in config.lisp

* (ql:quickload :shasht)
To load "shasht":
  Load 1 ASDF system:
    shasht
; Loading "shasht"
[package shasht].........
(:SHASHT)
* (let ((shasht:*write-alist-as-object* t)) 
    (shasht:write-json '(("fu" . 1) ("bar" . (("quux" . 7))))))
{
  "fu": 1,
  "bar": {
    "quux": 7
  }
}
(("fu" . 1) ("bar" ("quux" . 7)))

[Symbolics]

If you are using JupyterLab there is also an extension for Vega-Lite that allows one to pass the JSON based Vega-Lite description right to the front end. Eventually I may make some wrapper code for this.

Do you have an example of passing Vega-Lite JSON to directly to the front end? If, say, we were to encode a spec with an alist like this:

(let ((shasht:*write-alist-as-object* t))
  (shasht:write-json
'(("$schema" . "https://vega.github.io/schema/vega-lite/v5.json")
 ("description" . "A simple bar chart with embedded data.")
 ("data"
  ("values"
   . #((("a" . "A") ("b" . 28)) (("a" . "B") ("b" . 55))
       (("a" . "C") ("b" . 43)) (("a" . "D") ("b" . 91))
       (("a" . "E") ("b" . 81)) (("a" . "F") ("b" . 53))
       (("a" . "G") ("b" . 19)) (("a" . "H") ("b" . 87))
       (("a" . "I") ("b" . 52)))))
 ("mark" . "bar")
 ("encoding"
  ("x" ("field" . "a") ("type" . "nominal") ("axis" ("labelAngle" . 0)))
  ("y" ("field" . "b") ("type" . "quantitative"))))
   ))

Is there a way to have Jupyter Lab render the plot?

[yitzchak]

You don't need to serialize the JSON. You need to pass the JSOWN style json to jupyter:inline-result.

(jupyter:inline-result
  (jsown:new-js
    ("$schema" "https://vega.github.io/schema/vega-lite/v3.json")
    ("description" "A simple bar chart with embedded data.")
    ("data" (jsown:new-js
      ("values" (list
        (jsown:new-js ("a" "A") ("b"  28)) (jsown:new-js("a" "B") ("b" 55))  (jsown:new-js("a" "C") ("b" 43)) 
        (jsown:new-js("a" "D") ("b" 91)) (jsown:new-js("a" "E") ("b" 81))  (jsown:new-js("a" "F") ("b" 53)) 
        (jsown:new-js("a" "G") ("b" 19)) (jsown:new-js("a" "H") ("b" 87))  (jsown:new-js("a" "I") ("b" 52))))))
    ("mark" "bar")
    ("encoding" (jsown:new-js
      ("x" (jsown:new-js ("field" "a") ("type" "ordinal")))
      ("y" (jsown:new-js ("field" "b") ("type" "quantitative"))))))
  "application/vnd.vegalite.v3+json")

Also, it looks like you no longer need to install the separate extension as it now included by default with JupyterLab.

[Symbolics]

I was thinking of a way to save a JSON spec in a variable, or as part of a plot class. The jsown is a bit wordy to work with as a lisp-side spec. Are there any ways an alist can be coerced into the JSOWN (or other format) without inserting a bunch of jsown:new-js function calls into the DSL?

[yitzchak]

The JSOWN objects are just alists with a :obj head.

'(:obj ("a" . "A") ("b" . 28))

[Symbolics]

I see. In fact if you look at the new-js output:

(:OBJ ("$schema" . "https://vega.github.io/schema/vega-lite/v3.json")
 ("description" . "A simple bar chart with embedded data.")
 ("data" :OBJ
  ("values" (:OBJ ("a" . "A") ("b" . 28)) (:OBJ ("a" . "B") ("b" . 55))
   (:OBJ ("a" . "C") ("b" . 43)) (:OBJ ("a" . "D") ("b" . 91))
   (:OBJ ("a" . "E") ("b" . 81)) (:OBJ ("a" . "F") ("b" . 53))
   (:OBJ ("a" . "G") ("b" . 19)) (:OBJ ("a" . "H") ("b" . 87))
   (:OBJ ("a" . "I") ("b" . 52))))
 ("mark" . "bar")
 ("encoding" :OBJ ("x" :OBJ ("field" . "a") ("type" . "ordinal"))
  ("y" :OBJ ("field" . "b") ("type" . "quantitative"))))

I can see the raw JSOWN.

I was thinking that the alist->JSON encoding happened in one of the cells, and then was passed to Jupyter, but it seems instead that Jupyter receives the JSOWN spec to interpret. Are we able to influence this path and somehow send the final JSON spec to the Vega renderer?

[yitzchak]

Its JSOWN style because common-lisp-jupyter is using JSOWN and inline-result just passes the contents directly to the message encoder which handles the serialization to JSON since the Jupyter protocol is JSON layered on top of ZeroMQ.

It might be possible to add some options to inline-result which influence the serialization. I'll add it as an item in the shasht PR.

[yitzchak]

Because of the switch to the shasht JSON library this has changed a bit.

(jupyter:inline-result
  '(:object-alist
     ("$schema" . "https://vega.github.io/schema/vega-lite/v4.json")
     ("description" . "A simple bar chart with embedded data.")
     ("data" . (:object-alist
                 ("values" . (:object-alist ("a" . "A") ("b" . 28))
                             (:object-alist ("a" . "B") ("b" . 55))  
                             (:object-alist ("a" . "C") ("b" . 43)) 
                             (:object-alist ("a" . "D") ("b" . 91)) 
                             (:object-alist ("a" . "E") ("b" . 81))
                             (:object-alist ("a" . "F") ("b" . 53)) 
                             (:object-alist ("a" . "G") ("b" . 19)) 
                             (:object-alist ("a" . "H") ("b" . 87))
                             (:object-alist ("a" . "I") ("b" . 52))))))
     ("mark" . "bar")
     ("encoding" . (:object-alist
       ("x" . (:object-alist ("field" . "a") ("type" . "ordinal")))
       ("y" . (:object-alist ("field" . "b") ("type" . "quantitative"))))))
  "application/vnd.vegalite.v4+json")

Please note that :object-plist works also.

I am working on adding a simple VegaLite function that takes care of the mime type in #72 and possibly allows for sending proper alists.

[Symbolics]

It looks like this is related to issue #80 that I just opened. #72 has been merged; does it allow for sending proper alists?

[yitzchak]

I've switched to shasht so the jsown style encoding won't work anymore. In shasht JSON objects are represented by hashtables, (:object-alist ("fu" . 1)) or (:object-plist "fu" 1). JSON arrays are are represented by vectors or non null lists. nil is true and t is true.

There is also a convenience function (j:make-object) for making hash tables if you need it.

[Symbolics]

The plot specifications for Vega-Lite are all manipulated as alists. How can we get shasht to encode them? The example above:

(jupyter:inline-result
  '(:object-alist
     ("$schema" . "https://vega.github.io/schema/vega-lite/v4.json")
     ("description" . "A simple bar chart with embedded data.")
     ("data" . (:object-alist
                 ("values" . (:object-alist ("a" . "A") ("b" . 28))
                             (:object-plist ("a" . "B") ("b" . 55))  
                             (:object-alist ("a" . "C") ("b" . 43)) 
                             (:object-alist ("a" . "D") ("b" . 91)) 
                             (:object-alist ("a" . "E") ("b" . 81))
                             (:object-alist ("a" . "F") ("b" . 53)) 
                             (:object-alist ("a" . "G") ("b" . 19)) 
                             (:object-alist ("a" . "H") ("b" . 87))
                             (:object-alist ("a" . "I") ("b" . 52))))))
     ("mark" . "bar")
     ("encoding" . (:object-alist
       ("x" . (:object-alist ("field" . "a") ("type" . "ordinal")))
       ("y" . (:object-alist ("field" . "b") ("type" . "quantitative"))))))
  "application/vnd.vegalite.v4+json")

Looks like it requires :object-alist to be inserted into the alist structure. I don't see an easy way to do that in my quick look at shasht. Did I miss it?

[yitzchak]

shasht does have a way to do this (shasht:*write-alist-as-object*), but it is not going to work in this case because the JSON that you are passing to inline-result is just a fragment in a larger JSON structure. Basically, this (j:inline-result "foo" "application/vnd.vegalite.v4+json") gets sent as message content in an execute_result message:

{
  "execution_count": 3,
  "data": {
    "application/vnd.vegalite.v4+json": "foo"
  },
  "metadata": {}
}

When you use :display t the message type is display_data instead. Which is displayed in the notebook, but not retrievable as a REPL result via * or /.

Probably the long term answer is to provide some keys to control the conversion on inline-result such as :object-alist t to convert alists to the right internal format. I don't want to do that by default because alists and plists are not unambiguous structures.

[Symbolics]

Is there a short-term work-around?

[yitzchak]

Well you could write a recursive function that looks for alists and does (cons :object-alist my-alist) on them. But more specifically, how are you generating the data?

[Symbolics]

There are wrappers around the 'grammar of graphics' for Vega-Lite. Examples and documentation explain it better than I could here.

[Symbolics]

Perhaps the path of least resistance is to revert to an older version and hold there until shasht alist processing is worked out? I upgraded to get the markdown streams, which happened in commit cda458b on 26 April. Jsown was removed in 2511acb on 3 March.

Do you think it would be possible to revert to a version prior to 2511acb and then cherry pick the markdown stream changes?

[yitzchak]

alists are a fundamentally ambiguous, but convenient data structure in my opinion. My reading of your documentation and code seems to indicate thatt that you are supporting two different JSON encoders (yason and shasht) and two different environments (Jupyter and static web pages). This is probably making things even more complex.

I am currently investigating my own wrapper for VegaLite and I am starting to come to the conclusion that it would be best to delay the JSON encoding until the very last minute. In other words, encode the graph as combination of classes and then override shasht:print-json-value or yason:encode to do the last minute encoding. This could also be done with your data frames so that conversion to JSON is done only when the objects are serialized.

I know that doesn't necessarily answer your question. If you want to continue to use alists you could do the following.

(defun alistp (value)
  (and (listp value)
       (every #'consp value)))

(defun convert (value)
  (if (alistp value)
    (cons :object-alist
          (mapcar (lambda (pair)
                    (cons (car pair)
                          (convert (cdr pair))))
                  value))
    value))

Then you could do the this (j:vega-lite (convert alist-data))

[Symbolics]

I do use yason, but only because an earlier system I was using had it. I agree that alist encoding can be ambiguous, so some convention is probably required. Both jsown and yason produce the same alist output (depending on settings), and there is something to be said for consistency in implementations.

It might not be clear from the documentation (and if it isn't, I should change it), but the data frame is a data manipulation structure and the specification for a plot an alist. The alist turns out to be quite a convenient representation of Vega-Lite plots, and has the advantage of manipulation with standard functions and libraries. I had initially went down the object route (that's where the yason came from), but there was little return for the additional complexity.

If you're still considering options, you might want to look at using Lisp-Stat's vega-lite wrappers. It would be unfortunate to have duplicates of the same functionality in the Lisp community, and a lot of benefit to having a single vega-lite project to gain critical mass.

[Symbolics]

is (j:vega-lite .. working? I am trying to fix the breakage in the first Lisp-Stat notebook and when I try the suggestion above the conversion to shasht format seems to work:

(clj:vl-to-shasht online-bar-chart)
(:OBJECT-ALIST ("$schema" . "https://vega.github.io/schema/vega-lite/v5.json")
 ("data" :OBJECT-ALIST
  ("values"
   . #((("SOURCE" . "Other") ("COUNT" . 19))
       (("SOURCE" . "Wikipedia") ("COUNT" . 52))
       (("SOURCE" . "Library") ("COUNT" . 75))
       (("SOURCE" . "Google") ("COUNT" . 406)))))
 ("mark" . "bar")
 ("encoding" :OBJECT-ALIST
  ("x" :OBJECT-ALIST ("field" . "SOURCE") ("type" . "nominal")
   ("axis" :OBJECT-ALIST ("labelAngle" . 0)))
  ("y" :OBJECT-ALIST ("field" . "COUNT") ("type" . "quantitative"))))

but the call to vega-lite gives me:

interrupt: Execution interrupted

debugger invoked on a TYPE-ERROR in thread #<THREAD "SHELL Thread" RUNNING {1005D4FF13}>: The value "Other" is not of type LIST

The current thread is not at the foreground,
SB-THREAD:RELEASE-FOREGROUND has to be called in #<SB-THREAD:THREAD "main thread" RUNNING {1002320003}>
for this thread to enter the debugger.

[yitzchak]

Looks like the alist markers are missing in the vector. Try this.

(defun alistp (value)
  (and (listp value)
       (every #'consp value)))

(defun convert (value)
  (cond
    ((alistp value)
      (cons :object-alist
            (mapcar (lambda (pair)
                      (cons (car pair)
                            (convert (cdr pair))))
                    value)))
    ((vectorp value)
      (map 'vector #'convert value))
    (t
      value)))

[Symbolics]

That seems to vectorise everything:

(clj:vl-to-shasht online-bar-chart)
(:OBJECT-ALIST
 ("$schema"
  . #(#\h #\t #\t #\p #\s #\: #\/ #\/ #\v #\e #\g #\a #\. #\g #\i #\t #\h #\u
      #\b #\. #\i #\o #\/ #\s #\c #\h #\e #\m #\a #\/ #\v #\e #\g #\a #\- #\l
      #\i #\t #\e #\/ #\v #\5 #\. #\j #\s #\o #\n))
 ("data" :OBJECT-ALIST
  ("values"
   . #((:OBJECT-ALIST ("SOURCE" . #(#\O #\t #\h #\e #\r)) ("COUNT" . 19))
       (:OBJECT-ALIST ("SOURCE" . #(#\W #\i #\k #\i #\p #\e #\d #\i #\a))
        ("COUNT" . 52))
       (:OBJECT-ALIST ("SOURCE" . #(#\L #\i #\b #\r #\a #\r #\y))
        ("COUNT" . 75))
       (:OBJECT-ALIST ("SOURCE" . #(#\G #\o #\o #\g #\l #\e))
        ("COUNT" . 406)))))
 ("mark" . #(#\b #\a #\r))
 ("encoding" :OBJECT-ALIST
  ("x" :OBJECT-ALIST ("field" . #(#\S #\O #\U #\R #\C #\E))
   ("type" . #(#\n #\o #\m #\i #\n #\a #\l))
   ("axis" :OBJECT-ALIST ("labelAngle" . 0)))
  ("y" :OBJECT-ALIST ("field" . #(#\C #\O #\U #\N #\T))
   ("type" . #(#\q #\u #\a #\n #\t #\i #\t #\a #\t #\i #\v #\e)))))

[yitzchak]

Probably just have to put a tighter restriction on.

(defun alistp (value)
  (and (listp value)
       (every #'consp value)))

(defun convert (value)
  (cond
    ((alistp value)
      (cons :object-alist
            (mapcar (lambda (pair)
                      (cons (car pair)
                            (convert (cdr pair))))
                    value)))
    ((typep value '(vector t *))
      (map 'vector #'convert value))
    (t
      value)))