Docs about serialization

[souenzzo]

I'm using ubergraph on backend and I want to serialize in some "json friendly" format. What are possibilities?

[Engelberg]

Ubergraph data structures don't serialize well out of the box because of their use of atoms to hold cached hash values. There's probably a way to workaround that, but I think the simplest approach would be to convert the graph into a simple edn data structure containing enough info to rebuild the graph.

Try this code out:

(defn ubergraph->edn [g]
  {:allow-parallel? (:allow-parallel? g),
   :undirected? (:undirected? g),
   :nodes (vec (for [node (nodes g)] [node (attrs g node)]))
   :directed-edges (vec (for [edge (edges g) :when (directed-edge? edge)]
                             [(src edge) (dest edge) (attrs g edge)]))
   :undirected-edges (vec (for [edge (edges g) :when (and (undirected-edge? edge) (not (mirror-edge? edge)))]
                               [(src edge) (dest edge) (attrs g edge)]))})

(defn edn->ubergraph [{:keys [allow-parallel? undirected? nodes directed-edges undirected-edges]}]
  (-> (ubergraph allow-parallel? undirected?)
      (add-nodes-with-attrs* nodes)
      (add-directed-edges* directed-edges)
      (add-undirected-edges* undirected-edges)))

This turns the graph to and from a serializable Clojure data structure. To turn it into json, I recommend the cheshire library which should easily turn this to and from json. I've used cheshire very successfully in the past.

Of course, this code assumes that your nodes themselves are simple Clojure data that can be easily converted to json by cheshire.

Please let me know how this works out for you. This is something I haven't thought about before, but now that you've brought it up, it seems like the kind of thing that might be generally useful, so may warrant adding to ubergraph.

[souenzzo]

Sorry, wrong click (open/close). I will try it. But now I'm looking to viz-graph. If it uses viz, it serializes to dot language, so serialization is possible. I will try both approaches and give some feedback.

[Engelberg]

Yes, viz-graph uses a similar strategy as the code I showed above, as does uber/pprint. The main thing that is missing from this approach is the specific UUIDs used to store the edge information. When you deserialize the data structure, it will generate new UUIDs as it builds the graph. Also, the way I wrote this deserialization routine will effectively fill in missing attribute maps nil with the empty map {}. However, the ubergraph API is written in such a way to make those distinctions meaningless, for example, attrs returns the empty map {} regardless of whether it is holding nil or {} and equality and hashing are overridden so that the specific UUIDs don't matter. So you get back an "equivalent" graph, which is all that should really matter.

If you want a more literal serialization, the strategy would be to override the print-dup method for Ubergraph, like so:

(defmethod print-dup ubergraph.core.Ubergraph [o w]
  (print-ctor o (fn [o w] (print-dup (:node-map o) w) (.write w " ") (print-dup (:allow-parallel? o) w) (.write w " ") (print-dup (:undirected? o) w) (.write w " ") (print-dup (:attrs o) w) (.write w " ")
                  (print-ctor (:cached-hash o) (fn [o w] (print-dup (:cached-hash o) w)) w)) w))

Then, you can serialize to a string with:

(binding [*print-dup* true] (pr-str my-graph))

and deserialize with read-string.

This gives you back exactly the same data structure you had before. I didn't mention this technique before because this uses Clojure's ability to serialize to a string, and you specifically said you wanted to serialize to a json data structure. (I guess in some sense, a string counts as json data).

If you wanted to serialize to a more compact binary form, you could use something like nippy and override its multimethods similarly to what I did above for Ubergraph.

As before, let me know if this is useful for you, so I can consider for this general inclusion.

[souenzzo]

Far from ideal

(defn to-svg [g]
  (with-redefs [dorothy.core/save! (fn [graph f & [options]]
                                     (let [bytes (dorothy.core/render graph (merge options {:binary? true}))
                                           output (new ByteArrayOutputStream)]
                                       (jio/copy bytes output)
                                       (.toString output)))]
    (uber/viz-graph g {:save {:format :svg}})))

can solve my problem. It's not exactly serialization of graph, because there are several decorations in my graph, this seems to be the only sane solution (without reimplement all the decoration on the front).

I think that a possible standard solution could be

ubergraph.core/export-to-dorothy-disgraph - A primitive version of viz-graph, that I can manually pipe into d/disgraph d/dot and do stuff.

dorothy.core/save - ("not your problem") A primitive version of save! that returns a buffer or receives a writer, so I can do this output to an HTTP interface or something like (I can output PNG directly on HTTP!!)....

If you are interested, I can do a pull request ;)

[Engelberg]

I don't fully understand what your overall objective is. You talk about "serializing the graph", but you seem to be meaning something different than what I assumed, and your interpretation has something to do with visualization.

[Engelberg]

I've included the general-purpose serialization improvements that we discussed early on this thread as an official part of ubergraph (0.4.0). I think what you're saying is that ideally you'd like to have a version of viz-graph that stops and returns the result after the call to dot, without proceeding to save a file. Is that correct?

[souenzzo]

My initial goal was "ubergraph->edn and edn->ubergraph". sorry for the mess thank you so much :D