Add support for update

[jeluard]

Virtual DOM like libraries are gaining traction in JavaScript / ClojureScript world thanks to their hability to simplify UI construction while still being efficient. Hiccup representation and macro compilation might be leveraged to improve efficiency of underlying algorithms.

First approach: diff / patch

ReactJS (and most others) relies on a diff / patch mechanism to update a live element.

(defn my-list
  [m]
  [:ul#id.class
    (for [u (:users m)]
      [:li {:id (:id u) :class "some-class"}
        [:span {:class "some-other-class"} (:name u)]])])

(def m1 {:users [{:id 1 :name "john"}]})
(def h1 (my-list m1))
(let [el (hipo/create h1)]
  (.appendChild js/document.body el))

Update would then be:

(def m2 {:users [{:id 1 :name "john"} {:id 2 :name "paul"}]})
(let [h2 (my-list m2)]
  (hipo/update! el (hipo/diff h1 h2)))

Where diff generates a diff representation of both hiccup vectors and patch! applies this diff to the live DOM element. This process implies a complete tree diff to identify potential changes. See [1] for more details about how ReactJS does this.

Second approach: macro compilation

Now for most cases a complete diff is not necessary as the general element shape is usually kept intact. In our example only li children change, in one of 3 ways:

• the text content of the span child an identified li is updated
• an identified li is removed
• an identified li is added

(position changes might be optimised in term of DOM operations but are essentially combination of removal and addition)

If we had to hand write the update! code it would look like:

(defn update-li!
  [el m1 m2]
  (let [n2 (:name m2)]
    (when-not (= (:name m1) n2)
      (set! (.. el -firstChild -textContent) n2))))

(defn update!
  [el m1 m2]
  (when-not (= (:users m1) (:users m1))
    (doseq [[lel ml1 ml2] ...] ; identify updated lis
      (update-li! lel ml1 ml2)))
    (doseq [lel lels] ; identify removed lis
      (.remove iel))
    (doseq [[i m] ms] ; identify new lis
      (insert-at! el i (create m)))))

This code only focuses on changes to the :users key and associated hiccup tree and ignores everything else. In essence hiccup vectors are not directly diffed but we use the underlying data to drive the diff.

Obviously while optimal it is very time consuming and error prone to create such function for every component of your application.

The goal here is to have a macro generates those update! method for us. When the macro can't compile anymore the original approach (diff/patch) is used for the remaining trees.

Proposed syntax

(deftemplate my-list
  [m]
  [:ul#id.class
    (for [u (:users m)]
      [:li {:id (:id u) :class "some-class"}
        [:span {:class "some-other-class"} (:name u)]])])

(def el (my-list m1))
(.appendChild js/document.body el)

...

(hipo/update-state! el m2)

deftemplate would generate at compilation time the most optimal possible update! function by identifying where and how m is used. In the extreme case (m is not used, only static data) update won't do anything!

Obviously this example is a simplistic case but it should highlight the general idea.

[1] http://calendar.perfplanet.com/2013/diff/ http://facebook.github.io/react/docs/reconciliation.html http://facebook.github.io/react/docs/multiple-components.html#child-reconciliation

Note that ReactJS is moving to a new internal element representation that is surprisingly similar to hiccup syntax. See https://facebook.github.io/react/blog/2014/10/14/introducing-react-elements.html#third-party-languages

[jarohen]

(same note as posted to CLJS list - feel free to continue the conversation wherever's easiest :) )

If I understand you correctly, the approach you're looking into for Hipo is what Flow (https://github.com/james-henderson/flow) currently offers - you refer to it as option 2 - expanding the Hiccup forms at compile time and generating the necessary update code. In fact, the code you specify is quite similar to the code that Flow generates :) One area that Flow doesn't currently do a lot in is static analysis - inferring (where possible) the state -> element dependencies at compile-time. It currently does this at runtime, but I'd like to move as much of this as possible to compile-time in the next version of Flow. I've had a few thoughts regarding how to do it, but haven't made much headway as yet - if you've got ideas about what could be inferred, what properties such an analyser should look for, or how it could be implemented, it'd be great to chat through them - it seems like we've got very similar plans here :)

Flow does, however, maintain as much of the DOM as possible - you say '[in] most cases a complete diff is not necessary as the general element shape is usually kept intact. In our example only li children change, in one of 3 ways: ...'. I absolutely agree - this is one of the core implementation principles of Flow (you can see it in action at https://github.com/james-henderson/flow/blob/0.3.0-branch/src/flow/forms/node.cljx#L81, if you're interested - each node is only created once ($el, here) and we can return that node every time, just updating the styles/attrs/children if necessary)

Obviously, if I've misunderstood the aims behind Hipo, let me know :) Likewise, if Flow is doing a fair proportion of what you want to achieve in Hipo, but is missing a killer feature, or isn't quite how you'd like to express UIs, please let me know - I'd really appreciate the feedback!

As an aside, if it helps, I did consider your option 1, but decided against it on the grounds that Facebook/React/Om etc will probably do a much better job of fast DOM diffing than I could ever manage! If nothing else, given the popularity of Om+React within the ClojureScript community, any alternative solution based on DOM diffing would have to be much better to justify fragmenting the community :) Don't let me put you off though - if nothing else, it'd be a very interesting/challenging project!

Cheers,

James

[jeluard]

Yes I believe you understand right! There is on gotcha though: you cannot rely fully on the macro solution as there is always the possibility that some info is not known at compile time (say retrieving data using an HTTP client). In this case unless the macro walking is very sophisticated you will need a fallback solution: in my case this is the traditional diff based approach.

About React

Frankly the diff part is not rocket science and there are already a bunch of alternative implementation doing much better perf wise. And we have access to macro for some crazy magic. Now there is always the possibility that I am missing a crucial point :)

Your point about fragmentation is a strong one. Unfortunately React story here (with the React components) is pretty weak IMHO. You just can’t compete with Web Components and google pushing very strong behind.

Also React does a lot of things on top of the plain diffing. Specifically ClojureScript libraries have to hack around the fact React is not immutable first which sounds backwards to me. Finally I feel much better without anything Facebook related in my codebases but that’s just a matter of taste ;)

[jarohen]

Yes I believe you understand right! There is on gotcha though: you cannot rely fully on the macro solution as there is always the possibility that some info is not known at compile time (say retrieving data using an HTTP client). In this case unless the macro walking is very sophisticated you will need a fallback solution: in my case this is the traditional diff based approach.

Agreed :) Even without retrieving data through an HTTP client, I think it's still pretty difficult to know the result (or even result shape) of an arbitrary function call and, as you say, this then requires a fallback. (Flow's diffing fallback, incidentally, is implemented at https://github.com/james-henderson/flow/blob/0.3.0-branch/src/flow/dom/diff.cljx - it's about 100LoC so probably plenty of room for performance improvements). Where I'm thinking about going with Flow is to understand the shape of the underlying state (rather than the DOM) at compile time and then, for any given change in state, we can then infer what changes need to happen on the DOM.

I also agree with your points about React - I can't believe this wouldn't be quicker if it was immutable although, if I understand correctly, Om/Reagent mitigate a fair bit of this by overriding 'shouldComponentUpdate' to take immutable data structures into account?

Openly, it seems like the current implementation of Flow and your proposals for Hipo have a lot in common - is there anything in Flow/missing from Flow that's stopping you from using it? Chances are, if you're missing functionality, or think it's not lightweight enough, other people do as well - and if that's the case, I'd obviously like to try to fix it :)

[jeluard]

I can't believe this wouldn't be quicker if it was immutable ...

True. Still there is a bunch of stuff involved that are not needed in a hiccup / ClojureScript first implementation. The performance cost gain is probably marginal here but performance is the main React (and Om) selling point.

is there anything in Flow/missing from Flow that's stopping you from using it

I really need to dig more into it. My main issue so far is that it does too much. I like the idea of a lightweight library I can bend to my need of the day. I might not always need dynamic component. Or might want to wrap my component as a Web Component (using lucuma). Or use datascript for data access. Flow uses f/el to create component, probably to allow reuse. Here I believe Web Components are a better solution and thus I would not need that extra layer. Another point is I stick with 100% hiccup support so that server side rendering is transparent.

I feel like it is simpler to provide that kind of flexibility with a lower level library. It makes it also easier to reason about and potentially push optimizations further.