custom cell propagation logic

[thedavidmeister]

currently javelin checks if a value is = before propagating it and this is hardcoded.

the rationale is that we can save more CPU "downstream" in the graph than we spend calculating the equality of old/new values at each step of the propagation "upstream".

this is a fair and valuable assumption most of the time, but is not always true, for example i ran into an edge case with a datascript db in a cell. in this case, every new transact! call against the cell triggers an equality check where:

• the equality check can be very expensive as every datom in the db needs to be checked individually
• the equality check is almost guaranteed to return false as transact! almost always changes something in the db

in my personal experience, this unnecessary equality check for a modestly sized (< 500 datoms) datascript db was adding up to 13ms to UI updates in a hoplon app, which makes it hard to avoid jank in some situations.

datascript offers other challenges too, as entities are equal as long as their entity ids are equal, regardless of the values of their attributes... this means that entities cannot usefully be the values of cells, as regardless of their value in the db, they never propagate under a straight = check.

even ignoring datascript, when we look at hoplon, the main consumer/implementation of javelin we run into even more problems such as https://github.com/hoplon/hoplon/issues/194 when we need to trigger callbacks with side effects in response to upstream cells changing their value, but using the downstream cell's value.

examples of this:

• triggering a :focus event whenever an upstream cell's predicate is satisfied using the true value of a downstream cell (workarounds include juggling uuids or cycling artificially between true/false - but false triggers a focusout which can be undesirable)
• trying to sort a list of form fields based on user input (see the above issue)
• adding an add-watch to a cell to respond to updates with a callback (add-watch also seems to only trigger callbacks after an equality check, whereas native atoms unconditionally trigger the callback)

i found the last example odd as the update callback for a lens is always triggered, even if sequential reset! calls against a lens "set" the lens to an equal value, but reset! calls against a normal cell don't trigger any callbacks if subsequent values are equal.

ultimately though, my understanding is that the inclusion of the = check in cell propagation is purely a performance tweak, effectively working as a network of "mini caches", and has nothing to do with the "correctness" of javelin (if anything it can undermine the correctness). most cache systems need a mechanism to deal with or explicitly avoid side effects and unusual invalidation logic eventually...

[thedavidmeister]

maybe relevant https://github.com/hoplon/javelin/issues/35

[kennytilton]

when we need to trigger callbacks with side effects in response to upstream cells changing their value, but using the downstream cell's value.

The classic reactive glitch! I was amused to see MobX hit on the same solution as Cells: any time a formulaic value is read (such as by a side-effect callback) the dataflow engine ensures it is current. Now define "current". :) I was again interested to see MobX came up with a different approach than Cells to currency. They propagate out hard and soft warnings about values needing to be recalculated. In Cells I have a simple integer "pulse" and every cell keeps track of the pulse with which it is current. hth.

[alandipert]

I think I've learned enough to finally have an opinion on this. I think it's an awesome idea.

Javelin was designed to seamlessly wrap Clojure data, using Clojure's own value semantics as the propagation trigger. Working in Clojure data is usually pleasant, and event semantics are easily simulated with counters.

However, working with different data structures that stretch Clojure's own idioms - mutable ones, in particular - demand pluggable equality, particularly when such structures cannot easily participate in Clojure's world of values.

One way to achieve this is to add to the cell constructor and equality function. Another, which I have prototyped in Clojure, is to factor "propagation-ability" into an API that anything can participate in. For example, a JavaScript Array might extend a hypothetical "Propagate" protocol that augments Array to take dependencies and be changed in a way that triggered Javelin.

In one system I recently prototyped in JS, the unit of cell and value was a mutable collection of tuples similar to datascript. When the tuples changed, dependencies received a diff - the set up tuples added and tuples removed - instead of the full new set of tuples. Thus, consumers were free to either maintain their own full or partial dependency values, or to just process diffs incrementally.

This empowered dependencies that managed DOM resources to perform incremental updates.

[alandipert]

Oh, here's a list of things in the wider ecosystem that have either moved my thinking forward or were just interesting:

• https://github.com/datalogui/datalog (closest thing to what I made with the tuple diff)
• https://materialize.com/ (from Frank McSherry who's done a lot with "differential" data)
• https://www.starbeamjs.com/ (seen but not studied)

[alandipert]

Also worth mentioning: Javelin is already meaningfully extensible here since ClojureScript's = is extensible via cljs.core/IEquiv protocol. Micha pointed this out to me.

So, one way to get Javelin to propagate the way you want currently is to customize IEquiv/-equiv for a type you own and stick those in cells.