idea: perf gains and data abstraction

I had been pondering the idea of adding a new layer of abstraction over your data source originally to support (frontend) use cases such as:

use different storage for different perf characteristics, like using a datascript/doxa db for some data and a hashmap for others
partition multiple datascript/doxa dbs for the same reasons you partition a backend db.

The implementation change would be that your data source is a hashmap whose values are reagent atoms.

The probing question behind this is "when the data changes, why do we have to run every layer2 sub to find out what changed?" Can we change the design of the library to support atomic change detection?

Some API ideas:

(defonce my-app-storage 
    {:datascript (r/atom (datascript/db ,,,))
     :todo/id {1 (r/atom {:todo/id 1 :todo/name "todo one"})}}
     :users/id {1 (r/atom {:user/id 1 :user/name "user one"})}})

;;( reg-sub <top level key> <subscription name key> [usual subsription handler arguments])
(reg-sub :datascript :some-app-sub (fn [datascript-db] ,,,))
(reg-sub :users/id :my-users-sub  (fn [users] ,,,))

or have a separate API specifically for layer 2 subs. - all layer 2 subs are individual ratoms.

I wasn't sure this was a good idea, but coincidentally Vincent was thinking up similar ideas here: https://www.youtube.com/watch?v=oDiZxi5FRRc&t=2817s so perhaps it is worth looking into.

Like all designs there are tradeoffs. Your data manipulation code would have to be aware of these ratoms.

Some ideas:

Implement a hashmap-like custom data type that handles this abstraction

https://funcool.github.io/clojurescript-unraveled/#associative

assoc and get would just detect if you have a deref'able and deref it. The test if this works is that you should be able to use fulcro and its merge functions and have those continue working.

The subscriptions library side would be easy enough to change to support this using a middleware pattern at the implementation level of reg-sub, that way you can opt in to this as a user.

edit:

oh wow - had an insight:

add a new fn to the API something like reg-layer2-sub which takes one hashmap of args and returns a path vector into app db. This path will be used to create a reagent cursor into the reactive atom appdb.

(reg-layer2-sub ::todos (fn [args] [:todo/id]))

In the library implementation this "handler" will produce a reagent.ratom/RCursor type instead of a reaction. This will allow non-breaking changes to the API.

The idea is that all layer2 subs in an application will use this method and thus when appdb changes only the cursors whose values have changed will fire. This should be much more efficient because there is no computation function, the cursor will stop propagation using a reference equality check: https://github.com/reagent-project/reagent/blob/9ddf191a6a7338d226161e8af0d7ecc0bc2a6bae/src/reagent/ratom.cljs#L265

For applications that use a hashmap an additional idea would be to force the use of this API by having reg-sub only be used for creating layer3 subs (remove the single computation fn arity, or throw on its use).

matterandvoid-space / subscriptions

idea: perf gains and data abstraction #5