Architecture Improvement: Priority Queue as Central Processing Mechanism

[awkay]

One of the big weaknesses of the internal architecture is the fact that transactions, which cause all forward movement of the library, are processed "immediately" on the calling thread and have no facility for dealing with nesting (generally undesirable, but sometimes structurally necessary), timing, and most importantly: data dependencies.

The transact! mechanism, as it exists, has always had an ordering semantic for a single tx expression, but only for the optimistic updates and operations that happen on the same remote (a simple queue). This has been expanded over time to try to address this in a somewhat ad-hoc manner: ptransact! and incubators more useful variant of the same-named function allow for a slightly expanded form of runtime order dependency: that of running something after the remote behavior. This latter expansion essentially provides what vanillajs accomplishes with promises and then.

Some of the weaknesses of all of the above systems are:

• The flat nature of transactions has the advantage of simplicity, but unfortunately it is too simplistic. Sometimes the side effect of a transaction needs to be a new transaction. Being able to specify when that new transaction takes effect in a way more trustworthy than setTimeout is needed. (setTimeout, for example, doesn't compose well when there might be more than one thing being deferred..you get arbitrary ordering).
• No ability to reliably say X comes after Y in a compositional sense. Similar to the prior item, but there may be various dependency "reasons". E.g. "X comes after the remote response of Y", "X runs immediately after 'whatever is already scheduled to run' (imply Y)", "X should run as soon as possible, but after Y which was scheduled by time", "X should run after everything that is scheduled to run", etc.
• Difficult to analyze the current "list of things to do" and potentially cancel something.

I propose extending the internal architecture of Fulcro to support a much richer form of ordering and dependency management for operations. Specifically I plan to place a form of priority queue in place between the running of transact! and the processing of those items.

This would also affect loads, and would allow post mutations to become first-class mutations that are queued into this same priority queue. The current existence of load-action might be affected by this change, though, since the current implementation of loads can bypass the direct use of transact by parasitically "morphing" the remote of the mutation in which they are used. This has always been a hack around the original inherited architecture of Om Next. We will technically still need to support this use-case (which is just rewriting the remote AST), since it is widely adopted and used.

[awkay]

Some Design Aspects to Refine:

• [ ] What are the dependency cases? I.e. what should a new transaction be allowed to indicate as a "thing to wait on"?
• [ ] What notation should be used to indicate the dependencies?
• [ ] Is the dependency specification at the "transaction" level or "mutation" level?
• [ ] How do we cancel something in the queue
• [ ] Instrumentation hooks
• [ ] Dependency combos: e.g. "run 200ms after the queue is otherwise empty and idle", "Run before any transaction containing the mutation X (implies latest possible position that is still before X)", "Run after all of the optimistic activity that was queued at the time of being added to the queue"

[claudiuapetrei]

Hi,

Reading the https://github.com/fulcrologic/fulcro/blob/feature/react-play/docs/RFCs/RFC-transaction-semantics.adoc

Sounds really great :smile:

A few thoughts, don't know if there any good, hard to figure out all the use-cases and implications:

• Having group leader seems a bit hard to reason about (for separate transacts) first thing that came to mind was a group "registry" that would encode the semantics, that way different order or removing a transaction would not affect the behavior.
• Don't know if it's currently possible but would be nice for nested transactions like (transact this [(a) (b) (c)]) if we could have a way for a to terminate the rest. A special return value that would cause transact not to run b & c.
• Would be nice to have a way to tell the transact that I don't want it to refresh the UI (mutation and/or group level) even though I run them with a component or reconciler + ident

[awkay]

Thanks for the feedback @claudiu-apetrei .

• The group leader idea has to do with some known context wanting to group together the known transactions in the presence of calls where you don't know what else might be submitted. It is never intended to be something where transacts spread across the code share the same group, except in more global cases. Global use cases that are trivial to reason about are application startup (a :startup group). Localized uses of groups are things like the dynamic router, where the transacts the router is submitting need to go before anything a route target might submit.

• For "terminate the rest": Remember that there is an optimistic phase and pessimistic, and any group of things you put together like this are really meant to run as a group. That is the external semantic of transact (you submit a sequence of operations). The proposal does give you a way to indicate how the sequence runs...I'll think about this one. I've typically resisted this notion because it falls under flow control, and I don't want to move too far towards a programming language within the language (your suggestion is akin to adding when-not for some special condition). You can already do flow control semantics by writing a new mutation that does the composition and control, and the priority queue semantics that I'm suggesting adding give you a "proper way" to call new transactions within a transaction (e.g. the rule suppressing a run of transact in a mutation can be relaxed).

• UI refresh. I'd be interested in hearing your use case. It isn't actually hard to implement, but it had not occurred to me to be a need.

[claudiu-apetrei]

Hi,

• For the group would something like (transact-with-options {...} (transact! ...) (let [n (f x)] (transact! [(a n])) also be a possible option ? Was my first thought when reading the spec and was wondering in what scenarious having group leader would be a better fit.

• Not exactly sure what could be the downsides. Was recently thinking how nice it would be to have (ptransact! this [(validate-form)(submit)(change-route)]).
  To have this I would just need to have a way to abort from the mutation (validation fails or i reach the submit error). Ex returning ::f/stop Since they have side-effects seems like without a break in flow you have to move composition somewhere in the internal implementation of the mutation most of the time

• Was thinking of situations where preload-data for a component thats not mounted yet and the use set-query to change the route. Transact on reconciler would do a full refresh. Cant remember if adding the ref still triggered the full refresh. Thought it might be nice to have a way to load data thats not on screen and specify to fulcro that no extra ops need to be done (good chance this might be over optimization or something that you would only see if you need to keep 60fps in the UI)

[awkay]

So, I don't think you quite understand the group leader stuff...perhaps re-read the RFC? Your example is waaay too syntactic (e.g. extending the language, not the semantics). I really don't want to go that route.

On "abort": You just store that flag in your state, and any of the following mutations can look for it and become a noop...that is the intended place for flow control: in the mutations themselves.

On refresh: what does it matter if it does a refresh when there is nothing affected on-screen? That is more an issue of optimization, and doesn't belong in the semantics. The other :after options on the semantics, I think, can possibly solve any real cases, and UI display is another one where what you put in state is under your control, and what is in state is what renders.

On the optimization side: I do plan on letting you specify the scheulding of transaction queue processing, where you could indicate they are done on RAF, to prevent more than 60fps processing.

[claudiu-apetrei]

@awkay Thank you for answering my questions smile On the abort part. I sometimes use a marker in state but feels like I'm coupling things and most of the time I go with having a mutation implementation mutation-name* and just compose those in a single mutation. It's pretty ok, but just feels like there's something missing.

I get why adding flow control might me a bad idea and source of complexity. Would you consider passing the return of mutations to the next one in the new implementation ? Ex: (transact! this `[(a) (b) (c {:x 1})]) the return value from a would be available in b and the return value from b be available in c. For ptransact if there is no remote action fired it will pass the optimistic return value otherwise the ok or error return value.

[awkay]

Hi, I got on a bit of a ramble here. The short answer to your question is "you can already do that via app state", the slightly longer answer is "nesting mutations might solve your problem", and then I go off on some related thoughts and rambling... :)

It really would not be hard to put an atom in env that can be used to track the progress of a tx, but it would have to be an atom (the env is actually closed over when the mutation is dispatched, not every time a section runs). I'm not sure this makes much more sense than just leveraging what you already have (the state atom) as a means of communication.

That said, I am introducing (with queues) a formalism that allows for a transact to safely appear within an action. This alone is huge change, as it allows the decision logic of a mutation in the top-level transaction to submit additional top-level transactions (to the end of the queue, of course). This is a form of application-centric flow control in a way, since logic in an action can choose additional transactions to submit that are really not visible from the original. I've resisted this for a long time, because I see it as a way for chains of complicated logic to get out of control at the mutation layer. That said, I've come to the conclusion that it is not necessary or desirable to constrict where a new top-level transaction can be submitted, as long as you control how that is interpreted.

So, for your return value question: I think there are two answers in Fulcro 3:

• The traditional Fulcro 2 answer: put the value in app state and let the next mutation find it there. This is exactly equivalent to a "return value", you just have to name it.
• The Fulcro 3 answer: Use the optimistic action of the first to submit the second. That is:

;; somewhere in UI
(transact! x `[(f)]`)

(defmutation f [params]
  (action [env] 
    ;; Fulcro 3 only, or sort-of ok as `ptransact!` in 2.
    (transact! env `[(g {:data-from-f 42})]`)))

In terms of "feels like something is missing". I get that for Fulcro 2. It's been nagging at me for years as well, and I have seen it as a nesting problem. What is the best way to deal with the idea that there is logic "within" a transaction that needs to be expressed?

In the past the answer has been "write mutation helpers", but then we found cases where only ptransact! would work. With even more complicated scenarios (composition of dynamic routers with state machines that allows their own form of nesting and cross-communication) has it finally become more obvious that the real central problem is that transact! itself "runs when called" instead of using a queue. This means that the actions inside of that call should not call transact! because that has the effect of "interrupting" the top-level transaction and processing a nested one within it. Nested transactions are just a big ball of complexity. The top-level reasoning just gets destroyed. It is my hope that moving this to a queue will give us a more solid model (that isn't based on the opaque js setTimeout event queue). Reasoning about a single tx (no matter from where it is run) is now able to be "locally atomic" in the sense that (at least for local optimistic effects) an optimistic transaction will run as a single serialized unit at some future time that will not mix with or interfere with those that were submitted before it. (like the SERIALIZABLE isolation level of SQL). At the end of the day this is really what we mean when we talk about "transactions": a group of operations that runs together. We can further ensure that the remote effects of a transaction preserve their serial ordering and grouping (per remote).

Note that the setTimeout solution (which is what ptransact! and others do to avoid this problem) was already essentially a queue solution: js does in fact run these in the "order of submission". The proposed solution makes this "our queue", and therefore amenable to customization, analysis, and finer control.

My primary goal for this expansion is to clean up the API and make the use of transact! make more sense throughout the entire program. I'm almost certainly not going to implement the :after clauses (they are too complex and mostly not necessary), but I am giving transactions a "findable" ID, allowing for some kinds of transaction middleware and algorithm customization, and making the API usable in a consistent manner everywhere. This is also leading to a natural way to further minimize network traffic through "grouping", which isn't going to require the option described in the RFC...I've found a better way to do it I think.

[awkay]

The transaction processing has been rewritten in F3, and contains many of the ideas in this issue. It is also somewhat pluggable so that it can be tuned more easily over time.