In @odk/web-forms there is currently an informal separation between what we're calling the "engine" (largely contained in packages/odk-web-forms/src/lib/xforms) and the UI (large contained in packages/odk-web-forms/src/components). As we've discussed the migration of the UI to Vue, we've also discussed these related goals:

Formalize the engine/UI boundary
Make explicit that we intend the engine to be web framework-agnostic
- In so doing, potentially continue to maintain the current Solid UI in tandem with Vue
Treat use of reactivity in the engine (currently also using Solid) as an implementation detail
- This use of reactivity is explicitly not considered part of the interface contract at the engine/UI boundary
- Present use of Solid for the purpose is not necessarily under reconsideration in the scope of the Vue migration

This raises important questions about the interface of the boundary itself. We discussed the importance of this yesterday, and will be discussing design in more detail today. I wanted to take the opportunity to attempt to frame this discussion, and to introduce some prior art for consideration.

Assumptions

First I'll lay out a few assumptions about goals for this interface. Some of these are restating assumptions we've been aligning on in recent days/weeks, the rest fall out of these discussions (so hopefully nothing here is particularly controversial!):

Engine responsible for XForms domain logic: The responsibility for business logic pertaining to XForms (i.e. the ODK XForms spec)—e.g. parsing forms into a particular runtime schema, form-defined computation, dependency resolution and synchronization—is the domain of the engine. Pushing these aspects of business logic out to any particular UI client would be challenging to maintain and test, and would risk drift between multiple prospective UI clients.
UI responsible for reflecting and interacting with form state: The responsibility of any given UI client—in terms of the engine's business logic—is to present form structure and state to a user, and to provide means for the user to interact with the form to manipulate its state.
Engine-UI interface responsible for conveying updates: Implied by 1 and 2:
- the engine has a responsibility not just to initiate form state, but to update state as the dependencies of a given computation are updated;
- a UI client, in turn, has a responsibility of reflecting its presentation of such updates, and updating its interactive affordances accordingly
- the interface between the two, then, must provide a means to convey updates
Reads and state updates are most pressing design concern: While we should thoughtfully address all aspects of the engine-UI interface, reading state—especially reading engine-driven updates to state—are the most pressing unanswered question in the current interface. In the interest of making important short-term progress (and enabling effective iteration from there), it's likely prudent to limit the scope of this design effort to this read/updates concern. Expanding that scope to reconsider the current write interface may also be wise if the write interface is directly impacted.
Encpasulation as interface contract: The interface between engine and UI should not overly expose inner workings of how business logic is achieved, and that includes any particular notion of internal reactivity which might be used to that end. Notwithstanding Hyrum's Law, any such implementation detail should be treated as incidental and subject to change.

More concretely: just because the engine may currently use Solid reactivity to perform updates internally does not mean that Solid reactivity is the interface UI clients use to consume updates.
Encapsulation as design guidance: It isn't a hard requirement, but likely a very good guidance, to intentionally obscure these sorts of implementation details, and in particular to intentionally distinguish aspects of the engine-UI interface concerned with reads/updates from the present reactive internals.
Prior art as design guidance: In contrast, we are not inventing a wheel from whole cloth. Prior art being an excellent source of inspiration for interface design, it might be prudent to adopt interface patterns from another existing reactive implementation. In particular, it might be worth exploring where there is overlap between individual UI frameworks' interfaces to the outside world—which is to say, common themes for embedding non-framework functionality into idiomatic framework code.

Lastly, I think this is more a requirement than an assumption, but it's one we didn't discuss at length and one I think we should keep in mind:

Form values are polymorphic: while the current implementation largely ignores this by implementing values as strings, there's some hint of it in #14 (but I'd caution that I'm referencing it more as an example of the requirement, less as an example of prior art). In any case, the XForms domain deals with values of a variety of data types, and in some cases collections of values (as in #14). I don't know if we need to address this requirement directly in the scope of this particular design, but we should at least keep it in mind, as it will have implications for client UIs.

Prior art

With those assumptions stated, I think it might be good to seed the design discussion with some reference to prior art that might inspire us. For each instance of prior art, I'll include a minimal pseudo-code example of how it might be applied.

Callbacks

Callbacks are a built-in language feature, and a persistently common idiom for receiving updates over time. Example:

interface EngineNode<T> {
    getValue: () => T;
    onUpdate: (callback: (updatedValue: T) => void) => void;
}

In this example there is still a distinct method to get the value. While a callback API can be executed synchronously, it's difficult to convey this in the interface itself. Many recent callback-based APIs provide a similar companion API for synchronous retrieval, often superceding the callback when a value is present. (Example: MutationObserver's takeRecords).

For this example, and several of the others, we might also consider an update-receiving method which returns void as an opportunity to combine the more likley use cases (get initial value, and any subsequent updates). I think this kind of combination can be confusing if not designed and documented carefully, but I'm calling it out as a permutation which could apply to several prior art examples.

Callbacks are also likely to appear as part of other interfaces. Realistically, nearly all of the other ways to express "receive updates over time" will be specializations of this underlying language feature, or specific conventions around it.

Events (DOM)

A browser-native idiom also commonly used to convey updates over time is the DOM Event type, with associated dispatch and listener APIs.

interface UpdatedValueEvent<T> extends CustomEvent {
    data: T;
}

interface EngineNode<T> {
    getValue: () => T;

    // or `addEventListener(...)`, etc
    on: (
        eventName: 'update',
        callback: (event: UpdatedValueEvent<T>) => void
    ) => void;

    // or `dispatchEvent(...)`, etc
    trigger(event: UpdatedValueEvent<T>): void;
}

This tends to look like callbacks with a little bit more ceremony. A distinguishing feature of DOM events is the ability for them to propagate from a given EventTarget to another (such as the concept of "bubbling" in the actual DOM node tree). A corresponding downside of this feature is that it can be challenging to follow events to their source, both at runtime (call stacks become more convoluted, and may be broken by asynchrony) and in source code (static analysis of e.g. method calls often does not extend to these sorts of event dispatch and propagation).

Observables (user-land)

Observables are a concept used in many reactivity libraries (such as RxJS) or supported for compatiblity by others (such as Solid, which uses roughly the compatibility interface defined below).

interface Subscriber<T> {
    next?: (updatedValue: T) => void;
    complete?: () => void;
    error?: (error: unknown) => void;
}

interface EngineNode<T> {
    getValue: () => T;
    subscribe: (subscriber: Subscriber<T>): void;
}

Observables have semantics which overlap with another language feature, iterators. For the purposes of this discussion, we can also consider this a lense on how we might "abuse iterators" (and/or generators), as I quipped on our call yesterday.

In contrast with a basic callback, Subscriber has an object interface, capable of receiving additional information beyond value updates:

complete upon termination of value updates over time (possibly analogous in our usage to the removal of a repeat instance; this also further likens the concept to iterators)
error called when an error condition occurs over time

We could also conceivably expand this subscriber/observer interface to convey other semantic information.

In any case, this example is a helpful reminder that we'll want to consider aspects of state-over-time besides values, validation and termination among them.

Observables (future-spec)

There have been some efforts to standardize Observable, in the past as a potential JavaScript language feature (TC39), and more recently as a potential web platform feature (WICG). This more recent effort seems to have some real momentum (as it's backed by Google). RxJS is mentioned as a reference implementation, and Solid as a compatible interface, so another example would be redundant.

This is called out separately from user-land Observables specifically because it's an active standardization effort, and because it provides additional API discussion we might consider pertinent.

Subscribe-on-read ("signals", Vue's `ref`, etc)

Discussed a bit above is the concept of combining the interface for synchronous (e.g. first) read, and updates over time. This concept has been popularized in part by Solid's term "signal"—which other frameworks like Preact and Angular have adopted. Other frameworks provide a similar mechanism by a different name: Vue uses ref. The actual interface varies, but the gist is that accessing a value also establishes a subscription to future updates to that value.

Not to bias any particular framework's implemtnation or terminology, the following example uses a contrived name and interface for the concept:

interface SubscribingReader<T> {
    read: () => T;
    write: (value: T) => T;
}

interface EngineNode<T> {
    value: SubscribingReader<T>;
}

In order to actually implement subscribe-on-read, a system with this sort of runtime-defined reactivity must provide some means to establish a scope (or context, but not to be confused with what many UI frameworks call "context" for other uses) where subscriptions are tracked. Some solutions implicitly couple this scoping to other APIs with related concerns. Others, also implicitly, achieve scoping with a compiler (usually while addressing other concerns like templating). There are also more explicit APIs for establishing a tracking scope, but these tend to be less common, and generally more complex from an interface perspective.

Et cetera

I've tried to highlight some of the more common themes above, but of course this is not exhaustive. A very interesting resource I came upon while getting up to date on the Observable standardization efforts is A General Theory of Reactivity (GTOR). There are other areas of prior art apart from what's termed "reactivity", though it's unclear whether we'd find value in exploring those for our purposes.

Forgive me if this is naive/nonsense. Can the UI layer get a reference to an object that the engine layer is responsible for mutating? Right now EntryState has Solid reactive primitives in it, but couldn't there be an object beyond that with nearly the same shape but using domain types that gets updated by what is now EntryState, that is the output of the engine, and then the UI layer uses its reactive primitives to observe changes on that object?

That’s essentially the “signal” option. Some mechanism is needed to inform the consumer that a mutation occurred so it can, er, “react” to the update.

Sigh, as a consequence of trying to rush posting before our meeting, I neglected to include a really big assumption:

“Read” and “update” is discussed here in terms of “value”, but the same concerns apply to a bunch of other similar parts of the interface: relevant, readonly, etc. All of these will be state managed by the engine, which any UI client will need to read synchronously and get updates over time. A corresponding assumption is that we will want to use the same (or at least a thematically similar) mechanism for all of these.

I see, that makes sense. In that case, the UI layer would be entirely responsible for identifying updates based on whatever the framework in use provides to do so, right? The UI layer would have the responsibility of binding to the changing data provided by the engine and updating the interface based on it.

Right, so the design task here is settling a means to convey updates that would be suitable for use in the UI framework’s own state tracking paradigm.

I guess what I'm wondering is whether "just mutation" is a possibility.

I don't think it is. Arbitrary mutations are not observable from the outside without some specific notifying mechanism. There have been some historical attempts to introduce arbitrary external observation of mutation, both in the language and the web platform, and it's... well, a sore spot for many. But the very short version is, no, something needs to coordinate it.

Although this does sort of hint at another angle to think about it: "bring your own mechanism". We'd still need to have some sort of meta-interface to accept that, but it might be worth considering what that would look like and whether it'd be a better interface for UI clients.

Every kind of reactivity I've used does require an explicit value wrapper at creation time but I have to say my brain is really fighting it in this context. Maybe it's because I haven't gone deep in frontend frameworks and somehow I've convinced myself that they had solved this problem of just watching any old value!

For callbacks, it'd be something like this?

When the UI is instantiated and bound to the engine, each component registers callbacks for all the things that are reactive (value, label, constraint, etc)
The engine calls the corresponding callback when a signal changes

Or maybe it's one callback per form control instead of for each value that can change

Another understanding check: Doing something like returning the new state (or a diff of what's changed) out of the setter or having a single callback that returns the same would be essentially implementing a reducer concept outside of any UI system's reactivity model?

A couple of additional notes that may be of interest:

The "build your own reducer" concept is what Collect does for reactivity in field lists: https://github.com/getodk/collect/pull/3023 This uses JR's equivalent of EntryState which gets updated when the equivalent of setValue is called. It's basically exactly what @sadiqkhoja described. The Collect context is different because we're typically only updating the UI for a few questions in a field list. But also if what the UI layer does with the new state is update values wrapped in its reactive primitives, we'd get all the clever rendering tricks of the engine, I believe.
My understanding is that it's also similar to what JSONForms (which I think we have some spiritual alignment with) does. Here's the Vue binding initializing the reducer: https://github.com/eclipsesource/jsonforms/blob/2bfe9aa32a71afb7f0434b60ac9ed47855e4658b/packages/vue/src/components/JsonForms.vue#L129
It looks like JSONForms attempted to use an Observable implementation as an interoperability point but backed out of it: https://github.com/eclipsesource/jsonforms/blob/2bfe9aa32a71afb7f0434b60ac9ed47855e4658b/packages/core/src/util/type.ts#L152

I'm not ready to advocate for a particular solution but wanted to share these connections I've made.

Every kind of reactivity I've used does require an explicit value wrapper at creation time but I have to say my brain is really fighting it in this context. Maybe it's because I haven't gone deep in frontend frameworks and somehow I've convinced myself that they had solved this problem of just watching any old value!

If it helps to think of it from another angle, even if they could observe arbitrary mutations, they'd need to know which to observe. And for our use in UI clients, we'd probably want to express which values might change in the interface as guidance for clients.

For callbacks, it'd be something like this?

When the UI is instantiated and bound to the engine, each component registers callbacks for all the things that are reactive (value, label, constraint, etc)

The engine calls the corresponding callback when a signal changes

Or maybe it's one callback per form control instead of for each value that can change

The original writeup was meant to be less prescriptive about this, but for the sake of discussion I think we can even frame it as both rather than or (and perhaps extend that to other hypothetical interfaces besides callbacks). So a still-limited-but-expanded example might look something like:

type Read<T> = () => T;

type Write<T> = (value: T) => T;

type Callback<T> = (value: T) => void;

// Not necessarily a requirement/recommendation, but likely to come up when
// thinking about how a client might consume this API in the real world.
type CancelCallback = () => void;

type RegisterCallback<T> = (callback: Callback<T>) => CancelCallback;

// `Value` type parameter here anticipates addressing polymorphic field types
interface ValueNodeState<Value = string> {
  // Value state:
  //
  // - ✅ can be read synchronously
  // - ✅ can be written (typically)
  // - ✅ can be updated by the engine
  getValue: Read<Value>;
  setValue: Write<Value>;
  onValueChange: RegisterCallback<Value>;

  // We could of course choose to use native getters/setters rather than e.g.
  // `get`-/`set`-prefixed method names. To me they convey the same intent, but
  // it might be less clear to others/in a client context. I'll use getters below
  // to be slightly more brief, but the intent isn't to favor one or the other.
  get value(): Value;
  set value(value: Value);

  // Note that the following (and perhaps more I'm not thinking of) can cause
  // value to change in the engine:
  //
  // - `calculate`
  // - `setvalue`
  // - `setgeopoint`
  // - changes to `relevant`
  //
  // For the (primary) "end user filling a form" use case, `relevant` state is
  // important because it should affect other aspects of the UI, but we may want
  // to treat these others as internal (again for the primary use case).

  // Relevant state:
  //
  // - ✅ can be read synchronously
  // - ❌ cannot be written by clients, only engine
  // - ✅ can be updated by the engine
  get isRelevant(): boolean;
  onRelevantChange: RegisterCallback<boolean>;

  // Readonly state:
  //
  // - ✅ can be read synchronously
  // - ❌ cannot be written by clients, only engine
  // - ✅ can be updated by the engine
  get isReadonly(): boolean;
  onReadonlyChange: RegisterCallback<boolean>;

  // ... and so on for each aspect of state...

  // And a callback (or other interface) for **any** state update.
  onStateChange: RegisterCallback<{
    value: Value;
    relevant: boolean;
    readonly: boolean;
    // ...
  }>;
}

// It's worth noting that the `on`-prefixed callback register methods start to
// sound superficially like events, and we may even choose to frame them as
// such, even if we distinguish them from the more conventional
// event/propagation approach. We might also choose to have a single method
// to register callbacks, with a first parameter specifying what value to
// track changes on (and maybe that first parameter is optional, tracking
// any change to state when not specified). E.g.
interface ValueNodeState<Value = string> {
  watch: (state: 'value', callback: Callback<Value>) => CancelCallback;
  watch: (state: 'relevant', callback: Callback<boolean>) => CancelCallback;
  watch: (state: 'readonly', callback: Callback<boolean>) => CancelCallback;
  watch: (callback: Callback<{ /* all state */ }>) => CancelCallback;
}

Doing something like returning the new state (or a diff of what's changed) out of the setter or having a single callback that returns the same would be essentially implementing a reducer concept outside of any UI system's reactivity model?

Not necessarily. The engine state itself being reactive would give us a fair bit of flexibility on how to produce either the full state (we could effectively serialize it if we want) or e.g. a state change log (we could essentially just define a log entry schema, push to it, filter by action identifier/time range/whatever; not unlike the concept of a database "audit table"). But as a client interface, either would potentially impose some sort of reducer need onto the client.

I spent some more time yesterday thinking about a full state tree option, and I think that burden would depend significantly on the client's own model of input/rendering. Like, React (and Preact, and perhaps others) would be well positioned to deal with it because the full state tree could effectively be passed straight down to components as props. I think this is part of why I started thinking of it in terms of VDOM—a VDOM is designed specifically with at least the mental model that UI is a function of state, and so it's at least conceptually well suited to process the entire application state on every change.

I think a diff/log interface would inherently impose some sort of reducer burden on most if not all clients, and probably one where the logic itself would be identical, but the correlation between affected data (and affected components) would be very framework specific. I'd imagine many of the frameworks with a React-like concept of "context" would want to use that, but they'd still have to handle reconciliation more or less the same way.

1- Are changes really happening over time?

yes - If I think from an individual field/node/question's perspective then it's value, relevancy, label, constraint can change over time without user interacting with it.

No - If I think from the full form / EntryState, then state of the form changes only because of the user interaction. It's a serve and return mechanism. I could be wrong here but to me it's a single unit of work. If user does A then engine performs series of effects as a single unit - there's no chance that effects are partially applied.

2- JsonForm

First, I found the code really hard to understand because I have never worked with Redux. However, I was able to find out that they were using redux in their first version and later got rid of it.

For me most interesting line is this: https://github.com/eclipsesource/jsonforms/blob/2bfe9aa32a71afb7f0434b60ac9ed47855e4658b/packages/vue/src/components/JsonForms.vue#L275

Unlike redux, dispatch method of their core (engine) returns the updated state and then the client updates its reactive state.

Confession: I still don't understand how the change event on an individual renderer is propagated to the top level component i.e. JsonForms in case of Vue where dispatch method is defined. But I have confirmed by adding a breakpoint in the devtool that every change calls this method.

What are your thoughts on adopting redux pattern where our engine could act like a store and clients could dispatch(actions) and subscribe on the store? This will definitely change a lot of things in our existing implementation :(

3- What to expose from the engine?

LN asked related question above as well. How many callbacks/subscriptions/etc (whatever approach we pick) client of the engine has to define. For every mutable property or one for each control/question/field or one for the whole EntryState.

How each option will look like from clients perspective:

Option 1: one for each mutable state

const entryState = engine.createEntryState(formDefination or xml);

for(child in entryState.children){
    child.value.subscribe(...);
    child.relevant.subscribe(...);
    child.label.subscribe(...);

    // do same for all child.children - recursion
}

Option 2: one per control

const entryState = engine.createEntryState(formDefination or xml);

for(child in entryState.children){
    child.subscribe((key, value) => {...}); // key being value or label or relevant....

    // to same for all child.children - recursion
}

Option 3: just one subscription

const entryState = engine.createEntryState(formDefination or xml);

entryState.subscribe((path, value) => {... });

4- Inject reactive `factory`

I mentioned this idea yesterday's meeting as well, it's not fully baked and probably most difficult to implement in the engine and maybe not all the clients would be able to work with it. But here is what I think it could look like:

Engine

function createEntryState(formDefinition, reactiveFactory){
  ...
  for(element in formDefinition.elements){
      const [value, setValue] = reactiveFactory.create("");
      const [relevant, setRelevant] = reactiveFactory.create(true);
      ....
  }
}

Client (Vue)

const factory = {
  create: (initialValue) {
    const r = shallowRef(initialValue)
    const get = () => r.value
    const set = (v) => { r.value = v }
    return [get, set]
  }
  // maybe factory has to provide computed() and effect() methods as well
};
const entryState = engine.createEntryState(formDefinition, factory);

Client (Solid)

const factory = {
  create: (initialValue) {
    return createSignal(initialValue)
  }
};
const entryState = engine.createEntryState(formDefinition, factory);

5 - Just old fashion `return`

Thinking purely from the client's perspective and ignoring the current state of the engine and subsequent implementation (sorry). This interface seems so simple and initiative to me

const entryState = engine.init(formDef);

// update(path, value) => return only fields that changed
const partialState = entryState.update('children[0].value', 'new value');

patch(entryState, partialState);

or update could return change/transaction logs (I couldn't think of an example where order would be messed up)

const entryState = engine.init(formDef);

const logs = entryState.update('children[0].value', 'new value');

for(log in logs){
  entryState[log.path] = log.value; // using [] for simplicity, proprobly would need something like R.assocPath
}

From engine's perspective (assuming it has a way to track all the changes occurred due to a write action from the client) calling the callback function or returning the partial/full state is the same thing:

update(path, value) {
  state[path] = value;
  applyTheSideEffects();
  callback(state);
  or
  return state;  
}

1- Are changes really happening over time?

yes - If I think from an individual field/node/question's perspective then it's value, relevancy, label, constraint can change over time without user interacting with it.

No - If I think from the full form / EntryState, then state of the form changes only because of the user interaction. It's a serve and return mechanism. I could be wrong here but to me it's a single unit of work. If user does A then engine performs series of effects as a single unit - there's no chance that effects are partially applied.

This is an excellent, insightful question. And I think it might help us frame not just this aspect of design, but many others that either have already come up or will in due time.

Of course, both of these answers are correct. Moreover, I think it might help to step back even a bit further, to a point I've raised a few times in the past. Namely, considering what an XForm is.

XForms as program

As I've raised the point with @lognaturel, each individual XForm is a program. More specifically, an XForm is a declarative, and purely functional, program[^purity]. In theory, it could be compiled to any sufficiently expressive programming language—where the product of compilation would be equivalent to the XForm itself. And with some affordances for user interaction as part of the program's event loop (or similar), an entire XForm could be expressed as a function. I say all of this to strongly reinfroce the "no" answer to the question, are changes happening over time?

XForms as spreadsheet analogue

On the other hand, I think of a friend with whom I sometimes casually discuss work topics. This friend has an interest in programming, but has a more experienced background in project management. When I've tried to explain XForms in terms the friend understands, I continuously find myself gravitating towards spreadsheets as an analogue.

Of course, spreadsheets can similarly be reduced to the same previous definition. Spreadsheets are far and away the most popular programming tool, are overwhelmingly declarative and purely functional, and any individual spreadsheet could likewise be described as a program. But a spreadsheet is also more than that.

Granted, I haven't personally built spreadsheet software. So it's possible my instincts on the topic are way off. But when I think about how I would approach building, say, a subset of Excel with these properties:

A structured tabular format of sheets and cells
A set of formulas which compute (and continuously recompute) values from input cells into output cells
Interactivity consistent with the format, where a user:
- Is presented sheets and cells
- Enters data into some cell, which is input to another cell's formula...
- ... sees their input applied to the formula in that other cell
Multiple platform targets

I expect that many of the kinds of questions we're asking here and in related discussions would arise. And I would tend to think toward a model similar to the one emergent in web-forms:

A single core component responsible for the semantics of spreadsheet sheets, and computing formulas expressed therein
A data model exposing a structure to clients (per platform in this case) which is consistent between them, and sufficient for each to represent the state of both a given sheet and each of its cells
Some means of conveying the derived state (by way of formulas in this case) of individual cells directly between core and client, such that each client can present them accordingly

Being purely functional, it is absolutely possible and (according to pertinent theory) perfectly reasonable to convey the entire state of the program for any change. This is referential transparency.

In the context of the component structure described above however, each client would have direct access to sub-structures within the full state, and updates would tend to be issued by clients against a given sub-structure. Abstractly, we could call this direct sub-structure access a "lens". If updating a given lens would only effect its referenced sub-structure without effects on other aspects of the state, we could skip this whole conversation and go with the much more comfortable update(lens, value) -> lens. But where things get tricky is expressing update(lens, value) -> state.

Being a spreadsheet, it's also highly likely that, given a continuous loop of update(state, path, value) -> state (or update(lens, value) -> state), a set of N clients integrating 1 computational core would produce N+1 implementations of certain aspects of the spreadsheet-specific domain logic. Like an XForm, part of that domain logic is the dependency graph defined by the spreadsheet itself. A client interface of this sort tends toward a set of clients each establishing their own dependency graph in tandem with the one implemented in the core.

In other words, it seems highly likely (at least to me) that update(lens, value) -> state implies the eventual existence of N+1 implementations of at least some subset of the core functionality. This might start as parallel correlations between cells (in spreadsheet terms; or nodes in XForms terms), but the more worrying likelihood is that certain aspects of the computations themselves would find their way out to clients.

Sticking with the multi-platform spreadsheet analogy, it would be problematic if =SUM(A1:A10) produced different values on Windows and Linux. My concern is that an interface like update(lens, value) -> state encourages redundancies between clients and core, making that a more likely prospect.

This can be avoided to some extent with a well established separation of concerns, and a persistent discipline to preserve it. But some degree of redundancy is inherently unavoidable with this interface:

The core necessarily has a model of the computational graph. When a given computation produces a change at a given node in that graph, the core must know where that change occurred.
The interface update(lens, value) -> state does not convey this locality of change, making it opaque to clients. But each client must also apply these changes where they occur. As such, each client must then reproduce the information which is lost by this interface.

In the absence of any other information, I believe these points would strongly support an approach like reducers, or an event log. But I want to reiterate that the core knows where changes occur and clients want to know where changes occur. Given we have language features and reference to quite a few models of reactivity which would preserve this locality, it would be a shame to withhold it.

As I mentioned on Friday, I'm leaning toward a two-pronged approach to this design. I feel less strongly now than I did then about the specifics of those prongs, but I do feel strongly that it would be best if we:

Keep, rather than removing, the ability to convey updates where they occur
Also provide a more principled, funcional interface, such as returning the full state from setters

[^purity]: There are some notable exceptions to both of these which are worth considering in a broader discussion (jr: URLs at least theoretically break both the declarative and functional model by introducing networked resources; setgeopoint likewise depends on IO; certain XPath functions are inherently or conditionally non-deterministic), but I think we can put them aside for this discussion.

Summary of design decisions

I wanted to make sure to capture the decisions we made, both in terms of how it concretely impacts the engine-UI interface and the general goals informing the concrete decisions.

I also call out some considerations for particulars along the way. I've tried to make sure these thoughts are clearly additive.

Goals clarified

Satisfy both general categories of interface:
- Produce unmediated state resulting from a change, at the point the change is made
- Preserve locality of observably changed values where changes occur
Provide a minimal, effective, framework-agnostic means for engine and UI to collaboratively establish observable state changes, so that:
- The engine can change values over time according to the semantics in its domain
- Clients can react to those changes and update its presentation and interactive behavior according to their own domain
Continue to convey, at least in the interface's types (and to the extent reasonable), which aspects of state may change, and which parts of the application may change them

Client-facing setters return full state

To satisfy the first interface category, where the interface exposes a way to set state, it should return the effective state resulting from the change including:

The state change issued directly by the client
Any dependent computations (calculate, relevant, etc) affecting nodes elsewhere in the state tree
- -

Consideration: as we've discussed in at least a couple calls, we currently have some flexibility in terms of whether effects are applied synchronously. While the implementation is fully synchronous now, that hasn't been a guarantee in the interface. The implication of this decision is that we'll make that guarantee explicit, and the most likely alternative would be to return Promise<State> (which I don't think we'd find appealing). That said, I think we can consider a middle ground where the fully resolved state is produced synchronously-but-on-demand. Which is to say...

Where we currently have:

interface AnyNode<T> {
  setValue: (value: T) => T;
}

The decision we made probably implies:

interface AnyNode<T> {
  setValue: (value: T) => EntryState;
}

But we could also satisfy it with something like:

type Thunk<T> = () => T;

interface AnyNode<T> {
  setValue: (value: T) => Thunk<EntryState>;
}

The primary (potential) benefit would be performance flexibility: we could choose to defer full state resolution internally, unless a client explicitly asks for it by calling the thunk.

A secondary benefit would be flexibility of the interface itself: if it turns out that most clients don't consume this, we could consider deprecation or breaking it off into a more specialized API without too much disruption.

Bring your own reactive, mutable object interface

To satisfy the other category—locally observable changes—we decided to allow clients to supply a single, simple factory function. We deferred settling on more general language, but the factory function would produce what is sometimes called a "store" (or some specialization of that term). More specifically, we decided that:

The factory function itself will accept an object of any unspecified shape, and produce an object of the same shape and value as its input.
The produced object will accept arbitrary mutations, which the engine will use to record state changes as they become effective.
The underlying implementation of the factory function is unspecified, with the expectation that a client supplying it will be able to use that implementation to set up bindings appropriate for mutations in the engine to be observable in the client.

In terms of types, the interface would for the factory function itself would be something like:

/**
 * An implicitly reactive object. Web Forms clients determine the mechanism of
 * reactivity, and the Web Forms engine produces reactive state by mutating
 * the object directly.
 */
type OpaqueReactive<T> = T;

/**
 * Creates an implicitly reactive object.
 */
type OpaqueReactiveFactory = <T>(state: T) => OpaqueReactive<T>;

Examples of existing functions which directly satisfy this interface include:

Vue's reactive
Solid's createMutable
- -

Consideration: factory interface minimalism

Other solutions might require some extra setup or ceremony to satisfy this input. For instance, in the design discussion we also referenced Pinia as another compatible implementation. Its interface to define a store is more involved, but a simplistic (and potentially problematic) example of how Pinia could be used might look something like this:

import { defineStore } from 'pinia';

const factory: OpaqueReactiveFactory = <T>(state: T) => {
  return defineStore('⚠️⚠️⚠️ UH OH THIS IS NOT GENERIC ⚠️⚠️⚠️', {
    state: () => state
  });
};

It's unclear whether the extreme simplicity of the factory's interface is actually general enough for use with Pinia. It's also unclear whether we'll need to solve problems like this, or when we would prioritize it if we do.

Consideration: factory interface type refinement

In other conversations, we've discussed a related interface decision (which remains unspecified) namely: the initialization call site between client and engine. While we didn't fully specify that, the decisions here have implications for it:

We've made explicit that the factory is generic in terms of returning an object in the shape of its input
We've in turn made the reasonable expectation/assumption that the factory is provided "at the top level", i.e. when initializing the engine
We did not specify whether the factory should be able to produce a subtype of its input (and whether the engine state type should be parameterized to reflect that)

In other words, we haven't specified that this should work:

type HypotheticalReactive<T> = T & {
  [SOME_SYMBOL](): SOME_ADDITIONAL_BEHAVIOR;
}

declare const hypotheticalFactory: <T>(state: T) => HypotheticalReactive<T>;

const state = engineState(form, {
  reactiveFactory: hypotheticalFactory,
});

state[SOME_SYMBOL]();

Consideration: reactivity is inherently optional

The factory interface we've specified can be satisfied without actually implementing any reactivity at all. A client could hypothetically supply the identity function:

const state = engineState(form, {
  reactiveFactory: (state) => state,
});

I expect that most clients will want to provide a reactive implementation. Possible exceptions that come to mind:

A client which isn't concerned with reflecting changes at all because it checks them directly (the ported JavaRosa Scenario is a likely case)
A client with a substantially different user interface which doesn't map neatly to the form's structure and flow (such as hypothetical dev tools)

Distinct types for static, stateful, writable properties

While settling on these decisions, we also agreed that we should continue to convey information to clients about which aspects of form state are either:

statically read-only (e.g. representations of form schema)
stateful but read-only by clients (e.g. whether a field is relevant)
stateful and writable by clients (e.g. field values)

These distinctions are currently handled by a combination of:

Type conventions:
- readonly property modifiers generally correspond to static values
- get accessors and/or thunk-like types generally correspond to stateful values which clients read, but only the engine writes
- Client-writable APIs currently use explicit method calls like field.setValue(value)
Fairly granular exposure of runtime implementations directly corresponding to those types

The "bring your own reactive mutable" approach imposes (or at least strongly suggests) some limitations on reasonably prserving the runtime aspects of this. But we can continue to provide the type-level distinctions (likely with some adjustments).

Consideration: explicit type conventions

We specified that we would maintain distinctions between static, stateful read-only, and client-writable values. We didn't specify how. It's worth considering establishing explicit type conventions for this now, to reduce confusion and rework overall.

I'd like to suggest a simplification of the existing conventions:

interface AnyNode<T> {
  /**
   * This property, and any of its sub-properties, is fully static. It is either
   * derived from the form definition itself, or conveys some other information
   * directly from the engine to the client which will not change during a
   * user session involving the form state.
   */
  readonly foo: Whatever;

  /**
   * This property is readable by the client, and might be changed by the engine.
   * It should not be written by the client.
   */
  get bar(): SomethingElse;

  /**
   * Just like `bar`. Calling out a readonly array case to emphasize that the
   * value itself is produced by the engine, even if it's a collection. A client
   * should not mutate the collection by adding or removing items.
   */
  get bat(): readonly OtherThings[];

  /**
   * This property is readable and writable by the client. Explicitly defining
   * the type with a getter and setter conveys this, and can be called out in
   * documentation as an explicit indicator for that intent.
   */
  get quux(): PotentiallyClientInput;
  set quux(input: PotentiallyClientInput);

  /**
   * Just like `quux`. Calling out a generic case to relate the example back to
   * concerns around polymorphic values.
   */
  get value(): T;
  set value(input: T);
}

(An alternative convention would be to wrap client-facing types with methods corresponding to each case. I'll leave that example out unless there's interest in seeing it.)

Consideration: derive client types from engine types?

We discussed the fact that certain aspects of the client-facing interface should be more restricted than within the engine. This most obviously applies to stateful/client-readonly properties:

The engine should write to these fields to satisfy XForms semantics
Clients should not write to those fields

This necessarily implies that engine and clients interact with distinct types for the same values. To aid keeping these in sync without unnecessary maintenance burden, I'd suggest we consider taking on the upfront burden (in terms of some type complexity) so client types can be derived (largely or completely) from engine types.

Aaaaand I already realized that the suggestions around types conflict with returning full state to clients on writes.

Having noticed that the type conventions I was suggesting conflict with the goal of returning fully resolved state when clients issue writes to the engine, I took a bit of time to do some hammock-driven development. My thinking on this feels like a refinement on a bunch of the other ideas above, and might help to simplify some other questions.

I think we can better address all of:

Differentiation of static/readonly from client-reactive/readonly
Resolve full state on client demand, otherwise preserve internal flexibility to defer full resolution internally
How/where to invoke reactive factory
The potential contradiction between "reactive factory is optionally reactive" and "should the engine leverage client's reactivity implementation"

... with a few specific changes to the state structure, a clarification about the scope of client writes, and an upfront decision on the relationship between internal and external reactive state.

Concretely name exactly what clients can write, now.
Structurally delineate static and reactive portions of a node, at the node level.
Structurally delineate reactive client-readonly from client-writable as well.
Mirror reactive state: engine (internal, mutable) -> client (readonly)

What can clients write?

After form initialization, there are only three categories of client-writable engine state:

Form language (global across a form entry)
Individual values on nodes which accept values
Repeat count (where it isn't controlled by the form with jr:count or jr:noAddRepeat)

I don't believe there are any other XForms-specified aspects of state which a client should write, as it'll either be computed by the engine (e.g. calculate, relevant, etc for node bindings, dynamic aspects of labels, etc) or populated as a parsing responsibility (e.g. external instances).

Anything else the client might write is a presentation concern (e.g. collapsing a group might be something a client does, but doesn't impact the state of the form entry itself).

Structural delination of static-readonly, client-readonly

Rather than using (just) type conventions to convey these, we can convey them by isolation into specific sub-structures for a given node.

Every node will have some static aspects, established by parsing the form definition:

interface NodeDefinition {
  // ... further details intentionally omitted. This aspect is not currently
  // affected by this design consideration. We can (should) also consider
  // refining static-definition-design stuff. But it's out of scope here.
  // For the purposes of this discussion: everything in a node's definition is,
  // and should continue to be, explicitly `readonly`.
}

// Strawman name, not an actual suggestion. This is currently called `NodeState`,
// but I'll recommend using that name differently below.
interface EngineNode {
  readonly definition: NodeDefinition;

  // ... more below ...
}

Every node will also have at least some reactive, client-readonly state:

// Note: this name is currently used as the base interface for the actual nodes
// in the engine state tree. As above, the strawman replacement name for that
// concept frees up this name for this more appropriate usage.
interface NodeState {
  get readonly(): boolean;
  get relevant(): boolean;
  get required(): boolean;
  // ...
}

interface EngineNode {
  // Static/readonly
  readonly definition: NodeDefinition;

  // Client-reactive/readonly
  readonly currentState: NodeState;
}

Some nodes will have additional client-reactive/readonly state, which will tend to correspond to their client-writable state.

Structural delination of client-writable state

Many (but not all!) nodes have some client-writable state. But notably, there is no type of node concerned with more than one client-writable category! The examples below go through these cases, each with a subtype for additions to their client-reactive/readonly state, and corresponding client-writable setters (where applicable).

For the "entry"/top level node, form language is additional state which a client can both read and write:

interface EntryNodeState extends NodeState {
  get language(): string;
}

interface EntryNode {
  // Static/readonly. I know we should resolve this naming discrepancy. Just
  // referencing the existing name for clarity.
  readonly definition: XFormDefinition;

  // Client-reactive/readonly
  readonly currentState: EntryNodeState;

  // Client-writable
  //
  // - Only the language can be written to the entry node
  // - Writing returns the entire application state (first category of
  //   write -> read contract)
  // - `EntryNode` already separates reactive-read access, so we can still
  //   defer full state resolution internally and resolve it on client demand,
  //   specifically on get access to its `currentState` (if we choose)
  setLanguage(language: string): EntryNode;
}

For leaf (non-structural, value-bearing) nodes, the of the node itself is additional state the client can both read and write:

interface ValueNodeState<T = string> extends NodeState {
  get value(): T;
}

interface ValueNode<T = string> extends EngineNode {
  readonly definition: WhateverAppropriateStaticDefinition;
  readonly currentState: VaueNodeState<T>;
  setValue(value: T): EntryNode;
}

Repeats are a bit more complicated. I'll use the shorthand "controlled" to mean repeats where the number of instances is determined by the form definition and engine (jr:count/jr:noAddRepeat), and "uncontrolled" where the client can add or remove instances (i.e. the count is client-writable). These names are meant to illustrate, not necessarily as naming recommendations. Without further ado:

  // ... not important for this design, just know that it's a type of node...
interface RepeatInstanceNode extends EngineNode {}

// The state of all repeat sequences has:
//
// - A count which can be read by clients and might change over time
// - A corresponding set of instances
// - Indexes at which existing instances are positioned
//
// These can all be represented as an array.
interface RepeatSequenceState extends NodeState {
  get instances(): readonly RepeatInstanceNode[];
}

// All repeat sequence nodes have:
//
// - A static definition
// - Readable state including `count` and `instances`
//
// They differ by whether the number if instances is set by the client or just
// the engine, so here is a base type for the common stuff.
interface BaseRepeatSequenceNode extends EngineNode {
  readonly definition: RepeatSequenceDefinition;
  readonly currentState: RepeatSequenceState;
}

// Naming for illustration...
interface ControlledRepeatSequenceNode extends BaseRepeatSequenceNode {
  // Engine determines repeat count, nothing writable here
}

// Naming for illustration...
interface UncontrolledRepeatSequenceNode extends EngineNode {
  // Clients can control repeat count. The client domain concepts are:
  //
  // - Add instance
  // - Remove instance
  //
  // These unfortunately don't have direct parity with the state's `instances`
  // array. But I think breaking parity is probably okay, to keep the client-write
  // interface simple and its capabilities clear.
  //
  // Presently, Enketo presents a UI where a single instance is added to the end
  // of a sequence, and any individual instance may be removed. I believe this is
  // consistent with UI capabilities in Collect (but may be corrected if wrong).
  // In any case, other investigation on the topic gives me reason to believe
  // there may be a desire for more flexibility than that. These add/remove write
  // method signatures are intended as a strawman to anticipate greater
  // flexibility, but they're not prescriptive.
  addInstances(count?: number, afterIndex?: number): EntryNode;
  removeInstances(startIndex: number, count?: number): EntryNode;
}

Mirror reactive state

In each of these cases, the structural delineation also serves as a clear indicator of where the client's reactive factory is applicable: currentState. It's trivial at the type level to represent that sub-structure as engine-mutable, and client-readonly. It's also trivial at the implementation level to know where the factory should be invoked and what its reactive value should include. E.g. (obvious pseudocode oversimplification):

const initNode = (definition: NodeDefinition, reactiveFactory: OpaqueReactiveFactory): EngineNode => {
  return {
    definition,
    currentState: reactiveFactory({
      readonly: false,
      relevant: true,
      // ...
    }),
    children: definition.children.map((child) => {
      return initNode(child, reactiveFactory);
    }),
  };
}

It would be tempting to think about piggybacking on the reactive factory to handle internal engine effects. Among other reasons, this won't work because the reactivity aspect of it is inherently optional. But it does give a good sense of how we could structure internal reactive state and propagate it to client reactive state. Mirror it (again obvious pseudocode oversimplification):

const initNode = (definition: NodeDefinition, reactiveFactory: OpaqueReactiveFactory): EngineNode => {
  const initialState = {
    readonly: false,
    relevant: true,
    // ...
  };

  const internalState = internalReactiveFactory(initialState);
  const clientState = reactiveFactory(initialState);

  internalEffect(() => {
    for (const key in internalState) {
      clientState[key] = internalState[key];
    }
  });

  return {
    definition,
    currentState: clientState,
    children: definition.children.map((child) => {
      return initNode(child, reactiveFactory);
    }),
  };
}

Setter methods would write to internal state (triggering any reactive effects, all in turn being mirrored to client state).

// ...
  const internalState = /* ... */;

  return {
    definition,
    currentState: clientState,
    children: definition.children.map((child) => {
      return initNode(child, reactiveFactory);
    }),

    setValue(value) {
      internalState.value = value;
    },
  };
// ...

getodk / web-forms

Initial read interface design between engine and UI #45

Assumptions

Prior art

Callbacks

Events (DOM)

Observables (user-land)

Observables (future-spec)

Subscribe-on-read ("signals", Vue's `ref`, etc)

Et cetera

1- Are changes really happening over time?

2- JsonForm

3- What to expose from the engine?

4- Inject reactive `factory`

5 - Just old fashion `return`

1- Are changes really happening over time?

Summary of design decisions

Goals clarified

Client-facing setters return full state

Bring your own reactive, mutable object interface

Distinct types for static, stateful, writable properties

What can clients write?

Structural delination of static-readonly, client-readonly

Structural delination of client-writable state

Mirror reactive state

getodk / web-forms

Initial read interface design between engine and UI #45

Assumptions

Prior art

Callbacks

Events (DOM)

Observables (user-land)

Observables (future-spec)

Subscribe-on-read ("signals", Vue's ref, etc)

Et cetera

1- Are changes really happening over time?

2- JsonForm

3- What to expose from the engine?

4- Inject reactive factory

5 - Just old fashion return

1- Are changes really happening over time?

Summary of design decisions

Goals clarified

Client-facing setters return full state

Bring your own reactive, mutable object interface

Distinct types for static, stateful, writable properties

What can clients write?

Structural delination of static-readonly, client-readonly

Structural delination of client-writable state

Mirror reactive state

Subscribe-on-read ("signals", Vue's `ref`, etc)

4- Inject reactive `factory`

5 - Just old fashion `return`