Immer-like patches

[universse]

Hi. Any chance this library could support something like immer patches? It would be an efficient mechanism to implement undo history and distribute state changes (e.g. via websocket), since only state deltas are stored.

Something similar was mentioned here https://github.com/pmndrs/valtio/issues/89.

Immer patches for references https://immerjs.github.io/immer/patches/.

[dai-shi]

I have huge desire to keep the core minimal (to provide a minimal api for React's useMutableSource interface). Besides, unlike immer, it doesn't track patches. In fact, it works totally differently. The similar goal but from 180 degree different angle.

I'd be happy to have "undo" capability in utils. Would you or anyone like to tackle it?

Serializing delta is another feature to tackle. I wonder if there's any existing library we could use.

[redbar0n]

Check out Serializr for serializing/deserializing.

[redbar0n]

to provide a minimal api for React's useMutableSource interface

Could you elaborate a bit on the reasoning behind this?

[redbar0n]

Here's an idea, inspired by event-driven microservice architectures on the back-end (and recently mirrored in an anecdote about photoshop internals, and similar to what @thelinuxlich was talking about over at https://github.com/pmndrs/valtio/issues/89). Pardon me for the elaboration. If it turns out not to be relevant for Valtios direction, then hopefully it could serve as some elucidation for future libraries.

When a user action is received that would change the state of a component, it could:

1. Snapshot the local state used in the component.
2. Store the event in a local log of events.
3. Apply the event (aka. user action) to the state.
4. Snapshot the new local state.
5. Calculate the diff/patch between the snapshots. Possibly by using immer-patches, or with some persistent data structure with Mori or ImmutableJS, or only by introspecting the structural sharing of Valtio's own snapshot.
6. Notify direct listeners (either by event, or by a wrapper func like immer-patches' produceWithPatches), and send the serialized diff/patch of the state to them (and potential external third-parties). Alternatively: Assume the third party is in sync at some point, and from then on only send the events, so they can apply the same actions to the same state on their end.
7. By default, add the event to an append-only global event log, if it is not explicitly turned off for that event, for performance. (Or maybe the global event log could be generated only when needed, from the local ones. This is easier in a not-physically-distributed system like the client-side is, since you can easily impose a strict ordering condition like a global event counter.)

This way, each component can have its own undo-redo feature, which can be opted out of the global state log for performance. This is useful if you want to enable users to undo only particular parts of their interactions (with certain components), without affecting other components in the process (rewinding the entire UI..). But it is also useful as a general way of handling distributed state changes/log. An undo on a component would generally append a log entry for the undo event for that component in the global state log. The app-wide event log can be transmitted to the server with support requests to track down bugs. It could also be used for app-wide time travel.

Currently, it seems like solutions either focus on always doing nr 7 (Redux, Zustand), or, to get performance, always do nr 2 (Recoil, Jotai). Whereas Valtio & MobX would allow you to stay off the UI thread (even putting state management into a service worker if you like/need), which ensures the UI always stays responsive. Enabling UI as an afterthought, and improving testability, and even swapping out the rendering library for something better, when that time comes. However, MobX uses classes and forces you to wrap components with observer(). Whereas Valtio avoids that, and has a simple proxy object and the more familiar hook-like useProxy(state).

The proxy API of Valtio is nice, and I think it has a lot of potential. Also, when you have a proxy object, which lets you intercept anything, you can basically implement everything under-the-hood, preserving the nice and clean API.

What's nice is to be able to mutate state, but have it be performed in an immutable fashion. It's also nice to use persistent data structures with structural sharing, like I have the impression Valtio already does. Furthermore, it's good to be able to access the events/actions operating on the state, but also be able to access/generate the corresponding/resulting patches/diffs to the state.

I think it could be possible to have a library that has:

• The nice and straightforward API of Valtio.
• Atoms used internally, like Jotai. But not exposed to the user unless fine-grained control is necessary.
• Update state outside React, like Zustand/Redux/Valtio. Which means it could potentially be used with vanilla JS, Vue, Angular and Svelte too.
• Support for Suspense, like Jotai/Valtio. Possibly with a React-specific plugin or by exposing the library to React through a hook.
• Snapshot state at a given point in time: globally, locally, or at any level of the nested state tree. Maybe like hookstate does.
• Access to the log of events (aka. user actions) that operated on the state. This should take less space than storing the state patches/diffs/deltas, with increasing benefit as the state changes become larger. Besides, if you are able to get a snapshot at any point in time, then you could produce a delta on demand. By snapshotting before a change, applying the event, snapshot again, and get the difference between the snapshots. The persistent data structure could maybe make this easier/faster.
• Code splitting like Jotai.
• Access to Redux devtools, like Zustand.

It could look something like this:

import { proxy, useProxy } from "valtio-variant";

const counter = proxy({
  count: 0, // would be turned to a Jotai Atom internally
  unusedCount: 0,
  add: (n: number) => {
    counter.count += n;
  }
});

const Counter = () => {
  // useProxy(counter); // this Valtio API avoided, so that no babel plugin is needed
  // const snap = useSnapshot(counter) // this Valtio API avoided, since it's unintuitive to read from snap and mutate the counter source
  const state = useProxy(counter); // inspired by useAtom API of Jotai
  return (
    <div>
      {state.count} <button onClick={() => state.add(3)}>+3</button>
    </div>
  );
};

export default Counter;

What do you think?

[dai-shi]

There are lots of stuff and I'm not sure if I understand everything.

const state = useProxy(counter);
return <>state.count</>

This is simply impossible to my knowledge. If it were, I guess mobx would have done it already. So, one novel part of valito is useSnapshot. This is also concurrent friendly.

The other novel part is snapshot which creates a snapshot on demand. So,

const state = proxy({ count: 1 })
++state.count // this doesn't create a new snapshot
++state.count // this doesn't create a new snapshot either
snapshot(state) // this does create a snapshot for the first time

This may or may not explain why I'd want to avoid adding patch feature in core. I'd like to have it in utils.

---

Update state outside React for Jotai

There's some demand on this. And, technically it's possible with v1 provider-less mode. Because we are not sure if we can keep the capability in the future version when we use newer features in React, that usage is not supported.

[redbar0n]

Actually, the useProxy function I was suggesting is just a rename of useSnapshot. Except that you wouldn't use ++state.count to modify it directly, but rather only state.add(1). Maybe the proxy object fields could be made immutable/readonly internally, to disallow any direct mutation like ++state.count. I just thought it would be a better API if the user didn't have to think about snapshots, but there were a 1-to-1 match (reading the code) between the proxy object created and the implicit snapshot returned by a function with the same name (useProxy). So that useProxy doesn't have to be a macro.

Actually, rather than the details of the potential API, it's much more interesting to discuss the overall architectural suggestion. Is it possible to achieve such a distributed state system, with some or all of the aforementioned features, with either Valtio, Zustand or Jotai, or any other library you know of?

[dai-shi]

Actually, the useProxy function I was suggesting is just a rename of useSnapshot.

Oh, I misunderstand this. So, this is already possible. You can try import { useSnapshot as useProxy } from 'valtio'.

Actually, rather than the details of the potential API, it's much more interesting to discuss the overall architectural suggestion. Is it possible to achieve such a distributed state system, with some or all of the aforementioned features, with either Valtio, Zustand or Jotai, or any other library you know of?

Well, I know all of the three and if it were possible, I would have done combining all to create a dream state manager. (though, the selling point of zustand is the bundle size, so it can't be achieved.) That said, I've spent a lot of time with react-tracked and learned some people like proxy solution but others don't, so there couldn't exist the combination, unfortunately. Besides having too many features is bad for cognitive load.

[redbar0n]

Thanks. I think many like mutation based API's because they are simpler and more intuitive. As evidenced by the success of Immer. I also think the reason some don't like proxy solutions is because it is based on mutation, which is scary (shared mutable state and all...), because when scaling up it can easily become hard to know when and where something was mutated (debugging). So immutability is all the rage these days. But imagine if the proxy provided a mutation based API, but was actually immutable under the hood...

Meaning: it would copy-on-write (though structurally share data, like Valtio), but also provide a log of which actions were performed (for time-travelling and patch generation) and from where in the code (for tracking/analytics and debugging).

[akutruff]

I've been writing something similar. I have a proxy that records first writes.

Below is the write function for a proxy implementation I'm working on because even though valtio is really great, it doesn't include delta tracking. Given the amount of proxying and side-car WeakMaps<> being used in proxy based state tracking this is a drop in the bucket to enable for those that wish to opt-in. I only included the code for JS objects. Arrays, maps, sets work as well.

In a nutshell, there's a [CURRENT_DELTA] property on state objects that records the PREVIOUS value of properties and writes the new value directly to the target object. This is important as other frameworks like Immer focus on the new values. This subtle change has several benefits.

(Note: Instead of adding properties to the state object, a separate WeakMap can be used for storing deltas so ignore that detail please.)

The first time a property value on the proxy is written we add the object to a modified list. Right afterwards, the current value of the property is recorded prior to mutating the underlying object.

At the end of the action handler, we just copy the deltas to a history and clear the recorded values. This is faster than doing full snapshot comparison since we've been tracking modification the entire time.

const modified = new Set<any>();

export const objectProxyHandler: ProxyHandler<any> = {

    //Typical proxy pattern for creating wrapper proxies as hierarchy is accessed.  Nothing special here for deltas really.
    get(target: any, propertyKey: PropertyKey, receiver?: any) {
        switch (propertyKey) {
            case PROXY: {
                return receiver;
            }
            default: {
                const propertyValue = Reflect.get(target, propertyKey);
                return propertyValue?.[PROXY] ?? (
                    typeof propertyValue !== 'object' || !propertyValue
                        ? propertyValue
                        : createRecordingProxy(propertyValue));
            }
        }
    },

    set: function (target: any, propertyKey: PropertyKey, value: any) {
       // CURRENT_DELTA tracks change.  It was added to the target when the proxy was created. Could be lazy initialized too.
        const delta = target[CURRENT_DELTA];
        if (!delta.hasOwnProperty(propertyKey)) {
            modified.add(target);

            const currentValue = target[propertyKey];
            if (currentValue) {
                target[CURRENT_DELTA][propertyKey] = currentValue;
            } else {
                target[CURRENT_DELTA][propertyKey] = undefined;
            }
        }

        return Reflect.set(target, propertyKey, value);
    },
};

// call at the end of actions.
export function commitDeltas() {
    const changes: = new Map<any, any>();

    for (const target of modified) {

        const delta= target[CURRENT_DELTA];

         changes.add(target, delta);
         //clear recorded delta value for next time
         target[CURRENT_DELTA] = Object.defineProperty({}, TARGET, {
            value: target, configurable: true, writable: true, enumerable: false
         }
    }
    modified.clear();
    return changes;
}

[dai-shi]

I feel like I'm convinced that the mental model of valtio is different from that of immer or alike. And, I think it's the novel part and the flip side of immer.

So, my proposition stays: We definitely want "history" feature like undo/redo in utils not in core (however, adding some kind of hook in core for extension might be possible), and it might be something different from immer-like patches, implementation-wise.

I wish someone who gets familiar with valtio internals to help, but in the mean time, can anyone help understand how the "history" feature would be like, api-wise?

[akutruff]

shrug You're doing change tracking in core. It's the source of truth.

Not sure what to say about the blanket rule of not touching core as a priority. It's your party, but it gives me hesitation to digging in further and making PR's.

[dai-shi]

My apologies if I discouraged your contribution. We have a huge desire to keep the core minimal, and believe the "history" feature should be opt-in. Not all users need that feature.

Yes, it's tied to the core, and I can work on it because of the huge desire. But I'm not sure if I fully get the requirement of the feature. Would you or anyone help designing the api? any possible ideas?

[redbar0n]

state.previous seems like an obvious and intuitive start for an API. I imagine it could be enabled if the history util is installed, which would plug in to the Valtio proxy under-the-hood. That might be better, to keep a small API surface and encapsulation, than to expose tools from a utils with which to manipulate the state, like previousHistory(state) or similar. Just my 2 cents.

[dai-shi]

Thanks. How about this?

import { proxy } from 'valtio'
import { enableHistory, goPreviousHistory, goNextHistory } from 'valtio/utils'

const state = proxy({ count: 0 })
enableHistory(state)
state.count += 1
goPreviousHistory(state)
console.log(state.count) // ---> 0

Should we throw an error if there's no history to go back, or return false?

---

Looking at OP

It would be an efficient mechanism to implement undo history and distribute state changes (e.g. via websocket)

The enableHistory doesn't solve the second use case, which is true because valtio is not patch based. So, it actually makes sense (at least to me). (btw, I'm interested in yjs and my side project (far from complete) is valtio-yjs.)

[redbar0n]

It’s kinda the opposite of what I was suggesting. But if you go with that then I reckon that ‘false’ is better than an error, since clicking undo once to many wouldn’t be exceptional. Besides, a ‘hasPreviousHistory(state)’ returning false would be good for setting the accessibility of an undo button (greying it out when no previous history exists).

[zcaudate]

I'm kinda against this being added to valtio core because the whole thing is super simple and really composable.

It's super easy to add a history solution by having two proxies - one showing the current state and one storing history state - implementing a listener on the current proxy to push to the history state.

@akutruff your code could be written as a subscribe function to enhance an existing proxy.

This is how I'd imagine how state of the two proxies are changing:

a = proxy({value: 1})

historyA = proxy({})
var [goBack, goForward] = recordHistory(a, historyA)
historyA // {log: [{value: 1}] at: 0}

a.value = 2
historyA // {log: [{value: 1},{value: 2}] at: 1}

a.value = 3
historyA // {log: [{value: 1},{value: 2},{value: 3}] at: 2}

goBack(historyA)
historyA // {log: [{value: 1},{value: 2},{value: 3}] at: 1}
a // {:value 2}

a.value = 4
historyA // {log: [{value: 1},{value: 2},{value: 4}] at: 2}

[zcaudate]

If I were to Implement it, recordHistory(Current, History) would be

• adding a synchronous subscribe function:
  Valtio.subscribe(Current, () => updateHistory(Current, History), true)
• create some state {disableUpdate: false} - to avoid cyclic update dependencies
• goBack and goForward would set disableUpdate to true, change the current state of both Current and History and the set disableUpdate back to false again.
• initialise the starting state.

I think that's it.

[zcaudate]

you may want to use https://github.com/immutable-js/immutable-js for peace of mind that someone doesn't change the history out from under you as well as getting better memory usage for large history.

[akutruff]

@zcaudate Thanks for the response. Questions:

How would you handle nested objects?

const person = proxy({ name: { first: 'james', last: 'bond' } });

person.name.last = 'wethers';    

If I wanted history on the name property then would the ProxyHandler for person need to also call history() on the nested object?

var [goBack, goForward] = recordHistory(a, historyA)

For that API, does it need to be in a linked list fashion where goBack and goForward captured historyA instead of something thats less stateful on the operations?

const previous = goBack(historyA);  // historyA is not affected
const previousPrevious = goBack(previous); // previous is not affected

Regardless of whether proxies are used or where the code is modified - It seems weird to hide the state of the history and not have this be patch based where the deltas are tracked and exposed so that API consumers can use that information directly without having to reverse engineer it themselves. I want a timeline of changes so I can write rich code using the history.

[zcaudate]

@akutruff: I think you have to consider two things about your system:

1. how events propagate
2. how state is managed

if a system needs history as a feature, then you implement it according to your needs.

History is state and state management always has it's constraints and biases

• do you want speed over storage?
• do you have heavily nested documents or simple maps?
• do you have cyclic dependencies in your objects?
• do you need security on your objects?

Each of theme demands certain design decisions in your programs.

---

I'm not sure how valtio handles nesting but you could just snapshot the whole thing and clone it just to be safe. But I would recommend using proxies purely as a watchable node and not a full state management solution.

In clojure (where all datastructures are immutable), there is a thing called an atom -> which is basically a proxy. Most clojure based systems are implemented based just on that

https://clojuredocs.org/clojure.core/swap! https://clojuredocs.org/clojure.core/reset!

---

So yeah, if you need a robust state management solution, I'd go with immutablejs and doing atomic swaps on immutable data but if you just want a simple history implementation, then the approach I outlined should work fine.

[zcaudate]

Also immer doesn't really solve the history/state problem. it just makes it nicer to do updates on state -> but you can still mess around with object tree that it's manipulating. I like to think of it as more of a library for doing simple performant state updates rather than use it as a state management solution.

[akutruff]

I am familiar with state management. No education necessary.

Snapshotting the whole thing is not an option if you have lots of objects. That's the case I'm dealing with.

Definitely will not be using immutable-js. It's too intrusive. One of the main reasons I'm looking at this project (and previously immer) is that I want a plain old JS object hierarchy that I can write in a mutable style.

Nested objects and normalized data are a min requirement for me as well.

[zcaudate]

I don't know then. Going by what you wrote, I think I'd be culling the objects first or just saving the deltas and then reconstructing as needed.

If all of those objects are mutable and pointing at each other, you're going to have a hard time with history.

[dai-shi]

It’s kinda the opposite of what I was suggesting.

@redbar0n Oh, that's not my intention. I just want to learn what history api people have in mind, and I THOUGHT I picked your idea. Can you draft your api idea?

[zcaudate]

@dai-shi: going back to the comment from #171,

I'd like a function subscribeDiff that would propagate the changes in the proxy to it's listeners: would it be possible the changes to proxy-compare as a event source?

a = proxy({})
subscribeDiff(a, console.log)
a.hello = 1 // [["+", ["hello"] 1]] - or a similar format to express diffs
a.hello = 2 // [["-", ["hello"] 1] ["+", ["hello"] 2]]
a.world = 3 // [["+", ["world"] 3]]
delete a.hello // ["-", ["hello"] 2]

then history would be just be patching and unpatching the deltas.

[dai-shi]

would it be possible the changes to proxy-compare as a event source?

Yes, and no. Technically, it may help, but in practice the current proxy-compare doesn't depend on it. It only depends on object immutability. Given that proxy-compare is pretty stable and used in multiple projects, we don't want to change it. (Here's a top secret: the primary reason I developed valtio is to utilize proxy-compare.)

then history would be just be patching and unlatching the deltas.

This example is good and I assume OP should have a similar idea to this. Actually, this has been requested before, and I was kind of between positive and negative. Now, I understand this would resolve this issue, and hopefully you could work on the history feature based on it. Let me draft a PR.

[dai-shi]

89 is reopened.

[dai-shi]

(btw, I'm interested in yjs and my side project (far from complete) is valtio-yjs.)

https://github.com/dai-shi/valtio-yjs