Derived scoped atom

[yf-yang]

These days I've gone through the abstract model of jotai and bunshi.

In my understanding, jotai seperates React component tree and "state dependency graph", make them independent, and allow them to be connected in any manner by calling useAtom. In order to allow any component to access any atom, atoms should be defined in the global scope.

Then the reusability issue occurs, so jotai offers Provider, but Provider has its own limitation: Under the cover of a Provider component, whatever atom we are calling with useAtom would create a new independent state. The fundamental issue is that there is no API to distinguish whether or not we want to "share" an atom with a component within the Provider, or we want to create a new one.

Therefore, with jotai-scope, Provider offers an additional parameter, when accessing atoms in the parameter array, new instances (new indendent useState calls) are created. Other atoms (states) are still globally shared.

If the understanding is correct, then in the atom dependency graph:

• if a derived atom appears in the parameter of scoped Provider, but its dependent atom is globally shared, then it means we create multiple atom instances which subscribe to the same atom (the dependency "edge" in the graph are copied). When the dependent atom changes, all the derived atom should change accordingly. This is not the case in this demo, so I'm wondering what's your thought.
• if a derived atom's dependent atom appears in the parameter of scoped Provider, but the derived atom is globally shared, seems it is still a valid case (the derived atom also has multiple instances). I am not sure if this is a clear behavior. To make things clear, maybe throw an error is better, but in user's use cases, maybe that could be frequently used, especially when they want to make reusable components and have multiple copies of them simultaneously.
• if a derived atom and its dependent both appears in the parameter of scoped Provider, then their dependency relationship disappears.

Therefore, I suggest we clarify the dependency rules just like bunshi. I'd like to compare jotai-scope with bunshi. In my understanding, jotai-scope is pretty like a simplified version of molecule:

const jotaiScopeEquivalentMolecule = molecule((mol_DEPENDENCY_NOT_ALLOWED, scope) => {
  sharedAtoms = []
  // sharedAtomScopes are those atoms NOT passed to Provider
  // this simplified molecule only accepts atom type as scope value
  for (const anAtomScope of sharedAtomScopes) {
    sharedAtoms.push(scope(anAtomScope));
  }
  return sharedAtoms;
})

Note that in this molecule, atoms passed by sharedAtomScopes are shared, but in jotai-scope, atoms passed to Provider have indenpendent instances.

I want to make this comparision so we can define a similar dependency rule. For example, we call those atoms provided to Provider scoped atoms, then

• scoped atoms' dependents are also scoped even if they are not specified.
• unscoped atom's scoped dependents are not taking any effect.

This is a draft lack of considerations. In fact, there are cases that multiple Providers with different sets of atoms provided nested together, and atoms can have intertwined dependency relationship. It really helps to think as a graph theory graph, I believe we can find some mathematically clean rules.

[dai-shi]

Thanks for opening the discussion up. Yes, your understanding is basically correct. If I were to correct something, the Provider in Jotai is designed as expected, but it's not flexible enough, so "bunshi" and "jotai-scope" are born. These use cases are out of the scope of Jotai core and left to the third-party libraries in ecosystem.

Now, I understand your confusion, because I had the same one. It's still unsolved, but let it go to get feedback like this. Yes, my goal is to make jotai-scope a lesser version of jotai-molecules=bunshi. The implementation is pretty different currently. Please read the code and you will see jotai-scope actually uses multiple stores, whereas bunshi uses just one.

Yes, for now, jotai-scope would work fairly well for primitive atoms, but for derived atoms the behavior is a bit confusing.

Please feel free to give suggestion for clean rules. The ground requirement for jotai-scope is that it should be a much simpler implementation and not full featured, compared to bunshi. Otherwise, there's no point of building a different one.

[yf-yang]

Well, I find the rules may be pretty straightforward.

Rules:

• Each Provider provides a new store instance.
  • NOTE: The store is shared by all the atoms passing to the Provider. The outmost store (without any explicit Provider) is the global store.
• A useAtom call must exist within a React component, and given a React component, we are able to get the store, so each [component, atom] pair maps to an unique atom instance (I am not sure if it should be called an "atom instance" or something)

Those two rules are enough to define vertices in the dependency graph, and for edges we just connect them together.

After checking the createScope code, now I am confused why derived atoms are not working. I'll try to debug the demo.

[yf-yang]

I suspect that vertices (atom instances) are created at the correct store, but current jotai-scope version is unable to handle edges between different stores.

https://github.com/pmndrs/jotai/blob/21326503cfeaca7689ac90c40b2e627ece5ed1a7/src/vanilla/store.ts#L383

Here, when derived atom are accessing their dependent atoms, they perform the lookup within the same store, so if an atom has different store (scope) with its dependent atom, the edge cannot be established, it will never subscribe to the correct dependent atom.

That's my analysis, but I cannot understand current jotai implementation very well, still need your precise judgment

[yf-yang]

Check this demo, previous demo is misleading because it is using useAtom and useScopedAtom simultaneously. I also suggest use useScopedAtom only in the example to avoid confusion.

[dai-shi]

The jotai store implementation only handles one store, so reading from a different store isn't possible.

when derived atom are accessing their dependent atoms, they perform the lookup within the same store

so, that's true by design.

we have two options: a) use multiple stores and somehow define clean rules, or b) use single store and create different atoms like bunshi.

I also suggest use useScopedAtom only in the example to avoid confusion.

Feel free to open a PR to improve examples.

[dai-shi]

(Or, the third option c) to develop a special store that reads atom value from the parent provider's store. But, it will be very complicated implementation-wise, and easy to become less performant.)

[yf-yang]

Well, I think the rule is clear enough for vertices (atom instances), and there is no need for extra rules for edges. In other words, given component, we merely find atoms and their dependent atoms' instances (which means their actual store), and connect them together, then things are done. That is conceptually sound. I did try option c and found that scope is coupled with React component tree, but store is not, so yes it would be pretty difficult. i have an idea to expose an additional param of readAtomState and readAtom (and setter functions), the param is the map from atom to their actual store, so that not too many patches are added. Anyway, we still need to change jotai's implementation. What's your opinion? BTW, I'm curious if this library will be merged to jotai one day? It seems everything is still compatible, the only one exception is that jotai Provider creates new atom instances if they are not explicitly specified (and they can't), but ScopeProvider creates new atom instances if they are explicitly specified. They are just dual. Maybe jotai-scope can accept breaking change since it's new and have < 100 weekly downloads……? If so, then it makes more sense to add an additional parameter in jotai to support jotai scope. If you are interested I can write a sample implementation. I can't get your point of option b, does it mean implement something like bunshi or just use bunshi to achieve atom dependency?

[dai-shi]

We don't currently plan to patch jotai core for jotai-scope nor merge jotai-scope into there. That's the given constraints. So, the challenge is that we implement the feature in this repo only. As jotai-scope doesn't hit v1, and still new, so breaking change is fine.

[yf-yang]

All right, I found the rule has some flaws. For the following example:

const primitiveAtom = atom(0);
const scopedDerivedAtom = atom((get) => get(primitiveAtom) + 1);
const derivedAtom1 = atom((get) => get(scopedDerivedAtom) + 1);
const derivedAtom2 = atom((get) => get(primitiveAtom) + 3);

const Counter = () => {
  const [p, setP] = useScopedAtom(primitiveAtom);
  const sd = useScopedAtomValue(scopedDerivedAtom);
  const d1 = useScopedAtomValue(derivedAtom1);
  const d2 = useScopedAtomValue(derivedAtom2);
  return (
    <>
      <div>
        <span>p: {p}</span>
        <button type="button" onClick={() => setP((x) => x + 1)}>
          increment
        </button>
      </div>
      <div>
        <span>sd: {sd}</span>
        <span>d1: {d1}</span>
        <span>d2: {d2}</span>
      </div>
    </>
  );
};

const App = () => {
  return (
    <div>
      <h1>Global</h1>
      <Counter />
      <h1>First Provider</h1>
      <ScopedProvider atoms={[scopedDerivedAtom]}>
        <Counter />
      </ScopedProvider>
      <h1>Second Provider</h1>
      <ScopedProvider atoms={[scopedDerivedAtom]}>
        <Counter />
      </ScopedProvider>
    </div>
  );
};

Given the rule

A useAtom call must exist within a React component, and given a React component, we are able to get the store, so each [component, atom] pair maps to an unique atom instance.

We can draw the following graph: Here,

• a vertex => a useAtom call
• text => atom
• color => store/scope
• [text, color] pair => atom instance

Now take a look at those d1s, it has only one scope/store, the global one, so there should be exactly one d1 instance. However, d1's dependency atom sd has 3 instances, so 1 instance dependents on 3 instances of the same atom, which is wrong.

Solution 1: bunshi-alike rule

If we change to something similar to bunshi's rule:

• When calling useAtom, we find the scope/store of the atom itself, and find the scope/store of all of its dependencies (parent, ancestor...), which forms a dependency scope set. Two dependency scope set is the same iff all of their elements are the same. Each distinct dependency scope creates one new atom instance.

With the new rule, the graph becomes this:

And more complex example:

const primitiveAtom = atom(0);
const scopedDerivedAtom1 = atom((get) => get(primitiveAtom) + 1);
const scopedDerivedAtom2 = atom((get) => get(primitiveAtom) + 2);
const derivedAtom1 = atom((get) => get(scopedDerivedAtom1) + 2);
const derivedAtom2 = atom((get) => get(scopedDerivedAtom2) + 2);
const derivedAtom3 = atom(
  (get) => get(scopedDerivedAtom1) + get(scopedDerivedAtom2),
);

const Counter = () => {
  const [p, setP] = useScopedAtom(primitiveAtom);
  const sd1 = useScopedAtomValue(scopedDerivedAtom1);
  const sd2 = useScopedAtomValue(scopedDerivedAtom2);
  const d1 = useScopedAtomValue(derivedAtom1);
  const d2 = useScopedAtomValue(derivedAtom2);
  const d3 = useScopedAtomValue(derivedAtom3);
  return (
    <>
      <div>
        <span>p: {p}</span>
        <button type="button" onClick={() => setP((x) => x + 1)}>
          increment
        </button>
      </div>
      <div>
        <span>sd1: {sd1}</span>
        <span>sd2: {sd2}</span>
        <span>d1: {d1}</span>
        <span>d2: {d2}</span>
        <span>d3: {d3}</span>
      </div>
    </>
  );
};

const App = () => {
  return (
    <div>
      <h1>Global</h1>
      <Counter />
      <h1>First Provider Layer 1</h1>
      <ScopedProvider atoms={[scopedDerivedAtom1]}>
        <Counter />
        <h1>First Provider Layer 2</h1>
        <ScopedProvider atoms={[scopedDerivedAtom2]}>
          <Counter />
        </ScopedProvider>
      </ScopedProvider>
      <h1>Second Provider Layer 2</h1>
      <ScopedProvider atoms={[scopedDerivedAtom2]}>
        <Counter />
        <h1>Second Provider Layer 1</h1>
        <ScopedProvider atoms={[scopedDerivedAtom1]}>
          <Counter />
        </ScopedProvider>
      </ScopedProvider>
    </div>
  );
};

There are five counters, the graph is like:

Solution 2, scoped derived atom takes no effect

It makes no sense to put a derived atom to Provider, since its source has exactly one instance, even if we create multiple instances of the derived atom, they are actually the same, so all the vertices in the examples above can be transparent-colored. Then, why not just ignore them? I haven't examined this idea thoroughly, if that's correct (is it easy to implement?), then things would become much easier.

---

Graphviz source code of those three images:

digraph G {
    rankdir=BT;

    n1 [label="p"];
    n2 [label="sd"];
    n3 [label="d1"];
    n4 [label="d2"];

    n5 [label="sd", fillcolor=cornsilk1, style=filled];
    n6 [label="d1"];
    n7 [label="d2"];

    n8 [label="sd", fillcolor=burlywood, style=filled];
    n9 [label="d1"];
    n10 [label="d2"];

    n2 -> n1;
    n3 -> n2;
    n4 -> n1;

    n5 -> n1;
    n6 -> n5;
    n7 -> n1;

    n8 -> n1;
    n9 -> n8;
    n10 -> n1;
}

digraph G {
    rankdir=BT;

    n1 [label="p"];
    n2 [label="sd"];
    n3 [label="d1"];
    n4 [label="d2"];

    n5 [label="sd", fillcolor=cornsilk1, style=filled];
    n6 [label="d1", fillcolor=cornsilk1, style=filled];
    n7 [label="d2"];

    n8 [label="sd", fillcolor=burlywood, style=filled];
    n9 [label="d1", fillcolor=burlywood, style=filled];
    n10 [label="d2"];

    n2 -> n1;
    n3 -> n2;
    n4 -> n1;

    n5 -> n1;
    n6 -> n5;
    n7 -> n1;

    n8 -> n1;
    n9 -> n8;
    n10 -> n1;
}

digraph G {
    rankdir=BT;

    n1 [label="p"];
    n2 [label="sd1"];
    n3 [label="sd2"];
    n4 [label="d1"];
    n5 [label="d2"];
    n6 [label="d3"];

    n7 [label="sd1", fillcolor=cornflowerblue, style=filled];
    n8 [label="sd2"];
    n9 [label="d1", fillcolor=cornflowerblue, style=filled];
    n10 [label="d2"];
    n11 [label="d3", fillcolor=cornflowerblue, style=filled];

    n12 [label="sd1", fillcolor=cornflowerblue, style=filled];
    n13 [label="sd2", fillcolor=tomato, style=filled];
    n14 [label="d1", fillcolor=cornflowerblue, style=filled];
    n15 [label="d2", fillcolor=tomato, style=filled];
    n16 [label="d3", fillcolor=slateblue1, style=filled];

    n17 [label="sd1"];
    n18 [label="sd2", fillcolor=violetred1, style=filled];
    n19 [label="d1"];
    n20 [label="d2", fillcolor=violetred1, style=filled];
    n21 [label="d3", fillcolor=violetred1, style=filled];

    n22 [label="sd1", fillcolor=royalblue, style=filled];
    n23 [label="sd2", fillcolor=violetred1, style=filled];
    n24 [label="d1", fillcolor=royalblue, style=filled];
    n25 [label="d2", fillcolor=violetred1, style=filled];
    n26 [label="d3", fillcolor=purple, style=filled];

    n2 -> n1;
    n3 -> n1;
    n4 -> n2;
    n5 -> n3;
    n6 -> n2;
    n6 -> n3;

    n7 -> n1;
    n8 -> n1;
    n9 -> n7;
    n10 -> n8;
    n11 -> n7;
    n11 -> n8;

    n12 -> n1;
    n13 -> n1;
    n14 -> n12;
    n15 -> n13;
    n16 -> n12;
    n16 -> n13;

    n17 -> n1;
    n18 -> n1;
    n19 -> n17;
    n20 -> n18;
    n21 -> n17;
    n21 -> n18;

    n22 -> n1;
    n23 -> n1;
    n24 -> n22;
    n25 -> n23;
    n26 -> n22;
    n26 -> n23;
}

[yf-yang]

Maybe we can just recommend developers avoid sending derived atom to Provider, it's the simplest solution 😂.

However, solution 1 (bunshi-alike scope rule) is still needed, consider this example:

const primitiveAtom = atom(0);
const scopedAtom = atom(1);
const derivedAtom1 = atom((get) => get(scopedAtom) + 1);
const derivedAtom2 = atom((get) => get(primitiveAtom) + get(scopedAtom) + 3);

const Counter = () => {
  const [p, setP] = useScopedAtom(primitiveAtom);
  const [s, setS] = useScopedAtom(scopedAtom);
  const d1 = useScopedAtomValue(derivedAtom1);
  const d2 = useScopedAtomValue(derivedAtom2);
  return (
    <>
      <div>
        <span>p: {p}</span>
        <button type="button" onClick={() => setP((x) => x + 1)}>
          increment
        </button>
      </div>
      <div>
        <span>s: {s}</span>
        <button type="button" onClick={() => setS((x) => x + 1)}>
          increment
        </button>
      </div>
      <div>
        <span>d1: {d1}</span>
      </div>
      <div>
        <span>d2: {d2}</span>
      </div>
    </>
  );
};

const App = () => {
  return (
    <div>
      <h1>Global</h1>
      <Counter />
      <h1>First Provider</h1>
      <ScopedProvider atoms={[scopedAtom]}>
        <Counter />
      </ScopedProvider>
      <h1>Second Provider</h1>
      <ScopedProvider atoms={[scopedAtom]}>
        <Counter />
      </ScopedProvider>
    </div>
  );
};

demo link conceptual visualization:

image source:

digraph G {
    rankdir=BT;

    n1 [label="p"];

    n2 [label="s"];
    n3 [label="d1"];
    n4 [label="d2"];

    n5 [label="s", fillcolor=cornsilk1, style=filled];
    n6 [label="d1", fillcolor=cornsilk1, style=filled];
    n7 [label="d2", fillcolor=cornsilk4, style=filled];

    n8 [label="s", fillcolor=burlywood, style=filled];
    n9 [label="d1", fillcolor=burlywood, style=filled];
    n10 [label="d2", fillcolor=burlywood4, style=filled];

    { rank=max; n1; }
    { rank=source; n2; n5; n8; }

    n3 -> n2;
    n4 -> n1;
    n4 -> n2;

    n6 -> n5;
    n7 -> n1;
    n7 -> n5;

    n9 -> n8;
    n10 -> n1;
    n10 -> n8;
}

[dai-shi]

wow, nice graphs. that reminds me of mermaid support in markdown.

Maybe we can just recommend developers avoid sending derived atom to Provider, it's the simplest solution

I thought the simplest solution (as implemented now) is to recommend always send derived atoms to Provider, if you want to use it under the tree.

I assumed that people want to list up atoms that are scoped, but maybe that was my wrong assumption. I think people want to list atoms that are read from the parent provider (= paththrough atoms. The implementation will be tricker, but we can try.

[yf-yang]

I thought the simplest solution (as implemented now) is to recommend always send derived atoms to Provider, if you want to use it under the tree.

As the last example illustrates, there exists cases that one derived atom depends on atoms from different scopes. Then things are broken. I do think it would be quite common in a deep nested tree with lots of reusable components, what do you think?

I also want to explain the problem from another perspective: atoms' conceptual dependency graph is decoupled from React component tree. It is something independent of the tree. However, by implementing "scope" with React.Context, the scope is coupled with the tree. Therefore, even we do not explicitly scope those derived atoms, their scope are defined by their dependency atoms' scopes when calling useAtom with the derived one, so it is inevitable.

Moreover, when one derived atom depends on atoms from multiple different scopes, we can't even tell which scope it is in, so in these graphs, I use a different color to represent those atoms. A rule similar to bunshi is needed, the scope of the derived one's key is a set of multiple scopes (if I got it right, bunshi use the deepCache to achieve such behavior). In other words, scopes are not indexed by Provider, they are indexed by the union of Provider and the derived atom's dependency scopes.

[dai-shi]

I also want to explain the problem from another perspective

Great point. In that perspective, do you have any idea like a simplified API like bunshi?

[yf-yang]

I think I got it work: Example: https://github.com/jotaijs/jotai-scope/issues/4#issuecomment-1756697537 https://codesandbox.io/s/eloquent-joliot-lnr46v?file=/src/atom.jsx

This implementation is modified from https://jotai.org/docs/guides/core-internals#second-version. Be aware that I clearly distinguish atom instance and atom in the demo implementation's naming.

Concepts

In this world, we have four conecpts.

Atom

In this world, we divide atom (config) and atom instance into two concepts. An atom by default has an anonymous atom instance, this setting is compatible with current jotai implementation.

Scope

Each ScopeProvider creates a scope. It is implemented by React Context, so it is a concept in React component tree.

Atom instance

In React component tree, each atom is mapped to an atom instance. The instance is created if the ScopeProvider explicitly declares that it creates a new instance of the atom. Otherwise, the atom inherits its instance from its parent scopes. This is similar to bunshi.

Store

Now store is no longer the store of atoms, it is the store of atom instances.

---

When there is no scope exists, all the atom has exactly one instance, which is the anonymous instance itself. The store falls back to the store of atoms, that is current jotai world. By splitting scope and store into two concepts, we can still make it compatible with the original store implementation.

Abstraction

Now let's consider this model in an abstract dependency graph. atom = the class of vertices. atom instance = a vertex. store = the graph. scope is a React component tree concept, by calling useAtom within the specific scope, a vertex of a vertex class is created in the dependency graph.

Breaking change

ScopeProvider is incompatible with Provider. For ScopeProvider, declared atoms' instances are created, undeclared atoms' instances are inherited from their parent scope (parent ScopeProvider). For Provider, a new store is created, it's just like undeclared atoms' instances are created. Those two behaviors are incompatible. Personally I'd prefer current ScopeProvider implementation. Moreover, I think there is no specific use case for using Provider/create different stores.

Other notes

After several tries I find it is inevitable to change jotai's implementation. For derived atoms, we have to modify the getter/setter to implement the concept of scope.

[yf-yang]

BTW I find that it is painful to deal with readAtom/writeAtom's type signatures to patch jotai 😅, maybe passing a function instead of the map would be better.

const scope = new Map<AnyAtom, AnyAtom>()
function getAtomInstace<V>(atom: Atom<V>): Atom<V> {
  return (scope.get(atom) ?? atom) as Atom<V>;
}

[dai-shi]

I haven't read your code yet, but it sounds like the idea is similar to the original one of mine (not committed). I gave it up because it can't work with derived atoms.

Personally I'd prefer current ScopeProvider implementation.

Yeah, one key point is that our implementation should be much far simpler than bunshi. Otherwise, we should use bunshi.

to change jotai's implementation

One thing that is acceptable would be using this instead of config in atom.ts. It allows us to extend the config.

[yf-yang]

Maybe you can take a look of the implementation. I don't think it is bunshi, the key difference is bunshi makes a single instance for each ScopeProvider, but here each atom provided to ScopeProvider has an instance. I did try to wrap bunshi, but it is hard to deal with this part.

[yf-yang]

I also tried to add a property in atom, but it is not achievable because atom is something outside of the React world while scope is inside. Therefore the atom itself is not sufficient to distinguish scopes, at least we need to pass something more to readAtom/writeAtom to distinguish them.

[dai-shi]

    const scope = new Map(parentScope);
    for (const anAtom of atoms) {
      // create a new Atom instance by making a copy
      scope.set(anAtom, { ...anAtom });

Yeah, I know what you mean. I did the same before. So, the challenge is how we could implement it very simply. I know the capability is different from bunshi, which allows to read props.

[yf-yang]

implement it very simply.

It means we are still trying to avoid adding a new parameter to readAtom/writeAtom, right? I think I can give a more thorough proof why this is not viable on the basis of previous comment https://github.com/jotaijs/jotai-scope/issues/4#issuecomment-1763584796. I will think twice if there exists a better approach before I give a conclusion.

[dai-shi]

we are still trying to avoid adding a new parameter to readAtom/writeAtom, right?

Correct. I'm retrying my original idea with the following change.

One thing that is acceptable would be using this instead of config in atom.ts. It allows us to extend the config.

[dai-shi]

https://codesandbox.io/s/funny-wood-4h2p9n?file=/src/App.jsx with #5.

[yf-yang]

Close the issue in favor of #5