Change signature of reducers to be the same as react components with hooks

[inverted-capital]

With the recent modifications removing direct programmer access to the chain continuation primitives (promise, resolve, reject), and the addition of concurrency for the system reducer, the following problem arises:

What is the difference between responding to an action and returning a state, and how should this be mapped to the common language operation of returning from a function ?

I suggest we also add the problem of how to make large disparate state available to the programmer, and present a solution that can be implemented across any modern language, with an implementation in javascript being the primary programming model I would like feedback on. The cause for this problem was the introduction of concurrency that is used for the system reducer, in order that the chain should not block on any system action, which means multiple pending states need to be tracked.

I am assuming that responding to a request, as well as returning a state, are accepted as both being equally vital to our programming model.

The problem can be split in two:

Backend representation on chain

From the chains perspective, it is taking request and reply primitives off the channels, fetching the stored state (which might include some IPFS links) and begin given back the next state. This should be sufficient to mimic all programming language implementations of sync and async programming. As proof I suggest that we can indeed implement redux style state returning, as well as react style hooked components using this same backend representation. Probably callbacks, events, and generators are possible too.

The key assumption being that any programming language allows halting and disposing any time a call is made into our supplied interface functions, so that the programmer supplied functions can be rerun once reply primitives have been received.

So what the chain expects back is an object that contains a returned state, and an optional array of request primitives in the order they were made for re-execution purposes, then an optional single reply primitive which is the response to the received request. The presence of this reply signals that no further re-execution is required, and that no further tracking of asynchronous chain communication is required. Note that request primitives also track system calls like getState and remote queries.

Frontend presentation to the programmer

The frontend being the interface supplied to the programmer, each language would be free to add whatever pattern they prefer. For javascript we should pick our favorite, and allow config to switch between other preferences, or implement them using our favorite and signaling to config directly or with a higher order reducer. This allows programmers to use whatever style they are comfortable with.

I propose that we move from a pure function that takes current state plus a request and returns some state, to a function that takes in only some request, and returns what will be sent as a reply to that request, with access to state using an async api. The api is:

1. const state = await useState() which fetches the state of the chain, if required. Being async ensures that it is correctly resolved between concurrent executions of the function, and also provides the context with an opportunity to resolve any IPFS links, allowing for limitless state sizes to be handled with the same interface as used for concurrency.
2. await setState( state ) which requires that this thread first fetched the state without any other async breaks, ensuring no race conditions between threads

These two methods, plus the return value of the function being the reply to the request, should be sufficient to do everything encountered so far, and permit non blocking concurrent request processing, which can optionally switched off.

[inverted-capital]

Another benefit here is that execution can be frozen during a useState() call, to allow more opportunities to cut the block creation process in the event of slow to resolve IPFS queries.

[dougbear]

In terms of backend, (state, action) => (state', effects || ⊥) is a signature that it sounds like you're describing here, and it's a good generic representation of what it means to perform a stateful computation. The multivalued effects (which can be more actions), and the termination value ⊥ describe a tree with compute in the nodes that eventually terminates. The (state) => (state') describes the state updating as the overall process visits the next compute node.

For the frontend, I like the simplicity of (request) => (reply) and the idea of a hook-like language for addressing the state inside the function. It makes the separation of concerns very clear. It creates a clear sense of hierarchy in the programmers mind that the main purpose of any given code unit is to listen and respond, and that we might have a secondary interest in modifying the state. It also gives a clear foundational abstraction in useState and setState to express higher level hook concepts compositionally (ie, setting a specific path in the state is just a specified version of setState, such as setPath = (path, state) => setState(setIn(path, state)).

One thing I immediately think when I see await useState() and await setState() is that the await statements are a bit deceptive. From the perspective of the program (and by extension the programmer), these calls are resolved synchronously. From the perspective of the runtime, we know the state will be hydrated through as many rehydration events as necessary to present a fully hydrated subtree to the caller, but I don't think the entity processing the useState/setState call requires the await to behave asynchronously.

I think in the broadest possible view, leaving the await intact all the way through to userland code will simplify the codebase and unify system and user codebases, but it comes at the cost of possibly triggering confused mental model for the user.

[inverted-capital]

That's a great way to describe the signature. Addressing your concerns about the async await hook calls and expanding on the suggested composition of setState down using setPath, I offer the following changes:

1. Method changed to await useLockedState() => ( state, async setLockedState( state' ) )
2. Including the path variable as an optional parameter, and making it go across chains using the pathing conventions, instead of into state, which can still be included by decorating all methods with their setPath equivalents. Eg: await useLockedState( '../some/other/path' ) and await useLockedStateIn( '../some/other/path', [ 'some', 'path', 'in', 'state' ] )
3. Adding a way to set the state without locking it, by clobbering it using async useState() => state and async setState( state' ) which can both take path params

This has the benefit of allowing a chain to hold multiple locks across the app complex, to do atomic updates to state and rollback on fail. Scoping using the setPath extensions means less data traffic, particularly in the case of large states.

Hopefully this makes the async model we present to programmers easily understandable since it should be obvious that if you want to lock on a different chain, you have to wait for that to occur, then if you want to lock your own chain, it is the same interface, because fundamentally reading your own state is the same operation as reading from an arbitrary path, in this model. Whilst if the chain is in single threaded mode locking self state would be practically synchronous, but if multithreaded, you would definitely have to wait to know you had excluded other running instances of yourself.

The async model is definitely easier to write the system codebase, hopefully with these changes it also makes sense to the user codebase authors too ?

[inverted-capital]

having useState be an explicit call also means we do not have to load it before invocation which can save a lot of time, and then if the devs scope the request down to a subpath in the state, we can fetch even less.

[inverted-capital]

An epic gain for this model is the composability of multiple concurrent block proposals. The responder signature here is much more forgiving of changing the execution orders where multiple proposals are present, without having to re-execute anything. If the state changes where conflict can occur are scoped to narrow parts of the state, then even fewer conflicts can occur.

[inverted-capital]

It may be better to make the reducer be an async iterator ? Then yield would be used, which would give us a way to legitimately pause execution, rather than the simulation we provide. I do think this might be a little too repulsive to many devs, since not many people use generators, and even fewer use async generators 🤷

[dougbear]

I never understood the cultural repulsion to generators. Co-recursion is a very sound pattern for distributed systems. I would go so far as to say its the reason people like go. With async you get one shot to talk back to your caller, with generators there's many.

I would welcome more legitimate use cases of yield.

[inverted-capital]

Ok well lets say that the first time theres an issue with the callsites module which handles the async hooks, that we'll switch over to using async generators. Generators is way simpler to write system code for - hopefully it isn't too much of a shock for devs.