search

choojs / choo

:steam_locomotive::train: - sturdy 4kb frontend framework
https://choo.io/
MIT License
6.78k stars 595 forks source link

[discussion] how can we improve state management? #252

Closed yoshuawuyts closed 7 years ago

yoshuawuyts commented 8 years ago

As I'm working on a real-world application I'm noticing there's friction using stores, and I'm becoming unsure if namspacing is the best construct available.

Nested data structures are hard to model and use. Say we're building a giant form website with multiple forms on it, we might have the fields of "form", "questions" and "fields". We might perhaps want to track the state of these separately; as having a single blob make up 90% of our application doesn't quite feel right. "form", "questions" and "fields" all have their own little bits of logic and what not.

rdbms

An approach we could do here is have each of these be expressed in a relation of "one-to-one", "many-to-many", "many-to-one" (oneToOne, many, fk in RDBMS-speak respectively). This would allow us to define the fields between the respective items. And flatten them in our representation, making it easier to view and reason about. There's prior art in redux-orm and querying a redux store

The redux docs state the following:

In a more complex app, you’re going to want different entities to reference each other. We suggest that you keep your state as normalized as possible, without any nesting. Keep every entity in an object stored with an ID as a key, and use IDs to reference it from other entities, or lists. Think of the app’s state as a database.

There's also reselect which creates higher level views on top of normalized data, and updates to match. I like this idea because it matches the idea of how I think about data.

namespaces

But all of this raises the question: how useful are namespaces at this point? If our data is completely intertwined, and method calls on values rely on all data being available (e.g. "how many fields are completed on this form?"), then what's the point of having namespaces? In a sense we'd be implementing namespaces as a way to interact with our flat data.

Some considerations we might need to make regarding namespaces:

I suspect it's definitely possible, and would shrink the codebase significantly

I feel the endgame of RDBMS and selectors is to implement a graph structure where relationships can be used to query data (e.g. "give me all questions of form foo", "give me all fields of question bar").

Now what I wonder is if we could leapfrog over the implementation details, of RDBMS and selectors and implement a graph structure directly on top of a single object. This would be similar in spirit to @mcollina's levelgraph.

Now my knowledge on databases isn't great, and I don't have any prior experience implementing databases (relational, graph or otherwise) so any input here would be grand.

edit: for those unfamiliar with graph databases, I've found a great article explaining the differences between graph and RDBMS databases: https://neo4j.com/developer/graph-db-vs-rdbms/. Especially the chart showing the same data expressed in the different paradigms is interesting I found:

relational

relational architecture example

graph

graph architecture example

wrapping up

I'd be keen to hear people's thoughts on this. Like I said this post is coming from seeing choo's existing abstractions break down at a larger scale, and the desire to do better. Thanks for making it this far down the post; let me know what you think :v:

Related issues

donaldpipowitch commented 8 years ago

I don't know. Maybe this is an interesting link, too: https://circleci.com/blog/why-we-use-om-and-why-were-excited-for-om-next/ It is about Om Next and why it uses graphs for data management.

tdeekens commented 8 years ago

Really hard questions.

Slightly feel a tendency to reselect over an orm-y solution. Maintaining nested data often becomes error prone and managing it other than deriving from a store feels contra redux and I'd fear losing out of advantages.

I feel sceptical in wrapping data in objects in general. I'd rather but it in data structure if need be e.g. a graph. Is there maybe any way to create views towards the single data structure in a way like lenses (http://ramdajs.com/docs/#lens)?

kemitchell commented 8 years ago

I'm no longer using choo for commonform.org directly, but I've retained a good bit of its structure, including namespaces, the reducer-action dichotomy, and Object.assign/xtend mutations. I'm still riding the train ;-D

Namespacing actions feels very worthwhile. Then again, I suppose that could be accomplished with a naming convention alone, without any built-in, object-based dispatch mechanism. An EventEmitter can handle prefixes fine.

As for namespaces as a data segregation method, it's been tough. My application has a few "view modes" corresponding to entirely different ways the page can look at any given time: There's "browsing" views for listing content, "editor" views for viewing and changing content, and so on. I've ended up doing terrible things, like firing off reductions from the router to clear out state for views that are now hidden, or getting around the data barriers by encoding information in URLs that hit the router.

Then there's the inevitable problem of wanting to write an action handler that needs state from multiple namespaces. The effect of that structural limitation is to glom lots of state together under ever-larger namespaces. That's in direct tension with modularizing (meaning, in choo, namespacing) actions and reducers. It's the old which-code-goes-with-which-data OOP problem again.

At first blush, the idea of "importing" relations or graph structures, wholesale, kind of turns my stomach. Either would bring with it a whole mound of concepts dwarfing all the other fundamental primitives in choo combined. "If you wish to make a choo app from scratch, you must first understand relational theory?" Then again, it's clear the point of choo is to offer something more than just a more modular view concept, like React, with room to plug and play your choice of data management solution.

I'll put some more though on it. My gut tells me it would still be awesome if choo took up even a little bit more of the state-management problem, even if it still left substantial room to plug in roll one's own approach, at a more-than-TodoMVC level of complexity.

typeetfunc commented 8 years ago

I recommend to pay attention to Datalog systems - Datomic/DataScript. D Datalog is more flexible, dynamic and powerful than SQL(RDBMS), but more complicated. The main problem of Datascript is "non-fractality" - there is no possibility to make query for nested databases. But I am think it can be solved.

jonaskello commented 8 years ago

I like the ideas of namespaces in Choo. I always try to modularize each "feature" in the app so it does not know anything at all about other "features" in the app. And preferably not about any library either. Kind of what is mentioned in this old talk. However I also always end up with the problem that some data is needed by multiple features. In react-redux-saga apps I solve this in the sagas as they have access to the full state. But that couples the features together through state which is not good.

I mainly have two kinds of data, UI-state data (transient), and database data (data that came from the server where it is persisted in a database). Sometimes UI-state data need to shared, like when a currently selected product has information shown in two features. I can have the currently selected product id at one place in the state and both features render from that so they are in sync. There is also need to share database data, for example data such as a product name that need to be shown in two different features.

One idea would be to get looser coupling by having data-duplication but not code duplication by way of sharing reducer logic. For example, if a return from a server call causes an action like PRODUCT_DATA_LOADED, we could put this data in one place in the state. But each feature has it's own sub-part of state which it is only allowed to read so it will not be able to get the product data. So instead we can have every feature store this same data in it's sub-state. So this means every feature must have reducer logic to do this. But that reducer logic code can be shared. So the code would be shared but the data would be duplicated in the state (as it appears in multiple sub-states). Same can be done if an action such as SELECTED_PRODUCT_ID_CHAGE comes. Every feature could store the new id in it's own sub-state.

I think the advantage of this would be that a single feature is not dependent on state from any other feature and therefore can easily be re-used. But I have not tried this and do not know if it would work out in a larger project. Maybe someone else have tried it?

mantoni commented 8 years ago

It should be easy enough to allow choo apps to be nested. What I mean is that multiple parts of an application can usually function entirely by themselves, only requiring few "external" objects (like a WebSocket) that could be passed down in a function call.

In one of the biggest apps I wrote, we came up with a component model convention that worked like this:

exports.component = function (options) {

  // Initialize stuff here

  return {
    html: some_template_thingy,
    build: ($target) => {
      // Bind component to $target
    }
  };
});

What I like about this approach is that it's a convention, it's not bound to any framework or a library. Also, one can launch and test each component in an application separately. We usually had an /integration folder with a launchable application for each component.

I imagine this can be reduced down for choo to something like this:

exports.component = function (options) {
  const comp = choo();
  // ...
  return comp;
};

The question is, how would I "mount" components together? I would like to be able to somehow achieve this:

const other_thingy = require('other-thingy');

exports.component = function (options) {
  const other = other_thingy.component();

  return choo({
    thingy: other // allow ${component.thingy} to be used in the component html
  });
};

With a technique like this, I would prefer keeping choo's current flexibility to let a developer choose whether they want namespaces in their models or not. Some small components might not need namespaces, while larger ones can still have the separation.

What do you think?

jonaskello commented 8 years ago

@mantoni I'm not sure I follow, but wouldn't requiring one thingy within another thingy couple them together? I think it would be nicer if thingys/components/modules/features/parts (need to find a good name for a stand-alone application part) only were coupled by actions, so that one thingy only would expect some actions to be passed to it, and produce some other actions. So the only way for a thingy to see the outside world would be through actions.

mantoni commented 8 years ago

Yes, good points. Let me elaborate.

The example I made shows a simple composition where thingy is a building block that is used by the outer component. Yes, this means coupling them, like a motor that you put in a car. To stay in the picture, loose coupling would mean adding seats to the car, which are other components that are completely unrelated to the motor. However, the frame of the car puts them all together and has to know about them naturally.

Does that answer your question?

ajoslin commented 8 years ago

I've encountered the same problem and have had the same doubts about Choo's state management at scale.

Once you have hundreds of components and hundreds of models, it will just not work to have everything global and have string namespaces for everything.

All I can say is Mercury, despite its flaws, has the best model for state management I've seen in JS. In my opinion, the best three things about its state story are:

  1. Extremely strict rules for how you mutate state
  2. Strict separation and modularization of state atoms (each Mercury component is basically an app)
  3. Very powerful way to react to and watch state changes via Observ API

Mercury has a lot of flaws in terms of understandability, but we should learn from its good parts (and Elm/Om).

I'd caution against copying Redux too much... It falls apart at scale in similar ways to Choo's current state system.

Generally, I agree with the idea behind @mantoni's comments -- one of the biggest problems with the current system is that "everything is global." If we moved to smaller apps that can require() and compose each other, we can approach a much better system.

I will try to elaborate with more specifics later. Let's keep this discussion going.

toddself commented 8 years ago

It would be possible for the result of calling choo to return a unique object with its own closed over mode store and then to provide communication between these unique instances with a pull-stream (or other type) mechanism perhaps provided by a .use type plugin

This would allow us to maintain isolated components and the ability to define a well-known interface for communication between them.

On Sep 10, 2016, at 11:55, Andrew Joslin notifications@github.com wrote:

I've encountered the same problem and have had the same doubts about Choo's state management at scale.

Once you have hundreds of components and hundreds of models, it will just not work to have string namespaces for everything.

All I can say is Mercury, despite its flaws, has the best model for state management I've seen in JS, with its strict rules for mutations, strict separation and modularization, and easy reaction to changes via Observ. And a lot of its ideas are copied from Om/Elm.

I agree with the idea behind @mantoni's comments -- one of the biggest problems with the current system is that "everything is global." If we moved to smaller apps that can require() and compose each other, we might approach a much better system.

I will try to elaborate on more specifics later... Let's keep this discussion going.

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub, or mute the thread.

ahdinosaur commented 8 years ago

All I can say is Mercury, despite its flaws, has the best model for state management I've seen in JS.

:+1: to Mercury as an inspiration. i'd say though, i'm happy Redux popularized a global state atom over many isolated component states, because with a single state atom it is much easier to reason about time travel, logging, etc. maybe we can have the best of both worlds.

If we moved to smaller apps that can require() and compose each other, we can approach a much better system.

yes!

with inu (and the more opinionated sweetener inux) i'm trying to find a balance between Elm, where modules are more or less manually required into another, and something like choo where things are more automatic. in inux i've been experimenting with optionally namespaced state like redux.combineReducers (similar to choo), but namespaced actions using unique action type strings (so to fire an action from a namespace you require in the unique action type string or a corresponding action creator).

for a wildly different approach that i'm very interested in: dominictarr/depject, used to create the modular user interface patchbay. here, each "module" specifies what "plugs" it exposes and creates any new "sockets" (by also specifying how corresponding plugs will be combined into a socket). this leads to module graphs like this graph of patchbay modules.

thanks for opening this discussion @yoshuawuyts, keen to hear more about the experience of people building apps in the real-world, curious about the trade-offs between being manual and explicit (for example, userland namespacing by wrapping a reducer with a single function) or automatic and sugary.

myobie commented 8 years ago

The current isolation namespaces provide is inadequate, but IMO it's not a problem of needing access to data through a more sophisticated query mechanism. I agree with some of the other commenters that the problem is being able to offer a well defined interface between "models". Part of the problem is that "models" are not "things" that can be referred to (since there really is only one model).

"namespaces" are really "subsets of the state", although not for dispatch. My current solution is to strictly enforce the "subset" idea by wrapping send for effects and subscriptions to prepend the current namespace. This means I can have a load effect in each namespace and can simply send('load') in a subscription in the same namespace. Here is an example function to fully namespace a model which is similar to something I'm using right now.

The only thing this means is that every now and then I need to "cross send" between namespaces like send('emails:showEmailsForSelectedFolder', selectedFolder). For right now I'm fine with that, but I'd love to be able to pass something to my folders namespace that gives it access to emails so I can test these two parts of my code in isolation by passing in a dummy emails at test time. (Also, since sends can be chained together, I don't see a need to be able to modify state in two namespaces from the same reducer as I can just do many sends if required.)

As far as views are concerned, I like that views are handed the total state from the router and they just go down from there. I don't see a need to complicate that; I can choose to otherView(state.folders, prev.folders, send) if I want to. I don't believe the state's layout has to match the UI's layout.

How can we isolate stores/apps/models better without complicating things? Or are we trying to solve different problems?

YerkoPalma commented 8 years ago

Before sharing my opinion, I must say that before choo I used vue a lot, so I'm quite influenced by the that framework.

I think that there are two good points on what have been said, 1) Add more strict rules and 2) isolate choo components as a bunch of micro-apps. In my understandindg, this would allow to write apps like this

const choo = require('choo')
const html = require('choo/html')
const app = choo()

const loginComponent = choo.extend({
  model: require('./models/login'),
  view: require('./views/login')
})

const dashboardComponent = choo.extend({
  model: require('./models/dashboard'),
  view: require('./views/dashboard')
})

app.router((route) => [
  route('/', loginComponent ),
  route('/dashboard', dashboardComponent ),
])

const tree = app.start()
document.body.appendChild(tree)

The benefit of the before code is that every component act as an individual app (with the app constant being the root component) with it's own state, model and view, now is necesary to define a way to communicate between components, share some parts of their states. In vue, this is accomplished with vuex, each component has it's own data and props, but the state is global. This mean to add a way to define local component data/state and a global state to share across components.

I hope I made myself clear.

mcollina commented 8 years ago

Let me add my 2 cents, thanks @yoshuawuyts for involving me in the conversation.

The problems with choo are:

  1. there are no models nor controllers in the "MVC" terms. There is a global state, global list of reducers and a global list of effects.
  2. the way to call the reducers or the effects is the same, and calling send it multiple times within the same flow results in unknown effects.
  3. lacks of "components"/module/reuse: I cannot tie a view to a specific part of the state.
  4. missing of a global choo.send.

IMHO we should get rid of the concept of models. There is state, and a user can add reducers or effects on a choo instance. This would enable people to experiment more on their own way of doing things.

We should provide a shim layer with the current API.

I'm not sure if it is possible, but it might be amazing if we could run two instances of choo together for two different part of the page.

Regarding ORM vs graphs: I do not think it's Choo responsibility. The state holds what is displayed. If there is an ORM/graph behind, the user should feed that into the state.

Another interesting data model can be modelling it via Event Sourcing (http://martinfowler.com/eaaDev/EventSourcing.html).

jonaskello commented 8 years ago

I think event sourcing is an interesting concept which is what I tried to describe in my previous comment. Also the components/modules/thingys could work together without requiring each other or having to declare interfaces if the collaboration between them is modeled as event collaboration (http://martinfowler.com/eaaDev/EventCollaboration.html). The interface would then just be the actions and the coupling would be very low.

Specifically this part of the event collaboration link is relevant when it comes to state management:

A consequence of this is that the responsibility of managing state shifts. In request collaboration you strive to ensure that every piece of data has one home, and you look it up from that home if you want it. This home is responsible for the structure of data, how long it's stored, how to access it. In the event collaboration scenario the source of new data is welcome to forget the data the second it's passed to its Message Endpoint.

chrisber commented 8 years ago

@jonaskello https://medium.com/@fagnerbrack/do-you-think-its-a-good-thing-abstracting-useful-information-in-libraries-instead-of-standing-on-9a5b327f644e#.vu2f4ells

nichoth commented 8 years ago

Agreeing with @mcollina, you don't want to be thinking about ORM, relationships, or graphs. That should be delegated to your database or other library, which then would provide an API for you to hook into. You should not have "models" in the traditional/ORM sense, that's what the database is for. What you want is a big observable black box of state that is faceted so that you can hook into it at various levels.

Using a contrived example we can think of an app with books and movies. I would like to see an API using functional operators (map, filter, etc), but I haven't had time to try it yet. 

var booksEvents = db.books.getChangesStream()
var movieEvents = db.movies.getChangesStream()

var booksStore = BooksStore(db.booksInitialState())
var moviesStore = MoviesStore(db.moviesInitialState())

var sendDbEvents = booksStore({
    create: (state, event) => newState
    delete: (state, event) => newState
})
// we have closed around the internal state for "books", and we are
// able to use the same event names without collisions.
var sendOtherEvents = booksStore({
    create: (state, event) => newState
    delete: (state, event) => newState
})

booksEvents
    .map(toOurEventNames())
    .subscribe((ev) => sendDbEvents(ev))

// each `store.state` is an instance of observ-struct because it is
// the best observable API. We can observe state at any level of granularity.
// But it would be good to not expose the `.set` method, so that only the
// reducers are able to update state.
var appState = observStruct({
    books: booksStore.state,
    movies: moviesStore.state
})

// The views only care about when the state has updated, but they are
// also observable, because they need to emit actions. 
var bookEvents = BooksView(appState.books)
booksEvents.on('submit', (ev) => db.newBook(ev))

// We have created a feedback loop of `view.pipe(database).pipe(view)`.
// What I want to know is how granular the feedback loop structure can be,
// and what can we say about it if we restrict the operations involved? This
// is what people call "fractal", where the application structure is the same
// as the structure of sub-components. The database is basically the same also —
// a change log of operations over time.

Looking at the original comment, it seems like maintaining view state is the issue really, not persisted state. Maybe it would be good to look at it from the perspective of what is not to like about the mercury pattern, using a tree of observ-struct instances. It is lightweight in terms of lines of code, but requires more developer time (I think). React is the other end of the spectrum, lots of code, easy to use. Dealing with persisted view state across re-renders is actually one of the most painful parts of UI right now, in my opinion.

mcollina commented 8 years ago

Let me go on the "mad science" path.

Let's consider:

  1. a view layer that emits events when action happens
  2. an immutable state
  3. a set of handlers that react to events, and possibly update the state

I consider an event as a js object, and the handlers can subscribe for changes using pattern matching, see https://github.com/mcollina/bloomrun for that (the size of this might be an issue, but we can work on that, there is plenty of room for improvements). We can shortcut the pattern with https://github.com/mcollina/tinysonic. The state is just a view of the sequence of events that happened, and it is easy to hydrate and de-hydrate.

I'm throwing in @mcdonnelldean because he worked on https://github.com/mcdonnelldean/nanite

YerkoPalma commented 8 years ago

@yoshuawuyts Is there any "conclusion" about this discussion? Is the state management of choo going to change in the mean time?

brechtcs commented 8 years ago

So here's the two things I like most about choo:

So basically I would happily adapt to any state management changes that respect these advantages.

One caveat I'd like to add though, is to be careful using large-scale apps as a guideline. Apps at scale will by definition be complex, often in very different ways, and if you start to import all that complexity in a framework like choo, what you end up with is something like Angular 2. So while there might be improvements possible in choo proper, I think a large part of the solution will always have to be a chapter (or series of chapters) on 'choo at scale' in choo-handbook.

yoshuawuyts commented 8 years ago

Heya, thanks for the replies everyone! - Even though I've been a bit silent, I've read it all, and y'all put out some great stuff. What I'm thinking right now is:

How does that sound?

ps. Also docs everywhere; we're using selectors on a real world project rn and it seems to work alright enough - just a few tweaks could probably make it better C:

bengourley commented 7 years ago

Just thought I'd chime in on this. Nothing significant to add solution-wise, but want to voice some friction I'm experiencing with namespaced stores. This could either be resolved with a rejig in my app, or add weight to the argument that namespaced stores aren't helpful. Let's see!

So in my app the logged in user's auth credentials (a jwt) live in the top level app model with no namespace. This is all well and good for views, because they recieve the entire state tree (for all those occasions when I want to render the jwt to the screen 😁).

However, when I want to trigger an effect on a namespaced model (e.g. files:load) which results in an xhr to an API server, I don't have access to the credentials in the app-level state. For my app I'm having to just pull this in from somewhere else. This feels like I'm busting out of the framework to achieve something which I think should be doable from within.

Any ideas on how I can rework this, or is this exactly the type of reason why namespaces have come in to question?

Random unthought through proposition: what if effects had access to the entire state tree?

aknuds1 commented 7 years ago

@bengourley I also experienced some friction with namespaced models not having access outside of the namespace. I was able to work around it, but I can see it becoming a problem. My impression so far is that namespacing of models is nice, in the way that modularizing your app is, but not so sure that restricting access between them is going to work.

myobie commented 7 years ago

@bengourley and @aknuds1 I have dealt with the sharing problem by copying the data across namespaces during update. For the credentials problem I've made an effect auth:updateCreds and that calls a reducer in the auth namespace but also one in the files namespace also.

I don't have a way to do this in a declarative way (this property of this state should always match this other property of this state) and I have to manually make sure that I never update the cloned state in any way other than through the auth effect. It's an idea.

josephluck commented 7 years ago

@myobie, @bengourley , @aknuds1 - I've encountered the same problem. I've worked around this by having a global state singleton that is kept in sync with choo's state using the onStateChange hook. This singleton can be imported in to the models that require state from other namespaces, but I've also wrapped effects to include global state under a $root key using the wrapEffects hook. Not ideal, but it works.

vuex handles effects nicely IMO. Effects have access to the entire global state, whereas mutations only have access to state from the namespace.

I draw a lot of similarities with vuex simply because it's so easy to use, but here's my humble two cents regarding state in choo. When effects and mutations in vuex are called, they return the result of the effect / mutation. This makes it real easy to use promises instead of callbacks if that's your jam, or immediately access the updated state of a reducer during an effect. I haven't had any trouble with views and state, but being able to access state outside of choo's views and models would be really handy for use cases such as authentication.

myobie commented 7 years ago

@josephluck I really like the idea that reducers are only for updating local state while effects have access to the global state. @yoshuawuyts have you considered this?

About using global state outside of choo: I have found subscriptions to be more than good enough. They always force me to have good boundaries and so far I have always been able to do what I want with a simple EventEmitter as a message bug into a subscription.

yoshuawuyts commented 7 years ago

Yeah I've thought about this - it might make sense, and if prior art in Vue shows this is a good idea then it might definitely be something we could adopt

On Thu, 29 Dec 2016, 15:43 Nathan, notifications@github.com wrote:

@josephluck https://github.com/josephluck I really like the idea that reducers are only for updating local state while effects have access to the global state. @yoshuawuyts https://github.com/yoshuawuyts have you considered this?

About using global state outside of choo: I have found subscriptions to be more than good enough. They always force me to have good boundaries and so far I have always been able to do what I want with a simple EventEmitter as a message bug into a subscription.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/yoshuawuyts/choo/issues/252#issuecomment-269639629, or mute the thread https://github.com/notifications/unsubscribe-auth/ACWlepxFUHPxx5faMDrAtjZ2vT3DKh0Eks5rM8cMgaJpZM4J5FAq .

bengourley commented 7 years ago

So the proposition, if I understand correctly:

Sounds good to me ✅

forresto commented 7 years ago

Great rabbithole of a thread here, thanks for the links.

I was looking for a simple way to cache some expensive state, and found reselect to be functional / legible enough to work quite easily with choo. Here's my PR pulling some expensive stuff to cached selectors: https://github.com/softfab/tshirt/pull/2

yoshuawuyts commented 7 years ago

We figured it out - we're now using an event emitter and all state woes are solved. Check out master for the latest version; run npm i choo@next for the next version of choo. Thanks for the discussion everyone!

Closing because choo@5 will introduce a new API. See #425 for the merged PR. Thanks!

adminy commented 3 years ago

I think a way to import emit and state would have been nice addition. Even emit so it doesn't get passed around from function to function would have been an awesome improvement.