Expanding the falcor cache into plain JS objects

[ekosz]

Hey all. I'm currently creating a new library to help connect falcor and redux. What I am trying to accomplish is allowing developers to pull data from the single redux store synchronously like normal, but having some of that data get populated by falcor.

You can see how I'm doing that here. My current approach is a little naive. As results come back from falcor I just merge them into a master JS object that redux components can then pull from. This unfortunately loses much of the power that falcor provides like its expiration and LRU caching.

What I would like to do instead would be something like:

falcor.onChange(() => store.dispatch(updateFalcor(falcor.expandCacheToJS())))

That has the advantage of always keeping the redux store update to date with the falcor cache, but it also means we're doubling (or more) the memory footprint as we're storing the cache data twice.

Another option I can think of is using ES6 getters to create an object that hides the falcor implimentaion. But that would still require a way of getting the JS values out of the cache in a synchronous manner.

Do does anyone have any suggestions? I think falcor + redux is a powerful combination once the data sharing aspects are solved.

[falcor-build]

We are actually exploring a very similar problem at the moment. Maybe we could do a hang out or something. It would be interesting to share some ideas.

JH

On Oct 13, 2015, at 11:09 AM, Eric Koslow notifications@github.com wrote:

Hey all. I'm currently creating a new library https://github.com/ekosz/redux-falcor to help connect falcor and redux https://github.com/rackt/redux. What I am trying to accomplish is allow developers to pull data from the single redux store synchronously like normal, but having some of that data instead get populated by falcor.

You can see how I'm doing that here https://github.com/ekosz/redux-falcor/blob/7bbc617df9fd1dd256da1ad8f1e8391c1cc9d4b6/src/reducer.js. My current approach is a little naive. As results come back from falcor I just merge them into a master JS object that redux components can then pull data from. This unfortunately loses much of the power that falcor provides like expiration and its LRU caching.

What I would like to do instead would be something like:

falcor.onChange(() => store.dispatch(updateFalcor(falcor.expandCacheToJS())))

That has the advantage of always keeping the redux store update to date with the falcor cache, but it also means we're doubling (or more) the memory footprint as we're storing the cache data twice.

Another option I can think of is using ES6 getters https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Functions/get to create an object that hides the falcor implimentaion. But that would still require a way of getting the JS values out of the cache in a synchronous manner.

Do does anyone have any suggestions? I think falcor + redux is a powerful combination once the data sharing aspects are solved.

— Reply to this email directly or view it on GitHub https://github.com/Netflix/falcor/issues/572.

You received this message because you are subscribed to the Google Groups "Falcor" group. To unsubscribe from this group and stop receiving emails from it, send an email to falcor+unsubscribe@netflix.com. To post to this group, send email to falcor@netflix.com. To view this discussion on the web visit https://groups.google.com/a/netflix.com/d/msgid/falcor/Netflix/falcor/issues/572%40github.com https://groups.google.com/a/netflix.com/d/msgid/falcor/Netflix/falcor/issues/572%40github.com?utm_medium=email&utm_source=footer .

[tivac]

@jhusain You gotta start responding from the right GH account XD

@ekosz I was just looking at redux-falcor this morning and wondering about this exact problem when i saw the merging code. Good timing!

I would love to see some traction on this as well, it would enable some really valuable interop for falcor IMO.

[ekosz]

@falcor-build / @jhusain Yeah a hangout would be great. But you probably have smarter people than me working on this haha.

[trxcllnt]

@ekosz I was just thinking about this problem last Thursday. One possibility is to stick the bound models into redux's JSON store, then call getVersion() at each level in shouldComponentUpdate.

@jhusain possible ideas I've had so far are React-immutable-helpers-compatible JSON output, or a way of telling the Model, "reuse the same Object for all JSON output (and update the version on parent nodes if required)."

[bfitch]

We're trying to figure out a similar solution for cerebral. Discussed here: https://github.com/cerebral/cerebral/issues/120

[ekosz]

Hello. I wanted to ping this thread again, show what I'm working on, and ask for a bit of help.

I've come to the point where I've realized redux-falcor isn't very useful without being a direct representaion of the falcor cache. I've created a new project called falcor-expand-cache, which is where I'm currently trying to create a solution for expanding out the falcor cache into plain JS objects. That way whenever falcor changes its own cache, I can replace the redux version from scratch rather than always trying to patch the differences.

The problem I've hit with my very simple solution is that it blows its call stack on even moderate sized caches.

RangeError: Maximum call stack size exceeded

I've tried replacing with Array.reduce with bluebird's Promise.reduce to both parrelize the process and try and get around the call stack issue, but I instead get segfaults and OOM errors.

If anyone is working on something similar or has any suggestions I would be very greatful. Thanks!

edit - Ignore the call stack issue, it has to do with circular references. I need to figure out a solution there.

[ekosz]

Welp, I ended up rewriting my solution to be getter based instead of trying to expand out the entire cache. My solution now creates a "cache-like" object whose properties are all lazy. Only when the property is call it will transform the requests path into a JS object. These properties also follow refs properly.

The lastest code is here. I'm going experiment hooking this into redux-falcor and see how it fares.

[trxcllnt]

@ekosz I've started a project that takes an alternate approach. It doesn't need to expand the cache into JSON objects on each get, because it's built around an Observable of Models that emits each time the cache is changed. You can check it out here: https://github.com/trxcllnt/reaxtor

[ekosz]

@trxcllnt That looks great! Unfortunately I have yet to learn RxJS, but it's on my todo list. Ultimately I think I'm fighting against the tide here. It seems (at least to me) that redux and falcor are just a tad too different to really connect on a meaningful level. Which is a real pain as I've based my entire stack on those two technologies haha.

[ThePrimeagen]

@jhusain feel free to chime in here as you have spent a great deal of time thinking about this exactly...

[bfitch]

@ekosz I don't have much to add other than "I feel your pain". I've been struggling to integrate falcor with baobab, which is an immutable JS tree that uses cursors and getters/setters for managing client state.

My idea was that I could 'embed' the falcor cache within baobab:

let model = new falcor.Model(onChange: onChangeCallback);
let tree = new Baobab({falcor: model});

and then whenever falcor's cache changes (get, set, call on some path), I could simply brute force set the new state:

function onChangeCallback {
  tree.set('falcor', model.getCache)
}

Baobab has React bindings so that whenever one of it's cursors changes, it will trigger a re-render in any component that has "subscribed" to that cursor. As you can probably guess this strategy didn't work well because any change would require a re-render of the entire component tree since you can't leverage shouldComponentUpdate, specifically comparing objects by reference (previousState === newState).

Other than the brute force approach, I've been puzzling how to determine what paths have changed in falcor so I can "sync" that to the baobab tree intelligently. I haven't gotten far and feel like I've reached the same conclusion as you: I'm simply "doing it wrong".

Anyway, I am still really excited about falcor, but am at a dead end as to how it should fit into the wider client side state management story. How should it integrate with other "client only" state that you don't want to send to the server? Unfortunately, it can't be our "one model" yet cause we need to deal with so much more than just data sent over the wire. The pain is worse with "single state store" architectures like Redux or cerebral that want to centralize all state management into one object/atom.

Again, huge thanks to the Falcor team for this awesome project and thanks for any pointers you can provide. :)

[greim]

Why not have a synchronous getter on the model that reads from the cache, but as a side effect calls to the router and makes LRU updates? The idea being that instead of fetching on init, putting the resolved value in a state object, then rendering off the state object, you'd just render off the graph directly using that synchronous method, building the cache up naturally as you go.

This is what my lib ng-falcor does anyway. But it has to cheat by reading internally from model._root.cache. In my Angular templates I have have an ngf object which is called from templates, and the data just drops into place:

<div ng-repeat="n in range(0, 20)">
  <div ng-if="ngf.has('todos', n)">
    <div>Name: {{ ngf('todos', n, 'name') }}</div>
    <div>Completed: {{ ngf('todos', n, 'completed') }}</div>
  </div>
</div>

[bfitch]

Hey @greim thanks for pointing me to your lib. Just to make sure I understand, are you maintaining your own (LRU) cache on top of falcor? Or is the trick that you can bind templates directly with your synchronous (model._root.cache) getter?

[greim]

@bfitch - It doesn't do anything but pull out whatever's in the cache. LRU updating and router fetching is still managed as normal by the Falcor model, since internally it calls model.getValue() as a side effect in order to activate LRU, fetching, etc, but the ModelResponse is discarded and it returns whatever's in the cache at the moment.

https://github.com/greim/ng-falcor/blob/eddde9ec861cc2ff7e637c3ae0811273abb62549/src/index.js#L35

Basically yeah, with this approach your templates bind directly to the graph, as it were. The initial pass render will of course be empty (unless you've bootstrapped the model cache) but that's how all frameworks work nowadays anyway.

[edited for clarity]

[bfitch]

@greim Ah I see. Thanks for the tip!

[bfitch]

I'm really silly. facepalm I can put whatever client only data I want in the falcor cache. My real problem is synchronously accessing that model data whenever it changes so I can re-render the component tree.

[jhusain]

First of all I want to acknowledge that there are real challenges here. Merging model information into a single store presents several interesting problems to be solved.

This is something we are actively working on. I'm in the process of exploring different strategies with different tradeoffs. We are closing in on some solutions and we will share them as soon as we've fully thought them through.

[greim]

@bfitch - In angular we trigger a digest update instead of re-rendering the component tree, but it's analogous. I've been doing it with a generic onChange on the model like so. Something similar should be super easy in React by rerendering the root component in response to that event.

[greim]

@jhusain - I'm curious to see what you come up with. It seems to me that a model could simply be a store (rather than merging its data into a separate store) if only it had a synchronous getter, no?

[bfitch]

@greim yeah, could definitely do that, but it would be quite a performance hit re-render the whole tree on any change. Baobab/Cerebral react apps leverage cursors and immutable data to short circuit re-renders for most components (using shouldComponentUpdate).

I just don't know if I could forge ahead after getting that out of the box with immutable single state stores.

[greim]

Makes sense. That's a nice guarantee to have. I'm trying to wrap my brain around how cursors would work with a graph structure.

[wmertens]

(warning, armchair programming follows)

Another approach is to consider falcor as one source of props, and redux as another orthogonal one. In react, all props are merged so an update from either rerenders as needed. Redux would be storing metadata and wrangling mutations, falcor handles getting and caching server state.

This is similar to displaying an image, at most redux cares about its url but the actual loading and display is done by the browser without involving redux.

To enable server-sourced events (like mutation results) to trigger redux actions, there would need to be a way to subscribe to a piece of the json graph which gets converted into actions as appropriate. This subscription probably needs to be dynamic based on redux state (and thus currently showing components etc).

In other words, use each library for what it does best.

[overture8]

We had a bit of a discussion about this on the Cerebral project here. One of the ideas there was to think of Falcor as an "intelligent ajax library with caching" - this makes total sense in my eyes and made it a lot easier for me to understand how to use Falcor with a complementary library like Cerebral/Redux.

Maybe this is an oversimplification of the problem but it has worked great for me so far.

[bfitch]

I followed @ekosz's lead and have a basic cerebral integration working here: https://github.com/bfitch/cerebral-falcor-module incase anyone is interested.

[trxcllnt]

Here's my solution -- you don't need to expand the cache into one large object if rendering is asynchronous: https://github.com/trxcllnt/reaxtor

[ekosz]

I finally released version 3 of redux-faclor. It now makes the interaction between redux and falcor very smooth. What still needs to happen in some internal performance increases as it currently replaces the entire falcor state tree on every update.

[greim]

if rendering is asynchronous

I'll be curious to see whether the concept of async rendering takes off in the future. In the here and now though React or Angular don't have any concept of async rendering, so that excludes large swaths of the world :/

[jean-morissette]

Any updates on the integration of Falcor with Redux?

[trxcllnt]

@jean-morissette @ekosz's redux-falcor library seems to have really matured; I'd recommend that route if you're looking to use Falcor with React + Redux.

[steveorsomethin]

I'm currently performing issue triage as we get ready to perform a proper release, and closing/tagging as I go.

+1 to Paul's answer. While activity seems to have slowed against redux-falcor, it should still be capable of working with falcor. There are no plans for a netflix owned redux integration project at this time however.