how do you time travel backwards once the server db has changed?

[faceyspacey]

Hey Matt, I asked this question as a private message to you in the Meteor Forums too--I just wanted to post it here in case you haven't been to the Meteor forums in a while.

[mattkrick]

that's an interesting question, but no good answer (at least not one that could be used in production). Basically, you'd have to snapshot your database & store a queue of mutations applied. Current state would equal your snapshotted DB + each action from the queue. A query would have to apply each mutation before it could execute. If you read the redux-optimist source code you'll get an idea.

Alternatively, for every mutation you could create a counter mutation & store that in a queue. That means before every write you'd have to do a read & then create a mutation that achieves the current state. That's a headache.

The correct way to do this (in NoSQL) is to use arrays instead of primatives. For example, instead of having the last time the user logged in, you'd keep an array of every log in the user ever had. In SQL, this would be a separate table for each entity (eg DateTime, Price, Quantity). At $20/Gb, there's no good reason to do this for every field.

[faceyspacey]

You're familiar with the totally awesome Tonicdev right? http://tonicdev.com

Read their first blog article:

http://blog.tonicdev.com/2015/09/10/time-traveling-in-node.js-notebooks.html

How they restore state (by snapshotting all of "userspace" hosting your code) is based on this: http://criu.org/Main_Page

Basically, it seems that in all practical use cases Time Traveling is mainly hype--the second one of your actions gets state from the server side db, it's broken. It also makes me wonder about hot reload since the code reloaded could trigger not just db changes, but also some sort of client side state not properly consumed by Redux's state engine: for example increment a global counter when it shouldn't be incremented or "erase" opaque values stored within a closure since new functions making use of returned functions and their enclosed private variables will be different than already loaded non-changed code that has access to the original returned functions and their enclosed private variables, which could have be changed, e.g. an incremented private integer.

As far as the storage price problem--I'm not sure i get the importance of that one since time traveling is primarily important during development, where you usually have extremely small databases of sample/test data. What's the use case for time traveling in production? Maybe I'm missing something there. ..Basically, it seems we can do a shitload of snapshotting only during dev to enable complete/universal isomorphic time traveling.

What I am aiming to do is provide "universal time traveling" for Mongo within Meteor, implementing something like what you mentioned here:

Current state would equal your snapshotted DB + each action from the queue. A query would have to apply each mutation before it could execute.

..as creating counter mutations for all possible Mongo modifiers--like you said--would be a major headache. Likely possible though. I still got a lot of research to do before I can determine if snapshotting all of Mongo (often) is out of the question. It does seem though that the snapshot only has to happen for each "commit" rather than for every single change to the db, which was my concern at first. You just repeat all mutations since the last snapshot, e.g. since your last page refresh or intentional commit, which seems to be what you're indicating Redux is doing.

Let me know what you think about the viability of this for Mongo (and Minimongo--will have to do both, since often people use just client-side-only collections). Oh, and ReactiveDict as well. Right now I'm looking at this as a not-too-unreasonable surmountable task, and one that could provide a lot of value to the Meteor community, and ultimately anyone using Redux with Mongo.

ps. I wonder if CRIU would be the better approach, as that can capture all server side state, not just Mongo. If that did end up being a great option--then the question is how to do that for the client runtime as well (which is where this might go overboard, since you'd likely have to have a custom build of Chrome to do it, but maybe a Chrome extension gives enough low level access I'm not currently aware of). This to me seems like the ultimate solution--because then you're not stuck in the current Redux dilemma where only tightly managed "state" can be time traveled. It really seems that while this remains a "leaky abstraction" it's unusable--it basically needs to be perfect for every possible scenario.

[faceyspacey]

also, a quick related question: hot module replacement with webpack obviously suffers the same problems mentioned above about functions (and their enclosed private variables) already returned from previous code? i.e. webpack can replace a chunk of code, but it has no way of knowing or modifying previous usages of the code already in the runtime.

Great for purely functional aspects of your code, but crippled elsewhere. Is that the idea?

[mattkrick]

The oplog gives you everything you need to recreate state. Just open a new database & apply each row of the oplog. But personally, I don't see the value add. it's really easy to test if a write succeeded: it's persisted on disk & has no side effects. I only care about things that affect the view layer. adding a doc to the client cache triggers an animation or reorder. adding a doc to a database doesn't.

[faceyspacey]

In Meteor, the client cache (minimongo) and the server database are one and the same. The view layer is tightly intertwined with what's in the actual database. I suspect that to also become the case with GraphQL and Relay as it--and support for subscriptions--takes off.

For example, if you are trying to test the update of a document and everything happens almost correctly (not just animations, but perhaps emails sent out, linked documents get updated, etc), you can't effectively use time traveling to debug this situation when after you rewind the document already has properties previously assigned. For example, an email won't go out indicating you changed your email address if it was the same the second time around.

I'm unsure how we're not seeing eye to eye on this point. Perhaps you're just saying that view layer stuff is "good enough" and I would like the whole enchilada (server side-effects, perhaps client side-effects as well unrelated to functional rendering of a component tree). I'd say likely even the view layer is far from good enough--you're gonna need a rewind of the documents in the db to their initial state for many things to be debuggable the way you want it.

if this is the only example of side effects one might care about:

it's really easy to test if a write succeeded: it's persisted on disk & has no side effects.

that's just the tip of the ice berg.

regarding:

adding a doc to the client cache triggers an animation or reorder. adding a doc to a database doesn't.

my point here is if you rewind to an earlier action that populates the client side cache with data from the database, well then it will have data it shouldn't have yet. Then when it goes to insert a document, then you have 2 where you expected one.

It seems Redux is only good for basic instrumenting of your components, i.e. functionally where you wanna make sure the inputs and outputs of components operate correctly, but not for more "end to end"-like run-throughs and tests/debugging. Not for the more common "integration debugging" developers do probably more than anything. It's just going to be unclear what you're looking at; when and where what came from the database; why is it on the screen when it shouldn't have this data yet. Take as an example when a certain view element shouldn't exist unless certain props on a corresponding object exist, and you're debugging the hiding/showing (no animation involved) of this view component--you're going to be wondering why it's already showing when you rewind to before you perform the necessary action to make it show; you're going to have a hard time testing whether the view component code that hides/shows the element is working.

[mattkrick]

You're breaking the first rule of redux. Actions are pure, synchronous functions. You can timetravel all day & it'll never hit the server, but it will adjust your client cache. Look at this repo, if you time travel a _SUCCESS then the doc reverts back to meta.sync = false, meaning it's optimistically added. if you time travel the request, then the doc is gone all together. I have no need or desire to replay anything on the server.

[faceyspacey]

this file: https://github.com/mattkrick/meatier/blob/aea37a5aa16df43b5a283a8128e30634549aeb8a/src/universal/redux/middleware/optimisticMiddleware.js

?

[faceyspacey]

basically you gotta store the result of, say, collection.insert both in Redux state and in the mongo collection, correct?

[faceyspacey]

So I assume the way you would utilize meteor subscriptions with Redux would be to observe a collection you have a subscription for, and then when new documents are added, you then add them to Redux, correct?

If that's correct, I'm wondering what happens when, through time traveling, you repeat the same action over and over again, inserting more documents into the db, then Redux will keep getting them all (because of the observer connected to Redux).

Unless you're not utilizing Meteor's realtime functionality and are only getting documents on page load, there has to be something that will notify Redux when you have new documents. How are duplicates prevented? Additional code to manually check if Redux already has it in either: A) the action that will perform the insertion and prior to storage in Redux state or B) in the action triggered by observer added which then prevent re-adding it to Redux state??

[mattkrick]

minimongo runs the show in meteor, that's not easily avoidable or extendible. you're better off building your own meteor functionality than trying to hack extra functionality into it.

[faceyspacey]

what happens when you're rethinkdb observer discovers new docs? don't you have an action that's triggered that ultimately stores the doc as state within redux?

[mattkrick]

i have a function that creates an action. The action isn't coupled to the database, which is why I can mock any previous database state locally but hitting undo. https://github.com/mattkrick/meatier/blob/master/src/universal/redux/ducks/lanes.js#L106-L115

MInimongo does this too, but it mutates the client cache, which is why it can't tell you what changed after the fact. That's up to you to implement through collection.observe, which is really, really clunky to use inside a reactive tracker computation.

[faceyspacey]

I got it 100% now. Thanks. You just kinda deal with things separately, but perform additional checks to see if and what data you already have.

...that seems to be the big debate: the functional stuff is clunky interface-wise, but tracker will likely be clunky implementation wise. Interface-wise, ideally, you don't need to perform those checks of whether you have old_val or new_val in application code and the system isomorphically knows exactly where you're at.

I think we can abstract and automate this all for a perfect redux + minimongo/tracker package.

The predictability of the purely functional approach is great, but I think there's still room for implicit abstractions. Facebook's building so low level all explicit and whatnot, purposely because the expect other developers to build abstractions on top of it. Here's my post from last week all about that: https://medium.com/@faceyspacey/tracker-vs-redux-implicit-vs-explicit-df847abcc230#.jbsvn3z68

We don't gotta code in the exact Facebook style because Sebastian Markbage (lead Facebook React developer I'm sure you know of) says they are avoiding implicit abstractions themselves. The problem is some abstractions are leaky--but I'll take a rock solid implicit abstraction any day if it means I can write and maintain half as many lines of code. I know you're all of over with Meteor, but I think there's some real goodness in Tracker that Redux's only response for is a shitload of boilerplate. You're trading readability for predicability, and my hunch is our decisions can be more fine grained: functional purity where needed, and automated implicit abstractions where reliable. It might be cumbersome work, but I think we could build a solid abstraction that achieves just that by combining Redux + Minimongo/tracker.

[mattkrick]

Yeah...Tracker is trash. For small apps, it's novel because things magically update. For large apps, you're wondering what line, what function, what file, what package made that change. You're wondering if an old observer is still open somewhere. You're crawling your entire function call stack because if an observer is somewhere inside an autorun, it'll quietly stop your observe & create a new one. You're calling Meteor.defer to hack race conditions in the flush cycle. You're wonder what autorun got called twice and screwed everything up (idempotence would be nice...). I'll take predictability & boilerplate over magic any day.

Here's the right way to do it, coming soon https://github.com/zenparsing/es-observable

[faceyspacey]

that's throwing the baby out with the bathwater. There are many things that can be done to improve that situation, and let you still code implicitly with way less code than Redux. I'm also working on a few tracker abstractions that resolve all the tracing you're talking about.

One of the biggest issues is people simply don't use the fields option in their finders, and then their helpers end up being re-run a million times more than they need to. If you only need one field in your helper, than you should have {fields: {foo: 1}}. Like, nobody does that. That said, it's a nuisance--almost as bad as all the props you gotta pass from component to component in React ;). I'm working on a "property-level cursors" which through transpilation automatically fills in your fields. That's just one of many ways to box Tracker in. I got a whole lot to say about optimizing tracker. Everyone's jumping ship like yourself to the functional world. That's great. I'm on board too. But, there's a reason Lisp didn't take over decades ago. There's things you can do with abstractions that are more challenging in functional code. OOP is flatter and therefore more readable. In general, my biggest gripe is I don't want to be forced to down the "explicitness" Kool-aid. I want to be able to have the best in-class most implicit abstraction possible where I feel I can rely on it. You already use "magic" in plenty of places where you just accept it--you'd accept it in the View layer over all the functional boilerplate if it was rock solid. Point blank period. ..The observable stuff is great, but automatically making your function reactive where you use a reactive data source is hard to beat. Tracker just needs to be doubled-down on and optimized.

[mattkrick]

Agree to disagree I suppose. With meatier, i build faster, scale higher, and debug less. At the end of the day, that's all that matters.