Use Vue instead of React

[mitar]

React is so 2016.

[karth295]

Not sure if you're just trolling, but you should provide more context for your proposal, lol.

[toomim]

If I read our current Readme, I would have the same reaction as Mitar. It implies that Statebus users have to learn React first before using Statebus, which implies that you need to know the React API, which sucks, and would make me think "you should switch to Vue" too.

I'm fixing the readme.

We use React as a virtual dom implementation internally, but encapsulate its API with our own. In the future I plan to swap out React with Preact, virtual-dom, or our own diff-sync. This will improve performance, but barely affect the API.

[toomim]

@mitar, did I get you right?

[mitar]

No, in fact I am trying to suggest to really use Vue internally. From my reading about statebus it seems a much better fit. From my benchmarks, it is only 2x slower than raw dom changes, while React is 5x slower:

• https://github.com/mitar/meteor-blaze-benchmark
• https://github.com/meteor/blaze/issues/131#issuecomment-254079886

It uses a combination of reactivity and smart virtual dom. I think this is the best combination possible. So instead of rendering everything every time to virtual dom and diffing against current state of virtual dom (React), it has full reactivity support so it detects which parts of data changed and rerenders only that part of virtual dom first.

Because of that it is really to map to state bus. State bus would just output state, Vue would make that state into getters and setters automatically, to add reactivity to any interaction with the state, and render it. If state changes, Vue rerenders.

Vue already contains reactivity engine, so you do not have to roll your own. It seems statebus is currently using your own for your reactive functions.

You do not need to do any manual optimizations to get this performance.

It is also smaller than React. 19kb min+gzip runtime (in you do not want to compile templates on the client side). It can be done as templates or directly and everything in between (strings, templates inside <template> tags, etc.). It can be used with separation of concerns (template separate) or without.

It allows nice way to do animations.

[mitar]

In some way, if statebus is new HTTP, programming for new HTTP could just be (Vue extended) HTML. It is not the only way to use Vue, but just an example.

So the example would then maybe be more like:

<div id="messages">
    <div v-for="message in messages">
      {{message.content}
    </div>
  <new-message />
</div>

<script type="text/x-template" id="new-message">
  <div>
    <input v-model="new_message.text" type="text" />
    <button v-on:click="send">Send</button>
  </div>
</script>

<script type="statebus">
Vue.component 'new-message',
  template: '#new-message'
  data: ->
    new_message: fetch('new_message')
  methods:
    send: (event) ->
      chat = fetch('/chat')
      chat.messages or= []

      chat.messages.push
        key: "/message/#{random_string()}"
        content: this.new_message.text

      save(chat)

      this.new_message.text = ''

random_string = ->
  Math.random().toString(36).substring(3)
</script>
<script src="https://stateb.us/client6.js"></script>

BTW, I love the choice of CoffeeScript. :-)

[toomim]

Ok. Yes, Vue is faster and smaller than React, but so is Preact and virtual-dom. We will switch from React when it's time to optimize for performance, and compare all the options.

We're also working on our own diff-sync engine that generalizes virtual doms with collaborative editing and delta-compressed network transfer. This will become core to the Statebus protocol.

These features aren't unique to Vue:

It uses a combination of reactivity and smart virtual dom. I think this is the best combination possible. So instead of rendering everything every time to virtual dom and diffing against current state of virtual dom (React), it has full reactivity support so it detects which parts of data changed and rerenders only that part of virtual dom first.

Every virtual-dom library does that, including React. That's why React components don't re-render unless props or state have changed.

Vue would make that state into getters and setters automatically, to add reactivity to any interaction with the state, and render it.

Vue (like mobx) creates getters and setters with Object.defineProperty. We've prototyped this approach too, but we don't like it because it requires properties to be known in advance. The future of change-detection is ES6 Proxy objects.

It allows nice way to do animations.

That's nice, thanks. Travis has built an animation library too. I haven't investigated them, but I should at some point.

[toomim]

In some way, if statebus is new HTTP, programming for new HTTP could just be (Vue extended) HTML.

I like your line of thought! I do want to merge HTTP and HTML, but the Vue syntax in that example is overly complex way to merge them. There are (1) templates, (2) components, and (3) html tags, and then for specifying algorithms there are (4) vue html directive attributes (e.g. v-for) and (5) scripts.

We can merge (1), (2), and (3) into a single type of thing, which I've been calling "widgets", and the coffeescript syntax lets us embed them cleanly into regular scripts, merging (4) and (5). This is much simpler, yes?

Am I missing some advantages of the vue approach that you like?

FWIW, I have also partially implemented an HTML syntax for statebus code, like this:

<div>
  <my_custom_widget foo="bar">Stuff!</my_custom_widget>
</div>

<script type="widgets">
ui.MY_CUSTOM_WIDGET = (attrs, children) ->
  blah = whatever
  DIV
    style: whatever
    "My foo is #{attrs.foo}"
    children
</script>

Or like this:

<div>
  The current bitcoin price is <run-here>fetch('state://toom.im/bitcoin_price')</run_here>
</div>

...which will show a live updating bitcoin price, since <run-here> will re-run when anything fetched in it changes.

[mitar]

You are right. Proxy would be much better, but sadly we are still not there. :-(

I think that maybe my confusion is coming from the fact that I expected statebus to deal with state, and then I can plug on top of that any rendering library I like. That it will be like state.get() and state.set() and this is all that it does. This whole deal of also dealing with DOM is surprising to me.

Maybe a layered architecture here could be beneficial? Split this into statebus-state and statebus-dom and then people can use just state, or then can use also statebus-infused and endorsed dom?

[mitar]

You can of course wrap Vue into something else so that you do not have to expose everything. Your last example is similar to single file components in Vue.

What I am probably saying is that instead of making one more syntax for HTML templates, use something existing. I would suggest Vue syntax. It is close to Handlebars.

Another thing I am saying is that I prefer separate HTML template instead of mixing representation with code.

[mitar]

One other reason to use existing libraries like that is that you can reuse components available. I like web components, but Vue is very similar and seems more ready.

[mitar]

We're also working on our own diff-sync engine that generalizes virtual doms with collaborative editing and delta-compressed network transfer.

Any pointers to more information about this?

[toomim]

Thanks for these questions!

Statebus is already split similarly to your suggestion:

• statebus.js: abstract state management
• client.js: handy helpers and connectors for typical browser code (dom, react)
• server.js: typical helpers we use for nodejs servers

Client.js supports "typical" usage with react, but is also designed to connect to other front-ends and back-ends. You can slot statebus into any layer in the stack of state between the database and the pixels on screen.

Diffsync: A big insight we've had is that the dom is also state. There's an old state, and a new state, with every change. Because the dom is expensive to update, react et al diff the old and new versions and apply just a patch to the browser. We will do the same thing over the network. The network is expensive to send data across, so we will diff changes and send just a patch across it. This is how react's virtual dom innovation is generalized by statebus -- we are implementing react across the whole stack. Diffsync also enables collaborative editing for free. People can edit different parts of state simultaneously, and their patches will be merged together automatically without conflict. We haven't finished diffsync yet, but have some prototypes.

Oh, we also have some prototypes of ES6 Proxy support. It's almost ready. All major browsers support it now.

I can write more later, when I'm not on a phone at goodwill :p

[mitar]

Have you seen these arguments against using proxy. :-)

The fact that you are using a special method for saving state I think would mean that you do not really need a proxy anyway, no?

[mitar]

Because the dom is expensive to update, react et al diff the old and new versions and apply just a patch to the browser.

Just be careful here. They do not diff old state of DOM and new state, but old internal representation of DOM with new internal representation of DOM. DOM itself is even expensive to query for the state (if you do it outside of requestAnimationFrame) because you might trigger layout engine to be able to compute you the values you are asking for. See more about layout thrashing.

The network is expensive to send data across, so we will diff changes and send just a patch across it.

This is what Meteor does, BTW. And from experience there it seems like not one approach fits all. For example, Meteor for now diffs only at top level fields of the state. This works for most cases, but:

• some people prefer deep diffing, because if you have a subarray that might be large
• some people prefer no diffing at all because state on the server grows as you keep the state of the client (each server has to keep the state about the state of the client, so that you can do the diff on the server to know what to send to the client)

The downside of the first approach is that deep diffing in fact is expensive. (This is why some libraries are trying to get everything to immutable values, so that they can then just compare references.)

For the second approach the use case are chat applications. Messages are generally short, but you might have thousands of millions of users. So diffing at any level is not useful there.

People can edit different parts of state simultaneously, and their patches will be merged together automatically without conflict.

Yes, this is not a problem. Meteor already does that. The problem is when they are editing the same part of the state. :-) For that maybe check WOOT.

So Meteor is already doing all that, or in general I think this is a common pattern these days. You have state which you sync from server to client and from client into DOM. And from DOM into a modifier to the server state. And then that change on the server goes back to the client (or clients).

One interesting thing Meteor is doing is also latency compensation: when you change a state on the client, it applies automatically optimistically on the client, and sends it to the server, and then when server applies (and verifies) the modification, it sends it to the client (or clients) and then the initial client or rollbacks or commits the temporary change.

[toomim]

Just be careful here. They do not diff old state of DOM...

Yes, I know. We're doing the same thing. Our state is versioned, like git. When we do a diff, we compare the old version of the vdom state with the new version of the vdom state, and from that generate a patch, and then apply that patch to the real dom.

As for the server-diffing for network compression: yes, we keep a copy of state for every client on the server. But this is not that much data. For a Facebook page, for instance, we're talking maybe ... 70KB of state to maintain on the server? Compare this with the gigabytes of images and videos that Facebook is already storing on your behalf. Secondly, consider that websites commonly cache the entire rendered HTML in memcached—and this too is larger than the underlying state that generates the HTML. We will often shrink the data that needs to be cached.

On the other hand, deep diffing can be expensive in CPU time if you do it wrong. An optimal tree-diff is O(N^3). But we will get it down to roughly O(N) with good heuristics, and O(1) when the whole system operates on patches end-to-end and never requires a diff. For instance, an ES6 Proxy implementation of state can catch array modifications as they happen and auto-generate a patch, without having to diff the entire array. Then this patch can transmit throughout the statebus, and a full diff will never need to happen. So these systems will be architected for diffsyncpatch, but optimized such that all operations are patches, and everything happens in O(1) time.

It will work great for chat applications. :)

[toomim]

What diffing are you referring to with Meteor? Have any links? I'm quite familiar with ddp, but there's no diffing in it.

When you referenced WOOT, I think you're specifically referring to two people editing a single string? We handle that too—I think that's the coolest part. Every TEXTAREA in statebus will be collaborative by default, with multiple cursors for multiple users. Our diff algorithm operates on full arbitrary JSON. It diffs, patches, and merges edits to strings, and objects and arrays. Two people can only clobber each other if they modify the same number or boolean, or the same character within a string. Statebus applications will be automatically multi-user collaborative without additional programming effort.

This will work because we're baking differential synchronization into the web protocol itself. Consider how git lets multiple people edit the state of their files independently, and then they can push their updated versions to each other, and merge edits together, on completely different computers. We're going to do the same thing, but applied to arbitrary structured JSON, and happening with every keystroke or mouse click, and built into the web protocol, at every state:// URL. Every state change is the equivalent of a "commit" in git, and branches, merges, diffs, and patches transparently, behind the scenes.

FWIW, this automatically does the equivalent of "latency compensation," aka "optimistic updates," just like how git lets you update your local repository (committing new versions) before you've fetched and merged changes from a server or other peers. However, like meteor, we've also given authoritative computers control over their authoritative state — the server can abort a client's change with t.abort() in its save handler, and the client will roll it back.

The network protocol for all this is real simple. I haven't finalized the patch encoding yet, but here's how computers communicate the fetching and saving of different versions:

{fetch: "foo", version: "h2h32lj23j"}     // Get version "h2h32lj23j" of "foo"
{fetch: "foo", parent: "h2h32lj23j"}      // Get a patch of "foo" w.r.t. version h2h32lj23j
{save: {key: "foo", text: "hello!"},      // Push a new version of "foo"
 version: "277xy38",
 parents: ["98s7", "eiwu"]                // ...merging two parent versions together with 3-way recursive merge
}

A node will remember which versions it knows its peers have seen, and send them patches with respect to those versions. It will delete versions when they are no longer needed. And it can garbage collect them aggressively if it needs to—in the worst case, it just has to re-send the current data instead of a diff.

[toomim]

Re: proxy:

Have you seen these arguments against using proxy. :-)

1. The first is that they aren't supported in old browsers, and yes I know that.
2. The second is that Proxy(obj) != obj. That's only a problem if you program with both obj and Proxy(obj). With our Proxy design, you program using only the Proxy. (We have a special escape hatch to get the underlying JSON state: you just function call the proxy. Example: state.foo gets you the foo state, and state.foo() gets you the underlying JSON.) Here are some instructions if you're curious.

The fact that you are using a special method for saving state I think would mean that you do not really need a proxy anyway, no?

Correct. The proxy API is optional. It wraps calls to fetch and save to make programming cleaner.

[mitar]

As for the server-diffing for network compression: yes, we keep a copy of state for every client on the server. But this is not that much data.

I would really suggest you talk to Meteor developers about "not much data". I really depends on the scale you are working on. But I agree, for my level of projects I have not yet had issue with that.

Check this video where Rocket.Chat application talks about their scaling issues with Meteor.

But we will get it down to roughly O(N) with good heuristics, and O(1) when the whole system operates on patches end-to-end and never requires a diff.

Yes, I think this is a general approach many take. Find some heuristic (like id key in an array). Check how Meteor is doing diff of sequences.

Also, for string diffing, see fast-diff.

For instance, an ES6 Proxy implementation of state can catch array modifications as they happen and auto-generate a patch, without having to diff the entire array.

Yes, Vue is doing a similar thing.

What diffing are you referring to with Meteor? Have any links? I'm quite familiar with ddp, but there's no diffing in it.

No, DDP is just a communication protocol. Diffing is done on the server side, inside what is called mergebox. Check out this video for more information.

Meteor is doing diff only on top-level fields by default. I think it is a reasonable compromise. In any case, you should allow this to be configurable, based on stories I learned.

When you referenced WOOT, I think you're specifically referring to two people editing a single string?

I was mostly thinking of their interesting solution to improving operational transform which could be used probably for any state merging.

Every TEXTAREA in statebus will be collaborative by default, with multiple cursors for multiple users

Besides multiple cursors (and text selections), a useful property is also to remember authorship of a particular part of the text. Then one could color the text based on who contributed what.

We're going to do the same thing, but applied to arbitrary structured JSON

Have you looked into ottypes? And json1. That is from same people doing quill/sharejs/sharedb. They also made statecraft which shares a lot in common with statebus. Have you seen it?

Also this guide is a very interesting.

So I think there are some places here for reusing existing work. I think diffing is not enough, you have to apply those diffs in a correct order. Especially if you also want to support offline use (that I can modify state being offline and once I reconnect my state gets synced with the online version). Are you planning to support that?

So in which order you apply changes that is the question. And different approaches try to help with that: operational transform (OT) (using vector clocks), WOOT (adding explicit previous state, to help with resolving), CRDT (order of diffs do not match because they are defined to be commutative) (BTW, check CRDTs, if you haven't yet.)

All of those you can define not just on text strings but on structures like JSON as well. And people have been doing that already, see links above. So something is how you describe a diff, but something else is how you apply a series of potentially conflicting diffs.

Every state change is the equivalent of a "commit" in git, and branches, merges, diffs, and patches transparently, behind the scenes.

The issue is what to do when there is a conflict. With git, you ask the user. For text editors you try to guess the intent of users. And this is why it is a bit harder. I am not sure if your diffing is magical, but in general guessing the intent is pretty hard and this is why you have all those things like OT. So comparison with git is not a good one. That is a simple approach. Any scheme for collaboration is doing versioning and branches/forks. The question is how to do merging.

I think in fact that the main reason why automatic merging is hard and why we have so much different attempts on addressing it is because we are merging diffs. I think this is a wrong approach. We should not have stateA and stateB and compute a diff between them. Because that does not tell you how stateA became stateB, but only what are changes. So you loose all information about intent.

For example, in coding, often one indents the code. Or cuts and paste it. Or copies a block to a new location. If you do a diff, no matter if you do it at string level or AST level (which is slightly better) you do not have information on this available, especially if the copied part was further modified a bit, so it is not a perfect copy anymore at the moment when you do a diff.

What I think statebus could do and have an unique opportunity to do, because DOM is also seen as part of the whole stack, is to keep track at this immediate level of changes. You could know exactly what was copied from where and store it as such operation. So if somebody copies a block, and another user edits the same block, then merged result would be copied block twice with the edits applied. In a normal diffing and patching this would be a merge conflict you would not know how to resolve.

To keep track of copies, every time anyone selects any text on statebus-based app, you would wrap it with some <span id="source-123456">...</span> and when then user pastes this in, you has this ID to resolve where it came from (you would allow always just pasting into content editable and all standard inputs would be just styles content editables). Or maybe you could just remember what was selected every time something is selected, and if you detect that content is added, you check if this was recently selected and guess that it is coming from there (but there could be false positives for short strings like one character).

So my experience is that merging issues happen because you loose information about every step of a transformation, and that transformations are too basic (like remove/insert, instead of copy/paste, indent and so on), and because they are not done at every step, but just at arbitrary state commits. This also looses information about intent of users.

I haven't finalized the patch encoding yet, but here's how computers communicate the fetching and saving of different versions

Maybe look into JSONPatch. I think it might be interesting to use a standard, just because then it would improve interoperability.

A node will remember which versions it knows its peers have seen, and send them patches with respect to those versions. It will delete versions when they are no longer needed.

This is pretty simplistic approach in comparison with things like operational transform. I am not an expert in this, but I think they are not making it more complicated unnecessary. So maybe this simple approach does not work so well as you seem to believe it does.

One more thing. Have you looked into JSON LD? I think it might be an interesting way to use to store URLs to which every field corresponds to. So you would not use it to provide semantic information about each field, but to record a source of each filed, but as a way to convert a JSON like:

{
  "@context": "http://example.com/statebus/",
  "@id": "12345",
  "name": "John Lennon",
}

Into, for example:

{
  "http://example.com/statebus/#id": "12345",
  "http://example.com/statebus/#name": "John Lennon",
}

This does not look like anything special. But when you then merge two such JSONs from two sources, you could get:

{
  "http://example.com/statebus/#id": "12345",
  "http://example.com/statebus/#name": "John Lennon",
  "http://wikipedia.com/statebus/#pictureUrl": "http://wikipedia.com/john_lennon.jpg",
}

JSON-LD defines a nice way to make such JSONs compact (or expanded), with tooling to do so. And it is a standard.

You can also use it to give semantic meaning to fields as well, beyond just basic number, string, array types available in JSON. This would be great because then it would be possible to automatically generate forms and UIs in statebus directly from state information (because state would also encode semantics).

[mitar]

We have a special escape hatch to get the underlying JSON state: you just function call the proxy. Example: state.foo gets you the foo state, and state.foo() gets you the underlying JSON.

I think this is really bad API. So error prone. Maybe do something like state._json.foo instead. So that it is clear that you are operating on JSON. Or even state.toJSON().foo.

Will be returned JSON mutable? Maybe freeze it. :-)