Add optional support for channel rendering instead of Ajax

[smpallen99]

I'll be integrating this project with my chat app soon. The chat app uses channels for rendering server side and pushes HTML to the client over channels. So, I would like to add optional support for channel rendering. Ultimately, I'll be adding live form updating with this approach.

[tmbb]

What dores the ready label mean?

[smpallen99]

I'm using waffle.io for task management. Is a label generated by the app. It means that the issue is ready to be worked on.

[tmbb]

Have you already decided that you want to push HTML to the browser instead of pushing json and having the browser update the UI using js?

[tmbb]

I'll just rehash some things I said in the chat so that they're visible here.

There is a very good JS view layer called vuejs with a very easy to use template lanuage that's really hard to beat for individually designed pages. It's great when used with phoenix channels for realtime UI updates. You basically push json from the server and vuejs will build the UI from the state. The "normal" way to use it is to compile the js templates with brunch on the server and serve them as a normal script. This is not giod for somethig like talon, in which we want to customize things as much as possible from elixir templates or code. However, vuejs also supports templates embedded in HTML and can compile them on the browser, with a little speed penalty and larger js artifact (because it has to load the compiler). I haven't tried or benchmarked this option, but as long as we serve uvejs from a CDN performance should be good. This option allows us to generate the vuejs templates on the server.

An advantage of using something like vuejs is the ecosystem already built around it, especially regarding form components, client side validation and AJAX validation. Integrating with components not originally built for vuejs is also very easy. For the admin part of talon, this could be very convenient. For the non-admin part it might be more restrictive.

The other option is to use something like Drab, which @smpallen99 has been looking into. This is less restrictive because it allows you to do whatever you want from Elixir, but might make the architecture more complex and makes client side validation impractcal (due to latency).

I think that a disadvantage of something lie Drab is that it might make it harder to add incremental updates to the state. Imagine you're collaboratively editing a document/form/whatever. You might want to send diffs to other clients connected to the same page. In this situation it's probably easier to send the diffs and allow the js in the browser tobuild the local state from the previous state and the diff.

[smpallen99]

@dsnider and I were talking about this topic today. The idea we have right now is the concept of a series of default components and optional components. They can either be server side components and client side. We are thinking of provides some integration support like GraphQL, for example. So, the design has the option of using the default components, or use/create optional components.

I'm not sure how this is going to look yet. I want to get the the rest of the datatypes implemented and some more association support. Then we should have a better idea of the patterns and abstractions we need.

[tmbb]

I believe that state representation should be a priority and should guide the implementation of the UI components. So let me suggest something which you'll probably hate as being too opinionated. Let's say we want to go hardcore full realtime, since it seems like the scope of this project is way beyond a simple admin interface. The most realtime web interface I know is Google Docs, where two people can edit the same document at the same time. In the spirit of "realtime or bust", let's strive to emulate this.

The goal of a highly interactive application screen (what you see in the screen at each time) is to edit a lump of data. If someone else is editing that same data at the same time, we want that change to affect the screen in real time.

So, a bona fide real time component must provide, client-side:

• a way to display itself given the initial state
• a way of sending state changes in real time to the server
• a way to display itself given changes to the state that come from the server, if possible without disrupting the user's editing flow. Taking inspiration from Google Docs, it should be possible for one user to edit one part of the text, while another user edits another part of the text.

For most datatypes, this can be done by communicating state directly. For text widgets, we need a more sophisticated diff format, and a way to handle race conditions, but I'm sure the Elixir or Erlang ecosystem must have a ready made solution.

The authoritative state lives in the database, of course (possibly with acaching layer or something), but from the point of view of the client, the state live in the client and only changes are communicated from the server to the client. The client could (and should) periodically poll the server for the latest state in case the state becomes corrupted, but the normal mode of operation would be keeping a very fat client with a ton of state and get updates from the server.

Now, how does this mesh with associations? Imagine we are editing a member of the table A, that has associations with table B and C. The rows of table A are:

• A.id (primary key)
• A.f1 (a field)
• A.f2 (another field)
• A.f3 (yet another field)
• A.b_id (foreign key)
• A.c_id (foreign key)

We will need 3 channels:

1. one that listens on all updates on the particular element we are editing (element A.id) and to that element's deletion
2. one that listens on inserts and deletions on table B (so that we know when association changes)
3. one that listens on inserts and deletions on table C (so that we know when association changes)

Channels 2 and 3 are easy to setup server side if we are using a wrapper around the persistence backend.

There would be a global channel listener that would listen to these channels and dispatch the state changes to the appropriate components so that they could update themselves. Components that don't with to be real time could just ignore the remote state changes. Components that want to be realtime but don't want to deal with handling the deltas, could just query the server for the updated state when a delta arrives, at the cost of a server roundtrip.

This solution has a number of advantages:

• The server doesn't need to keep client state, just fire some deltas and let the client deal with it. Race conditions could be solved by serverside timestamps on the deltas

• The state deltas being passed around is pure data (JSON, msgpack, XML, etc.), and not fancy JS calls like in Drab.

• Developing a component requires only: 1) render the UI from state, 2) render the UI from the previous state and a delta. Components need not (and should not) listen to channel events directly. That work should be delegated to the global listener.

Vue.js offers a natural way of implementing these ideas, but it's not the only one. Having a bunch of jqueryui components which has lots of ready made components of very high quality controlled by a global event listener is also an option. The only real restriction here is that we must have a fat client and a thin server.

I know this means we have to write javascript instead of elixir (ugh!), but it does seem to offer a simpler way of handling the collaborative editing problem.

[tmbb]

A problem with this approach lies with associations and permissions. Maybe the user doesn't have access to all elements of table B, and in that case sending some updates of table B over the channel will leak information. This would require several channels (maybe even one per user) and some mechanism for the server to know whom he should send the insert, update and delete events. This might be computationally expensive: each event requires checking if a certain user can know about that change, for all users subscribing to updates to that association.

[tmbb]

For realtime collaborarive text editing look into quiljs and it's native document document format, the delta format which is optimized for operational transformation and has an elixir implementation. Now that I've read a little about this it seems like you can't just fire deltas as they and actually need an elixir process per document to coordinate simultaneous edits. Hopefully this won't be too resource intensive. Maybe we can batch several changes client side instead of sending each keystroke. In any case the implementation is way over my skill level, and that's why I'm being lazy and offering suggetsions instead of implementing a prototype.

The delta format works for both plain text and rich text. It's json so it can be packed into msgpack and I suspect it would be easy to pack it even further.

Other datatypes, like numbers, booleans, dates, timestamps, etc would not benefit from this and should have their values transmitted directly over the wire.

PS: I've just noticed that this collaborative text editor example is an argument against "one channel per DB row or 1 channel per table"... Since this would require some server side state, it makes more sense to have the server side plumbing for such an editor as a standalone module (maybe even a standalone app, I don't know enough to speculate on this) and have the ui component communicate directly with the server through a dedicated channel. This would make the component reusable outside talon. I'd like to amend my suggestion by saying that there should be one channel per field and not one channel per editable field and that fields should communicate with the server. This negates much of the point of having a standardized representation of state over the wire, but I'm still biased towards communicating data or deltas and let the client resolve the state into an HTML representation.