Move to a glitch/etherpad/gdocs-like autosave model

[wlach]

The "explicit save" model is so 2003. Let's have the autosave experience that users expect these days.

Goals

(note: not all of these will necessarily be implemented for the first pass!)

1. Allow "save-as-you-type" functionality
2. Allow multiple users to concurrently modify a notebook
3. Allow a user to propose a set of changes to be merged back into a source notebook
4. Allow tracking which users are responsible for which revisions to a notebook

Implementation details

• Client stores a copy of the last accepted/acknowledged copy of the data on the server, along with its current representation.
• Every 30 seconds, if there are any changes to the document, client will attempt to contact server and submit its changes vs. the last known server revision. Server should respond with an acknowledgement and a new representation of the jsmd for the client to use.
• When receiving a set of changes from the server, client may have to perform the following operations:
  • Delete cells
  • Add cells
  • Modify existing cells Not sure how hard this is, in theory this is the sort of thing that react makes easy.
• If communication with server fails for an intermittent reason, let client know prominently. Continue to store changes inside localStorage/indexedDb and sync at next opportunity.
• How to handle a giant merge conflict with server? Maybe pop up an error and offer to fork? Probably an edge case.
• Similarly, how to handle a permanent error? (e.g. user loses rights to a notebook?) Just hide their work? Again, probably an edge case.
• We will start just by storing the full contents of the notebook for every revision. If this stops scaling, we can look at xdelta or something similar to only store chunks. Diffs, etc. will be displayed by doing diffs on either the server side or client side, whichever is faster and makes more sense.

Prior art

• Documentation on etherpad storage system: https://github.com/ether/etherpad-lite/blob/develop/doc/easysync/easysync-full-description.pdf

[bcolloran]

love this proposal will! :+1:

[hamilton]

Every 30 seconds, if there are any changes to the document, client will attempt to contact server and submit its changes vs. the last known server revision

@wlach would the idea be to reduce that delta down as low as possible to make way for real-time collaboration?

[wlach]

Actually, yeah, 30 seconds is too infrequent. :) Let's see if we can't get realtime collaboration going using django channels. Seems like etherpad uses socket.io for this (basically the node equivalent).

[hamilton]

@wlach Yes, that seems right - kind of a canonical web sockets use case.

Solving this issue will also require solving #916 (notifying user of a severed connection), though I think we'll get most of the way toward that issue once we have the machinery in-place for this issue.

[hamilton]

@wlach here's a strange idea. What if we also enqueue evaluation? So I receive both the document and the steps toward reproducing the current state. As an opening bid, we don't validate state, and perhaps we maintain that we don't validate state going forward.

This could be an option that is toggle-able, so not the default. I can see use cases for this in collaborative editing (maintaining something along the lines of a similar eval history, if not kernel state which is a much harder problem), teaching that allows the driver to also explore things on their own in some way, etc.

[wlach]

@wlach here's a strange idea. What if we also enqueue evaluation? So I receive both the document and the steps toward reproducing the current state. As an opening bid, we don't validate state, and perhaps we maintain that we don't validate state going forward.

Interesting idea, I'm not quite sure how to present a specific set of evaluation steps to the user in an intuitive way though. I suspect that with iodide notebooks, because they are re-evaluated so often, there will strong pressure for them to be runnable in linear order.

[hamilton]

I think the evaluation steps would simply appear in the console, and the notebook gets them entirely from the server in that instance, and the server communicates the queue to the connected notebooks, and the notebook itself is responsible for the evaluation. So long as its in the console, I can see it being pretty easy to see who has evaluated what.

Also to be clear I don't think this is probably in-scope but I added the proposal label to this so we can talk freely about any hair-brained ideas, so here we are.

The OP feels like the right set of requirements and constraints.

[wlach]

I still want to do this but am going to put it on hold for a short while, as I attack lower hanging fruit. The new autosave implementation (#1131) goes some way to solving some real pain points which motivated this work.

[wlach]

Let's close this bug, we've implemented much of it, and our discussion/thinking in this area has evolved so that much of the discussion of the remainder is obsolete.