Posting Messages

[EGreg]

We need a mechanism for apps to be more responsive, and not wait for a roundtrip to the server. At the same time, we need a way to rollback these "optimistic" change when the server state turns out to be different than expected.

This issue is about changing and posting messages to streams which already exist on the server. Please create a branch for this issue and commit it only after it's completely tested.

We have get/set/clear methods on streams for managing attributes. Please create methods to manage stream.pendingFields like this:

setting pending: stream.content(value) // and so on for other fields getting final: stream.content() getting pending: stream.contentPending() pending clear: stream.content(null); // works for non core fields only

Make a few event factories similar to Stream.onFieldChanged:

Streams.onPendingField(publisherId, name, fieldname)
Streams.onPendingAttribute(publisherId, name, fieldname)
stream.onPendingField(fieldName)
stream.onPendingAttribute(attributeName)

Add event factories to handle posting of a message:

Streams.onPost(type, mtype)
Streams.Stream.onPost(publisherId, name, mtype)
stream.onPost(mtype)

When Message.post triggers events in the above factories, it should pass the Q.Promise returned from Q.request. (See https://github.com/EGreg/Q/issues/3)

Internally, Message.post should maintain an incrementing mid for each browser window. It should be posted as one of the fields of the message, and it should be sent by PHP to Node.js and included in the result of $message->exportArray() in PHP and Node.js only if the userId and clientId matches.

Finally, add a method similar to Q.page() to handle pending messages:

Streams.Message.pending = function(publisherId, name, mtype, handler) {
  function Streams_pending_handler(message, promise) {
    var rollback = handler.call(this, arguments); // apply interface changes
    if (!rollback || typeof rollback !== "function") {
      throw new Q.Exception("Streams.pending: please provide a way to roll back");
    }
    Streams.Message.pending.rollback.set(function () {
      rollback(); // this is a closure
    }, message.mid, true); // prepend = true so we can rollback in the opposite order

    promise.onFail().set(function () {
      // should we schedule a retry? or just roll back?
      var event = Streams.Message.pending.rollback;
      event.handle.call(this, arguments);
      event.removeAllHandlers();
    });
  }
  Streams.onPost(publisherId, name, mtype).add(Streams_pending_handler);
}
Streams.Message.pending.rollback = new Q.Event();

and then before you trigger _messageHandlers in the Streams.onEvent('post') handler, check the order of the incoming messages. If there is a discrepancy (figure out how to detect this), do the following:

var event = Streams.Message.pending.rollback;
event.handle.call(this, arguments);
event.removeAllHandlers();

Please test that the side effects of the messages really are being rolled back in the right order.

Note: this feature does not implement offline streams & messages support. That will be done in another feature. This feature is just to increase perceived responsiveness by exposing events for implementing interface feedback.

[EGreg]

This implementation can actually be improved. First of all, instead of mid we can have "expected ordinals".

When posting a message, if the ordinal that comes back is equal to the expected ordinal, you can already be sure that the message doesn't have to be rolled back.

In addition, messages can be posted in batches with only one batch request outstanding per stream. This will preserve the order in which messages are posted. For example, if the current messageCount is 7, and we post a message with an expected ordinal 8, and while the request is going, the client wants to post messages with expected ordinals 9 and 10, those messages will be queued until the request from message "8" will come back. If there was a failure for some reason, then message "8" will be added to the front this batch. In any case (failure or success, and even if the resulting ordinal wasn't expected), the batch is sent with the messages in it.

So Streams/message POST should in fact accept multiple messages at a time, perhaps as JSON. And the client should be responsible for using this mechanism in order to preserve the order of messages.

If the client has a high latency, then messages will be posted in order but may take some time to be posted. The client interface will show immediate feedback, and if the messages that come in do not match the expected ordinals, then the "optimistic" interface feedback will be rolled back and the messages from the server will be handled by the players.

[EGreg]

In short -- we have to implement the scheme described above.

1) Extend Streams.Stream methods to getting/setting/clearing fields. Make event factories for onPost of messages, and maybe one for beforeSave.

2) Make Q.Message.pending function for registering handlers to "optimistically" update the interface in a browser window, when certain types of messages are posted to a stream. And these handlers will return the rollback functions (which are closures that roll back the interface changes). These functions should be added to the Q.Streams.Message.pending.rollback event with the key being the ordinal of the message! (Maybe with "Streams.Message.pending " prepended.)

3) Make sure all messages are posted in the right order. Expand Streams/message POST to handle JSON of one or more messages being posted, with "expected ordinals". Have the client make one queue per stream, so if a message request is still pending for that stream, they queue up and get sent as a batch. If the request fails, then prepend the previously attempted messages to the queue. In any case try again. If the queue has become too large, stop allowing posting of messages, and just throw an exception or something, which can be displayed in a non-blocking notice.

4) Since the server is now saving all the messages in the same order they were posted, the client can inspect the messages and see if the expected ordinals match the actual ordinals. If so, then the client simply fires the onMessage handlers. For as long as this is the case, it should remove the "rollback" handlers one by one, based on the key derived from the ordinal of the message.

5) Otherwise, upon encountering the first ordinal where it is not the case, it should call Q.Streams.Message.pending.rollback.handle() which would execute all the remaining rollbacks, and then would go ahead and trigger the onMessage events as usual. It would be very nice to pass wasRolledBack to the onMessage handlers about whether the optimistic interface update was rolled back or not for that message. So a typical handler that a developer would write would check wasRolledBack and, if the interface update already did all the work, return or maybe just set opacity of something to 1.