Store supplies messages in different order than intended

[jcookems]

Dev Estimate: 5 Test Estimate: 5

To simulate lots of messages, added code to the Send button click in the browser code (edited index.jade)

for (i = 0; i < 100; i++) {
    socket.emit('user message', "foo " + i);
}

The messages that show up in the chat app connected to a different node (that is, where the messages travel via Service Bus) are in a different order:

Send Recieved
0    0  
2    1  OutOfOrder
1    2  OutOfOrder
3    3  
4    4  
6    5  OutOfOrder
5    6  OutOfOrder
7    7
... 

Investigating more deeply, I added output to servicebusinterface.js to snoop on the values as they were being sent out and revived, and they were out of order there as well.

There are two places where the problem could be occurring:

1. The module is just queuing a bunch of async tasks to be executed without a strong order, so Node is just processing them in whatever order it feels like. To fix that, we'd have to impose a strong order when uploading.
2. Perhaps the Service Bus topic, when receiving a lot of messages at about the same time, just processes them in the order received. Fixing that would be more difficult as we'd have to wait for confirmation from the topic before sending the next message.

Or, we can fix at a higher level and address the throughput issues at the same time, by batching messages before sending. Then, the receiver gets one big batch, and the messages will be in the intended order.

[christav]

Do we know if the messages are being sent to the store (SbStore.publish method) in or out of order? If they're being sent to the store out of order, then that's socket.io's doing, and our assumptions about the message ordering is wrong.

If they're arriving ordered to the store, then there's a problem on our end.

[jcookems]

I logged the messages in the module received from the store, before getting sent to SB. So the store is sending them in order. And the Socket.IO is getting the messages in order from the client (in a batch actually).

[glennblock]

This is by design. Service Bus does not make ordering guarantees when over HTTP due to the inherent latency of the protocol. If ordering is critical, the application will have to handle it. Per Clemens there is an ID that customers can use.

[jcookems]

I read that WebSockets will ensure the correct ordering of messages, even if the underlying packets are exchanged out-of-order.

Does Socket.IO make ordering guarantees when broadcasting from one node to all the other nodes?

If so, then we need to ensure that our store implementation respects order, which would probably involve us writing the code to re-sequence the messages by ID, or by batching.

[glennblock]

Websockets is orthagonal. That's a protocol for peer to peer communication between server and client, just as XHR long polling. In this case we're talking about server to server.

Socket.io ensures that any messages it receives are sent to the client in the right order. In this case the issue is that the messages are coming in from SB out of order because there are no session guarantees as it's HTTP.

With our SB store, the same ordering is still in effect. The issue here is that the messages coming from SB are out of order.

[jcookems]

OK, I've clarified the title of this issue. To summarize:

1. SocketIO store contract says that messages will come out the same order they went in.
2. Our store does not respect the ordering (because of how we use Service Bus)

So we need to change our store implementation to respect the ordering.

[christav]

This should be delivering in the correct order now.

[jcookems]

To be clear, there are two parts to this fix. One is batching, which solves most of it. The other is the new messagesequencer code. I've tried this with a modified chat app to send lots of messages in a short time, and it works as expected. I also looked over the code to see if there were any issues, but it looks fine.