new way to do downward sync

[sojharo]

This task is to modify current sync so that it becomes efficient. Discussed in :

https://docs.google.com/document/d/1OuH-Fr3b2TjBNDRpIzDMdI7dzf4exd15EWAxiWyU5k4/edit

This task is for downward sync

[jekram]

@sojharo Please see the comments Q: Are the performance issue we are seeing. Is the issue happening because of upward sync or downward sync? Which is the main culprit?

Q: In our testing how many messages were in the queue when we saw this issue?

Q: Why the client would not update with the notification services? When the server has to send something, it sends the notification to the iOS and Android, and these are queued by the notification services. When the client comeback up would it not get these notifications and try to get these data?

Q: What do we know in testing WhatsApp how this is handled by WhatsApp?

@sumairasaeed

[jekram]

Q: In the current client we are only doing incremental sync on both Android and iOS?

[sojharo]

Q: Are the performance issue we are seeing. Is the issue happening because of upward sync or downward sync? Which is the main culprit?

Main Culprit is the upward sync. As stated in document also, it runs move http requests.

Q: In our testing how many messages were in the queue when we saw this issue?

Mostly, the messages are two or three. However, there other type of http requests which are done too i.e. to check status of sent messages, to send the status of received messages.

Q: Why the client would not update with the notification services? When the server has to send something, it sends the notification to the iOS and Android, and these are queued by the notification services. When the client comeback up would it not get these notifications and try to get these data?

Because of short life of push notification, it doesn't reach device if device was offline. So device will not know that there was a push notification to get the data.

Q: What do we know in testing WhatsApp how this is handled by WhatsApp?

I asked similar question on quora and found that whatsapp does polling of messages on server instead of doing sync. https://www.quora.com/How-often-should-a-chat-app-like-WhatsApp-sync-with-a-server

Q: In the current client we are only doing incremental sync on both Android and iOS?

Yes, it is incremental sync. The amount of data is not problem as it is always small. The number of http requests sent to server from android is problem.

[jekram]

Q: Are the performance issue we are seeing. Is the issue happening because of upward sync or downward sync? Which is the main culprit?

Main Culprit is the upward sync. As stated in document also, it runs move http requests. Thanks.

Q: In our testing how many messages were in the queue when we saw this issue?

Mostly, the messages are two or three. However, there other type of http requests which are done too i.e. to check status of sent messages, to send the status of received messages.

Do you not think that we should be able to handle 2-3 messages. I do not see this in testing Android. I see this more on iOS. I would think we we should be able to handle 2-3 messages. I am afraid we have other issue and this resign would not that problem. Have done iOS trace on client during this time frame to validate this issue? We are going from 2-3 message to 1 message and how much of a difference it would make.

Q: Why the client would not update with the notification services? When the server has to send something, it sends the notification to the iOS and Android, and these are queued by the notification services. When the client comeback up would it not get these notifications and try to get these data?

Because of short life of push notification, it doesn't reach device if device was offline. So device will not know that there was a push notification to get the data.

I thought the push notification is queued for couple of months? Are we not doing double request right now?

Q: What do we know in testing WhatsApp how this is handled by WhatsApp?

I asked similar question on quora and found that whatsapp does polling of messages on server instead of doing sync. https://www.quora.com/How-often-should-a-chat-app-like-WhatsApp-sync-with-a-server

Thanks. What is polling means?

Q: In the current client we are only doing incremental sync on both Android and iOS?

Yes, it is incremental sync. The amount of data is not problem as it is always small. The number of http requests sent to server from android is problem.

How many message we typically send. I thought it was 2-3 messages. Or is it hundred of messages?

Q: Can we quantify what the time synch go to after these changes?

[sojharo]

Q: In our testing how many messages were in the queue when we saw this issue?

_Mostly, the messages are two or three. However, there other type of http requests which are done too i.e. to check status of sent messages, to send the status of received messages.

Do you not think that we should be able to handle 2-3 messages. I do not see this in testing Android. I see this more on iOS. I would think we we should be able to handle 2-3 messages. I am afraid we have other issue and this resign would not that problem. Have done iOS trace on client during this time frame to validate this issue? We are going from 2-3 message to 1 message and how much of a difference it would make._

Yes, as I discussed the amount of data is not the problem the number of http requests we run is problem. Sync is not only done for chat but also for group chat, chat status for 1-1 and group chat, new groups created while offline, and then downward sync. It is the number of http request that we want to reduce.

Q: Why the client would not update with the notification services? When the server has to send something, it sends the notification to the iOS and Android, and these are queued by the notification services. When the client comeback up would it not get these notifications and try to get these data?

_Because of short life of push notification, it doesn't reach device if device was offline. So device will not know that there was a push notification to get the data.

I thought the push notification is queued for couple of months? Are we not doing double request right now?_

iOS doesn't get push later when it was offline. However, on Android it gets for group chat messages. Nevertheless, the upward sync (which is main problem) is to send data from mobile to server which will be opposite to push work.

Q: What do we know in testing WhatsApp how this is handled by WhatsApp?

_I asked similar question on quora and found that whatsapp does polling of messages on server instead of doing sync. https://www.quora.com/How-often-should-a-chat-app-like-WhatsApp-sync-with-a-server

Thanks. What is polling means?_

They poll the messages on the server in queues. When the device comes online, they push the message to device thus nothing is stored on their server and therefore they don't need sync. They are using different set of technologies i.e. XMPP, ejabbered on their backend.

Q: In the current client we are only doing incremental sync on both Android and iOS?

_Yes, it is incremental sync. The amount of data is not problem as it is always small. The number of http requests sent to server from android is problem.

How many message we typically send. I thought it was 2-3 messages. Or is it hundred of messages?_

It is 2-3 messages. The amount of data will not affect us. The number of http requests will.

Q: Can we quantify what the time synch go to after these changes?

I couldn't completely understand the question. The time sync will take after these changes. It would reduce the time as it would only do two http requests to server: one to do upward sync (to send pending data) and one to do downward sync (to get data pending on server)

[sojharo]

I have completed the work on this and pushed the code to github and now publishing to production.

[jekram]

Thanks

[jekram]

FYI @sumairasaeed