Message stuck is pending queue

[jekram]

@sojharo

This is the screen shot that happened yesterday

Now it happened again today. I went out of app several times and it is still in pending.

[sojharo]

I have worked on this and have fixed the problem. I need to do more testing on this between android and iOS also.

[jekram]

you need to do more testing or this is resolved?

[jekram]

what release this is fixed in?

[jekram]

This is not fixed

Message is stuck in Pending state on CloudKibo for 30 min while I can send messages of Whats App

[jekram]

BTW: This is reproducible....

Send multiple chats t "a" and all of them will be in "pending" then go to B and send chat messageto "B' and that would in pending also go back to "a" and now status chages to sent go back to 'B' and now the staus is sent

[sojharo]

what is the state of Internet in all these step? when do we close the Internet and then open it back.

[jekram]

Trust me nothing to do with the Internet. I reproduced it three times. Why the internet would be down the first time and then when I come back the internet would be up? Also during this time I was able to send via Whatsapp

[sojharo]

so it showed pending on sending of message while Internet was on. I have seen your messages in the logs. These messages have reached the server so server has sent the acknowledgement back to the device for each message. the device must update the status of each message.

[jekram]

ok. I am reporting what I see

[sojharo]

I kept the 10 chat messages in pending status and didn't open the Internet for 6 hours. I opened the Internet and then opened the app and all the messages were sent. The status got updated from pending to sent. The logs on server shows my messages sent to server:

[jekram]

what do you show for me?

[jekram]

so we need to figure out why the sender status is incorrect?

[sojharo]

Let me send message to your number using cloudkibo and see your messages back. I want to understand why the status is not changed on your side.

[sojharo]

I did lot of testing on this and instead of "pending" status error, I got the error of "delivered" status. In this, the delivered message status was between the seen message status. Like in the following image:

I have solved this problem in current issue. I would do more testing to reproduce the "pending" status error. The error of "delivered" status was also not easily reproduced. I am sure with more testing I would be able to reproduce "pending" error

[jekram]

@sojharo

What is the next step here? How much work is left?

Refresh me the design document on this logic? Add a link here.

We should store all design in the Repo when it has been agreed

[sojharo]

I did more testing in this and this time I got the error of "pending". I have solved the error now. The solution is in version 38 that I will push now. There were two issues in this task, one was solved one day ago. It was in "delivered" status and other error was in pending. I have solved both now.

Here is the design documents:

Use Case 1

Use Case 2

Use Case 3

https://github.com/Cloudkibo/CloudKibo/blob/master/cloudkibo_documentation/chatstatusdoc.md

Long discussion was done here.

https://github.com/Cloudkibo/Android/issues/122

[jekram]

Thanks

[jekram]

It is still happening. I sent two messages and then after that rest of the messages have gone into pending. In between these I think I pressed call button by mistake (I think. I am not 100% sure), but now everything is in pending.

[jekram]

Msg 1 & 2 never sent. We need more through testing. It has happened multiple times.

We have logic issue. with TWO things

1) Message should not be stuck in pending queue 2) Msg 3,4, 5 should have "seen" status

Screen shot of sender:

Screen shot of receiver

[jekram]

The pending issue happened again.

Why my outbound message is in pending when I can receive the message. So It, not the network.

[sojharo]

I have worked on it but still it is not fully resolved. This happens when during our chat, android goes to sleep mode. When he wake the android next time, our messages are not going to server and remain in pending. When it goes to sleep the socket gets disconnected. I have tried connecting it on wake, but it gets connected twice or thrice thus creating more than one socket connections with server.

[jekram]

Thanks for the update.

On my phone the issue can be reproduced very easily. So think about how you want to debug

[sojharo]

I spent lot of time on this. The issue is still happening. The problem is socket.io is little bit slow in connecting or disconnecting. i.e. if socket connection is broken between server and mobile, both don't know about this until next heart-beat. So, until android knows that socket.io connection is broken, socket.io library shows connection as active. So we think that socket is connected and we send the message. It doesn't reach in real-time.

I have added one logic to reconnect with the server if socket gets disconnected while application was in background or phone was locked. With this logic, when we unlock the device, or bring the application back to focus, it reconnects with the server. However, it is painfully slow. It takes around 10 to 20 seconds to connect. Still in this case, the message is not sent and we have to just go to previous app screen and come back to chat to send it. However, this time we don't need to close the application which starts sync. Still this is not good solution.

I tried several ways today. Later, I just thought that we don't use socket.io to send the message and send it using API and then in API we call push notification to send the message to receiver. The other option is to call the sync to send the pending messages whenever socket gets connected. I have started to code on server side to implement the former. However, if you think the later option is better then I would start working on that one.

[jekram]

@sojharo

The pending message was a very rare issue before, and now it has become a norm in the last 2 weeks.

This has nothing to do with app going into the background. I will send two messages, and then the third message goes into pending.

It is also happening on both of my devices on a regular basis.

Also, a number of time I am getting receiving messages and the send would be in pending.

This use to work and this is change-related, and as this is happening with your device also, we need to debug it. Instead of creating new logic.

I would also rule out server side code as we not seeing this problem on iOS.

I do not understand when you say send it using API? Would API not using Socket.io to send the message?

[jekram]

Let's understand the issue first before we change direction.

[jekram]

Also, we ned to have consistent design (as much as possible) between iOS and Android.

[jekram]

This is a logic issue - nothing to do with something else. How we can received the message and can't send the message.

[sojharo]

Yes, I have debugged it a lot before I thought of going to the other logic.

Yes, you are right the issue was very rare before but still it happened some times. Socketio connection is not very stable. The reason for this issue is that we have stopped socket.io service running in background when app closes. We did this after discussion that we already have push notification to receive the messages so we should not keep socket.io listening in background. This is the main reason behind the problem.

I think this problem also happens on iOS sometimes and sumaira had discussed with me once.

I am able to reproduce it in this way. I open the app and keep sending messages. I can send as many messages as I want. It doesn't stop after 2nd or 3rd message as it is happening with you. Later, I just leave the device there and after some time device gets locked and application goes in background. After, some time I open the device and app comes to foreground. When I send the message at this time, it goes in pending.

We are receiving messages because they are not using socket.io. The receiving message is coming through push notification and then API to fetch it. Sending of message is on different channel and receiving is on different channel.

I have debugged it and as I discussed in above comment that now it automatically connects when app comes to background (and also if app was in foreground and there was socket connection broken), however, it takes time to establish the connection this time. Sometimes 20 seconds. Also, as discussed in previous comment, now the pending message is sent and we don't need to restart the application. It goes when we leave chat screen and go to chat list and then come back to chat screen.

I have discussed two options above: one of them is to change the logic i.e. change the channel from socket.io to using API. Other option is to do sync as soon as socket is connected.

I do not understand when you say send it using API? Would API not using Socket.io to send the message?

API would be on server and server uses push notification to send message to other client. Similarly, API would use the push notification to send the message to other client. Receiving client would always receive message using push.

I thought of API as it is stateless and doesn't depend on active connection. But when socketio disconnects and I try connecting again at this it takes more time to establish connection. This is the core reason I wanted to switch to API. Remember, we had once discussed Azure way of sending push from device. It was also using API. However, I have also discussed other option, which is not change in logic. Let me try the other option before and work more on it.

[jekram]

@sojharo

I do not understand of this fully. So we need to have more discussion on it.

We are way behind schedule, so we to trade between tactical solution and long term solution.

1. If the main reason of this issue is: "Socketio connection is not very stable. The reason for this issue is that we have stopped socket.io service running in background when app closes. We did this after discussion that we already have push notification to receive the messages so we should not keep socket.io listening in the background. This is the main reason behind the problem"

Do you not think we should keep Socket.io in the background? It will consume battery but on a tactical basis, we would be back to the working product.

What remember the change we made was on the input side and decided that we would not use Socket.io for input, but we left the output to Socket.io.

The key thing I believe that we should not create a regression. By fixing one code, we break something else.

1. I am all for changing to another method. However, we need to balance it how long it would take. It always start with a small change and then days become week and week become month. Without fully understanding it implication we would no do it.

When we have a tactical solution why we would not do that first?

[jekram]

@sojharo

1. Please help me understand when there is a pending message and I go out of the app and come back the message is sent right away. What is the logic that causes it to send?
2. Have we made the change to make socket.io running in the backgroud?

[sojharo]

I worked on it and created the following diagram to understand how it would be done using HTTP API to send the message.

In the following Azure article, they are also using http to send messages from the mobile to azure notification hub which will then send to all other devices using push notification.

Further, as per our discussion I kept the socket.io running in background and I had published the app with version 66 for this change.

Please help me understand when there is a pending message and I go out of the app and come back the message is sent right away. What is the logic that causes it to send?

When we go out of app and come back to app (if app restarts), we do run the sync code which sends our pending messages to server. It connects to socket first and then sends our pending messages through it.

Have we made the change to make socket.io running in the backgroud?

I had not made this change in version 65. So, if you used version 65 to test then no such change was made. However, in version 66, as per our discussion I kept socket.io running in background.

[jekram]

@sojharo

Please send a link to the Azure article

Please describe when you say run the "sync code". We have used this term in a broad fashion.

Normally things are in pending when lost connectivity. Should we not build the logic around when network is restored

[sojharo]

Here is the link:

https://azure.microsoft.com/en-us/documentation/articles/notification-hubs-android-push-notification-google-fcm-get-started/

Yes, we run the upward sync too at start of application which sends pending messages to the server or pending message status to server.

When we lose connectivity, we have the logic to start sync when network is restored.

[sojharo]

We would close this issue and carry on the remaining work from this https://github.com/Cloudkibo/Android/issues/208

[sojharo]

removing the label resolved.

[Assadbintahir]

I tested it today.

Steps:

1- First I disconnected the internet on Device1. 2- Sent 10 test messages from Device1 to Device2. 3- Reconnected the internet on Device1. 4- Instantly, all the pending messages were changed to sent on Device1. 5- All messages were received on Device2.

[jekram]

@Assadbintahir

I think @sojharo was able to reproduce it. Please check with him.

The way I was testing put the phone in Airplane mode send the message and then turn off Airplane mode. Then I was able to reproduce it on both iOS and Android.

@sumairasaeed

[sojharo]

Salaam,

Assad and I tested it together after I had fixed the issue. I am publishing the next version which contains the fix. Just doing one final testing before pushing it.

On Tue, Oct 4, 2016 at 2:20 AM, Cloudkibo notifications@github.com wrote:

@Assadbintahir https://github.com/Assadbintahir

I think @sojharo https://github.com/sojharo was able to reproduce it. Please check with him.

The way I was testing put the phone in Airplane mode send the message and then turn off Airplane mode. Then I was able to reproduce it on both iOS and Android.

@sumairasaeed https://github.com/sumairasaeed

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/Cloudkibo/Android/issues/157#issuecomment-251230605, or mute the thread https://github.com/notifications/unsubscribe-auth/AFitCSWHYEkMNBC2sIeQZe3SxnnCdH_iks5qwXGIgaJpZM4JQRP0 .

Regards,

Sojharo

[jekram]

Thanks

[sojharo]

I just completed the thorough testing of use cases beside this task. I found just one small bug. It was sending pending messages on Internet coming back. However, during that sync, if we went to chat list screen and came back, it that sync twice. Receiver received the message twice. However, I have fixed that issue as well. Then did testing again. Now, the issue is resolved. I have published the 69 version.

However, the incorrect status are still there due to #204 .

[jekram]

Sojharo I do not see 69. Did it got pushed

[sojharo]

Some problem has occurred on play store. It has published both versions 68 and 69. Here is the error shown on their portal:

Let me do something

[sojharo]

I have deactivated 68 now. It is processing 69. It would be available in 30 minutes.

[jekram]

Thanks