Web - Chat disappears after being sent

kbecciv commented 1 year ago

If you haven’t already, check out our contributing guidelines for onboarding and email contributors@expensify.com to request to join our Slack channel!

Action Performed:

Open any DM with another user
Send a message
Note after a few minutes that the message is no longer there

Expected Result:

The message is sent.

Actual Result:

The message has disappeared.

Workaround:

Unknown

Platforms:

Which of our officially supported platforms is this issue occurring on?

[ ] Android / native
[ ] Android / Chrome
[ ] iOS / native
[ ] iOS / Safari
[x] MacOS / Chrome / Safari
[ ] MacOS / Desktop

Version Number: 1.3.33-4 Reproducible in staging?: n/a Reproducible in production?: n/a If this was caught during regression testing, add the test name, ID and link from TestRail: Email or phone of affected tester (no customers): Logs: https://stackoverflow.com/c/expensify/questions/4856 Notes/Photos/Videos: Any additional supporting documentation

his occurred late last night and I didn’t catch it happening to record a screenshot. I haven’t been able to reproduce it this morning, but multiple guides have reported messages disappearing after sending their customer #admins rooms. The logs from the time I attempted to send the message (19:39 UTC 27/06/23) say:

[hmmm] [PusherConnectionManager] Channels Error 1006 ~~ error: '[type: 'PusherError' data: '[code: '1006' message: 'The operation couldn't be completed. Socket is not connected']']' userAgent: 'New%20Expensify/1.3.31.3 CFNetwork/1408.0.4 Darwin/22.5.0' [hmmm] [PusherConnectionManager] WebSocketError ~~ error: '[type: 'WebSocketError' error: '[isTrusted: '' message: 'The operation couldn't be completed. Socket is not connected']']' userAgent: 'New%20Expensify/1.3.31.3 CFNetwork/1408.0.4 Darwin/22.5.0'

Expensify/Expensify Issue URL: Issue reported by: Milja Kljajic Slack conversation: https://expensify.slack.com/archives/C049HHMV9SM/p1687969003963889

View all open jobs on GitHub

Issue Owner

Current Issue Owner: @mountiny

melvin-bot[bot] commented 1 year ago

Triggered auto assignment to @kadiealexander (Bug), see https://stackoverflow.com/c/expensify/questions/14418 for more details.

melvin-bot[bot] commented 1 year ago

Bug0 Triage Checklist (Main S/O)

[ ] This "bug" occurs on a supported platform (ensure Platforms in OP are ✅)
[ ] This bug is not a duplicate report (check E/App issues and #expensify-bugs)
- If it is, comment with a link to the original report, close the issue and add any novel details to the original issue instead
[ ] This bug is reproducible using the reproduction steps in the OP. S/O
- If the reproduction steps are clear and you're unable to reproduce the bug, check with the reporter and QA first, then close the issue.
- If the reproduction steps aren't clear and you determine the correct steps, please update the OP.
[ ] This issue is filled out as thoroughly and clearly as possible
- Pay special attention to the title, results, platforms where the bug occurs, and if the bug happens on staging/production.
[ ] I have reviewed and subscribed to the linked Slack conversation to ensure Slack/Github stay in sync

kadiealexander commented 1 year ago

Have asked about steps to take to reproduce this issue here.

kadiealexander commented 1 year ago

Assigning @iwiznia to help with nailing down the reproduction of this via logs, as per this thread. Please let me know if I should approach this a different way Ionatan, I'm a bit stumped on how to reliably reproduce this!

iwiznia commented 1 year ago

ok device logs are here https://www.expensify.com/_devportal/tools/logSearch/#query=blob%3A%22AuthenticatePusher%22%20AND%20email%3A%22k.alexander%40expensify.com%22%20AND%20timestamp%3A%5B2023-07-03T00%3A00%20TO%202023-07-06T23%3A59%5D&index=_all You can see it calling AuthenticatePusher over and over Then if you look at the logs from the server, you see that it fails with 407 authToken expired: https://www.expensify.com/_devportal/tools/logSearch/#query=request_id%3A(%227e0c993709e5ee9e-SJC%22)%20AND%20timestamp%3A%5B2023-07-03T04%3A06%3A19.503Z%20TO%202023-07-03T06%3A06%3A19.503Z%5D&index=_all I am assuming pusher calls go through a different flow that do not handle authToken expiring properly... need to investigate a bit more.

melvin-bot[bot] commented 1 year ago

@iwiznia, @kadiealexander Eep! 4 days overdue now. Issues have feelings too...

melvin-bot[bot] commented 1 year ago

@iwiznia, @kadiealexander 6 days overdue. This is scarier than being forced to listen to Vogon poetry!

iwiznia commented 1 year ago

Fuck I totally forgot about this, will try to work on it or ask someone to take it

melvin-bot[bot] commented 1 year ago

@iwiznia @kadiealexander this issue was created 2 weeks ago. Are we close to a solution? Let's make sure we're treating this as a top priority. Don't hesitate to create a thread in #expensify-open-source to align faster in real time. Thanks!

melvin-bot[bot] commented 1 year ago

@iwiznia, @kadiealexander Eep! 4 days overdue now. Issues have feelings too...

kadiealexander commented 1 year ago

@iwiznia any thoughts on this one? Should we find someone else who might have some bandwidth?

iwiznia commented 1 year ago

I actually can't reproduce the error you were getting. I tried forcing authenticate pusher to return 407 but that is triggering the authentication correctly and not getting stuck in a loop. Long story short, I don't know what the problem was....

kadiealexander commented 1 year ago

It's not my error I just got the Bug0 assignment, I'm not able to reproduce this either.

@miljakljajic is this still happening for you?

melvin-bot[bot] commented 1 year ago

@iwiznia @kadiealexander this issue is now 3 weeks old. There is one more week left before this issue breaks WAQ. What needs to happen to get a PR in review this week? Please create a thread in #expensify-open-source to discuss. Thanks!

miljakljajic commented 1 year ago

I think it's related to having a ton (thousands) of newDot policies which is why you're struggling to reproduce it. I think we discussed having Applause test these types of wave 1 support bugs by setting up accounts with a similar number of policies?

iwiznia commented 1 year ago

Can you reproduce the issue now @miljakljajic or was it a one off?

miljakljajic commented 1 year ago

Not for a couple of weeks at least but I've asked in #guides as I wasn't the only person experiencing this issue.

miljakljajic commented 1 year ago

Stilll reproducible for at least three of our Guides:

Describe the exact steps you took to arrive at the issue: I sent a message to my user yesterday in which he stated he did not receive. I can see the message in public mode however, I do not see the message in the incognito mode. Now it's stating I do not have access to the chat via incognito. Your email: elle.jackson@team.expensify.com Platform (e.g. desktop app, chrome, iOS, android): Desktop App version: v1.3.43-7 Report ID: 3975761511991100 Screenshots of what you’re seeing in the app & Console - image (42) image (41) image (40) image (39)

iwiznia commented 1 year ago

Damn, I am very lost on the issue. More investigation is needed, but I am going on sabbatical, so please reassign another engineer

melvin-bot[bot] commented 1 year ago

Triggered auto assignment to @mountiny (Engineering), see https://stackoverflow.com/c/expensify/questions/4319 for more details.

kadiealexander commented 1 year ago

Not overdue! @mountiny we have a weird issue affecting our Guides teams, and it's almost impossible to reproduce reliably. Could you please look into this and see if there's anything we can do (such as adding logs) to narrow down this bug?

mountiny commented 1 year ago

From what I could skim over I think we should wait for the reliable updates to be in place and also the onyx improvememnts Margelo will plan to add, seems like really niche bug, combination of potentially missed pusher updates and some flakiness in Onyx due to high amount of stuff to store which margelo is looking into

miljakljajic commented 1 year ago

Niche as in rare? Because it has affected at least 4 people that we know of, and probably more people who never noticed that the message wasn't sent.

mountiny commented 1 year ago

@miljakljajic Sorry, yes, niche in a sense that its rare and its not clear how to reproduce reliably. We can wait on the reliable updates and onyx improvements to land in app and see if this will still be experienced by some.

The wave1 improvements should also make this better for guides so I would prefer to wait before spending too much time on this if it will self resolve.

kadiealexander commented 1 year ago

Still on hold.

mountiny commented 1 year ago

We are close but still on hold

miljakljajic commented 1 year ago

Happened again this week - I responded to this customer's message on the 21st on web, but the message didn't send.

WhatsApp Image 2023-08-24 at 09 47 06

mountiny commented 1 year ago

@miljakljajic has it been only that one particular message and you were able to successfully send messages to other users around the same time?

miljakljajic commented 1 year ago

I actually can't remember which other messages I sent around the same time to confirm. It's possible this was the only message, because I use email as much as possible now.

mountiny commented 1 year ago

I remember there was the issue with queued onyx updates being stuck when you got malformed response from server or you have changed your name for example and because you know so many people the request timed out but it just kept retrying indefinitely blocking other request (like sending a message from happening). Seems like this could be the case too, for John Schuster too it happened when his timezone changed

miljakljajic commented 1 year ago

Ah, I see. Strange because I haven't changed timezone, or name, this week...

miljakljajic commented 1 year ago

I just noticed another instance of this - a message that I sent to dk@oci-group.co.uk on Aug 18th also didn't send:

miljakljajic commented 1 year ago

I sent my 10am around 7:30 this morning and again the message hasn't gone through:

mountiny commented 1 year ago

Reliable updates are almost out, I think this might also be related to just large accounts and bad performance.

mountiny commented 1 year ago

This still doesn't not have a clear reproduction steps

mountiny commented 11 months ago

I have not experienced this and I dont think we had any more widespread or helpful information for how to reproduce.

I thin I would prefer to close this issue in sake of clean ing up plate and focusing on current strategic goals.

Feel free to reopen if this is reproducible

Expensify / App