HIGH: [Reliability] Background sync issue - combination of API request latency and closing client before HTTPS receives API response causes all clients to fall out of sync

muttmuure commented 6 months ago

If you haven’t already, check out our contributing guidelines for onboarding and email contributors@expensify.com to request to join our Slack channel!

Version Number: v1.4.49-3 Reproducible in staging?: Yes Reproducible in production?: Yes If this was caught during regression testing, add the test name, ID and link from TestRail: Email or phone of affected tester (no customers): mattmo2025@gmail.com Logs: https://stackoverflow.com/c/expensify/questions/4856 Expensify/Expensify Issue URL: Issue reported by: @muttmuure Slack conversation: https://expensify.slack.com/archives/C0671JDRKQW/p1708058905414929

Action Performed:

Have the same account open with two devices (one web, one mobile)
Enable client-side logging
Kill the app on the mobile device
Go offline on the mobile device
Open the network queue on your web device
Send messages until you've reached 100 requests in the network queue to create latency between HTTPS data and API requests
Close app on web device
Reopen app on mobile device
Reopen app on web device

Expected Result:

All of the onyx updates shown locally on the web client will (eventually) sync to the mobile client

Actual Result:

It seems like the latency of the HTTPS request being converted to an API request and then closing the web client, causes the Onyx data on the mobile client to fall out of sync with the web client

We do not have a retry mechanism either on the web client or the mobile client:

On the web client, upon reopening the app, we do not retrieve older HTTPS requests and retry turning them into API requests
On the mobile client, we log that 'Update received was older than or the same as current state has received an update that is older than the current state' which means that the app stops looking for further updates

This causes a synchronization problem from one client to another, one devices LHN and message history appears different to another.

Workaround:

It resolves itself in a piecemeal fashion (chat report by chat report) if you engage with that individual chat report. The mobile client detects that there are missing updates by calling GetMissingOnyxUpdates for that individual chat report.

For realtime chat this doesn't have a workaround.

Platforms:

Which of our officially supported platforms is this issue occurring on?

[X] Android: Native
[X] Android: mWeb Chrome
[X] iOS: Native
[X] iOS: mWeb Safari
[X] MacOS: Chrome / Safari
[X] MacOS: Desktop

Screenshots/Videos

Add any screenshot/video evidence

View all open jobs on GitHub

{ "time": "2024-03-09T14:01:46.301Z", "level": "DEBUG", "message": "[info] [OnyxUpdateManager] Applying success or failure data from request without onyxData from response - \"\"" }, { "time": "2024-03-09T14:01:46.299Z", "level": "DEBUG", "message": "[info] [OnyxUpdateManager] Update received was older than or the same as current state, returning without applying the updates other than successData and failureData - \"\"" }, { "time": "2024-03-09T14:01:46.298Z", "level": "DEBUG", "message": "[info] Finished API request - {\"command\":\"ReconnectApp\",\"jsonCode\":200,\"requestID\":\"861b994d5ba56401-LHR\"}" },

melvin-bot[bot] commented 6 months ago

Triggered auto assignment to @stephanieelliott (Bug), see https://stackoverflow.com/c/expensify/questions/14418 for more details.

melvin-bot[bot] commented 6 months ago

@muttmuure Uh oh! This issue is overdue by 2 days. Don't forget to update your issues!

melvin-bot[bot] commented 6 months ago

@muttmuure Eep! 4 days overdue now. Issues have feelings too...

danieldoglas commented 6 months ago

No updates on this one yet, need to finish other tasks before I can look into it

danieldoglas commented 5 months ago

I think this will be solved with https://github.com/Expensify/Expensify/issues/381960, so I'll move this to weekly.

Expensify / App