[SMARTSCAN] [$500] MEDIUM: Update the workspace chat report preview component when a receipt scan completes

[trjExpensify]

If you haven’t already, check out our contributing guidelines for onboarding and email contributors@expensify.com to request to join our Slack channel!

---

Action Performed:

Prerequisite: Use the example receipt provided below, because it will likely hit an image parser and complete super fast to test this.

1. Create a workspace on staging.new.expensify.com
2. Invite a member to it
3. As the member, sign-in and navigate to your workspace chat on the workspace
4. Upload a receipt
5. Observe the "Receipt scan in progress..." report preview component
6. Give it 15-30s or so to finish scanning
7. Navigate away from the workspace chat (i.e tap < to the LHN or click the report preview component)
8. Navigate back to the workspace chat

Expected Result:

The report preview component should update with the receipt data automatically without needing to navigate away and back to the chat

Actual Result:

The report preview component does not update

Workaround:

Yes, as described above to navigate away and back.

Platforms:

Which of our officially supported platforms is this issue occurring on?

• [ ] Android / native
• [ ] Android / Chrome
• [x] iOS / native
• [x] iOS / Safari
• [x] MacOS / Chrome / Safari
• [ ] MacOS / Desktop

Version Number: v1.3.64-0 Reproducible in staging?: Y Reproducible in production?: Y If this was caught during regression testing, add the test name, ID and link from TestRail: Email or phone of affected tester (no customers): Logs: https://stackoverflow.com/c/expensify/questions/4856 Notes/Photos/Videos:

mWeb safari reproduction: https://github.com/Expensify/App/assets/16232057/019912f4-8c89-415f-92c6-8f1f363a033e

iOS native app reproduction: https://github.com/Expensify/App/assets/16232057/2fa42280-be97-49d9-a62a-5de4699bf94c

Mac Chrome reproduction: https://github.com/Expensify/App/assets/16232057/13172aa7-0abe-435e-aa24-8e25e9225077

Example receipt to use:

Expensify/Expensify Issue URL: Issue reported by: @trjExpensify Slack conversation: Internal

View all open jobs on GitHub

CC: @Gonals @MariaHCD @JmillsExpensify @dylanexpensify

Upwork Automation - Do Not Edit

• Upwork Job URL: https://www.upwork.com/jobs/~01eae90794b2b26fd4
• Upwork Job ID: 1699178450385817600
• Last Price Increase: 2023-10-02

Issue Owner

Current Issue Owner: @MariaHCD

[melvin-bot[bot]]

Current assignee @trjExpensify is eligible for the Bug assigner, not assigning anyone new.

[melvin-bot[bot]]

Job added to Upwork: https://www.upwork.com/jobs/~01eae90794b2b26fd4

[melvin-bot[bot]]

Bug0 Triage Checklist (Main S/O)

• [x] This "bug" occurs on a supported platform (ensure Platforms in OP are ✅)
• [x] This bug is not a duplicate report (check E/App issues and #expensify-bugs)
  • If it is, comment with a link to the original report, close the issue and add any novel details to the original issue instead
• [x] This bug is reproducible using the reproduction steps in the OP. S/O
  • If the reproduction steps are clear and you're unable to reproduce the bug, check with the reporter and QA first, then close the issue.
  • If the reproduction steps aren't clear and you determine the correct steps, please update the OP.
• [x] This issue is filled out as thoroughly and clearly as possible
  • Pay special attention to the title, results, platforms where the bug occurs, and if the bug happens on staging/production.
• [x] I have reviewed and subscribed to the linked Slack conversation to ensure Slack/Github stay in sync

[melvin-bot[bot]]

Current assignee @trjExpensify is eligible for the External assigner, not assigning anyone new.

[melvin-bot[bot]]

Triggered auto assignment to Contributor-plus team member for initial proposal review - @0xmiroslav (External)

[s-alves10]

This looks like a backend issue When money request with receipt is finished, it returns the transaction with {receipt: {state: SCANREADY} When scan is completed, this state is changed to SCANCOMPLETE. We need to receive pusher event for this scan status change, but no pusher event is received. This is the root cause

[imranaalam]

Problem Statement:
Users who upload a receipt to a workspace chat encounter an issue where the "Receipt scan in progress..." preview doesn't automatically update to display the receipt data after the scan completes. The expected behavior is for this update to happen automatically, but users currently have to navigate away and return to the chat to see the updated data.

Root Cause:

1. Pusher Event Absence: The application is designed to await a Pusher event to signal when the scan status transitions to SCANCOMPLETE. Our investigation suggests that this Pusher event isn't being received as anticipated.
2. State Management & Rendering: The ReportPreview component is closely tied to the Onyx state. If there's an inconsistency in how Onyx updates its state or if the component doesn't re-render in response to state changes, it can lead to the observed issue. The sections of code, especially those using isReceiptBeingScanned() and other related utilities, play a crucial role in this process.

Proposed Solution:

1. Pusher Event Logging: Establish comprehensive logging for all incoming Pusher events. This action will confirm whether the SCANCOMPLETE event is getting lost in transit or not being sent at all.
2. Backend Verification: Delve deeper into the backend, ensuring that it's programmed to send the SCANCOMPLETE event. The backend should be reviewed to ascertain that the event emission process is flawless.
3. Onyx State Checks: Scrutinize the sections of the codebase where the Onyx state related to receipt scanning is adjusted. This scrutiny will ensure that the state updates appropriately after each receipt scan.
4. Forced Component Refresh: As a diagnostic tool, introduce a mechanism, even if temporary, to force the ReportPreview component to refresh at regular intervals. This approach can help clarify if the issue lies in the component's re-rendering mechanism or in the state management/data flow.

   
   const [, forceUpdate] = useReducer(x => x + 1, 0);

useEffect(() => { const interval = setInterval(() => { forceUpdate(); }, 5000);

return () => clearInterval(interval);

}, []);

**Alternative Solution**:  
Should the Pusher event issue be confirmed, contemplate the initiation of a polling mechanism. This mechanism will carry out periodic checks on the receipt's state until its status reaches `SCANCOMPLETE`. While this might introduce some performance overhead compared to the real-time push mechanism, it can act as a temporary fix. However, the optimal long-term solution would be to rectify the Pusher event's emission and reception process.

[MonstaBug]

The problem is scanning is async (on the backend) and we don't know when the scanning has been completed. It worked when you switch between reports/chat because it refetchs the report data and then knows it has been completed.

So we either do polling or use pusher

[melvin-bot[bot]]

📣 @MonstaBug! 📣 Hey, it seems we don’t have your contributor details yet! You'll only have to do this once, and this is how we'll hire you on Upwork. Please follow these steps:

1. Get the email address used to login to your Expensify account. If you don't already have an Expensify account, create one here. If you have multiple accounts (e.g. one for testing), please use your main account email.
2. Get the link to your Upwork profile. It's necessary because we only pay via Upwork. You can access it by logging in, and then clicking on your name. It'll look like this. If you don't already have an account, sign up for one here.
3. Copy the format below and paste it in a comment on this issue. Replace the placeholder text with your actual details. Format:

   Contributor details
   Your Expensify account email: <REPLACE EMAIL HERE>
   Upwork Profile Link: <REPLACE LINK HERE>

[melvin-bot[bot]]

@trjExpensify, @0xmiroslav Huh... This is 4 days overdue. Who can take care of this?

[melvin-bot[bot]]

@trjExpensify, @0xmiroslav Huh... This is 4 days overdue. Who can take care of this?

[trjExpensify]

@0xmiroslav @Gonals @MariaHCD, what do you think about this one. Should we take it internal?

[melvin-bot[bot]]

📣 It's been a week! Do we have any satisfactory proposals yet? Do we need to adjust the bounty for this issue? 💸

[MariaHCD]

Hmm, we already have a pusher event in the backend to update the transaction data in Onyx once the receipt scan is complete here

Can we get the receiptID, perhaps there's something else we can find via the logs

[trjExpensify]

Sure! Here's a fresh one with receiptID: 657488558

https://github.com/Expensify/App/assets/16232057/536f8afa-33bd-48d4-b205-28a0fb7268ed

[imranaalam]

Problem Statement:

Users who upload a receipt to a workspace chat encounter an issue where the "Receipt scan in progress..." preview doesn't automatically update to display the receipt data after the scan completes. The expected behavior is for this update to happen automatically, but users currently have to navigate away and return to the chat to see the updated data.

Root Cause:

1. State Management & Reactivity: The Onyx.connect mechanism, which establishes a reactive connection between the Onyx data store and the components or utilities, might be experiencing delays or inconsistencies in its callbacks. This can lead to outdated or incorrect data in the allTransactions variable.
2. Data Retrieval & Integrity: The getLinkedTransaction function relies on accurate and timely data from the allTransactions collection. If there's an absence of the IOUTransactionID or inconsistencies in the allTransactions data, it can impact the UI updates.
3. Receipt Scanning State Analysis: The isReceiptBeingScanned function may not be accurately reflecting the real-time state of receipt scanning due to backend inconsistencies or Onyx update delays.

Proposed Solution:

1. Backend/Data Source Verification: Ensure that data updates related to transactions are timely and accurate in the backend. Verify the flow of how and when ONYXKEYS.COLLECTION.TRANSACTION gets updated.
2. Frontend/UI Examination: Identify components that rely on the utility functions discussed and ensure they react appropriately to data changes. Implement additional logging to capture discrepancies in data flow or UI updates.
3. Enhanced Error Handling: Strengthen error handling in both utilities and components to capture any missing or inconsistent data. This can prevent minor issues from escalating and provide clearer insights when problems arise.
4. User Feedback Mechanism: Implement UI feedback mechanisms to inform users about potential delays in scanning or if manual intervention is required.

Alternative Solution:

Should the root cause be deeply embedded in the state management system or if there are significant delays in data updates, consider introducing a polling mechanism. This mechanism will periodically check the state of the receipt until its status reaches SCANCOMPLETE. While this might introduce some performance overhead, it can act as a temporary fix until the core issues with Onyx's reactivity or backend data updates are addressed.

[imranaalam]

Problem Statement:

Users who upload a receipt to a workspace chat encounter an issue where the "Receipt scan in progress..." preview doesn't automatically update to display the receipt data after the scan completes. Users currently have to navigate away and then return to the chat to see the updated data.

Root Cause:

1. Linked Transaction Retrieval: The function TransactionUtils.getLinkedTransaction is pivotal for fetching the transaction details. If this function encounters delays or inconsistencies in retrieving transaction details, it can lead to discrepancies in the UI.
2. State Management & Memoization: The OptionRowLHNData component uses memoization to optimize rendering. If the variables it depends on, like fullReport or linkedTransaction, don't update correctly, the memoized value may not reflect the most recent state.
3. Scanning State Determination: The function TransactionUtils.isReceiptBeingScanned determines if a receipt is in the scanning process. If this function's logic isn't accurate or if the data it depends on isn't updated in real-time, it could lead to the observed issue.

Proposed Solution:

1. Data Verification: Thoroughly inspect the TransactionUtils.getLinkedTransaction and ReportActionsUtils.getParentReportAction functions to ensure they always fetch accurate data.
2. State Management Review: In the OptionRowLHNData component, review the memoization logic. Ensure that all dependent variables are updating correctly. Introduce logging mechanisms to track these updates.
3. Scanning State Check: Deep dive into the TransactionUtils.isReceiptBeingScanned function. Ensure its logic correctly determines the scanning state. Verify where and how the data, determining the scanning state, gets updated.
4. Enhanced Error Handling: Strengthen error handling mechanisms. Implement comprehensive logging to capture any missing or inconsistent data.

Alternative Solution:

1. State Management Overhaul: If the root cause is deeply embedded in the state management system or the memoization logic, consider revamping the state management strategy. This could involve moving away from memoization or using a different state management library.
2. Backend Polling: If real-time updates prove challenging, contemplate initiating a polling mechanism. This mechanism will periodically check the receipt's state until its status is SCANCOMPLETE. This can act as a temporary solution until the core issues are addressed.

[0xmiros]

@imranaalam I see 3 proposals from you. Can you please remove 2 and consolidate into one?

[0xmiros]

Awaiting better proposals

[dylanexpensify]

@MariaHCD were you able to find anything from @trjExpensify's post?

[s-alves10]

I verified again that frontend doesn't receive the pusher event. Can anyone check this?

[MariaHCD]

Checking the logs from @trjExpensify's test:

SIkoAQ | virt2.sjc | 2023-09-13 11:42:31 939 | tom@expensify.com | Hit - Updating transaction after applying rules ~~ transactionID: '2584992638075293377'
SIkoAQ | virt2.sjc | 2023-09-13 11:42:31 939 | tom@expensify.com | Hit - Updating transaction on a free policy expense report or IOU ~~ transactionID: '2584992638075293377'
SIkoAQ | virt2.sjc | 2023-09-13 11:42:31 939 | tom@expensify.com | Bedrock\Client - Starting a request ~~ command: 'EditMoneyRequest' clusterName: 'auth' headers: '[authToken: '<REDACTED>' transactionID: '2584992638075293377' modifiedExpenseReportActionID: '0' created: '2023-09-07' amount: '2000' currency: 'USD' comment: '' merchant: 'SaaStr Cantina' idempotent: '1' commitCount: '20765590422' requestID: 'SIkoAQ' lastIP: '' writeConsistency: 'ASYNC' priority: '500' timeout: '290000' logParam: 'we@dont.know']'

Looks like we definitely sent the data for the transactions key via pusher:

SIkoAQ | virt2.sjc | 2023-09-13 11:42:32 233 | tom@expensify.com | Expensify\Onyx - Sending command response via Pusher ~~ 0: '[onyxMethod: 'merge' key: 'transactions_2584992638075293377']'

SIkoAQ | virt2.sjc | 2023-09-13 11:42:32 349 | tom@expensify.com | Bedrock\Client - Starting a request ~~ command: 'InsertOnyxUpdates' clusterName: 'auth' headers: '[onyxData: '[0: '[updates: '[0: '[onyxMethod: 'merge' key: 'transactions_2584992638075293377' value: '[amount: '-2000' date: '2023-09-07' merchant: 'SaaStr Cantina' currency: 'USD' receipt: '[receiptID: '657488558' source: 'https://www.expensify.com/receipts/w_9c37363057a594c401dd7fb0d87ad4e971d0ec27.png' state: 'SCANCOMPLETE']']']']' accounts: '[0: '281396']']']' idempotent: '1' logParam: 'tom@expensify.com' commitCount: '20765590473' requestID: 'SIkoAQ' lastIP: '127.0.0.1' writeConsistency: 'ASYNC' priority: '500' timeout: '290000']'

SIkoAQ | virt2.sjc | 2023-09-13 11:42:32 391 | tom@expensify.com | [push] Push_Service - Sending chunked event ~~ name: 'multipleEvents' channel: 'private-encrypted-user-accountID-281396' data: '[lastUpdateID: '5441835' previousUpdateID: '5441832' updates: '[0: '[data: '[0: '[key: 'transactions_2584992638075293377' onyxMethod: 'merge' value: '[amount: '-2000' currency: 'USD' date: '2023-09-07' merchant: 'SaaStr Cantina' receipt: '[receiptID: '657488558' source: 'https://www.expensify.com/receipts/w_9c37363057a594c401dd7fb0d87ad4e971d0ec27.png' state: 'SCANCOMPLETE']']']']' eventType: 'onyxApiUpdate']']']'

[MariaHCD]

I verified again that frontend doesn't receive the pusher event.

@s-alves10 how are you verifying that the frontend doesn't receive the pusher event?

[s-alves10]

I verified by log

[s-alves10]

@MariaHCD

Will you check if the accountID is correct from the log?

[MariaHCD]

Just checked, the accountID in the channel we're sending to (channel: 'private-encrypted-user-accountID-281396' in https://github.com/Expensify/App/issues/26830#issuecomment-1718855492) definitely belongs to @trjExpensify

[MariaHCD]

Wasn't able to reproduce the issue on MacOS/Chrome on staging:

https://github.com/Expensify/App/assets/12268372/c7a9d326-2ac4-4c6a-aa5c-feb1754d4150

Maybe this issue is more obvious on native mobile? 🤔

@s-alves10 Which platforms have you tested on?

[trjExpensify]

Wasn't able to reproduce the issue on MacOS/Chrome on staging:

In the video of yours @MariaHCD you navigate to the expense report to wait for it to complete. If you stay on the workspace chat for a bit the report preview component should update but, but you'll notice in mine I have to navigate from it to get it to update. Here it is on MacOS Chrome:

https://github.com/Expensify/App/assets/16232057/d7333f04-4a3e-4710-aedf-f19eef5107b4

[0xmiros]

100% reproducible on latest main

https://github.com/Expensify/App/assets/97473779/8de8acb2-4d4a-4675-b18d-1ac82150ff4c

This doesn't depend on platforms

[Gonals]

It is not immediate, as parsing takes a little bit (more on staging/prod, as there are, potentially, many receipts, each of which creates a job with competing priorities), but this is working fine for me on dev, and the component is updated a few seconds after the job is completed, which is the expected behavior 🤷

[MariaHCD]

If you stay on the workspace chat for a bit the report preview component should update

Oh, interesting, I tested again and remained on the workspace chat. Definitely saw that it did not update the report preview 🤔

https://github.com/Expensify/App/assets/12268372/b9ec5fb9-4bdb-4a01-b496-cde34297bf62

@s-alves10 @0xmiroslav Can you double check if the transaction in Onyx is being updated with the receipt details as expected? Because we're definitely sending the transaction data via pusher in the backend and our logs do show the same as well. I wonder if it is the App that isn't rendering the updated transaction in the ReportPreview component here?

[s-alves10]

@MariaHCD

No, the frontend doesn't receive the pusher events and so onyx transaction is not updated.

[MariaHCD]

Would you mind giving me the email address that you tested with?

[s-alves10]

I just ran a test with sangar.work1028+009@gmail.com

[MariaHCD]

We definitely sent the pusher data for the Onyx key transactions_1078602410102294240 🤔 @s-alves10 can you confirm that was the Onyx data you checked?

x3T6ok | virt1.rno | 2023-09-15 07:26:43 237 | sangar.work1028+009@gmail.com | [push] Push_Service - Sending event ~~ channel: '[0: 'private-encrypted-user-accountID-14942190' 1: 'private-encrypted-user-accountID-14868802']' eventName: 'onyxApiUpdate' eventData: '[{"onyxMethod":"merge","key":"transactions_1078602410102294240","value":{"amount":-2000,"date":"2023-09-04","merchant":"SaaStr Cantina","currency":"USD","receipt":{"receiptID":658015732,"source":"https:\/\/www.expensify.com\/receipts\/w_f7f7171539b192e2ab2d0717e2fb9ee076681237.png","state":"SCANCOMPLETE"}}}]'

x3T6ok | virt1.rno | 2023-09-15 07:26:43 237 | sangar.work1028+009@gmail.com | [push] Push_Service - Sending chunked event ~~ channel: '[0: 'private-encrypted-user-accountID-14942190' 1: 'private-encrypted-user-accountID-14868802']' eventName: 'onyxApiUpdate' eventData: '[{"onyxMethod":"merge","key":"transactions_1078602410102294240","value":{"amount":-2000,"date":"2023-09-04","merchant":"SaaStr Cantina","currency":"USD","receipt":{"receiptID":658015732,"source":"https:\/\/www.expensify.com\/receipts\/w_f7f7171539b192e2ab2d0717e2fb9ee076681237.png","state":"SCANCOMPLETE"}}}]' chunkSize: '1'

The data we sent:

[
   {
      "onyxMethod":"merge",
      "key":"transactions_1078602410102294240",
      "value":{
         "amount":-2000,
         "date":"2023-09-04",
         "merchant":"SaaStr Cantina",
         "currency":"USD",
         "receipt":{
            "receiptID":658015732,
            "source":"https:\/\/www.expensify.com\/receipts\/w_f7f7171539b192e2ab2d0717e2fb9ee076681237.png",
            "state":"SCANCOMPLETE"
         }
      }
   }
]

[s-alves10]

Let me check again

[s-alves10]

Really strange. Frontend doesn't receive any events and transaction receipt status isn't updated. The frontend does receive other events like message

[MariaHCD]

Yeah, definitely weird. I'll try pointing my dev App to the staging API and see if the issue still occurs.

[melvin-bot[bot]]

@trjExpensify, @0xmiroslav Whoops! This issue is 2 days overdue. Let's get this updated quick!

[trjExpensify]

Thanks, @MariaHCD!

[melvin-bot[bot]]

@MariaHCD @trjExpensify @0xmiroslav this issue was created 2 weeks ago. Are we close to approving a proposal? If not, what's blocking us from getting this issue assigned? Don't hesitate to create a thread in #expensify-open-source to align faster in real time. Thanks!

[MariaHCD]

Focused on https://github.com/Expensify/Expensify/issues/316043 today. Getting to this tomorrow.

[melvin-bot[bot]]

📣 It's been a week! Do we have any satisfactory proposals yet? Do we need to adjust the bounty for this issue? 💸

[MariaHCD]

Tested the app locally while pointing to the staging API. And I do think there is some discrepancy between the Onyx data sent via Pusher after the scan completes and the Onyx data that is set once the report is opened 🤔 Specifically for the amount, merchant and currency fields.

Left: transaction Onyx data sent via Pusher after the scan is complete, Right: transaction Onyx data once OpenReport is called:

Internal logs:

[updates: '[0: '[data: '[0: '[key: 'transactions_8138414820431815831' onyxMethod: 'merge' value: '[amount: '-2000' currency: 'USD' date: '2023-09-08' merchant: 'SaaStr Cantina (E)' receipt: '[receiptID: '659128384' source: 'https://www.expensify.com/receipts/w_75fe0ca9dfa7c03ca8c9f56da25bac00ec2bc099.png' state: 'SCANCOMPLETE']']']']' eventType: 'onyxApiUpdate']']' lastUpdateID: '14726669' previousUpdateID: '14726358']

[MariaHCD]

@Gonals once the scan is complete and when we update the transactions Onyx key here, shouldn't we be setting the modifiedAmount, modifiedCurrency, modifiedMerchant fields instead of the amount, currency and merchant fields?

[MariaHCD]

So two things potentially:

1. In the API, we should be setting the modifiedAmount, modifiedCurrency, modifiedMerchant fields here
2. We might also need to update the total field in the report Onyx key since we use that in the report preview here so it seems like the amount doesn't update after the scan completes because we don't send the updated total in the Onyx data here

Going to continue testing/investigating tomorrow.

[MariaHCD]

Got a draft backend PR in place, I'll do some more testing.

[MariaHCD]

Pushed the backend PR out for review. We can also test it on staging once it is deployed.

[MariaHCD]

Backend PR was deployed. I tested on staging and the report preview is updated after the scan is complete:

https://github.com/Expensify/App/assets/12268372/9f30db92-c520-4a1a-b213-2755e7ab040e

However, I noticed two things:

1. The updated transaction details are actually pushed to Onyx pretty quickly but the changes take a while to show in the report preview. Could it be that the ReportPreview component isn't updating when the new transaction details are available? cc: @0xmiroslav

2. The report total was sent by the API as 0 so the report preview isn't showing the correct amount. Looking into this one. (internal logs)

ffyqIs | virt1.sjc | 2023-10-02 04:36:50 561 | maria+2@expensifail.com | [push] Push_Service - Sending chunked event ~~ name: 'multipleEvents' channels: '[0: 'private-encrypted-user-accountID-8910926']' data: '[updates: '[0: '[data: '[0: '[key: 'transactions_7076274334862224452' onyxMethod: 'merge' value: '[modifiedAmount: '-2000' modifiedCreated: '2023-09-08' modifiedCurrency: 'USD' modifiedMerchant: 'SaaStr Cantina (E)' receipt: '[receiptID: '662139566' source: 'https://www.expensify.com/receipts/w_74b4d517c431235930da8283113164f8deefbabd.png' state: 'SCANCOMPLETE']']']' 1: '[key: 'report_3732056035654371' onyxMethod: 'merge' value: '[total: '0']']']' eventType: 'onyxApiUpdate']']' lastUpdateID: '19261406' previousUpdateID: '19261404']'

[melvin-bot[bot]]

📣 It's been a week! Do we have any satisfactory proposals yet? Do we need to adjust the bounty for this issue? 💸