Closed trjExpensify closed 8 months ago
Current assignee @trjExpensify is eligible for the Bug assigner, not assigning anyone new.
Job added to Upwork: https://www.upwork.com/jobs/~01eae90794b2b26fd4
Platforms
in OP are ✅)Current assignee @trjExpensify is eligible for the External assigner, not assigning anyone new.
Triggered auto assignment to Contributor-plus team member for initial proposal review - @0xmiroslav (External
)
This looks like a backend issue
When money request with receipt is finished, it returns the transaction with {receipt: {state: SCANREADY}
When scan is completed, this state is changed to SCANCOMPLETE
. We need to receive pusher event for this scan status change, but no pusher event is received.
This is the root cause
Problem Statement:
Users who upload a receipt to a workspace chat encounter an issue where the "Receipt scan in progress..." preview doesn't automatically update to display the receipt data after the scan completes. The expected behavior is for this update to happen automatically, but users currently have to navigate away and return to the chat to see the updated data.
Root Cause:
SCANCOMPLETE
. Our investigation suggests that this Pusher event isn't being received as anticipated.ReportPreview
component is closely tied to the Onyx state. If there's an inconsistency in how Onyx updates its state or if the component doesn't re-render in response to state changes, it can lead to the observed issue. The sections of code, especially those using isReceiptBeingScanned()
and other related utilities, play a crucial role in this process.Proposed Solution:
SCANCOMPLETE
event is getting lost in transit or not being sent at all.SCANCOMPLETE
event. The backend should be reviewed to ascertain that the event emission process is flawless.ReportPreview
component to refresh at regular intervals. This approach can help clarify if the issue lies in the component's re-rendering mechanism or in the state management/data flow.
const [, forceUpdate] = useReducer(x => x + 1, 0);
useEffect(() => { const interval = setInterval(() => { forceUpdate(); }, 5000);
return () => clearInterval(interval);
}, []);
**Alternative Solution**:
Should the Pusher event issue be confirmed, contemplate the initiation of a polling mechanism. This mechanism will carry out periodic checks on the receipt's state until its status reaches `SCANCOMPLETE`. While this might introduce some performance overhead compared to the real-time push mechanism, it can act as a temporary fix. However, the optimal long-term solution would be to rectify the Pusher event's emission and reception process.
The problem is scanning is async (on the backend) and we don't know when the scanning has been completed. It worked when you switch between reports/chat because it refetchs the report data and then knows it has been completed.
So we either do polling or use pusher
📣 @MonstaBug! 📣 Hey, it seems we don’t have your contributor details yet! You'll only have to do this once, and this is how we'll hire you on Upwork. Please follow these steps:
Contributor details
Your Expensify account email: <REPLACE EMAIL HERE>
Upwork Profile Link: <REPLACE LINK HERE>
@trjExpensify, @0xmiroslav Huh... This is 4 days overdue. Who can take care of this?
@trjExpensify, @0xmiroslav Huh... This is 4 days overdue. Who can take care of this?
@0xmiroslav @Gonals @MariaHCD, what do you think about this one. Should we take it internal?
📣 It's been a week! Do we have any satisfactory proposals yet? Do we need to adjust the bounty for this issue? 💸
Hmm, we already have a pusher event in the backend to update the transaction data in Onyx once the receipt scan is complete here
Can we get the receiptID, perhaps there's something else we can find via the logs
Sure! Here's a fresh one with receiptID: 657488558
https://github.com/Expensify/App/assets/16232057/536f8afa-33bd-48d4-b205-28a0fb7268ed
Users who upload a receipt to a workspace chat encounter an issue where the "Receipt scan in progress..." preview doesn't automatically update to display the receipt data after the scan completes. The expected behavior is for this update to happen automatically, but users currently have to navigate away and return to the chat to see the updated data.
allTransactions
variable.getLinkedTransaction
function relies on accurate and timely data from the allTransactions
collection. If there's an absence of the IOUTransactionID
or inconsistencies in the allTransactions
data, it can impact the UI updates.isReceiptBeingScanned
function may not be accurately reflecting the real-time state of receipt scanning due to backend inconsistencies or Onyx update delays.ONYXKEYS.COLLECTION.TRANSACTION
gets updated.Should the root cause be deeply embedded in the state management system or if there are significant delays in data updates, consider introducing a polling mechanism. This mechanism will periodically check the state of the receipt until its status reaches SCANCOMPLETE
. While this might introduce some performance overhead, it can act as a temporary fix until the core issues with Onyx's reactivity or backend data updates are addressed.
Users who upload a receipt to a workspace chat encounter an issue where the "Receipt scan in progress..." preview doesn't automatically update to display the receipt data after the scan completes. Users currently have to navigate away and then return to the chat to see the updated data.
TransactionUtils.getLinkedTransaction
is pivotal for fetching the transaction details. If this function encounters delays or inconsistencies in retrieving transaction details, it can lead to discrepancies in the UI.OptionRowLHNData
component uses memoization to optimize rendering. If the variables it depends on, like fullReport
or linkedTransaction
, don't update correctly, the memoized value may not reflect the most recent state.TransactionUtils.isReceiptBeingScanned
determines if a receipt is in the scanning process. If this function's logic isn't accurate or if the data it depends on isn't updated in real-time, it could lead to the observed issue.TransactionUtils.getLinkedTransaction
and ReportActionsUtils.getParentReportAction
functions to ensure they always fetch accurate data.OptionRowLHNData
component, review the memoization logic. Ensure that all dependent variables are updating correctly. Introduce logging mechanisms to track these updates.TransactionUtils.isReceiptBeingScanned
function. Ensure its logic correctly determines the scanning state. Verify where and how the data, determining the scanning state, gets updated.SCANCOMPLETE
. This can act as a temporary solution until the core issues are addressed.@imranaalam I see 3 proposals from you. Can you please remove 2 and consolidate into one?
Awaiting better proposals
@MariaHCD were you able to find anything from @trjExpensify's post?
I verified again that frontend doesn't receive the pusher event. Can anyone check this?
Checking the logs from @trjExpensify's test:
SIkoAQ | virt2.sjc | 2023-09-13 11:42:31 939 | tom@expensify.com | Hit - Updating transaction after applying rules ~~ transactionID: '2584992638075293377'
SIkoAQ | virt2.sjc | 2023-09-13 11:42:31 939 | tom@expensify.com | Hit - Updating transaction on a free policy expense report or IOU ~~ transactionID: '2584992638075293377'
SIkoAQ | virt2.sjc | 2023-09-13 11:42:31 939 | tom@expensify.com | Bedrock\Client - Starting a request ~~ command: 'EditMoneyRequest' clusterName: 'auth' headers: '[authToken: '<REDACTED>' transactionID: '2584992638075293377' modifiedExpenseReportActionID: '0' created: '2023-09-07' amount: '2000' currency: 'USD' comment: '' merchant: 'SaaStr Cantina' idempotent: '1' commitCount: '20765590422' requestID: 'SIkoAQ' lastIP: '' writeConsistency: 'ASYNC' priority: '500' timeout: '290000' logParam: 'we@dont.know']'
Looks like we definitely sent the data for the transactions key via pusher:
SIkoAQ | virt2.sjc | 2023-09-13 11:42:32 233 | tom@expensify.com | Expensify\Onyx - Sending command response via Pusher ~~ 0: '[onyxMethod: 'merge' key: 'transactions_2584992638075293377']'
SIkoAQ | virt2.sjc | 2023-09-13 11:42:32 349 | tom@expensify.com | Bedrock\Client - Starting a request ~~ command: 'InsertOnyxUpdates' clusterName: 'auth' headers: '[onyxData: '[0: '[updates: '[0: '[onyxMethod: 'merge' key: 'transactions_2584992638075293377' value: '[amount: '-2000' date: '2023-09-07' merchant: 'SaaStr Cantina' currency: 'USD' receipt: '[receiptID: '657488558' source: 'https://www.expensify.com/receipts/w_9c37363057a594c401dd7fb0d87ad4e971d0ec27.png' state: 'SCANCOMPLETE']']']']' accounts: '[0: '281396']']']' idempotent: '1' logParam: 'tom@expensify.com' commitCount: '20765590473' requestID: 'SIkoAQ' lastIP: '127.0.0.1' writeConsistency: 'ASYNC' priority: '500' timeout: '290000']'
SIkoAQ | virt2.sjc | 2023-09-13 11:42:32 391 | tom@expensify.com | [push] Push_Service - Sending chunked event ~~ name: 'multipleEvents' channel: 'private-encrypted-user-accountID-281396' data: '[lastUpdateID: '5441835' previousUpdateID: '5441832' updates: '[0: '[data: '[0: '[key: 'transactions_2584992638075293377' onyxMethod: 'merge' value: '[amount: '-2000' currency: 'USD' date: '2023-09-07' merchant: 'SaaStr Cantina' receipt: '[receiptID: '657488558' source: 'https://www.expensify.com/receipts/w_9c37363057a594c401dd7fb0d87ad4e971d0ec27.png' state: 'SCANCOMPLETE']']']']' eventType: 'onyxApiUpdate']']']'
I verified again that frontend doesn't receive the pusher event.
@s-alves10 how are you verifying that the frontend doesn't receive the pusher event?
I verified by log
@MariaHCD
Will you check if the accountID
is correct from the log?
Just checked, the accountID in the channel we're sending to (channel: 'private-encrypted-user-accountID-281396'
in https://github.com/Expensify/App/issues/26830#issuecomment-1718855492) definitely belongs to @trjExpensify
Wasn't able to reproduce the issue on MacOS/Chrome on staging:
https://github.com/Expensify/App/assets/12268372/c7a9d326-2ac4-4c6a-aa5c-feb1754d4150
Maybe this issue is more obvious on native mobile? 🤔
@s-alves10 Which platforms have you tested on?
Wasn't able to reproduce the issue on MacOS/Chrome on staging:
In the video of yours @MariaHCD you navigate to the expense report to wait for it to complete. If you stay on the workspace chat for a bit the report preview component should update but, but you'll notice in mine I have to navigate from it to get it to update. Here it is on MacOS Chrome:
https://github.com/Expensify/App/assets/16232057/d7333f04-4a3e-4710-aedf-f19eef5107b4
100% reproducible on latest main
https://github.com/Expensify/App/assets/97473779/8de8acb2-4d4a-4675-b18d-1ac82150ff4c
This doesn't depend on platforms
It is not immediate, as parsing takes a little bit (more on staging/prod, as there are, potentially, many receipts, each of which creates a job with competing priorities), but this is working fine for me on dev, and the component is updated a few seconds after the job is completed, which is the expected behavior 🤷
If you stay on the workspace chat for a bit the report preview component should update
Oh, interesting, I tested again and remained on the workspace chat. Definitely saw that it did not update the report preview 🤔
https://github.com/Expensify/App/assets/12268372/b9ec5fb9-4bdb-4a01-b496-cde34297bf62
@s-alves10 @0xmiroslav Can you double check if the transaction in Onyx is being updated with the receipt details as expected? Because we're definitely sending the transaction data via pusher in the backend and our logs do show the same as well. I wonder if it is the App that isn't rendering the updated transaction in the ReportPreview component here?
@MariaHCD
No, the frontend doesn't receive the pusher events and so onyx transaction is not updated.
Would you mind giving me the email address that you tested with?
I just ran a test with sangar.work1028+009@gmail.com
We definitely sent the pusher data for the Onyx key transactions_1078602410102294240
🤔 @s-alves10 can you confirm that was the Onyx data you checked?
x3T6ok | virt1.rno | 2023-09-15 07:26:43 237 | sangar.work1028+009@gmail.com | [push] Push_Service - Sending event ~~ channel: '[0: 'private-encrypted-user-accountID-14942190' 1: 'private-encrypted-user-accountID-14868802']' eventName: 'onyxApiUpdate' eventData: '[{"onyxMethod":"merge","key":"transactions_1078602410102294240","value":{"amount":-2000,"date":"2023-09-04","merchant":"SaaStr Cantina","currency":"USD","receipt":{"receiptID":658015732,"source":"https:\/\/www.expensify.com\/receipts\/w_f7f7171539b192e2ab2d0717e2fb9ee076681237.png","state":"SCANCOMPLETE"}}}]'
x3T6ok | virt1.rno | 2023-09-15 07:26:43 237 | sangar.work1028+009@gmail.com | [push] Push_Service - Sending chunked event ~~ channel: '[0: 'private-encrypted-user-accountID-14942190' 1: 'private-encrypted-user-accountID-14868802']' eventName: 'onyxApiUpdate' eventData: '[{"onyxMethod":"merge","key":"transactions_1078602410102294240","value":{"amount":-2000,"date":"2023-09-04","merchant":"SaaStr Cantina","currency":"USD","receipt":{"receiptID":658015732,"source":"https:\/\/www.expensify.com\/receipts\/w_f7f7171539b192e2ab2d0717e2fb9ee076681237.png","state":"SCANCOMPLETE"}}}]' chunkSize: '1'
The data we sent:
[
{
"onyxMethod":"merge",
"key":"transactions_1078602410102294240",
"value":{
"amount":-2000,
"date":"2023-09-04",
"merchant":"SaaStr Cantina",
"currency":"USD",
"receipt":{
"receiptID":658015732,
"source":"https:\/\/www.expensify.com\/receipts\/w_f7f7171539b192e2ab2d0717e2fb9ee076681237.png",
"state":"SCANCOMPLETE"
}
}
}
]
Let me check again
Really strange. Frontend doesn't receive any events and transaction receipt status isn't updated. The frontend does receive other events like message
Yeah, definitely weird. I'll try pointing my dev App to the staging API and see if the issue still occurs.
@trjExpensify, @0xmiroslav Whoops! This issue is 2 days overdue. Let's get this updated quick!
Thanks, @MariaHCD!
@MariaHCD @trjExpensify @0xmiroslav this issue was created 2 weeks ago. Are we close to approving a proposal? If not, what's blocking us from getting this issue assigned? Don't hesitate to create a thread in #expensify-open-source to align faster in real time. Thanks!
Focused on https://github.com/Expensify/Expensify/issues/316043 today. Getting to this tomorrow.
📣 It's been a week! Do we have any satisfactory proposals yet? Do we need to adjust the bounty for this issue? 💸
Tested the app locally while pointing to the staging API. And I do think there is some discrepancy between the Onyx data sent via Pusher after the scan completes and the Onyx data that is set once the report is opened 🤔 Specifically for the amount, merchant and currency fields.
Left: transaction Onyx data sent via Pusher after the scan is complete, Right: transaction Onyx data once OpenReport is called:
Internal logs:
[updates: '[0: '[data: '[0: '[key: 'transactions_8138414820431815831' onyxMethod: 'merge' value: '[amount: '-2000' currency: 'USD' date: '2023-09-08' merchant: 'SaaStr Cantina (E)' receipt: '[receiptID: '659128384' source: 'https://www.expensify.com/receipts/w_75fe0ca9dfa7c03ca8c9f56da25bac00ec2bc099.png' state: 'SCANCOMPLETE']']']']' eventType: 'onyxApiUpdate']']' lastUpdateID: '14726669' previousUpdateID: '14726358']
@Gonals once the scan is complete and when we update the transactions Onyx key here, shouldn't we be setting the modifiedAmount, modifiedCurrency, modifiedMerchant fields instead of the amount, currency and merchant fields?
So two things potentially:
total
field in the report Onyx key since we use that in the report preview here so it seems like the amount doesn't update after the scan completes because we don't send the updated total in the Onyx data hereGoing to continue testing/investigating tomorrow.
Got a draft backend PR in place, I'll do some more testing.
Pushed the backend PR out for review. We can also test it on staging once it is deployed.
Backend PR was deployed. I tested on staging and the report preview is updated after the scan is complete:
https://github.com/Expensify/App/assets/12268372/9f30db92-c520-4a1a-b213-2755e7ab040e
However, I noticed two things:
0
so the report preview isn't showing the correct amount. Looking into this one. (internal logs)ffyqIs | virt1.sjc | 2023-10-02 04:36:50 561 | maria+2@expensifail.com | [push] Push_Service - Sending chunked event ~~ name: 'multipleEvents' channels: '[0: 'private-encrypted-user-accountID-8910926']' data: '[updates: '[0: '[data: '[0: '[key: 'transactions_7076274334862224452' onyxMethod: 'merge' value: '[modifiedAmount: '-2000' modifiedCreated: '2023-09-08' modifiedCurrency: 'USD' modifiedMerchant: 'SaaStr Cantina (E)' receipt: '[receiptID: '662139566' source: 'https://www.expensify.com/receipts/w_74b4d517c431235930da8283113164f8deefbabd.png' state: 'SCANCOMPLETE']']']' 1: '[key: 'report_3732056035654371' onyxMethod: 'merge' value: '[total: '0']']']' eventType: 'onyxApiUpdate']']' lastUpdateID: '19261406' previousUpdateID: '19261404']'
📣 It's been a week! Do we have any satisfactory proposals yet? Do we need to adjust the bounty for this issue? 💸
If you haven’t already, check out our contributing guidelines for onboarding and email contributors@expensify.com to request to join our Slack channel!
Action Performed:
Prerequisite: Use the example receipt provided below, because it will likely hit an image parser and complete super fast to test this.
<
to the LHN or click the report preview component)Expected Result:
The report preview component should update with the receipt data automatically without needing to navigate away and back to the chat
Actual Result:
The report preview component does not update
Workaround:
Yes, as described above to navigate away and back.
Platforms:
Which of our officially supported platforms is this issue occurring on?
Version Number:
v1.3.64-0
Reproducible in staging?: Y Reproducible in production?: Y If this was caught during regression testing, add the test name, ID and link from TestRail: Email or phone of affected tester (no customers): Logs: https://stackoverflow.com/c/expensify/questions/4856 Notes/Photos/Videos:mWeb safari reproduction: https://github.com/Expensify/App/assets/16232057/019912f4-8c89-415f-92c6-8f1f363a033e
iOS native app reproduction: https://github.com/Expensify/App/assets/16232057/2fa42280-be97-49d9-a62a-5de4699bf94c
Mac Chrome reproduction: https://github.com/Expensify/App/assets/16232057/13172aa7-0abe-435e-aa24-8e25e9225077
Example receipt to use:
Expensify/Expensify Issue URL: Issue reported by: @trjExpensify Slack conversation: Internal
View all open jobs on GitHub
CC: @Gonals @MariaHCD @JmillsExpensify @dylanexpensify
Upwork Automation - Do Not Edit
Issue Owner
Current Issue Owner: @MariaHCD