[HOLD for payment 2024-03-14] [$500] [MEDIUM] Smartscan: Show PDF receipt thumbnails

trjExpensify commented 10 months ago

Coming from here. CC: @sobitneupane @pradeepmdk @Gonals

If you haven’t already, check out our contributing guidelines for onboarding and email contributors@expensify.com to request to join our Slack channel!

Version Number: v1.3.98-5 Reproducible in staging?: Y Reproducible in production?: Y If this was caught during regression testing, add the test name, ID and link from TestRail: Email or phone of affected tester (no customers): Logs: https://stackoverflow.com/c/expensify/questions/4856 Expensify/Expensify Issue URL: Issue reported by: @trjExpensify Slack conversation: https://github.com/Expensify/App/pull/30747#issuecomment-1802321587

Action Performed:

Request Money > Scan > Upload a PDF receipt > choose a participant
Observe there isn't a preview of the PDF receipt in the thumbnail on the confirmation page
Click confirm to create the request
Observe there isn't a preview of the PDF receipt in the thumbnail on the report preview component in the chat
Click the report preview component
Observe there isn't a preview of the PDF receipt in the thumbnail on the request preview component in the expense/iouReport
Click the request preview component
Observe there isn't a preview of the PDF receipt in the thumbnail on the receipt preview component in the request view

Expected Result:

PDF receipts should show a preview in the relevant thumbnails throughout the app.

P.s this also includes in the split preview component, so let's make sure it works there.

Actual Result:

Generic "empty" thumbnail previews are displayed for PDF receipts.

Workaround:

Yes, they can tap the thumbnail to see the PDF but they'd really not know to do that.

Platforms:

Which of our officially supported platforms is this issue occurring on?

[ ] Android: Native
[ ] Android: mWeb Chrome
[ ] iOS: Native
[ ] iOS: mWeb Safari
[ ] MacOS: Chrome / Safari
[ ] MacOS: Desktop

Screenshots/Videos

Confirmation screen:

Report preview on the chat:

Request preview on the expense/iouReport:

Receipt preview on the request:

View all open jobs on GitHub

Upwork Automation - Do Not Edit

Upwork Job URL: https://www.upwork.com/jobs/~01f90fbd2f6a7c49f2
Upwork Job ID: 1725113274629943296
Last Price Increase: 2023-11-23

melvin-bot[bot] commented 10 months ago

Current assignee @trjExpensify is eligible for the Bug assigner, not assigning anyone new.

melvin-bot[bot] commented 10 months ago

Job added to Upwork: https://www.upwork.com/jobs/~01f90fbd2f6a7c49f2

melvin-bot[bot] commented 10 months ago

Bug0 Triage Checklist (Main S/O)

[ ] This "bug" occurs on a supported platform (ensure Platforms in OP are ✅)
[ ] This bug is not a duplicate report (check E/App issues and #expensify-bugs)
- If it is, comment with a link to the original report, close the issue and add any novel details to the original issue instead
[ ] This bug is reproducible using the reproduction steps in the OP. S/O
- If the reproduction steps are clear and you're unable to reproduce the bug, check with the reporter and QA first, then close the issue.
- If the reproduction steps aren't clear and you determine the correct steps, please update the OP.
[ ] This issue is filled out as thoroughly and clearly as possible
- Pay special attention to the title, results, platforms where the bug occurs, and if the bug happens on staging/production.
[ ] I have reviewed and subscribed to the linked Slack conversation to ensure Slack/Github stay in sync

melvin-bot[bot] commented 10 months ago

Triggered auto assignment to Contributor-plus team member for initial proposal review - @0xmiroslav (External)

tienifr commented 10 months ago

Proposal

Please re-state the problem that we are trying to solve in this issue.

Show PDF receipt thumbnails

What is the root cause of that problem?

We're getting thumbnail for receipt here

https://github.com/Expensify/App/blob/69a33583877f26d7c446ba231af6fc2c477cca5b/src/libs/ReceiptUtils.ts#L31

We do not have the logic to get the thumbnail of pdf

-> Default image is shown

What changes do you think we should make in order to solve the problem?

We can lean on the source to get the thumbnail for pdf.

Here's the sample source https://www.expensify.com/receipts/{id}.pdf -> the thumbnail should be https://www.expensify.com/receipts/{id}.jpg.1024.jpg

    const isReceiptPDF = Str.isPDF(filename);
...

 if(isReceiptPDF){
            return {thumbnail: `${path.substring(0, path.length-4)}.jpg.1024.jpg`, image: path};
        }

What alternative solutions did you explore? (Optional)

BE will return the thumbnail and use it to shown on FE side

Result

https://github.com/Expensify/App/assets/113963320/c7dff2d8-4bfe-45c2-b1fe-ddefd2f3c0be

trjExpensify commented 10 months ago

@0xmiroslav can you review this please? Also, @tienifr confirming in the results we're solving for all the thumbnails noted in the OP?

eh2077 commented 10 months ago

Proposal

Please re-state the problem that we are trying to solve in this issue.

We want to show PDF thumbnail in the process of uploading pdf receipt.

What is the root cause of that problem?

The root cause of this issue is that, currently, we don't have an implementation of previewing PDF source by thumbnail. The PDF source here means expensify source like https://www.expensify.com/chat-attachments/7486855856104260927/w_1c2aef0dea2f6e30acd3ca31c50286af09a4d3c9.pdf or local file source starts with blob:// or file://.

Currently, uploading PDF attachment from chat can be previewed before upload and the PDF thumbnail is only available after it's been successfully uploaded and the backend will create an image source url for the PDF, like, https://www.expensify.com/chat-attachments/7486855856104260927/w_1c2aef0dea2f6e30acd3ca31c50286af09a4d3c9.jpg.1024.jpg

Intuitively, we should be able to preview the PDF thumbnail before uploading it to backend because we're able to preview PDF pages before uploading.

The new feature to have a ThumbnailPDF which works on both server and local uri like what Image component supports is great complement to the App, especially for the use case like creating an IOU request and the file is not yet been uploaded to backend and the offline use case.

What changes do you think we should make in order to solve the problem?

We can add an implementation of ThumbnailPDF which previews pdf uri (both server and local uri) in thumbnail. On the web platform, we can refer the implementation of PDFView by only rendering the first page of PDF file as the thumbnail. We can convert the first page to image, like this example.

We can also update react-pdf to 7.x.x and then use the thumbnail feature provided. There's some extra rendering issue need to be solved if we choose to upgrade.

On native platform, we can make use of the singlePage property of https://github.com/wonday/react-native-pdf.

This solution will need more effort to implement.

What alternative solutions did you explore? (Optional)

melvin-bot[bot] commented 10 months ago

📣 It's been a week! Do we have any satisfactory proposals yet? Do we need to adjust the bounty for this issue? 💸

trjExpensify commented 10 months ago

@0xmiroslav bump!

trjExpensify commented 10 months ago

Same. @0xmiroslav can you review these proposals today, please? They've been hanging out for 2 weeks.

0xmiros commented 10 months ago

reviewing today

trjExpensify commented 10 months ago

Perfect, thanks.

trjExpensify commented 10 months ago

@0xmiroslav did you review? 2 weeks is a pretty long time, and PDFs are pretty common receipt types. Thanks!

0xmiros commented 10 months ago

We definitely need thumbnail from backend like chat pdf attachment. But do we need to show thumbnail for local pdf (before uploading) as well? cc: @trjExpensify

0xmiros commented 10 months ago

When show local thumbnail, need to match getting thumbnail logic with backend for consistency Going to pull up engineer directly before proposal approval, since this requires help from backend 🎀 👀 🎀

melvin-bot[bot] commented 10 months ago

Triggered auto assignment to @luacmartins, see https://stackoverflow.com/c/expensify/questions/7972 for more details.

trjExpensify commented 10 months ago

But do we need to show thumbnail for local pdf (before uploading) as well?

Yeah, we should show you a preview of the PDF like we do a preview of the local image attachment prior to upload.

0xmiros commented 10 months ago

But do we need to show thumbnail for local pdf (before uploading) as well?

Yeah, we should show you a preview of the PDF like we do a preview of the local image attachment prior to upload.

It's actually full preview, not thumbnail. So there's no pre-existing solution to use.

luacmartins commented 10 months ago

@0xmiroslav should I review any specific proposal?

0xmiros commented 10 months ago

@0xmiroslav should I review any specific proposal?

There's no acceptable proposal yet since we agreed to show thumbnail for local pdf as well. But I need your help sharing backend logic to get thumbnail for pdf so that we can apply same to frontend.

luacmartins commented 10 months ago

We simply replace the extension .pdf with .1024.jpg, so the resulting url is filename.1024.jpg. Is that what you needed?

melvin-bot[bot] commented 10 months ago

@trjExpensify @luacmartins @0xmiroslav this issue was created 2 weeks ago. Are we close to approving a proposal? If not, what's blocking us from getting this issue assigned? Don't hesitate to create a thread in #expensify-open-source to align faster in real time. Thanks!

0xmiros commented 10 months ago

We simply replace the extension .pdf with .1024.jpg, so the resulting url is filename.1024.jpg. Is that what you needed?

ok 2 more things:

always shows first page as thumbnail
max size is 1024, keep same ratio as original pdf size. i.e. 649x1024

luacmartins commented 10 months ago

I think we should show the first page as thumbnail, just like we do for regular pdf attachments. As for the ratio, I'm not sure what you mean. We should show the preview in the same format/dimensions we show other receipt file types, no?

0xmiros commented 10 months ago

As for the ratio, I'm not sure what you mean. We should show the preview in the same format/dimensions we show other receipt file types, no?

It's preview size in the app. I meant the real size of generated image. If you see pdf in chat attachment, the size will be something like w x 1024 even though preview in report action item is much smaller.

luacmartins commented 10 months ago

Yes, we scale the thumbnail to fit a bounding box of 1024x1024 keeping its aspect ratio and only scaling down if the image is larger than 1024px.

melvin-bot[bot] commented 10 months ago

@trjExpensify @luacmartins @0xmiroslav this issue was created 2 weeks ago. Are we close to approving a proposal? If not, what's blocking us from getting this issue assigned? Don't hesitate to create a thread in #expensify-open-source to align faster in real time. Thanks!

0xmiros commented 10 months ago

We're waiting proposals based on expected behavior discussed

melvin-bot[bot] commented 10 months ago

@trjExpensify @luacmartins @0xmiroslav this issue was created 2 weeks ago. Are we close to approving a proposal? If not, what's blocking us from getting this issue assigned? Don't hesitate to create a thread in #expensify-open-source to align faster in real time. Thanks!

tienifr commented 10 months ago

@luacmartins Can you share the logic to generate the thumbnail of pdf

We simply replace the extension .pdf with .1024.jpg, so the resulting url is filename.1024.jpg. Is that what you needed?

we have to generate the filename.1024.jpg before replace the extension .pdf with .1024.jpg.

Or maybe we can use the library to do that like react-native-pdf-thumbnail. cc @0xmiroslav

eh2077 commented 10 months ago

@0xmiroslav Can I have your feedback on my proposal https://github.com/Expensify/App/issues/31432#issuecomment-1820184229 ?

0xmiros commented 10 months ago

We can add an implementation of ThumbnailPDF which previews pdf uri (both server and local uri) in thumbnail. On the web platform, we can refer the implementation of PDFView by only rendering the first page of PDF file as the thumbnail. We can convert the first page to image, like this example.

Have you tried to integrate your solution into our app?

luacmartins commented 10 months ago

I'm not sure what you mean, if you already have filename.1024.jpg why would you need to replace the extension?

trjExpensify commented 10 months ago

Bump @eh2077 on @luacmartins's question!

eh2077 commented 10 months ago

Proposal

Updated proposal to use the new thumbnail feature of react-pdf on web.

@0xmiroslav I'll share a demo about the POC later.

eh2077 commented 10 months ago

I'm not sure what you mean, if you already have filename.1024.jpg why would you need to replace the extension?

@luacmartins When choosing a pdf receipt to scan from local disk, we're in the middle of flow and the pdf receipt is not yet uploaded to the backend, so in the MoneyRequestConfirmPage, we can't preview the pdf receipt in thumbnail, see the confirm screen shared by @trjExpensify

I guess the backend should have some logic to convert pdf to image, like this SO

eh2077 commented 10 months ago

@trjExpensify Is it acceptable to show pdf receipt only when online? If so, then we can consider uploading the pdf to backend to get the thumbnail before confirming the request, which will be easier to implement though it's not perfect when offline.

luacmartins commented 10 months ago

@eh2077 we need the thumbnail to work offline too.

I guess the backend should have some logic to convert pdf to image, like this SO

Yes, the backend uses ImageMagick to convert the pdf to jpg.

trjExpensify commented 9 months ago

@eh2077 we need the thumbnail to work offline too.

Agreed 👍

melvin-bot[bot] commented 9 months ago

@trjExpensify @luacmartins @0xmiroslav this issue is now 3 weeks old. There is one more week left before this issue breaks WAQ and will need to go internal. What needs to happen to get a PR in review this week? Please create a thread in #expensify-open-source to discuss. Thanks!

dylanexpensify commented 9 months ago

@eh2077 any update here?

luacmartins commented 9 months ago

@eh2077 @0xmiroslav anything else that you need from the backend for now?

trjExpensify commented 9 months ago

Bump on the question above: @eh2077 @0xmiroslav

0xmiros commented 9 months ago

I don't think we need more from backend for now

melvin-bot[bot] commented 9 months ago

@trjExpensify @luacmartins @0xmiroslav this issue is now 4 weeks old and preventing us from maintaining WAQ, can you:

Decide whether any proposals currently meet our guidelines and can be approved as-is today
If no proposals meet that standard, please take this issue internal and treat it as one of your highest priorities
If you have any questions, don't hesitate to start a discussion in #expensify-open-source

Thanks!

melvin-bot[bot] commented 9 months ago

Current assignee @0xmiroslav is eligible for the Internal assigner, not assigning anyone new.

dylanexpensify commented 9 months ago

@0xmiroslav what's next here? Mind giving us an ETA?

0xmiros commented 9 months ago

We're waiting proposals

dylanexpensify commented 9 months ago

@eh2077 @tienifr any chance you'd like to update proposals?

trjExpensify commented 9 months ago

Yeah, @tienifr @eh2077 we're super close on this one. Can you confirm in the next day or two before we look elsewhere?

Expensify / App