[HOLD for payment 2024-03-14] [$500] [MEDIUM] Smartscan: Show PDF receipt thumbnails

[trjExpensify]

Coming from here. CC: @sobitneupane @pradeepmdk @Gonals

If you haven’t already, check out our contributing guidelines for onboarding and email contributors@expensify.com to request to join our Slack channel!

---

Version Number: v1.3.98-5 Reproducible in staging?: Y Reproducible in production?: Y If this was caught during regression testing, add the test name, ID and link from TestRail: Email or phone of affected tester (no customers): Logs: https://stackoverflow.com/c/expensify/questions/4856 Expensify/Expensify Issue URL: Issue reported by: @trjExpensify Slack conversation: https://github.com/Expensify/App/pull/30747#issuecomment-1802321587

Action Performed:

1. Request Money > Scan > Upload a PDF receipt > choose a participant
2. Observe there isn't a preview of the PDF receipt in the thumbnail on the confirmation page
3. Click confirm to create the request
4. Observe there isn't a preview of the PDF receipt in the thumbnail on the report preview component in the chat
5. Click the report preview component
6. Observe there isn't a preview of the PDF receipt in the thumbnail on the request preview component in the expense/iouReport
7. Click the request preview component
8. Observe there isn't a preview of the PDF receipt in the thumbnail on the receipt preview component in the request view

Expected Result:

PDF receipts should show a preview in the relevant thumbnails throughout the app.

P.s this also includes in the split preview component, so let's make sure it works there.

Actual Result:

Generic "empty" thumbnail previews are displayed for PDF receipts.

Workaround:

Yes, they can tap the thumbnail to see the PDF but they'd really not know to do that.

Platforms:

Which of our officially supported platforms is this issue occurring on?

• [ ] Android: Native
• [ ] Android: mWeb Chrome
• [ ] iOS: Native
• [ ] iOS: mWeb Safari
• [ ] MacOS: Chrome / Safari
• [ ] MacOS: Desktop

Screenshots/Videos

Confirmation screen:

Report preview on the chat:

Request preview on the expense/iouReport:

Receipt preview on the request:

View all open jobs on GitHub

Upwork Automation - Do Not Edit

• Upwork Job URL: https://www.upwork.com/jobs/~01f90fbd2f6a7c49f2
• Upwork Job ID: 1725113274629943296
• Last Price Increase: 2023-11-23

[melvin-bot[bot]]

@trjExpensify, @luacmartins, @0xmiroslav Uh oh! This issue is overdue by 2 days. Don't forget to update your issues!

[melvin-bot[bot]]

@trjExpensify, @luacmartins, @0xmiroslav Whoops! This issue is 2 days overdue. Let's get this updated quick!

[luacmartins]

Still waiting on proposals

[trjExpensify]

@eh2077 @tienifr, can you confirm if you'll be able to update in the next couple of days? Else @luacmartins, I think you might need to grab this one if that's okay with you. It's a HIGH issue and it's coming up in conversation with the real-world usage we have right now with the generalists policy and TU policy.

[0xmiros]

Another option is to double bounty given it's high priority and requires more efforts to implement

[AmjedNazzal]

Proposal

Please re-state the problem that we are trying to solve in this issue.

Show PDF receipt thumbnails

What is the root cause of that problem?

We do not have a thumbnail coming back from the backend for PDF files, So when we attempt to get a thumbnail for the PDF from ReceiptUtils.getThumbnailAndImageURIs, it will give us path for image and ReceiptGeneric for a thumbnail since PDF extention doesn't match any logic in the function.

Furthermore we currently have no logic to handle a preview for PDF files, as it is now we also try to handle that using ReceiptUtils.getThumbnailAndImageURIs, resulting in the same behaviour when we actually shouldn't try to retrieve the preview like that since we aren't making the backend call yet.

What changes do you think we should make in order to solve the problem?

Firstly we could handle the PDF preview before the submission and backend call by using react-pdf as we do for PDF preview while sending attachments, the only difference is that we would capture a preview image rather than using PDFPreview components, we can do this within MoneyTemporaryForRefactorRequestConfirmationList:

let receiptImage;
let receiptThumbnail;

const [PDFReceiptImage, setPDFReceiptImage] = useState('');

const generateThumbnail = useCallback(() => {
    const offScreenCanvas = document.createElement('canvas');
    let PDFRecieptImageCapture;
    pdfjs.getDocument(transaction.receipt.source).promise
        .then(pdf => pdf.getPage(1))
        .then(page => {
            const viewport = page.getViewport({ scale: 1 });
            offScreenCanvas.width = viewport.width;
            offScreenCanvas.height = viewport.height;
            const context = offScreenCanvas.getContext('2d');

            page.render({ canvasContext: context, viewport }).promise
            .then(() => {
                PDFRecieptImageCapture = offScreenCanvas.toDataURL('image/jpeg');
                setPDFReceiptImage(PDFRecieptImageCapture? PDFRecieptImageCapture : ReceiptGeneric);
            })
        })
    }, [transaction]);

if(receiptPath && receiptFilename && transaction){
    if ((Str.isPDF(transaction.filename || translate('attachmentView.unknownFilename')))) {
        generateThumbnail();
    }else {
        ({ image: receiptImage, thumbnail: receiptThumbnail } = ReceiptUtils.getThumbnailAndImageURIs(transaction, receiptPath, receiptFilename));
    }     
}

....
return(
    ....

    {(receiptImage || receiptThumbnail) && (
        <Image
            style={styles.moneyRequestImage}
            source={{ uri: receiptThumbnail || receiptImage }}
            // AuthToken is required when retrieving the image from the server
            // but we don't need it to load the blob:// or file:// image when starting a money request / split bill
            // So if we have a thumbnail, it means we're retrieving the image from the server
            isAuthTokenRequired={!_.isEmpty(receiptThumbnail)}
        />
    )} 
    {PDFReceiptImage && (
        <Image 
            source={{ uri: PDFReceiptImage }}
            style={styles.moneyRequestImage} 
            isAuthTokenRequired={false} 
        />
    )}

This results in a preview image like shown in the following screenshot:

Now as for the thumbnail itself, we could add a const in ReceiptUtils.getThumbnailAndImageURIs to check if the file is PDF and then add that check to the if statement that adds .1024.jpg, assuming the backend will start having a thumbnail with that name within the transaction object

const isPDFImage = Str.isPDF(filename);
....
if (isReceiptImage || isPDFImage) {
    return {thumbnail: `${path}.1024.jpg`, image: path};
}

Please keep in mind that the visual aspect can be improved further by for example adding a loading spinner while the pdf is getting rendered and captured similar to how we do in attachments

[trjExpensify]

👋 @0xmiroslav can you prioritise the review on this one, please? Thanks!

[0xmiros]

yes, I will provide feedback tomorrow

[trjExpensify]

Thanks so much! Somewhat related, but it seems like loading the PDF in the receipt modal has broken again somehow, so the team approving PDF receipts can't see them at all now in /app so that's increased the urgency on this.

Asked a question on the OG issue for that about looking into it, but might be something easier to take care of here.

[AmjedNazzal]

@trjExpensify I haven't dug deep into the other issue you've mentioned, but from a quick glance I can see a couple of issues: 1) The url we pass to PDFView will look like this "/receipts/w_4904c6352b77bcf981b69797e5cde1c3dbc5e851.pdf" which will get automatically appended whatever we are running the app from, for example if we are running the dev build, the url will have dev appended in it and will look like this "https://dev.new.expensify.com:8082/receipts/w_4904c6352b77bcf981b69797e5cde1c3dbc5e851.pdf", meanwhile the backend specifies the url in transactions object to be "https://www.expensify.com/receipts/w_4904c6352b77bcf981b69797e5cde1c3dbc5e851.pdf" since we want to access the url in the actual expensify DB.

2) Even if I try correcting the url to "expensify.com", whenever pdfjs tries to access that, it will also fail which could mean there's also a problem in the BE side of things.

Would love to get insight from an engineer on why the actual url in onyx transaction is failing.

[luacmartins]

@AmjedNazzal do you see any console errors or request errors when trying to access the URL from pdfjs? Can you access the URL directly in your browser? This url works for me in the browser.

[AmjedNazzal]

@luacmartins Yes I am getting console error, even if I try this on prod which means I would be trying to access the url while logged in the correct user in prod and so the path will be correct, I get this console error:

And if I copy the url, it's actually the exact same url as the one I'm seeing in transaction, but if I try to open it in browser I get this:

[trjExpensify]

I think Sobit figured out the regression here: https://github.com/Expensify/App/issues/27680#issuecomment-1866052293

[AmjedNazzal]

@0xmiroslav any updates? thank you!

[melvin-bot[bot]]

@trjExpensify, @luacmartins, @0xmiroslav Whoops! This issue is 2 days overdue. Let's get this updated quick!

[0xmiros]

sorry I will provide update this week

[melvin-bot[bot]]

@trjExpensify, @luacmartins, @0xmiroslav Whoops! This issue is 2 days overdue. Let's get this updated quick!

[melvin-bot[bot]]

@trjExpensify, @luacmartins, @0xmiroslav Still overdue 6 days?! Let's take care of this!

[melvin-bot[bot]]

@trjExpensify, @luacmartins, @0xmiroslav 6 days overdue. This is scarier than being forced to listen to Vogon poetry!

[trjExpensify]

👋 @0xmiroslav this is a HIGH priority issue in wave5, can you prioritise this proposal review today please?

[0xmiros]

@AmjedNazzal you proposed only web solution. How about native?

[0xmiros]

@eh2077 thumbnail in react-pdf v7 doesn't work when offline?

[eh2077]

@eh2077 thumbnail in react-pdf v7 doesn't work when offline?

Sorry for delay just back from holiday. The new thumbnail feature should work but the v7 will break the pdf preview feature in the way we do in v6. I'll dig deeper and update soon.

[0xmiros]

react-pdf is being updated to v7 in #33083 so you can research based on that codebase

[AmjedNazzal]

@0xmiroslav You are right that my original solution cannot handle native, to simplify and cleanup this and make code that can be overall useful in the future as well, we could create a component for thumbnails with the following steps:

1) We would completely remove the logic for generating thumbnails and move it to a seperate new component called FileThumbnail and only include that within MoneyTemporaryForRefactorRequestConfirmationList:

{(receiptPath && receiptFilename && transaction) && (
<FileThumbnail
    transaction={transaction}
    receiptPath={receiptPath}
    receiptFilename={receiptFilename} 
/>
)} 

2) We then create the new component with index.js and index.native.js, in the index.js we use the original solution in my proposal:

function FileThumbnail ({transaction, receiptPath, receiptFilename}){
    ....
    let receiptImage;
    let receiptThumbnail;

    const [PDFReceiptImage, setPDFReceiptImage] = useState('');

    const generateThumbnail = () => {
        const offScreenCanvas = document.createElement('canvas');
        pdfjs.getDocument(transaction.receipt.source).promise
            .then(pdf => pdf.getPage(1))
            .then(page => {
                const viewport = page.getViewport({ scale: 1 });
                offScreenCanvas.width = viewport.width;
                offScreenCanvas.height = viewport.height;
                const context = offScreenCanvas.getContext('2d');

                page.render({ canvasContext: context, viewport }).promise
                .then(() => {
                    setPDFReceiptImage(offScreenCanvas.toDataURL('image/jpeg'));
                })
            })

    };

    if(receiptPath && receiptFilename && transaction){
        if ((Str.isPDF(transaction.filename || translate('attachmentView.unknownFilename')))) {
            generateThumbnail();
        }else {
            ({ image: receiptImage, thumbnail: receiptThumbnail } = ReceiptUtils.getThumbnailAndImageURIs(transaction, receiptPath, receiptFilename));
        }     
    }
    return (
        <>
        {(receiptImage || receiptThumbnail) && (
            <Image
                ....
            />
        )} 
        {PDFReceiptImage && (
            <Image 
                source={{ uri: PDFReceiptImage }}
                style={styles.moneyRequestImage} 
                isAuthTokenRequired={false} 
            />
        )}
        </>
    );

}

But in index.native.js, we can install and implement the use of PdfThumbnail from react-native-pdf-thumbnail which is a very simple library specifically made for this within the generateThumbnail function:

import PdfThumbnail from "react-native-pdf-thumbnail";
function FileThumbnail ({transaction, receiptPath, receiptFilename}){

....

 const generateThumbnail = async () => {
    const { uri } = await PdfThumbnail.generate(transaction.receipt.source, 0);
    setPDFReceiptImage(uri);
};

if(receiptPath && receiptFilename && receiptFile){
    if ((Str.isPDF(receiptFile.name || translate('attachmentView.unknownFilename')))) {
        generateThumbnail();
    }else {
        ({ image: receiptImage, thumbnail: receiptThumbnail } = ReceiptUtils.getThumbnailAndImageURIs(transaction, receiptPath, receiptFilename));
    }     
}

...

[melvin-bot[bot]]

⚠️ Looks like this issue was linked to a Deploy Blocker here

If you are the assigned CME please investigate whether the linked PR caused a regression and leave a comment with the results.

If a regression has occurred and you are the assigned CM follow the instructions here.

If this regression could have been avoided please consider also proposing a recommendation to the PR checklist so that we can avoid it in the future.

[trjExpensify]

How is this linked to a deploy blocker? 🤔

Separately, @0xmiroslav can you provide feedback on https://github.com/Expensify/App/issues/31432#issuecomment-1875909723

[0xmiros]

I am inclined to using thumbnail feature in react-pdf v7. So it makes sense to hold this for https://github.com/Expensify/App/pull/33083

[0xmiros]

How is this linked to a deploy blocker? 🤔

False alarm. @sobitneupane referenced this issue and deleted that comment

Not a deploy blocker. It is reproducible in production as well. I believe it will probably handled in #31432.

[trjExpensify]

I am inclined to using thumbnail feature in react-pdf v7. So it makes sense to hold this for https://github.com/Expensify/App/pull/33083

Hm, I'm hesitant to add even further delay to fixing this as it's impacting a core flow, but if it's going to break again in a week I see the point. I think we need to up the urgency on getting https://github.com/Expensify/App/pull/33083 to the finish line though and treat it as a deliverable for this issue and #wave5 if we're now reliant on it. CC: @artus9033 & @jjcoffee to put this on your radar.

[trjExpensify]

Popped a hold in the title as that's getting finished.

[luacmartins]

Still on hold for https://github.com/Expensify/App/pull/33083

[trjExpensify]

Same, held on the linked PR.

[trjExpensify]

Still held, getting close though!

[eh2077]

@trjExpensify We don't need to consider the case of uploading a password protected pdf receipt right? Currently, uploading a protected pdf receipt results in uploading failure error.

[trjExpensify]

Sorry, not sure I'm quite following. Are you saying any upload of a password protected PDF is resulting in a failure? Maybe a quick screen recording would be helpful?

[eh2077]

Yes, here's the clip

https://github.com/Expensify/App/assets/117511920/547edfa3-1335-479e-bc4f-6e2444024b95

[trjExpensify]

Got it. If we don't allow password protected PDFs for receipt upload, we should be preventing that with the "file type not supported" alert modal. Remind me, do they have a special extension?

[eh2077]

No, the password protected PDFs have same .pdf extension as normal ones. I think we need to read them to know if they're password protected.

[eh2077]

react-pdf is being updated to v7 in #33083 so you can research based on that codebase

POC update

POC branch for testing: https://github.com/eh2077/App/commit/1ed6dbab2c14baa226312f5bc35d2f6901184b2a

1. Web implementation uses Thumbnail of react-pdf v7
2. Native implementation uses the singlePage property of react-native-pdf

We can create a new component PDFThumbnail which is similar to the ThumbnailImage component. The PDFThumbnail can display a PDF source as a thumbnail. It works for both local file and remote source, online and offline modes.

POC videos

1. Web online

https://github.com/Expensify/App/assets/117511920/909e1f9c-79fc-4eac-8420-6b88b0122826

2. Web offline

https://github.com/Expensify/App/assets/117511920/f7438222-6205-489f-9dd2-7d808ee8d8a9

4. Native

Android online	Android offline

[eh2077]

@trjExpensify Can you help to check if the above demo videos look good to you overall?

[trjExpensify]

No, the password protected PDFs have same .pdf extension as normal ones. I think we need to read them to know if they're password protected.

I see, okay. In which case, what's with the failed upload error? Granted, the receipt scanning will always fail to extract the receipt details because we can't see the content of the PDF to read it, but if the file extension is simply .pdf, we shouldn't be failing on upload.

@trjExpensify Can you help to check if the above demo videos look good to you overall?

Unfortunately, none of those vids are loading for me.

[melvin-bot[bot]]

Triggered auto assignment to Contributor Plus for review of internal employee PR - @s77rt (Internal)

[mallenexpensify]

@s77rt reassigning, please take over as C+. If you don't have bandwidth, unassign yourself. Thanks

[trjExpensify]

FYI: PR for react-pdf is deployed to prod, so I took this off hold officially.

Vids still don't load for me, but don't block on that if @s77rt & @luacmartins want to proceed and give this a review now.

[s77rt]

@tienifr Thanks for the proposal. Your RCA is correct. However your solution will only work if online and if the receipt is uploaded already.

[s77rt]

@eh2077 Thanks for the proposal. Your RCA is correct. The solution looks good to me. I have just one addition which is that we should always try to use the .1024.jpg thumbnail first since it's an actual image and faster to render.

🎀 👀 🎀 C+ reviewed Link to proposal

[melvin-bot[bot]]

Current assignee @luacmartins is eligible for the choreEngineerContributorManagement assigner, not assigning anyone new.

[s77rt]

@AmjedNazzal Thanks for the proposal. Your RCA is correct. Your solution is close to what @eh2077 is suggesting. But I think it's better to use the the built in feature that react-pdf offers. Also I don't see the need for a new dependency react-native-pdf-thumbnail.

[luacmartins]

I agree with @s77rt's assessment and the videos looked good to me. Let's go ahead with it!