[HOLD for payment 2024-07-22] [HOLD for payment 2024-07-17] [$250] [Performance] [MEDIUM] Add "Export Onyx state" option to Troubleshoot menu and four finger tap

muttmuure commented 3 months ago

If you haven’t already, check out our contributing guidelines for onboarding and email contributors@expensify.com to request to join our Slack channel!

What performance issue do we need to solve?

e.g. memory consumption, storage read/write times, React native bridge concerns, inefficient React component rendering, etc.

When your mobile client's Onyx state falls out of sync with web, it is hard to diagnose which value has been missed or has been incorrectly inserted, especially when a refresh, sign in, or reload of the app on the client can solve the issue.

What is the impact of this on end-users?

List specific user experiences that will be improved by solving this problem e.g. app boot time, time to for some interaction to complete, etc.

This is more of a debugging tool, since Onyx impacts on pretty much everything you see in app, it can impact on a multitude of end user functionality.

The most important end user impact of a debugging tool which exports the Onyx state is around privacy. We need to sanitize the authToken, emails and messages of the data since those are private and shouldn't be shared with anyone.

List any benchmarks that show the severity of the issue

Please also provide exact steps taken to collect metrics above if any so we can independently verify the results.

This is more of a debugging tool, so this isn't something that will show benchmarks.

Proposed solution (if any)

Please list out the steps you think we should take to solve this issue.

Add a "Export Onyx State" option to the Troubleshoot menu.

This downloads the values in the Onyx keyvaluepairs table to a txt file. The authTokens, emails and messages in the file are redacted

List any benchmarks after implementing the changes to show impacts of the proposed solution (if any)

Note: These should be the same as the benchmarks collected before any changes.

Platforms:

Which of our officially supported platforms is this issue occurring on?

[ ] Android: Native
[ ] Android: mWeb Chrome
[ ] iOS: Native
[ ] iOS: mWeb Safari
[ ] MacOS: Chrome / Safari
[ ] MacOS: Desktop

Version Number: Reproducible in staging?: Reproducible in production?: Email or phone of affected tester (no customers): Logs: https://stackoverflow.com/c/expensify/questions/4856 Notes/Photos/Videos: Any additional supporting documentation Expensify/Expensify Issue URL: Issue reported by: Slack conversation:

View all open jobs on Upwork

Upwork Automation - Do Not Edit

Upwork Job URL: https://www.upwork.com/jobs/~014fd561267f73f88e
Upwork Job ID: 1803048668106098615
Last Price Increase: 2024-06-18

Issue Owner

Current Issue Owner: @muttmuure

melvin-bot[bot] commented 3 months ago

Auto-assigning issues to engineers is no longer supported. If you think this issue should receive engineering attention, please raise it in #whatsnext.

melvin-bot[bot] commented 2 months ago

Huh... This is 4 days overdue. Who can take care of this?

burczu commented 2 months ago

Hi, I am Bartek from Callstack - expert contributor group. I would like to work on this issue.

burczu commented 2 months ago

Daily update: I’ve been working on the proposal for this issue today, mostly on finding the way to collect the whole onyx snapshot as a string (to be able to save it as a txt file later), I hope I’ll be able to post something by the end of tomorrow - I’ll keep you posted. I’ve also created a draft PR in GH.

burczu commented 2 months ago

Daily update: I think I’ve found the way to collect all the Onyx data for web environment and already tested it; for native I think I’ve also find the way but it still needs testing. I believe the proposal is behind the corner and I’ll be able to post it tomorrow.

burczu commented 2 months ago

Daily update: the solution for collecting the Onyx data and saving as a text file is done and can be checked in the draft PR. I've forgotten that we also need to mask data such as authToken, emails etc, so this is the last thing I need to sort out.

melvin-bot[bot] commented 2 months ago

@burczu Whoops! This issue is 2 days overdue. Let's get this updated quick!

burczu commented 2 months ago

not overdue

burczu commented 2 months ago

Proposal

Please re-state the problem that we are trying to solve in this issue.

When your mobile client's Onyx state falls out of sync with web, it is hard to diagnose which value has been missed or has been incorrectly inserted, especially when a refresh, sign in, or reload of the app on the client can solve the issue.

What is the root cause of that problem?

There is no specific problem - this is more like a new feature. We want to add a "Export Onyx State" option in the Troubleshoot menu in the App. Pressing it should download the values in the Onyx keyvaluepairs table as a *.txt file.

What changes do you think we should make in order to solve the problem?

Add new button to the `Troubleshoot` menu

First we need to add the new button in the Troubleshoot section. To do so, we need to add new translation (initialSettingsPage.troubleshoot.exportOnyxState) in en.ts and es.ts files

exportOnyxState: 'Export Onyx state',

exportOnyxState: 'Exportar estado Onyx',

Next, we need to add the button in the /src/pages/settings/Troubleshoot/TroubleshootPage.tsx, here:

https://github.com/Expensify/App/blob/b278b8f2ec103eed98215b7068c14cc8d029deea/src/pages/settings/Troubleshoot/TroubleshootPage.tsx#L63-L69

The code after addition:

const baseMenuItems: BaseMenuItem[] = [
    {
        translationKey: 'initialSettingsPage.troubleshoot.clearCacheAndRestart',
        icon: Expensicons.RotateLeft,
        action: () => setIsConfirmationModalVisible(true),
    },
    {
        translationKey: 'initialSettingsPage.troubleshoot.exportOnyxState',
        icon: Expensicons.Download,
        action: exportOnyxState,
        },
];

The exportOnyxState action method implementation will be described in the last step.

Reading data from Onyx

Next, we need to add the code that will be responsible for reading data from Onyx's keyvaluepairs table. We need to maintain two different implementations for web and mobile, because on web we use the IndexedDB database, and on mobile we use the SQLite database.

To do it, we need to create the ExportOnyxState folder inside /src/libs, and create two files there: index.ts that will be responsible for web implementation and index.native.ts for mobile implementation.

IndexedDB on web

Here's the implementation of reading data from Onyx stored in the IndexedDB on web (using IDBDatabase available in the browser) in the index.ts file:

const readFromIndexedDB = () =>
new Promise<Record<string, unknown>>((resolve) => {
    let db: IDBDatabase;
    const openRequest = indexedDB.open('OnyxDB', 1);
    openRequest.onsuccess = () => {
        db = openRequest.result;
        const transaction = db.transaction('keyvaluepairs');
        const objectStore = transaction.objectStore('keyvaluepairs');
        const cursor = objectStore.openCursor();

        const queryResult: Record<string, unknown> = {};

        cursor.onerror = () => {
            console.error('Error reading cursor');
        };

        cursor.onsuccess = (event) => {
            const {result} = event.target as IDBRequest<IDBCursorWithValue>;
            if (result) {
                queryResult[result.primaryKey as string] = result.value;
                result.continue();
            } else {
                resolve(queryResult);
            }
        };
    };
});

SQLite on mobile

Here's the implementation of reading data from Onyx stored in the SQLite on native devices (using react-native-quick-sqlite we already use in the App) in the index.native.ts file:

const readFromIndexedDB = () => new Promise((resolve) => {
    const db = open({name: 'OnyxDB'});
    const query = 'SELECT * FROM keyvaluepairs';

    db.executeAsync(query, []).then(({rows}) => {
        // eslint-disable-next-line no-underscore-dangle
        const result = rows?._array.map((row) => ({[row.record_key]: JSON.parse(row.valueJSON as string)}));

        resolve(result);
        db.close();
    });
});

Masking fragile data

Now that we have date retrieved from the Onyx database we need to mask some fragile data such as authToken or emails. Do to it, we can simply iterate over the keys of retrieved object, check if its the fragile data and replace it with the *** string. Let's add another method to the ExportOnyxState lib (index.ts):

const maskFragileData = (data: Record<string, unknown>): Record<string, unknown> => {
    const maskedData: Record<string, unknown> = {};

    for (const key in data) {
        if (Object.prototype.hasOwnProperty.call(data, key)) {
            const value = data[key];
            if (typeof value === 'string' && (Str.isValidEmail(value) || key === 'authToken' || key === 'encryptedAuthToken')) {
                maskedData[key] = '***';
            } else if (typeof value === 'object') {
                maskedData[key] = maskFragileData(value as Record<string, unknown>);
            } else {
                maskedData[key] = value;
            }
        }
    }

    return maskedData;
};

As we will use the same code for both web and native in this case, can simply import the above method in the index.native.ts and export it with no modifications.

Sharing result as `*.txt` file

The last thing is to share the retrieved data as a text file. We also need to do it in a different way for web and for mobile.

Web

For web implementation we can simply use the download attribute of the a element that can be temporarily added to the DOM:

// eslint-disable-next-line @lwc/lwc/no-async-await,@typescript-eslint/require-await
const shareAsFile = async (value: string) => {
    const element = document.createElement('a');
    element.setAttribute('href', `data:text/plain;charset=utf-8,${encodeURIComponent(value)}`);
    element.setAttribute('download', 'onyx-state.txt');

    element.style.display = 'none';
    document.body.appendChild(element);

    element.click();

    document.body.removeChild(element);
};

Mobile

For mobile implementation we can use react-native-fs and react-native-share libraries that we already use in the App:

// eslint-disable-next-line @lwc/lwc/no-async-await
const shareAsFile = async (value: string) => {
    try {
        // Define new filename and path for the app info file
        const infoFileName = `onyx-state.txt`;
        const infoFilePath = `${RNFS.DocumentDirectoryPath}/${infoFileName}`;
        const actualInfoFile = `file://${infoFilePath}`;

        await RNFS.writeFile(infoFilePath, value, 'utf8');

        const shareOptions = {
            urls: [actualInfoFile],
        };

        Share.open(shareOptions);
    } catch (error) {
        console.error('Error renaming and sharing file:', error);
    }
}

Wrapping things up in the `exportOnyxState` method

At last, we need to utilize all the methods created above in the exportOnyxState method in the TroubleshootPage.tsx:

const exportOnyxState = () => {
    ExportOnyxState.readFromOnyxDatabase().then((value: Record<string, unknown>) => {
        const maskedData = ExportOnyxState.maskFragileData(value);

        ExportOnyxState.shareAsFile(JSON.stringify(maskedData));
    });
};

What alternative solutions did you explore? (Optional)

n/a

Reminder: Please use plain English, be brief and avoid jargon. Feel free to use images, charts or pseudo-code if necessary. Do not post large multi-line diffs or write walls of text. Do not create PRs unless you have been hired for this job.

melvin-bot[bot] commented 2 months ago

Job added to Upwork: https://www.upwork.com/jobs/~014fd561267f73f88e

melvin-bot[bot] commented 2 months ago

Triggered auto assignment to Contributor-plus team member for initial proposal review - @ZhenjaHorbach (External)

ZhenjaHorbach commented 2 months ago

@burczu 's proposal looks cool 💪 🎀👀🎀 C+ reviewed

melvin-bot[bot] commented 2 months ago

Triggered auto assignment to @hayata-suenaga, see https://stackoverflow.com/c/expensify/questions/7972 for more details.

hayata-suenaga commented 2 months ago

@muttmuure, thank you for creating this issue 😄

I have two questions:

Who will be reviewing this txt file? Will it be internal Expensify engineers or contributors?
Who will generate and share this file? Will they be internal employees, contributors, or regular users who receive instructions from Concierge on how to do so?

muttmuure commented 2 months ago

Who will be reviewing this txt file? Will it be internal Expensify engineers or contributors?

Expensify engineers and contributors will review the txt file in #newdot-quality to debug UX and platform inconsistency issues caused by the onyx state.

Who will generate and share this file? Will they be internal employees, contributors, or regular users who receive instructions from Concierge on how to do so?

Anyone can do it. We discussed adding a warning that the data could contain sensitive information - do you think that would be useful here?

hayata-suenaga commented 2 months ago

Thank you for the detailed answer 😄 Could I ask one last question?

How are we expecting the user to send the text file? Are they going to download the text file on their device and then send it over a chat on New Expensify to Concierge?

(I guess internal engineers and contributors can send them through Slack or post them on GitHub)?

ZhenjaHorbach commented 2 months ago

@muttmuure @hayata-suenaga Sorry for the misunderstanding 😅 But can C+ review the PR? Or do we still need to discuss the details?

hayata-suenaga commented 2 months ago

@ZhenjaHorbach, you can review the PR. To move this issue ahead, I'm assigning @burczu. We can discuss the details in the PR if necessary 🚀

hayata-suenaga commented 2 months ago

@burczu, please raise a PR when you have time 🙇

ZhenjaHorbach commented 2 months ago

@burczu, please raise a PR when you have time 🙇

Cool thank you 😀 I'll do a review today in this case

We already have PR https://github.com/Expensify/App/pull/43552

burczu commented 2 months ago

@ZhenjaHorbach Please hold on now - there are some things I need to amend in the PR to reflect some changes in requirements:

masking fragile data should be optional, so I need to implement toggle button to allow enabling/disabling it while generating the text file
only messages inside the report action should be masked instead filtering out all the report actions as its done now

I'll let you know when the changes are ready and you could start your review then.

burczu commented 2 months ago

@ZhenjaHorbach I think the PR is ready to review now.

blimpich commented 2 months ago

@hayata-suenaga @ZhenjaHorbach @burczu can you handle this issue as follow up for this issue/PR?

ZhenjaHorbach commented 1 month ago

@hayata-suenaga @ZhenjaHorbach @burczu can you handle this issue as follow up for this issue/PR?

@burczu Could you create a PR when you are free, please ?

melvin-bot[bot] commented 1 month ago

Reviewing label has been removed, please complete the "BugZero Checklist".

melvin-bot[bot] commented 1 month ago

The solution for this issue has been :rocket: deployed to production :rocket: in version 9.0.5-13 and is now subject to a 7-day regression period :calendar:. Here is the list of pull requests that resolve this issue:

https://github.com/Expensify/App/pull/43552

If no regressions arise, payment will be issued on 2024-07-17. :confetti_ball:

For reference, here are some details about the assignees on this issue:

@burczu does not require payment (Contractor)
@ZhenjaHorbach requires payment (Needs manual offer from BZ)

melvin-bot[bot] commented 1 month ago

The solution for this issue has been :rocket: deployed to production :rocket: in version 9.0.6-8 and is now subject to a 7-day regression period :calendar:. Here is the list of pull requests that resolve this issue:

https://github.com/Expensify/App/pull/43552

If no regressions arise, payment will be issued on 2024-07-22. :confetti_ball:

For reference, here are some details about the assignees on this issue:

@burczu does not require payment (Contractor)
@ZhenjaHorbach requires payment (Needs manual offer from BZ)

ZhenjaHorbach commented 1 month ago

BugZero Checklist: The PR fixing this issue has been merged! The following checklist (instructions) will need to be completed before the issue can be closed:

[x] [@ZhenjaHorbach] The PR that introduced the bug has been identified. Link to the PR:

Not a regression. This is a new feature.

[x] [@ZhenjaHorbach] The offending PR has been commented on, pointing out the bug it caused and why, so the author and reviewers can learn from the mistake. Link to comment:

Not a regression. This is a new feature.

[x] [@ZhenjaHorbach] A discussion in #expensify-bugs has been started about whether any other steps should be taken (e.g. updating the PR review checklist) in order to catch this type of bug sooner. Link to discussion:

NA

[x] [@ZhenjaHorbach] Determine if we should create a regression test for this bug.

[x] [@ZhenjaHorbach] If we decide to create a regression test for the bug, please propose the regression test steps to ensure the same bug will not reach production again.

Regression Test Steps

Log in to the App
Navigate to Accout Settings -> Troubleshoot section
Press the "Export Onyx state" button
Wait for the file to be generated and saved (you may be asked to decide where to save the file)
Check the content of the saved file (it should contain the Onyx state as a string with emails and authTokens masked by "***" string)

Do we agree 👍 or 👎?

ZhenjaHorbach commented 1 month ago

Not overdue Waiting for payment !

melvin-bot[bot] commented 1 month ago

Payment Summary

Upwork Job

Contributor: @burczu is from an agency-contributor and not due payment
ROLE: @ZhenjaHorbach paid $(AMOUNT) via Upwork (LINK)

BugZero Checklist (@muttmuure)

[x] I have verified the correct assignees and roles are listed above and updated the neccesary manual offers
[x] I have verified that there are no duplicate or incorrect contracts on Upwork for this job (https://www.upwork.com/ab/applicants/1803048668106098615/hired)
[x] I have paid out the Upwork contracts or cancelled the ones that are incorrect
[x] I have verified the payment summary above is correct

melvin-bot[bot] commented 1 month ago

@burczu, @muttmuure, @ZhenjaHorbach, @hayata-suenaga 6 days overdue. This is scarier than being forced to listen to Vogon poetry!

muttmuure commented 1 month ago

Handling

muttmuure commented 1 month ago

Invited @ZhenjaHorbach

ZhenjaHorbach commented 1 month ago

Invited @ZhenjaHorbach

Done !

Christinadobrzyn commented 1 month ago

hey I have this GH https://github.com/Expensify/App/issues/44841 that's also ready for payment.

The PRs are different but is this the same job? I've already paid out @etCoderDysto in this GH for this PR.

Should we pay @ZhenjaHorbach once for both jobs or are these separate jobs?

@burczu @hayata-suenaga

hayata-suenaga commented 1 month ago

@burczu ?

muttmuure commented 1 month ago

It should be one payment since this was the original issue and the linked issue is a follow-up/polish issue caught in testing

muttmuure commented 1 month ago

Paid

Expensify / App