Closed muttmuure closed 1 month ago
Auto-assigning issues to engineers is no longer supported. If you think this issue should receive engineering attention, please raise it in #whatsnext.
Huh... This is 4 days overdue. Who can take care of this?
Hi, I am Bartek from Callstack - expert contributor group. I would like to work on this issue.
Daily update: I’ve been working on the proposal for this issue today, mostly on finding the way to collect the whole onyx snapshot as a string (to be able to save it as a txt file later), I hope I’ll be able to post something by the end of tomorrow - I’ll keep you posted. I’ve also created a draft PR in GH.
Daily update: I think I’ve found the way to collect all the Onyx data for web environment and already tested it; for native I think I’ve also find the way but it still needs testing. I believe the proposal is behind the corner and I’ll be able to post it tomorrow.
Daily update: the solution for collecting the Onyx data and saving as a text file is done and can be checked in the draft PR. I've forgotten that we also need to mask data such as authToken, emails etc, so this is the last thing I need to sort out.
@burczu Whoops! This issue is 2 days overdue. Let's get this updated quick!
not overdue
When your mobile client's Onyx state falls out of sync with web, it is hard to diagnose which value has been missed or has been incorrectly inserted, especially when a refresh, sign in, or reload of the app on the client can solve the issue.
There is no specific problem - this is more like a new feature. We want to add a "Export Onyx State" option in the Troubleshoot
menu in the App. Pressing it should download the values in the Onyx keyvaluepairs
table as a *.txt file.
Troubleshoot
menuFirst we need to add the new button in the Troubleshoot
section. To do so, we need to add new translation (initialSettingsPage.troubleshoot.exportOnyxState
) in en.ts
and es.ts
files
exportOnyxState: 'Export Onyx state',
exportOnyxState: 'Exportar estado Onyx',
Next, we need to add the button in the /src/pages/settings/Troubleshoot/TroubleshootPage.tsx
, here:
The code after addition:
const baseMenuItems: BaseMenuItem[] = [
{
translationKey: 'initialSettingsPage.troubleshoot.clearCacheAndRestart',
icon: Expensicons.RotateLeft,
action: () => setIsConfirmationModalVisible(true),
},
{
translationKey: 'initialSettingsPage.troubleshoot.exportOnyxState',
icon: Expensicons.Download,
action: exportOnyxState,
},
];
The exportOnyxState
action method implementation will be described in the last step.
Next, we need to add the code that will be responsible for reading data from Onyx's keyvaluepairs
table. We need to maintain two different implementations for web and mobile, because on web we use the IndexedDB
database, and on mobile we use the SQLite
database.
To do it, we need to create the ExportOnyxState
folder inside /src/libs
, and create two files there: index.ts
that will be responsible for web implementation and index.native.ts
for mobile implementation.
Here's the implementation of reading data from Onyx stored in the IndexedDB on web (using IDBDatabase
available in the browser) in the index.ts
file:
const readFromIndexedDB = () =>
new Promise<Record<string, unknown>>((resolve) => {
let db: IDBDatabase;
const openRequest = indexedDB.open('OnyxDB', 1);
openRequest.onsuccess = () => {
db = openRequest.result;
const transaction = db.transaction('keyvaluepairs');
const objectStore = transaction.objectStore('keyvaluepairs');
const cursor = objectStore.openCursor();
const queryResult: Record<string, unknown> = {};
cursor.onerror = () => {
console.error('Error reading cursor');
};
cursor.onsuccess = (event) => {
const {result} = event.target as IDBRequest<IDBCursorWithValue>;
if (result) {
queryResult[result.primaryKey as string] = result.value;
result.continue();
} else {
resolve(queryResult);
}
};
};
});
Here's the implementation of reading data from Onyx stored in the SQLite on native devices (using react-native-quick-sqlite
we already use in the App) in the index.native.ts
file:
const readFromIndexedDB = () => new Promise((resolve) => {
const db = open({name: 'OnyxDB'});
const query = 'SELECT * FROM keyvaluepairs';
db.executeAsync(query, []).then(({rows}) => {
// eslint-disable-next-line no-underscore-dangle
const result = rows?._array.map((row) => ({[row.record_key]: JSON.parse(row.valueJSON as string)}));
resolve(result);
db.close();
});
});
Now that we have date retrieved from the Onyx database we need to mask some fragile data such as authToken
or emails
. Do to it, we can simply iterate over the keys of retrieved object, check if its the fragile data and replace it with the ***
string. Let's add another method to the ExportOnyxState
lib (index.ts
):
const maskFragileData = (data: Record<string, unknown>): Record<string, unknown> => {
const maskedData: Record<string, unknown> = {};
for (const key in data) {
if (Object.prototype.hasOwnProperty.call(data, key)) {
const value = data[key];
if (typeof value === 'string' && (Str.isValidEmail(value) || key === 'authToken' || key === 'encryptedAuthToken')) {
maskedData[key] = '***';
} else if (typeof value === 'object') {
maskedData[key] = maskFragileData(value as Record<string, unknown>);
} else {
maskedData[key] = value;
}
}
}
return maskedData;
};
As we will use the same code for both web and native in this case, can simply import the above method in the index.native.ts
and export it with no modifications.
*.txt
fileThe last thing is to share the retrieved data as a text file. We also need to do it in a different way for web and for mobile.
For web implementation we can simply use the download
attribute of the a
element that can be temporarily added to the DOM:
// eslint-disable-next-line @lwc/lwc/no-async-await,@typescript-eslint/require-await
const shareAsFile = async (value: string) => {
const element = document.createElement('a');
element.setAttribute('href', `data:text/plain;charset=utf-8,${encodeURIComponent(value)}`);
element.setAttribute('download', 'onyx-state.txt');
element.style.display = 'none';
document.body.appendChild(element);
element.click();
document.body.removeChild(element);
};
For mobile implementation we can use react-native-fs
and react-native-share
libraries that we already use in the App:
// eslint-disable-next-line @lwc/lwc/no-async-await
const shareAsFile = async (value: string) => {
try {
// Define new filename and path for the app info file
const infoFileName = `onyx-state.txt`;
const infoFilePath = `${RNFS.DocumentDirectoryPath}/${infoFileName}`;
const actualInfoFile = `file://${infoFilePath}`;
await RNFS.writeFile(infoFilePath, value, 'utf8');
const shareOptions = {
urls: [actualInfoFile],
};
Share.open(shareOptions);
} catch (error) {
console.error('Error renaming and sharing file:', error);
}
}
exportOnyxState
methodAt last, we need to utilize all the methods created above in the exportOnyxState
method in the TroubleshootPage.tsx
:
const exportOnyxState = () => {
ExportOnyxState.readFromOnyxDatabase().then((value: Record<string, unknown>) => {
const maskedData = ExportOnyxState.maskFragileData(value);
ExportOnyxState.shareAsFile(JSON.stringify(maskedData));
});
};
n/a
Reminder: Please use plain English, be brief and avoid jargon. Feel free to use images, charts or pseudo-code if necessary. Do not post large multi-line diffs or write walls of text. Do not create PRs unless you have been hired for this job.
Job added to Upwork: https://www.upwork.com/jobs/~014fd561267f73f88e
Triggered auto assignment to Contributor-plus team member for initial proposal review - @ZhenjaHorbach (External
)
@burczu 's proposal looks cool 💪 🎀👀🎀 C+ reviewed
Triggered auto assignment to @hayata-suenaga, see https://stackoverflow.com/c/expensify/questions/7972 for more details.
@muttmuure, thank you for creating this issue 😄
I have two questions:
txt
file? Will it be internal Expensify engineers or contributors?Expensify engineers and contributors will review the txt file in #newdot-quality to debug UX and platform inconsistency issues caused by the onyx state.
Anyone can do it. We discussed adding a warning that the data could contain sensitive information - do you think that would be useful here?
Thank you for the detailed answer 😄 Could I ask one last question?
How are we expecting the user to send the text file? Are they going to download the text file on their device and then send it over a chat on New Expensify to Concierge?
(I guess internal engineers and contributors can send them through Slack or post them on GitHub)?
@muttmuure @hayata-suenaga Sorry for the misunderstanding 😅 But can C+ review the PR? Or do we still need to discuss the details?
@ZhenjaHorbach, you can review the PR. To move this issue ahead, I'm assigning @burczu. We can discuss the details in the PR if necessary 🚀
@burczu, please raise a PR when you have time 🙇
@burczu, please raise a PR when you have time 🙇
Cool thank you 😀 I'll do a review today in this case
We already have PR https://github.com/Expensify/App/pull/43552
@ZhenjaHorbach Please hold on now - there are some things I need to amend in the PR to reflect some changes in requirements:
I'll let you know when the changes are ready and you could start your review then.
@ZhenjaHorbach I think the PR is ready to review now.
@hayata-suenaga @ZhenjaHorbach @burczu can you handle this issue as follow up for this issue/PR?
@hayata-suenaga @ZhenjaHorbach @burczu can you handle this issue as follow up for this issue/PR?
@burczu Could you create a PR when you are free, please ?
Reviewing
label has been removed, please complete the "BugZero Checklist".
The solution for this issue has been :rocket: deployed to production :rocket: in version 9.0.5-13 and is now subject to a 7-day regression period :calendar:. Here is the list of pull requests that resolve this issue:
If no regressions arise, payment will be issued on 2024-07-17. :confetti_ball:
For reference, here are some details about the assignees on this issue:
The solution for this issue has been :rocket: deployed to production :rocket: in version 9.0.6-8 and is now subject to a 7-day regression period :calendar:. Here is the list of pull requests that resolve this issue:
If no regressions arise, payment will be issued on 2024-07-22. :confetti_ball:
For reference, here are some details about the assignees on this issue:
BugZero Checklist: The PR fixing this issue has been merged! The following checklist (instructions) will need to be completed before the issue can be closed:
- [x] [@ZhenjaHorbach] The PR that introduced the bug has been identified. Link to the PR:
Not a regression. This is a new feature.
- [x] [@ZhenjaHorbach] The offending PR has been commented on, pointing out the bug it caused and why, so the author and reviewers can learn from the mistake. Link to comment:
Not a regression. This is a new feature.
- [x] [@ZhenjaHorbach] A discussion in #expensify-bugs has been started about whether any other steps should be taken (e.g. updating the PR review checklist) in order to catch this type of bug sooner. Link to discussion:
NA
[x] [@ZhenjaHorbach] Determine if we should create a regression test for this bug.
[x] [@ZhenjaHorbach] If we decide to create a regression test for the bug, please propose the regression test steps to ensure the same bug will not reach production again.
Regression Test Steps
Do we agree 👍 or 👎?
Not overdue Waiting for payment !
@burczu, @muttmuure, @ZhenjaHorbach, @hayata-suenaga 6 days overdue. This is scarier than being forced to listen to Vogon poetry!
Handling
Invited @ZhenjaHorbach
Invited @ZhenjaHorbach
Done !
hey I have this GH https://github.com/Expensify/App/issues/44841 that's also ready for payment.
The PRs are different but is this the same job? I've already paid out @etCoderDysto in this GH for this PR.
Should we pay @ZhenjaHorbach once for both jobs or are these separate jobs?
@burczu @hayata-suenaga
@burczu ?
It should be one payment since this was the original issue and the linked issue is a follow-up/polish issue caught in testing
Paid
If you haven’t already, check out our contributing guidelines for onboarding and email contributors@expensify.com to request to join our Slack channel!
What performance issue do we need to solve?
e.g. memory consumption, storage read/write times, React native bridge concerns, inefficient React component rendering, etc.
When your mobile client's Onyx state falls out of sync with web, it is hard to diagnose which value has been missed or has been incorrectly inserted, especially when a refresh, sign in, or reload of the app on the client can solve the issue.
What is the impact of this on end-users?
List specific user experiences that will be improved by solving this problem e.g. app boot time, time to for some interaction to complete, etc.
This is more of a debugging tool, since Onyx impacts on pretty much everything you see in app, it can impact on a multitude of end user functionality.
The most important end user impact of a debugging tool which exports the Onyx state is around privacy. We need to sanitize the authToken, emails and messages of the data since those are private and shouldn't be shared with anyone.
List any benchmarks that show the severity of the issue
Please also provide exact steps taken to collect metrics above if any so we can independently verify the results.
This is more of a debugging tool, so this isn't something that will show benchmarks.
Proposed solution (if any)
Please list out the steps you think we should take to solve this issue.
Add a "Export Onyx State" option to the Troubleshoot menu.
This downloads the values in the Onyx keyvaluepairs table to a txt file. The authTokens, emails and messages in the file are redacted
List any benchmarks after implementing the changes to show impacts of the proposed solution (if any)
Note: These should be the same as the benchmarks collected before any changes.
Platforms:
Which of our officially supported platforms is this issue occurring on?
Version Number: Reproducible in staging?: Reproducible in production?: Email or phone of affected tester (no customers): Logs: https://stackoverflow.com/c/expensify/questions/4856 Notes/Photos/Videos: Any additional supporting documentation Expensify/Expensify Issue URL: Issue reported by: Slack conversation:
View all open jobs on Upwork
Upwork Automation - Do Not Edit
Issue Owner
Current Issue Owner: @muttmuure