[Regression test required] [$500] HIGH: Add `Avatar > About > Troubleshooting` section

[quinthar]

Strategy: A billion users means a billion problems. To diagnose and solve them at scale, we capture client-side logs and upload centrally.

Problem: Some problems can't be diagnosed cleanly after the fact from the logs, for a few reasons:

• External contributors don't have access to production logs.
• Internal engineers do have production access, but they're often cumbersome to access.
• The logs are often ambiguous, such as if you have multiple devices connected and tabs open at the same time -- reverse engineering what happened for a particular client is confusing and time consuming.
• Observing logs in realtime is cumbersome or sometimes impossible, which slows down and complicates testing.
• Even if you have all the access you need, when talking with a user sometimes it's difficult to find the logs specific to them.

Solution: Let's expand our ability to remotely diagnose production devices with better dev tools built into the app itself. Specifically:

• Add a Troubleshooting section to the Avatar > About page, just above Report a bug
  • This will always be available, both on staging and production builds
• Move the following Test preferences settings into this new menu:
  • Use staging server
  • Force offline
  • Simulate failing network requests
  • Authentication status: Invalidate
  • Device credentials: Destroy
• Add Reset and refresh, which wipes Onyx clean except for the minimum needed to call OpenApp (so you don't need to reauthenticate)
• Add View console which opens a simple console:
  • This would first open up and initialize with all locally stored logs, so you can scroll back and see what happened
  • It would also append new logs in realtime
  • There would also be a Save button that just copies all available logs into a text file that is saved on the device
  • Add a Share button that brings up a Search selector and just posts the logs into the selected room (and then leaves the user navigated there, so they can talk about them). I'm imagining this would be useful for chatting with a real world user and asking them to do something and send you the logs.
  • At the bottom is a command line that works similar to the Chrome console, where you can execute arbitrary JavaScript, with the results output to the console

This is blocking https://github.com/Expensify/App/issues/33040.

Upwork Automation - Do Not Edit

• Upwork Job URL: https://www.upwork.com/jobs/~01c2a08b8b8ae90e21
• Upwork Job ID: 1744202378047897600
• Last Price Increase: 2024-01-15
• Automatic offers:
• fedirjh | Reviewer | 28115102

[melvin-bot[bot]]

Job added to Upwork: https://www.upwork.com/jobs/~01c2a08b8b8ae90e21

[melvin-bot[bot]]

Triggered auto assignment to @greg-schroeder (NewFeature), see https://stackoverflowteams.com/c/expensify/questions/14418#:~:text=BugZero%20process%20steps%20for%20feature%20requests for more details.

[melvin-bot[bot]]

Triggered auto assignment to Contributor-plus team member for initial proposal review - @fedirjh (External)

[kmwamasali]

Proposal

Please re-state the problem that we are trying to solve in this issue.

External contributors don't have access to production logs. Internal engineers do have production access, but they're often cumbersome to access. The logs are often ambiguous, such as if you have multiple devices connected and tabs open at the same time -- reverse engineering what happened for a particular client is confusing and time consuming. Observing logs in realtime is cumbersome or sometimes impossible, which slows down and complicates testing. Even if you have all the access you need, when talking with a user sometimes it's difficult to find the logs specific to them.

What is the root cause of that problem?

What changes do you think we should make in order to solve the problem?

Expand app ability to remotely diagnose production devices with better dev tools built into the app itself. Specifically:

Add a Troubleshooting section to the Avatar > About page, just above Report a bug This will always be available, both on staging and production builds Move the following Test preferences settings into this new menu: Use staging server Force offline Simulate failing network requests Authentication status: Invalidate Device credentials: Destroy Add Reset and refresh, which wipes Onyx clean except for the minimum needed to call OpenApp (so you don't need to reauthenticate) Add View console which opens a simple console: This would first open up and initialize with all locally stored logs, so you can scroll back and see what happened It would also append new logs in realtime There would also be a Save button that just copies all available logs into a text file that is saved on the device Add a Share button that brings up a Search selector and just posts the logs into the selected room (and then leaves the user navigated there, so they can talk about them). I'm imagining this would be useful for chatting with a real world user and asking them to do something and send you the logs. At the bottom is a command line that works similar to the Chrome console, where you can execute arbitrary JavaScript, with the results output to the console

[shubham1206agra]

Can I directly work on this without a proposal?

[Tony-MK]

Proposal

Please re-state the problem that we are trying to solve in this issue.

Some problems can't be diagnosed cleanly after the fact from the logs.

What is the root cause of that problem?

The settings page is not optimal for diagnosing.

What changes do you think we should make in order to solve the problem?

Troubleshoot

1) In the AboutPage, create a menu item titled Troubleshoot above the ReportBug menu item on this line Sample Component

 { 
     translationKey: 'initialSettingsPage.aboutPage.troubleshoot', 
     icon: Expensicons.Troubleshoot, 
     action: waitForNavigate(Report.navigateToTroubleshoot), 
 },

Test preferences

1) Add a new menu item called Test preferences to the settings page. 2) Create a menu item on the settings menu for the new page. 3) Move the necessary menu items (use staging server, force offline, etc..) to the new page.

Reset and refresh

1) Add another item in the Test preferences page called Reset and refresh. 2) Set its action to a created function to refresh the page after setting every onyx collection, except for data required to call the OpenApp endpoint, to null.

View console

1) Add a new menu item called View console to the Test preferences page. 3) Bind on console.log, console.info, console.info, etc... depending on the type of messages we want to collect, then render before/after calling console.stdlog.apply. 2) Create a component to get the logs from the console and render them. 4) Create an input component for the js script and execute the input script using the inbuilt eval function. I recommend a feature to access the history of input scripts and replay if desired. 5) Add a share button on the header component to take the required logs and send them to the appropriate channels.

[quinthar]

Please submit proposals, especially for how you intend to do the console feature. The more detail you provide, the greater likelihood we select your proposal.

On Sun, Jan 7, 2024, 7:57 PM Antony Kithinzi @.***> wrote:

Proposal Please re-state the problem that we are trying to solve in this issue.

Some problems can't be diagnosed cleanly after the fact from the logs, for a few reasons What is the root cause of that problem?

The settings page is not optimal for diagnosing. What changes do you think we should make in order to solve the problem?

Add a new page in settings called Test preferences. Then move the necessary components to the new page. Create a menu item on the settings menu for the page.

Then add another item in Test preferences page called Reset and refresh. This option will clear the onyx collection except for the auth data. Finally, it will refresh the page.

— Reply to this email directly, view it on GitHub https://github.com/Expensify/App/issues/34082#issuecomment-1880353158, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAEMNUQ4EYCWKZ7HZ4VAK63YNNVEDAVCNFSM6AAAAABBQ2U4NSVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTQOBQGM2TGMJVHA . You are receiving this because you authored the thread.Message ID: @.***>

[samsonmobisa]

This is a thoughtful and comprehensive proposal that tackles the challenge that's encountered in diagnosing and solving issues at scale. I appreciate the clarity in the suggested changes, especially the addition of the Troubleshooting section and the integration of better dev tools within the app.

The proposed enhancements, such as the Console View, Reset and Refresh option, and the ability to Share Logs, would significantly improve the ability to remotely diagnose production devices. I particularly like the idea of having a command line for executing JavaScript, resembling the Chrome console.

I'm eager to contribute to the implementation of these features. Before diving in, is there any specific technical considerations or preferences regarding the implementation? Additionally, if there are any specific testing scenarios or edge cases to keep in mind, please provide guidance.

Looking forward to collaborating on this improvement

[melvin-bot[bot]]

📣 @samsonmobisa! 📣 Hey, it seems we don’t have your contributor details yet! You'll only have to do this once, and this is how we'll hire you on Upwork. Please follow these steps:

1. Make sure you've read and understood the contributing guidelines.
2. Get the email address used to login to your Expensify account. If you don't already have an Expensify account, create one here. If you have multiple accounts (e.g. one for testing), please use your main account email.
3. Get the link to your Upwork profile. It's necessary because we only pay via Upwork. You can access it by logging in, and then clicking on your name. It'll look like this. If you don't already have an account, sign up for one here.
4. Copy the format below and paste it in a comment on this issue. Replace the placeholder text with your actual details. Format:

   Contributor details
   Your Expensify account email: <REPLACE EMAIL HERE>
   Upwork Profile Link: <REPLACE LINK HERE>

[Tony-MK]

I have some questions about the View console expected result.

1) Should the View console page be full screen because long lines of code will be displayed and the console input component be wide enough to type in scripts?

2) Should we have a separate output log component to differentiate the output from the user-performed scripts? Another option is to color or tag it differently from the rest of the application's console log.

3) Are we going to preserve logs after the window refreshes?

[samsonmobisa]

Contributor details Your Expensify account email: samsonmobisa62@gmail.com Upwork Profile Link: https://www.upwork.com/freelancers/~018e78ebb343f52495

[melvin-bot[bot]]

✅ Contributor details stored successfully. Thank you for contributing to Expensify!

[samsonmobisa]

Have already submitted my details

[Piyush-Kumar-Ghosh]

Problem Overview: The current challenge involves diagnosing issues at scale, where existing logs may not provide a clear understanding of problems. External contributors lack access to production logs, and even for internal engineers with access, retrieving logs can be cumbersome and time-consuming. The logs' ambiguity, coupled with difficulties in observing real-time logs, further complicates issue resolution.

Proposed Changes:

1. Troubleshooting Section:

   • Add a Troubleshooting section to the Avatar > About page, above Report a bug.
   • Always available on staging and production builds.

2. Test Preferences Settings:

   • Move Test preferences settings to the Troubleshooting menu.
   • Settings include Use staging server, Force offline, Simulate failing network requests, Invalidate Authentication status, and Destroy Device credentials.

3. Reset and Refresh:

   • Introduce a Reset and refresh option in Troubleshooting.
   • Wipes Onyx clean, retaining only essentials for OpenApp calls, avoiding ReAuthentication.

4. View Console:

   • Add a View console option in Troubleshooting.
   • Opens a console with locally stored logs, allowing real-time updates.
   • Includes a Save button for exporting logs to a text file on the device.

5. Share Logs:

   • Include a Share button in Troubleshooting.
   • Facilitates posting logs into a selected room for collaboration.
   • Enhances communication with users for issue resolution.

6. Command Line:

   • Integrate a command line at the bottom of the console.
   • Allows execution of arbitrary JavaScript, akin to the Chrome console.

Implementation Details:

1. Troubleshooting sections should have a clear UI design, ensuring easy navigation.
2. Test preferences settings must seamlessly transition to the Troubleshooting menu.
3. Reset and refresh should be implemented efficiently to maintain essential data.
4. View consoles should offer a user-friendly interface, supporting both existing and real-time logs.
5. Share logs feature should provide options for selecting rooms and streamline the collaboration process.
6. Command line functionality should resemble the Chrome console, with proper error handling.

Expected Results:

1. Improved accessibility for external contributors and internal engineers.
2. Enhanced clarity in log interpretation and diagnosis.
3. Streamlined troubleshooting process with efficient tools.
4. Real-time log observation for quicker issue resolution.
5. Seamless collaboration through log sharing.

I would appreciate your feedback and insights on this proposal. Looking forward to discussing this further.

[melvin-bot[bot]]

📣 @Piyush-Kumar-Ghosh! 📣 Hey, it seems we don’t have your contributor details yet! You'll only have to do this once, and this is how we'll hire you on Upwork. Please follow these steps:

1. Make sure you've read and understood the contributing guidelines.
2. Get the email address used to login to your Expensify account. If you don't already have an Expensify account, create one here. If you have multiple accounts (e.g. one for testing), please use your main account email.
3. Get the link to your Upwork profile. It's necessary because we only pay via Upwork. You can access it by logging in, and then clicking on your name. It'll look like this. If you don't already have an account, sign up for one here.
4. Copy the format below and paste it in a comment on this issue. Replace the placeholder text with your actual details. Format:

   Contributor details
   Your Expensify account email: <REPLACE EMAIL HERE>
   Upwork Profile Link: <REPLACE LINK HERE>

[Piyush-Kumar-Ghosh]

Contributor details Your Expensify account email: piyushkumarghosh1@gmail.com Upwork Profile Link: https://www.upwork.com/freelancers/~017087736e94a4f9b8

[melvin-bot[bot]]

✅ Contributor details stored successfully. Thank you for contributing to Expensify!

[b4s36t4]

Proposal

Please re-state the problem that we are trying to solve in this issue.

HIGH: Add Avatar > About > Troubleshooting section

What is the root cause of that problem?

NA - Feature request.

What changes do you think we should make in order to solve the problem?

1. Creating a new Menu i.e TroubleShooting.

We already have About page in which we show multiple menu's under it, we should use MenuItem component to render the new TroubleShoot item.

Once user clicks on it, use Navigation to navigate to the troubleshoot page.

2. Moving Test Preferences to TroubleShoot page, since this Troubleshoot will be a new page, we need to move MenuItem's from the Preferences page.

3. Reset and Refresh, we only need to wipe the onyx keys which doesn't require the user to be login again, this means we need to remove the data inside onyx that are not needed.

The majority of data we store are related to reports, transactions, reportActions, policy, reportDraftComment, network.

To Simply say we should be having onyx data related to following keys account, session, credentials, loginList, user, after clearing the keys we should do refresh i.e window.location.reload in browser and some native code for native devices to restart the app. Maybe using react-native-restart.

4. View console

   1. Since console might contain a big chunk of lines i.e in real scenario people can be using the application for days where logs can be in 1000s of lines. We have logs stored in Log.logLines which we can use to render logs.

   2. How, rendering well the logs are large list, which can block the low-end browsers to hang and doesn't give good UX. We should be using some virtualized list to render the logs, I prefer react-virtual for web and react-native-fast-list for native devices or we can simply use the SectionList that we're using to render the reportActions.

   3. Realtime logs rendering, well at the current code we cannot do with the current implementation, to resolve it we should be having some realtime i.e a variable that causes re-render with the code.

   4. Saving, we should be generating a blob with type text/plain and download it using native tools, i.e for web we should be using url.createObjectURL and same implementation or react-native-fs to create the file.

   5. Sharing, when we click on Share we should snapshot the current logs (not new logs, because we're thinking about the current logs) and use the basic logic to share the logs.

Implementation specifics / issues

Realtime rendering is the main issue here which will require some observable implementation.

Why? we're not saving logLines in the react specific code which doesn't let us use some react specific sections i.e useState or so. While figuring out the solution, I found the following package https://jotai.org/docs/guides/react-native which doesn't need any react specific code to create observable and can also be used with react that gives the ability to cause re-renders. So we should be creating a atom that stores the logs and should be exported and used in App codebase.

We have a limit of 10000 characters inside composer, but we don't know that logLines does contain the 10000 characters which will block the implementation of Share feature. Maybe we can wrap the logs in a file and share it with the selected chat. @quinthar you're view on this?

What alternative solutions did you explore? (Optional)

NA

[samsonmobisa]

Hello Expensify Team

Your comprehensive plan demonstrates a deep understanding of the challenges at hand and offers a thoughtful and well-structured solution. I particularly appreciate the attention to detail in addressing each aspect of the problem. The proposed Troubleshooting section, along with the seamless integration of Test Preferences and the Reset and Refresh feature, reflects a strategic approach to streamline user support and enhance debugging processes.

The innovative use of observable atoms from the jotai package for real-time log rendering showcases a keen problem-solving mindset. Your consideration of potential issues, such as the character limit for sharing logs, reveals a thorough analysis of the implementation specifics.

This proposal not only tackles the immediate problem but also reflects a commitment to ensuring a robust, user-friendly application. I'm impressed by the holistic approach, covering everything from the user interface to the technical intricacies of real-time rendering.

I am ready and eager to collaborate on implementing these enhancements. Your feedback and guidance will be invaluable as we work together to create a more efficient and reliable workflow.

[quinthar]

What's the next step? Who is doing it and when will it be done?

[Piyush-Kumar-Ghosh]

According to guidelines, All the proposals are reviewed by the Core Team from which one gets selected and then needs to do some follow up steps.

After that the contributor starts the work.

I haven't contributed to this project before so I would love to work on it.

[melvin-bot[bot]]

@greg-schroeder, @fedirjh Uh oh! This issue is overdue by 2 days. Don't forget to update your issues!

[greg-schroeder]

Bump @fedirjh for proposal review

[melvin-bot[bot]]

📣 It's been a week! Do we have any satisfactory proposals yet? Do we need to adjust the bounty for this issue? 💸

[melvin-bot[bot]]

@greg-schroeder, @fedirjh Whoops! This issue is 2 days overdue. Let's get this updated quick!

[melvin-bot[bot]]

@greg-schroeder, @fedirjh Huh... This is 4 days overdue. Who can take care of this?

[samsonmobisa]

Contributor details Your Expensify account email: samsonmobisa62@gmail.com Upwork Profile Link: https://www.upwork.com/freelancers/~018e78ebb343f52495

[melvin-bot[bot]]

✅ Contributor details stored successfully. Thank you for contributing to Expensify!

[greg-schroeder]

bump @fedirjh

[quinthar]

Bump, what's the next step here? I think @TMisiukiewicz has already volunteered. Who are we waiting on? What's the next step, and when will it happen?

[TMisiukiewicz]

As discussed with @muttmuure in #33040 , I want to take this over and start working on a solution. Can I get assigned to this issue?

[melvin-bot[bot]]

📣 @fedirjh 🎉 An offer has been automatically sent to your Upwork account for the Reviewer role 🎉 Thanks for contributing to the Expensify app!

Offer link Upwork job

[quinthar]

@TMisiukiewicz sounds great, go for it!

[TMisiukiewicz]

Quick update from my side:

• created Troubleshooting page and linked it with routing
• created "Reset and refresh" and "View console" menu items
• moved "Test preferences" section to "Troubleshooting" screen
• created basic implementation for "Reset and refresh", it works most of the times but I'm still figuring out what exact Onyx keys should be preserved from cleanining.

Do we have any design for the new page? If we want to follow the design of other pages, what i need to know is:

• icon to use with "Troubleshooting" menu item
• lottie illustration to use in the page header
• icon for "Reset and refresh" menu item
• icon for "View console" menu item
• design for console view (?)

[fedirjh]

cc @Expensify/design

[dubielzyk-expensify]

Unsure exactly what the console view looks like, but here's my take on how we could design the page. cc @Expensify/design

[shawnborton]

I love that! I think that's a great illustration choice, and I actually think we have an animated version of it where the planet in the background rotates and the screen flashes: https://www.dropbox.com/scl/fi/k6n1k21839t4gs6m8s3x5/ED-workspaces-1-030323-kjs-4.gif?rlkey=f4iqy7paop13448l4dylrc92h&dl=0

Small nitpick, but I think the top page header would stay with a blue background even as content on the page scrolls. Check the existing examples in product to see how that works.

[TMisiukiewicz]

Thanks for the design! For those kind of illustrations we use Lottie in the app - is there any way to get it as a .lottie file?

Unsure exactly what the console view looks like

I think it could be something similar to the console in the browser, just matching the mobile view and some other requirements mentioned above

https://github.com/Expensify/App/assets/13985840/9aedd9e1-579b-4111-bbdc-450486990aef

[shawnborton]

@TMisiukiewicz try this for the animation: https://www.dropbox.com/scl/fi/s1r6oty2q20fyydl148e6/ED-workspaces-1-030323-kjs-4.json?rlkey=f3hwuv9i5xcjlzuhynp0dwsrb&dl=0

[TMisiukiewicz]

Works nicely, thank you! 👍

[TMisiukiewicz]

I also need Spanish translations for the following phrases:

• Troubleshoot
• Reset and refresh
• View console
• Use the tools below to help troubleshoot the Expensify experience. If you encounter any issues, please submit a bug

For wiping the Onyx state, I preserve the Onyx keys that seem to be static or relevant to user account: account, currency_list, device_id, isFirstTimeNewExpensifyUser, isLoadingApp, isSidebarLoaded, loginList, nvp_priorityMode, preferredLocale, preferredTheme, session, tryFocusMode, user, userWallet. Can someone who is more experienced with Onyx take a look and verify if I can safely remove something from this list, or if there are any missing keys I need to add?

Can we split work on this issue into 2 PRs? First would introduce the Troubleshooting page and "Reset and refresh" feature and second one would contain the console. The console feature will probably take some time, so this way we could ship the "Reset and refresh" feature faster

[muttmuure]

cc @cead22 @pecanoro

[pecanoro]

@muttmuure Please, follow this if you need some translations! https://stackoverflowteams.com/c/expensify/questions/10316/10317

[muttmuure]

Oops, sorry!

[dubielzyk-expensify]

Small nitpick, but I think the top page header would stay with a blue background even as content on the page scrolls. Check the existing examples in product to see how that works.

My bad on the explanation. It was actually a variation with and without an illustration. Not meant to be a scrolled version 😅 But yeah we should do what you said

[shawnborton]

Ah got it, all good and sounds good to me!

[shawnborton]

For this screenshot here though, looks like it doesn't quite match your mockup @dubielzyk-expensify - I think you are using blue600 or blue700 as the BG?

[TMisiukiewicz]

@shawnborton Good catch, thanks!

Additionally, what icon should I use for Troubleshoot menu item? The one you see on the screenshot below is a placeholder for now

[shawnborton]

Hmm good question. Curious what the @Expensify/design thinks but perhaps a lightbulb or the "!" (exclamation) icon could work?

[dannymcclain]

I like the lightbulb or the "!" icon. I lean toward the "!" but I want to make sure it doesn't feel too alarming haha.