Benchmark and identify front end bottlenecks for loading reports

neil-marcellini commented 5 months ago

Problem

cc @roryabraham @neil-marcellini

As part of https://github.com/Expensify/Expensify/issues/305196 we need to look forward and identify problems that might still occur after server-side pagination is implemented. Specifically, we need to better understand:

What minimum device we will support (for each platform)
The upper bounds of what the devices can handle when we continuously load new reports onto the device.
We should keep in mind things like: memory, device storage, LHN rendering performance.

Why is this important?

Ultimately, we'd like to present a complete future-proof solution for pagination. Not considering device limitations leaves us less informed about the requirements of our design. e.g. our pagination system might work great on sign in, but may continue to irrecoverably degrade.

Solution

Let's work together with an external expert to answer a series of questions related to frontend performance in the context of report pagination.

We're thinking of doing a manual test where an increasing amount of reports are loaded into the app. 500, 1k, 5k, 10k, etc. Then we can use a profiler, and added code for timing/logging to understand where the bottle necks and limits are. Rather than focusing on optimizing right now, let's focus on gathering evidence and finding the limits to inform our design of report pagination.

Here are some additional questions we would like to have answered with evidence.

Which part of the app breaks/bottlenecks first when overloaded with reports? render performance, ram, storage?
What is the max report count the app can handle without noticeable slow downs?
How long would it take a user to accumulate that many reports with pagination, on average? Basically, for how long do you have to scroll down to load the max amount of reports?

For the external assignee, please let me know if you have questions or suggestions about the plan. Please also post a proposal of how you plan to test.

Issue Owner

Current Issue Owner: @neil-marcellini

melvin-bot[bot] commented 5 months ago

Triggered auto assignment to Contributor-plus team member for initial proposal review - @Ollyws (External)

neil-marcellini commented 5 months ago

We're discussing the plan here, and I'm going to assign Tomasz Lesniakiewicz from Callstack once he comments on this issue.

rinej commented 5 months ago

Hello, I am Tomasz from Callstack, please assign me to the issue. Thanks :)

rinej commented 5 months ago

TLDR: The app starts to lag between 200 and 500 standard reports

Problem Statement

We aim to identify the bottleneck in loading reports and determine after how many reports the app starts to lag and break. Our testing focuses on a real-life scenario using a real device without employing any testing libraries.

1. Approach and Limitations

Testing Device

Device: Mid-range Android - Samsung A34 6GB RAM/128GB

Testing Methodology

Report Creation: We created a function that uses buildOptimisticChatReport and runs openReport, which triggers API calls to generate real reports. Our goal was to create 'pure' reports without tasks, expenses, or attachments to simulate a standard user experience.
Limitations: This approach generates only standard reports without additional elements like expenses or tasks. This provides a rough estimate of when the app starts to lag, but it is not a comprehensive solution. For example, a user with 100 standard reports may experience better performance than a user with 50 complex reports. Please see the implementation getOrderedReportID, it is very complex and its performance depends on SO many factors. For this approach, we will create just normal reports to get rough estimates as fast as possible. Creating a more complex approach would be possible but would require much more effort.

2. Results

We tested the app by increasing the number of reports incrementally from 50 to 500. Here are the observations:

50 Reports: App functions normally.
100 Reports: App functions normally.
200 Reports: App functions normally.
500 Reports: App becomes laggy, reports do not fully load, and the app's startup time increases significantly (please see below videos)

Supporting Evidence

Videos:

**200 Reports (Normal Performance)**:

https://github.com/Expensify/App/assets/2650904/59fa93c0-1d64-4933-ab30-046d78c6e560

**500 Reports (Laggy Performance)**:

https://github.com/Expensify/App/assets/2650904/373a52ce-e2da-490b-ad55-c34e06aa3be9

Flashlight Results:

200 Reports (Normal Performance)

500 Reports (Laggy Performance)

Additional metrics

email	report length	App startup metrics TTI (ms)	Additional metric
tomasz.lesniakiewicz+100@callstack	75	12810	load_search_options: 365ms, search_render: 1435ms
tomasz.lesniakiewicz+100@callstack	100	11523	load_search_options 415ms, search_render 1624ms
tomasz.lesniakiewicz+100@callstack	200	12155	load_search_options 342ms, search_render 1731ms
tomasz.lesniakiewicz+100@callstack	500	22558, 13345, 90123!	search_render 3601ms, load_search_options 462ms

3. Summary

Based on our results, the app starts to lag between 200 and 500 reports on a mid-range Android device. However, this assessment is based solely on standard reports. Adding elements like personal details, more complex reports with tasks, expences or more report actions could significantly impact performance. Mimicking real-life scenarios for high-traffic accounts would be challenging.

Monitoring Recommendations

To better identify and manage performance issues, implementing a monitoring system would be great. Tools like Sentry can help detect users experiencing performance issues and we could check the number of personal details, reports, and other factors affecting performance. This would be invaluable for spotting and addressing performance issues

For further discussion on monitoring, please refer to the thread on monitoring with Sentry https://callstack-hq.slack.com/archives/C05LX9D6E07/p1715870874641849?thread_ts=1715377508.170269&cid=C05LX9D6E07

marcaaron commented 5 months ago

@rinej This is perfect, thanks! Do you think we could also produce a similar test for a mid range desktop + iOS device? I am guessing it would be better, but would be good to understand by how much.

neil-marcellini commented 5 months ago

Awesome work @rinej! Thank you. I especially like the additional metrics like load_search_options 342ms, search_render 1731ms. Would you please provide a little more info about how those are measured, and could you also measure similar metrics for the LHN specifically?

rinej commented 5 months ago

@neil-marcellini the additional metrics I took directly from what we already have in the code base I used: CHAT_FINDER_RENDER: 'search_render' and LOAD_SEARCH_OPTIONS: 'load_search_options',. Please have a look on the src/CONST.ts file and CONST.TIMING variable. There are other metric which we can check.

What do you mean metric for LHN? Do you mean loading the reports list on the LHN? If yes, I think that app startup metric would work for that case. It shows how long we need to wait fot the app to became usable and first thing that load is the reports list

neil-marcellini commented 5 months ago

Ah ok great, thanks for explaining. Yeah I guess the app startup metric could work for the LHN. Maybe it would be better to specifically measure the part of startup related to sorting the reports / deciding which should be showed in the LHN.

rinej commented 5 months ago

Ok, I will now focus on benchmarking the Desktop and iOS versions of the app to identify the bottlenecks causing performance issues. Based on recent audits, I expect significantly better results and larger report amounts.

Historically, the app has always performed much slower on Android, so I am unsure if those measurements will be useful, especially since we aim for optimal performance on Android devices as well. Please let me know your thoughts.

muttmuure commented 5 months ago

thank you for keeping up the pressure on this!

melvin-bot[bot] commented 5 months ago

@rinej, @Ollyws, @neil-marcellini Whoops! This issue is 2 days overdue. Let's get this updated quick!

rinej commented 5 months ago

Hello, I am finalizing collecting the data, tomorrow, I will post the summary

rinej commented 5 months ago

Hello, @neil-marcellini I am sharing newest results, hope that would be useful!

Problem Statement

Following the initial performance testing on Android, we performed similar tests on iOS and desktop platforms. The aim was to identify after how many reports the app starts to lag or break, without focusing on specific metrics due to platform differences. We recorded videos to illustrate the app's behavior as the number of reports increased.

Important Note

These tests were conducted using basic reports. Adding more complex elements such as attachments, tasks, and subitems would likely provide different results.

Testing Methodology

Desktop: Tested on a Mac Pro. iOS: Tested on an iPhone 15 in a simulator.

Results

Desktop (Mac Pro)

Report Number	Performance
75	✅
200	✅
1000	✅
1500	❌ / ✅
2000	❌

iOS (iPhone 15 Simulator)

Report Number	Performance
75	✅
200	✅
1000	✅
1500	✅
2000	❌ / ✅

Supporting Evidence

Desktop (Mac Pro)

**75 Reports** ✅

https://github.com/Expensify/App/assets/2650904/c89cf170-a5b4-4c25-bd80-137771f21736

**200 Reports** ✅

https://github.com/Expensify/App/assets/2650904/8d1abc83-58e9-4a7b-8952-aa2f38a93b5e

**1000 Reports** ✅

https://github.com/Expensify/App/assets/2650904/8bf42695-5a59-441c-acd7-9a37dd911f26

**1500 Reports** ❌ / ✅

https://github.com/Expensify/App/assets/2650904/a77fe72c-7df9-4d43-a3f6-67a691a39bd2

**2000 Reports** ❌

https://github.com/Expensify/App/assets/2650904/48f8600d-575d-4ac7-ad54-5de941cb9a9c

iOS (iPhone 15 Simulator)

**500 Reports** ✅

https://github.com/Expensify/App/assets/2650904/655d4467-039f-4600-8ddd-464e244f6bf9

**1000 Reports** ✅

https://github.com/Expensify/App/assets/2650904/0d22caa0-a50e-4c16-bc4d-b2275e6db367

**1500 Reports** ✅

https://github.com/Expensify/App/assets/2650904/813a3db1-f919-40a7-8998-de3ff52397c7

**2000 Reports** ❌ / ✅

https://github.com/Expensify/App/assets/2650904/9f5439ad-a803-4853-914e-bb1aa6497e17

Summary

Both desktop and iOS platforms perform much better than Android.

Desktop (Mac Pro):

Issues with loading reports start at 1500, though searching still works almost fine. At 2000 reports, the app becomes laggy.

iOS (iPhone 15 Simulator):

Performs the best among all platforms. Even at 2000 reports, the app behaves fine, with occasional issues in report loading.

Android:

Issues begin after reaching 500 reports.

❗️Please note that these results are based on basic reports. More complex report structures would provide different results.

melvin-bot[bot] commented 5 months ago

@rinej, @Ollyws, @neil-marcellini 6 days overdue. This is scarier than being forced to listen to Vogon poetry!

neil-marcellini commented 5 months ago

Wow very interesting, thanks. I wonder why Android is so much slower, is it just a much less powerful device, or are there some Android specific inefficiencies in our app or in react native?

I'm going to close this out now since we've collected enough information for now.

rinej commented 5 months ago

There might be several factors behind the performance differences between Android and iOS devices. iOS devices are better optimized because they support a much smaller range of devices. This allows for seamless integration and optimization. In contrast, Android needs to support a wide variety of devices from different manufacturers, leading to less efficient optimization.

Additionally, iOS offers more efficient memory management and provides a smoother user experience, largely due to its software.

When we conducted the performance audit, we focused mostly on Android because it is easier to spot performance issues, which become much more visible and apparent more quickly.

Expensify / App