search

Expensify / App

Welcome to New Expensify: a complete re-imagination of financial collaboration, centered around chat. Help us build the next generation of Expensify by sharing feedback and contributing to the code.
https://new.expensify.com
MIT License
3.48k stars 2.84k forks source link

MEDIUM: [Tracking] Implement more E2E tests and add new metrics #30265

Open mountiny opened 12 months ago

mountiny commented 12 months ago

The E2E tests are now set up and working again, lets expand on their options and add more flows and metrics:

@Szymon20000 @hannojg

hannojg commented 12 months ago

:wave:

hannojg commented 12 months ago

@perunt

perunt commented 12 months ago

👋

mountiny commented 12 months ago

Also Assigning @AndrewGable as he will most likely help us too 🤝 feel free to unassign if you prefer

hannojg commented 12 months ago

Update:

There was again a false positive on the e2e test runs. Added this PR trying to minimise those errors:

hannojg commented 12 months ago

Update:

First PR for adding tests for typing is up:

melvin-bot[bot] commented 11 months ago

⚠️ Looks like this issue was linked to a Deploy Blocker here

If you are the assigned CME please investigate whether the linked PR caused a regression and leave a comment with the results.

If a regression has occurred and you are the assigned CM follow the instructions here.

If this regression could have been avoided please consider also proposing a recommendation to the PR checklist so that we can avoid it in the future.

mountiny commented 11 months ago

Lies, it was a different pr causing the issue

melvin-bot[bot] commented 10 months ago

This issue has not been updated in over 15 days. @AndrewGable, @hannojg, @mountiny, @perunt eroding to Monthly issue.

P.S. Is everyone reading this sure this is really a near-term priority? Be brave: if you disagree, go ahead and close it out. If someone disagrees, they'll reopen it, and if they don't: one less thing to do!

mountiny commented 10 months ago

@hannojg @perunt how is it going?

hannojg commented 10 months ago

(Note: we are internally working on a system to make the e2e tests even better, more news on that soon! ™️ )

mountiny commented 10 months ago

Thanks!

dangrous commented 10 months ago

adding myself here for reviewing the chat switching one!

hannojg commented 10 months ago

Chat switching is merged 🎊 !

@perunt I think we can add "number of renders when we open chat (it should be 1-2)" to the same test as second metric?

Also next up would be the search page test!

melvin-bot[bot] commented 10 months ago

Reviewing label has been removed, please complete the "BugZero Checklist".

melvin-bot[bot] commented 10 months ago

The solution for this issue has been :rocket: deployed to production :rocket: in version 1.4.14-6 and is now subject to a 7-day regression period :calendar:. Here is the list of pull requests that resolve this issue:

If no regressions arise, payment will be issued on 2023-12-28. :confetti_ball:

After the hold period is over and BZ checklist items are completed, please complete any of the applicable payments for this issue, and check them off once done.

For reference, here are some details about the assignees on this issue:

mountiny commented 9 months ago

I think we might want to take a better look into how we are using the E2E tests now and make sure we already getting some use of it before adding more scenarios. In theory it can be very powerful tool, but so far we have not really been using the results of the tests to stop any PRs/ deploys. Come more convo here cc @blimpich @roryabraham @iwiznia

melvin-bot[bot] commented 9 months ago

@AndrewGable, @dangrous, @hannojg, @mountiny, @perunt Whoops! This issue is 2 days overdue. Let's get this updated quick!

blimpich commented 9 months ago

My 2 cents: E2E tests should be used incredibly sparingly. E2E tests are powerful but they're also expensive to make, maintain, and debug. I don't have much context on Expensify's E2E suite but I'd strongly lean in the direction of investing in many small, inexpensive unit tests instead of more E2E tests.

I'd also agree that if we aren't even using our current E2E test suite to stop deploys or cause code freezes, that there is no real reason why we even should care about the E2E test suite. The test suite as it stands is providing little to no value, since we currently can go a week with it being broken in main and no one noticing (see this slack thread).

Ignore this, I should have brought this up internally, I misunderstood and don't think this was the right place for me to make this comment.

mountiny commented 9 months ago

This is not overdue, we are continuing as normal

melvin-bot[bot] commented 9 months ago

@AndrewGable, @dangrous, @hannojg, @mountiny, @perunt Huh... This is 4 days overdue. Who can take care of this?

hannojg commented 9 months ago

This is currently on hold by the effort to change how we mock the API. Discussion is here:

https://expensify.slack.com/archives/C035J5C9FAP/p1703237225747209

mountiny commented 9 months ago

Working on the API mocking

melvin-bot[bot] commented 9 months ago

@AndrewGable, @dangrous, @hannojg, @mountiny, @perunt Whoops! This issue is 2 days overdue. Let's get this updated quick!

mountiny commented 9 months ago

Solution for the API mocking was provided, we need to test it

mountiny commented 9 months ago

@hannojg whats the latest here?

hannojg commented 9 months ago
hannojg commented 8 months ago

Update: the e2e tests seem to be broken right now and we are working on finding out why. Conversation is happening here.

hannojg commented 8 months ago

The e2e tests became broken again, mentioned and fixed here.

We are working on:

Once thats all merged and stable we will work on the next e2e tests (hopefully 🔜 ™️ )

mountiny commented 8 months ago

@hannojg monitoring the reliability of existing tests

AndrewGable commented 7 months ago

@hannojg is working on this

hannojg commented 7 months ago

After this PR was merged:

we are monitoring stability again, then moving on to implementing more tests!

mountiny commented 7 months ago

@hannojg what is the latest on this one?

hannojg commented 7 months ago

The tests are semi stable, currently 15% of them fail during running. Still want to iron that out!

mountiny commented 6 months ago

@hannojg what are the next steps on this issue from your side?

hannojg commented 6 months ago

We are now actively working on implementing the tests that align with the core metrics:

https://expensify.slack.com/archives/C05LX9D6E07/p1711455442814489?thread_ts=1711010712.155509&cid=C05LX9D6E07

mountiny commented 6 months ago

@hannojg how is this looking?

hannojg commented 6 months ago

We have two PRs open waiting for final review / clarifications:

mountiny commented 5 months ago

@hannojg What's next now?

hannojg commented 5 months ago

Next is to allocate time for this task in our team and move forward! Nothing blocking as far as I can see. We'll pick this up again next week!

melvin-bot[bot] commented 4 months ago

This issue has not been updated in over 15 days. @dangrous, @hannojg, @perunt eroding to Monthly issue.

P.S. Is everyone reading this sure this is really a near-term priority? Be brave: if you disagree, go ahead and close it out. If someone disagrees, they'll reopen it, and if they don't: one less thing to do!

roryabraham commented 4 months ago

We were chatting in slack about pointing these E2E tests at staging and using them to help figure out our "high water point" at which the performance degrades critically with an increasing number of reports across various devices.

hannojg commented 4 months ago

@roryabraham I remember we discussed doing a one-off report for those "high water points" here

Is this something we should look at?

cc @kirillzyusko

mountiny commented 4 months ago

I believe @rinej is working on that

hannojg commented 4 months ago

@kirillzyusko lets finish this one!

Which tests are currently missing? I see that we are still missing:

I think for the onyx tests we wanted some tests where we don't load the full app, but just run some performance tests with mock data in onyx to catch performance regressions in onyx.

kirillzyusko commented 4 months ago

@hannojg yes, you are right. I think for number of renders we only track amount of re-renders in Composer.

Also I think we'll need to revise the approach for tracking number of re-renders, because now we rely on the fact, that if parent was re-rendered, then children that are not wrapped in memo will be re-rendered. But with react-compiler it doesn't work anymore, because it'll memoize everything automatically 👀

kirillzyusko commented 3 months ago

Also I think we'll need to revise the approach for tracking number of re-renders, because now we rely on the fact, that if parent was re-rendered, then children that are not wrapped in memo will be re-rendered. But with react-compiler it doesn't work anymore, because it'll memoize everything automatically 👀

Just as a proof of my words - below I attached a report (after react-compiler was merged). And as we can see (first of all it really works 😅) that amount of renders went down from 2 to 0. I assume we have the same amount of re-renders, but since we were relying on an approach described above -> we have modified numbers now, so eventually at some point of time we'll need to revisit that.

Performance Comparison Report 📊

Significant Changes To Duration

There are no entries

Meaningless Changes To Duration

Show entries | Name | Duration | | --------------------------------------------- | ----------- | | App start nativeLaunch | 46.421 ms | | App start nativeLaunchEnd_To_appCreationStart | 50.533 ms | | App start appCreationEnd_To_contentAppeared | 590.857 ms | | App start appCreation | 57.070 ms | | App start contentAppeared_To_screenTTI | 985.[12](https://github.com/Expensify/App/actions/runs/9660820703/job/26648859627#step:17:13)2 ms | | App start TTI | 1727.189 ms | | App start runJsBundle | 441.492 ms | | App start regularAppStart | 0.022 ms | | Load Search Options | 2[13](https://github.com/Expensify/App/actions/runs/9660820703/job/26648859627#step:17:14).100 ms | | Open Chat Finder Page TTI | 1237.410 ms | | Composer typing rerender count | 0.000 ms | | Chat opening | 166.850 ms | | Chat TTI | 565.710 ms | | Comment linking | 480.110 ms |
Show details | Name | Duration | | --------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | App start nativeLaunch | **Baseline**
Mean: 46.474 ms
Stdev: 4.519 ms (9.7%)
Runs: 39 39 41 41 41 41 41 42 42 42 43 43 43 43 43 44 44 44 44 44 44 44 44 44 45 45 45 45 46 46 46 46 47 47 47 47 47 48 48 48 48 49 49 50 50 50 51 51 51 51 52 52 54 54 57 58 59

**Current**
Mean: 46.421 ms
Stdev: 5.231 ms (11.3%)
Runs: 39 40 40 40 40 40 41 41 42 42 42 42 42 42 42 42 43 43 43 43 43 43 44 44 45 45 45 46 46 46 46 46 46 46 46 47 47 47 48 48 48 49 50 50 51 51 52 52 52 52 53 54 54 55 59 60 61 | | App start nativeLaunchEnd_To_appCreationStart | **Baseline**
Mean: 49.867 ms
Stdev: 8.401 ms (16.8%)
Runs: 36 37 37 38 39 40 40 41 41 42 42 42 43 43 44 44 44 44 44 45 45 45 46 46 46 47 48 48 48 49 49 49 50 51 51 51 52 52 52 52 52 53 53 54 54 56 57 57 57 58 59 59 60 60 63 63 64 66 69 75

**Current**
Mean: 50.533 ms
Stdev: 8.401 ms (16.6%)
Runs: 38 39 39 40 40 41 41 41 42 42 42 43 43 43 44 44 44 45 45 45 45 47 47 47 47 47 47 48 48 49 49 50 51 51 52 52 52 53 53 53 53 54 54 55 55 56 57 57 57 57 57 60 60 61 62 67 69 70 70 72 | | App start appCreationEnd_To_contentAppeared | **Baseline**
Mean: 586.533 ms
Stdev: 43.644 ms (7.4%)
Runs: 497 506 5[14](https://github.com/Expensify/App/actions/runs/9660820703/job/26648859627#step:17:15) 516 520 524 529 529 534 537 538 539 540 545 551 554 561 564 575 576 576 578 580 581 582 582 582 586 587 588 589 589 590 590 591 595 596 596 602 603 610 611 612 613 617 617 617 618 622 622 627 627 639 642 645 648 656 662 685 690

**Current**
Mean: 590.857 ms
Stdev: 31.269 ms (5.3%)
Runs: 519 524 530 537 538 543 544 546 553 555 565 574 577 577 577 579 580 580 582 582 584 586 587 587 587 588 588 592 592 593 594 596 597 597 598 598 602 610 610 611 612 6[15](https://github.com/Expensify/App/actions/runs/9660820703/job/26648859627#step:17:16) 616 618 620 621 621 623 626 627 628 629 631 637 650 655 | | App start appCreation | **Baseline**
Mean: 61.390 ms
Stdev: 10.279 ms ([16](https://github.com/Expensify/App/actions/runs/9660820703/job/26648859627#step:17:17).7%)
Runs: 46 48 49 50 51 51 51 51 51 52 52 52 53 53 53 54 54 54 54 54 54 55 55 55 55 56 58 59 59 60 60 60 61 61 62 62 62 62 63 64 65 66 68 69 69 69 71 71 72 72 72 74 74 75 75 80 85 87 92

**Current**
Mean: 57.070 ms
Stdev: 6.948 ms (12.2%)
Runs: 44 45 46 47 48 48 48 50 50 50 51 51 51 51 51 52 52 54 54 54 55 55 55 55 56 56 56 57 57 57 58 58 58 59 59 59 59 59 59 59 60 60 61 61 62 62 62 63 63 64 66 69 69 71 71 72 74 | | App start contentAppeared_To_screenTTI | **Baseline**
Mean: 1065.342 ms
Stdev: 76.461 ms (7.2%)
Runs: 907.8019419999328 915.2966060000472 922.8915379999671 923.1415490000509 936.7570619999897 941.0060360000934 949.8935799999163 982.3591420000885 983.5623409999534 990.5694039999507 998.6525149999652 999.3633300000802 999.6344999999274 1006.9684609998949 1007.7[17](https://github.com/Expensify/App/actions/runs/9660820703/job/26648859627#step:17:18)413000064 1016.4667450000998 1020.0574990001041 1022.4106940000784 1023.4612030000426 1023.9040620001033 1033.2889250000007 1041.210217999993 1044.919617000036 1048.5714539999608 1050.9262329998892 1058.0397880000528 1064.8353329999372 1064.9087469999213 1065.360864999937 1065.9842719999142 1073.4153340000194 1081.0659119999036 1081.4825420000125 1081.8841289998963 1082.1053369999863 1082.308960000053 1082.649822999956 1090.1506209999789 1090.231924999971 1090.3712689999957 1106.3483090000227 1107.380[18](https://github.com/Expensify/App/actions/runs/9660820703/job/26648859627#step:17:19)49999 1114.1298309999984 1115.2328929998912 1115.4641899999697 1116.5955270000268 1124.08[19](https://github.com/Expensify/App/actions/runs/9660820703/job/26648859627#step:17:20)810000248 1124.3340419998858 1128.1248780000024 1131.7707030000165 1140.8266129998956 1165.3738959999755 1165.58939[20](https://github.com/Expensify/App/actions/runs/9660820703/job/26648859627#step:17:21)001108 1169.9072310000192 1173.0982860000804 1173.8252739999443 1182.3338279998861 1197.9031980000436 1206.981495999964 1[21](https://github.com/Expensify/App/actions/runs/9660820703/job/26648859627#step:17:22)5.5781890000217

**Current**
Mean: 985.1[22](https://github.com/Expensify/App/actions/runs/9660820703/job/26648859627#step:17:23) ms
Stdev: 82.303 ms (8.4%)
Runs: 816.1936719999649 828.1428680000827 842.0098999999464 849.1310709998943 854.7382259999868 864.8342579999007 890.2964689999353 897.9933609999716 899.5491200000979 900.2879840000533 903.3480809999164 907.226885000011 907.6650960000698 9[23](https://github.com/Expensify/App/actions/runs/9660820703/job/26648859627#step:17:24).9573609998915 924.5785950000864 924.6644840000663 940.2367819999345 940.8640999998897 947.6720690000802 949.1422260000836 949.1836399999447 949.1871100000571 949.3551290000323 954.6597519998904 956.0930820000358 956.447179999901 958.2914809999056 967.0230159999337 972.9453010000288 973.3050220001023 977.215649999911 982.6516629999969 998.3009860001039 1005.8315520000178 1006.9743359999266 1007.1177580000367 1007.7846750000026 1015.1390990000218 1015.2986280000769 1015.7307490000967 1016.4739329998847 1022.6975750001147 1023.8839889999945 10[24](https://github.com/Expensify/App/actions/runs/9660820703/job/26648859627#step:17:25).0584040000103 1032.2667839999776 1056.6662920000963 1073.7304310000036 1074.429194000084 1082.1811830000952 1082.3930500000715 1082.4438249999657 1098.0842269998975 1098.9[25](https://github.com/Expensify/App/actions/runs/9660820703/job/26648859627#step:17:26)2840001136 1099.4649529999588 1107.1244930000976 1107.71744700009 1113.516067000106 1114.0042310000863 1132.7914040000178 1133.428477999987 | | App start TTI | **Baseline**
Mean: 1811.642 ms
Stdev: 110.432 ms (6.1%)
Runs: 1570.1415490000509 1605.7570619999897 1610.891537999967 1612.8935799999163 1613.2966060000472 1631.8019419999328 1643.5623409999534 1648.3591420000885 1649.3633300000802 1650.0060360000934 1677.717413000064 1692.9040620001033 1702.6344999999274 1716.5694039999507 1759.4667450000998 1761.1053369999863 1764.0574990001041 1769.4825420000125 1771.0659119999036 1774.0397880000528 1778.4106940000784 1779.968460999895 1791.210217999993 1800.5714539999608 1803.46120300004[26](https://github.com/Expensify/App/actions/runs/9660820703/job/26648859627#step:17:27) 1808.360864999937 1823.649822999956 1824.6525149999652 1828.9087469999213 1830.3712689999957 1836.7707030000165 1840.919617000036 1843.8841289998963 1847.4153340000194 1848.8353329999372 1851.1506209999789 1854.308960000053 1856.984[27](https://github.com/Expensify/App/actions/runs/9660820703/job/26648859627#step:17:28)19999142 1862.2328929998912 1871.2889250000007 1875.1248780000024 1887.4641899999697 1891.1298309999984 1892.0819810000248 1894.3340419998858 1894.9262329998892 1895.9072310000192 1899.3483090000227 1914.231924999971 1914.8266129998956 1915.3801849999 1917.5955270000268 1922.3738959999755 1929.9031980000436 1934.8252739999443 1953.098[28](https://github.com/Expensify/App/actions/runs/9660820703/job/26648859627#step:17:29)60000804 1953.5893920001108 1963.3338279998861 2005.5781890000217 2034.981495999964

**Current**
Mean: 1727.189 ms
Stdev: 107.662 ms (6.2%)
Runs: 1503.1428680000827 1507.1936719999649 1533.1310709998943 1538.0098999999464 1550.2879840000533 1555.8342579999007 1556.3480809999164 1589.9933609999716 1594.5785950000864 1596.2367819999345 1601.1871100000571 1624.0930820000358 1626.226885000011 1626.7382259999868 1627.6644840000663 1654.549120000098 1668.[29](https://github.com/Expensify/App/actions/runs/9660820703/job/26648859627#step:17:30)64689999353 1671.9573609998915 1687.6650960000698 1695.447179999901 1705.02[30](https://github.com/Expensify/App/actions/runs/9660820703/job/26648859627#step:17:31)159999337 1713.3050220001023 1724.1836399999447 1726.3551290000323 1728.2914809999056 1728.8640999998897 1730.7846750000026 1732.1422260000836 1735.1177580000367 1745.651662999997 1747.9743359999266 1749.8[31](https://github.com/Expensify/App/actions/runs/9660820703/job/26648859627#step:17:32)5520000178 1750.215649999911 1750.9453010000288 1753.6720690000802 1755.6975750001147 1762.7307490000967 1765.4739[32](https://github.com/Expensify/App/actions/runs/9660820703/job/26648859627#step:17:33)9998847 1766.6597519998904 1770.2986280000769 1776.3009860001039 1777.1390990000218 1778.8839889999945 1779.6662920000963 1789.2667839999776 1797.71744700009 1801.0584040000103 1803.7304310000036 1831.1811830000952 1839.3930500000715 1844.9252840001136 1846.0842269998975 1864.0042310000863 1864.1244930000976 1864.429194000084 1864.4649529999588 1889.4438249999657 1910.428477999987 1925.516067000106 1931.7914040000178 | | App start runJsBundle | **Baseline**
Mean: 440.915 ms
Stdev: 31.211 ms (7.1%)
Runs: 368 379 387 395 395 397 402 403 406 406 409 409 414 414 419 420 423 425 425 427 431 4[33](https://github.com/Expensify/App/actions/runs/9660820703/job/26648859627#step:17:34) 433 434 435 436 440 442 443 444 444 445 447 447 447 448 448 449 454 455 456 458 458 460 463 468 468 469 471 473 473 475 478 481 482 489 493 508 513

**Current**
Mean: 441.492 ms
Stdev: 28.374 ms (6.4%)
Runs: 384 385 386 387 397 399 400 402 408 409 412 416 420 420 425 426 427 430 430 432 432 432 4[34](https://github.com/Expensify/App/actions/runs/9660820703/job/26648859627#step:17:35) 437 440 440 441 445 446 447 447 448 448 448 449 450 452 452 453 454 454 455 455 455 457 459 465 467 469 470 470 472 475 476 481 487 489 498 504 | | App start regularAppStart | **Baseline**
Mean: 0.023 ms
Stdev: 0.002 ms (10.3%)
Runs: 0.017089999979361892 0.017699999967589974 0.01818799995817244 0.01827000011689961 0.018554000183939934 0.0187180000[35](https://github.com/Expensify/App/actions/runs/9660820703/job/26648859627#step:17:36)732985 0.01977499993517995 0.019896999932825565 0.01989799994044006 0.02030499977990985 0.02087399992160499 0.02123999991454184 0.021321999840438[36](https://github.com/Expensify/App/actions/runs/9660820703/job/26648859627#step:17:37)6 0.02140299999155104 0.021403000224381685 0.021646999986842275 0.021727999905124307 0.021769999992102385 0.021849999902769923 0.021850999910384417 0.022013999987393618 0.022461000131443143 0.02254199981689453 0.022582999896258116 0.02258300012908876 0.022664000047[37](https://github.com/Expensify/App/actions/runs/9660820703/job/26648859627#step:17:38)079 0.022704999893903732 0.022745999973267317 0.022786000045016408 0.022948999889194965 0.02294900012202561 0.02298999996855855 0.023194000124931335 0.02327399980276823 0.023314999882131815 0.0235190000[38](https://github.com/Expensify/App/actions/runs/9660820703/job/26648859627#step:17:39)5046 0.023558999877423048 0.023600999964401126 0.023640999803319573 0.023680999875068665 0.023681999882683158 0.024252000031992793 0.02437300002202[39](https://github.com/Expensify/App/actions/runs/9660820703/job/26648859627#step:17:40)16 0.024495000019669533 0.02474000002257526 0.0248209999[40](https://github.com/Expensify/App/actions/runs/9660820703/job/26648859627#step:17:41)85729 0.024983000010252 0.024984000017866492 0.02518699993379[41](https://github.com/Expensify/App/actions/runs/9660820703/job/26648859627#step:17:42)4 0.02547200000844896 0.025594000006094575 0.0258390000090003 0.025919999927282333 0.025959999999031[42](https://github.com/Expensify/App/actions/runs/9660820703/job/26648859627#step:17:43)5 0.026286000153049827 0.026448999997228384 0.027180999983102083

**Current**
Mean: 0.022 ms
Stdev: 0.003 ms (12.8%)
Runs: 0.01745599997229874 0.017577999969944358 0.0176189998164773 0.018229999812319875 0.01831099996343255 0.018514000112190843 0.01871700002811849 0.019287000177428126 0.019328000023961067 0.019328000023961067 0.019368999870494008 0.01977499993517995 0.019816000014543533 0.019896999932825565 0.01989799994044006 0.020141000160947442 0.020141999935731292 0.020183000015094876 0.02034499985165894 0.020345000084489584 0.020385999931022525 0.02050799992866814 0.020751999923959374 0.02079300000332296 0.020913999993354082 0.021036999998614192 0.02123999991454184 0.021280999993905425 0.021484999917447567 0.02148500015027821 0.021607000147923827 0.02168800006620586 0.02172800013795495 0.021850999910384417 0.021890999982133508 0.021932000061497092 0.022012999979779124 0.022827000124379992 0.022867999970912933 0.023275000043213367 0.023641000036150217 0.024088000180199742 0.024088999954983592 0.024373000022023916 0.024536000099033117 0.02474000002257526 0.025228000013157725 0.025430999929085374 0.0256759999319911 0.026203999994322658 0.026285999920219183 0.02636700007133186 0.026814999990165234 0.027180999983102083 0.027222000062465668 0.029541000025346875 | | Load Search Options | **Baseline**
Mean: 215.810 ms
Stdev: 15.900 ms (7.4%)
Runs: 187.6241859998554 188.15079699992202 189.3496509999968 190.11763500003144 190.5600179999601 191.2069500000216 191.7418619999662 192.7706299999263 193.29142200015485 199.33841999992728 199.9901129999198 201.3704830002971 202.27295000012964 205.46675600018352 205.87142000021413 207.3771569998935 207.7867439999245 208.11596699990332 209.21459899982437 209.2482089996338 210.2288000001572 210.27315300004557 212.57914200006053 212.72977699991316 212.9648849996738 213.21862800000235 213.52095600031316 214.11641399981454 215.16345199989155 215.73836300009862 217.7987069999799 218.44311500014737 218.49833199987188 218.6028639995493 218.76013199985027 219.09814399993047 221.77832000004128 224.21106000011787 225.00195300020278 225.20874000014737 225.68843599967659 226.3619389999658 227.2277020001784 227.28361000027508 227.28812699997798 227.8553059999831 227.86226400034502 228.46329799992964 230.8099360000342 230.8671059999615 230.94376600021496 233.64123499998823 236.1510820002295 236.4513339996338 241.75793500011787 247.57438199967146 249.14916999964043 252.78511600010097

**Current**
Mean: 213.100 ms
Stdev: 15.295 ms (7.2%)
Runs: 179.6796059999615 186.46903500007465 190.09228500002064 190.5006510000676 192.99190200003795 193.0804440001957 193.30700699985027 194.13647399982[43](https://github.com/Expensify/App/actions/runs/9660820703/job/26648859627#step:17:44)7 194.97448699967936 195.29024200001732 195.31978299980983 195.3685710001737 195.54191099992022 197.5196940000169 199.6716310000047 201.90002500033006 203.11031099967659 204.49642000021413 204.76607300015166 206.48881000000983 207.07328300038353 208.9291179999709 209.98046899959445 210.40722699975595 212.0319409999065 212.41699200030416 212.6582430000417 216.0651039998047 216.43379699997604 216.7900390001014 216.89347400004044 217.84602899989113 218.96207699971274 219.21671500010416 219.50752800004557 220.14009599993005 221.03352800011635 221.03857500012964 221.14009599993005 221.41162099968642 221.6957610002719 222.19934000028297 222.6646729996428 222.98185299988836 223.0439039999619 224.09293700009584 225.1695559998043 227.60632300004363 229.25406899955124 230.33272300008684 230.39489699993283 233.8977049998939 235.01298000011593 235.75081399967894 238.08630400011316 238.8903810000047 250.9480789997615 | | Open Chat Finder Page TTI | **Baseline**
Mean: 1183.998 ms
Stdev: 57.718 ms (4.9%)
Runs: 1049.1873379999306 1070.65002499986[44](https://github.com/Expensify/App/actions/runs/9660820703/job/26648859627#step:17:45) 1091.3407799997367 1093.3906260000076 1100.8560799998231 1106.7984219999053 1111.3887950000353 1114.4760750001296 1125.1623539999127 1134.828614000231 1137.7478849999607 1138.5943610002287 1139.1834720000625 1139.372802999802 1139.5280770002864 1139.805136000039 1141.2639580001123 1146.276041999925 1146.4478360000066 1151.9972340003587 1152.890585000161 1155.1714679999277 1155.269328000024 1156.8361410000362 1157.2795820003375 1157.4157310002483 1158.2241620002314 1169.0699880002066 1174.953288000077 1181.7361249998212 1184.7868249998428 1186.482340999879 1191.9135340000503 1199.40730800014 1203.1869720001705 1214.8201910001226 1215.7211919999681 1218.0220139999874 1218.7476820000447 1220.501221000217 1225.2565520000644 1226.3272310001776 1228.6660569999367 1228.710694000125 1229.2508960003033 1229.7907730001025 1232.983196000103 1240.5893149999902 1240.7472740001976 1242.1815189998597 12[45](https://github.com/Expensify/App/actions/runs/9660820703/job/26648859627#step:17:46).8743489999324 1248.7130950000137 1248.7187099996954 1250.643962000031 1251.3100589998066 1252.8673109998927 1255.250326000154 1255.9399010003544 1270.9145919997245 1344.4232180002145

**Current**
Mean: 1237.410 ms
Stdev: 85.441 ms (6.9%)
Runs: 1056.287558000069 1062.01550400001 1098.9885260001756 1108.5623380001634 1112.0209559998475 1116.9572350000963 1124.5742600001395 1130.574137999909 1138.5173349999823 1143.43933199998 1157.0199390002526 1159.6073809999507 1160.4798590000719 1166.092244999949 1169.2841799999587 1171.2209479999729 1183.5891929999925 1186.5815029996447 1187.3382979999296 1197.3414309998043 1208.0832519996911 1210.9396979999729 1212.4313570000231 1213.7568359998986 1219.5778410001658 1220.9313159999438 1226.4167490000837 1229.743530999869 1231.2180590000935 1232.7742509997915 1241.372844000347 1242.3741[46](https://github.com/Expensify/App/actions/runs/9660820703/job/26648859627#step:17:47)0000165 1243.3283700002357 1245.799682999961 1253.8372810003348 1255.6403000000864 1260.2985029998235 1266.7641209997237 1271.83203099994 1273.852295000106 1278.9971929998137 1282.7675789999776 1287.2347420002334 1289.0062259999104 1299.5423999996856 1306.2535410001874 1306.8204350001179 1312.210367999971 1312.2255859998986 1318.6031909999438 1326.724731999915 1334.040934999939 1346.6018480001949 1346.867065999657 1349.7342129996978 1349.9047860000283 1373.926270000171 1403.1051030000672 1409.0791429998353 1419.4740809998[47](https://github.com/Expensify/App/actions/runs/9660820703/job/26648859627#step:17:48)5 | | Composer typing rerender count | **Baseline**
Mean: 2.000 ms
Stdev: 0.000 ms (0.0%)
Runs: 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2

**Current**
Mean: 0.000 ms
Stdev: 0.000 ms (NaN%)
Runs: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 | | Chat opening | **Baseline**
Mean: 144.959 ms
Stdev: 6.599 ms (4.6%)
Runs: 134.66532400017604 135.12951700016856 135.66825400013477 136.[48](https://github.com/Expensify/App/actions/runs/9660820703/job/26648859627#step:17:49)018399998546 137.52022300008684 138.20890300022438 138.2253020000644 139.0252680000849 139.7583010001108 139.76721200020984 139.94966599997133 140.440510999877 140.51761899981648 140.68684900039807 140.8257650001906 140.87076900014654 141.05513500003144 141.07238799985498 141.22603400005028 141.54764899984002 141.60290500009432 141.6551510002464 141.7930899998173 142.13582399999723 142.19893399998546 142.75113999983296 143.33548999996856 143.66524200001732 143.85380100039765 145.12675000028685 145.1649989997968 145.6087239999324 145.70882199984044 145.71130400011316 145.83064700011164 146.2416179999709 146.3520510001108 146.36348400032148 147.00370300002396 147.28881900012493 147.55061899963766 147.90820399997756 147.92561799986288 149.02136199967936 150.31978400005028 151.28027400001884 153.69934100005776 153.95800799969584 154.4168289997615 156.9649659995921 157.28751600021496 160.70922800013795 161.990233999677 162.7268880000338

**Current**
Mean: 166.850 ms
Stdev: 12.657 ms (7.6%)
Runs: 141.99723300011829 144.2677810001187 144.35876500001177 145.1138510000892 145.99772100010887 149.32535899989307 149.53910399973392 150.03788299998268 152.4448239998892 152.45634000003338 152.56005899980664 153.10286499978974 153.11889600008726 159.0053309998475 160.40144800022244 161.34920200007036 161.5302320001647 161.79292799998075 162.3262120001018 162.61356600001454 164.56384300021455 165.2255859998986 165.512940000277 165.82426000013947 166.455851000268 167.01375399995595 167.04829900013283 167.40071600023657 169.56388300005347 169.5749109997414 170.3244630000554 170.60412599984556 170.78129100007936 170.88480600016192 171.63403300009668 171.98120100004598 172.35278399987146 172.98783400002867 173.89680999983102 174.24361200025305 174.73632800020278 175.04858400020748 175.34790099970996 176.57291600015014 176.94393000006676 177.11149099981412 177.92614699993283 180.98140500020236 182.7631829995662 183.49198399996385 184.63033000007272 187.30403700005263 188.45646200003102 188.98754899995402 195.2517499998212 | | Chat TTI | **Baseline**
Mean: 476.442 ms
Stdev: 30.748 ms (6.5%)
Runs: 434.2028810000047 437.42981000011787 440.26863600034267 441.4828290003352 441.9324139999226 446.1614179997705 446.4310710001737 446.7055259998888 447.65071700001135 448.7163[49](https://github.com/Expensify/App/actions/runs/9660820703/job/26648859627#step:17:50)999886 449.24047899991274 449.6853030002676 449.82076000003144 450.2500409996137 450.9113360000774 450.969034999609 451.24296100018546 452.19966599997133 452.81559300003573 454.2888600002043 454.53576700016856 456.12874399963766 457.02852400019765 457.63704500021413 458.65694200014696 459.0605069999583 462.0355630000122 463.3727220003493 466.6648770002648 466.9698080001399 467.61844900017604 472.4313559997827 475.7039799997583 479.0117190000601 479.1413169996813 479.52579799992964 481.0275070001371 481.9576820000075 482.4648040002212 483.312663000077 484.4969489998184 485.2662769998424 485.91410299995914 492.203572999686 495.68957499973476 500.7[50](https://github.com/Expensify/App/actions/runs/9660820703/job/26648859627#step:17:51)52900006995 501.087728000246 503.28108800016344 504.24967499962077 504.2858070000075 509.18249499984086 509.9790040003136 512.5047209998593 518.9646809999831 532.6081550000235 532.9685880001634 534.7239990001544 545.7838129997253 548.5006919996813 557.3652350003831

**Current**
Mean: 565.710 ms
Stdev: 30.276 ms (5.4%)
Runs: 496.25329599995166 510.42346200020984 510.9249669997953 512.8595790001564 513.2970790001564 518.8798019997776 520.0266920002177 536.0394290001132 536.6150309997611 538.2088629999198 538.3254800001159 540.3033849997446 542.1627200003713 542.3945309999399 543.8304039998911 545.8713389998302 546.221923999954 550.6606040000916 550.6827389998361 552.5664880000986 553.3953860001639 554.043864000123 557.8120929999277 558.5373129998334 561.6619469998404 563.2617600001395 566.0299889999442 567.0897220000625 567.4525150000118 568.374307999853 569.0174569999799 570.2483320003375 572.0125739998184 572.2180989999324 572.6911220001057 573.6181649998762 575.0176190002821 576.9523119996302 577.2944739996456 578.8898519999348 580.112548999954 580.962849999778 581.153239000123 581.2606609999202 582.7671309998259 583.2261969996616 585.7374269999564 593.535116000101 594.7001950000413 596.4340010001324 601.5576980002224 606.8865559999831 608.2360849999823 609.4044590000995 609.8971350002103 610.0624190000817 613.2183429999277 615.7701820000075 639.7746989997104 | | Comment linking | **Baseline**
Mean: 479.941 ms
Stdev: 94.333 ms (19.7%)
Runs: 329.9139000000432 343.6412349999882 345.00870699994266 353.8523350004107 356.29203299991786 365.00968400016427 368.8347169999033 369.1029059998691 369.19584200065583 370.2426760001108 377.52616299968213 378.4616290004924 381.8841559998691 387.89355499949306 391.8345539998263 394.7826339993626 401.44868999999017 415.91259799990803 419.47282000025734 419.6573490006849 420.69120300002396 433.2966719996184 437.5493980003521 441.165731000714 449.463175999932 454.1970620006323 455.46671599987894 463.6658939998597 469.843913000077 470.22635999973863 470.660442000255 477.4665130004287 481.33044399973005 483.81404699990526 492.84175700042397 503.6965330000967 507.8522140001878 508.1464029997587 510.19746899977326 512.2441819999367 522.2788909999654 533.435344000347 535.5654710000381 537.1874599996954 540.5981049994007 548.4115810003132 5[51](https://github.com/Expensify/App/actions/runs/9660820703/job/26648859627#step:17:52).3695480003953 564.5918780001812 568.8046059999615 571.7102049998939 580.5857340004295 600.5196130005643 607.0284420000389 630.4750570002943 634.9811599999666 637.326253999956 639.7915850002319 658.9739180002362 663.3828539997339 685.6864419998601

**Current**
Mean: 480.110 ms
Stdev: 129.942 ms (27.1%)
Runs: 290.7557779997587 298.0888269999996 301.43457099981606 307.4532060008496 318.6726479995996 319.74788499996066 323.0387369999662 326.82503299973905 328.23693799972534 337.19824200030416 338.0709229996428 350.3486730000004 359.3664959995076 360.7155770007521 362.8916020002216 363.6289880000986 370.4261879995465 377.2214360004291 380.35819499986246 380.4551599994302 382.2853199997917 385.06620300002396 385.9578049997799 388.3621829999611 402.1499429997057 427.3040360007435 452.9881600001827 463.32287599984556 475.5500900000334 478.95011400058866 482.47977699991316 485.61649600043893 490.5386150004342 496.2018639994785 509.42675900040194 518.3033050000668 521.200154999271 522.1745200008154 541.8848069999367 552.8354080002755 554.4312340002507 558.0227460004389 567.8896490000188 568.1798509992659 568.9352619992569 580.8712160000578 587.284831000492 604.4516600007191 605.9324139999226 615.2051189998165 624.5551350004971 633.5082599995658 650.5862229997292 665.2394620003179 671.7214359994978 686.9343670001253 687.4185389997438 704.3919680006802 721.1540120001882 792.3304449999705 |
dangrous commented 2 months ago

hi friends! just checking in - where are we on this one? let me know if I can help in any way!

kirillzyusko commented 2 months ago

@dangrous I think we are currently (still) trying to fix e2e tests. And only after that we can start to write new e2e tests 👀

dangrous commented 1 month ago

how are we looking?