search

getsentry / sentry-react-native

Official Sentry SDK for React Native
https://sentry.io
MIT License
1.59k stars 337 forks source link

fix(test): Flaky E2E Android Metrics App Start Time #4292

Open antonis opened 6 days ago

antonis commented 6 days ago

:loudspeaker: Type of change

⚠️ Depends on: https://github.com/getsentry/action-app-sdk-overhead-metrics/pull/23

:scroll: Description

Since the main activity of the application is displayed before RN start executing and thus before the JS SDK inits this PR:

:bulb: Motivation and Context

Fixes https://github.com/getsentry/sentry-react-native/issues/3413

:green_heart: How did you test it?

CI

:pencil: Checklist

:crystal_ball: Next steps

skip-changelog

github-actions[bot] commented 6 days ago

iOS (legacy) Performance metrics :rocket:

  Plain With Sentry Diff
Startup time 1225.24 ms 1231.35 ms 6.10 ms
Size 2.36 MiB 3.10 MiB 753.36 KiB

Baseline results on branch: main

Startup times

Revision Plain With Sentry Diff
77680ecf266a860bfb896a48f2e1beb05d774bb2+dirty 1231.81 ms 1238.35 ms 6.54 ms
1faf8e36aa9f577383f75fcefacad011fe82a691+dirty 1214.87 ms 1222.83 ms 7.97 ms
7bc4d758f2bdfa0bbb51d2a9119418c0f500e893+dirty 1233.40 ms 1229.56 ms -3.83 ms
d7401ac44acf524fe4d9a6a525f2cc13c70ce9e5+dirty 1252.38 ms 1275.04 ms 22.66 ms
8ae23a70cf23a4f7a3c54784f496e7af813b7a48+dirty 1230.02 ms 1227.62 ms -2.40 ms
5a22220fafc6352a2e07e20943dda325a4cf67f5+dirty 1209.49 ms 1220.94 ms 11.45 ms
80b2ce3d0ebe86bf1196944a2036912f6670295c+dirty 1265.92 ms 1268.60 ms 2.69 ms
b95b8afb28c7421bc27d61aca08c99e7957777f2+dirty 1221.39 ms 1228.52 ms 7.13 ms
d0bf49472b8e02ea40eab48e2e861b603450233e+dirty 1289.40 ms 1298.40 ms 9.00 ms
f06c879d164a5bd44945c6e000e25ce3f7c14172+dirty 1252.64 ms 1259.66 ms 7.02 ms

App size

Revision Plain With Sentry Diff
77680ecf266a860bfb896a48f2e1beb05d774bb2+dirty 2.36 MiB 3.10 MiB 753.42 KiB
1faf8e36aa9f577383f75fcefacad011fe82a691+dirty 2.36 MiB 3.08 MiB 736.75 KiB
7bc4d758f2bdfa0bbb51d2a9119418c0f500e893+dirty 2.36 MiB 3.10 MiB 752.58 KiB
d7401ac44acf524fe4d9a6a525f2cc13c70ce9e5+dirty 2.36 MiB 2.83 MiB 481.14 KiB
8ae23a70cf23a4f7a3c54784f496e7af813b7a48+dirty 2.36 MiB 3.10 MiB 752.42 KiB
5a22220fafc6352a2e07e20943dda325a4cf67f5+dirty 2.36 MiB 2.92 MiB 570.21 KiB
80b2ce3d0ebe86bf1196944a2036912f6670295c+dirty 2.36 MiB 2.84 MiB 486.98 KiB
b95b8afb28c7421bc27d61aca08c99e7957777f2+dirty 2.36 MiB 3.14 MiB 793.32 KiB
d0bf49472b8e02ea40eab48e2e861b603450233e+dirty 2.36 MiB 2.83 MiB 481.15 KiB
f06c879d164a5bd44945c6e000e25ce3f7c14172+dirty 2.36 MiB 2.88 MiB 530.42 KiB
github-actions[bot] commented 6 days ago

iOS (new) Performance metrics :rocket:

  Plain With Sentry Diff
Startup time 1235.94 ms 1229.96 ms -5.98 ms
Size 2.92 MiB 3.66 MiB 758.63 KiB

Baseline results on branch: main

Startup times

Revision Plain With Sentry Diff
77680ecf266a860bfb896a48f2e1beb05d774bb2+dirty 1237.65 ms 1236.24 ms -1.41 ms
1faf8e36aa9f577383f75fcefacad011fe82a691+dirty 1223.38 ms 1220.56 ms -2.82 ms
7bc4d758f2bdfa0bbb51d2a9119418c0f500e893+dirty 1222.13 ms 1216.39 ms -5.74 ms
d7401ac44acf524fe4d9a6a525f2cc13c70ce9e5+dirty 1288.10 ms 1289.54 ms 1.44 ms
8ae23a70cf23a4f7a3c54784f496e7af813b7a48+dirty 1233.67 ms 1229.52 ms -4.15 ms
5a22220fafc6352a2e07e20943dda325a4cf67f5+dirty 1246.18 ms 1249.61 ms 3.43 ms
80b2ce3d0ebe86bf1196944a2036912f6670295c+dirty 1245.12 ms 1262.04 ms 16.92 ms
b95b8afb28c7421bc27d61aca08c99e7957777f2+dirty 1235.60 ms 1242.06 ms 6.46 ms
d0bf49472b8e02ea40eab48e2e861b603450233e+dirty 1266.20 ms 1267.52 ms 1.32 ms
f06c879d164a5bd44945c6e000e25ce3f7c14172+dirty 1285.14 ms 1285.86 ms 0.72 ms

App size

Revision Plain With Sentry Diff
77680ecf266a860bfb896a48f2e1beb05d774bb2+dirty 2.92 MiB 3.66 MiB 758.54 KiB
1faf8e36aa9f577383f75fcefacad011fe82a691+dirty 2.92 MiB 3.64 MiB 742.61 KiB
7bc4d758f2bdfa0bbb51d2a9119418c0f500e893+dirty 2.92 MiB 3.66 MiB 757.15 KiB
d7401ac44acf524fe4d9a6a525f2cc13c70ce9e5+dirty 2.92 MiB 3.40 MiB 488.06 KiB
8ae23a70cf23a4f7a3c54784f496e7af813b7a48+dirty 2.92 MiB 3.66 MiB 757.67 KiB
5a22220fafc6352a2e07e20943dda325a4cf67f5+dirty 2.92 MiB 3.48 MiB 575.81 KiB
80b2ce3d0ebe86bf1196944a2036912f6670295c+dirty 2.92 MiB 3.40 MiB 492.75 KiB
b95b8afb28c7421bc27d61aca08c99e7957777f2+dirty 2.92 MiB 3.69 MiB 794.16 KiB
d0bf49472b8e02ea40eab48e2e861b603450233e+dirty 2.92 MiB 3.40 MiB 488.08 KiB
f06c879d164a5bd44945c6e000e25ce3f7c14172+dirty 2.92 MiB 3.44 MiB 533.24 KiB
antonis commented 5 days ago

Though the AppReadyActivity is launched I failed to detect it with Appium https://github.com/getsentry/action-app-sdk-overhead-metrics/pull/23. I'll stop for now and revisit the issue later.

antonis commented 5 days ago

A better simpler approach could be to just emit the app ready timestamp from RN and then parse it from appium to calculate the total load time.

2024-11-20 20:46:18.098 16742-16837 ReactNativeJS           com.testappplain                     I  'AppReadyTimestampMs:', 1732128378092
antonis commented 4 days ago

A better simpler approach could be to just emit the app ready timestamp from RN and then parse it from appium to calculate the total load time.

This failed too since capturing the logs on CI did not work (metric lib branch https://github.com/antonis/action-app-sdk-overhead-metrics/commits/antonis/add-log-timestamp-regex/).

I'm stopping this effort for now.

vaind commented 4 days ago

A better simpler approach could be to just emit the app ready timestamp from RN and then parse it from appium to calculate the total load time.

Yes, this looks like a good idea. We may not even need to post the timestamp, just any message and we can grab log timestamps from logcat

antonis commented 4 days ago

Yes, this looks like a good idea.

Thank you for iterating on this @vaind 🙇

What I tried with this approach is:

This didn't work either since the log wasn't emitted or captured on ci (it worked on my machine with a release build 😅).

I guess the reason might be similar to why capturing the 2nd activity did not work.

We may not even need to post the timestamp, just any message and we can grab log timestamps from logcat

Good idea. This can simplify the implementation further.

Any ideas on what I might be missing are more than welcome 🙇