Open bruno-garcia opened 2 years ago
@vaind would you mind updating this issue with the latest changes? Some checkboxes are done already, and some issues can be closed I suspect.
The size analyses bit could be replaced by https://docs.emergetools.com/reference/get_analysis Not telling that we should do this, just sharing that some tools are getting specialized in this field.
What we want to measure:
captureMessage
,captureException
,startTransaction
,addBreadcrumb
, performance profilingWhat do we measure against:
main
branch and store it somewhere (gcloud)main
branchmain
and post in the PR as a comment. See actionGeneral approach
The goal is to run benchmarking on Sauce labs and check what’s the overhead (e.g. of the SDK during App start). We can break the build if the overhead is more than a magic number or a specific percentage. Profiling benchmark in Java does that - they break the test if the overhead is more than 5%. They also break the build if the overhead exceeds an absolute threshold.
Related:
Tracking issues, PRs and the current status
Java
The recently added (profiling-related) in-app benchmarks are based on Espresso (see https://github.com/getsentry/sentry-java/pull/2013 & https://github.com/getsentry/sentry-java/pull/2077. There has been a discussion about using Macrobenchmark in the PR that introduced the benchmarks, with the result being inconclusive and going ahead with the in-app benchmarks for now but evaluate Macrobenchmark.
Additionally, there's been an experiment with Jetpack Macrobenchmark in GDP to measure app startup - https://github.com/getsentry/sentry-android-gradle-plugin/pull/317