mlykotom
commented
4 months ago If StartupMode.COLD is used, the app process is killed between the
execution of setupBlock and measureBlock to allow for app preparation
without starting the process.
If you need the process to remain active, use
StartupMode.WARM, which restarts activities without restarting the process, or
set startupMode to null and call killProcess() within the setupBlock.
You can subscribe to b/278214396 to get more updates about this in the future.
Thank you for reporting this! We added information about this to DAC as well :)
Component used: androidx.benchmark:benchmark-macro-junit4
Version used: 1.2.4
Devices/Android versions reproduced on: Pixel 6 Pro Android 14
I have setup benchmark module more than a year ago and everything was working fine back then. I haven't been running or checking benchmarks since then, because there were no high prio performance issues. I have however, been updating library versions on constant basis, including macrobenchmark and other testing libraries.
Now, I decided to come back to this and update the user flows, benchmark current code and generate new baseline profiles. But my tests started to fail. Specifically,
FrameTimingMetricwithStartupMode.COLDstarted to fail at themeasureBlock. LeavingstartupMode=nullmakes tests run finally, but I would rather prefer to have app killed and restarted between iterations to be able to measure across multiple instances of the app.The error I get is all about UiAutomator not being able to find any object by resId. Here are some details in the SO question I submitted earlier https://stackoverflow.com/q/78591545/6272369
I'm not allowed to share my project, but this issue is reproducible in current project of macrobenchmark codelab https://github.com/android/codelab-android-performance/tree/end
There are however, compilation errors that needs to be fixed and
StartupMode.COLDneeds to be added first before running the benchmark.What can be done on my side to be able to run FrameTimingMetric benchmarks with StartupMode.COLD?
P.S. I would actually also like to ask you to go through codelab https://developer.android.com/codelabs/android-macrobenchmark-inspect and update or filling missing steps. Just following the steps in the codelab is not enough to be able to run the benchmark. There seems to be missing steps or instructions when starting from the
mainbranch. Andendbranch code doesn't match with what is described in the codelab. For example, codelab includes StartupMode.COLD at the step #10, but it is not there when checking outendbranch code.