search

MobileNativeFoundation / bluepill

Bluepill is a reliable iOS testing tool that runs UI tests using multiple simulators on a single machine
BSD 2-Clause "Simplified" License
3.19k stars 232 forks source link

Ordering test bundles by test execution times read from an input json file #342

Closed ravimandala closed 5 years ago

ravimandala commented 5 years ago

Introducing an configuration to provide a json file with test case level execution times and subsequently use the execution times to order the test bundles from longest to shortest, for better concurrency. \cc @ob Ref: https://github.com/linkedin/bluepill/issues/336

ravimandala commented 5 years ago

I'll incorporate the comments and add a couple of tests.

ravimandala commented 5 years ago

@oliverhu I have incorporated Oscar and Shawn's comments and added some unit tests. Can you please review? Thanks.

ravimandala commented 5 years ago

Same changes are available on https://github.com/linkedin/bluepill/pull/347 but on a different base (tag - v4.1.1).

ob commented 5 years ago

Oh, one more thing... where is the timings JSON file supposed to come from? I didn't see changes where Bluepill will emit it.

ravimandala commented 5 years ago

Looks good. I think the only piece missing is that Bluepill should generate the JSON file with the times. After all, it has all the information at the end, when it does the trace-profile.json file...

This is an external input to Bluepill and it is not directly derived from any specific thing that Bluepill emits. Based on my experimentation, times from one single execution are NOT reliable estimates for the next execution. In our case, we calculate medians of execution times of a test over the last 7 days.