Best way to generate estimated test execution times in Bluepill

[ravimandala]

We are working on issue https://github.com/linkedin/bluepill/issues/336 which orders the test bundles by estimated test execution times provided as an input (re: json file) to Bluepill. Please refer to the implementation in this PR https://github.com/linkedin/bluepill/pull/342.

The current implementation expects this file to be provided as an input, but it would be an order of magnitude better if Bluepill can generate such file as an output artifact.

Let's ignore the fact that Bluepill already generates a lot of outputs, and assume one more output file wouldn't harm.

Option#1:

The simplest is to generate a file based on only the times from the current execution, just like trace-profile.json, but lightweight key-value pairs.

Cons: The current test execution time alone need not be a great predictor/estimate of the next execution time. It is not, at least from our experimentation.

Option#2:

Generating a running average of execution times by using the times from the input file and times from the current execution.

Cons: Outliers in execution times can corrupt the running average. A moving time window technique may eventually phase out outliers and prevent them from ruining the running average forever, but we need to maintain past execution times along with timestamps, which is heavy-weight.

Option#3:

To avoid outliers, a median based solution (median of past executions in a moving time window) but also need times with timestamps.

I am looking for suggestions on option#3 or any other alternatives to provide more accurate estimates, given the constraints Bluepill has. \cc @ob

[ob]

I'm leaning towards Option #1 and leaving the computation to a script outside of Bluepill... it seems it'd be more flexible.

[chenxiao0228]

@ravimandala are you working on this?