This PR fixes an error in the cost calculation in that we were not handling pre-emption. That is, a shard could get preempted and thus we'd have multiple cost measurements for that shard.
I've addressed this by looking for a single 'representative' instance of each shard. But despite this, I found there is still non-determinism in the values written to the cost table. So, I've reset the thresholds we use for calculating if the test 'passes'.
We may need some time to deal with finding real numbers (that is, my threshold may now be too conservative).
Passing test runs:
All Chromosomes here.
Chromosome 20/X/Y here.
This PR fixes an error in the cost calculation in that we were not handling pre-emption. That is, a shard could get preempted and thus we'd have multiple cost measurements for that shard. I've addressed this by looking for a single 'representative' instance of each shard. But despite this, I found there is still non-determinism in the values written to the cost table. So, I've reset the thresholds we use for calculating if the test 'passes'. We may need some time to deal with finding real numbers (that is, my threshold may now be too conservative).
Passing test runs: All Chromosomes here. Chromosome 20/X/Y here.