In order for this to meaningful, that means that we need to have reference prices for all tasks in the workload that are accurate down to around 1e-8. That may actually be infeasible, but it would be interesting if it were a spur or a challenge to people.
Setting of Accuracy Levels
My feeling is that there should be a way to define this in a way that depends on $n$, but I'm not sure of the maths right now. I'd have to do some thinking, but it feels like the maximum allowed error should grow as a function of $\sqrt{n}$, both for statistical and numerical reasons.
From the document:
Feasibility
In order for this to meaningful, that means that we need to have reference prices for all tasks in the workload that are accurate down to around 1e-8. That may actually be infeasible, but it would be interesting if it were a spur or a challenge to people.
Setting of Accuracy Levels
My feeling is that there should be a way to define this in a way that depends on $n$, but I'm not sure of the maths right now. I'd have to do some thinking, but it feels like the maximum allowed error should grow as a function of $\sqrt{n}$, both for statistical and numerical reasons.