This is currently using a branch of ALBench and needs to use a pypi release before we merge.
In addition to the confidence array in the metadata, this add a certainty array based on the ALBench computed certainty. We also save "certainty_info" which is a dict of "type" (the type used for computing certainty), "percentiles", and "cdf". Percentiles and cdf are both computed for a selected set of values. For percentiles, this could show how we are approaching some sort of limit, but since the certainty values aren't on the same range for each certainty type, I'm not sure how we would interpret this. And, the computed cdf values are probably worthless on certainty types where the certainty values aren't in the range [0,1].
This is currently using a branch of ALBench and needs to use a pypi release before we merge.
In addition to the confidence array in the metadata, this add a certainty array based on the ALBench computed certainty. We also save "certainty_info" which is a dict of "type" (the type used for computing certainty), "percentiles", and "cdf". Percentiles and cdf are both computed for a selected set of values. For percentiles, this could show how we are approaching some sort of limit, but since the certainty values aren't on the same range for each certainty type, I'm not sure how we would interpret this. And, the computed cdf values are probably worthless on certainty types where the certainty values aren't in the range [0,1].