MStarmans91
commented
9 months ago That can happen. In the performance_all_0.json, for all performance metrics, by default 95% confidence intervals are reported. To construct these, quoting the WORC paper:
When a single dataset is used, and thus a ktest = 100 random-split cross-validation, 95% confidence intervals of the performance metrics are constructed using the corrected resampled t-test, thereby taking into account that the samples in the cross-validation splits are not statistically independent (Nadeau and Bengio, 2003).
The relevant function can be found here: https://github.com/MStarmans91/WORC/blob/master/WORC/plotting/compute_CI.py#L36. Practically, this means that we apply a correction to the standard deviation, which generally widens the confidence interval. This can cause the confidence interval to both go below 0 or above 1. As such a value in practice of course cannot occur, you can cap it in the expacted range of (0, 1) when reporting it in e.g. a paper. But I decided to keep it this way to show you the true output of the computation.
Let me know if any unclearity remains.
xinyiwan
Describe the bug In the final performance reports (
performance_all_0.json), the max value in the specificity range is over 1.WORC configuration no customization
fastr trace
To Reproduce
Expected behavior Specificity values in the rage of (0,1)
Desktop (please complete the following information):
Additional context no.