This R package provides scoring mechanisms for computational challenges and implements the bayesBootLadderBoot approach for avoiding test data leakage.
Apache License 2.0
3
stars
0
forks
source link
add handling of when reference column is not the best #25
This change will handle the case for when the reference column is not the best set of predictions, i.e. baseline model. When this happens, the inversion of K (BF) will change, depending on which submission it is being compared to.
I did test this change with a sample bootstapped matrix to ensure the behavior will not change when the reference column is the best:
> computeBayesFactorWhereRefIsNotBest(boot, refPredIndex=1, invertBayes=F) %>%
+ as_tibble(rownames = "submission") %>%
+ rename(bayes = value)
# A tibble: 3 x 2
submission bayes
<chr> <dbl>
1 team A 0
2 team B 8.09
3 team C Inf
> computeBayesFactor(boot, refPredIndex=1, invertBayes=F) %>%
+ as_tibble(rownames = "submission") %>%
+ rename(bayes = value)
# A tibble: 3 x 2
submission bayes
<chr> <dbl>
1 team A 0
2 team B 8.09
3 team C Inf
(Adding change for @allaway )
This change will handle the case for when the reference column is not the best set of predictions, i.e. baseline model. When this happens, the inversion of K (BF) will change, depending on which submission it is being compared to.
I did test this change with a sample bootstapped matrix to ensure the behavior will not change when the reference column is the best: