I wanted to flag a potential issue that I'd noticed. I believe that the p-value calculations aren't necessarily robust to different specifications of a sharp hypothesis that is not zero. In particular, in the conduct_ri.R function, there is a place to calculate a two-tailed p-value by tagging extreme observations:
summary.ri <- function(object, p = NULL, ...) {
if(is.null(p)){p <- object$p}
if (p == "two-tailed") {
object$sims_df <-
within(
object$sims_df,
extreme <- abs(est_sim) >= abs(est_obs)
)
} else if (p == "lower") {
object$sims_df <-
within(
object$sims_df,
extreme <- est_sim <= est_obs
)
} else if (p == "upper") {
object$sims_df <-
within(
object$sims_df,
extreme <- est_sim >= est_obs
)
} else {
stop('p must be either "two-tailed" (the default), "lower", or "upper".')
}
> However, extreme values should be tagged as: extreme <- abs(est_sim - sharp_hypothesis) >= abs(est_obs - sharp_hypothesis) so that this p-value is valid for non-zero null hypotheses (looking for simulated values as or more extreme than the observed value).
I implemented the changes and added a test
This PR addresses a user issue:
summary.ri <- function(object, p = NULL, ...) { if(is.null(p)){p <- object$p} if (p == "two-tailed") { object$sims_df <- within( object$sims_df, extreme <- abs(est_sim) >= abs(est_obs) ) } else if (p == "lower") { object$sims_df <- within( object$sims_df, extreme <- est_sim <= est_obs ) } else if (p == "upper") { object$sims_df <- within( object$sims_df, extreme <- est_sim >= est_obs ) } else { stop('p must be either "two-tailed" (the default), "lower", or "upper".') }