hyunjimoon
commented
3 years ago I am not sure about the use of underpower, though. The described situation seems to be related to false-positive, i.e. unable to detect misspecified models is type one error. Power is 1-beta. Underpower has high type two error and therefore highly false-negative; overly conservative and saying all models are misspecified for example.
Package could provide the scope of prior parameter based on SBC result as follows:
experiment results on different prior settings
Figure 10: SBC histogram with different prior parameters and shape
SBC histograms corresponding to different prior distributions, normal, t, and invese gamma are compared in Figure10. ρ parameter for inverse gamma prior is interesting. Heavy tail prior could have caused large variance in simulated y values and therefore imposed small values on data averaged posterior sample of ρ .
concerns
The following problems have been raised and should be addressed.
SBC will be extremely under-powered to detect misspecification if the prior dispersion is vastly different than the posterior dispersion, either greater than or less than. But that means that you cannot use the SBC to compare vastly different prior parameters, since the power of the test may be quite different for different prior parameters. In other words, you may find that certain prior parameters "pass" the SBC test simply because the SBC test no longer has any power to detect misspecification, rather than because those prior parameters are actually giving you more reliable results. That seems like a very real concern.
The final step of SBC is a check for uniformity of the q_i, and this check is a frequentist test. If you had an infinite number of replications for SBC (M = \infty), then there would be no need for a frequentist test, you would simply see whether the q_i were exactly linear. But because you run for finite M, you need to account for the randomness in SBC (specifically, in steps 1 and 2) with a frequentist test for deviation from uniformity, and worry about the power of that test to detect deviations from uniformity.
I'm trying to point out here that when the prior and posterior dispersion are very mismatched, then the deviation from uniformity will be quite small, and this final frequentist test will be under-powered. In general, the power of the frequentist test depends on the prior parameters. And so using the SBC to choose which prior parameter to use will inherently favor using prior parameters for which SBC is under-powered.
solutions
For this, truncating the prior to, 1.25 * the box containing all posterior samples, for example, before running SBC might be one solution. This we can bound the dispersion and power among our prior parameter comparison.