The current approach adopted in ssdtools for calculating model averaged confidence intervals based on a simple weighted arithmetic mean yields coverage values that fall substantially short of their notional 95% level, even at quite high samples sizes.
Simulations show that the weighted sample and the log transformed bootstrap mata.wald methods provide similar coverage and perform the best across all methods that do not yield negative lower confidence bounds. Of these two approaches, the weighted sample method appears slightly more robust across datasets and the sample sizes examined compared to the transformed boostrap mata.wald method. The weighted sample method makes no underlying assumption about the distribution of the estimates, and is therefore more robust compared to the other methods.
In addition to being more robust than the other methods, the weighted sample method also has the advantage of not requiring a full number of bootstrap samples to be drawn from each distribution. Rather a sample proportional to the weight of each distribution can be drawn such that the total bootstrap samples equals the number required. This will substantially speed up bootstrap based CI estimation, whilst still yielding a robust result.
The current approach adopted in ssdtools for calculating model averaged confidence intervals based on a simple weighted arithmetic mean yields coverage values that fall substantially short of their notional 95% level, even at quite high samples sizes.
Simulations show that the weighted sample and the log transformed bootstrap mata.wald methods provide similar coverage and perform the best across all methods that do not yield negative lower confidence bounds. Of these two approaches, the weighted sample method appears slightly more robust across datasets and the sample sizes examined compared to the transformed boostrap mata.wald method. The weighted sample method makes no underlying assumption about the distribution of the estimates, and is therefore more robust compared to the other methods.
In addition to being more robust than the other methods, the weighted sample method also has the advantage of not requiring a full number of bootstrap samples to be drawn from each distribution. Rather a sample proportional to the weight of each distribution can be drawn such that the total bootstrap samples equals the number required. This will substantially speed up bootstrap based CI estimation, whilst still yielding a robust result.
A modified fork of ssdtools is available at https://github.com/beckyfisher/ssdtools/tree/keep_samples that saves the bootstrap samples for each distribution to allow implementation of the method.