bcgov / ssdtools

An R package to fit and plot Species Sensitivity Distributions (SSDs)
https://bcgov.github.io/ssdtools/
Apache License 2.0
31 stars 16 forks source link

Implement weighted bootstrap sample method for estimating CI #270

Closed beckyfisher closed 11 months ago

beckyfisher commented 2 years ago

The current approach adopted in ssdtools for calculating model averaged confidence intervals based on a simple weighted arithmetic mean yields coverage values that fall substantially short of their notional 95% level, even at quite high samples sizes.

Simulations show that the weighted sample and the log transformed bootstrap mata.wald methods provide similar coverage and perform the best across all methods that do not yield negative lower confidence bounds. Of these two approaches, the weighted sample method appears slightly more robust across datasets and the sample sizes examined compared to the transformed boostrap mata.wald method. The weighted sample method makes no underlying assumption about the distribution of the estimates, and is therefore more robust compared to the other methods.

In addition to being more robust than the other methods, the weighted sample method also has the advantage of not requiring a full number of bootstrap samples to be drawn from each distribution. Rather a sample proportional to the weight of each distribution can be drawn such that the total bootstrap samples equals the number required. This will substantially speed up bootstrap based CI estimation, whilst still yielding a robust result.

A modified fork of ssdtools is available at https://github.com/beckyfisher/ssdtools/tree/keep_samples that saves the bootstrap samples for each distribution to allow implementation of the method.

joethorley commented 2 years ago

This is really important.

joethorley commented 11 months ago

Done in main branch as multi method.