Pacific-salmon-assess / samSim

3 stars 0 forks source link

Is the forecast error lognormally distributed? #20

Closed catarinawor closed 3 years ago

catarinawor commented 3 years ago

The forecast error , if used, is generated in this line I think this is equivalent to a lognormally distributed variable with mu =forecastMean and sigma=forecastSig. Is that correct?

seananderson commented 3 years ago

Yes, but median not mean since it isn't bias corrected and removing the tails.

catarinawor commented 3 years ago

Oh, hi Sean! =) So... should we apply the bias correction? I'm not sure it would matter a lot here, but it is also not a complex change. @CamFreshwater @carrieholt @krHolt On a curiosity note: Any specific reason for using qnorm(runif()) instead of rnorm()? --maybe limit sampling on the tails?

seananderson commented 3 years ago

If I remember correctly, the lack of bias correction was on purpose, possibly for historical reasons, but I could be remembering the wrong project. I imagine the quantile-uniform approach was used to crop the tails of the distribution to avoid extremes. It's a nice trick for converting one distribution to another. One use is 'randomized-quantile' residuals. E.g. https://github.com/pbs-assess/sdmTMB/blob/master/R/residuals.R to turn non-Gaussian residuals into something that should be normal. It's the same trick used in the DHARMa R package.

CamFreshwater commented 3 years ago

Yes I remember debating the bias correction issue and believe we decided to keep it unadjusted for historical reasons and because even recent papers differ in whether or not it's applied. I know @carrieholt went on a bit of a deep dive here so she can probably weigh in more. I also think Sean's correct on the quantile-uniform approach, but I have to admit that it was a design that Carrie was already using in her original chum CSAS model so I can't speak directly to the rationale!

carrieholt commented 3 years ago

Yes, bias correction was left out of forecast because forecasts for P salmon generally don't include it. However, we are including bias correction in log-normal distributions in population dynamics model, and so, when we start to use the observation error component of the model again down the road, we should also adjust those. Cam, I have a table that we used in the write-up for the recent SBC CK RPA (in press) that documents when/where log-normal bias corrections have been included for salmon assessments and projections that might be helpful (CSAS and primary lit).
As for the randomized-quantile approach, yes that was a legacy of some old code where we inputted a set of runifs to ensure the same random numbers were used for each scenario, and not necessarily meant to avoid extremes.

catarinawor commented 3 years ago

I will open another issue to remind us to include the bias correction option for forecasts whenever that option is used.