[Feature]: Do we need to put bias adjustment fraction within the likelihood?

Bai-Li-NOAA commented 2 years ago

Is your feature request related to a problem? Please describe.

While developing the recruitment likelihood component, the implementation subgroup came across a question that asked if we need the bias adjustment fraction within the likelihood. Can't we just use the standard likelihood formulation without modification?

The standard negative log-likelihood (nll) is

$nll = \frac{n}{2}ln(2\pi)+\frac{1}{2}{\sum{y=1}}^n(\frac{r{y}^2}{\sigma{R}^2}+ln(\sigma{R}^2))$

where $n, r{y}, and \sigma{R}$ are the number of data points, recruitment deviations in log space, and standard deviation in log space respectively.

The nll that includes bias adjustment fraction based on A.3.10 from Methot and Wetzel 2013 is

$nll = \frac{n}{2}ln(2\pi)+\frac{1}{2}{\sum{y=1}}^n(\frac{r{y}^2}{\sigma{R}^2}+b{y}ln(\sigma{R}^2))$ where $b{y}$ is the bias adjustment fraction.

But we are not sure about back-transforming the equation 2 out to the original likelihood form and how the derivation justified:

$\frac{1}{\sqrt{2\pi}^n {\sqrt{\sigma{R}^2}}^{\sum{b{y}}}}e^{-{\prod{y=1}}^n \frac{r{y}^2}{2\sigma_{R}^2}}$

Currently, we have implemented the standard nll (equation 1) in FIMS for M1. Do we need to modify the code and include bias adjustment fraction?

@KyleShertzer-NOAA, @iantaylor-NOAA, @Andrea-Havron-NOAA, please feel free to edit the description.

Describe the solution you would like.

NA

Describe alternatives you have considered

NA

Statistical validity, if applicable

No response

Describe if this is needed for a management application

No response

Additional context

No response

Code of Conduct

[X] I agree to follow this project's Code of Conduct

iantaylor-NOAA commented 2 years ago

Explanation of bias adjustment in equation (2) from Methot and Taylor (2011) is as follows. This is less a formal statistical derivation and more a pragmatic solution that provides for desirable properties. @Rick-Methot-NOAA, please feel free to add more info on this.

ChristineStawitz-NOAA commented 2 years ago

We are inclined to leave this as is and revisit post-M1 since existing models without the bias adjustment fraction in the likelihood (AMAK, MAS, BAM, ASAP) seem to yield comparable results.

@kellijohnson-NOAA mentions this likely only matters if you are missing many years of age comp data and have only a couple years in the beginning that aren't modeled fully and the rest of the years $b_y$ is close to 1

Rick-Methot-NOAA commented 2 years ago

Agree with Kelli. When models are supplied with good age data for entire time series, the issue of bias adjustment is ignorable. Bias adjustment is most important when age data are weak or missing. SS3 models that use only length data and still try to estimate recdevs are the situation most in need of bias adjustment concept.

Cole-Monnahan-NOAA commented 1 year ago

@Rick-Methot-NOAA Do we even need to do this manual bias correction to recruitment when TMB has a generic form built in already? This is a penalized ML thing to do and we're not longer bound by ADMB's limitations.

https://doi.org/10.1016/j.fishres.2015.11.016

In this paper, we have demonstrated that a new epsilon estimator improves upon the bias-correction algorithm previously introduced by Methot and Taylor (2011) for models that treat recruitment deviations as random effects.

Rick-Methot-NOAA commented 1 year ago

Good question Cole. I hope you can avoid the bias adjustment for the reasons demonstrated in that paper by Jim and Kasper. The paper by Ian and me showed that the bias adjustment was unnecessary when using ADMB's MCMC and SS3 turns off the recruitment bias adjustment when running in MCMC mode. I do not know if TMB will adequately deal with the situation that requires the bias adjustment ramp in ADMB's MLE. So, I recommend that someone try TMB with a ramp in data quality regarding recruitment and see if TMB adequately adjusts for that.

Cole-Monnahan-NOAA commented 1 year ago

@james-thorson-noaa could you chime in on this topic? The basic question is whether FIMS should build in the recruitment bias adjustment ramp or if your generic method totally supersedes that?

@Rick-Methot-NOAA The ramp option won't match the generic one, so how would we set up this experiment? Put the ramp into WHAM and compare the two? Compare it to MCMC?

Andrea-Havron-NOAA commented 1 year ago

@timjmiller is looking into bias adjustment as part of WHAM's research track and could potentially provide guidance on how we develop this in FIMS.

Rick-Methot-NOAA commented 1 year ago

I would:

simulate a time series of recruitment at some sigmaR
sample composition data from the generated population. Let the effN of that sampling increase over time
run MLE and MCMC estimation to see if TMB's MLE can reproduce the arithmetic mean recruitment in the early years.

Cole-Monnahan-NOAA commented 1 year ago

OK I'll try it with a simple WHAM model. I'm assuming something like effN=seq(5,100, len=nyrs) would suffice?

timjmiller commented 1 year ago

I am not sure why it is better for Recruitment or any other log-normal RV to be mean unbiased vs. median unbiased? When we do simulation studies we often look at median unbiasedness, right? At least for variables that are bounded below by 0.

On Tue, Aug 1, 2023 at 1:26 PM Richard Methot @.***> wrote:

I would:

simulate a time series of recruitment at some sigmaR

sample composition data from the generated population. Let the effN of that sampling increase over time

run MLE and MCMC estimation to see if TMB's MLE can reproduce the arithmetic mean recruitment in the early years.

— Reply to this email directly, view it on GitHub https://github.com/NOAA-FIMS/FIMS/issues/185#issuecomment-1660781902, or unsubscribe https://github.com/notifications/unsubscribe-auth/AEIGN7HDAC3QABEJEMZ6OUDXTE34BANCNFSM52CVN2FA . You are receiving this because you were mentioned.Message ID: @.***>

-- Timothy J. Miller, PhD (he, him, his) Research Fishery Biologist NOAA, Northeast Fisheries Science Center Woods Hole, MA 508-495-2365

James-Thorson-NOAA commented 1 year ago

Cole -- I imagine you could explore both epsilon bias-correction and MCMC posterior mean estimators...?

On Tue, Aug 1, 2023 at 10:43 AM Cole Monnahan @.***> wrote:

OK I'll try it with a simple WHAM model. I'm assuming something like effN=seq(5,100, len=nyrs) would suffice?

— Reply to this email directly, view it on GitHub https://github.com/NOAA-FIMS/FIMS/issues/185#issuecomment-1660803768, or unsubscribe https://github.com/notifications/unsubscribe-auth/AL62VMQ3OBIDN2NUWGPJVZ3XTE52NANCNFSM52CVN2FA . You are receiving this because you were mentioned.Message ID: @.***>

James-Thorson-NOAA commented 1 year ago

I assume there's some national standards guidance but have never dug in ... changing the definition of B0 from average unfished biomass to median unfished biomass seems like a big change for many stocks :0

On Tue, Aug 1, 2023 at 11:05 AM Tim Miller @.***> wrote:

I am not sure why it is better for Recruitment or any other log-normal RV to be mean unbiased vs. median unbiased? When we do simulation studies we often look at median unbiasedness, right? At least for variables that are bounded below by 0.

On Tue, Aug 1, 2023 at 1:26 PM Richard Methot @.***> wrote:

I would:

simulate a time series of recruitment at some sigmaR

sample composition data from the generated population. Let the effN of that sampling increase over time

run MLE and MCMC estimation to see if TMB's MLE can reproduce the arithmetic mean recruitment in the early years.

— Reply to this email directly, view it on GitHub https://github.com/NOAA-FIMS/FIMS/issues/185#issuecomment-1660781902, or unsubscribe < https://github.com/notifications/unsubscribe-auth/AEIGN7HDAC3QABEJEMZ6OUDXTE34BANCNFSM52CVN2FA>

. You are receiving this because you were mentioned.Message ID: @.***>

-- Timothy J. Miller, PhD (he, him, his) Research Fishery Biologist NOAA, Northeast Fisheries Science Center Woods Hole, MA 508-495-2365

— Reply to this email directly, view it on GitHub https://github.com/NOAA-FIMS/FIMS/issues/185#issuecomment-1660833497, or unsubscribe https://github.com/notifications/unsubscribe-auth/AL62VMV4VPUTXP6FUQJAHPLXTFALPANCNFSM52CVN2FA . You are receiving this because you were mentioned.Message ID: @.***>

Rick-Methot-NOAA commented 1 year ago

Tim - population abundance must be mean unbiased because the catch coming from that population is mean unbiased.

Cole-Monnahan-NOAA commented 1 year ago

I worked up a simple simulation example using WHAM (script to reproduce but note you have to install a modified branch of WHAM b/c it doesn't ADREPORT recruits). The surprising result is that bias correction does almost nothing in this example, even if I turn the age comp data way down. sigmaR=1.35 so it's really big.

The key code modifications are

effN <- floor(seq(5,100, len=44))
effN[10:20] <- 1
input2$data$catch_Neff[,1] <- effN
input2$data$index_Neff <- cbind(effN,effN)*0+1

I left some early fishery age data so it could estimate initial age structure, but then have some years with only a single MN sample for the fishery. The two surveys have 1 in every year just so I don't have to change the code as much.

It's possible (likely?) I'm doing the simulation wrong and I'll check again tomorrow. @timjmiller can probably most efficiently check this if it's worth pursuing further.

timjmiller commented 1 year ago

@Cole Monnahan - NOAA Federal @.***> The log recruits should be adreported as the first column of log_NAA_rep. At any rate, I haven't noticed the epsilon bias correction for random effects to produce different estimates unless you get away from years where there are data and where the transformation is more than a simple exponentiation of a normal RE.

On Tue, Aug 1, 2023 at 7:38 PM Cole Monnahan @.***> wrote:

I worked up a simple simulation example using WHAM (script to reproduce https://github.com/Cole-Monnahan-NOAA/wham/blob/add_recruit_reporting/inst/example_scripts/test_bias_adjust.R but note you have to install a modified branch of WHAM b/c it doesn't ADREPORT recruits). The surprising result is that bias correction does almost nothing in this example, even if I turn the age comp data way down. sigmaR=1.35 so it's really big.

[image: image] https://user-images.githubusercontent.com/58868176/257663699-69514f1c-397a-4fc6-b2a7-a427f91c869e.png

The key code modifications are

effN <- floor(seq(5,100, len=44)) effN[10:20] <- 1 input2$data$catch_Neff[,1] <- effN input2$data$index_Neff <- cbind(effN,effN)*0+1

I left some early fishery age data so it could estimate initial age structure, but then have some years with only a single MN sample for the fishery. The two surveys have 1 in every year just so I don't have to change the code as much.

It's possible (likely?) I'm doing the simulation wrong and I'll check again tomorrow. @timjmiller https://github.com/timjmiller can probably most efficiently check this if it's worth pursuing further.

— Reply to this email directly, view it on GitHub https://github.com/NOAA-FIMS/FIMS/issues/185#issuecomment-1661245311, or unsubscribe https://github.com/notifications/unsubscribe-auth/AEIGN7BRQERRMGGTVSN7FGDXTGHNFANCNFSM52CVN2FA . You are receiving this because you were mentioned.Message ID: @.***>

-- Timothy J. Miller, PhD (he, him, his) Research Fishery Biologist NOAA, Northeast Fisheries Science Center Woods Hole, MA 508-495-2365

James-Thorson-NOAA commented 1 year ago

Yeah, my experience testing in TMB using CCSRA suggests that we'd expect the Methot-Taylor method (i.e., correcting by exp(SE^2/2)) to break down when there's large covariances in estimated rec-devs. This can occur when there's some information about initial depletion and you're modelling rec-devs occurring prior to that information (i.e., some constraint on the sum of exponentiated rec-devs). See e.g. the SRA scenarios of Fig. 2 here https://www.sciencedirect.com/science/article/pii/S0165783615301399. I think this occurs mainly in the NWFSC assessments, which start the model and associated rec-devs well prior to the first data (It seems like AFSC and NEFSC assessment typically start around the onset of good data)

Jim

On Tue, Aug 1, 2023 at 7:54 PM Tim Miller @.***> wrote:

@Cole Monnahan - NOAA Federal @.***> The log recruits should be adreported as the first column of log_NAA_rep. At any rate, I haven't noticed the epsilon bias correction for random effects to produce different estimates unless you get away from years where there are data and where the transformation is more than a simple exponentiation of a normal RE.

On Tue, Aug 1, 2023 at 7:38 PM Cole Monnahan @.***> wrote:

I worked up a simple simulation example using WHAM (script to reproduce < https://github.com/Cole-Monnahan-NOAA/wham/blob/add_recruit_reporting/inst/example_scripts/test_bias_adjust.R>

but note you have to install a modified branch of WHAM b/c it doesn't ADREPORT recruits). The surprising result is that bias correction does almost nothing in this example, even if I turn the age comp data way down. sigmaR=1.35 so it's really big.

[image: image] < https://user-images.githubusercontent.com/58868176/257663699-69514f1c-397a-4fc6-b2a7-a427f91c869e.png>

The key code modifications are

effN <- floor(seq(5,100, len=44)) effN[10:20] <- 1 input2$data$catch_Neff[,1] <- effN input2$data$index_Neff <- cbind(effN,effN)*0+1

I left some early fishery age data so it could estimate initial age structure, but then have some years with only a single MN sample for the fishery. The two surveys have 1 in every year just so I don't have to change the code as much.

It's possible (likely?) I'm doing the simulation wrong and I'll check again tomorrow. @timjmiller https://github.com/timjmiller can probably most efficiently check this if it's worth pursuing further.

— Reply to this email directly, view it on GitHub https://github.com/NOAA-FIMS/FIMS/issues/185#issuecomment-1661245311, or unsubscribe < https://github.com/notifications/unsubscribe-auth/AEIGN7BRQERRMGGTVSN7FGDXTGHNFANCNFSM52CVN2FA>

. You are receiving this because you were mentioned.Message ID: @.***>

-- Timothy J. Miller, PhD (he, him, his) Research Fishery Biologist NOAA, Northeast Fisheries Science Center Woods Hole, MA 508-495-2365

— Reply to this email directly, view it on GitHub https://github.com/NOAA-FIMS/FIMS/issues/185#issuecomment-1661412493, or unsubscribe https://github.com/notifications/unsubscribe-auth/AL62VMWPRLUCCCMDZIJ5EXTXTG6PJANCNFSM52CVN2FA . You are receiving this because you were mentioned.Message ID: @.***>

Cole-Monnahan-NOAA commented 1 year ago

@timjmiller my script certainly corroborates that finding. I did find a mistake I was taking mean of the posterior so I switched to median and now they all match very closely.

But I'm confused b/c I have some years with very little age data and sigmaR is big so the bias correction should be impactful in those cases. I also tried with bias_correct_pe=0 and it made no difference. So I'm at a bit of a loss there. @James-Thorson-NOAA is this unexpected from your perspective?

James-Thorson-NOAA commented 1 year ago

Cole

Before answering, can you show the example table: [image: image.png] but also including a 4th column with the MLE SE for each row?

I'm guessing that the CV is very low (which would also explain why the mean and median for MCMC are so similar). I wouldn't expect any type of bias-correction to matter for a stock that has low SEs (as I'm guessing this one does)

On Wed, Aug 2, 2023 at 9:26 AM Cole Monnahan @.***> wrote:

@timjmiller https://github.com/timjmiller my script certainly corroborates that finding. I did find a mistake I was taking mean of the posterior so I switched to median and now they all match very closely.

But I'm confused b/c I have some years with very little age data and sigmaR is big so the bias correction should be impactful in those cases. I also tried with bias_correct_pe=0 and it made no difference. So I'm at a bit of a loss there. @James-Thorson-NOAA https://github.com/James-Thorson-NOAA is this unexpected from your perspective?

— Reply to this email directly, view it on GitHub https://github.com/NOAA-FIMS/FIMS/issues/185#issuecomment-1662537194, or unsubscribe https://github.com/notifications/unsubscribe-auth/AL62VMXI5V4JDM4M3JPX76LXTJ5SNANCNFSM52CVN2FA . You are receiving this because you were mentioned.Message ID: @.***>

ChristineStawitz-NOAA commented 1 year ago

@James-Thorson-NOAA your image didn't come through - can you add it again please?

James-Thorson-NOAA commented 1 year ago

I just meant the same image that Cole had sent earlier (three columns showing MLE, epsilon and MCMC mean)

On Wed, Aug 2, 2023 at 10:11 AM Christine Stawitz - NOAA < @.***> wrote:

@James-Thorson-NOAA https://github.com/James-Thorson-NOAA your image didn't come through - can you add it again please?

— Reply to this email directly, view it on GitHub https://github.com/NOAA-FIMS/FIMS/issues/185#issuecomment-1662624711, or unsubscribe https://github.com/notifications/unsubscribe-auth/AL62VMT5RPOLLIMFJSBS4E3XTKC3FANCNFSM52CVN2FA . You are receiving this because you were mentioned.Message ID: @.***>

Rick-Methot-NOAA commented 1 year ago

In Cole's example, the estimated recruitments are showing large changes between years, which means there is a lot more data informing them than many of the long time series situations that the bias ramp in SS3 is designed for.

A typical situation with sigmaR=0.4 (current Petrale sole assessment) has the se of each recent log recruitment down to 0.1, whereas the se of the early recruitments is equal to 0.4 because they are only informed by sigmaR. All the early recruitments have a MLE value of 0.0 and they show nil autocorrelation. Info from Ian Taylor

Cole-Monnahan-NOAA commented 1 year ago

I'm trying to produce a situation like Rick's using a packaged WHAM example but it is turning out to be difficult. I got something close with these estimated recruits:

but it is quite contrived. I also can't get MCMC to work b/c the early Fdevs wander off to +/- Inf for reasons I don't understand. But we can compare the MLE estimates with and without the bias correction, below are the first 15 years of estimates of recruits. So it's definitely doing more now, but I'm surprised to see that sometimes the correction decreases and sometimes increases the estimate of recruits.

Really we should find an SS3 model like Rick's example that has been ported to WHAM or some other TMB platform so that we can more directly test this effect. Does anyone have that lying around? I can do the legwork.

>    uncorrected corrected mle_se
1        26240     26240      0
2        15331     15104   6503
3         5252      5648   3680
4         9068      8252   5526
5        63903     66185 111040
6        12197      9759  13889
7        14055     13389  17324
8        17901     23222  25598
9        20123     16308  26755
10       20119     17816  27244
11       22049     27124  31902
12       24268     23625  34294
13       23718     18418  31945
14       18823     25157  25127
15       17124     14196  15839

James-Thorson-NOAA commented 1 year ago

My memory is that a negative correction is not possible using Methot-Taylor, but does in fact happen using epsilon correction (and make sense) when the covariance is non-zero (and I think highly negative) between rec-devs.

but I agree that it would be easier to find a model that already has a non-trivial estimate of the bias-ramp, if the goal is to investigate simple ways of replacing the bias-ramp using options in TMB.

Rick-Methot-NOAA commented 1 year ago

Sorry Cole, I cannot help with porting to WHAM.

While on this bias adjustment topic, a side issue is that large recruitments are estimated more precisely than weak recruitments (see trend in the green values). Also note that sometimes the se on the early recruitments is >sigmaR. This might be consequence of sum-to-zero constraint. The lower se on large recruitments might be due to: (a) use of multinomial logL; (b) data are length comps and age comps with ageing error, so small recruitments disappear into the noise. A corollary of this as noted in M&T(2011) is that large positive recdevs will get estimated, but -2sigmaR devs don't get estimated because they are lost in the noise and the sigmaR penalty draws them towards 0.0.

timjmiller commented 1 year ago

Is there a declining trend in recruitment over time through the data period though? Just wondering if this is due to poorer estimation of recruitment at the end of the time series.

On Wed, Aug 2, 2023 at 5:32 PM Richard Methot @.***> wrote:

Sorry Cole, I cannot help with porting to WHAM.

While on this bias adjustment topic, a side issue is that large recruitments are estimated more precisely than weak recruitments. [image: image] https://user-images.githubusercontent.com/6250016/257946704-65be585c-5e47-465f-b3d3-c6353be55f9d.png

— Reply to this email directly, view it on GitHub https://github.com/NOAA-FIMS/FIMS/issues/185#issuecomment-1662991243, or unsubscribe https://github.com/notifications/unsubscribe-auth/AEIGN7E4NYGV2ZW7I4PKBD3XTLBNVANCNFSM52CVN2FA . You are receiving this because you were mentioned.Message ID: @.***>

-- Timothy J. Miller, PhD (he, him, his) Research Fishery Biologist NOAA, Northeast Fisheries Science Center Woods Hole, MA 508-495-2365

Rick-Methot-NOAA commented 1 year ago

Tim - no trend in recruitment. And Ian & I saw the same pattern in our simulation work in 2011 which had a much larger sample size than one example time series.

Cole-Monnahan-NOAA commented 1 year ago

I think we're at a bit of a standstill on this issue. To really investigate this we need a more thorough example that uses the ramp in SS3 and is ported to WHAM or another TMB platform. Maybe @kellijohnson-NOAA knows of one?

If not, I suggest that we not include the bias adjustment ramp in FIMS. Thorson and Kristensen provide a theoretical argument and examples that the generic bias adjustment is better. It's also not used in other platforms. People bridging from SS3 can rerun their models without the ramp to aid that process, then turn bias correction on once in FIMS. If it turns out the ramp is necessary then we can revisit it.

Rick-Methot-NOAA commented 1 year ago

Much prefer to see your TMB example completed first. Can you rerun it with zero data for the first 10 years of the time series? Are estimated recruitments during those years mean or median unbiased?

James-Thorson-NOAA commented 1 year ago

If the goal is just a TMB model showing this, would you be satisfied with an experiment in CCSRA (here https://github.com/James-Thorson/CCSRA)?

It seems to be difficult to find a case where the bias-correction ramp does anything interesting using WHAM (I'm not sure why). As Cole said, there's no sense in testing the epsilon correction as a solution to the bias-correction ramp until we have a TMB model that has an interesting bias-correction ramp.

Jim

On Tue, Aug 8, 2023 at 4:57 PM Richard Methot @.***> wrote:

Much prefer to see your TMB example completed first. Can you rerun it with zero data for the first 10 years of the time series? Are estimated recruitments during those years mean or median unbiased?

— Reply to this email directly, view it on GitHub https://github.com/NOAA-FIMS/FIMS/issues/185#issuecomment-1670461849, or unsubscribe https://github.com/notifications/unsubscribe-auth/AL62VMX66CGCJQQKVTCCFX3XULG5TANCNFSM52CVN2FA . You are receiving this because you were mentioned.Message ID: @.***>

Rick-Methot-NOAA commented 1 year ago

I do not think CCSRA will demonstrate what we need because it only has data in the final year. The ramp concept is only relevant in situations where we have a trend over time in data quality regarding some RE.

James-Thorson-NOAA commented 1 year ago

Rick,

CCSRA started with demonstrating that option (hence the name), but in fact it fits an index and marginal-age comps in any year. I've used in several times for other purposes, e.g., see Fig. 2 here https://doi.org/10.1016/j.fishres.2015.11.016 which used CCSRA and could be replicated.

On Wed, Aug 9, 2023 at 8:00 AM Richard Methot @.***> wrote:

I do not think CCSRA will demonstrate what we need because it only has data in the final year. The ramp concept is only relevant in situations where we have a trend over time in data quality regarding some RE.

— Reply to this email directly, view it on GitHub https://github.com/NOAA-FIMS/FIMS/issues/185#issuecomment-1671578441, or unsubscribe https://github.com/notifications/unsubscribe-auth/AL62VMS2JH4CFRPUSGUWSZDXUOQVVANCNFSM52CVN2FA . You are receiving this because you were mentioned.Message ID: @.***>

Rick-Methot-NOAA commented 1 year ago

Thanks Jim. seems that suitable example can be created with CCSRA.

Cole-Monnahan-NOAA commented 1 year ago

If we can agree that's a suitable example and it already shows improved performance compared to the ramp then I think we're safe to close this issue with the conclusion to not include any bias correction into FIMS and let that be done in TMB instead. Any remaining objections? If not I'll close this out by the end of the week.

James-Thorson-NOAA commented 1 year ago

Just saw your response Cole ... I spent a couple minutes simplifying the tutorial.R script on github https://github.com/James-Thorson/CCSRA/tree/main/examples, and please see that attached.

I ran it with age-comps using the linear-DM and epsilon bias-correction. The rec-devs RecDev_hat are here, where first column is estimate, 2nd is SE, and 3rd is epsilon bias-correction. Note that final RecDev_hats go to zero, and bias-correction has no effect on those (continuing below). [image: image.png]

next we can see RecMult_t = exp(RecDev_hat). The terminal year plug-in estimator for RecMult_t in those final years is 0.83, which is exp(0-0.6^2/2) where 0.6 is the fixed sigmaR value. However, the epsilon bias-correction estimator for terminal years is 0.985, which is close to the bias-corrected value of 1.00 that is the known true value.

[image: image.png]

Maybe I didn't need to break out the actual demo here, but please feel free to explore!

On Wed, Aug 9, 2023 at 8:44 AM Cole Monnahan @.***> wrote:

If we can agree that's a suitable example and it already shows improved performance compared to the ramp then I think we're safe to close this issue with the conclusion to not include any bias correction into FIMS and let that be done in TMB instead. Any remaining objections? If not I'll close this out by the end of the week.

— Reply to this email directly, view it on GitHub https://github.com/NOAA-FIMS/FIMS/issues/185#issuecomment-1671656844, or unsubscribe https://github.com/notifications/unsubscribe-auth/AL62VMRBCOIJWN4CNENCGUDXUOV7HANCNFSM52CVN2FA . You are receiving this because you were mentioned.Message ID: @.***>

Rick-Methot-NOAA commented 1 year ago

Looks good to proceed with the TMB approach.

jimianelli commented 1 year ago

Thanks Jim T. minor point but tried tutorial.R and devtools::install_github("James-Thorson/CCSRA", ref="dev") failed but: devtools::install_github("James-Thorson/CCSRA") worked...maybe no branch called dev?

On Wed, Aug 9, 2023 at 9:55 AM Jim Thorson @.***> wrote:

Just saw your response Cole ... I spent a couple minutes simplifying the tutorial.R script on github https://github.com/James-Thorson/CCSRA/tree/main/examples, and please see that attached.

I ran it with age-comps using the linear-DM and epsilon bias-correction. The rec-devs RecDev_hat are here, where first column is estimate, 2nd is SE, and 3rd is epsilon bias-correction. Note that final RecDev_hats go to zero, and bias-correction has no effect on those (continuing below). [image: image.png]

next we can see RecMult_t = exp(RecDev_hat). The terminal year plug-in estimator for RecMult_t in those final years is 0.83, which is exp(0-0.6^2/2) where 0.6 is the fixed sigmaR value. However, the epsilon bias-correction estimator for terminal years is 0.985, which is close to the bias-corrected value of 1.00 that is the known true value.

[image: image.png]

Maybe I didn't need to break out the actual demo here, but please feel free to explore!

On Wed, Aug 9, 2023 at 8:44 AM Cole Monnahan @.***> wrote:

If we can agree that's a suitable example and it already shows improved performance compared to the ramp then I think we're safe to close this issue with the conclusion to not include any bias correction into FIMS and let that be done in TMB instead. Any remaining objections? If not I'll close this out by the end of the week.

— Reply to this email directly, view it on GitHub https://github.com/NOAA-FIMS/FIMS/issues/185#issuecomment-1671656844, or unsubscribe < https://github.com/notifications/unsubscribe-auth/AL62VMRBCOIJWN4CNENCGUDXUOV7HANCNFSM52CVN2FA>

. You are receiving this because you were mentioned.Message ID: @.***>

— Reply to this email directly, view it on GitHub https://github.com/NOAA-FIMS/FIMS/issues/185#issuecomment-1671800958, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAUW7YUZUFDU5RTDOMGUHT3XUO6HXANCNFSM52CVN2FA . You are receiving this because you are subscribed to this thread.Message ID: @.***>

-- Jim Ianelli

Cole-Monnahan-NOAA commented 1 year ago

Closing this with conclusion to not include the bias adjustment in FIMS and let TMB do it externally. We can revisit once we have a handful of SS3 models ported to FIMS and can explore the impacts of the ramp more thoroughly.

NOAA-FIMS / FIMS