SS forecast option 5=input annual F scalar issues

[k-doering-NOAA]

Large SD for forecasting values in 1 year found in a model sent by Katie Siegfried indicates something has gone awry with this option (at least in the particular model). See email thread "problems in with SS forecast option 5=input annual F scalar" for additional details.

[Rick-Methot-NOAA]

1. I reproduced the problem
2. note that it is just one year. It is not a special year (nothing else changing at that time), and the point estimate for that year seems entirely normal.
3. changing the value of the input F scalar from 0.63 to 0.40 removes the problem
4. changing the inflection of the control rule from 0.40 to anything else (closest I tried was 0.45) also causes the problem to go away.

So my hunch is that this is a particular numerical issue. It could be that it is due to some gradient getting calculated right next to a bin boundary. But with it only showing up in the variance and not the point estimate, I'm not sure how I would even go about investigating a change in the code to make the code robust to this issue.

If I was to make a next step at investigating, I would do a profile on my point #4 above to narrow down exactly where this happens and would then look closely at forecast-report.sso for additional clues. I'd reduce forecast years to 10 just to make the files and runtime a bit more manageable.

[k-doering-NOAA]

I added a "wontfix" label, @RickMethot , does this seem appropriate? I'm thinking that issues like this that can't be solved should be tagged somehow, in case we stumble upon a similar problem in the future, or if SS users want to look through our issues themselves in the future. Or maybe a different label should be used?

[iantaylor-NOAA]

I went back to the Pacific Hake model where I had tested forecast option 5 with a 100-year forecast. All input and output files including the full r4ss plots directory are at https://drive.google.com/drive/folders/1rA0V--0b9r6KUx3hDMHgaA-s0xyhxoI7

Changing the control rule reference points to very small values makes the problem go away, but to me that suggests that the control rule settings themselves are mixed up in the issue.

As shown in the first 2 figures below, the stock is stable with Bratio around 13% for the first 40 years of forecast (multiple generations) and then declines before stabilizing again around 7%. It crosses below the 10% reference point in 2083 (which should lead to no F and some rebuilding, but doesn't). Uncertainty in Bratio peaks close to this point, in 2085, where the Bratio is 0.084 with a StdDev of 524.

Also strange is an oscillating pattern in the F values reported in the time-series at the start of the forecast (third figure).

[Rick-Methot-NOAA]

Good work Ian. Your example uses HarvestPolicy option = 1. The SEFSC used 2. Can you please run your script using each of the 4 options?

On Thu, Dec 10, 2020 at 10:06 AM Ian Taylor notifications@github.com wrote:

I went back to the Pacific Hake model where I had tested forecast option 5 with a 100-year forecast. All input and output files including the full r4ss plots directory are at https://drive.google.com/drive/folders/1rA0V--0b9r6KUx3hDMHgaA-s0xyhxoI7

Changing the control rule reference points to very small values makes the problem go away, but to me that suggests that the control rule settings themselves are mixed up in the issue.

As shown in the first 2 figures below, the stock is stable with Bratio around 13% for the first 40 years of forecast (multiple generations) and then declines before stabilizing again around 7%. It crosses below the 10% reference point in 2083 (which should lead to no F and some rebuilding, but doesn't). Uncertainty in Bratio peaks close to this point, in 2085, where the Bratio is 0.084 with a StdDev of 524.

Also strange is an oscillating pattern in the F values reported in the time-series at the start of the forecast (third figure).

[image: image] https://user-images.githubusercontent.com/4992918/101809592-96a3f900-3acc-11eb-808d-2e4660e81c15.png [image: image] https://user-images.githubusercontent.com/4992918/101809611-9c014380-3acc-11eb-9176-6200dd1b6cce.png [image: image] https://user-images.githubusercontent.com/4992918/101811547-0f0bb980-3acf-11eb-928b-fd82d64cf6df.png

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/nmfs-stock-synthesis/stock-synthesis/issues/105#issuecomment-742695458, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABPV4IAMILMF27HVWMCMDC3SUEE2RANCNFSM4USFMO3Q .

[Rick-Methot-NOAA]

The control rule code uses a join function to transition from the slope to the constant F. This is a prime candidate for the cause of the unstable behavior. This implies that the problem is not related to use of Forecast option 5 and should occur with other options also. I will write some temporary code to essentially unit test the control rule calculations to see how smooth they are in the vicinity of the transition.

[iantaylor-NOAA]

Policy option 2 produced linear point estimates of Bratio but steadily increasing uncertainty throughout the forecast as shown in the screenshot below. Options 3 and 4 produced very similar results to 1 and 2, respectively (based on quick visual comparison).

A steady increase in forecast uncertainty (reflected in Bratio, SSB, and Recruitment) seems realistic as a constant F applied to an uncertain stock SHOULD lead to divergent outcomes depending on whether it is sustainable or not. However, the amount of uncertainty is way to large considering that the stock can't go below 0 and the stock-recruit relationship should keep it from going too high.

 Label    Value  StdDev 
 SSB_2119 476990 3.07536e+08

[iantaylor-NOAA]

I forgot to note that the hake models for all 4 policy options are now on Google Drive in separate folders within the one I linked above: https://drive.google.com/drive/folders/1rA0V--0b9r6KUx3hDMHgaA-s0xyhxoI7

[Rick-Methot-NOAA]

I looked more deeply at the hake example and found that the pattern was due to the choice of a very high forecast F (5.0) interacting with a F constraint applied in the control file (1.5).

When I ran with forecast F at 0.4, I get the expected result: That is: F in 2020 and 2021 is right at the input value (0.4) and it has no variance because it is fixed (note that this cause the Hessian to fail to invert). Then in 2022, B declines below 0.4 so F also declines due to the control rule and it now shows noticeable variance. All changes after that are smooth with no anomalies.

ALSO, I note that this model has time-varying selectivity parameters and that the forecast selectivity is set to average over the last 5 years of the time series. This means that selectivity in the forecast is much less variable than selectivity in the time series. An alternative configuration would have the selectivity parameters continue to have random devs into the forecast period and to change the forecast selectivity to use annual calculations rather than an average of recent years. I tried this alternative and, as expected, it had a large impact on the variance patterns in the forecast. Please bring this to the attention of the hake team.

[iantaylor-NOAA]

Thanks for looking at this Rick. I should have thought to try a more reasonable F. However, isn't there still an open question about why the HCR didn't set F to 0 when the stock fell below the minimum stock size threshold?

[iantaylor-NOAA]

@kellijohnson-NOAA can you share Rick's comment above about including time-varying selectivity devs into the forecast. I presume that similar approaches could be used for both this status-quo setup and the semi-parametric alternative.

[Rick-Methot-NOAA]

I re-ran the example after changing maxpossF to 5.0 to match the intended max F in the forecast file. With this run, I get biomass stabilizing at 0.13 and the F stabilizing at about 1.78.

If I increase the bottom of the ramp to 0.14 (from 0.1), then the resultant ABC buffer is very small (it does not go to 0.0, the code forces a small value of 0.0001) and the F and catch are also small. I also inserted some test code to verify that the relationship between ABC buffer and SSB is curvilinear.

I also noticed that the run with the bottom of ramp at 0.14 produced an oscillatory behavior in which F went from tiny one year (because of the ramp) to near max (5.0) the next year. Not surprising given the difference between maturation curve and the selection curve and the high F in this toy example.

[Rick-Methot-NOAA]

Going back to the original hake example and the discrepancies that Ian noted. I now see a little unexpected drift happening and will investigate.

[iantaylor-NOAA]

Thanks for all the time on this Rick.

I hadn't really thought through how oscillations are a logical consequence of the impact of a really high Fs where all selected stock can get wiped out in a single year. In a high-F situation, I think the recommendation has been to use the Fs as parameters options anyway.

If don't find any smoking gun with your ongoing investigations. it seems fair to just close this issue and focus on the proposed "no-ramp" HCR option described in #83. Let me know if I can help on either front.

[Rick-Methot-NOAA]

I have revised the SS code to apply the max_harvest_rate limit at the beginning of the forecast routine so it is applied to all aspects of forecast. This smooths out the hake forecast and it reaches an equilibrium earlier. A warning has been added if the max_harvest_rate limit causes the forecast_F to be limited. Alas, none of this changes the strange variance in 2026 that occurs in the SEFSC example. I will close this issue.

[kellijohnson-NOAA]

Thank you both for looking into this. I have been following (silently) and I will make sure that the JTC is notified regarding hake. Food for thought here is that we only do three year projections, but I did see some of the discussion above was about variance in short term forecasts.

[Rick-Methot-NOAA]

Glad you are focusing on short-term projections. Long projections with use of level 3 forecast loop is not advised. I should add a recommendation/warning in this regard.

[k-doering-NOAA]

See comment about a new error msg: https://github.com/nmfs-stock-synthesis/stock-synthesis/commit/dfbd736f96077178005965b80e5408023fcf2b07#r45254687