search

pfmc-assessments / nwfscDiag

Tool to run diagnostics for U.S. West Coast groundfish stock assessments using Stock Synthesis
https://pfmc-assessments.github.io/nwfscDiag/
0 stars 2 forks source link

Add option to apply retro to user specified window of years #12

Open brianlangseth-NOAA opened 2 years ago

brianlangseth-NOAA commented 2 years ago

To fully diagnose potential sources of retrospective patterns it would be add functionality to specify a set window of years to do a retrospective analysis on. Similar to this would be to automate the exclusion of entire datasets as well, which could serve both as a sensitivity, but also to explore fixes of possible retrospective patterns.

kellijohnson-NOAA commented 2 years ago

@brianlangseth-NOAA you can do this by using the model_settings argument of retro_wrapper(). https://github.com/nwfsc-assess/nwfscDiag/blob/55f49fb16b540ed2752482dd46774d79791d9d83/R/retro_wrapper.R#L68 I am not sure if you can run it multiple times in the same directory because the naming of the directory is done automatically which would put the results in the same folder. Regarding automating exclusion of entire datasets, I have moved this to a new issue. Keeping issues focused on a single topic helps future users search them and helps keep individual threads to a manageable size.

kellijohnson-NOAA commented 2 years ago

Perhaps @brianlangseth-NOAA you could suggest changes to the documentation, maybe more links to r4ss::SS_doretro() to make it clear what options users might want to change?

brianlangseth-NOAA commented 2 years ago

@kellijohnson-NOAA You are right in that years can be specified within the function, upon which all years after that point are removed. Thank you for reminding me of that functionality.

There remains a lack of capability to automatically remove only a window of years, and not all years after that window, as well as to remove years at the back end of the time series (i.e. a reverse retrospective). Neither of these are a true retrospective run but rather diagnostics for exploring effects of data on our models.

The reverse retrospective can be done by iteratively changing

  1. the #_StartYr in the .dat file, and
  2. updating the first year of main recr_devs and the recdev_early_start in the .ctl file, if needed.

Utilizing the set up from retro_wrapper could make this automated and easy to explore a range of starting years in one fell swoop.

The window approach would be hard because r4ss::Do_retro() changes the starter file retro year, which is coded within SS3 itself https://github.com/r4ss/r4ss/blob/82abd14df22bc6ad39fb9b59e371d2e16944a805/R/SS_doRetro.R#L140 I dont see a way to adjust with the current retrospective setting. A hacky way would be to add a negative value for any data in the window of years, and then advance that, which may be tricky to automate. Certainly not as easy as adjusting just a few lines.

iantaylor-NOAA commented 2 years ago

I think that if we choose to get better at modifying our data files through r4ss::SS_readdat() and r4ss::SS_writedat() where removing data for any subset of years/fleets/data-types could be pretty easy to script, e.g. https://github.com/nwfsc-assess/lingcod/blob/main/models/remove_ages_south.R#L40-L42.

If excluding a window of years is a common task, we could write a function to loop across all data types to exclude a window of years to make it all the easier within the data file workflow.