Open John-Wiedenmann opened 2 years ago
cmlegault
commented
2 years ago Thanks John. I haven't looked at the results yet, but can answer your question. When time was not a factor, then the results are summarized over the short and long term results for all variables except interannual variability in catch, which was computed across the entire feedback period.
John-Wiedenmann
commented
2 years ago I'm still a little confused when you say that when time is NOT a factor there are values for short and long. For example in the table "table.heatmap.ibm.retro.fhist_median.csv" a single value is reported for SSB, C, and F ratios - and the results are separated by IBM, catch or M, and fishing history. So the values reported here were calculated over the entire feedback period, correct? That is how I interpreted it, anyway. For the ones with time as a factor you separated them out by short vs long and other factors (table.heatmap.ibm.retro.time_median.csv shows results by time AND retro source), and those are the ones with the NAs for IAV for the reason you described. Is that accurate?
cmlegault
commented
2 years ago Not quite. The values that were collected were the short and long time period SSB/SSBmsy, F/Fmsy, Catch/MSY, probability overfished, and probability overfishing. When time was not a factor, the results for the short and long term were included in the medians (or means for the probabilities) for these five variables. Only the interannual catch variability was calculated over the entire feedback period.
John-Wiedenmann
commented
2 years ago Ah, thanks. I get it now.
On Sep 28, 2021, at 12:55 PM, Chris Legault @.***> wrote:
Not quite. The values that were collected were the short and long time period SSB/SSBmsy, F/Fmsy, Catch/MSY, probability overfished, and probability overfishing. When time was not a factor, the results for the short and long term were included in the medians (or means for the probabilities) for these five variables. Only the interannual catch variability was calculated over the entire feedback period.
— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/cmlegault/IBMWG/issues/58#issuecomment-929406782, or unsubscribe https://github.com/notifications/unsubscribe-auth/ACXZRW3G7NAFDDA2WAAODUDUEHXQPANCNFSM5E5ZJMPQ. Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.
liz-brooks
commented
2 years ago i uploaded a ppt explaining some interpretation of the full distribution of SSB/SSBmsy, F/Fmsy, and Catch/MSY results for all methods by scenario. i think it gives a more complete picture of risk probabilities relative to targets/thresholds. the ppt is in Manuscript/some_results.pptx and refers to plots in Manuscript/tables_figs/*.ecdf.cairo.png (plots are also in the ppt). to me, it now looks like the SCAA performs worse than what i think are two top contenders (DLM and PBS, or whatever we're calling plan b smooth now) for several scenarios with missing catch as the source of the retro. take a look and let me know if you agree with my conclusions (plot specific conclusions are in the "Notes" panel below the ppt slides). if so, i'm happy to add this text to the manuscript. i tried to provide a guide for how to interpret the plots, but if you still find it non-intuitive let me know. the plots summarize a lot of info, so if they are too busy they could be revised (simplified) or included as supplementary info or none of the above.
cheers liz
On Tue, Sep 28, 2021 at 12:58 PM John-Wiedenmann @.***> wrote:
Ah, thanks. I get it now.
On Sep 28, 2021, at 12:55 PM, Chris Legault @.***> wrote:
Not quite. The values that were collected were the short and long time period SSB/SSBmsy, F/Fmsy, Catch/MSY, probability overfished, and probability overfishing. When time was not a factor, the results for the short and long term were included in the medians (or means for the probabilities) for these five variables. Only the interannual catch variability was calculated over the entire feedback period.
— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub < https://github.com/cmlegault/IBMWG/issues/58#issuecomment-929406782>, or unsubscribe < https://github.com/notifications/unsubscribe-auth/ACXZRW3G7NAFDDA2WAAODUDUEHXQPANCNFSM5E5ZJMPQ . Triage notifications on the go with GitHub Mobile for iOS < https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android < https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>.
— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/cmlegault/IBMWG/issues/58#issuecomment-929408829, or unsubscribe https://github.com/notifications/unsubscribe-auth/ACFL5LLGFVIWBGWLMCR6WA3UEHX3FANCNFSM5E5ZJMPQ . Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.
--
Liz Brooks, PhD
Operations Research Analyst
Population Dynamics Branch
NOAA/NMFS
Northeast Fisheries Science Center
166 Water Street phone: 508.495.2238
Woods Hole, MA 02543 fax: 508.495.2393
apologies if this is a repeat... not sure i hit reply all:
i uploaded a ppt explaining some interpretation of the full distribution of SSB/SSBmsy, F/Fmsy, and Catch/MSY results for all methods by scenario. i think it gives a more complete picture of risk probabilities relative to targets/thresholds. the ppt is in Manuscript/some_results.pptx and refers to plots in Manuscript/tables_figs/*.ecdf.cairo.png (plots are also in the ppt). to me, it now looks like the SCAA performs worse than what i think are two top contenders (DLM and PBS, or whatever we're calling plan b smooth now) for several scenarios with missing catch as the source of the retro. take a look and let me know if you agree with my conclusions (plot specific conclusions are in the "Notes" panel below the ppt slides). if so, i'm happy to add this text to the manuscript. i tried to provide a guide for how to interpret the plots, but if you still find it non-intuitive let me know. the plots summarize a lot of info, so if they are too busy they could be revised (simplified) or included as supplementary info or none of the above.
cheers liz
On Tue, Sep 28, 2021 at 12:58 PM John-Wiedenmann @.***> wrote:
Ah, thanks. I get it now.
On Sep 28, 2021, at 12:55 PM, Chris Legault @.***> wrote:
Not quite. The values that were collected were the short and long time period SSB/SSBmsy, F/Fmsy, Catch/MSY, probability overfished, and probability overfishing. When time was not a factor, the results for the short and long term were included in the medians (or means for the probabilities) for these five variables. Only the interannual catch variability was calculated over the entire feedback period.
— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub < https://github.com/cmlegault/IBMWG/issues/58#issuecomment-929406782>, or unsubscribe < https://github.com/notifications/unsubscribe-auth/ACXZRW3G7NAFDDA2WAAODUDUEHXQPANCNFSM5E5ZJMPQ . Triage notifications on the go with GitHub Mobile for iOS < https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android < https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>.
— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/cmlegault/IBMWG/issues/58#issuecomment-929408829, or unsubscribe https://github.com/notifications/unsubscribe-auth/ACFL5LLGFVIWBGWLMCR6WA3UEHX3FANCNFSM5E5ZJMPQ . Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.
--
Liz Brooks, PhD
Operations Research Analyst
Population Dynamics Branch
NOAA/NMFS
Northeast Fisheries Science Center
166 Water Street phone: 508.495.2238
Woods Hole, MA 02543 fax: 508.495.2393
Thanks Liz, these are helpful. And yes, SCAA in aggregate looks great, but when you break it out by scenario it does look worse than some other options. It's really conservative with the M retro source and not with catch. It performed much worse when the F returns to Fmsy in the base period which I found interesting (see Figure 3 in the manuscript). Based on the separated results I would argue that Ismooth (formerly Plan B), DLM, ES-Frecent, and the catch curve methods are better overall, though the CC methods are pretty conservative at times.
liz-brooks
commented
2 years ago ok, thanks for the response. do you want me to contribute text along the lines of what i summarized in those slides? and if so, what are your thoughts on where/whether the ecdf figures below (i'm leaning towards supplemental, since they are kinda big and there are 6 of them). i could add some words tomorrow, now that it looks like we'll be open for business, if that doesn't interfere with anyone elses planned word-crafting.
On Thu, Sep 30, 2021 at 1:00 PM John-Wiedenmann @.***> wrote:
Thanks Liz, these are helpful. And yes, SCAA in aggregate looks great, but when you break it out by scenario it does look worse than some other options. It's really conservative with the M retro source and not with catch. It performed much worse when the F returns to Fmsy in the base period which I found interesting (see Figure 3 in the manuscript). Based on the separated results I would argue that Ismooth (formerly Plan B), DLM, ES-Frecent, and the catch curve methods are better overall, though the CC methods are pretty conservative at times.
— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/cmlegault/IBMWG/issues/58#issuecomment-931500775, or unsubscribe https://github.com/notifications/unsubscribe-auth/ACFL5LJAGUYGLMPXW44OMDLUESJRTANCNFSM5E5ZJMPQ . Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.
--
Liz Brooks, PhD
Operations Research Analyst
Population Dynamics Branch
NOAA/NMFS
Northeast Fisheries Science Center
166 Water Street phone: 508.495.2238
Woods Hole, MA 02543 fax: 508.495.2393
The summaries John produced are interesting, but still hide quite a bit of detail. For example, the figure below shows how one point in the F/Fmsy figure when plotted by retro type and Fhist is derived. The 8 scenarios correspond to time period (Long or Short), number of selectivity blocks (1 or 2), and catch advice multiplier (Applied or Reduced). The violin plots show the distributions of the 1,000 realizations. The dots show the median value for that scenario. The dashed red line shows the median across all 8 scenarios (this is the value that is plotted in John's figure). Note the red dashed line does not do a great job describing these simulation results due to the large difference between the short and long time periods. The short time periods are clearly suffering from an F=2 cap, which is leading to the lines along the x-axis in Liz's ecdf plots. Just want to make sure we all understand what is happening in these summary plots. (If the plot doesn't show up in GitHub issues, I'll send it via email)
cmlegault
commented
2 years ago I agree SCAA does not look as clear cut of a winner as it did based on the smaller number of scenarios originally examined. However, I would argue that in the long term its performance is still as good if not better than all the IBMs examined, based on the S/Smsy vs C/MSY plot anyway. The CCs, DLM, and Ismooth all have some large biomasses with very low catch in contrast to SCAA. I think SCAA really struggles in the short term, where F=2 causes the population to essentially crash, but it is able to rebuild the stock and still produce reasonable catches for the fishery in the long term.
John-Wiedenmann
commented
2 years ago Very interesting Chris. Essentially the median (red line) is the low point in the middle of a bimodal distribution of the short and long term. I find it interesting how bad it does in the short term though, but in the long term it lines up more with the missing catch with F = 2.5 x Fmsy. In other words, when the population is really low the retro-adjusted SCAA is better than when the biomass is higher (when catch is misreported).
I’m not sure how others feel about how we want to report things in terms of splitting things out to summarize results, so feedback from the group would be great. Here, even though the median of the short and long-term isn’t really representative of either, it still characterizes that it’s not great under this scenario. The fact that it crashes the population but then allows for a rebound is really interesting, but the crashing part is concerning. I think that we definitely want to emphasize the SCAA, both good and bad, and that we’ll want to report things more in depth for this method than others.
On Sep 30, 2021, at 4:50 PM, Chris Legault @.***> wrote:
I agree SCAA does not look as clear cut of a winner as it did based on the smaller number of scenarios originally examined. However, I would argue that in the long term its performance is still as good if not better than all the IBMs examined, based on the S/Smsy vs C/MSY plot anyway. The CCs, DLM, and Ismooth all have some large biomasses with very low catch in contrast to SCAA. I think SCAA really struggles in the short term, where F=2 causes the population to essentially crash, but it is able to rebuild the stock and still produce reasonable catches for the fishery in the long term.
— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/cmlegault/IBMWG/issues/58#issuecomment-931655864, or unsubscribe https://github.com/notifications/unsubscribe-auth/ACXZRWZXRDSYV73XMO6MY53UETESDANCNFSM5E5ZJMPQ. Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.
John-Wiedenmann
commented
2 years ago All, just a reminder if you could go through the Results section and Figures to edit / comment.
JDeroba
commented
2 years ago So I read the results section and fixed some typos. I think it is well written and strikes a good balance between too much and too little, clearly highlighting the salient points. I did not pour over the plots and tables to confirm that I agree with all the conclusions, but there seems to be some good minds on that task. I'll try to make time for that soon. I agree that reporting more details of the SCAA methods seems warranted. In particular, the point JW mentioned above in this thread, "It performed much worse when the F returns to Fmsy in the base period", is interesting and I think needs to be more clearly highlighted (and eventually explained) somewhere. In terms of splitting results, I'd vote to split them when it mattered. For example, even if the SCAA method performed poorly in the short and long-term with catch as the retro source, and the median captures that, the fact that the effect was way worse in the short-term is still a result worth reporting. Nice work @John-Wiedenmann
I took a stab at writing the Results and adding some Figures, but there is more to be done. Many thanks to Chris for pushing the output tables for me to use as a starting point. I created a folder MS_tables_figs that includes the table summaries that have all PMs combined, as well as some Figures I created using these summary tables (Figure 1, 3 and 4) in the manuscript. Figures 2 and 5 are currently just pulled straight from the report and will need some attention if we intend to keep them.
So, please have a look at the Figures and let me know your thoughts (e.g., too busy), other things we want to convey within a figure or with new figures. Also think about the organization of writing and style within the Results, and how it can be improved. For me this was really the first time I looked at things really trying to tease apart some of the interactions with the different scenarios, and learned some new things. That said, I may have misinterpreted things or missed some other big picture things we want to address.
I also have a question for Chris - some of the output tables were separated by time (short- or long-term), but other were not. In the "not" cases, were the values calculated for the entire time period or something different?