gapindex is not appropriately integrating the pre-start date composite species code from the SPECIES_YEAR table in biomass estimates

[EmilyMarkowitz-NOAA]

Issue Description

The gapidnex R package is not calculating estimates for pre-confidence (or SPECIES_YEAR.START_DATE) species codes as listed in the SPECIES_YEAR table (or future version of this table). This issue applies to every species listed in the SPECIES_YEAR table, but I'll use arrowtooth and Kamchatka flounder as an example below. This echos but is slightly different to https://github.com/afsc-gap-products/gap_products/issues/37.

According to the SPECIES_YEAR table, the 10111 Atheresthes sp. (arrowtooth [10110] and Kamchatka [10112] flounder) composite species code should be used for all arrowtooth and Kamchatka flounder caught before 1991. However, there are only 0s, except for the one time that the 10111 code was earnestly used in 2023 (AKFIN_BIOMASS table: AREA_ID = 99901 AND SPECIES_CODE = 10111). All rows before 1991 should be populated with non-zero estimates because these species were caught (and likely listed as arrowtooth flounder [10110]), but scientists were not confident about their IDs.

!! It's important that both pre- and post-time stanza estimates are calculated so we can accurately calculate total annual biomass for each `AREA_ID. !!

For the record, you'll see in the next two screenshots that arrowtooth (10110) and Kamchatka (10112) flounder estimates are correctly cropped by gapindex to the post-1992 time-series.

Arrowtooth flounder: (AKFIN_BIOMASS table: AREA_ID = 99901 AND SPECIES_CODE = 10110)

Kamchatka flounder: (AKFIN_BIOMASS table: AREA_ID = 99901 AND SPECIES_CODE = 10112)

Proposed Solution

Add calculating the pre-SPECIES_YEAR.START_DATE species composite codes in gapindex.

[zoyafuso-NOAA]

I just want to flesh this out a little bit more after our office hour session.

• Rock soles: 10261 and 10262 do not exist prior to 1996, just 10260. The 10260 is still used after 1996. Zero records for NRS and SRS will continue to be removed prior to 1996 (artifact of gapindex).
• REBS: 30051 and 30052 don’t exist until 2006 (except for a vouchered BSRF in 2005), 30050 complex code is used prior to 2006 and is still used. The current workflow is to remove RERF and BSRF records prior to 2006. The proposed modification is to aggregate RERF and BSRF SPECIES_CODEs to the complex-level SPECIES_CODE from 1982-2006 and to remove filled zeros for RERF and BSRF prior to 2006.
• Dusky/Dark RFs: 30151 and 30152 do not exist until 1996 (with a handful of duskies prior), 30050 complex code is used prior to 1996 and is seldom used after 1996. The current workflow is to remove dusky and dark RFs prior to 1996. The proposed modification is to aggregate them to the complex code from 1982-1996 and to remove filled zeros for dusky RF and dark RF prior to 1996.
• ATF/Kams: Both species codes (10110 and 10112) existed prior to 1992 but we are only confident in those two SPECIES_CODE values from 1992-on. The ATF/Kam complex code (10111) is seldom used in RACEBASE.CATCH but does exist. The current workflow is to remove ATF and Kams prior to 1992. The proposed modification is to aggregate ATF and Kam records to the complex-level SPECIES_CODE 10111 from 1982-1991 and to remove filled zeros for ATF and Kams prior to 1992.
• Skates: SPECIES_CODEs 435, 455, 471, 472, existed prior to 1999 but we are only confident in those SPECIES_CODE values from 1999-on. The current workflow is to remove these SPECIES_CODE values pre-1999. The proposed modification is to aggregate them to the genus-level SPECIES_CODE (405) from 1982-1999 and to remove filled zeros for each of these SPECIES_CODE values prior to 1999.
• Skates: SPECIES_CODE 480 existed prior to 1988 but only in the AIGOA. The AIGOA standard time series starts in the early 1990s so these earlier 480 records from the 1980s will not affect the current workflow.
• Skate egg cases: SPECIES_CODEs 436, 456, 473, 474, 481 existed only after 1999 and will not affect the current workflow.

[zoyafuso-NOAA]

Sorry, one more clarification for discussion: this only for the CPUE and BIOMASS tables, right? We’re not worried about size and age compositions, right?

[EmilyMarkowitz-NOAA]

Oh, good point. My vote is that any complexing decisions we make on this should be consistent across all data products (aka CPUE, biomass/abundance, age comp, and length comp tables). Am I right in thinking that in the case of the age and length comp tables, we would simply not produce estimates for species before the SPECIES_YEAR.START_DATE listed, because we do not provide complexed age/length comps. Is that what we are currently doing/what you are alluding to in your comment?

[zoyafuso-NOAA]

Yes, we currently are not providing size and age compositions for aggregated species_code complexes in GAP_PRODUCTS nor was this the case in the historical workflows, and it sounds like it will be okay for this to continue to be the case going forward.

[zoyafuso-NOAA]

This issue is slowly dissolving me inside because it should be so simple. Another complication is that within the Bathyraja 405 species complex, SPECIES_CODE values 435, 455, 471, and 472 have a start year of 1999 while species_code 480 has a start date of 1988. That creates three temporal stanzas:

• 1982-1987: 405 consists of SPECIES_CODE values (405, 435, 455, 471, 472, and 480)
• 1988-1999: 405 consists of SPECIES_CODE values (405, 435, 455, 471, and 472), 480 is separated from the 405 complex
• 2000-on: 405, 435, 455, 471, 472, and 480 are reported as separate codes

I'm sorry about throwing wrenches here I just want to make sure that this additional work makes sense and is feasible before I go ahead and work on creating the workflow to accommodate this in the production run.

[Ned-Laman-NOAA]

I recommend that we invite the taxonomic brains to advise us here. Maybe there's a way out of this that's less complicated than having several temporal stanzas where the same code means different things in each.

[EmilyMarkowitz-NOAA]

Agreed - thanks for thinking through this issue from all of the different angles, @zoyafuso-NOAA ! It seems way more complicated than we originally anticipated. @SarahFriedman-NOAA and @ThaddaeusBuser-NOAA , do either of you have any insight on a best path forward for this issue?

[SarahFriedman-NOAA]

Thanks for the careful discussion on this topic. Unfortunately, I am struggling to think of another way to handle this issue. I think the proposed solution is probably the best way forward. Though I will say that I don't think the same code means different things in each temporal stanza. Essentially, everything just being grouped at the genus-level (at least for skates) and that seems quite consistent for me. I understand that that category consists of a few different species codes, but I fail to see how that is substantially different than even the modern day usage of 405.