TIP Full Tag Up 9/22/2023

[KatGodwin-NOAA]

Agenda for September 22

Progress report:

• Data Flagging project
• Historical Data project

Progress

Last full team meeting was Sept. 7; Next one is Oct. 5?

Data Flagging project

• Flagging visualization work used in Caribbean Index and Discards meeting 9/12
• Met with David to discuss flagging procedures that work across our projects 9/12
• Continued work with sub-milestones of "Identify trends in reporting consistency across variables" and "Explore data with figures"
  • Created ggplots of area-time data reporting consistency across PR and VI separately for Yellowtail Snapper focusing on the variables:
    • SAMPLE_WEIGHT_KG
    • STANDARDGEARNAME_1
    • OBS_WEIGHT_KG
    • Note: across all variables, data shows consistent lack of reporting in:
      • St. Thomas between 1990 and 2005
      • St. Croix between 2010 and present
      • St. John all years except 1985 and 2009 to 2011
    • Note: OBS_WEIGHT_KG reporting decreases in Puerto Rico over time since 2000, with only 4 recordings since 2020
  • Document notes in Quarto

Historical Data project

• Completed part 1 of step 2 of historical/Oracle dataset comparison
  • 13 of the 16 muni codes represented align between the historical and Oracle data
  • 1 muni code (Vega Baja) is only represented in the historical dataset
  • 2 muni codes (Ceiba and Guayama) are only represented in the Oracle dataset
• Next is part 2 of step 2: matching the PLACE_ID codes

1. Look at how many unique interviews there for a given date
   1. Interview date vs historical vs oracle # of interviews
   2. Correct dates to historical dates
2. Areazip correction
   1. Look for sample_county_code table codes that have muni/county names that correspond to codes
   2. Look in oracle tables and tip manuals: Once fixed, update reporting_area_zip
3. Match gear codes
4. Convert old species codes to itis
5. How many length records for each species
6. Possibly find # of matching and mismatching lengths

Next Steps

• Continued work on data consistency visualization across area and time
• Comparison of historical PLACE_ID codes and gear codes to current PLACE_ID codes and gear codes within data
• Follow up from last meeting: setting up a confidentiality meeting to discuss moving some code to github

Sub Milestones

Milestone 1 - Send funding to Contracting Agency and Cooperative Institution (DONE)

Milestone 2 - Develop data flagging system to be programed - 9/1/2023

July

• Start exploring data with figures (in process)
  • Flagging needs to include data visualization work (oddities are often only noted when the data are plotted)
• Reach out to projects for documents/updates (Completed)
  • Better understand complementary projects
• Start annotated list of resources (in process)
  • Revisit specific flags from existing work; identify additional relevant work
• Schedule status quo meetings for island specific (?) metadata review
• Identify trends in reporting consistency across variables (table showing what has been reported and with what consistency?) (in process)
• Revisit specific flags from existing work (S57, S46) (completed)
• Revisit sub-milestones

August

• Document new flags (in process)
• Finalize reference library of past efforts (in process)
• Revisit sub-milestones

Milestone 3 - Develop data visualization to be programed - 10/6/2023

September

• Leverage documentation compiled for milestone 2
• Meet with Molly to explore past SEDAR flagging automation
• Incorporated documented flags into data visualization system
• Inform Milestone 4 - Develop data filtering and binning processes to be programed - 11/3/2023