TIP Full Tag Up 10/5/2023

[KatGodwin-NOAA]

Agenda for October 5:

Progress report:

• Data Flagging project
• Historical Data project

Progress

Last full team meeting was Sept. 7; Next one is Oct. 19?

Data Flagging project

• [x] Continued work with sub-milestones of "Identify trends in reporting consistency across variables" and "Explore data with figures"
  • [x] Created ggplots of area-time data reporting consistency across PR and VI separately for Stoplight Parrotfish and Yellowtail Snapper focusing on the variables:
    • COUNTY_SAMPLED
    • SAMPLE_WEIGHT_KG
    • STANDARDGEARNAME_1
    • OBS_WEIGHT_KG
    • LENGTH1_MM
    • Note: Yellowtail Snapper (focus on PR, STT, STJ)-
      • across all variables, data shows consistent lack of reporting in:
        • St. Thomas between 1990 and 2005
        • St. Croix between 2010 and present
        • St. John all years except 1985 and 2009 to 2011
      • OBS_WEIGHT_KG reporting decreases in Puerto Rico over time since 2000, with only 4 recordings since 2020
      • 2 inaccurate LENGTH1_MM records in 2016 and 2022 Puerto Rico, others with questionable accuracy
    • Note: Stoplight Parrotfish (focus on STX) -
      • Sparse SAMPLE_WEIGHT_KG recordings since 2020
      • Gear most frequently used are pots and traps
      • 5 inaccurate OBS_WEIGHT_KG records in 2016 in Puerto Rico
      • Lack of reporting in St. Croix from about 2011 to 2016 across variables
  • [x] Document notes with Quarto
• [x] Met with Molly to review code previously used in SEDARS for flagging and data consistency

Historical Data project

• [x] Established date glitch between Oracle and Historical datasets:
  • Dates misalign from Jan 1 to early April and early November to Dec. 31. The historical records are labeled as one day later than the Oracle records for these sets of dates.
  • Theory: the oracle records have a time value when the historical do not. The time values for the misaligned dates are 23:00:00 while the time value for the aligned dates are 00:00:00. Possibly human error or system glitch caused the time input to be incorrect, thus pushing the dates back by one hour in Oracle to the previous day.
• [x] Aligned Muni codes between datasets
• [x] Began work with Place_ID codes between datasets
  • [x] Met with David and Steve to discuss information sharing between projects to aid in area code alignment
• [x] Met with Sarina to create functions to pull Oracle data by year and state_code
• [x] Began gear_code comparison using tables received from Sarah

Next Steps

• Continued work on data consistency visualization across area and time focusing on utilizing past SEDAR strategies
• Comparison of historical GEAR_CODES current gear codes within data
• Explore Area_codes from other projects and compare to historical areazip_codes
• Follow up from last meeting: setting up a confidentiality meeting to discuss moving some code to github

Sub Milestones: Data Flagging Project

Milestone 1 - Send funding to Contracting Agency and Cooperative Institution (DONE)

Milestone 2 - Develop data flagging system to be programed - 9/1/2023

July

• Start exploring data with figures (in process)
  • Flagging needs to include data visualization work (oddities are often only noted when the data are plotted)
• Reach out to projects for documents/updates (Completed)
  • Better understand complementary projects
• Start annotated list of resources (in process)
  • Revisit specific flags from existing work; identify additional relevant work
• Schedule status quo meetings for island specific (?) metadata review
• Identify trends in reporting consistency across variables (table showing what has been reported and with what consistency?) (in process)
• Revisit specific flags from existing work (S57, S46) (completed)
• Revisit sub-milestones

August

• Document new flags (in process)
• Finalize reference library of past efforts (in process)
• Revisit sub-milestones

Milestone 3 - Develop data visualization to be programed - 10/6/2023

September

• Leverage documentation compiled for milestone 2
• Meet with Molly to explore past SEDAR flagging automation
• Incorporated documented flags into data visualization system
• Inform Milestone 4 - Develop data filtering and binning processes to be programed - 11/3/2023

Sub Milestones: Data Flagging Project

1. Look at how many unique interviews there for a given date
   1. Interview date vs historical vs oracle # of interviews
   2. Correct dates to historical dates
2. Areazip correction
   1. Look for sample_county_code table codes that have muni/county names that correspond to codes
   2. Look in oracle tables and tip manuals: Once fixed, update reporting_area_zip
3. Match gear codes
4. Convert old species codes to itis
5. How many length records for each species
6. Possibly find # of matching and mismatching lengths

[mollystevens-noaa]

Gulf of Mexico commercial TIP data processing that was built off of SEDAR80 work can be found here

[mollystevens-noaa]

Table of approximate max sizes defined by Caribbean TIP data in Stevens et al. 2019 and compiled by Refik