Progress update 2/22/2024

[KatGodwin-NOAA]

Sub Milestones: Data Flagging Project

Milestone 1 - Send funding to Contracting Agency and Cooperative Institution (DONE)

Milestone 2 - Develop data flagging system to be programed - 9/1/2023

July

• Start exploring data with figures (in process)
  • Flagging needs to include data visualization work (oddities are often only noted when the data are plotted)
• Reach out to projects for documents/updates (Completed)
  • Better understand complementary projects
• Start annotated list of resources (in process)
  • Revisit specific flags from existing work; identify additional relevant work
• Schedule status quo meetings for island specific (?) metadata review
• Identify trends in reporting consistency across variables (table showing what has been reported and with what consistency?) (complete)
• Revisit specific flags from existing work (S57, S46) (completed)
• Revisit sub-milestones

August

• Document new flags (in process)
• Finalize reference library of past efforts (in process)
• Revisit sub-milestones

Milestone 3 - Develop data visualization to be programed - 10/6/2023

September

• Leverage documentation compiled for milestone 2
• Meet with Molly to explore past SEDAR flagging automation (completed)
• Explore SEDAR flagging automation with current SEDAR species ( in progress)
• Incorporated documented flags into data visualization system (in progress)
• Inform Milestone 4 - Develop data filtering and binning processes to be programed - 11/3/2023

Milestone 4 - Develop data filtering and binning processes to be programed - 11/3/2023

October

• Move code to GitHub (complete)
• Incorporate flags and filters from visualization into filtering and binning process (in process)
• Practice flags with SEDAR 84 (in progress)

Sub Milestones: Data Flagging Project

1. Look at how many unique interviews there for a given date(complete)
   1. Interview date vs historical vs oracle # of interviews(complete)
   2. Correct dates to historical dates (complete)
2. Areazip correction (in progress)
   1. Look for sample_county_code table codes that have muni/county names that correspond to codes (complete)
   2. Look in oracle tables and tip manuals: Once fixed, update reporting_area_zip/PLACE_ID
3. Match gear codes (complete)
4. Convert old species codes to itis (in progress)
5. How many length records for each species
6. Possibly find # of matching and mismatching lengths

Outline of process completed

TIP data flagging and automation

The following is the order in which exploration took place:

0. preparation
   • learn the dataset
   • identify key resource documents
1. variable reporting consistency across all islands and species
   • how many variables have 100% or 0% reporting consistencies
   • which variables fluctuate in reporting consistencies and why
   • identify relevant variable for length-weight composition
   • visualize reporting consistency across time
2. length-weight relevant variable investigation by species
   • data visualization - visually identify outliers in length and weight variables
   • county_sampled over time
   • sample_weight over time by county_sampled
   • gear utilization over time by county_sampled
   • obs_weight over time by county_sampled
   • length over time by county sampled
   • statistical exploration of length values
   • Molly Stevens's SEDAR size-comp script exploration
   • collaborate with Kyle Dettloff on Generalized Mixed Model Linear analysis of measured lengths relative to gear and time
3. Quarto document to streamline SEDAR size comp analysis with flags included
4. streamlined scripts of all flagging and visualization completed in this project

Historical flagging and comparison

1. build script to compare each data set by year
   • compare interview dates
   • compare areas sampled
   • compare gear used
   • compare species sampled
2. identified a possible bug in ORACLE relating to daylight savings time shifts
3. mapped updated site specific muni codes in PR

[mollystevens-noaa]

We have this dataset from R work, what are the easy development wins? How are we going to store and disseminate the data?

• Developer that Michael Holley is hiring could be a good fit for this
• Alan Lowther says her "to do" list is already very long
• Developer time is spread thin fixing problems across the board
  • We have the funding, how will that money be spent? Another developer is an option
  • Could do short term ; can't have federal staff working on this
• How do we go about formalizing what we want and need?
  • Sandra was participating in meetings and providing live guidance
  • What are our new goals for development given the staff turnover?
  • Could back off visualizing data, and include simple flags and improved data extractions
• What data products are we trying to build?
  • R work can be split into an improved data extraction (developed by the app dev) and R shiny app or package
  • Grander vision is data warehousing ; Matt Grossi put in an IRA proposal to streamline these ; improved storage and access
• Next steps:
  • $75K in CIMAS to improve Caribbean TIP data extraction
  • Shiny app from improved TIP data extraction? - Match variable names to standard Gulf/SA SEDAT to collaborate w Vivian's group?
  • How to initiate a project w app dev? - the process hasn't started yet - can initiate in OMI portal by "Requesting new service, project, or application"

[mollystevens-noaa]

OMI Helpdesk link: https://apps-st.fisheries.noaa.gov/jirasm/servicedesk/customer/portal/3/SEFSCSM-1985

TIPS SEDAT table for Gulf and SA: SEDAT.TIPS_EXTRACTION_MV@SECDW_DBLK