TIP Full Tag Up 9/7/2023

[KatGodwin-NOAA]

Agenda for September 7

Progress report:

• Data Flagging project
• Historical Data project

Progress

Last full team meeting was Aug. 24; Next one is September 21?

Data Flagging project

• [x] Continued work with sub-milestone of "Identify trends in reporting consistency across variables"
  • Created ggplots of variables with inconsistent reporting across PR and VI separately (reporting consistency between 1-99%)
  • Document notes in Quarto
• [x] Continued progress on sub-milestone "Explore data with figures"
  • Began looking at data consistency over area and time by species

Historical Data project

• [x] Moved to step 2 of historical/Oracle dataset comparison

1. Look at how many unique interviews there for a given date
   1. Interview date vs historical vs oracle # of interviews
   2. Correct dates to historical dates
2. Areazip correction
   1. Look for sample_county_code table codes that have muni/county names that correspond to codes
   2. Look in oracle tables and tip manuals: Once fixed, update reporting_area_zip
3. Match gear codes
4. Convert old species codes to itis
5. How many length records for each species
6. Possibly find # of matching and mismatching lengths

Next Steps

• Continued work on data consistency visualization across area and time
• Comparison of historical areazips and gear codes to current areazips and gear codes within data
• Follow up from last meeting: setting up a confidentiality meeting to discuss moving some code to github

Sub Milestones

Milestone 1 - Send funding to Contracting Agency and Cooperative Institution (DONE)

Milestone 2 - Develop data flagging system to be programed - 9/1/2023

July

• Start exploring data with figures (in process)
  • Flagging needs to include data visualization work (oddities are often only noted when the data are plotted)
• Reach out to projects for documents/updates (Completed)
  • Better understand complementary projects
• Start annotated list of resources (in process)
  • Revisit specific flags from existing work; identify additional relevant work
• Schedule status quo meetings for island specific (?) metadata review
• Identify trends in reporting consistency across variables (table showing what has been reported and with what consistency?) (in process)
• Revisit specific flags from existing work (S57, S46) (completed)
• Revisit sub-milestones

August

• Document new flags (in process)
• Finalize reference library of past efforts (in process)
• Revisit sub-milestones

Milestone 3 - Develop data visualization to be programed - 10/6/2023

• Leverage documentation compiled for milestone 2
• Inform Milestone 4 - Develop data filtering and binning processes to be programed - 11/3/2023

[KatGodwin-NOAA]

Follow up with @mollystevens-noaa on confidentiality process. Possibly look into confidentiality automation?

[mollystevens-noaa]

I've pasted below our notes from the NMFS Posit Connect Forum. A lot of the conversations regarding confidential data and Posit Connect directly apply to GitHub Enterprise (IMO). Lmk if you have any questions on what's pasted below, and I'll keep yall posted as these conversations continue!

Posit Connect notes from 8/16/23: Confidential fisheries data handling

• Lots of people have workflows based on confidential data
• Right now, our advice is even if it is private, don’t put any confidential data up on Posit connect
• Questions about GitHub enterprise cloud and confidential data - currently not allowed
• New rule around confidential data handling associ. With Magnuson that will come out in the next few months
• Maybe it would be nice by the end of the year to scope out the problem and vision about what we want to do with confidential data, not what it is.
• Currently, implementation of “rules of thumb” differ by region. When models require confidential data at PIFSC, just stay away from Posit and GitHub
• NDA signing process not shared across centers

Posit Connect notes from 6/21/23: Confidential data/database connections - what is and isn’t allowed on posit connect?

• This keeps coming up from users, so perhaps we should add guidance to our documentation, if we can. Asked Scott, who needs to ask someone else. Not given clear guidance on Jira, either.
• Up to WOC about this issue
• How often do science centers offer training on how to use confidential data?. GC doesn’t even always have a clear answer
  • NE needs to use confidential fisheries data a lot - send out a powerpoint
  • PI offered once since 2020
  • GARFO uses NDAs, as does SE. But other science centers don’t use this
  • Rule of 3 common (different interpretations, however)
• Current guidance: NMFS_Guidelines_Rauch_Mar_2008_Cyr_Jul_2009.pdf
• Would fedramp certification help us allow confidential data
• Could we connect to databases with non-confidential data? Or is the connection a security issue?
• Firewalls may have also been an issue for working with databases
• WOC is a different system than the databases in the individual FMCs