nwfsc-fram / warehouse

FRAM Data Warehouse - Public
Other
5 stars 1 forks source link

National survey data - one stop shop? #96

Open kellijohnson-NOAA opened 3 years ago

kellijohnson-NOAA commented 3 years ago

Data team - Thank you for creating the data warehouse for FRAM. We use it monthly if not more within NWFSC/FRAM/PEP to pull survey data. I am curious if you know of any initiatives towards standardizing the availability of data across the centers? For example, it would be great if one could download all Alaska, Washington, Oregon, and California data from a single place. NOAA has taken the initiative to work with PacFIN, AKFIN, and others for commercial and recreational data creating an online repository that ensures data remain confidential. I recently talked with @EmilyMarkowitz-NOAA from the Alaska Fisheries Science Center, who is trying to get data from the Bering Sea Surveys online in a non-csv form, she thought there was potential to add survey data to the link I gave for commercial data. @Curt-Whitmire-NOAA do you know of any initiatives to create a NOAA-wide database to house information from white ships? Anyone know where data from the east coast is stored?

cc @chantelwetzel-noaa @MargaretSiple-NOAA

EmilyMarkowitz-NOAA commented 3 years ago

Thanks for starting up this convo, @kellijohnson-NOAA! I am stoked to see the end of our csv-linked page and the start of API-driven and public-friendly interactive data portals. I wrote up this proposal with our AFSC IT division director to OST IT that might be helpful for others thinking of doing the same thing. I used to work in OST so I have an in with the folks running the FOSS landing page. I am happy to introduce others to the FOSS team and start to build a unified data sharing space. We are hoping we may be able to get our data on FOSS before the end of the year!

Curt-Whitmire-NOAA commented 3 years ago

@kellijohnson-NOAA, for NOAA white ships, I'm only aware that SCS data are sent to NCEI. Additional data collected specific to each survey is typically the responsibility of the chief scientist. As an example, I understand that @aliciabillings-noaa might be chatting with NCEI in the coming months about submitting acoustics data to NCEI.

After a quick search of InPort, it appears each survey group handles data delivery differently (shocking!). Perhaps it would be useful to determine common data attributes for each survey, and develop APIs where such data attributes could be served via a NMFS enterprise portal (like FOSS). NOAA's Deep-Sea Coral Research and Technology Program adopted a similar workflow where DSCRTP-funded cruises were required to submit data (using standard attributes) to the DSC national database. I think it's been pretty successful, but funding was the big hook in that case.