search

USEPA / EPATADA

This R package can be used to compile and evaluate Water Quality Portal (WQP) data for samples collected from surface water monitoring sites on streams and lakes. It can be used to create applications that support water quality programs and help states, tribes, and other stakeholders efficiently analyze the data.
https://usepa.github.io/EPATADA/
Creative Commons Zero v1.0 Universal
39 stars 18 forks source link

Alternative workflow for big data retrieval #384

Open cristinamullin opened 5 months ago

cristinamullin commented 5 months ago

Is your feature request related to a problem? Please describe.

Big data retrieval is challenging for WQP/TADA users. Consider including an example using a completely different alternative workflow for big data retrieval than what we have in TADA_BigDataRetrieval in DataDiscoveryRetrieval.R. Considering applying this different approach to an EPA use case in a new vignette: https://github.com/USEPA/TADA/issues/362

Describe the solution you'd like

Matt Ross (Colorado State) suggested using targets.

USGS already created a targets vignette: https://waterdata.usgs.gov/blog/wqp-large-pull-targets/ Another USGS vignette: https://waterdata.usgs.gov/blog/large_sample_pull/ USGS target repo: https://github.com/DOI-USGS/ds-pipelines-targets-example-wqp

Describe alternatives you've considered

Still exploring...

Reminders for TADA contributors addressing this issue

New features should include all of the following work:

lkoenig-usgs commented 5 months ago

Hello! 👋 I've periodically followed the progress on the TADA package and am really excited to see it come together. I created the USGS targets workflow with other USGS colleagues - if you have any questions or want to compare notes on our approach to big data pulls, please don't hesitate to reach out!