USEPA / EPATADA

This R package can be used to compile and evaluate Water Quality Portal (WQP) data for samples collected from surface water monitoring sites on streams and lakes. It can be used to create applications that support water quality programs and help states, tribes, and other stakeholders efficiently analyze the data.
https://usepa.github.io/EPATADA/
Creative Commons Zero v1.0 Universal
40 stars 18 forks source link

Max web service URL #363

Open cristinamullin opened 1 year ago

cristinamullin commented 1 year ago

Describe the bug

Users are interested in querying the WQP by a long list of parameters (sites or characteristics), but their queries will not work if they input too many parameters. There appears to be a max URL length for web service calls. Ideally, we will be able to add some logic to the TADA data retrieval functions to handle very long parameter lists (chunk queries) automatically.

Currently, the max within any given filter appears to be 30-45, which would align/make sense with the 2,083 character max for Chrome.

I think dataRetrieval uses Curl for the queries. Regardless, it's probably a good idea to restrict URL lengths. Not sure if there is an option to specify to use POST for the queries using dataRetrieval? https://www.waterqualitydata.us/webservices_documentation/#queries-using-post

Reminders for TADA contributors addressing this issue

New features should include all of the following work:

ldecicco-USGS commented 10 months ago

dataRetrieval uses GET unless the URL gets too long, then it switches to a POST. However, just recently it seems like the limits for how big a POST can be has been restricted compared to what it use to be. I'm still looking into if it is in fact a new restriction or if I just hadn't noticed it until recently. Unfortunately at the moment the POSTs don't seem to give the flexibility over the GET that I was assuming we'd see.