search

DOI-USGS / dataRetrieval

This R package is designed to obtain USGS or EPA water quality sample data, streamflow data, and metadata directly from web services.
https://doi-usgs.github.io/dataRetrieval/
Other
256 stars 85 forks source link

POST in WQP not working? #686

Closed ldecicco-USGS closed 5 months ago

ldecicco-USGS commented 6 months ago

Describe the bug WQP POST for a long list of sites use to work, now doesn't.

To Reproduce Steps to reproduce the behavior:

library(dataRetrieval)
df <- data.table::fread("sites.csv", data.table = FALSE)

x <- readWQPdata(
  siteid = df$sites,
  characteristicName = "Temperature, water"
)
Forbidden (HTTP 403).The following url returned no data:

sites.csv

Expected behavior This should work via POST

ldecicco-USGS commented 6 months ago

Well, generally the POST still seems to work. One interesting thing I've learned while trying to debug this is that you cannot have a single character object > 4096 characters when you make it by just copy/pasting in the Console:

https://community.rstudio.com/t/does-console-impose-an-upper-limit-on-the-length-of-strings/12872/10

However, if you paste a few smaller characters together, you can make a single character that's longer than 4096. I don't think this is the cause of the error above, but it caused me to go down a pretty deep rabbit hole to figure out that out.

ldecicco-USGS commented 5 months ago

So, the POST does work. There were some new limits added on the WQP server which generated a 403 error (I think?) if the query body was too big. They took off those limits and now those same queries tend to timeout (instead of error).