search

DOI-USGS / dataRetrieval

This R package is designed to obtain USGS or EPA water quality sample data, streamflow data, and metadata directly from web services.
https://doi-usgs.github.io/dataRetrieval/
Other
256 stars 85 forks source link

Problem with the function readNWISdata #676

Closed fhmejia closed 9 months ago

fhmejia commented 9 months ago

Problem with the function readNWISdata I keep getting the following error when I run the function readNWISdata: Request failed [504]. Retrying in 1.8 seconds...

Code used:

dataTemp_all1<-readNWISdata(sites=c("14144800","14144900","14145500","14147500","14148000","14149011","14150000","14150290","14150800","14151000","14152000","14152500","14153500","14154500","14155500","14158100","14166000","14169000","14170000","14171000","14174000","441152123085600","441155123091900","441351123090900","441355123094600","441359123093800","441428123095600","441446123103500","441447123103000","441515123104000","441540123101900","441558123103500","441613123102600","444517123084900","14158500","14158740","14158790","14158798","14158850","14159200","14159500","14161100","14161500","14162050","14162200","14162450","14162500","14163000","14163150","14163900"), parameterCd="00010", service="uv", startDate="2011-01-01T00:00",endDate="2023-09-30T12:00", tz="America/Los_Angeles")

However, I do not get this error when I only run three sites: dataTemp<-readNWISdata(sites=c("14144800","14144900","14145500"), parameterCd="00010", service="uv", startDate="2011-01-01T00:00",endDate="2023-09-30T12:00", tz="America/Los_Angeles")

What is the maximum number of sites one can include with this function?

Thank you!

Francine

ldecicco-USGS commented 9 months ago

There's not a built in site limit. There's an upper limit to how many characters a URL can be, if that happens I think we automatically switch to a "POST" call instead of a "GET" call (small details in how to talk to the web services).

The 504 return means a timeout. So, while the request is OK, it's taking too long to get the data. Since you are asking for 12 years of "uv" data...I'm not that surprised. That's A LOT of data (~12 million rows of data per site).

For this uv data, I'd split it up, maybe even site-by-site. You could have a look at the 2 blog posts on large data queries. They are WQP focused, but the same loop or "target" approach could be used for this query.

Scripting: https://doi-usgs.github.io/dataRetrieval/articles/wqp_large_pull_script.html "target" pipeline: https://doi-usgs.github.io/dataRetrieval/articles/wqp_large_pull_targets.html

fhmejia commented 9 months ago

Thank you! yes, I was playing with the loops from one of your examples. It seems to be working, no time out yet!

You guys are so helpful. Thank you!!!!!!