cristinamullin
commented
10 months ago > TADA_BigDataRetrieval(
+ startDate = "1993-01-01",
+ endDate = "1993-12-31",
+ # huc = "null",
+ # siteid = "null",
+ # siteType = "null",
+ # characteristicName = "null",
+ # characteristicType = "null",
+ sampleMedia = c("Water", "water"),
+ # statecode = "null",
+ maxrecs = 250000,
+ applyautoclean = FALSE
+ )
[1] "Building site summary table for chunking result downloads..."
Request failed [504]. Retrying in 1.5 seconds...
Request failed [504]. Retrying in 1.5 seconds...
For: https://www.waterqualitydata.us/data/summary/monitoringLocation/search?zip=yes&dataProfile=periodOfRecord&mimeType=csv
Gateway Timeout (HTTP 504).Error in attr(retval, "queryTime") <- Sys.time() :
attempt to set an attribute on NULL
> test = TADA_BigDataRetrieval(
+ startDate = "1993-01-01", # consider downoading one year at a time
+ endDate = "2000-12-31",
+ # huc = "null",
+ # siteid = "null",
+ # siteType = "null",
+ # characteristicName = "null",
+ # characteristicType = "null",
+ sampleMedia = c("Water", "water"),
+ # statecode = "null",
+ maxrecs = 250000,
+ applyautoclean = FALSE
+ )
[1] "Building site summary table for chunking result downloads..."
Request failed [504]. Retrying in 1.8 seconds...
Request failed [504]. Retrying in 1.9 seconds...
For: https://www.waterqualitydata.us/data/summary/monitoringLocation/search?zip=yes&dataProfile=periodOfRecord&mimeType=csv
Gateway Timeout (HTTP 504).Error in attr(retval, "queryTime") <- Sys.time() :
attempt to set an attribute on NULL
> test = TADA_BigDataRetrieval(
+ startDate = "1993-01-01", # consider downoading one year at a time
+ endDate = "2010-12-31",
+ # huc = "null",
+ # siteid = "null",
+ # siteType = "null",
+ # characteristicName = "null",
+ # characteristicType = "null",
+ sampleMedia = c("Water", "water"),
+ # statecode = "null",
+ maxrecs = 250000,
+ applyautoclean = FALSE
+ )
[1] "Building site summary table for chunking result downloads..."
Request failed [504]. Retrying in 1 seconds...
Request failed [504]. Retrying in 1.1 seconds...
For: https://www.waterqualitydata.us/data/summary/monitoringLocation/search?zip=yes&dataProfile=periodOfRecord&mimeType=csv
Gateway Timeout (HTTP 504).Error in attr(retval, "queryTime") <- Sys.time() :
attempt to set an attribute on NULL
Describe the bug Yu-Chen Wang
I tried the TADA_BigDataRetrieval function to download the Aluminum data in Montana with the following command.
test1 <- TADA_BigDataRetrieval(statecode = "MT", characteristicName = "Aluminum")
This gives me the following error: a long URL with lots of site ID names and the HTTP 504 error.
In the source code, I can see that TADA_BigDataRetrieval tries to calculate the record numbers first and download sites with records fewer than 250,000 as chunks. However, in this case, there are too many sites with fewer records. It partially confirms my theory that the Water Quality Portal cannot take too many site ID, characteristicName, HUC, or any other inputs even the web service API guide states that multiple inputs are allowed.
It is thus challenging to design a workflow to download the data with lots of records. A single function call may be able to download the data, but the Water Quality Portal may time out or take a long time to run, and the users cannot know when it will finish. Using readWQPsummary to download the site ID and available parameters allows the users to download the data as chunks and monitor the progress. However, setting the chuck size too small would allow the process to continue but make it too slow.
Code to reproduce the behavior:
Expected behavior Query should return data.
Here are some other examples to try to optimize: • Dearborn HUC (10030102) for SANDS all types 2000-present. • Silver and hardness data for all of WY: 2018-01-01 to present. • Wyoming data for the last 5 years • Yellowstone watershed HUC for SANDS all types 2000-present
Suggestion Let's try adding a function input with a default number of sites (max_sites = 40). We can adjust the max number of sites after testing approx. how many cause this issue related to the max query length ).
Let's try adding a function input with a default number of characteristics (max_characteristics = 40). We can adjust the max number of sites after testing approx. how many cause this issue related to the max query length).
Need to do more research because the issue is the overall service length, so instead or in addition to the suggestions above, we might need to look across the function input parameters and give a warning if there are over X amount OVERALL.