search

DOI-USGS / dataRetrieval

This R package is designed to obtain USGS or EPA water quality sample data, streamflow data, and metadata directly from web services.
https://doi-usgs.github.io/dataRetrieval/
Other
263 stars 85 forks source link

qw bug with old dates? #485

Closed kknierim-usgs closed 5 years ago

kknierim-usgs commented 5 years ago

Having problems with readNWISqw() retrieving older samples with an expanded format. Have found this issue for samples generally pre-1940

Example -- works when expanded = FALSE, note site includes only year (no month day) dfx <- readNWISqw('295227091004501', '00940', "1930-01-01", "1939-12-31", expanded=FALSE, reshape=TRUE)

Example -- does not work with expanded = TRUE dfx <- readNWISqw('295227091004501', '00940', "1930-01-01", "1939-12-31", expanded=TRUE, reshape=TRUE)

Error = Error in charToDate(x) : character string is not in a standard unambiguous format

We found was an issue doing a large data retrieval where we are looping on decades and trying to pull all data, with expanded = TRUE

srwacast commented 5 years ago

Has this been looked into?

ldecicco-USGS commented 5 years ago

I think if there aren't valid dates, the reshape code code will not work (I think?). I can help write up a script to call the raw data, and reshape by hand (ie...write a little script). The site above doesn't seem to have any data other than the 1 observation in the 1930's. Do you have sites that had data in the old format, and also have more modern data?

Here's the link to the raw data:

https://nwis.waterdata.usgs.gov/nwis/qwdata?search_site_no=295227091004501&search_site_no_match_type=exact&multiple_parameter_cds=00940&param_cd_operator=AND&list_of_search_criteria=search_site_no,multiple_parameter_cds&group_key=NONE&sitefile_output_format=html_table&column_name=agency_cd&column_name=site_no&column_name=station_nm&inventory_output=0&rdb_inventory_output=file&TZoutput=0&pm_cd_compare=Greater%20than&radio_parm_cds=previous_parm_cds&qw_attributes=0&format=rdb&rdb_qw_attributes=expanded&date_format=YYYY-MM-DD&rdb_compression=value&qw_sample_wide=0&begin_date=1930-01-01&end_date=2018-12-31

kknierim-usgs commented 5 years ago

Thanks @ldecicco-USGS Could this issue be considered a bug that needs to be fixed in the DataRetrieval() code, so that readNWISqw() will work even if dates are invalid? The issue comes are we are doing large data retrievals

ldecicco-USGS commented 5 years ago

If you give me a site that has more than one measurement (so, the invalid date and valid dates), I can get you a script quickly on how to pull data, and maybe get an alternative up for using readNWISqw at least on Github...but getting the function updated on CRAN will take some time. Pulling the data from WQP might be an alternative too. But...once I have one (or better yet several) sites, I can look into it more.

kknierim-usgs commented 5 years ago

Thanks @ldecicco-USGS The sites I have with older data don't also have newer data, so don't have an example. We will just cutoff our retrieval date at 1940 (we don't have problems with samples 1940-present). Thank you!

ldecicco-USGS commented 5 years ago

That's all I've been able to find as well.... If anyone does find a site, that would make a great test case for this issue.