search

DOI-USGS / dataRetrieval

This R package is designed to obtain USGS or EPA water quality sample data, streamflow data, and metadata directly from web services.
https://doi-usgs.github.io/dataRetrieval/
Other
262 stars 85 forks source link

Are composite samples problematic for time zone in readNWISqw? #536

Closed aspaetzel closed 3 years ago

aspaetzel commented 4 years ago

Does dataRetrieval 2.7.6.9001 readNWISqw tz work with sites and pcodes with composite samples? 

remotes::install_github("USGS-R/dataRetrieval")
library(dataRetrieval)
x<- readNWISqw("01115114","00666","2010-10-01","2020-09-30",expanded = TRUE, reshape = FALSE, tz = "America/Jamaica")
tail(x[,c("sample_dt", "sample_tm", "sample_end_dt", "sample_start_time_datum_cd_reported", "startDateTime","sample_start_time_datum_cd")])
    sample_dt sample_tm sample_end_dt sample_start_time_datum_cd_reported       startDateTime sample_start_time_datum_cd
75 2019-07-10     11:30          <NA>                     America/Jamaica 2019-07-10 10:30:00            America/Jamaica
76 2019-07-22     11:30          <NA>                     America/Jamaica 2019-07-22 10:30:00            America/Jamaica
77 2019-07-22     21:21    2019-07-24                     America/Jamaica 2019-07-22 20:21:00            America/Jamaica
78 2019-08-27     10:00          <NA>                     America/Jamaica 2019-08-27 09:00:00            America/Jamaica
79 2019-09-24     04:51    2019-09-25                     America/Jamaica 2019-09-24 03:51:00            America/Jamaica
80 2019-09-26     12:00          <NA>                     America/Jamaica 2019-09-26 11:00:00            America/Jamaica

Expected behavior sample_start_time_datum_cd_reported should be EST/EDT sample_start_time_datum_cd should be America/Jamaica

If you change the code above to ask for pcode 00940, then it works. None of the chloride samples are composites.

The time conversions are correct, but the reported time zone is incorrect.

ldecicco-USGS commented 4 years ago

Composite samples are definitely problematic. I had to put in a special set of rules awhile back: https://github.com/USGS-R/dataRetrieval/blob/master/R/importRDB1.r#L218

It's especially hard with mixes of composite and not composite because of the way they are reported. I'll go see if I can retain the reported USGS words without screwing up any actual times.

ldecicco-USGS commented 4 years ago

Here's the results from the latest PR. I'm still exploring if there are other composite issues that I don't test:

x<- readNWISqw("01115114","00666","2010-10-01","2020-09-30",expanded = TRUE, reshape = FALSE, tz = "America/Jamaica")
> tail(x[,c("sample_dt", "sample_tm", "sample_end_dt", "sample_start_time_datum_cd_reported", "startDateTime","sample_start_time_datum_cd", "endDateTime")])
    sample_dt sample_tm sample_end_dt sample_start_time_datum_cd_reported
75 2019-07-10     11:30          <NA>                                 EDT
76 2019-07-22     11:30          <NA>                                 EDT
77 2019-07-22     21:21    2019-07-24                                 EDT
78 2019-08-27     10:00          <NA>                                 EDT
79 2019-09-24     04:51    2019-09-25                                 EDT
80 2019-09-26     12:00          <NA>                                 EDT
         startDateTime sample_start_time_datum_cd         endDateTime
75 2019-07-10 10:30:00            America/Jamaica                <NA>
76 2019-07-22 10:30:00            America/Jamaica                <NA>
77 2019-07-22 20:21:00            America/Jamaica 2019-07-24 12:11:00
78 2019-08-27 09:00:00            America/Jamaica                <NA>
79 2019-09-24 03:51:00            America/Jamaica 2019-09-25 00:48:00
80 2019-09-26 11:00:00            America/Jamaica                <NA>
ldecicco-USGS commented 3 years ago

@aspaetzel could you install dataRetrieval like this:

remotes::install_github("USGS-R/dataRetrieval")

and let me know if your composite samples look OK? I changed the code again...I think making it even more simple, but it would be great to get your eyes on it too.

aspaetzel commented 3 years ago

Yes, working perfectly now! Thank you SO much!

ldecicco-USGS commented 3 years ago

Awesome, thanks for checking.