search

DOI-USGS / dataRetrieval

This R package is designed to obtain USGS or EPA water quality sample data, streamflow data, and metadata directly from web services.
https://doi-usgs.github.io/dataRetrieval/
Other
259 stars 84 forks source link

tz issue with readNWISqw #526

Closed jmusgs closed 3 years ago

jmusgs commented 3 years ago

When pulling data from NWIS using readNWISqw and specifying a time zone, incorrect time information for the zone is returned

library(dataRetrieval) #using v 2.7.6 library(dplyr) rawNWISqwData <- readNWISqw("01193500","00940","2010-10-01","2020-09-30",expanded = TRUE, reshape = FALSE, tz = "EST")

print <- rawNWISqwData %>% select(sample_dt, sample_tm, sample_start_time_datum_cd_reported, startDateTime,sample_start_time_datum_cd)

returns:

tail(print,5)

sample_dt sample_tm sample_start_time_datum_cd_reported       startDateTime sample_start_time_datum_cd

38 2018-10-10 13:45 EDT 2018-10-10 13:45:00 EST 39 2019-05-23 15:15 EDT 2019-05-23 15:15:00 EST 40 2019-07-18 09:15 EDT 2019-07-18 09:15:00 EST 41 2019-10-02 11:45 EDT 2019-10-02 11:45:00 EST 42 2020-01-06 15:30 EST 2020-01-06 15:30:00 EST

It does not appear to be correcting the reported time code for the sample to the requested tz to be output

Using "America/Jamaica" does not seem to work either

ldecicco-USGS commented 3 years ago

EST won't work (it's not a valid R timezone). America/Jamaica looks like it's working to me though:

> x <- readNWISqw("01193500","00940","2010-10-01","2020-09-30",expanded = TRUE,
+                             reshape = FALSE,
+                             tz = "America/New_York")
> tail(x$startDateTime)
[1] "2018-07-19 11:15:00 EDT" "2018-10-10 14:45:00 EDT"
[3] "2019-05-23 16:15:00 EDT" "2019-07-18 10:15:00 EDT"
[5] "2019-10-02 12:45:00 EDT" "2020-01-06 15:30:00 EST"

> x <- readNWISqw("01193500","00940","2010-10-01","2020-09-30",expanded = TRUE,
+                             reshape = FALSE,
+                             tz = "America/Jamaica")
> tail(x$startDateTime)
[1] "2018-07-19 10:15:00 EST" "2018-10-10 13:45:00 EST"
[3] "2019-05-23 15:15:00 EST" "2019-07-18 09:15:00 EST"
[5] "2019-10-02 11:45:00 EST" "2020-01-06 15:30:00 EST"

Is that different than what you are seeing?

jmusgs commented 3 years ago

if I do America/Jamaica I get sample_dt sample_tm sample_start_time_datum_cd_reported startDateTime sample_start_time_datum_cd 38 2018-10-10 13:45 EDT 2018-10-10 13:45:00 America/Jamaica 39 2019-05-23 15:15 EDT 2019-05-23 15:15:00 America/Jamaica 40 2019-07-18 09:15 EDT 2019-07-18 09:15:00 America/Jamaica 41 2019-10-02 11:45 EDT 2019-10-02 11:45:00 America/Jamaica 42 2020-01-06 15:30 EST 2020-01-06 15:30:00 America/Jamaica

The first date and separate time column are the date and time stored in NWIS for the sample as the technician entered it. IN New England we stored all values as EST for water quality up until recently. So 13:45 EDT should be converted to 12:45 EST, and 09:15 EDT should be 08:15 EST etc.

If I use "America/New_York", I get

sample_dt sample_tm sample_start_time_datum_cd_reported startDateTime sample_start_time_datum_cd 38 2018-10-10 13:45 EDT 2018-10-10 14:45:00 America/New_York 39 2019-05-23 15:15 EDT 2019-05-23 16:15:00 America/New_York 40 2019-07-18 09:15 EDT 2019-07-18 10:15:00 America/New_York 41 2019-10-02 11:45 EDT 2019-10-02 12:45:00 America/New_York 42 2020-01-06 15:30 EST 2020-01-06 15:30:00 America/New_York

So in this case 13:45 EDT should to stay as 13:45 in America/New_York, however, the last value is correctly kept as the same.

Hope I am not missing something here. Thanks for your input.

jmusgs commented 3 years ago

Lastly, just ran using dataRetrieval v 2.7.4, which I had on another computer and "America/Jamaica" it converts correctly, using "EST" also converts correctly sample_dt sample_tm sample_start_time_datum_cd_reported startDateTime sample_start_time_datum_cd 38 2018-10-10 13:45 EDT 2018-10-10 12:45:00 America/Jamaica 39 2019-05-23 15:15 EDT 2019-05-23 14:15:00 America/Jamaica 40 2019-07-18 09:15 EDT 2019-07-18 08:15:00 America/Jamaica 41 2019-10-02 11:45 EDT 2019-10-02 10:45:00 America/Jamaica 42 2020-01-06 15:30 EST 2020-01-06 15:30:00 America/Jamaica

ldecicco-USGS commented 3 years ago

Sounds good, my goal is to take a look today!

jmusgs commented 3 years ago

Thanks!

From: Laura DeCicco notifications@github.com Sent: Friday, October 9, 2020 10:26 AM To: USGS-R/dataRetrieval dataRetrieval@noreply.github.com Cc: Mullaney, John R jmullane@usgs.gov; Author author@noreply.github.com Subject: [EXTERNAL] Re: [USGS-R/dataRetrieval] tz issue with readNWISqw (#526)

This email has been received from outside of DOI - Use caution before clicking on links, opening attachments, or responding.

Sounds good, my goal is to take a look today!

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHubhttps://github.com/USGS-R/dataRetrieval/issues/526#issuecomment-706213227, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AD7GNPCHYLKTY3U2WB7XCE3SJ4MPVANCNFSM4SGUFMQA.

ldecicco-USGS commented 3 years ago

OK, so here's what the lastest PR is returning:

x <- readNWISqw("01193500","00940","2010-10-01","2020-09-30",
                  expanded = TRUE,
                  reshape = FALSE,
                  tz = "America/Jamaica")

tail(x[,c("sample_dt", "sample_tm", "sample_start_time_datum_cd_reported", "startDateTime")])
    sample_dt sample_tm sample_start_time_datum_cd_reported       startDateTime

37 2018-07-19     10:15                                 EDT 2018-07-19 09:15:00
38 2018-10-10     13:45                                 EDT 2018-10-10 12:45:00
39 2019-05-23     15:15                                 EDT 2019-05-23 14:15:00
40 2019-07-18     09:15                                 EDT 2019-07-18 08:15:00
41 2019-10-02     11:45                                 EDT 2019-10-02 10:45:00
42 2020-01-06     15:30                                 EST 2020-01-06 15:30:00

x <- readNWISqw("01193500","00940","2010-10-01","2020-09-30",
              expanded = TRUE,
              reshape = FALSE,
              tz = "America/New_York")
tail(x[,c("sample_dt", "sample_tm", "sample_start_time_datum_cd_reported", "startDateTime")])
    sample_dt sample_tm sample_start_time_datum_cd_reported       startDateTime

37 2018-07-19     10:15                                 EDT 2018-07-19 10:15:00
38 2018-10-10     13:45                                 EDT 2018-10-10 13:45:00
39 2019-05-23     15:15                                 EDT 2019-05-23 15:15:00
40 2019-07-18     09:15                                 EDT 2019-07-18 09:15:00
41 2019-10-02     11:45                                 EDT 2019-10-02 11:45:00
42 2020-01-06     15:30                                 EST 2020-01-06 15:30:00

Sound right?

jmusgs commented 3 years ago

This looks correct. When you are done, let us know the best way to update to the revised version. Thanks for looking into this. Does this also fix the same issue with readNWISgwl?

ldecicco-USGS commented 3 years ago

Yup! Run this:

remotes::install_github("USGS-R/dataRetrieval")

and you should have the working version. I've got a few other issues to fix up before it heads to CRAN, but it shouldn't be too long (a couple of weeks maybe)

To make sure you've got it, check if the version is this:

packageVersion("dataRetrieval")
[1] ‘2.7.6.9001’
jmusgs commented 3 years ago

Thank you!

From: Laura DeCicco notifications@github.com Sent: Tuesday, October 13, 2020 4:52 PM To: USGS-R/dataRetrieval dataRetrieval@noreply.github.com Cc: Mullaney, John R jmullane@usgs.gov; Author author@noreply.github.com Subject: [EXTERNAL] Re: [USGS-R/dataRetrieval] tz issue with readNWISqw (#526)

This email has been received from outside of DOI - Use caution before clicking on links, opening attachments, or responding.

Yup! Run this:

remotes::install_github("USGS-R/dataRetrieval")

and you should have the working version. I've got a few other issues to fix up before it heads to CRAN, but it shouldn't be too long (a couple of weeks maybe)

To make sure you've got it, check if the version is this:

packageVersion("dataRetrieval")

[1] ‘2.7.6.9001’

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHubhttps://github.com/USGS-R/dataRetrieval/issues/526#issuecomment-708002522, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AD7GNPH5EG7JVUARB2WGY5LSKS42LANCNFSM4SGUFMQA.