mjbarryfb commented 2 years ago

Hi, I was trying to do some comparisons between data pull from nwmTools and the data from https://registry.opendata.aws/nwm-archive. I notice that there might be a 3 hour difference between the value reported back from nwmTools and the data store in the archive. I don't have very much experience in R programing so forgive me if there is a better way compare. My test code is below: `# Compare results from nwmTools and Registry of Open Data v 2.1

Download Test data from AWS and put in test directory.

library(nwmTools) library(AOI) library(dataRetrieval) library(nhdplusTools) library(ncdf4) library(lubridate)

ncpath <- "D:\temp\nwmTools\dataIn"

Find comid:

library(dataRetrieval)

Location of interest

featurInfo=(findNLDI(location = c(-91.9885, 30.2155))) comidin = as.integer(featurInfo$comid)

Get Data for location:

library("nwmTools") df <- readNWMdata( comid = comidin, startDate = "2016-04-05 12:00:00", endDate = "2016-04-06 12:00:00", tz = "UTC", version = 2.1, addObs = FALSE )

df <- data.frame(df)

Test data from AWS: https://noaa-nwm-retrospective-2-1-pds.s3.amazonaws.com/index.html#model_output/2016/

Reference: Registry of Open Data on AWS (https://registry.opendata.aws/nwm-archive)

Link Ref: https://noaa-nwm-retrospective-2-1-pds.s3.amazonaws.com/index.html

station_id <- comidin

inputlist <- c("201604051200.CHRTOUT_DOMAIN1.comp", "201604051300.CHRTOUT_DOMAIN1.comp", "201604051400.CHRTOUT_DOMAIN1.comp", "201604051500.CHRTOUT_DOMAIN1.comp", "201604051600.CHRTOUT_DOMAIN1.comp", "201604051700.CHRTOUT_DOMAIN1.comp", "201604051800.CHRTOUT_DOMAIN1.comp", "201604051900.CHRTOUT_DOMAIN1.comp", "201604052000.CHRTOUT_DOMAIN1.comp", "201604052100.CHRTOUT_DOMAIN1.comp", "201604052200.CHRTOUT_DOMAIN1.comp", "201604052300.CHRTOUT_DOMAIN1.comp", "201604060000.CHRTOUT_DOMAIN1.comp", "201604060100.CHRTOUT_DOMAIN1.comp", "201604060200.CHRTOUT_DOMAIN1.comp", "201604060300.CHRTOUT_DOMAIN1.comp", "201604060400.CHRTOUT_DOMAIN1.comp", "201604060500.CHRTOUT_DOMAIN1.comp", "201604060600.CHRTOUT_DOMAIN1.comp", "201604060700.CHRTOUT_DOMAIN1.comp", "201604060800.CHRTOUT_DOMAIN1.comp", "201604060900.CHRTOUT_DOMAIN1.comp", "201604061000.CHRTOUT_DOMAIN1.comp", "201604061100.CHRTOUT_DOMAIN1.comp", "201604061200.CHRTOUT_DOMAIN1.comp")

Get data values from the dates to compare

value <- c() time <- c() for (f in inputlist) { fsplit <- strsplit(f,".",fixed=TRUE)[[1]][1] dname <- "streamflow" ncfile <- sprintf("%s\%s",ncpath,f) ncin <- nc_open(ncfile) nc_atts <- ncatt_get(ncin, 0) curdate <- as.POSIXct(nc_atts$model_output_validtime,format="%Y-%m-%d%H:%M:%S",tz="UTC") stream_id <-ncvar_get(ncin,'feature_id') index <- which(stream_id %in% station_id) flow_array <- ncvar_get(ncin,dname) value <- c(value,flow_array[index]) time <- c(time,format(curdate)) nc_close(ncin) } df$compare <- value print(df)

Remove comments see dataset from compare to see df compare TS

dataout <- data.frame(time, value)

print(dataout) `

Output: comid dateTime flow_cms_v2.1 compare 1 21897353 2016-04-05 12:00:00 8.120000 8.10 2 21897353 2016-04-05 13:00:00 8.110000 8.09 3 21897353 2016-04-05 14:00:00 8.110000 8.08 4 21897353 2016-04-05 15:00:00 8.099999 8.08 5 21897353 2016-04-05 16:00:00 8.090000 8.07 6 21897353 2016-04-05 17:00:00 8.080000 8.06 7 21897353 2016-04-05 18:00:00 8.080000 8.05 8 21897353 2016-04-05 19:00:00 8.070000 8.04 9 21897353 2016-04-05 20:00:00 8.059999 8.02 10 21897353 2016-04-05 21:00:00 8.050000 8.00 11 21897353 2016-04-05 22:00:00 8.040000 7.98 12 21897353 2016-04-05 23:00:00 8.020000 7.96 13 21897353 2016-04-06 00:00:00 8.000000 7.94 14 21897353 2016-04-06 01:00:00 7.980000 7.91 15 21897353 2016-04-06 02:00:00 7.960000 7.88 16 21897353 2016-04-06 03:00:00 7.940000 7.85 17 21897353 2016-04-06 04:00:00 7.910000 7.83 18 21897353 2016-04-06 05:00:00 7.880000 7.80 19 21897353 2016-04-06 06:00:00 7.850000 7.77 20 21897353 2016-04-06 07:00:00 7.830000 7.74 21 21897353 2016-04-06 08:00:00 7.800000 7.70 22 21897353 2016-04-06 09:00:00 7.770000 7.67 23 21897353 2016-04-06 10:00:00 7.740000 7.64 24 21897353 2016-04-06 11:00:00 7.700000 7.61 25 21897353 2016-04-06 12:00:00 7.670000 7.57

mikejohnson51 commented 2 years ago

Hi @mjbarryfb! Thanks for the report. We are aware that 42 gaps in the v2.1 data (see #3). That should be fixed soon and then I will check if this is a data issue or a code issue (I am optimistic it is data). Thanks!

Mike

mjbarryfb commented 2 years ago

Hi Mike, Thanks for looking into this and sorry for reporting a duplicate issue.

Max Barry D 916.679.8882 hdrinc.com/follow-ushttp://hdrinc.com/follow-us

From: MikeJohnson-NOAA @.> Sent: Monday, April 11, 2022 9:44 AM To: mikejohnson51/nwmTools @.> Cc: Barry, Max @.>; Mention @.> Subject: Re: [mikejohnson51/nwmTools] Possible date gap between the NOAA National Water Model CONUS Retrospective Dataset and nwmTools (Issue #5)

CAUTION: [EXTERNAL] This email originated from outside of the organization. Do not click links or open attachments unless you recognize the sender and know the content is safe.

Hi @mjbarryfbhttps://nam12.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fmjbarryfb&data=04%7C01%7Cmax.barry%40hdrinc.com%7Cbe030bcb0d8c453beec608da1bda7c90%7C3667e201cbdc48b39b425d2d3f16e2a9%7C0%7C0%7C637852922453819110%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=RottnNT3RKJ8COQ4iWrOFTkUERsJhrtJypJYkTxDyf8%3D&reserved=0! Thanks for the report. We are aware that 42 gaps in the v2.1 data (see #3https://nam12.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fmikejohnson51%2FnwmTools%2Fissues%2F3&data=04%7C01%7Cmax.barry%40hdrinc.com%7Cbe030bcb0d8c453beec608da1bda7c90%7C3667e201cbdc48b39b425d2d3f16e2a9%7C0%7C0%7C637852922453819110%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=TAc0us3yLV7DrawiHnT1dSBWnb0ZB%2BGu14FsXUnoO5U%3D&reserved=0). That should be fixed soon and then I will check if this is a data issue or a code issue (I am optimistic it is data). Thanks!