mikejohnson51
commented
1 year ago Hello @jtkemper!
Thanks again for this issue and sorry for the delay.
This coincided with some development work we had after the CIROH meeting and the good news is that I am not seeing the same results. As part of this update I added an example timeseries - built from AWS - that we can compare against. Some basic findings are below. I would be interested to see if you see anything wrong with this go!
thanks again!
library(nwmTools)
# Read in sample timeseries
## https://github.com/mikejohnson51/nwmTools/blob/master/data-raw/example_timeseries.R
truth = readRDS(system.file("extdata", "sample_ts.rds", package = "nwmTools"))
# Date Range
range(truth$dateTime)
#> [1] "2010-10-01 00:00:00 UTC" "2010-10-02 23:00:00 UTC"
# COMID (associated with gage '04275500')
truth$comid[1]
#> [1] 9528411
# Read in data ...
read = readNWMdata(comid = truth$comid[1],
startDate = truth$dateTime[1],
endDate = truth$dateTime[nrow(truth)] )
# Plot 2 timeseries
plot(truth$dateTime, truth$streamflow,
xlab = "DateTime", ylab = "Q",
main = "Timeseries Comparision")
lines(read$dateTime, read$flow_cms_v2.1, col = 'red')
# Plot values against each other ...
plot(truth$streamflow, read$flow_cms_v2.1,
xlab = "DateTime", ylab = "Q",
main = "Value Comparision")
# Check differences
# Due to the nature of data transposition, and TDS delivery,
# there is a slight lose of precision in the data, however it is minor
plot(truth$streamflow - read$flow_cms_v2.1,
ylab = "Streamflow Difference",
main = "Precision Comparision", pch = 16)
abline(h = 0, col = "red", lwd = 2)
Created on 2023-06-21 by the reprex package (v2.0.1)
jtkemper
Hi Mike,
Over the course of playing around the nwmTools – fantastic package, by the way, thank you – it has come to my attention that there is a systematic offset of 41 hours in the retrospective v2.1 as downloaded using that package and the data downloaded either directly from the AWS S3 bucket or extracted from the netCDF files provided as a USGS ScienceBase release for the retrospective v2.1 runs.
To illustrate, I’ve attached two folders of plots from just a single month (Oct 2010) for 16 USGS gage locations in the Lake Champlain basin in Vermont. In the first folder (uncorr_oct2010), the green line represents the retrospective v2.1 streamflow as downloaded using nwmTools (readNWMdata), the purple represents the streamflow as pulled directly from the S3 bucket, and the black (diff) the difference between the two streamflow values at a given timestamp. The second folder (corr_oct2010) contains plots for the same time period and the same gages, but with 41 hours subtracted from the nwmTools-derived time stamps. The green line plots directly on top of the purple, and the difference between discharge at a given timestamp has gone to ~zero.
I’ve plotted just a single month so that the offset can be clearly seen, but this appears systematic across all times. As far as I can venture a guess, it seems to stem from the fact that the retrospective forecasts for streamflow as downloaded from nwmTools start at 1979-02-02 18:00, whereas the ones from ScienceBase and AWS start at 1979-02-01 01:00 (all times downloaded in UTC). Of course, I could be wrong on this!
Any guidance you may have would be greatly appreciated!
Cheers,
uncorr_oct2010.zip
corr_oct2010.zip