lindsayplatt
commented
4 years ago Further investigation reveals that those SD sites had data pulled down, but not enough to be considered "active" for 2019. Pulling data down for the sites that didn't meet the mark for "active" in SD will be my next step.
# Why are we missing SD?
# Did Something happen during the data munging?
# First, defining a bbox that is mostly SD but not entirely.
sd_bbox <- c(xmin = -104.471120, ymin = 42.447817, xmax = -96.097503, ymax = 46.192474)Pre-data munging data only near SD
dv_data <- readRDS("10_nwis_pull/out/nwis_dv_data.rds")
dv_sites <- unique(dv_data$site_no)
dv_sites_sd <- readNWISsite(dv_sites) %>%
filter(dec_long_va > sd_bbox[["xmin"]],
dec_long_va < sd_bbox[["xmax"]],
dec_lat_va > sd_bbox[["ymin"]],
dec_lat_va < sd_bbox[["ymax"]]) %>%
pull(site_no)
dv_data_sd <- dv_data %>% filter(site_no %in% dv_sites_sd)
dv_data_sd_2019 <- dv_data_sd %>%
mutate(year = format(Date, "%Y")) %>%
filter(year == "2019")
uv_data <- readRDS("10_nwis_pull/out/nwis_uv_data.rds")
uv_sites <- unique(uv_data$site_no)
uv_sites_sd <- readNWISsite(uv_sites) %>%
filter(dec_long_va > sd_bbox[["xmin"]],
dec_long_va < sd_bbox[["xmax"]],
dec_lat_va > sd_bbox[["ymin"]],
dec_lat_va < sd_bbox[["ymax"]]) %>%
pull(site_no)
uv_data_sd <- uv_data %>% filter(site_no %in% uv_sites_sd)
uv_data_sd_2019 <- uv_data_sd %>%
mutate(year = format(dateTime, "%Y")) %>%
filter(year == "2019")
# Now plot the sites that appear in both.
both_sites_sd_2019 <- readNWISsite(unique(c(dv_data_sd_2019$site_no, uv_data_sd_2019$site_no)))
# Where are those sites?
maps::map("state", "south dakota")
points(both_sites_sd_2019$dec_long_va, both_sites_sd_2019$dec_lat_va)
title("Pre-munge 2019 data")
Post-data munging data only near SD
all_data <- readRDS("20_data_munge/out/daily_flow.rds")
data_2019 <- all_data %>%
mutate(year = format(date, "%Y")) %>%
filter(year == "2019")
gages_2019_sd <- readNWISsite(unique(data_2019$site_id)) %>%
filter(dec_long_va > sd_bbox[["xmin"]],
dec_long_va < sd_bbox[["xmax"]],
dec_lat_va > sd_bbox[["ymin"]],
dec_lat_va < sd_bbox[["ymax"]])
# Where are those sites?
maps::map("state", "south dakota")
points(gages_2019_sd$dec_long_va, gages_2019_sd$dec_lat_va)
title("Post-munge 2019 data")
Post-summarization - filtering to those that are "active"
# Still have all of SD gages
# Now, test which are "active" by manually calculating that here
days_of_data_gages_2019_sd <- data_2019 %>%
filter(site_id %in% unique(gages_2019_sd$site_no)) %>%
group_by(site_id) %>%
summarize(count = n())
active_gages_2019_sd <- days_of_data_gages_2019_sd %>%
filter(count >= 335) %>%
pull(site_id) %>% readNWISsite()
maps::map("state", "south dakota")
points(active_gages_2019_sd$dec_long_va, active_gages_2019_sd$dec_lat_va)
title("Post-munge 2019 data, active gages only")
Final summary of what gages aren't counted as active
# After further investigation, it appears that many SD gages are
# dropped due to not having enough data to be counted as "active"
nrow(gages_2019_sd)
# [1] 143
nrow(active_gages_2019_sd)
# [1] 66
days_of_data_gages_2019_sd %>% filter(count < 335) %>% arrange(count) %>% as.data.frame()
# site_id count
# 06479520 22
# 06441000 90
# 06437500 141
# 06354480 214
# 06410000 214
# 05051600 215
# 06401500 222
# 06452320 242
# 06478000 242
# 06408700 271
# 06465700 287
# 06475000 289
# 06449500 292
# 06450500 292
# 06403300 300
# 06447000 300
# 06447230 300
# 06446000 301
# 06446500 301
# 06479215 301
# 06357800 302
# 06464100 302
# 06464500 303
# 06447450 304
# 06442600 305
# 06481480 306
# 06442130 308
# 06452000 308
# 06479438 308
# 06481000 308
# 06479500 309
# 06479525 309
# 06479770 309
# 06480000 309
# 06480650 309
# 06354881 310
# 06441500 310
# 06442900 310
# 06471000 310
# 06471200 310
# 06471500 310
# 06471800 310
# 06481500 310
# 06482000 310
# 06482610 310
# 06478690 311
# 06479010 311
# 06482020 311
# 06485910 311
# 06360500 313
# 06472000 313
# 06473000 314
# 06476000 314
# 06477000 314
# 06478600 314
# 06483950 314
# 06438000 315
# 06485500 315
# 06470878 317
# 05293000 320
# 06425500 320
# 05051300 323
# 06436198 323
# 06437000 323
# 06478500 323
# 05050000 324
# 05291000 324
# 06409000 324
# 06478513 324
# 05290000 325
# 06438500 325
# 06477500 329
# 06410500 330
# 06425100 330
# 06436180 330
# 06404998 334
# 06421500 334
limnoliver
Need to figure it out, but we noticed this in the gages through the ages viz. The count of gages in 2019 on gages through the ages matches what we have coming out of this pipeline, so something in here must be an issue. Here is a bit of what I did to evaluate this (haven't truly found the source of the error yet).
2019 "active" gages