nolangormley
commented
10 months ago It looks like this is not an is_latest mismatch issue. I checked the entirety of HHS data with this query and it came back empty (coming back from a bad sinus infection, so please tell me if I made a dumb error).
select
*
from epimetric_latest el
join (
select
geo_key_id,
signal_key_id,
time_type,
time_value,
max(issue) as max_issue
from epimetric_full
join geo_dim geo using (geo_key_id)
join signal_dim sig using (signal_key_id)
where
sig.source = 'hhs'
and time_type = 'day'
and geo.geo_type = 'state'
group by
geo_key_id,
signal_key_id,
time_type,
time_value
) latest_full
using (geo_key_id, signal_key_id, time_type, time_value)
where max_issue <> issue;
brookslogan
melange396
Actual Behavior:
hhsconfirmed_admissions_influenza_1ddata, it has mismatches with both the latest version of the upstream "timeseries" data set and with the latest version of the timeseries data archived in the archive data set.The above differences do not seem fully explainable by a single mismatched version propagating to future versions; it seems that multiple mismatched versions must have been ingested. The
time_valuerange of mismatches for the historical snapshot I tried were limited totime_values close to theas_ofdate, but thetime_valuerange of mismatches for the latest-version snapshot when I tried it yesterday, 2023-10-12, extended back farther in time, but also did not extend to the most recenttime_values.Unexplored possibility: perhaps the historical version mismatch on 2023-04-25 (first date I tried) was due to delays in acquisition (#1889), and the latest/2023-10-12 version mismatch could be due to some other reason.
Mismatches on latest, run on 2023-10-12, after API update due to 2023-10-11 upstream update
```r library(tidyverse) library(epidatr) library(epiprocess) disable_cache() test_as_of <- Sys.Date() # 2023-10-12 epidata_api_as_of <- pub_covidcast( "hhs", "confirmed_admissions_influenza_1d", "state", "day", "*", "*", as_of = as.character(test_as_of) ) socrata_archive_info <- RSocrata::read.socrata(url = "https://healthdata.gov/resource/qqte-vkut.json") %>% dplyr::arrange(update_date) %>% as_tibble() socrata_archive_as_of <- socrata_archive_info %>% filter(update_date <= test_as_of) %>% slice_max(update_date) %>% pull(url) %>% # Stop if there were multiple updates published to resolve ambiguity. Except # there were supposed to have been multiple updates on 2023-10-11, one only # revising values, and another that added new rows. So maybe not all updates # are recorded in this archive? {stopifnot(length(.) == 1L); .} %>% read_csv() socrata_timeseries <- RSocrata::read.socrata(url = "https://healthdata.gov/resource/g62h-syeh.json") %>% as_tibble() # saveRDS(socrata_timeseries, "socrata_timeseries_2023-10-12-circa-1530PT.rds") # socrata_timeseries <- readRDS("socrata_timeseries_2023-10-12-circa-1530PT.rds") epidata_api_as_of_reformatted <- epidata_api_as_of %>% select(geo_value, time_value, admissions = value) socrata_archive_as_of_reformatted = socrata_archive_as_of %>% transmute(geo_value = tolower(state), time_value = date - 1L, admissions = previous_day_admission_influenza_confirmed) socrata_timeseries_reformatted = socrata_timeseries %>% transmute( geo_value = tolower(state), time_value = as.Date(date) - 1L, admissions = previous_day_admission_influenza_confirmed ) joined <- list( api = epidata_api_as_of_reformatted, sarchive = socrata_archive_as_of_reformatted, timeseries = socrata_timeseries_reformatted ) %>% map2(names(.), ~ .x %>% rename("admissions_{.y}" := admissions) ) %>% reduce(full_join, by = c("geo_value", "time_value")) %>% {.} # mismatches between API as-of results and the Socrata/healthdata.gov archive # data as-of results, for this admissions signal, ignoring differences between # NA values & rows not existing: mismatches <- joined %>% filter(is.na(admissions_api) != is.na(admissions_sarchive) | !is.na(admissions_api) & admissions_api != admissions_sarchive) %>% {.} print(mismatches) #> # A tibble: 128 × 5 #> geo_value time_value admissions_api admissions_sarchive admissions_timeseries #>Mismatches between API and upstream snapshots for one `as_of` date in the past
```r test_as_of <- as.Date("2023-04-25") epidata_api_as_of <- pub_covidcast( "hhs", "confirmed_admissions_influenza_1d", "state", "day", "*", "*", as_of = as.character(test_as_of) ) socrata_archive_as_of <- socrata_archive_info %>% filter(update_date <= test_as_of) %>% slice_max(update_date) %>% pull(url) %>% # Stop if there were multiple updates published to resolve ambiguity. Except # maybe not all of these updates are reliably recorded in the archive data # set, based on 2023-10-12? {stopifnot(length(.) == 1L); .} %>% read_csv() epidata_api_as_of_reformatted <- epidata_api_as_of %>% select(geo_value, time_value, admissions = value) socrata_archive_as_of_reformatted = socrata_archive_as_of %>% transmute(geo_value = tolower(state), time_value = date - 1L, admissions = previous_day_admission_influenza_confirmed) joined <- list( api = epidata_api_as_of_reformatted, sarchive = socrata_archive_as_of_reformatted ) %>% map2(names(.), ~ .x %>% rename("admissions_{.y}" := admissions) ) %>% reduce(full_join, by = c("geo_value", "time_value")) %>% {.} # mismatches between API as-of results and the Socrata/healthdata.gov archive # data as-of results, for this admissions signal, ignoring differences between # NA values & rows not existing: mismatches <- joined %>% filter(is.na(admissions_api) != is.na(admissions_sarchive) | !is.na(admissions_api) & admissions_api != admissions_sarchive) %>% {.} print(mismatches) #> # A tibble: 54 × 4 #> geo_value time_value admissions_api admissions_sarchive #>Expected behavior
I expected these to match, based on covidcast hhs docs pointing to
covid_hosp_state_timeseriesdocs pointing to upstream "timeseries" data set.Context
I was trying to recreate the FluSight baseline model's forecasts from last season, which use the "truth" data here, which are acquired from the healthdata.gov timeseries archive data set.