GoogleCloudPlatform / covid-19-open-data

Datasets of daily time-series data related to COVID-19 for over 20,000 distinct locations around the world.
Apache License 2.0
472 stars 131 forks source link

Stringency Index now NaN #456

Closed nhartland closed 3 years ago

nhartland commented 3 years ago

Hi there,

First of all, many thanks for creating and maintaining this repo!

I've noticed that the stringency_index field is now NaN for a lot of countries. From what I can see the Oxford API is still returning sensible values here, but checking the provided file oxford-government-response.json I see lots of NaNs. This wasn't the case when I checked yesterday.

The effect can be clearly seen in the public BQ table:

SELECT
  date,
  country_name,
  stringency_index
FROM
  `bigquery-public-data.covid19_open_data.covid19_open_data`
WHERE
  country_name = "France"
  AND aggregation_level = 0
ORDER BY
  date DESC

Is returning a long list of empty values for stringency_index.

nhartland commented 3 years ago

Just as a follow up, I can only currently see stringency_index in the JSON file for the following keys on the 15th April 2021:

'CA_AB', 'CA_MB', 'CA_NB', 'CA_NL', 'CA_NS', 'CA_NT', 'CA_QC',
       'ES', 'FI', 'GA', 'GN', 'MR', 'NE', 'US_AK', 'US_AL', 'US_AR',
       'US_CA', 'US_CO', 'US_DC', 'US_DE', 'US_GA', 'US_IA', 'US_ID',
       'US_IL', 'US_IN', 'US_KS', 'US_LA', 'US_MA', 'US_MD', 'US_ME',
       'US_MI', 'US_MN', 'US_MO', 'US_MT', 'US_NC', 'US_ND', 'US_NE',
       'US_NH', 'US_NJ', 'US_NM', 'US_NV', 'US_NY', 'US_OH', 'US_OK',
       'US_OR', 'US_RI', 'US_SD', 'US_TN', 'US_TX', 'US_UT', 'US_VA',
       'US_WI', 'US_WY'

But checking the Oxford API with

curl https://covidtrackerapi.bsg.ox.ac.uk/api/v2/stringency/date-range/2021-04-15/2021-04-15

I can see a pretty complete record.

nhartland commented 3 years ago

On checking again today I've noticed that the Oxford API is actually intermittently dropping data, so most likely the problem source is there.

Reported at https://github.com/OxCGRT/covid-policy-tracker/issues/87