Data files getting HUGE

[phocks]

OK so we've come to a point where we're going to have to start separating some of this data, or putting it in a different format, or something like that.

If we could collect a list of stories that are using this data (@drzax @colingourlay), then we can make sure they continue to work when we update the data source.

Off the top of my head we have Simon's growth factor story https://www.abc.net.au/news/2020-04-10/coronavirus-data-australia-growth-factor-covid-19/12132478?nw=0 which could conceivably be rewritten to use the individual country data instead of the combined country data.

And we have the stories built off Colin's COVID chart builder tool. And the initial COVID stories we did...

Anyway if anyone has any other ideas please drop them below.

jb

[phocks]

Zipped (json files served using gzip) it's still not toooooo bad... but going forward into the future we'll probably want to do something about these files. Otherwise we can just make the decision to cut them off.

[phocks]

Current files being ingested (as far as I can tell)

https://www.abc.net.au/dat/news/interactives/covid19-data/hybrid-country-totals.json by Simon's growth factor story

https://www.abc.net.au/dat/news/interactives/covid19-data/places-totals.json by Colin's chart builder tool + embeds: also by our initial Corona stories here, and here

That's all, I think...

[phocks]

OK the lookup json is now here: https://www.abc.net.au/dat/news/interactives/covid19-data/places-lookup.json And data will be available under places/ eg. https://www.abc.net.au/dat/news/interactives/covid19-data/places/us.json

@drzax and @colingourlay to update their code eventually, then we can stop generating the HUGE combined files and close this issue!

[phocks]

Just bumping this again. places-totals.json is now above 7.3MB. I reckon we might wanna deprecate it pretty soon... @colingourlay @drzax

Will be releasing a new version with @katiaeirin 's vaccination data included shortly