Update the ACS/Census data that powers the demographic sorting on county tables

[mileswwatkins]

Tyler left behind some tooling to update these: https://github.com/nprapps/elections18-general/blob/e345242ccc768cc7d3884225cd313f1f73f3c3ab/fabfile/data.py#L438

[mileswwatkins]

You can see what some of the populated data files looked like in 2016 here: https://github.com/nprapps/elections16-general/tree/master/data. Hadn't remembered that all these files were checked-in and available to browse for last cycle!

[freejoe76]

Note that the ACS 5-year data for 2017 won't be released until December 2018, per https://www.census.gov/programs-surveys/acs/news/data-releases/2017/release-schedule.html

[freejoe76]

Other notes: I'll write up the workflow necessary to put together data/unemployment.csv (see unemployment.csv in 2016's repo), which is necessary in order to run fab data.save_old_data, which I'm pretty sure is crucial to generate the json files used for the data in the county tables.

[freejoe76]

There's also the data/twentyTwelve.csv spreadsheet (first three lines of 2016's file pasted below), which is also necessary for fab data.save_old_data, I don't know exactly what the workflow will be on that, I do know it's also necessary for #34.

id,raceid,racetype,racetypeid,ballotorder,candidateid,description,delegatecount,electiondate,electtotal,electwon,fipscode,first,incumbent,initialization_data,is_ballot_measure,last,lastupdated,level,national,officeid,officename,party,polid,polnum,precinctsreporting,precinctsreportingpct,precinctstotal,reportingunitid,reportingunitname,runoff,seatname,seatnum,statename,statepostal,test,uncontested,votecount,votepct,winner
0-polid-1918-None,0,General,G,2,1918,,0,2012-11-06,538,332,,Barack,True,False,False,Obama,2016-07-26T17:41:11.483Z,national,True,P,President,Dem,1918,1918,173375,0.9961,174047,,,False,,,National,US,False,False,62615406,0.50608,True
0-polid-893-None,0,General,G,1,893,,0,2012-11-06,538,206,,Mitt,False,False,False,Romney,2016-07-26T17:41:11.483Z,national,True,P,President,GOP,893,893,173375,0.9961,174047,,,False,,,National,US,False,False,59142004,0.478006,False

[freejoe76]

I found the place where the previous unemployment data started, and documented it in the code ( https://github.com/nprapps/elections18-general/commit/036dc60c321696dfc47773e8a4e74027ec2a2b56 ), so that's progress.

[freejoe76]

I was caught up in getting the data.update_old_data command running and neglected to mention in commit ac7aaeb that the update also included updates to all the state census data, except Alaska.

[freejoe76]

Also, the Alaska data updated in cf6fd21 is incomplete and will need to be fixed. I should have verified this before committing and before pushing.

[mileswwatkins]

@freejoe76: Since Alaska isn't organized into formal counties (it has really janky county-equivalents, that aren't used most places), the AP doesn't provide county-level data for Alaskan gubernatorial and US Senate (and presidential) elections. So we won't need Alaskan county info, no need to go the extra mile, and sorry for not remembering to mention this!

[freejoe76]

@mileswwatkins All good! I'm not sure exactly what the things were that went wrong but it's all fixed now, am closing this issue and merging to master.

[mileswwatkins]

@freejoe76, can you also update the footnote used on the state pages? Right now it still reads

Unemployment numbers from the Bureau of Labor Statistics (2015)

and there may be other sourcing bits that I've missed.

[mileswwatkins]

@freejoe76, you'd also asked whether this code needs to be in the back-end repo, or if it could live in the graphics repo. Tyler responded:

i think you're right, those could go in the frontend repo i was just keeping all the data products in one repo, but i ended up hand-copying stuff because of that separation