vooc-election-results

Pipeline infrastructure for live 2020 general election results for the Voice of Orange County.

Proposed pipeline plan

1. Fetch the results

   a. From each County Registrar of Voters b. From each relevant endpoint of the Secretary of States API

2. Compare the new results to the previous results.

   a. If different, store a "latest" copy and a datetime-stamped copy.

3. Parse out results for individual contests

   a. Every contest in OC b. Just the contests with supplementary results for OC contests c. The SoS results are already parsed out.

4. Compare and store parsed results

5. Compile final json for each contest with new parsed results

   a. For the vast majority of the OC contests, we will simply update the datetime stamp and the vote counts b. For contests, we will combine vote counts as well.

Bootstrapping

To set up your local dev environment, you need to install both sets of requirements:

pip install -r requirements.txt

pip install -r requirements-dev.txt

You'll also need to set a few environment variables. For this, I'm using direnv.

Deploying

Here's the stack:

• Code is is hosted in an AWS Lambda function.
• Data and cached HTML are stored in an AWS S3 bucket.
• Scheduling is handled via AWS EventBridge.

To deploy:

source deploy.sh

Which will:

• Make the bucket (if it doesn't already exist)
• Set the bucket's CORS policy (necessary to load the data in Observable)
• Package the source code
• Create the lambda function
• Set the environment variables in the lambda function
• Schedule the frequency of function calls

Managing the deployment

To update the deployment:

source update.sh

Note that:

• New files that need to be in the deployment should be specified in package.sh
• New environment variables that need to be set in the lambda function should be specified in set-env-vars.sh

If your update fails, you might try tearing down the entire deployment (except the S3 bucket):

source teardown.sh

And then re-deploying:

source deploy.sh

Updates will also reset the function call frequency. If you just need to modify the schedule, then:

source schedule.sh