The API need so to have a server somewhere that's querying a database and returning API results.
I wrote a README that outlines some ideal installation steps and usage for the ballotapi server, so that should be used as the starting point for writing the server.
Prototype checklist:
[x] Command line parsers Done
[x] Package installation config Done
[ ] Dataset loader (e.g. ballotapi load ...) - doesn't need to support all the formats/retrieval methods, just has to be able to load a minimal test dataset
[x] Webserver (e.g. ballotapi runserver) - doesn't need to support all the server features (daemons, signals, etc.), just has to run a basic uwsgi server
[ ] API endpoints (/elections, /precincts, /contests) - doesn't need to support all the parameters be efficient, just needs to return correct data for some requests
Thoughts on Dependencies
This server is expected to used at high scale by the public for ballot information right before an election, so stability and security are top priorities. Many citizens and organizations are depending on this system to keep running, so I don't want to get fancy with the architecture. The server doesn't need to do much, so I want to keep dependencies minimal and only use libraries that are battle tested and well supported.
Here's my current list of dependencies I think we need:
Programs:
python - Language for the server code.
postgres - Database for the ballot data.
postgis - Geospatial plugin for Postgres.
redis - Caching server for faster API responses.
Python libraries:
uwsgi - Production-quality web server for serving API requests.
psycopg2 - Library for connecting to the postgres database.
redis - Library for connecting to the redis caching server.
pyyaml - Library for reading yaml files when loading the data set.
I don't think we need a web framework (like Django REST framework) because I don't think the API is complex enough to warrant it. Also, I don't want to be tempted to expand the server functionality beyond barebones API responses. If we want to have more features than just API responses (e.g. authenticated users, admin interfaces, editing tools, settings pages, etc.), those should be separate apps using separate codebases.
Thoughts on ballotapi runserver
Rather than having a separate dev and production runtimes, I think we should just use uwsgi all the time for both development and production. So, I think the ballotapi runserver command should basically just spin up a child process of uwsgi that runs the server. That way, the only difference in running on development vs running on production is changing the --uwsgi-ini config files.
Thoughts on ballotapi load
Our ballot data set will be constantly changing, just like a code repository (pull requests, branches, reviews, sprints, etc.). So, the authoritative source format for our ballot data won't be in a postgres sql file. Thus, the loader for this server should be able to read sql dumps, tarballs, and git repos. That way, for developers, they can just keep loading the latest git repo version without having to wait for nightly sql dumps. And for production, we can load the latest sql dump file generated by ballotapi export.
Thoughts on file structure
ballotapi/__init__.py
ballotapi/setup.py (what pip uses to install the server)
ballotapi/ballotapi/__init__.py (version number)
ballotapi/ballotapi/cli.py (command line tool parsing)
ballotapi/ballotapi/runserver.py (webserver management)
ballotapi/ballotapi/load.py (dataset loading)
ballotapi/ballotapi/export.py (database dumping)
ballotapi/ballotapi/api/__init__.py
ballotapi/ballotapi/api/elections.py (/elections endpoints)
ballotapi/ballotapi/api/precincts.py (/precincts endpoints)
ballotapi/ballotapi/api/contests.py (/contests endpoints)
ballotapi/ballotapi/utils/*.py (various utilities and shortcut functions)
ballotapi/tests/*.py (tests for the server)
Thoughts on database schema
==Elections==
id - primary key (uuid???)
ocd_id - open civic data reference id, string (long max length???)
election_type - string (short max length???)
election_name - string (long max length???)
date - date (no timezone or time???)
short_info - text (no max length)
==Precincts==
id - primary key (uuid???)
election_id - foreign key to Elections
ocd_id - open civic data reference id, string (long max length???)
voting_info - jsonb
geo - multipolygon
==Contests==
id - primary key (uuid???)
election_id - foreign key to Elections
ocd_id - open civic data reference id, string (long max length???)
contest_type - string (short max length???)
contest_level - string (short max length???)
voting_method - string (short max length???)
voting_instructions - text (no max length)
title - string (short max length???)
question - text (no max length)
==Choice==
id - primary key (uuid???)
contest_id - foreign key to Contests
ocd_id - open civic data reference id, string (long max length???)
type - string (short max length???)
title - string (short max length???)
party - string (short max length???)
info - text (no max length)
==ContestPrecincts==
contest_id - foreign key to Contests
precinct_id - foreign key to Precincts
The API need so to have a server somewhere that's querying a database and returning API results.
I wrote a README that outlines some ideal installation steps and usage for the ballotapi server, so that should be used as the starting point for writing the server.
Prototype checklist:
ballotapi load ...) - doesn't need to support all the formats/retrieval methods, just has to be able to load a minimal test datasetballotapi runserver) - doesn't need to support all the server features (daemons, signals, etc.), just has to run a basic uwsgi server/elections,/precincts,/contests) - doesn't need to support all the parameters be efficient, just needs to return correct data for some requestsThoughts on Dependencies
This server is expected to used at high scale by the public for ballot information right before an election, so stability and security are top priorities. Many citizens and organizations are depending on this system to keep running, so I don't want to get fancy with the architecture. The server doesn't need to do much, so I want to keep dependencies minimal and only use libraries that are battle tested and well supported.
Here's my current list of dependencies I think we need:
python- Language for the server code.postgres- Database for the ballot data.postgis- Geospatial plugin for Postgres.redis- Caching server for faster API responses.uwsgi- Production-quality web server for serving API requests.psycopg2- Library for connecting to the postgres database.redis- Library for connecting to the redis caching server.pyyaml- Library for reading yaml files when loading the data set.I don't think we need a web framework (like Django REST framework) because I don't think the API is complex enough to warrant it. Also, I don't want to be tempted to expand the server functionality beyond barebones API responses. If we want to have more features than just API responses (e.g. authenticated users, admin interfaces, editing tools, settings pages, etc.), those should be separate apps using separate codebases.
Thoughts on
ballotapi runserverRather than having a separate dev and production runtimes, I think we should just use uwsgi all the time for both development and production. So, I think the
ballotapi runservercommand should basically just spin up a child process of uwsgi that runs the server. That way, the only difference in running on development vs running on production is changing the--uwsgi-iniconfig files.Thoughts on
ballotapi loadOur ballot data set will be constantly changing, just like a code repository (pull requests, branches, reviews, sprints, etc.). So, the authoritative source format for our ballot data won't be in a postgres sql file. Thus, the loader for this server should be able to read sql dumps, tarballs, and git repos. That way, for developers, they can just keep loading the latest git repo version without having to wait for nightly sql dumps. And for production, we can load the latest sql dump file generated by
ballotapi export.Thoughts on file structure
Thoughts on database schema