What is this?

A project of loosely coupled applications that force aspects of the political process to give each other a clean handshake ... with gloves on.

But seriously...

Assumptions
Quickstart
Ingesting And Baking Election Data
Config and Settings
Available Fabric Commands
Building A Mac OS Python Dev Environment

Assumptions

You are running OSX.
You are using Python 2.x
You have virtualenv and virtualenvwrapper installed and working.
You have MySQL installed

If any of these are not true, please skip ahead to Building a Mac OS Python dev environment

Quickstart To Get Up And Running

Clone this repo to wherever it is on your machine that you work on your projects
```
git clone git@github.com:SCPR/kpcc_backroom_handshakes.git
```
Change into that directory
```
cd kpcc_backroom_handshakes
```
Create your virtualenv and install the project/application requirements using pip
```
mkvirtualenv kpcc_backroom_handshakes
pip install -r requirements.txt
```
Make a copy of TEMPLATE_development.yml and rename it to development.yml
```
cp TEMPLATE_development.yml development.yml
```
Run fab makesecret and add the output on line 5 of development.yml
```
secret_key: "1=5avBguTW ... "
```

Assuming MySQL is installed in development.yml add username and password for your MySQL install.

database:
  host: "127.0.0.1"
  port: 3306
  database: "kpcc_backroom_handshakes"
  username: "root"
  password: ""

Assuming you have virtualenv and pip installed run fab bootstrap
- This single fabric command uses several functions to scaffold the project by:
  - Creating the database: fab create_db
  - Applying initial Django migrations: fab migrate
  - Load initial data fixtures: fab load_fixtures
  - Create the directory structure for the ballot_box application: fab build_data_dirs
  - Creating the Django superuser: python manage.py createsuperuser
  - Running the Django development server: fab lrun
Navigate to http://127.0.0.1:8000/ and you should arrive at the homepage that shows the elections we have processed using the application so far.

Ingesting And Baking Election Data

Assuming you're up and running successfully, let's attempt to see if we can access data from the most recent election...

In development.yml add the Slack Auth token and API key at line 80 and line 81. KPCC'ers can find this in our password manager program.
While in development.yml add the absolute path to the kpcc_backroom_handshakes/latest_builddirectory

Now you should be able to now run fab fetch_lac from the command line. If everything runs appropriately you should see the following output...

[] Executing task 'fetch_lac'
[localhost] local: python manage.py fetch_lac_results
INFO: manager_lac_results.py (def get_results_file 61):
*** Beginning Request ***
    * Success! http://rrcc.co.la.ca.us/results/2619mar17.ets responded with a file
    * Success! _2017_03_21_15_04_19_lac_2619mar17.ets downloaded
    * Success! _2017_03_21_15_04_19_lac_2619mar17.ets is a valid file
    * Skipping because lac_latest exists
    * Success! los-angeles-county-2017-municipal-primary.txt is ready to parse
    * Success! _2017_03_21_15_04_19_lac_2619mar17.ets is archived
    * Success: los-angeles-county-2017-municipal-primary.txt exists
*** Ending Request ***

INFO: manager_lac_results.py (def parse_results_file 101):
*****
we have new data to save and we'll update timestamps in the database
*****
INFO: manager_lac_results.py (def parse_results_file 119):  we've finished processing lac results

Task finished at 2017-03-21 15:04:27.048844

Done.

Once the script has finished processing, run fab build to bake out views as flat HTML pages that will be pushed to a S3 bucket and used to display results and charts to the people of the world.

Config and Settings

Still very much learning how to configure a collaborative project in different environments, I feel this is a good start to a path that allows for experimentation among collaborators without having to keep too many settings files in sync. Of course, like with all things, we'll learn something new that makes all of this seem silly.

The Files

settings_common.py contains middlware classes and installed apps common to the project.
settings_production.py contains references to Installed Apps, API, database and other configuration variables. config.yml holds configuration variables used in production. development.yml holds configuration variables used in development.
config.yml contains installed apps, API, database and other configuration variables for production
development.yml contains installed apps, API, database and other configuration variables for development
TEMPLATE_development.yml is the template for either config.yml or development.yml configuration variables.

The Variables

We started to add some variables to development.yml. Here's what all of the variables in that file stand for.

debug: Default is True. Set to False when in a production environment. Is echoed and used in variables contained in settings_common.py.
secret_key: Default is "". Use fab makesecret to create value to plug in here. Is used to provide cryptographic signing, and should be set to a unique, unpredictable value. Django will refuse to start if is not set.
internal_ips: Default is an empty list. Stands for a list of IP addresses, as strings, that:
- Allow the debug() context processor to add some variables to the template context.
- Can use the admindocs bookmarklets even if not logged in as a staff user.
- Are marked as "internal" (as opposed to "EXTERNAL") in AdminEmailHandler emails.
site_url: Default is "".
database: A dictionary containing the settings for all databases to be used with Django. It is a nested dictionary whose contents map a database alias to a dictionary containing the options for an individual database. Default engine is mysql.
- host: Default is "". Which host to use when connecting to the database. An empty string means localhost. Not used with SQLite.
- port: Default is "". The port to use when connecting to the database. An empty string means the default port. Not used with SQLite.
- database: Default is "". The name of the database to use. For SQLite, it’s the full path to the database file.
- username: Default is "". The username to use when connecting to the database. Not used with SQLite.
- password: Default is "". The password to use when connecting to the database. Not used with SQLite.
email: optional if you intend to generate emails from an application
- host: Default is localhost. The host to use for sending email.
- user: Default is "". Username to use for the SMTP server defined in EMAIL_HOST. If empty, Django won’t attempt authentication.
- password: Default is "". Password to use for the SMTP server defined in EMAIL_HOST. This setting is used in conjunction with EMAIL_HOST_USER when authenticating to the SMTP server. If either of these settings is empty, Django won’t attempt authentication.
- port: Default is 25. Port to use for the SMTP server defined in EMAIL_HOST.
- use_tls: Default is True. Whether to use a TLS (secure) connection when talking to the SMTP server. This is used for explicit TLS connections, generally on port 587.
installed_apps: Django applications that we've enabled by default out of the box.
- massadmin: Docs and code
- slacker_log_handler: Docs and code
- bakery: Docs and code
- election_registrar: Docs and code
- ballot_box: Docs and code
- newscast: Docs and code
- measure_finance: Docs and code
deployment_env: Variables to aid in automatic deployment of the project and its application.
- hosts: Default is "".
- project_name: Default is "".
- local_branch: Default is "".
- remote_ref: Default is "".
- requirements_file: Default is "".
- use_ssh_config: Default is False.
- code_dir: Default is "".
build: Variabes for the django-bakery python library that allows us to serve up views as flat HTML files that are pushed to an Amazon S3 bucket. Docs on settings variables are here.
- aws_bucket_name: Default is "". The name of the Amazon S3 "bucket" were you want to publish the flat files in your local BUILD_DIR. KPCC's is documented elsewhere.
- aws_access_key_id: Default is "". A part of your secret Amazon Web Services credentials. Necessary to upload files to S3. KPCC's is documented elsewhere.
- aws_secret_access_key: Default is "". A part of your secret Amazon Web Services credentials. Necessary to upload files to S3. KPCC's is documented elsewhere.
- aws_s3_host: Default is s3-accelerate.amazonaws.com, Amazon's accelerated upload service
- bakery_gzip: Default is True Opt in to automatic gzipping of your files in the build method and addition of the required headers when deploying to Amazon S3.
- build_dir: Absolute path to the location of the kpcc_backroom_handshakes/latest_build directory which is the location of where you want the flat files to be built.
- views: The list of views you want to be built out as flat files when the build management command is executed. The following are the views we've enabled by default:
  - "election_registrar.views.ElectionDetailView"
  - "ballot_box.views.BakedHomepageIndex"
  - "ballot_box.views.BakedFeaturedIndex"
  - "ballot_box.views.BakedResultsIndex"
  - "ballot_box.views.BakedEmbeddedDetail"
  - "measure_finance.views.InitialDetailView"
- bakery_cache_control: Set cache-control headers based on content type. Headers are set using the max-age= format so the passed values should be in seconds ('text/html': 900 would result in a Cache-Control: max-age=900 header for all text/html files). By default we set for html and javascript.
  - html: 300
  - javascript: 86400
- static_to_ignore: Default are "admin", "bootstrap", "bootstrap-rtl", "yaml"
api: api settings and keys for applications housed in the project.
- slack: Used for the slacker_log_handler module that posts data from project jobs to a specific channel in KPCC's Slack account.
  - token: Default is "". KPCC's token is documented elsewhere.
  - api_key: Default is "". KPCC's api key is documented elsewhere.
- propublica: Optional api key is using propublica's data api
  - api_key: Default is "".
- maplight: Optional api key is using maplight's data api
  - api_key: Default is "". KPCC's api key from the 2016 election cycle is documented elsewhere.
- headers: HTTP headers added to a request
  - from: String that is passed through URL requests made using the Requests library.
  - user_agent: String that is passed through URL requests made using the Requests library. Default is "Mozilla/5.0 (Windows; U; Windows NT 6.0; en-US) AppleWebKit/525.19 (KHTML, like Gecko) Chrome/1.0.154.53 Safari/525.19"

Available Fabric Commands

Functions That Fetch Our Election Data From Our Sources

election_night: run this on election night to fetch data from live elections and build out pages...
```
local("python manage.py election_night")
```
fetch_sos: shortcut for running the management command to fetch election results from the california secretary of state
```
local("python manage.py fetch_sos_results")
```
fetch_lac: shortcut for running the management command to fetch election results from the los angeles county
```
local("python manage.py fetch_lac_results")
```
fetch_sbc: shortcut for running the management command to fetch election results from the san bernardino county
```
local("python manage.py fetch_sbc_results")
```
fetch_oc: shortcut for running the management command to fetch election results from the orange county
```
local("python manage.py fetch_oc_results")
```

fetch_all: shortcut for fetch results from all four primary sources

local("python manage.py fetch_sos_results")
local("python manage.py fetch_oc_results")
local("python manage.py fetch_lac_results")
local("python manage.py fetch_sbc_results")

fetch_maplight: shortcut for fetch data from the maplight campaign finance api but it's specific to the general-2016-11-08 election.
```
local("python manage.py fetch_measure_finance")
```

Dump And Load Existing Election Data

dump_ballot_box: shortcut to dump data from ballot box as fixtures

local("python manage.py dumpdata ballot_box > ballot_box/fixtures/ballot_box.json")

load_ballot_box: shortcut to load ballot box data fixtures

local("python manage.py loaddata ballot_box/fixtures/ballot_box.json")

dump_registrar: shortcut to dump data from ballot box as fixtures

local("python manage.py dumpdata election_registrar > election_registrar/fixtures/election_registrar.json")

load_registrar: shortcut to load ballot box data fixtures

local("python manage.py loaddata election_registrar/fixtures/election_registrar.json")

dump_playlist: shortcut to dump data from ballot box as fixtures

local("python manage.py dumpdata newscast > newscast/fixtures/newscast-playlist.json")

load_playlist: shortcut to dump data from ballot box as fixtures

local("python manage.py loaddata newscast/fixtures/newscast-playlist.json")

dump_maplight: shortcut to dump data from ballot box as fixtures

local("python manage.py dumpdata measure_finance > measure_finance/fixtures/measure_finance.json")

load_maplight:shortcut to dump data from ballot box as fixtures

local("python manage.py loaddata measure_finance/fixtures/measure_finance.json")

dump_fixtures: shortcut to dump all data fixtures with logging

dump_registrar()
dump_ballot_box()
dump_playlist()
dump_maplight()

load_fixtures: shortcut to load all data fixtures with logging

load_registrar()
load_ballot_box()
load_playlist()
load_maplight()

django-bakery functions

build: Activates the django-bakery script to build the views specified in config.yml or development.yml
```
local("python manage.py build")
```
buildserver: Activates the django-bakery development server
```
local("python manage.py buildserver")
```
publish: Publishes views and static files in the build directory to Amazon S3
```
local("python manage.py publish")
```

Development Functions

lrun: shortcut for base manage.py function to run the dev server
```
local("python manage.py runserver")
```
make: shortcut for base manage.py function to sync the dev database
```
local("python manage.py makemigrations")
```
migrate: shortcut for base manage.py function to apply db migrations
```
local("python manage.py migrate")
```
test: shortcut for base manage.py function to create a superuser
```
local("python manage.py test")
```

set_featured:

local("python manage.py set_featured_contests")

zero_it:
```
local("python manage.py zero_out_data")
```

Bootstrapping Functions

requirements(): shortcut to install requirements from repository's requirements.txt
```
local("pip install -r requirements.txt")
```
superuser: shortcut for base manage.py function to create a superuser
```
local("python manage.py createsuperuser")
```
create_db:
makesecret: generates secret key for use in django settings
bootstrap: run tasks to setup the base project
syncstart: get in sync quickly when collaborating
```
requirements()
migrate()
load_fixtures()
```
syncend: end a working session and dump out potential changes to requirements and data
```
local("pip freeze > requirements.txt")
make()
dump_fixtures()
```

Building A Mac OS Python Dev Environment

Assumming homebrew is installed...

Install homebrew python

cd /System/Library/Frameworks/Python.framework/Versions
sudo rm Current
brew install python
brew doctor
which python
which pip
pip install --upgrade setuptools
pip install --upgrade distribute
pip install virtualenv
pip install virtualenvwrapper
python --version
source /usr/local/bin/virtualenvwrapper.sh
sudo ln -s /usr/local/Cellar/python/2.7.8_2 /System/Library/Frameworks/Python.framework/Versions/Current

Configure $PATH variables for python, virtualenv

# homebrew path
export PATH="/usr/local/bin:$PATH"

# virtualenvwrapper settings
export WORKON_HOME=$HOME/.virtualenvs
export PIP_VIRTUALENV_BASE=$WORKON_HOME
export PIP_RESPECT_VIRTUALENV=true
source /usr/local/bin/virtualenvwrapper.sh

Install MySQL via homebrew

brew remove mysql
brew cleanup
launchctl unload -w ~/Library/LaunchAgents/homebrew.mxcl.mysql.plist
rm ~/Library/LaunchAgents/homebrew.mxcl.mysql.plist
sudo rm -rf /usr/local/var/mysql
brew install mysql
ln -sfv /usr/local/opt/mysql/*.plist ~/Library/LaunchAgents

Getting mysql up and running

mysql.server start
mysql_secure_installation
mysql -u root -p
SHOW DATABASES;
SET default_storage_engine=MYISAM;

Docker Setup

Or, you can use Docker.

Prerequisites

Instructions

A Dockerfile is included, which will build the entire environment into a Docker image. This includes Python and dependencies needed for building requirements from source.

To build the image, run docker-compose up -d backroom-handshakes.

The build step can take several minutes, but you should only have to do it once.

If you don't have an instance of MySQL or MariaDB running already, docker-compose.yml includes a minimal MySQL setup. Just run docker-compose up -d mysql, and you should now have a development instance ready to use.

Open up your a shell for your container:

docker-compose exec backroom-handshakes /bin/sh

Now, within that shell, you can continue to install your required modules:

pip install -r requirements.txt

At this point, you can continue with the "quickstart" instructions listed above. The only difference is, if you decided to run the MySQL container, you should change the database host in development.yml to this:

  database:
    host: mysql
    port: 3306
    database: kpcc_backroom_handshakes
    username: root
    password: password

Notes

Remmber that build.build_dir in development.yml must be an absolute path, not relative. For development, simply setting this configuration to /home/latest_build works fine.

SCPR / kpcc_backroom_handshakes

readme