Miami-Dade Councilmatic

Keep track of what Miami-Dade County Council is doing.

:red_circle: Note: this project has a URL in a staging environment, but has no regular maintainer. If you are interested in continuing development, feel free to submit pull requests and/or contact a Code for Miami tech team member for additional information.

Setup

Install OS level dependencies:

Python 3.4 - Heads up: many OSX machines will use Python 2.x by default.
PostgreSQL 9.4 +

Install app requirements

We recommend using virtualenv and virtualenvwrapper for working in a virtualized development environment. Read how to set up virtualenv.

Once you have virtualenvwrapper set up,

brew install python3
which python3
mkvirtualenv --python=/usr/local/bin/python3.4 mia-councilmatic
workon mia-councilmatic
git clone https://github.com/Code-for-Miami/mia-councilmatic.git
cd mia-councilmatic
pip install -r requirements.txt

For Linux systems Install these packages:

python3.4
postgresql
virtualenv
virtualenvwrapper
postgresql-server-dev-9.4 (or if you're running a version other than 9.4, choose that package)

Then run:

mkvirtualenv --python=/usr/bin/python3.4 mia-councilmatic
workon mia-councilmatic
git clone https://github.com/Code-for-Miami/mia-councilmatic.git
cd mia-councilmatic
pip install -r requirements.txt

Afterwards, whenever you want to use this virtual environment to work on mia-councilmatic, run workon mia-councilmatic. To exit the virtual environment, run deactivate.

OPTIONAL: install django-councilmatic locally
If you plan on making changes to core councilmatic features (as opposed to Miami-specific stuff), you'll want to install django-councilmatic locally instead of installing from pypi.

cd ..
git clone https://github.com/datamade/django-councilmatic.git
cd django-councilmatic
python setup.py develop
cd ../mia-councilmatic/

Create your settings file

cp councilmatic/settings_deployment.py.example councilmatic/settings_deployment.py

Then edit councilmatic/settings_deployment.py:

DATABASES['default']['USER'] should be your PostgreSQL username
DATABASES['default']['NAME'] should be 'mia_councilmatic'
if you're setting up councilmatic for local development, use a dummy cache by setting CACHES['default']['BACKEND'] to 'django.core.cache.backends.dummy.DummyCache'. if you're deploying, leave it as is

Setup your database

Before we can run the website, we need to create a database. If you're setting up a development environment on your personal OSX computer, use an app such as Postgres.app, making sure the app is running.

createdb mia_councilmatic

If you use Linux and get a "Role does not exist" error, run

createdb mia_councilmatic -u YOUR_POSTGRESQL_USERNAME

Then, run migrations

python manage.py migrate --no-initial-data

Create an admin user - set a username & password when prompted

python manage.py createsuperuser

Importing data from the open civic data api

Run the loaddata management command. This will take a while, depending on volume (Some cities have ~70k bills, & taking ~1 hour to load the data).

python manage.py loaddata

By default, the loaddata command is smart about what it looks at on the OCD API. If you already have bills loaded, it won't look at everything on the API - it'll look at the most recently updated bill in your database, see when that bill was last updated on the OCD API, & then look through everything on the API that was updated after that point. If you'd like to load things that are older than what you currently have loaded, you can run the loaddata management command with a --delete option, which removes everything from your database before loading.

The loaddata command has some more nuance than the description above, for the different types of data it loads. If you have any questions, open up an issue and pester us to write better documentation.

Running Chicago Councilmatic locally

python manage.py runserver

navigate to http://localhost:8000/

Setup Search

Install Open JDK or update Java

On Ubuntu:

$ sudo apt-get update
$ sudo apt-get install openjdk-7-jre-headless

On OS X:

Download latest Java from http://java.com/en/download/mac_download.jsp?locale=en
Follow normal install procedure

Change system Java to use the version you just installed:

sudo mv /usr/bin/java /usr/bin/java16
sudo ln -s /Library/Internet\ Plug-Ins/JavaAppletPlugin.plugin/Contents/Home/bin/java /usr/bin/java

Download & setup Solr

wget http://mirror.sdunix.com/apache/lucene/solr/4.10.4/solr-4.10.4.tgz
tar -xvf solr-4.10.4.tgz
sudo cp -R solr-4.10.4/example /opt/solr

# Copy schema.xml for this app to solr directory
sudo cp solr_scripts/schema.xml /opt/solr/example/solr/collection1/conf/schema.xml

Run Solr

# Next, start the java application that runs solr
# Do this in another terminal window & keep it running
# If you see error output, somethings wrong
cd /opt/solr/example
sudo java -jar start.jar

Index the database

# back in the chi-councilmatic directory:
python manage.py rebuild_index

OPTIONAL: Install and configure Jetty for Solr

Just running Solr as described above is probably OK in a development setting. To deploy Solr in production, you'll want to use something like Jetty. Here's how you'd do that on Ubuntu:

sudo apt-get install jetty

# Backup stock init.d script
sudo mv /etc/init.d/jetty ~/jetty.orig

# Get init.d script suggested by Solr docs
sudo cp solr_scripts/jetty.sh /etc/init.d/jetty
sudo chown root.root /etc/init.d/jetty
sudo chmod 755 /etc/init.d/jetty

# Add Solr specific configs to /etc/default/jetty
sudo cp solr_scripts/jetty.conf /etc/default/jetty

# Change ownership of the Solr directory so Jetty can get at it
sudo chown -R jetty.jetty /opt/solr

# Start up Solr
sudo service jetty start

# Solr should now be running on port 8983

Regenerate Solr schema

While developing, if you need to make changes to the fields that are getting indexed or how they are getting indexed, you'll need to regenerate the schema.xml file that Solr uses to make it's magic. Here's how that works:

python manage.py build_solr_schema > solr_scripts/schema.xml
cp solr_scripts/schema.xml /opt/solr/solr/collection1/conf/schema.xml

In order for Solr to use the new schema file, you'll need to restart it.

Using Solr for more than one Councilmatic on the same server

If you intend to run more than one instance of Councilmatic on the same server, you'll need to take a look at this README to make sure you're configuring things properly.

Team

David Moore - project manager
Forest Gregg - Open Civic Data (OCD) and Legistar scraping
Cathy Deng - data models and loading
Derek Eder - front end
Eric van Zanten - search and dev ops

Errors / Bugs

If something is not behaving intuitively, it is a bug, and should be reported. Report it here: https://github.com/datamade/chi-councilmatic/issues

Note on Patches/Pull Requests

Fork the project.
Make your feature addition or bug fix.
Commit, do not mess with rakefile, version, or history.
Send a pull request. Bonus points for topic branches.

CodeforSouth / mia-councilmatic

readme