search

OpenBudget / budgetkey-data-pipelines

Budget Key data processing pipelines
8 stars 29 forks source link

readme

budgetkey-data-pipelines

Build Status

Budget Key data processing pipelines

What are we doing here?

The heart of the BudgetKey project is its rich, up-to-date quality data collection. Data is collected from over 20 different data sources, cleaned, normalised, validated, combined and analysed - to create the most extensive repository of fiscal data in Israel.

In order to get that data, we have an extensive set of downloaders and scrapers which get the data from government publications and other web-sites. The fetched data is then processed and combined, and eventually saved to disk (so that people can download the raw data without hassle), loaded to a relational database (so that analysts can do in-depths queries of the data) and pushed to a key-store value (elasticsearch) which serves our main website (obudget.org).

The frameworks we're using to accomplish all of this are called dataflows and datapackage-pipelines. These frameworks allow us to write simple 'pipelines', each consisting of a set of predefined processing steps. Most of the pipelines use of a set of common building-blocks, and some custom processors - mainly custom scrapers for exotic sources.

dataflows is used for writing processing flows individual sources.

datapackage-pipelines runs the flows, combines the results and creates aggregated datasets.

Quickstart on dataflows

The recommended way to start is by reading the README of dataflowshere - and then follow on to the interactive tutorial on Binder (you'll find the link there).

Then, try to write a very simple pipeline - just to test your understanding. A good task for that would be:

Working on issues (for newcomers)

Most issues you'd be working on will require you to write a 'Flow' to perform some scraping, analysing or data integration.

  1. The first step will always be to write that flow and make sure it works.
  2. Then, we would need to integrate it with the rest of the pipelines and perhaps with other, existing datasets.

Finish step (1) before starting step (2) - the entire system is quite complex but you shouldn't worry about understanding everything at first. Consult with Adam when (1) is done for guidance and support...

Quickstart on datapackage-pipelines

The recommended way to start is by reading the README of datapackage-pipelineshere- it's a bit long, so at least read the beginning and skim the rest.

I want to help - how?

As you can see, pipelines are sorted by domain and data source.

Higher up the tree, we find pipelines that aggregate different sources (e.g. companies + associations + ... -> all-entities). Finally, under the budgetkey directory we have pipelines that process and store the data for displaying on the website.

note: To understand a bit more on the difference between the different types of government spending, please read this excellent blog post.

What's currenty running?

To see what's the current processing status of each pipeline, just hop to the dashboard.

Developing a new pipeline

Installation

Installing Python 3.6+

We recommend using pyenv for managing your installed python versions.

On Ubuntu, use these commands:

sudo apt-get install git python-pip make build-essential libssl-dev zlib1g-dev libbz2-dev libreadline-dev libsqlite3-dev
sudo pip install virtualenvwrapper

git clone https://github.com/yyuu/pyenv.git ~/.pyenv
git clone https://github.com/yyuu/pyenv-virtualenvwrapper.git ~/.pyenv/plugins/pyenv-virtualenvwrapper

echo 'export PYENV_ROOT="$HOME/.pyenv"' >> ~/.bashrc
echo 'export PATH="$PYENV_ROOT/bin:$PATH"' >> ~/.bashrc
echo 'eval "$(pyenv init -)"' >> ~/.bashrc
echo 'pyenv virtualenvwrapper' >> ~/.bashrc

exec $SHELL

On OSX, you can run

brew install pyenv
echo 'eval "$(pyenv init -)"' >> ~/.bash_profile

After installation, running:

pyenv install 3.6.1
pyenv global 3.6.1

Will set your Python version to 3.6.1

Installing dataflows

$ pip install dataflows

Installation of the Package (if you want to run the system - not needed for a single pipeline development)

$ sudo apt-get install build-essential python3-dev libxml2-dev libxslt1-dev libleveldb-dev
$ python --version
Python 3.6.0+
$ sudo mkdir -p /var/datapackages && sudo chown $USER /var/datapackages/
$ make install
$ budgetkey-dpp
INFO    :Main                            :Skipping redis connection, host:None, port:6379
Available Pipelines:
- ./budget/national/changes/original/national-budget-changes
...

Running a Pipeline (if you want to run the system - not needed for a single pipeline development)

$ budgetkey-dpp run ./entities/companies/registrar/registry

The following files will be created:

Writing Tests

unit tests

$ make test
run a specific test / modify test arguments

any arguments added to tox will be added to the underlying py.test command

$ tox tests/tenders/test_fixtures.py

tox can be a bit slow, especially when doing tdd

to run tests faster you can run py.test directly, but you will need to setup the test environment first

$ pip install pytest
$ py.test tests/tenders/test_fixtures.py -svk test_tenders_fixtures_publishers

Using Docker Compose

Docker Compose can be used to run a full environment with all required services - similar to the production environment.

Installation

Loading datapackages to Elasticsearch

This method allows to load the prepared datapackages to elasticsearch, the data is then available for exploration via Kibana

This snippet will delete all local docker-compose volumes - so make sure you don't have anything important there beforehand..

It loads the first 100 rows from each pipeline, you can modify ES_LIMIT_ROWS below or remove it to load all data

docker-compose down -v && docker-compose pull elasticsearch db && docker-compose up -d elasticsearch db
export DPP_DB_ENGINE="postgresql://postgres:123456@localhost:15432/postgres"
export DPP_ELASTICSEARCH="localhost:19200"
for doctype in `budgetkey-dpp | grep .budgetkey/elasticsearch/index_ | cut -d"_" -f2 - | cut -d" " -f1 -`; do
    echo " > Loading ${doctype}"
    ES_LOAD_FROM_URL=1 ES_LIMIT_ROWS=100 budgetkey-dpp run ./budgetkey/elasticsearch/index_$doctype
done

Now you can start Kibana to explore the data

docker-compose up -d kibana

Kibana should be available at http://localhost:15601/ (It might take some time to start up properly)

Index name is budgetkey