Popgetter is a convenience tool for downloading census data from a number of different jurisdictions and coercing the data into common formats. The aim is that city or region scale analysis can be easily replicated for different geographies, using the most detailed, locally available data.
Popgetter DOES:
For each of the implemented countries:
Popgetter WILL
Popgetter DOES NOT:
At present, this is still a development project, so the first step is to clone
the repo and then install using pip, with the --editable
option:
conda
or venv
, but so far
pyenv
is the most tested with popgetter). eg:python -m venv popgetter_venv # create a virtual environment called `popgetter_venv`
source popgetter_venv/bin/activate # activate the virtual environment
git clone https://github.com/Urban-Analytics-Technology-Platform/popgetter.git
cd popgetter
pip install -e ".[dev]"
dagster dev
Open http://localhost:3000 with your browser to see the project.
You can start writing assets in popgetter/assets/
directory. New assets and
jobs will need to be added to the popgetter/__init__.py
file.
You can specify new Python dependencies in pyproject.toml
.
Tests are in the popgetter_tests
directory and you can run tests using
pytest
:
pytest
This is a Dagster project. The repo layout was initially
created with the
dagster project scaffold
command. It has been subsequently updated using the
copier command and the
Scientific Python template.
There is code, which predates the migration to Dagster in the previous_code
directory. In due course, this will be removed as the remaining countries are
migrated to Dagster. There are usage instructions for this old code in
previous_code/previous_code_usage.md
.