Scripts to download the data from the http://registry.threesixtygiving.org file registry
Install dependencies:
The datagetter is tested to run on python3.8
# Create a new python3 virtual environment
$ virtualenv --python python3 .ve
$ source .ve/bin/activate
# Install datagetter dependencies
$ pip install -r requirements.txt
If you want to install the datagetter on your environment (rather than run it from the source directory)
# see setup.py --help for more options
$ setup.py install
Basic usage to download all published 360Giving data run:
$ datagetter.py
$ datagetter.py --help
usage: datagetter.py [-h] [--no-download] [--local-registry LOCAL_REGISTRY] [--no-convert] [--no-convert-big-files] [--no-validate] [--data-dir DATA_DIR] [--threads THREADS] [--socks5 SOCKS5_PROXY] [--limit-downloads LIMIT_DOWNLOADS] [--schema-branch SCHEMA_BRANCH]
[--publishers PUBLISHER_PREFIXES [PUBLISHER_PREFIXES ...]]
optional arguments:
-h, --help show this help message and exit
--no-download Don't download any files only convert existing data
--local-registry LOCAL_REGISTRY
Use a local registry file rather than registry.threesixtygiving.org
--no-convert
--no-convert-big-files
--no-validate
--data-dir DATA_DIR
--threads THREADS
--socks5 SOCKS5_PROXY
Use a socks5 proxy to fetch publisher data. Example --socks5=socks5://host:port
--limit-downloads LIMIT_DOWNLOADS
Limit the number of file downloads
--schema-branch SCHEMA_BRANCH
Specify a git branch of the 360Giving schema
--publishers PUBLISHER_PREFIXES [PUBLISHER_PREFIXES ...]
Only download for selected publishers
If you are updating requirements.txt
please make sure you use version 3.8 of Python.