search

terraref / documentation

TERRAref documentation
https://docs.terraref.org
6 stars 7 forks source link

How to import trait database #153

Closed dlebauer closed 6 years ago

dlebauer commented 7 years ago

Based on https://github.com/terraref/reference-data/issues/111 and https://github.com/terraref/computing-pipeline/issues/302

Changes to load.bety.sh described here: https://github.com/PecanProject/pecan/issues/1406

How to import terraref data:

This is using the load.bety.sh script that has been updated by @robkooper and is currently on a branch here:

https://raw.githubusercontent.com/PecanProject/pecan/357b70e21592375827e9c55bc5e1ce8e3426bd78/scripts/load.bety.sh

Initial import

load.bety.sh -c -u -m 99 -r 0 -w https://terraref.ncsa.illinois.edu/bety/dump/bety0/bety.tar.gz
load.bety.sh -m 99 -r 6 -w https://terraref.ncsa.illinois.edu/bety/dump/bety6/bety.tar.gz

Scheduled sync

Now you should be able to setup cron job to periodically (we often use hourly or daily) import the data using the same as above, except remoe the -c flag from the first line :

load.bety.sh -m 99 -r 0 -w https://terraref.ncsa.illinois.edu/bety/dump/bety0/bety.tar.gz
load.bety.sh -m 99 -r 6 -w https://terraref.ncsa.illinois.edu/bety/dump/bety6/bety.tar.gz

Note that this will overwrite any changes that you have made to rows with id fields between 1-1 billion and 6 billion+1 to 7 billion.

dlebauer commented 7 years ago

To run a docker image w/ data. Contains daily import via cron:

# -p makes port visible to localhost
docker run --name betydb -p 5432:5432 terraref/betydb-postgis

## to run bash within container:
docker exec -it betydb bash

(note if deploying bety web application, use )

docker run --name betydb -p 5432:5432 pecan/bety
craig-willis commented 6 years ago

@dlebauer Should this move into the new sprint?

dlebauer commented 6 years ago

Yep

max-zilla commented 6 years ago

there is a PR to add this to documentation.