HakaiInstitute / cde

https://explore.cioos.ca
0 stars 1 forks source link

CDE - CIOOS Exploration and Data Discovery

Test

Testing a dataset

If you just want to see how a dataset is harvested by CDE:

  1. Start your python environment environment, conda create -n cde python=3.10;conda activate cde
  2. pip install -e ./harvester
  3. python -m cde_harvester --urls https://data.cioospacific.ca/erddap --dataset_ids ECCC_MSC_BUOYS
  4. See files in harvest folder

Starting using docker

  1. Install Docker and Docker compose. New versions of Docker include docker-compose
  2. Rename .env.sample to .env and change any settings if needed. If you are running on your local machine these settings don't need to change
  3. docker-compose -f docker-compose.production.yaml up -d to start all services. This will take a few minute to download, build, create the database schema.
  4. Start your python 3.10 or greater environment, eg conda create -n cde python=3.10;conda activate cde
  5. pip install -e ./harvester -e ./db-loader
  6. Run harvester to load data. From this directory, run: sh data_loader_test.sh to just load one dataset for testing purposes
  7. See website at http://localhost:8098

Development

Handy docker commands

See which cde services are running: docker-compose ps

Start all containers, the first time this runs it will build containers: docker-compose up -d

Tail logs: docker-compose logs -f

(Re/)Build and (re/)start all containers that have code changes: docker-compose -f docker-compose.production.yaml up -d --build

Rebuild database: (this will erase all your data)

docker-compose stop db
docker volume rm cde_postgres-data
docker-compose up -d db

Redis CLI: docker exec -it cde_redis_1 redis-cli

Flush redis tile cache: docker exec -it cde_redis_1 redis-cli FLUSHALL

Production deployment

From the production server,