If you just want to see how a dataset is harvested by CDE:
conda create -n cde python=3.10;conda activate cde
pip install -e ./harvester
python -m cde_harvester --urls https://data.cioospacific.ca/erddap --dataset_ids ECCC_MSC_BUOYS
harvest
folderdocker-compose
docker-compose -f docker-compose.production.yaml up -d
to start all services. This will take a few minute to download, build, create the database schema.conda create -n cde python=3.10;conda activate cde
pip install -e ./harvester -e ./db-loader
sh data_loader_test.sh
to just load one dataset for testing purposesTo run CDE locally, you will need Docker, Python and Node and a few terminal windows
Rename .env.sample from the root directory to .env and change any settings if needed. If you are running on your local machine these settings don't need to change
Start a local database using docker
:
docker-compose up -d db
Setup Python virtual env and install Python modules:
conda create -n cde python=3.10
conda activate cde
pip install -e ./downloader -e ./download_scheduler -e ./harvester -e ./db-loader
Start the API:
cd web-api
npm install
npm start
Start the download scheduler:
python -m download_scheduler
Start the frontend:
cd frontend
npm install
npm start
Harvest a single dataset and load CKAN data.
sh data_loader_test.sh
See website at http://localhost:8000
See which cde services are running:
docker-compose ps
Start all containers, the first time this runs it will build containers:
docker-compose up -d
Tail logs:
docker-compose logs -f
(Re/)Build and (re/)start all containers that have code changes:
docker-compose -f docker-compose.production.yaml up -d --build
Rebuild database: (this will erase all your data)
docker-compose stop db
docker volume rm cde_postgres-data
docker-compose up -d db
Redis CLI:
docker exec -it cde_redis_1 redis-cli
Flush redis tile cache:
docker exec -it cde_redis_1 redis-cli FLUSHALL
From the production server,
rename .env.sample
to production.env
and configure.
Delete old redis and postgres data (if needed):
sudo docker volume rm cde_postgres-data cde_redis-data
Start all services:
sudo docker-compose -f docker-compose.production.yaml up -d --build
Harvest data:
conda create -n cde python=3.10
conda activate cde
pip install -e ./harvester -e ./db-loader
sh data_loader.sh
Add a crontab entry for the scheduler to run nightly.
deploy frontend to Gitpages
API_URL=https://explore.cioos.ca/api npm run deploy