search

mjanez / ckan-docker

Custom CKAN Docker Compose Deployment
https://mjanez.github.io/ckan-docs
0 stars 5 forks source link

Feature - Separate containers for harvest (gather/consumer) and xloader #101

Open mjanez opened 9 months ago

mjanez commented 9 months ago

Info

Background

First a deployment was developed with all workers in the background in the main CKAN container, but in order to handle high availability and scalability, it is proposed to separate the workers into isolated containers, connected to each other.

More info: https://github.com/mjanez/ckan-docker/issues/102

To do

To allow horizontal scalability, it would be better to use workers in separate containers rather than as background cron jobs within the `ckan' container.

They communicate with the ckan container via redis, and need access to DB, solr and filesystem (config, storage, logs, etc.)

References:

Example

  gather_consumer:
    << : *default-common-ckan
    image: ${COMPOSE_PROJECT_NAME}-dcatapit_ckan:latest
    container_name: ${COMPOSE_PROJECT_NAME}-gather
    depends_on:
      ckan:
        condition: service_healthy
    entrypoint: /consumer-entrypoint.sh
    volumes:
      - ckan-data:/var/lib/ckan
      - ckan-config:/etc/ckan
      - ckan-logs:/var/log/ckan
      - ./docker/ckan/ckan_harvesting_gather.conf:/etc/supervisor/conf.d/ckan_harvesting.conf

  fetch_consumer:
    << : *default-common-ckan
    image: ${COMPOSE_PROJECT_NAME}-dcatapit_ckan:latest
    container_name: ${COMPOSE_PROJECT_NAME}-fetch
    depends_on:
      ckan:
        condition: service_healthy
    entrypoint: /consumer-entrypoint.sh
    volumes:
      - ckan-data:/var/lib/ckan
      - ckan-config:/etc/ckan
      - ckan-logs:/var/log/ckan
      - ./docker/ckan/ckan_harvesting_fetch.conf:/etc/supervisor/conf.d/ckan_harvesting.conf
mjanez commented 1 month ago

Check kartoza/ckanext-dalrrd_emc_dcpr solution: