mcuadros / ofelia

A docker job scheduler (aka. crontab for docker)
MIT License
2.96k stars 174 forks source link

No overlap between jobs #285

Open wuodar opened 2 months ago

wuodar commented 2 months ago

Currently, one can set no-overlap to true, to avoid two concurrent executions. But is it possible to avoid overlapping between services? E.g.

version: '3.7'

services:
  feature_pipeline:
    image: feature_pipeline
    container_name: feature_pipeline
    restart: always
    user: user
    build:
      context: .
      dockerfile: Dockerfile
    volumes:
      - ${DATA_VOLUME}:/home/user/app/data
    deploy:
        resources:
            limits:
              cpus: ${MAX_CPUS}
              memory: ${MAX_MEMORY}
            reservations:
              cpus: ${MIN_CPUS}
              memory: ${MIN_MEMORY}
    labels:
      ofelia.enabled: "true"
      ofelia.job-exec.feature_pipeline.schedule: ${PIPELINE_SCHEDULE}
      ofelia.job-exec.feature_pipeline.command: "python src/feature_pipeline.py"
      ofelia.job-exec.feature_pipeline.no-overlap: "true"

  inference_pipeline:
    image: inference_pipeline
    container_name: inference_pipeline
    restart: always
    user: user
    build:
      context: .
      dockerfile: Dockerfile
    volumes:
      - ${DATA_VOLUME}:/home/user/app/data
    deploy:
        resources:
            limits:
              cpus: ${MAX_CPUS}
              memory: ${MAX_MEMORY}
            reservations:
              cpus: ${MIN_CPUS}
              memory: ${MIN_MEMORY}
    labels:
      ofelia.enabled: "true"
      ofelia.job-exec.inference_pipeline.schedule: ${PIPELINE_SCHEDULE}
      ofelia.job-exec.inference_pipeline.command: "python src/inference_pipeline.py"
      ofelia.job-exec.inference_pipeline.no-overlap: "true"
      ofelia.job-exec.feature_pipeline.no-overlap: "true"

  ofelia:
    image: mcuadros/ofelia:latest
    restart: always
    depends_on:
      - feature_pipeline
    command: daemon --docker
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock:ro

In this example, I'd like to set ofelia.job-exec.feature_pipeline.no-overlap: "true" but not only for feature_pipeline service, but also avoid running concurrently feature_pipeline and inference_pipeline.

taraspos commented 1 month ago

No, that's not something Ofelia supports at the moment.