hashicorp / nomad

Nomad is an easy-to-use, flexible, and performant workload orchestrator that can deploy a mix of microservice, batch, containerized, and non-containerized applications. Nomad is easy to operate and scale and has native Consul and Vault integrations.
https://www.nomadproject.io/
Other
14.81k stars 1.94k forks source link

A periodic task was not scheduled and was not ran #23402

Closed EugenKon closed 3 months ago

EugenKon commented 3 months ago

Nomad version

Operating system and Environment details

Issue

The "periodic" task was not scheduled and was not run.

image

Actually nothing was run for the last day. You can see here only "Force launch" attempts: image

Reproduction steps

nomad job plan postgres-backup.hcl
nomad job run -check-index 17404 postgres-backup.hcl

Expected Result

The task should run periodically every 4 hours.

Actual Result

The task was not ran.

Job file

job "backup-database" {
  region = "co"
  datacenters = ["dc"]
  type = "batch"

  periodic {
    enabled   = true
    crons     = [ "* * */4 * * * *" ]
    time_zone = "America/Toronto"
    prohibit_overlap = true
  }

  # https://awesomeprogrammer.com/blog/2022/06/05/how-to-backup-postgres-database-with-nomad/
  group "db-backup" {
    task "postgres-backup" {
      driver = "raw_exec"

      config {
        command = "/bin/bash"
        args    = ["local/script.sh"]
      }

      ## !!! It would be nice to be able write a constraint that can target "meta" in the different task
      # constraint {
      #   attribute = "${meta.service.tag}" XXXX
      #   attribute = "${meta.database}"    XXXX
      #   operator  = "="
      #   value     = "postgres"
      # }

      env {
        NOMAD_TOKEN = "****-****-****"
      }

      template {
        destination = "local/script.sh"
        data        = <<-EOH
          set -ex

          env

          docker ps

          if [ -z "${DB_ALLOC_ID}" ]; then echo "We do not have a task with a database"; exit 1; fi

          nomad alloc exec -task postgres-task $DB_ALLOC_ID is_ready 120  || {
            docker run --rm planitar/wi-api:${WI_DOCKER_TAG}
              wi-cli devnotify error "Postgres backup FAILED (not ready)";
            exit 1;
          };
          nomad alloc exec -task postgres-task $DB_ALLOC_ID backup  || {
            docker run --rm planitar/wi-api:${WI_DOCKER_TAG}
              wi-cli devnotify error "Postgres backup FAILED";
            exit 1;
          };
          docker run --rm planitar/wi-api:${WI_DOCKER_TAG}
            wi-cli devnotify info "Postgres backup succeeded";

          echo "Backup finished";
        EOH
      }

      template {
        # https://developer.hashicorp.com/nomad/docs/job-specification/template#env
        # This option specifies to read the template back in as environment variables for the task.
        env         = true
        destination = "secrets/file.env"
        data = <<-EOH
          # As service 'postgres-node' is registered in Consul, we want to grab its 'alloc' tag
          {{- range $tag, $services := service "postgres-node" | byTag -}}
            {{- if $tag | contains "alloc" -}}
              {{- $dbAllocId := index ($tag | split "=") 1}}
              DB_ALLOC_ID="{{ $dbAllocId }}"
            {{- end -}}
          {{- end }}

          WI_DOCKER_TAG="{{key "project_name"}}"
        EOH
      }
    }
  }
}

Another job:

job "portal" {
  group "database" {
    task "postgres-task" {
      meta {
        database = "postgres"
      }
      ...
    }
  }
}
tgross commented 3 months ago

@EugenKon I think you need to adjust your cron expression. * * */4 * * * * says that the job should run once a minute every 4 days. DDG has a great expression parser for that: https://duckduckgo.com/?q=cron+*+*+*%2F4+*+*+*+*&ia=answer