Periodic jobs for system scheduler

[kak-tus]

Now (0.5.0rc1) nomad not support periodic jobs for system scheduler.

* Periodic can only be used with "batch" scheduler

I think, that it is normal use case that someone want to do periodic job on each host. Yes, I can use system scheduler with docker container with crond inside, but periodic+system is more simple to use, then cron-in-docker solution.

[jippi]

I don't think nomad ever supported periodic jobs for any type but batch - but having a way to run something across all hosts could be nice.

today we use rundeck for that :)

[dadgar]

Hey as @jippi mentioned, this has never been supported but it is a valid use case. Tagged as an enhancement. Thanks!

[oribaldi]

@dadgar Is there a way to simulate this with the batch scheduler? In other words, how can we run a periodic job on each host with the batch scheduler? Setting the distinct_host constraint is not enough.

[dadgar]

@oribaldi There really is no elegant workaround with out scheduler support. You could run the batch scheduler with distinct_host and a count equal to the number of nodes on the cluster but there is no guarantee that will target every node. It may be worth just running the system scheduler as is, and wrapping your "batch" job in a script that acts as a cron and starts up the task every N minutes.

Sorry that there isn't a great work around.

[oribaldi]

@dadgar Thanks for the reply! We will go for the cron job then.

[huntercatalan]

+1. for now, i have a master job that restarts scheduled system jobs maintained in a separate config.

[ketzacoatl]

I would love to use nomad as a distributed cron, and move my "management tasks" running on various CM systems over to nomad, but need this to do so.

[shantanugadgil]

My use case is to run a "cleanup" of the docker directory at '/var/lib/docker'. (even though I have image cleanup set to true in my agents)

[henyxia]

Any chance to have this feature developed any time soon? :pray:

[kcajf]

This would be super useful for important periodic cleanup jobs

[ndobbs]

This would be great to have, I have a few scenarios where a 'system' job could be scheduled periodically.

[shantanugadgil]

based on a suggestion by Tommy Alatalo in the Gitter room, you can leverage the restart section of the system job to mimic the periodic behavior:

ref: https://gitter.im/hashicorp-nomad/Lobby?at=5e4beee0d97c107ed25e3a91 (sorry, I could not find his previous reply, which made me post this)

[ndobbs]

based on a suggestion by Tommy Alatalo in the Gitter room, you can leverage the restart section of the system job to mimic the periodic behavior:

ref: https://gitter.im/hashicorp-nomad/Lobby?at=5e4beee0d97c107ed25e3a91 (sorry, I could his previous reply, which made me post this)

Thanks a ton!

[tino]

@jippi: I don't think nomad ever supported periodic jobs for any type but batch - but having a way to run something across all hosts could be nice.

As in: "I don't think it will ever"?

Is there any chance this enhancment will be added? It sounds like an easy add, as both the system & cron types already exist ☺️. But I haven't looked at the code.

today we use rundeck for that :)

Looks like a nice tool but I really think it would be something nomad could easily provide, saving me from having yet another tool to manage things in our infra... :)

[shantanugadgil]

@tino you could try the restart stanza trick to simulate periodic behavior ... ref: https://github.com/hashicorp/nomad/issues/1944#issuecomment-592162572

[tino]

I could but that means I have to fail the task, which will give noise. ("The restart stanza configures a tasks's behavior on task failure. Restarts happen on the client that is running the task.")

I currently implemented it by running a 24h sleep around the actual task in a loop. Also ugly, hence my request to the Nomad team to take another look at this 😉

[shantanugadgil]

@tino I too once had something similar ... a "while 1" loop with a sleep of 24h 😄

Just for the record ... I am mentioning the restart stanza here to get the hack to work. (I keep searching for this from time to time myself)

Note delay is the "cadence" with which you want the job to execute. interval should be double than the delay

group "something" {
  ...
  restart {
    interval = "4h"
    attempts = 1
    delay    = "2h"
    mode     = "delay"
  }
  ...

[lgfa29]

Thank you all the input and waiting for this feature. sysbatch jobs are now supported in Nomad 1.2 and they can be set as periodic just like regular batch jobs 🎉

Feel free to open new issues if you find any problems with sysbatch jobs 🙂

Closed by #9160.

[github-actions[bot]]

I'm going to lock this issue because it has been closed for 120 days ⏳. This helps our maintainers find and focus on the active issues. If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.