search

concourse / time-resource

a resource for triggering on an interval
Apache License 2.0
44 stars 33 forks source link

allow periodically scheduled tasks to produce even load on the system #40

Closed weberstephanhd closed 5 years ago

weberstephanhd commented 5 years ago

We have heavy peaks when e.g. a 30mins time trigger is being used by some ten short running monitoring jobs spread over many different pipelines. Our time trigger resource is configured like

- name: 30-min
  type: time
  source:
    interval: 30m

in every pipeline. With that we get heavy peaks every 30mins for 2-3 mins. Just increasing the size and amount of worker resources is propably not the right way if they then idle the other 27 mins... is there something like Jenkins' randomised scheduling feature using H:

To allow periodically scheduled tasks to produce even load on the system, the symbol H (for “hash”) should be used wherever possible. For example, using 0 0 for a dozen daily jobs will cause a large spike at midnight. In contrast, using H H would still execute each job once a day, but not all at the same time, better using limited resources.

=> https://github.com/jenkinsci/jenkins/blob/master/core/src/main/resources/hudson/triggers/TimerTrigger/help-spec.jelly

How would you spread the load of 50 2-3min jobs distributed over 20 independent pipelines so that each runs once in 30 mins?

vito commented 5 years ago

Planning to tackle this with https://github.com/concourse/rfcs/pull/27 which will give each job its own interval. 🙂

Also, we'll be addressing the worker overloading problem in general by implementing some form of queueing, so at least in the worst case you shouldn't end up with your workers dying. You can keep tabs on progress for that in https://github.com/concourse/concourse/issues/3695

I'm gonna close this since there's nothing that this resource can really do - the change has to happen in Concourse itself, as it's currently an architectural flaw that I plan on addressing with the aforementioned RFC. Thanks!