throttling not working as one would expect?

[thisIsLoading]

Hi,

i wanted to use throttling for a job that downloads files from a website. i dont want to limit the amount of jobs to be scheduled, just the amount of jobs that are executed within a certain amount of time. i figured one job every 5 seconds would be good enough to not trigger any 429s on the remote server, so i set this:

  good_job_control_concurrency_with(
    # Maximum number of jobs with the concurrency key to be
    # concurrently performed (excludes enqueued jobs)
    # Can be an Integer or Lambda/Proc that is invoked in the context of the job
    perform_limit: 1,

    # Maximum number of jobs with the concurrency key to be performed within
    # the time period, looking backwards from the current time. Must be an array
    # with two elements: the number of jobs and the time period.
    perform_throttle: [12, 1.minute],

    key: -> { self.class.name }
  )

which i thought would exactly do what i need.

however, when running the jobs i realized, that the concurrency control seems to be via an exception where it throws an error about an exceeded throttle:

and it re-schedules the job about 3 hours later(?)

There definitely are a lot of moving parts and my simple world view isnt enough. so, i clearly dont seem to understand how i would need to configure the way i want it to perform.

I thought, it would take this arry, would devide the duration by the number in [0] and then executes a job, waits the calculated amount and executes the next job.

can you help me out what i do wrong and how i get it to constantly executing jobs until the queue is empty, without long pauses?

thank you!

[bensheldon]

Your understanding of how Throttling actually works is correct. I tried to explain that in this section in the Readme: https://github.com/bensheldon/good_job?tab=readme-ov-file#how-concurrency-controls-work

You're seeing 3 hours because it is using retry_on ... wait: :polynomially_longer. You can add your own retry_on handler to your job with a fixed retry e.g.

retry_on(
          GoodJob::ActiveJobExtensions::Concurrency::ConcurrencyExceededError,
          attempts: Float::INFINITY,
          wait: -> (executions) { 30.seconds + (10 * Kernel.rand) } 
        )

The challenge with throttling and concurrency control is that there's a conflict between the goal of a general job queue (run tasks as quickly as possible) and a throttled queue (run tasks at a managed rate). GoodJob's "dequeue, check constraints, retry" pattern is the same one I've seen implemented elsewhere, but I'm open to contributions or outside inspiration.

[thisIsLoading]

i understand. thank you @bensheldon

as this was just the early stages of my project, i must admit i jumped ship to sidekiq after i found https://github.com/ixti/sidekiq-throttled which is doing exactly what i needed.

i still feel good_job is doing a better job than sidekiq, just not for this particular use case. unfortunately i dont feel ready enough to contribute anything (yet), so i had to take the easy exit.

with that said, thanks a lot for doing all this and providing this gem.

[ixti]

FWIW, I believe GoodJob's concurrency control is doing a better job than sidekiq-throttled. If I understand correctly, GoodJob does not deletes row from DB. Thus, message does not need to be removed just to get written back in case of throttling. That makes me believe throttling with GoodJob should be more efficient.