scalability: delaying large number of tasks

[assembler]

From reading the source, scheduler is going to handle_delayed_items each 5 seconds (sleep interval).

This method is going to take the next timestamp from the queue and process all tasks associated with that timestamp.

If i have huge amount of tasks which are scheduled (delayed) to be run like this:

• now -> task 1
• now + 1 second -> task 2
• now + 2 seconds -> task 3
• now + 3 seconds -> task 4
• now + 4 seconds -> task 5
• now + 5 seconds -> task 6
• ... and so on

Scheduler is going to run like this:

• now -> task 1
• now + 5 seconds -> task 2
• now + 10 seconds -> task 3
• now + 15 seconds -> task 4
• now + 20 seconds -> task 5
• now + 25 seconds -> task 6
• ... and so on

If these repeated tasks are infinite, scheduler will keep lagging more and more as the time passes.

I see two solutions to this:

1. set sleep interval to 1s
2. refactor handle_delayed_items to keep looping until next_delayed_timestamp reaches the current timestamp

[assembler]

or third option: refactor my app no to schedule tasks in intervals smaller than poll_sleep_amount (5 seconds)

[ksinkar]

@assembler, Can you give a unit test in a PR against which we can program and implement this feature. Maybe we can give a an external configuration setting for the sleep interval.

[rohitpaulk]

This seems to be handled now, tasks are enqueued in a loop until no more timestamps currently have tasks: https://github.com/resque/resque-scheduler/blob/5d7a4e1d4ef33c37e9c05c9a91236e7ef12e1529/lib/resque/scheduler.rb#L189-L192