query on schedule jobs

[ankitAtVauld]

• Which Faktory package and version? Faktory Enterprise 1.6.0

• Which Faktory worker package and version? faktory-worker nodejs 4.2.1

• Please include any relevant worker configuration. 3 Different queues, each with 1000 Jobs/sec. 1 of the queue has all the jobs, first scheduled and then enqueued. Our license allows total 200 connections, but they are under utilised because there may be a lag in moving jobs from scheduled state to enqueued state.

• Please include any relevant error messages or stacktraces None so far.

Are you using an old version? No

Have you checked the changelogs to see if your issue has been fixed in a later version? We seem to be on the latest version.

Issue Description:

We are scheduling like below: let at = DateTime.now().setZone("UTC").plus({ seconds: 3 }).toISO();js Job payload: {id}, { custom: { unique_until: "start" }, at }js

we notice that as the scheduled queue keeps increasing, however the number of enqueued jobs doesn't follow the rate at which the jobs are scheduled. This problem keep getting more prominent with an increasing number of scheduled jobs.

So, we would like to know if there is any throttling or rate limiting that we need to keep in mind while using the scheduling jobs. or anything else that you can suggest we look at. Our requirements are scheduling around 10K TPS for each job to enqueue after 2-3 seconds.

https://github.com/contribsys/faktory/blob/master/Changes.md https://github.com/contribsys/faktory/blob/master/Pro-Changes.md https://github.com/contribsys/faktory/blob/master/Ent-Changes.md

[mperham]

I don't know if Faktory will be able to keep up with that scheduled pace. The scheduler enqueues one job at a time with a network roundtrip. If you can enqueue your jobs directly rather than requiring that 3 second delay, that would be far more efficient. Can you explain the 3 second pause?

[mperham]

Especially if you are using a Remote Redis SaaS with Faktory Enterprise, where the latency may be 1ms or more.