nicoburns
commented
2 years ago My thinking was that I could schedule jobs with unique payloads to run on a scheduled cron interval but it doesn't appear that is how things work. Am I missing something?
The way I have approached this is to have two queues:
- A queue that takes a payload and does work based on that
- A "scheduler" queue that runs on a schedule, and simply creates tasks for the other queue
If I do that approach, it appears that you are reading in all rows in the schedule table every interval (i.e. every ~30 seconds) to determine which ones need to run. Seems like that is a lot of database queries to constantly be reading in that list of records so frequently. Does it scale to large number of scheduled items?
The table's indexed, so it doesn't have to scan every row. For really high queue volumes you probably would start running into issues, but we're talking several thousand queue tasks at least before you'd run into trouble.
timgit
on2air
After some digging into the code, I have a few questions on how the schedule feature works for recurring tasks.
It appears that you schedule a queue to repeat, not specific jobs, correct? This doesn't seem to fit my use case. I have a single queue with different jobs that in the past have utilized the singletonkey to differentiate themselves.
I guess I could treat the queues as different jobs by using a unique name for each and then use a dynamic listener to listen to all queues based on a wildcard in the name. Is that best practice?
If I do that approach, it appears that you are reading in all rows in the schedule table every interval (i.e. every ~30 seconds) to determine which ones need to run. Seems like that is a lot of database queries to constantly be reading in that list of records so frequently. Does it scale to large number of scheduled items?
My thinking was that I could schedule jobs with unique payloads to run on a scheduled cron interval but it doesn't appear that is how things work. Am I missing something?