search

weyoss / redis-smq

A simple high-performance Redis message queue for Node.js.
MIT License
596 stars 64 forks source link

[Feature] Support option for a missing interval in the scheduler. #95

Closed intech closed 1 year ago

intech commented 2 years ago

:wave: Hey!

I propose to add an option for the message scheduler, which will support two modes of work with intervals:

  1. Current mode by default. Task runs once per second 1,2,3,restart,9,10 Usage example: we make requests to the API, and it makes no sense to do it more often, for example, because of the rate limits.

  2. Strict mode. The task runs once per second 1,2,3,restart,4,5,6,7,8,9,10 were 4-9 jobs processed without delay. Usage example: we collect data per unit of time and cannot skip the time interval to avoid a gap in the data. We take the data time from the task.

I understand that we could solve the problem another way by polling and processing all unselected intervals at once in the first trigger of the scheduler. But, this complicates the monitoring and debugging of the system. 🤔

weyoss commented 2 years ago

@intech Hey! Thank you for opening this issue.

I was a bit busy with the v7.1 release and I needed some time to think about your suggestion.

So your point is about compensating the scheduler interruption time (down time), that may happen because of a consumer failure or any other reason, to avoid gaps for scheduled tasks that run periodically.

Please confirm or correct me if I am wrong.

intech commented 2 years ago

@weyoss You got that right!

weyoss commented 2 years ago

I believe it is not a good idea to try to compensate messages for missed scheduler ticks. Considering that a consumer is down for several hours or maybe days, once it is online, it would take him a huge amount of time/work to perform the following actions:

  1. To calculate missed ticks since the last time the scheduler was online;
  2. For each tick and for each periodic scheduled message calculate missed messages;
  3. And finally to publish those messages.

Additionally, this would require a complete scheduler logic rewrite.

The way the scheduler is currently implemented is simple: each second a scheduler tick is fired. For each tick, the scheduler pull out from the scheduled messages queue the list of expired/old messages with regards to the timestamp of the tick. Each message of the list is published and the message is scheduled for the next time if it is schedulable.

I am open for discussion.

weyoss commented 1 year ago

Closing as abandoned.