hibiken / asynq

Simple, reliable, and efficient distributed task queue in Go
MIT License
9k stars 659 forks source link

[FEATURE REQUEST] Support for Fixed-Length Queues in Asynq #869

Open FogDong opened 2 months ago

FogDong commented 2 months ago

Is your feature request related to a problem? Please describe. In many applications, particularly those involving resource-constrained environments or systems where old tasks need to be automatically discarded, having the ability to maintain fixed-length queues is crucial. This feature would allow developers to set a maximum size on queues, ensuring that the queue does not exceed a certain number of tasks. This is particularly useful for maintaining the latest N tasks and automatically discarding older tasks when new ones arrive.

Describe the solution you'd like I propose that Asynq supports an option to limit the size of task queues. When a queue reaches its maximum size, the system should automatically remove the oldest task(s) upon the addition of new tasks. This would ensure that the queue remains at a fixed length.

The implementation could involve:

  1. Extending the QueueConfig struct to include a new field, MaxSize, which defines the maximum number of tasks the queue can hold.
  2. Modifying the task enqueue logic to check the current queue size against MaxSize. If the queue size is at its maximum, the oldest task(s) would be dequeued or dropped before the new task is enqueued.
  3. Providing configuration options in the Asynq server settings to enable and configure this feature per queue.

Describe alternatives you've considered

Additional context If this feature is accepted by the Asynq community, I would be very interested in contributing to its development. I believe this feature would not only benefit my use case but also be a valuable addition for many other users facing similar requirements.