taskforcesh / bullmq

BullMQ - Message Queue and Batch processing for NodeJS and Python based on Redis
https://bullmq.io
MIT License
5.62k stars 365 forks source link

Store the queues defaultJobOptions independently from the jobs to reduce memory consumption #2625

Open lukas-becker0 opened 1 week ago

lukas-becker0 commented 1 week ago

I noticed that the defaultJobOptions are added to each job in redis, I thought they would be stored independently from the jobs.

I think it would be nice to store the defaultJobOptions in the queue and not add them to each job in redis, that would reduce the memory consumption e.g. when using custom job retain options for completed and failed jobs.

lukas-becker0 commented 6 days ago

Okay, I missed that we can at least do this for the retain options by passing them to the worker options as opposed to the queue options although that difference has not been documented.

https://docs.bullmq.io/guide/queues/auto-removal-of-jobs https://docs.bullmq.io/guide/workers/auto-removal-of-jobs

manast commented 5 days ago

Yes, we are trying to move some settings to the workers to reduce memory usage.