Support for Bull / BullMQ libraries

[manast]

Due to the increasing popularity of Upstash as a Redis hosting solution, many Bull/BullMQ users are trying to deploy their queues and getting all kinds of weird behaviors since, at the moment, Upstash is not compatible with any of these two libraries.

The reason is not working is because Bull/BullMQ requires to create keys inside the Lua scripts, but Upstash requires the keys to be explicitly known before the scripts are executed.

In order to resolve this issue, it would probably be needed to make changes both in Bull/BullMQ as well as in Upstash.

[enesakar]

hey @manast . Do you have anything you plan on your side? How possible is it to call the lua scripts with predefined keys? @mdogan knows better but what I understood is this is the only thing blocks us.

[manast]

@enesakar I am waiting for the dev team to come back to me so that we can discuss potential solutions.

[pellizzetti]

Ran into this issue today, did they managed to contact you @manast? Any updates @mdogan?

[manast]

I have been contacted and offered all my help to solve this issue, and it seems they looked a bit into it but I do not know the status or their plan.

[ChristianIvicevic]

I am curious since I just tested Bull agains Upstash via Nest.js myself and it seems to work. The only caveat being that there are so many commands that I'd go broke, but it seems to work regardless.

[chronark]

It works as long as all keys remain in memory. But we evict keys from memory after some time (they are of course durably stored on disk).

The problem is that bullmq is building dynamic keys inside of lua Scripts, which is highly discouraged by the official redis specs btw.

When executing a script on upstash, we first load all specified keys from disk to memory, but that's only possible for the ones that are explicitly provided as script arguments.

[chronark]

Important: to ensure the correct execution of scripts, both in standalone and clustered deployments, all names of keys that a script accesses must be explicitly provided as input key arguments. The script should only access keys whose names are given as input arguments. Scripts should never access keys with programmatically-generated names or based on the contents of data structures stored in the database.

from https://redis.io/docs/manual/programmability/eval-intro/#getting-started

[manast]

The problem is that bullmq is building dynamic keys inside of lua Scripts, which is highly discouraged by the official redis specs btw.

It has not been highly discouraged previously, although they have rephrased the recommendation recently. The reason for this recommendation is because Redis cannot fully take advantage of clustered setups, although there are workarounds for this https://docs.bullmq.io/bull/patterns/redis-cluster

Bull/BullMQ works really well on all Redis hosting solutions (Elasticache, GCP, Redis cloud, Redis Green, etc), but does not work on Upstash because it is a custom implementation that do not work exactly the same as the rest.

I have offered resources to try to find a solution but unfortunatelly this has not received enough priority from Upstash (or they think it is too difficult to solve) so it has not yet been resolved.

[manast]

For the record, I challenge anybody if they can find a bug in Bull/BullMQ related to using keys generated in LUA scripts when using any official Redis instance.

[ctrlaltdylan]

I'm a BullMQ user and a Upstash customer. I would transfer my hosting to Upstash if it was compatible with BullMQ. But this incompatibility has forced me to manage my background jobs on AWS.

[chronark]

@manast you are right it works with the official redis server implementation, because redis keeps everything in memory and our implementation moves entries to disk to allow us to provide the service at much lower cost than traditional redis vendors.

When implementing it, we were sticking with the guidelines and specs at redis.io and at this point the only solution would be to enable the option to keep everything in memory That would make things way more expensive and commercially unviable to be used with a queue library like bullmq or others. They tend to be very noice and are not well suited for a per-request-pricing-model

Unfortunately making bullmq compatible is not as easy as fixing a few bugs.

@ctrlaltdylan If you want to use a managed queue on Upstash, please try out QStash. It offers many of the features of bullmq and some unique ones such as being able to run with serverless functions and more.

[manast]

Unfortunately making Bullmq compatible is not as easy as fixing a few bugs.

We do not know actually, because we never discussed the possibility in detail...

[manast]

Closing to signal that no further actions are expected in this regard.

[enesakar]

@manast We have worked on a solution and include in the next release if it passes our perf tests.

[manast]

Impressive!

[ianjkaplan]

What is the status of this release? I have an app that uses Bullmq and want to migrate to fly.io and use upstash redis on fly. It would be nice not to have to change my queue implementation

[mdogan]

We are currently deploying the fix to all regions gradually, not sure whether it's deployed to fly.io regions at the moment.

But note that, this fix is only for older Bull library (specifically to support dynamic keys in Lua scripts), not for the new BullMQ. BullMQ uses Redis Streams to publish events but Upstash doesn't support streams api yet.

[antoinedc]

We are currently deploying the fix to all regions gradually, not sure whether it's deployed to fly.io regions at the moment.

But note that, this fix is only for older Bull library (specifically to support dynamic keys in Lua scripts), not for the new BullMQ. BullMQ uses Redis Streams to publish events but Upstash doesn't support streams api yet.

Any plans to support it?

[manast]

@antoinedc events in BullMQ are not a critical feature, meaning the queues will work without the events. Of course, the events may be important for you for debugging purposes, etc. If streams do not land in upstash, it is possible that we can have an events system based in pubsub again, as we had in Bull.

[antoinedc]

@antoinedc events in BullMQ are not a critical feature, meaning the queues will work without the events. Of course, the events may be important for you for debugging purposes, etc. If streams do not land in upstash, it is possible that we can have an events system based in pubsub again, as we had in Bull.

Hi Manast, thanks for the reply. I'm confused now 😅, does that mean that BullMQ works on Upstash then?

[manast]

I don't know... I am just saying that events are not critical so even if streams are not implemented in Upstash, it could work.

[mdogan]

We are currently deploying the fix to all regions gradually, not sure whether it's deployed to fly.io regions at the moment. But note that, this fix is only for older Bull library (specifically to support dynamic keys in Lua scripts), not for the new BullMQ. BullMQ uses Redis Streams to publish events but Upstash doesn't support streams api yet.

Any plans to support it?

Streams are in our roadmap but not in the short term unfortunately.

[mdogan]

@antoinedc events in BullMQ are not a critical feature, meaning the queues will work without the events. Of course, the events may be important for you for debugging purposes, etc. If streams do not land in upstash, it is possible that we can have an events system based in pubsub again, as we had in Bull.

Hi Manast, thanks for the reply. I'm confused now 😅, does that mean that BullMQ works on Upstash then?

No, it doesn't work at the moment. Because any call like redis.call("XADD", ...) or redis.call("XTRIM", ...) raises an error and script execution is terminated at that point. As an alternative, if they are called via redis.pcall(...) instead, then script will continue to run and return error reply for these unsupported commands.

I don't know the all use-cases for BullMQ, but it might be easier to optionally turn off events feature than implementing them based on pubsub.

[chronark]

Streams are unlikely to be available on upstash in the next few months, so let's discuss options:

1. replace stream calls with pcall to partially ignore errors and partially handle them gracefully.
2. allow the user to disable streams this would require adding a new argument to lots of scripts and pass that configuration around. Sounds like a lot of spaghetti to me.
3. Event system build on pubsub as mentioned here

@manast I'd like to work on this and submit a PR to bullmq, but I would appreciate your opinion on what's the best approach.

[manast]

The main reason for using streams in the global events system of BullMQ was to enable the creation of a user interface that can continuously monitor and display the status of jobs in real time. This approach is effective because using streams ensures that no events will be lost. Additionally, the user interface can always request events starting from a specific event ID, which allows for complete and accurate information to be displayed at all times.

This approach has not been used in any user interface that I know of. It is possible to achieve a similar result using other methods. For example, an "epoch" value that is incremented every time a new event is emitted can be used to determine if the data on the client is out of sync with the server. The client can request new data from the server if the epoch value of a received event is more than one higher than the current value. In the future, we can consider implementing the epoch approach to improve the events system.

So for now, I think we can revert to using standard PubSub.

[Corjen]

Hi! Is there any update on this? Would love to be able to use bullmq on upstash

[manast]

@corjen according to the compatibility table, redis streams are coming soon: https://docs.upstash.com/redis/overall/rediscompatibility

[sancar]

Streams implementation is almost ready. Right now, we are working on tests within the bullmq repository against upstash redis. We should be able to release it in a couple of weeks.

[tylertyssedal]

Hi @sancar ,

I saw that streams are still listed as coming soon in the compatibility table. Are there any updates regarding the release timeline?

Thank you!

[sancar]

Hi @tylertyssedal , testing phase took more than expected. We identified a couple of problems and fixed them. It will be released this week. We will start deploying gradually tough. so give it some more time to be available everywhere.

[barbinbrad]

Anyone able to run this successfully yet?

[chronark]

@barbinbrad we have deployed the required features to regional dbs in ap-north-east-1, eu-central-1 and partially to us-west-2 other regions and global dbs will follow soon

[manast]

Please let me know when this is generally available so that we can remove the current check on BullMQ for upstash hosts.

[chronark]

https://github.com/taskforcesh/bullmq/pull/1756

I've created a PR for it, I'll give you an update as soon as it is GA.

[nickjames00]

I am super excited for this feature! @chronark is support for streams GA? I just took a look at https://docs.upstash.com/redis/overall/rediscompatibility and it has a green check

[chronark]

yeah, we're just missing one region, everything else is ready. but as far as I understood, @manast wants to wait until all regions are compatible before https://github.com/taskforcesh/bullmq/pull/1756 will be merged

[chronark]

we just upgraded the last remaining region, fly.io databases will be upgraded today as well

and it looks like https://github.com/taskforcesh/bullmq/pull/1756 has been merged already 🥳

[assimovt]

has anyone tried using it already in prod?

[navaneeth-dev]

Can bullmq be used with upstash or not?

[chronark]

@assimovt @navaneeth-dev

yes you can. I'll close this issue to reflect that Please open a new issue if you find any problems

[CanRau]

What about the high command usage? 😬

[rizexor-endeva]

yes high command usage. use redis cloud instead

[kimitrii]

I'm almost contract a plan in upstash but my code use BullMq and when I saw one simple queue with two items generate more then 200 commands and keep counting after the execution, I literally gave up. It's make no sense? What the logic of it?

[manast]

Currently, the biz model of Upstash does not fit well with BullMQ, even if it's supported and works reliably. We are trying to minimize the number of commands issued but still, there are going to be a lot of commands per job unfortunately as there are a lot of things to book keep.

[navaneeth-dev]

I'm almost contract a plan in upstash but my code use BullMq and when I saw one simple queue with two items generate more then 200 commands and keep counting after the execution, I literally gave up. It's make no sense? What the logic of it?

use fast api with python. it has inbuilt background tasks

[mdogan]

Currently we (Upstash) are gathering statistics about empty commands (write commands which are not mutating the state and read commands returning empty response or null) generated by libraries like BullMQ, Sidekiq etc. We are discussing & considering not to account these commands, but it's not decided yet.

[barbinbrad]

Currently we (Upstash) are gathering statistics about empty commands (write commands which are not mutating the state and read commands returning empty response or null) generated by libraries like BullMQ, Sidekiq etc. We are discussing & considering not to account these commands, but it's not decided yet.

Thanks for the update Mehmet! I'm sure many are glad to hear that.

[MartinL83]

Currently we (Upstash) are gathering statistics about empty commands (write commands which are not mutating the state and read commands returning empty response or null) generated by libraries like BullMQ, Sidekiq etc. We are discussing & considering not to account these commands, but it's not decided yet.

Is there any update on this @mdogan ? I would like to use Upstash with BullMQ. I tested out Bull yesterday with 2 queues and 100 something jobs. That ate up 10k commands in one go. So its not realistic to use BullMQ currently with Upstash :/

[TheRealFlyingCoder]

Just jumping in here, as the Fly.io documentation now explicitly says:

Note that empty responses are not counted towards your bill. This prevents billing for standard polling behavior of tools like Sidekiq or BullMQ.

Although i've just deployed an app with BullMQ, with no jobs and 1 queue but it chewed up all 10K daily requests? 🤷

[manast]

@TheRealFlyingCoder BullMQ does not do a lot of polling, and in fact you can reduce it as much as you want with the drainDelay option (https://api.docs.bullmq.io/interfaces/v4.WorkerOptions.html#drainDelay). The problem is the amount of commands executed every time a job is added and processed, could be something between 10 and 20 commands per job.