Support for Bull / BullMQ libraries

[manast]

Due to the increasing popularity of Upstash as a Redis hosting solution, many Bull/BullMQ users are trying to deploy their queues and getting all kinds of weird behaviors since, at the moment, Upstash is not compatible with any of these two libraries.

The reason is not working is because Bull/BullMQ requires to create keys inside the Lua scripts, but Upstash requires the keys to be explicitly known before the scripts are executed.

In order to resolve this issue, it would probably be needed to make changes both in Bull/BullMQ as well as in Upstash.

[manast]

Although i've just deployed an app with BullMQ, with no jobs and 1 queue but it chewed up all 10K daily requests? 🤷

Do you have a log with the commands you were charged for that empty queue?

[mdogan]

Just jumping in here, as the Fly.io documentation now explicitly says:

Note that empty responses are not counted towards your bill. This prevents billing for standard polling behavior of tools like Sidekiq or BullMQ.

Although i've just deployed an app with BullMQ, with no jobs and 1 queue but it chewed up all 10K daily requests? 🤷

Normally empty responses are not billed but due to a bug they are still counted in the free-tier quota at the moment. The fix is being gradually deployed to all AWS and fly.io regions.

Still, not all commands are considered empty while BullMQ is idle. Especially EVAL/EVALSHA commands are counted (but not the empty commands embedded inside Lua script).

I did a quick test with a single worker and single queue in 5mins period. Here are the metrics:

 "DailyProcessedCommands": 445,
 "DailyWriteCommands": 184,
 "DailyReadCommands": 261,
 "DailyEmptyCommands": 335,

445 commands (184 writes, 261 reads) are sent by the idle BullMQ worker, and 335 of them are considered empty and not counted as billable. So still, 25% of the idle requests are billed.

These billed commands are mostly EVALSHA and some occasional SET and HGET calls.

[TheRealFlyingCoder]

Normally empty responses are not billed but due to a bug they are still counted in the free-tier quota at the moment. The fix is being gradually deployed to all AWS and fly.io regions.

Just to confirm @mdogan all 445 would have been counted as billed requests currently in Free Tier on fly due to the bug?

But once a fix is implemented 25% of idle requests will still be counted towards the daily limit?

[CanRau]

Very exciting this is being figured out, probably returning to upstash soonish then 🥰

Thanks a lot for all the work ❤️

[TheRealFlyingCoder]

Just checking in on the status of this... was there any work that we needed to do for Fly.io instances?

I have a development server online, with zero activity on it, and upstash is currently averaging 35K requests/day

So far I have 300K billable requests, 200K empty requests showing up on the usage for the month.

[manast]

@TheRealFlyingCoder which request is the one producing the empty 200k requests, BRPOPLPUSH by any chance?

[buggyhunter]

@TheRealFlyingCoder we would like to check deeper if you provide your database name?

[TheRealFlyingCoder]

@TheRealFlyingCoder which request is the one producing the empty 200k requests, BRPOPLPUSH by any chance?

So unless i'm crazy I can't find any way to check logs with Fly.io Upstash?

@buggyhunter it's 'theprintbar-dev-redis'

[mattapperson]

Just checking in with this. Has the issue of being charged for empty requests / using BullMQ very large queues been solved?

[TheRealFlyingCoder]

Just checking in with this. Has the issue of being charged for empty requests / using BullMQ very large queues been solved?

Not that i'm aware of... i had to switch to a paid subscription just to keep it alive.

My BullMQ with no activity (just polling) hits exactly 166K requests per day every day, of that ~50% are put down as billable

[manast]

I think you can change the "draingDelay" setting to something much larger, like 60 seconds or so, that would reduce the requests to "just" 13k or something. I hope I can work something up with the upstash team to make this much better.

@TheRealFlyingCoder BullMQ does not do a lot of polling, and in fact you can reduce it as much as you want with the drainDelay option (https://api.docs.bullmq.io/interfaces/v4.WorkerOptions.html#drainDelay). The problem is the amount of commands executed every time a job is added and processed, could be something between 10 and 20 commands per job.

Did you try this?

[TheRealFlyingCoder]

Did you try this?

Will push an update now and let you know if I notice a drop... I didn't update it originally because the wording seemed to suggest it wouldn't be the problem haha

At a default of 5 seconds it still shouldn't be hitting 160K/day unless i'm misunderstanding how it works

[lildesert]

I'm just sharing my experience here but while we're waiting for this to be fixed we deployed a redis on a fly.io app of our own following this repo https://github.com/fly-apps/redis/. It's been working well for our usage so far (which is pretty basic) and it's much more cost-efficient.

[jorisvdz]

Upstash launched a new project (https://github.com/upstash/queue), maybe this will be of interest to some of you?

[TheRealFlyingCoder]

Did you try this?

@manast Just coming back to this, Drain delay saw no improvement... and as a follow up I've continued to build out my solution which involves more queues and it's increasing exponentially.

I am now up to 2-3M requests per day with zero traffic hitting the server 😬😬😬

All of the graphs are flat, i'll share the stats and hopefully you'll see if anything stands out:

Daily Commands: 2m - 3m Daily Bandwidth: 150MB - 200MB Throughput: 20 Reads & 16 Writes per second Data Size: 24.38KB Connections: 138 Keyspace: 85 Hits/s: 37 Misses/s: 97

[manast]

@TheRealFlyingCoder if you want you can contact me via email and provide more details of your solution so that I can understand better where all these calls are going.

[manast]

I just made some tests with @TheRealFlyingCoder example code and with 1 worker it should consume around 700k commands per month, which at current costs is less than 1.5$/mo. There are a couple of things to note here however, I am assuming only "write" commands are charged, and I think that should be the case. Secondly, it would be even better if Upstash only charged for commands that actually changed the database. So for instance, if I delete an un-existing key, no charge for that command... If that was achieved, then for an empty queue there is by default only 1 write command every 30 seconds, or basically zero costs.

[TheRealFlyingCoder]

I'm also keen to know whether my solution has a flaw in it's implementation or whether that polling quantity is to be expected.

Because as I stated before it is ramping up a lot, I am up to 17 queues, running on each of my 3 dev servers.

Using @manast's 700K requests and 51 active queues, I would expect 35M writes/month now (~1M/day which is what i'm seeing in the usage) and a base operating cost of ~$75/month just from the write command polling.

The read commands are much higher and also seem to be charged though but I don't have the numbers on those (Fly.io upstash you can't see any real info)

[vincentdoerig]

For those using Upstash with fly.io, they just announced a $10/month fixed plan with no cap on the number of commands executed: https://community.fly.io/t/upstash-for-redis-new-10-mo-single-region-fixed-price-plan/19555.