search

taskforcesh / bullmq

BullMQ - Message Queue and Batch processing for NodeJS and Python based on Redis
https://bullmq.io
MIT License
5.96k stars 386 forks source link

Make BullMQ Compatible with Bun #2177

Open manast opened 1 year ago

manast commented 1 year ago

Bun has reached 1.0 (https://bun.sh/), and is getting a lot of traction. Being much faster than Node, it is of interest to make BullMQ compatible with it and see if we can also gain some extra performance. Furthermore, as more and more users starts using Bun, they will expect BullMQ to also work flawlessly with this new runtime.

Tests The major hurdle right now is in the tests. Currently, Bun does not support Mocha, even using some hacks that have popped up recently I could not make it work. Bun has its own built-in test framework that is supposed to be compatible with Jest. As we need to be able to run tests both using Node and Bun (we cannot just trust that because tests pass in one runtime they would pass the other), I think the only way forward is to port the tests to Jest, to begin with, and then try to run them with Bun. Also, the test files need to be renamed, as the current naming schema is not compatible with Bun. So we should change them from test_job.ts to job.test.ts.

Benchmarks As part of this issue, it would also be quite interesting to make some benchmarks to see the performance gains if any. I did some experiments some months ago and the improvement was not so impressive, around 20%. This could mean that the current bottlenecks in BullMQ are in Redis itself, it could also be that IORedis is not running as fast as it could (I have seen benchmarks where the old Redis npm module is actually faster than IORedis).

kibertoad commented 1 year ago

@manast Vitest is also Jest- and Bun-test compatible, and is (arguably) a better and more modern test framework; perhaps it could be considered as well?

kibertoad commented 1 year ago

Also it would be interesting to see a modern benchmark comparing recent versions of node-redis and ioredis. ioredis rather stagnated lately, while node-redis is still going strong, and was even officially embraced by Redis Labs.

manast commented 1 year ago

Also it would be interesting to see a modern benchmark comparing recent versions of node-redis and ioredis. ioredis rather stagnated lately, while node-redis is still going strong, and was even officially embraced by Redis Labs.

Yes, I am aware of it. However we are using some unique features of ioredis and getting all the connect/disconnect cycles correct took a lot of attempts, so I think switching back to the official Redis module is not exempt from risks...

manast commented 1 year ago

I will note https://vitest.dev/ as a viable alternative to Jest.

loicnestler commented 12 months ago

Is there already a roadmap planned for this? What are the necessary steps / changes in order to support the Bun runtime?

manast commented 12 months ago

@loicnestler mostly ports the tests to jest or vitest, so that we can verify that it works as the Node version. In my local tests it seems to be working though, so it may be just pure administrative work to port the tests.

nullndr commented 9 months ago

Any news on this?

manast commented 9 months ago

@ccollie is working on porting the tests to Vite, after that we can also run all the tests on bun and so officially support it. However, in my experiments a couple of months ago I noticed that the process could die unexpectedly with a segmentation fault, it may be more stable now, but I would not recommended it yet for production until it has been a bit more battle tested.

devdras commented 7 months ago

@ccollie @manast is there a branch where the tests are being converted to vitest?

ccollie commented 7 months ago

@devdras - https://github.com/ccollie/bullmq/tree/vitest It's a bit behind master but the work itself is not too difficult. Message me if you need pointers.

manast commented 6 months ago

@andras-fn do they fail with bun only or also with Node?

devdras commented 6 months ago

@manast I've only tested with Node. I'll test with Bun and confirm. Could you remove that tag please. Multiple git accounts goof on my part.

manast commented 6 months ago

I wouldn't bother to test with bun until they all work with Node.

ccollie commented 6 months ago

@ccollie would you mind giving me some pointers please? I've got the Vitest test's running. I can see there's a total of 467, 108 fail and 356 pass. Is the goal to look at why the 108 fail and make them work as intended?

That's the goal. The main issue is in converting async tests which take a done callback.