search

RomainLanz / adonis-bull-queue

Queue system based on BullMQ for AdonisJS
MIT License
147 stars 26 forks source link

BullMq Flows? #26

Open timganter opened 1 year ago

timganter commented 1 year ago

Hi there! Is support for BullMQ flows on the roadmap for this package? 🙂

RomainLanz commented 1 year ago

Hey @timganter! 👋🏻

I have never heard of it! Skimming through the doc, it indeed seems a nice feature to have. So expect to have it land on the next major release (the one to support AdonisJS V6).

timganter commented 1 year ago

That's awesome to hear! Thanks @RomainLanz ! 🎉

ThisIsMissEm commented 5 months ago

So just some small caveats: I've just been trying to get Bullmq Flows to work, and there some issues I've encountered:

  1. Flows basically destroy all types, like, everything you do with them will be unsafe. With Queues and Workers you can type them like: 
    
const DownloadWorker = new Worker<MediaDownloadData, MediaDownloadResult>(

export const MediaDownloadQueue = new Queue<MediaDownloadData, MediaDownloadResult>(



2. FlowProducer and FlowJobs do not at all like working with MemoryDB or Redis Cluster mode, I'm not sure why, but they just hang on producing jobs that the Worker can pick up. From my experience with Bullmq, hangs tend to happen when a connection cannot be established, but even passing in the correct connection args it just didn't work.

3. Good luck with typings on connections: Bullmq ships it's _own_ types for RedisConnection, and doesn't seem natively aware of RedisClusterConnection.

So basically: you loose typesafety with your jobs, and you'll hit into hard to debug connection issues when using MemoryDB or Redis Cluster mode.
RomainLanz commented 5 months ago

Thanks for the feedback! Maybe we should wait a bit before implementing them inside this package.

Also, it seems it may be "experimental".

Flows are a brand new feature in BullMQ, and although is implemented on a stable foundation there could be some unknown issues.

ThisIsMissEm commented 5 months ago

@RomainLanz yeah, it cost me a good few hours ripping them out and figuring out an alternative approach.