search

adonisjs / rfcs

💬 Sharing big changes with community to add them to the AdonisJs eco-system
52 stars 6 forks source link

Support for queues #4

Closed assertchris closed 3 years ago

assertchris commented 6 years ago

Brief history

Similar frameworks support a close integration of message queues, for parallel processing. This is useful for pulling slow tasks out of the request/response cycle in a way that is easier to manage than child processes.

What problem does it solve?

We can already achieve parallel processing via child processes. These are more difficult to manage and restart/terminate (from within NodeJS) than what is idea for an app developer. It would be great to build on one of the existing AdonisJS (or NodeJS) queue libraries, to allow deeper integration with things like email sending, events/listeners, and other deferrable tasks.

Proposal

Pull in one of the third-party queue libraries or build one from scratch; as part of the framework. Tie into the queue system with other parts of the framework, like mail sending and events/listeners. Document how to use these features.

Relevant links

https://www.npmjs.com/package/kue https://laravel.com/docs/5.6/queues

Are you willing to work on it?

Yes, but I don't think I could do it alone.

diego3g commented 6 years ago

I'm using https://github.com/nrempel/adonis-kue and it can be a great start to native queues in AdonisJS.

thetutlage commented 6 years ago

Yeah, I do have plans to implement queues as first party addon. @assertchris @diego3g can you guys explain, what all features you are looking from queues.

Also @assertchris, also I know you are a user of Laravel, is there something specific that you like about Laravel's implementation of queue and you found that missing in Node.js alternatives?

assertchris commented 6 years ago

The main thing is that Laravel's queue is deeply integrated into many aspects of the framework. Everywhere there's something that is common to do in a request/response cycle, as a way to push it into the queue.

I have never had it that easy in an AdonisJS app before; because the first step is finding a third-party library that isn't abandonware and figuring out how to get it working in my app, and the second step is changing every bit of how I do slow things so that it can work with the library I've found.

I want to be able to call an event() function, with an object that indicates it can be queued, and have it go to the queue. If the queue is configured to use Redis then that's what is used. if the queue is configured to be synchronous then the event is processed immediately. Then, it's effort less to prototype with a sync queue and deploy to a Redis-backed parallel processing queue in prod.

mourad-ghafiri commented 6 years ago

it will be a great thing to have a queue system in adonis 5 , hope it will support priority as well and a dashboard to visually monitor the queue :)

assertchris commented 6 years ago

@mouradghafiri38 a dashboard is probably quite far out of scope for this RFC.

thetutlage commented 6 years ago

For the most part, I do agree with all the comments in this thread. Do let me know, if anyone wants to work on this RFC, I will be more than happy to guide them or work together in collaboration.

Note: It may take another week or 10 days, before you can starting working on this RFC, since we have to take of RFC's with Fundamentals label first.

willvincent commented 5 years ago

Figured I would mention it here -- I just released adonis-ironium earlier this week. Based heavily on (read: borrowed much of the code from) Adonis-Kue, but it uses Ironium instead of Kue as the name suggests. Ironium provides a unified (simple) api for ironMQ, SQS and Beanstalkd.

It seems to me that the ideal solution here would provider a pluggable interface allowing any backend to be easily used with the same, or very similar, api. Laravel's queue system is pretty complete, and well documented, modelling after it wouldn't be a bad idea.

McSneaky commented 4 years ago

Necroing it little bit, since I think this could also solve #25 (scheduled tasks)

I think how Yii2 (PHP framework) handles queue is really good way to do it Basically they have queue that also works as task scheduler

I might remember some things wrong, since years have passed since I worked on it, but here's how it roughly worked. Maybe some of it was done in the company, not by Yii2 itself, but I still find it really good way

They write job to queue and set some dates for it: When was job queued, when shall job run and when did job actually run (might get delayed coz of heavy loads, downtime etc)

It basically works as scheduler too, since at the end of job you could just reschedule the same job to run again at x time.

They had queue separated into multiple pieces, like serializer, driver, executor and manager Serializer is to serialize and deserialize payload needed to run the job, may it be JSON, XML, whatever Driver is well, basically place where job is held, file, DB, Redis etc Manager reads and writes jobs from and to the queue. When job is read then lock is added to that job. When you have 10 instances running, then job can be picked up only by single manager. As soon as manager reads the job for execution it locks the job so no other manager will start executing it Executor is well, job executor. It get's job from manager and executes it. Executor can be anything, it can be just an class or fork or thread or another process

Such structure solves 2 things at once, queue and scheduling. It also gives quite a lot of flexibility. You can easily switch drivers, serializers and executors. Want to use Redis driver instead of Postgres? Not a problem, just switch the driver. Same with executor. Job is really heavilifting and would block NodeJS main thread? Offload it to other thread or fork or totally different process (like some Rust or C++ worker)