cjus
commented
6 years ago @sjmcdowall Thanks for your interest in Hydra. Lot's of great questions above. The shortest answer is that Hydra can be part of your solution but not, perhaps all of it. Hydra message features are not designed to replace a full-blown messaging queuing system. Great choices include Kafka, MQTT, RabbitMQ and many others.
It would take a blog post to answer all of the above. And we're painfully aware that there is a serious need for an article describing Hydra and messaging. Our own work at Flywheel Sports currently centers around this.
I'd like to invite you to join our HydraNPM Slack group where you can connect directly with others using Hydra. More info here: https://github.com/flywheelsports/hydra#join-us-on-slack
sjmcdowall
First -- loving the project. We are looking at this instead of a low-level Redis/Kue for implementing our microservices. However, I do have a question (or two) as it comes to our desires (and maybe best practices tips!) ..
We have a bunch of "work queues" for our services -- things that can take some time to process and the more often than not -- invoke another step in the "recipe" of what needs to be done. I see a bit of documentation on "Message Queues" but it's a bit vague per se and I don't really see any examples?
Also -- part of the "Health Check" our users want to see is the length of any particular "Message Queues" so we can see if they are growing and hence, maybe we need for service instances -- or if they idle a long time and thus maybe reduce the service instances. But I don't see where we can get the current message queue length?!
Also -- once a job is completed (either ok or error) and we mark is so .. how / when does it get removed from the "Queue" so it doesn't build up to "infinity"?
Also is there some examples on how to setup a good Redis cluster and how to use it? We most certainly want a HA environment and having a single Redis node appears to be a Single Point of Failure.. ?
Is there the idea of routing of messages based on "advanced" criteria other than just service name? Use case is this -- we have a "file processor" that .. no surprises ..processes files .. however, the physical location of the file may only be available some of the service instances (local to a single machine for example) etc. Thus the service is still "file-processor" but I would like a clean way to indicate there maybe constraints on to WHICH file-processor is suitable..
Since this is nodeJS (for us anyway) -- is there a way to set / limit the amount of concurrency a single instance can handle similar to Kue's "Processing Concurrency" argument to queue.process() ?
Are there best practices on how to "share" in an installation common needed configuration information on deployment (versus development). We will have an average of 6-10 compute nodes per install (at different customers) and hand configuring the Redis stuff maybe a PITA .. Just wondering how other people may have solved this ..
TIA and I'm excited about going forward ..