zpetterd / feathers-rabbitmq

7 stars 0 forks source link

Possible use cases #1

Open zpetterd opened 7 years ago

zpetterd commented 7 years ago

Currently this works in a very specific way, which is what I needed but I could see it being quite useful to other people.

In the /bin folder is a example of how to use it at the moment.

The problem I was trying to solve is using FeathersJS in a microservices environment were there is possibly multiple instances of a service running. Currently using the SocketIO provider all instances of the service would receive the events and as such process them. The idea here is that a server sets up a number of topics and then each service has a number of queues that are used for subscription to the events. A service client is identified by its appName.

One other advantage of this is reliable messaging in that if the client isn't connected RabbitMQ can be configured such that the pending messages are available when the client rejoins.

Is there other use cases that would be useful to cover and make possible?

idibidiart commented 7 years ago

@zapur1

this is great!

I'm trying to understand how your scheme would work and being new to Feathers I could use a little more clarity.

How is a 'Microservices setting' different than the scaling scheme currently in use (I believe it's based on node clusters) ... Why is the problem you describe unique to Microservices setting? I suppose if you're load balancing between clusters there is only one instance of each service in each cluster. I'm vague on how you structure the Feathers server and the individual services (that normally run within the Feathers server) in a Microservices settings? Could you describe it more clearly?

zpetterd commented 7 years ago

Certainly can, imagine you have a series of instances of Feathers like this

Something to note, in this case Service A doesn't directly correlate to a Feathers app service, it may have multiple service endpoints. In this case if all the services (A and B) are connected to API app, when API app emits a "created" event every every service instance will receive the event. That means if Service A was setup to always take a action when it receives a "created" event it would be duplicated across both of the A service instances.

The idea with what has been implemented in this provider is to use RabbitMQ to ensure only one of Service A #1 or Service A #2 receive the event. Only once delivery of events within a namespace is the exact problem I was trying to solve.

This could also be used for other situations were you still want the normal behaviour of all clients receiving events but you just want the reliability and scaling in messaging that comes from a messaging queue

idibidiart commented 7 years ago

I don't understand why services would consume mutation notifications from Feathers services? Those are meant for clients to update their state, not for other services to consume. If you want something to happen in an external system you should use after-hooks. If you want something to happen to Feathers connected data store then listen to the mutation event in client and create another mutation via whatever Feathers service.

No doubt a message queue is useful in case of Mircoservices, e.g. event sourcing. But it probably needs order guarantees like Kafka.

zpetterd commented 7 years ago

I'm not real sure if you fully understand the use case for this. Using after hooks would require them to be added to each service along with the fact that they would add a blocking action to the after hooks being complete.

The idea is that the event emitter simply emits all events and the consumer makes decisions about how it would like to receive those events. Kafka would be better in regards to receive and processing the messages but is much harder to implement in regards to infrastructure compared to RabbitMQ's simplicity.

zpetterd commented 7 years ago

Implementing this as a provider also allows RabbitMQ to be used for RPC

idibidiart commented 7 years ago

I thought the blocking action is the mutation and the after hook can fire and forget (call some async API, get Ack instantly, and move on... are after hooks not capable of doing that?) ... Can you describe your use case.

There is definitely a need for a messaging bus in a microservices arch. Just not sure of the detailed scheme. Can you educate us with diagrams and/or blog post?

kethan commented 7 years ago

Can I use java or python as a webservice for this ? Microservice with different programming language?

zpetterd commented 7 years ago

@kethan This is designed to be used with docs.feathersjs.com. Although it doesn't matter what other clients you use with RabbitMQ to receive the events.