search

mateodelnorte / servicebus

Simple service bus for sending events between processes using amqp.
MIT License
420 stars 66 forks source link

Support queues other than RabbitMQ #61

Open pawelbaranski opened 8 years ago

pawelbaranski commented 8 years ago

Hi,

Would you consider adding different queues than RabbitMQ ? Is there a way to easily replace RabbitMQ with Kafka, ZeroMQ, etc... ? I am having an issue where I'd like to use servicebus, bus our technology stack may not use RabbitMQ in a long run

mateodelnorte commented 8 years ago

I started a refactor to do just that, which is why there are two bus.js files and a rabbitmq folder, but ran out of time. More than happy to accept pull requests. ;) On Jan 14, 2016 7:09 AM, "Paweł Barański" notifications@github.com wrote:

Hi,

Would you consider adding different queues than RabbitMQ ? Is there a way to easily replace RabbitMQ with Kafka, ZeroMQ, etc... ? I am having an issue where I'd like to use servicebus, bus our technology stack may not use RabbitMQ in a long run

— Reply to this email directly or view it on GitHub https://github.com/mateodelnorte/servicebus/issues/61.

pawelbaranski commented 8 years ago

I would be happy to help, but I am just starting to improve my NodeJS skills. Maybe later :)

emadb commented 8 years ago

@mateodelnorte do you have an idea on how to implement this functionality (decoupling bus from transport)? I mean which refactoring steps are needed to have it I could try to help you.

mateodelnorte commented 8 years ago

Hi @emadb. Happy to have the help!

The first step will be modifying https://github.com/mateodelnorte/servicebus/blob/master/bus/index.js#L3 to accept a transport type and creating an implementation for redis (or whatever the provider will be) that will implement the same features that https://github.com/mateodelnorte/servicebus/blob/master/bus/rabbitmq/bus.js implements. The bus() function will then instantiate the appropriate transport and functionality should continue from there.

To provide a standard interface to implement, send+listen+publish+subscribe+close from https://github.com/mateodelnorte/servicebus/blob/master/bus/rabbitmq/bus.js should probably all have stubs implemented in https://github.com/mateodelnorte/servicebus/blob/master/bus/bus.js which all providers must implement, else they throw a new Error('bla is not implemented)' error.

That should get us to the point where basic functionality works, then we can hack through getting any additional things working like servicebus-retry, etc.

Would be happy to help with all of this.

emadb commented 8 years ago

Ok @mateodelnorte. I will give a try, I can't assure you to complete all the task but as soon as I have some free time I would like to help. One thing that I need is an in-memory bus, primarily for testing purpose and for dev scenario where you don't need (yet) a real bus. Having a redis implementation is also useful!

mateodelnorte commented 8 years ago

In memory could be performed either by zeromq or a combination of UDP and TCP messages. Zeromq would be the easier easier way to go, but introduces the single dependency.

If what you want to do is unit test your handlers / modules that use service bus, look at using something like mockrequire On Apr 12, 2016 9:42 AM, "Emanuele DelBono" notifications@github.com wrote:

Ok @mateodelnorte https://github.com/mateodelnorte. I will give a try, I can't assure you to complete all the task but as soon as I have some free time I would like to help. One thing that I need is an in-memory bus, primarily for testing purpose and for dev scenario where you don't need (yet) a real bus. Having a redis implementation is also useful!

— You are receiving this because you were mentioned. Reply to this email directly or view it on GitHub https://github.com/mateodelnorte/servicebus/issues/61#issuecomment-208910944

emadb commented 8 years ago

I implemented a first implementation of a FakeBus. Most methods throw an error (I just implemented pub-sub for testing). By now I pushed my code here: https://github.com/emadb/servicebus/commits/master

I tried to keep the old tests green and the default implementation is still on Rabbit, I simply added an optional argument on the constructor. Let me know if it is a good starting point or not :-)

archa347 commented 8 years ago

Are similar modifications needed to work with other AMQP compliant systems, or is this module truly RabbitMQ specific?

mateodelnorte commented 8 years ago

I haven't tried it on any vendors other than rabbitmq. Happy to merge PR's for additional brokers or transports.

mateodelnorte commented 8 years ago

@emadb I responded in your repo, but I'll parrot here.

A fake bus isn't beneficial here, as an external mocking framework can provide that - and having a fake bus internal to the servicebus library would only help in mocking out single process scenarios, where sb is most widely used in creating distributed systems.

archa347 commented 8 years ago

Alright, my company has decided to invest its time in ActiveMQ, which supports AMQP as well. I'll have to try it out and I'll let you know what comes up

emadb commented 8 years ago

@mateodelnorte Yes. I saw your response. I replied here:

Hi @mateodelnorte I totally agree with you. I was just trying to create a fake bus for "playing" with it and see if everything works fine. FakeBus is not to meant to stay there. The point on which I would like to have some feedback is the "factory method" to which you can pass the bus implementation to use: https://github.com/emadb/servicebus/blob/master/bus/index.js#L4

Is this the idea that you have in mind? If it's ok I go on on cleaning up the code, write some docs add some tests and..of course removing the FakeBus :-)

mateodelnorte commented 8 years ago

Hi @archa347 what were your results? I'm guessing we're far too tightly coupled to RabbiqMQ's AMQP specification to be simple plug-and-play. The AMQP standard is, in and of itself, all over the place with vendor implementations being much tighter than the standard.

bradoyler commented 8 years ago

@mateodelnorte Foresee any major issues with swapping in this package? https://github.com/markbirbeck/amqp-sqs

mateodelnorte commented 8 years ago

Interesting. I think that wrapper goes half way, and covers send/listen. It looks like for publish/subscribe and fanout you'd need to also implement a wrapper on top of SNS. Search for 'fanout' here: https://aws.amazon.com/sqs/details/.

Theoretically, though, I think it could be done.

I've started doing some experimental work with making servicebus use a provider model for transports: https://github.com/mateodelnorte/servicebus-rabbitmq I'm still finicking with the best way to separate the required modules into different packages without leaving servicebus, itself, as a mostly empty shell.

I think what would be required is making something like servicebus-rabbitmq (or whatever interface it ends up boiling down to) that implements amqp-sqs and a corollary to fanout/topic routing, perhaps being a wrapper around SNS. Something like https://www.npmjs.com/package/aws-sns-publish might help do the trick.

patrickleet commented 5 years ago

this is in progress in servicebus org - kafkabus on it's way