Support Kafka as an additional transport

[rhinof]

@vladshub @adiweiss @avigailberger @danielwitz @yuvmendel thoughts?

[rhinof]

@Prajna you can configure a kafka producer to wait for an ack from the elected leader or wait for akcs from all brokers in a replica (or none for non-resilient producing) https://kafka.apache.org/documentation/#producerconfigs On the consumer side it is just a matter of incrementing the consumers offset

[vladshub]

I think this could add a lot of value, and clean our code as well.

[rhinof]

@vladshub I agree, it will force us to decouple our message handling code from rabbitmq specifics that will promote for a better design and in addition provide kafka as a transport which will provide for a lot of value

[rhinof]

@vladshub I think the challenge and where we need to think about the correct abstractions is how we bridge the conceptual gap between rabbit and kafka in respect to what is the unit of parallelism. while in rabbit it is the consumer and in kafka it is the topic/partition.

[vladshub]

@vladshub I think the challenge and where we need to think about the correct abstractions is how we bridge the conceptual gap between rabbit and kafka in respect to what is the unit of parallelism. while in rabbit it is the consumer and in kafka it is the topic/partition.

I agree with you, however, I think it should be "hidden" in the implementation of the transport. The first stage we should decide on the interface which I started writing in PR #230 which should abstract the underlying implementation and we should decide if we have one transport instance which receives all of the messages/instances and then fans them out to all of the grabbit workers or should we have an instance of transport for each grabbit worker. Once that is done and we cleaned up the code from Rabbit "notions" it should be easy to implement the Kafka transport and in the future any additional. :)

[vladshub]

@rhinof also I was thinking about a transport registry which should look like the database/sql implementation in go

import "github.com/wework/grabbit/gbus"
import _ "github.com/vladshub/grabbit-kafka"
.
.
.

This way we would allow the addition of new transports seamlessly, and maybe it will be easier to adopt.

then we would be able to have things like:

url = "amqp://user:pass@host:port/vhost"
url = "nats://user:pass@host:port/vhost"
url = "kafka://user:pass@host:port/vhost"
url = "nsq://user:pass@host:port/vhost"
...

[rhinof]

@rhinof also I was thinking about a transport registry which should look like the database/sql implementation in go

import "github.com/wework/grabbit/gbus"
import _ "github.com/vladshub/grabbit-kafka"
.
.
.

This way we would allow the addition of new transports seamlessly, and maybe it will be easier to adopt.

then we would be able to have things like:

url = "amqp://user:pass@host:port/vhost"
url = "nats://user:pass@host:port/vhost"
url = "kafka://user:pass@host:port/vhost"
url = "nsq://user:pass@host:port/vhost"
...

I like the notion of following the driver model for hooking new transports however I still do not have a clear enough opinion on what would be the better approach to bridging the conceptual gaps between a broker model as rabbitmq and a distributed log like kafka (push vs pull, different units of parallelism etc,,)

[vladshub]

@rhinof , you are right that is why my suggestion is to start with just moving and abstracting away our current rabbit implementation, once it's in one place we can look at the 2 different approaches and see where this responsibility should lay in and how to model it.

[rhinof]

@vladshub ok, makes sense starting with something rough and refining along the way. I created the experimental-kafka branch (branched off of master) so we could work on