pubsub: generic type for asynchronous messaging

[zombiezen]

A common feature request is asynchronous messaging, like Pub/Sub. More research and design work is required to know exactly what this API should look like, but any design should work for (not necessarily saying it will be implemented):

• Amazon SQS
• Google Cloud Pub/Sub
• Azure Service Bus
• Open source standalone: RabbitMQ, Kafka, Redis

(Worth noting that the Contribute Bot proposed in #216 uses Pub/Sub as well: design doc.)

[asim]

We abstract pubsub as a Broker interface in go-micro which includes support for google pubsub, amazon sqs, kafka, redis, rabbitmq, nsq, nats, etc. It may help you understand how you want to start this https://github.com/micro/go-micro/blob/master/broker/broker.go

[ijt]

@zombiezen, nit: pubsub on AWS is SNS (simple notification service), sometimes combined with SQS (simple queuing service).

[ijt]

@asim, I'm not sure I understand how Broker in go-micro supports all the backends you listed. Can you say a bit more about how that works?

[asim]

See code in micro/go-plugins repo. Broker is an interface which provides a high level abstraction for pubsub. Go Plugins contains implementations of that interface for various messaging systems. These can be used anywhere a broker interface is accepted. You do lose some unique per message provider features but it also creates a simple pluggable messaging layer.

I imagine you guys will likely do something a bit different more similar to Blob and it's setup code.

[ijt]

Thanks @asim!

[ijt]

@asim, I'm surprised that SQS is used as a backend by itself since SQS is just a queuing service, not a pubsub service. I would have thought it would be a combination of SNS + SQS. How does this work?

[asim]

Anything which you can send a message to or receive from can be used for pubsub. Queue URL is used as the topic. The combination of SNS and SQS is not required but does provide a more feature rich option.

[stephnr]

I agree with @asim's point here. Forgive me for being the random new guy to jump in on this but I can chime in from an AWS perspective and would say that SQS falls into the umbrella of Pub/Sub systems. However, the action of "subscribing" to updates from SQS isn't as intuitive. Rather, you would need to do some kind of long-polling for info.

FYI: I would love to help in any way on this one. Long time lurker, eager to jump in and help <3.

[ijt]

I'm still skeptical about this. I want very simple semantics for pubsub on Go Cloud: when a message is published to a topic, all the subscribers to that topic get exactly one copy of that message. It may make sense to add a queue package to Go Cloud though.

[asim]

@ijt you need to think about messaging semantics. It's very unlikely you can build a library abstraction for pubsub without addressing the core building blocks. Think of the complexities of the other packages you offer. Queuing is handled by basically every messaging system. You may also need to consider message acking.

If the idea is to provide simple broadcasting then you'll need to be very clear about that upfront and maybe even reconsider the naming so as not to confuse people.

[ijt]

@asim to be clear, I don't intend Go Cloud pubsub to expose a push-style pubsub system like SNS. The plan is for it to work more like a pull-style pubsub system with subscription queues like GCP PubSub, SNS+SQS (https://www.youtube.com/watch?v=c_WNBmEc6EE), Azure Service Bus, or Kafka.

I like how AWS has provided broadcasting (SNS) and queueing (SQS) services ala carte in a composable way. At the same time, that doesn't appear to be the way most pubsub systems are structured.

[asim]

I think from an implementation standpoint it doesn't really matter because if my assumption is correct the plan is to support a variety of existing pubsub solutions in the same way as other packages here support other underlying systems. In which case the real design requirement is around the high level abstraction. Please see the go-micro broker to understand my ideas.

From my perspective you have something with methods Publish and Subscribe where both would take a topic and subscribe would optionally take a channel or queue name which would be used by the underlying implementation for whatever form of distribution it does for subscribers.

To be clear though, I don't personally feel strongly about whatever choices the Go Cloud team makes. I'm only voicing my opinion due to an initial conversation with Eno. I'll leave you to it now. Thanks.

[zombiezen]

@asim We really appreciate the input! @ijt is working on an initial design draft now. We want to review it a bit inside the team first before publishing. We'll post the link on this issue thread when it's ready. Once we have it up, if you see any pitfalls or have suggestions, we would very much appreciate the feedback.

[vangent]

The in-progress API is here: https://github.com/google/go-cloud/blob/master/internal/pubsub/pubsub.go

[ijt]

The pubsub subpackage has been added.