Current interface does not support a number of important use cases

[autodidaddict]

I wasn't sure whether to put all of the use cases in their own issues or bundle it up. I can always split this one later if people want. The core of this issue is that the current way in which wasi-messaging is designed (by current I mean what I can see in main, I'm not sure if there's work going on elsewhere) only supports a tiny fraction of the use cases required by application developers interacting with messaging systems.

Do not mandate Cloud Events

Forcing the publisher of messages to only ever use cloud events is unnecessarily prohibitive. Only a small fraction of publications on message brokers are actually events, and an even smaller portion of that ever really need the shape of a cloud event. I understand the desire for an envelope, but I don't think it's the responsibility of this interface to enforce the envelope shape. When we get into requests (below), the mandate of a cloud event switches from inconvenient to a blocking impedance mismatch.

Request/Reply

The current consumer interface only supports subscribe and unsubscribe. Consumers of message brokers need the ability to make a request and await a reply. This is perfectly normal to do synchronously so it wouldn't need to be done in collaboration with async/threads. I suggest supporting a call that looks like this:

request: func(b: broker, topic: string, data: list<u8>) -> result<list<u8>, error>
or...
request: func(b: broker, topic: string, timeout_ms: option<u8>, data: list<u8>) -> result<list<u8>, error>

As a convenience, we should probably give people the ability to make a request that expects multiple replies rather than just one. This is a basic necessity for message brokers that facilitates patterns like map/reduce, scatter/gather, polling, etc.

request-many: func(b: broker, topic: string, timeout_ms: option<u8>, max_results: option<u8>, data: list<u8>) -> result<list<list<u8>>, error>

The channel type doesn't feel appropriate here because that is an enum that distinguishes between queue and non-queue subscriptions. Making a request on a topic should never involve explicit knowledge if the target(s) are consuming via queue or not. In other words, as mentioned below, publication should be done on a topic, not a channel. The publisher/requester in a system should not have special knowledge of how that message will be consumed.

Publish should be on a topic, not a channel

Depending on which message brokers people have used in the past, people come with preconceptions about what consumers and producers should know about in advance. As a general rule, a publisher or requester should have no knowledge of the target of the message beyond that of the topic on which it is published. Whether there is a queue consumer, or multiple queue consumers, durable streams, or anything else listening on that topic should be unknown to the client.

Consumer and Producer Split is Confusing

In an environment where a component may either publish or request, is that component utilizing a consumer or a publisher interface? I think this split forces artificial barriers that are likely to confuse people rather than help them. A request in most brokers is a publish with a reply-to topic, so that actually combines both consumer and producer.

Naming things is hard, but I think a more intuitive interface split might be handler (receives messages) and client (makes requests, publishes messages, subscribes, unsubscribes).

[danbugs]

Thank you opening this issue, @autodidaddict ~ This is really valuable feedback. I've been recently getting in touch w/ messaging stakeholders, and, from that, I'm currently working on a significant re-write of the interface. Once that PR is in, I'll make sure to ping you for review, so we can ensure your concerns are addressed or that, at the very least, we have a plan for addressing them moving forward.

[autodidaddict]

How goes the interface refactor? If I can help in any way, even if it's to bounce ideas off of, let me know.

[danbugs]

How goes the interface refactor? If I can help in any way, even if it's to bounce ideas off of, let me know.

I'm hoping to have a PR for review by (hopefully) the end of this month, and I'd really appreciate your review then 👍 – might scrap the old 'stakeholder review' PR draft, and start a new one w/ feedback I got here from you, Steve, Clemens, and privately!

[clemensv]

I believe adding request/reply correlation into a messaging abstraction were a huge mistake because it favors one of many different flow patterns and complicates the implementation for everyone. If you add request/reply, why not add scatter/gather or continuous receives based on a trigger (like a subscribe gesture in MQTT)? Does request/reply run over two distinct paths or (as in MQTT) over a single connection? How does request/reply even work if the target is a topic and the requesting party doesn't (as it should) not know how many parties might reply? What do we do with a request/reply path on an interface in a system where the target is a click-stream telemetry system?

Request/reply is an app-level correlation activity. If one wants to have platform support for request/reply, layer some RPC framework on top of the messaging abstraction.

Let's not pollute the foundation.

[autodidaddict]

This is a great point. I've been thinking in terms of what functionality I wanted abstracted for me, but it isn't the wasi-messaging standard's job to abstract everything, just the most basic primitives. I was hoping in my suggestions here to make it so that the only code I would need to use is that generated by wit-bindgen. If we keep requesst/reply, ack/nack, gather out of the interface then everyone that generates their wit bindings will also need to do a little bit of ceremony to wrap these patterns around the primitives.

[danbugs]

As per my latest comment in #9 (i.e., https://github.com/WebAssembly/wasi-messaging/pull/9#issuecomment-1597808086), let me know if you're ok w/ me closing this, so we can create more individualized issues and to aid creating better targeted PRs, @autodidaddict .

[danbugs]

@autodidaddict ~ W/ the approaching merge of #9 , I'm closing this issue for now. If you find any of the things you outlined here are still not covered, I'd really appreciate it if you could open separate issues to help w/ making more targeted PRs.