stevelr
commented
1 year ago please bear with the the long-ish response. I want to articulate some goals, so if there's a difference in perspective we can address that before getting stuck in the details of apis.
The difference between basic- and stream- seems to be a difference in metadata associated with the message.
- both include subject (aka entity), format (aka type), and optional metadata
stream-also includes partition and offset
Is it possible to combine and simplify so that we end up with only one type of producer and one type of consumer?
If we unwrap all the layers, the full set of metadata considered by the various send/receive apis here is
- message data (
list<u8>) - subject (
string) - type, e.g., mime type (
string) - partition (
u32) - offset (
u32) - optional metadata (
list<tuple<...>>)
Sender and Receiver objects embody the subject name. I think that abstraction adds an unnecessary layer of complexity. Some messaging systems (at least NATS and MQTT, possibly others) permit subscribing to a set of topic with wildcards, so subject needs to be delivered with each message. In other words, in the general api, a subscriber can't receive a message without also receiving its subject. Similarly, a single producer can often transmit messages on multiple subjects. In both cases, if subject were a string parameter of send and receive, it would simplify usage and remove the need for the sender and receiver objects.
NATS, PubSub, an MQTTv5 include optional attributes (aka headers or properties) with each message so an optional tuple of properties also seems warranted, for both send() and receive()
The format parameter is sometimes useful. A few are enumerated: MQTT, HTTP, etc. If a format parameter is part of the api, it needs a 'raw' variant as a catch-all if none of the other formats applies. Perhaps a mime type would be more inclusive and handle even more variants. To decide whether it should be included in the send/receive signatures (or a message struct), and incurring the complexity and expense, we want to know how often it's needed. In most cases I've seen, the format is effectively communicated by the subject name and the documentation for the message stream. For example, if you're watching for changes to a database based on a set of keys, or watching a file system for changes to a file, the db and fs specify the format of the event. It's often specified along with the subject name of the event stream, in which case the publisher and subscriber already know the data format.
There are always tacit aspects of a message protocol that are part of its spec, in the documentation. If the implementers of the publisher and consumer have the same understanding of the protocol, and it doesn't very per message, it doesn't need to be sent with each message. Other protocol attributes that are often not explicit fields in the message are compression algorithm and encryption algorithm. In cases where additional or optional metadata is required to describe format, or other message attributes, the format can be included in a property map with other metadata (for example, the Content-Type value in the headers map in an HttpRequest), or the format and other metadata can be put into a wrapper object that becomes the new message (for example, a CloudEvents message has explicit metadata with a data field for the inner payload). If, in the vast majority of cases, format is tacit, or it can be included in metadata or the message object, it doesn't need its own field in the wasi messaging interface, either as a field of message or as a function parameter.
That leaves partition and offset. Although they are used with Kafka, and Kafka may be a common expected use case, it's not needed for a lot of other implementations, and harder to justify for inclusion in a common api used by all implementations. Every user of this api would have to go through a decision process on whether to interpret the offset parameter as a sequence, and wonder if they need to implement message reordering. If our goal is to make it easy to create lightweight WASM components, we don't want users to even have to think about message reordering - it should be in a service layer.
How can we enable WASM developers to use Kafka partitions, and QOS, and TTL, and other variations of messaging services that might be critical to their needs? In the short term, we have ways to handle variations: on the producer side, create components that accept additional service-specific parameters and pass a wrapper object as the message to the basic interface. On the consumer side, use a component that decodes the wrapper object and calls a subscriber that knows how to handle the extra parameters for receive. Similarly, wrappers can be created when subscribing, or when obtaining a broker handle.
I'm hopeful that the initial set of wasi interfaces can have a high enough level of abstraction to be broadly useful and simple to use, where api designers apply a high bar before deciding to add parameters that increase complexity for all users. Ideally, keeping these shared wasi interfaces simple will make them broadly applicable, and help adoption by shorten the learning curve for all developers.
One way to future-proof the basic api would be to allow config parameters when opening a broker. That could be an escape hatch used by wrapper components that need to pass implementation-specific parameters into the basic interface.
To summarize, I propose:
- combining basic-producer, basic-consumer , stream-producer, stream-consumer into
producer,consumer - combining sender, receiver into string, and including subject in each invocation of
sendandreceive(as a function parameter or field of themessageobject) - removing
partition,offset
This leave us with something like this:
interface broker{
open(config: option<metadata>) -> result<broker, error>
publish() -> result<producer, error>
/// subscribe to a subject or wildcard set of subjects
subscribe(subject: string) -> result<consumer, error>
}
interface producer {
use self.types.{error, message}
send(message: message) -> result<_, error>
}
interface consumer {
use self.types.{error, message}
receive() -> result<message, error>
receive_timeout(timeout: timeout-ms) -> result<message, error>
}
interface types {
record message {
subject: string,
data: list<u8>,
metadata: option<metadata>,
}
type metadata = list<tuple<string, string>>
/// timeout in milliseconds
type timeout-ms = u32
variant error {
timeout,
...
}
}
autodidaddict
danbugs
clemensv
guest271314
This PR contains changes across the all world interfaces and focuses to address feedback from:
Thank you everyone for your help and patience w/ this big refactor.
Below, I'll highlight the major changes to this interface. To see it actually being used w/ a dummy implementation, please refer to: https://github.com/danbugs/wasi-messaging-demo .
typesreceiver, andsenderhave been unified intoclient, and theclient's only function is to connect to some sort of named message exchange service.typealias to a string as the rest are consumer issues.guest-configurationrecordthat contains a list of subscriptions that a specific guest owns, and an optional key-value pair list of 'extensions', which can contain specific config required for a particular implementor (e.g., partitions/offsets, etc..)format-specis now anenumrather than astringtypealias to ensure implementors exhaustively support/address different types of data envelopes.messageno longer enforces CloudEvents. Instead, it's arecordwith a binary payload andcontent-type-type field (i.e.,format) for deserialization.consumerguest-configurationand its' capability to allows us to set specific partitions/offsets to read from,stream-consumerandbasic-consumerwere unified into a singleconsumerinterface.subscribe-try-receive/subscribe-receivemethods to fulfill therequest-replypattern – they aim to create an ephemeral subscription to receive from a channel until a specific timeout/indefinitely, respectively. Also, as for some implementors it's uncommon to return a single message, both methods returnlist<message>.update-guest-configuration, which allows for things like: setting a specific partition/offset to read messages from in the case of Kafka/EventHubs, or even just unsubscribe. It's meant to be a fit-all solution for any implementor-specific quirks to be addressed.guest-configurationproducerstream-producerandbasic-producerwere unified into a singleproducer.consumer,sendtakeslist<message>.guestconfigureto obtain a list of implementor-specific configurations (if needed), and a list of channels to subscribe and create a long-running client connection on the host side. At this point, the host can kill this component instance. (2) Once any messages come in for the subscriptions returned byconfigure, the host can spawn up a new component and call theguest'shandlerto take care of the incoming message. Equal to the motivations presented inconsumerandproducer,handlerreceives alist<message>.