magicmatatjahu
commented
3 years ago @fmvilas Awesome proposal! I love it ❤️ However, I have some problems with some parts of this proposal, maybe some things are obvious for you and for other people, but some things surprise me and I think their assumptions are wrong.
The main problem - message(s) in the channel
Why do you want to define the message(s) in the channel, not as currently in the operation level? Consider the first example you gave:
asyncapi: 3.0.0
channels:
userSignedUp:
address: user/signedup
message:
payload:
type: object
properties:
email:
type: string
format: email
operations:
sendUserSignedUpEvent:
channel: userSignedUp
action: send- How can I guess that by sending a given message the application underneath will process the
sendUserSignedUpEventoperation? This problem is more pronounced in the case ofOverride channel messages with operation-specific messagescase. - Go ahead. In other examples you defined on the channel the
servers(andremotesfield) with names of the existing servers. I guess that given channel is only available for particular server/remote. Ok... but what in case when someone want to define multiple operations (withreceivekind) for the different servers? - What in case when for given channel you wanna define
sendandreceiveoperation (we have still possibility in the2.X.XAsyncAPI). If you define the message asoneOf, then how to guess which message is forsendand which forreceive?
I think that in this case, message(s) should be only defined on the operation, not on the channel. Channels should be then treated only as "interface", which you must "implement" by the operations (also if you need by multiple operations). For me, in the current proposal, the operation is "hardcoded" to the given channel by the message. If I'm not right, please clarify the message field on the channel, because it really doesn't fit for me.
Splitting servers to the two groups
I like idea to clarify that given server is remote and application is connecting to it and another server "runs" the app, but... did you consider having kind in the Server object? Then we don't need the remotes Object. Also as I see, the remotes have this same shape as servers, so kind: remote (or remote: true) is enough for me.
Servers and remotes in the channel
I mentioned about it the first section:
... In other examples you defined on the channel the
servers(andremotesfield) with names of the existing servers. I guess that given channel is only available for particular server/remote...
but I think that you should clarify this thing in the proposal, because I know that you copied this from existing PR, but people can be surprised 😄
Adding request/reply support
Here I'm just going to think out loud because I'm not an expert in this topic. Maybe instead of an extra action type like request and the replay field we should reuse receive action and add the word operations or then, that would determine what should happen when a given operation is processed and whether it "produces" subsequent operations? Something like, if the payment in the e-shop was successful, send (reply to the broker) an operation that saves record in the database - otherwise create another operation that does rollback etc. This way we could support many more techniques than just request/reply.
Example:
operations:
checkPayment:
channel: payment
action: receive
then:
2XX: ... # "reply" operation when given operations success.
4XX:... # "reply" operation when given operations fails.In the example above, I use HTTP statuses to describe the state of the operation: success or failure. Probably something better can be invented.
It seems more generic to me. The idea itself is taken from CQRS.
👏🏼 for the proposal. If we go along with this (and I think we will after a dozen or so revisions), then every tool will have to be re-written from scratch 😂
fmvilas
iancooper
jonaslagoni
jessemenning
dalelane
ekozynin
joerg-walter-de
Relequestual
damaru-inc
derberg
nictownsend
tristanpye
TL;DR: What's new?
kindproperty. By default, all servers areremoteservers (e.g., brokers). However, if thekindproperty is set tolocal, that means the application is the one actually exposing the server (e.g., WebSocket or HTTP Server-Sent Events server).channelsobject is optional. The only two fields that are required areasyncapiandinfo.serversandchannelscan be defined insidecomponentsfor reusability purposes. This way, an AsyncAPI document may only contain acomponentsobject that can serve as an organization-wide menu of servers, channels, messages, etc.Abstract
For some years now, we've been discussing how to solve the publish/subscribe perspective problem. Some were arguing we should be giving priority to broker-based architectures because "they are the most common type of architecture AsyncAPI is used for". Although they're not wrong, by simply changing the meaning of
publishandsubscribekeywords, we'd be leaving many people and use cases out. We have to come up with a way to fix this problem without losing the ability to define client/server interactions too, which are especially useful for public WebSockets and HTTP Server-Sent Events APIs. And more importantly, we need a structure that will allow us to grow well and meet our vision.The latest Thinking Out Loud episodes with @lornajane, @dalelane, @damaru-inc, and @jmenning-solace have been key to better understand the problem. The intent of this proposal is to be a mix of the ideas and feedback I got from the community. I hope I managed to capture them well 😊
Foundational concepts
We gotta start from the beginning: an AsyncAPI file represents an application. This doesn't change. However, if no
channelsoroperationsare provided, the file becomes a "menu", a "library", a collection of reusable items we can later use from other AsyncAPI files.We have been very focused on the meaning of
publishandsubscribebut there is another key aspect of the spec that is confusing to many people:servers. As it is right now, theserverskeyword holds an object with definitions of the servers (brokers) we may have to connect to. However, it also holds information about the server our application is exposing (HTTP server, WebSocket server, etc.) In some tools, we have been incorrectly assuming that if someone specifieswsas the protocol, it means they want to create a WebSocket server instead of a WebSocket client. But what if someone wants to connect to a broker using the WebSocket protocol? This whole thing about the role of our application has been confusing all of us. As @lornajane pointed out on multiple occasions, an application can be both a server and a client, or just a server, or just a client. Therefore,serverscan't be made up of that mix. Exposed server interfaces and remote servers (usually brokers) have to be separated because —even though they look the same— they're semantically different.Remotes vs Local Servers
This proposal introduces the concept of a
remoteorlocalserver. Remote servers are those our application has to connect to. They're usually brokers but can also be other kinds of servers.On the other hand,
localservers are server interfaces our application exposes. Their URL defines where clients can reach them.Example
New
channelsandoperationsobjectsAnother common concern related to the current state of
publishandsubscribeis the channel reusability problem we encounter because these verbs are part of the channel definition. To avoid this problem, we remove the operation verbs from the channel definitions and move them to their own root objectoperations. Let's have a look at an example:There are a few new things here:
address: it's the logical address where you can find this channel. Usually, this is the topic name, the routing key, the URL path, etc.userSignedUpthen? This is the channel identifier. It's an identifier that serves to reference the channel from another part of the document or another document.user/signedupusing JSON Pointer would beuser~1signedupas opposed touser/signeduplike many people would think. That's highly unreadable and error-prone, therefore I'm introducing channel identifiers.channelhints against which channel is this operation performed andactionis the type of operation, i.e., we're "sending" or "receiving".sendUserSignedUpEvent? It's the operation identifier. Yes, the oldoperationIdis now mandatory and it's implicit, i.e., theoperationIdfield doesn't exist anymore.channela$refor JSON Pointer? To keep things simple. If you're referring to a channel here, it must be defined in thechannelsobject. If we allow$refhere, it means it can be dereferenced and therefore the channel ID would be lost. To make things easier and more consistent, this field is simply a string with the name of the channel ID.sendandreceive? No, it's not that I'm hatingpublishandsubscribealready 😅 I'm usingsendandreceivehere to avoid confusion for those thinking that AsyncAPI is only meant to describe pub/sub architectures. I think "send" and "receive" are pretty common verbs that shouldn't be linked to any super-very-special meaning.Organization and reusability at its best
I'm adding two new objects to the
componentsobject:serversandchannels. Some may be already wondering "why? don't we haveserversandchannelsalready in the root of the document?". To understand this decision, I think it's better if I just describe the reusability model I have in mind.Reusability
Some people expressed their interest in having a "menu" of channels, messages, servers, etc. They're not really interested in a specific application. In other words, they want an organization-wide "library". This is the
componentsobject. It's now possible to do something like the following:This is now a valid AsyncAPI file and it can be referenced from other AsyncAPI files:
As you can see, I'm not defining any channel or server in this file but instead, I'm pointing to their definitions in the org-wide document (
common.asyncapi.yaml). And this leads me to the other part of this section: "organization".Organization
The example above shows how we explicitly reference the resources (servers and channels) that our application is using. Having them inside
componentsdoesn't mean they are making any use of it. And that's key. The file is split into two "sections": application-specific and reusable items. Let's see an example:So for those of you who were wondering before "why? don't we have
serversandchannelsalready in the root of the document?": This is the reason, it's no different than it's right now in v2.x but I thought I'd make it more clear this time. Just because we put something incomponentsdoesn't mean the application is making any use of it.Life is better with examples
Microservices
Say we have a social network, a very basic one. We want to have a website, a backend WebSocket server that sends and receives events for the UI to update in real-time, a message broker, and some other services subscribed to some topics in the broker.
We'll define everything that's common to some or all the applications in a file called
common.asyncapi.yaml. Then, each application is going to be defined in its own AsyncAPI file, following the template{app-name}.asyncapi.yaml.The
```yaml asyncapi: 3.0.0 info: title: Organization-wide stuff version: 0.1.0 components: servers: websiteWebSocketServer: url: ws://mycompany.com/ws protocol: ws mosquitto: url: mqtt://test.mosquitto.org protocol: mqtt channels: commentLiked: address: comment/liked message: ... likeComment: address: likeComment message: ... commentLikesCountChanged: address: comment/{commentId}/changed message: ... updateCommentLikes: address: updateCommentLikes message: ... ```common.asyncapi.yamlfileThe
```yaml asyncapi: 3.0.0 info: title: Website Backend version: 1.0.0 servers: websiteWebSocketServer: $ref: 'common.asyncapi.yaml#/components/servers/websiteWebSocketServer' mosquitto: $ref: 'common.asyncapi.yaml#/components/servers/mosquitto' channels: commentLiked: $ref: 'common.asyncapi.yaml#/components/channels/commentLiked' likeComment: $ref: 'common.asyncapi.yaml#/components/channels/likeComment' commentLikesCountChanged: $ref: 'common.asyncapi.yaml#/components/channels/commentLikesCountChanged' updateCommentLikes: $ref: 'common.asyncapi.yaml#/components/channels/updateCommentLikes' operations: onCommentLike: action: receive channel: likeComment description: When a comment like is received from the frontend. onCommentLikesCountChange: action: receive channel: commentLikesCountChanged description: When an event from the broker arrives telling us to update the comment likes count on the frontend. sendCommentLikesUpdate: action: send channel: updateCommentLikes description: Update comment likes count in the frontend. sendCommentLiked: action: send channel: commentLiked description: Notify all the services that a comment has been liked. ```backend.asyncapi.yamlfileThe
```yaml asyncapi: 3.0.0 info: title: Website WebSocket Client version: 1.0.0 servers: websiteWebSocketServer: $ref: 'common.asyncapi.yaml#/components/servers/websiteWebSocketServer' channels: likeComment: $ref: 'common.asyncapi.yaml#/components/channels/likeComment' updateCommentLikes: $ref: 'common.asyncapi.yaml#/components/channels/updateCommentLikes' operations: sendCommentLike: action: send channel: likeComment description: Notify the backend that a comment has been liked. onCommentLikesUpdate: action: receive channel: updateCommentLikes description: Update the UI when the comment likes count is updated. ```frontend.asyncapi.yamlfileThe
```yaml asyncapi: 3.0.0 info: title: Notifications Service version: 1.0.0 servers: mosquitto: $ref: 'common.asyncapi.yaml#/components/servers/mosquitto' channels: commentLiked: $ref: 'common.asyncapi.yaml#/components/channels/commentLiked' operations: onCommentLiked: action: receive channel: commentLiked description: When a "comment has been liked" message is received, it sends an SMS or push notification to the author. ```notifications-service.asyncapi.yamlfileThe
```yaml asyncapi: 3.0.0 info: title: Comments Service version: 1.0.0 description: This service is in charge of processing all the events related to comments. servers: mosquitto: $ref: 'common.asyncapi.yaml#/components/servers/mosquitto' channels: commentLiked: $ref: 'common.asyncapi.yaml#/components/channels/commentLiked' commentLikesCountChanged: $ref: 'common.asyncapi.yaml#/components/channels/updateCommentLikes' operations: onCommentLiked: action: receive channel: commentLiked description: Updates the likes count in the database and sends the new count to the broker. sendCommentLikesUpdate: action: send channel: commentLikesCountChanged description: Sends the new count to the broker after it has been updated in the database. ```comments-service.asyncapi.yamlfilePublic-facing API
Another common use case for AsyncAPI is to provide a definition of a public-facing API. Examples of this are Slack, Gitter, and Gemini.
This would work differently than it is now. Instead of defining the server as we do with OpenAPI (and AsyncAPI v2), we'd have to define how a client would look like. For instance:
Currently, in v2, the spec lets you describe the server and infer the client from this description (as with OpenAPI). However, this causes a discrepancy with servers, especially in production systems. For instance, my server may be listening on port 5000 because it's behind a proxy. The client should be sending requests/messages to port 80 and they'll be forwarded by the proxy. If we define our port as 80 in the AsyncAPI file and generate a client, everything is ok but if we generate a server, our code will try to listen in port 80, which is not what we want. This means we can't infer clients from server definitions.
The drawback is that we'd have to define 2 files, one for the server and one for the client but the benefit is that we can be 100% accurate with our intents. Tooling can help auto-generating a client definition from a server definition.
Further expansion
Override channel messages with operation-specific messages
The new
operationsobject would help us define additional use cases, like those where we define what message is sent or received to/from a channel at the operation level. Let's see an example:Adding request/reply support
Adding support for the so-demanded request/reply pattern, would be as easy as adding a new verb and a
replykeyword. See example:FAQ
Does it serve as a base ground to meet our vision without a future major version?
I think so. Whenever we're ready to support REST, GraphQL, and RPC APIs, this structure should perfectly serve as a base ground.
Does it enable for better reusability of channels and other elements?
Yes. It completely decouples channels from operations and even clarifies the reusability model of the specification.
Does it allow users to define their whole architecture in a single file?
No, but it allows users to have a "common resources" AsyncAPI file where most of the information can reside.
Is it backward-compatible?
Absolutely not.
Does it remove the publish/subscribe confusion without introducing a new confusing term?
Yes. We keep the basic terms as before with just a bit of reorganization.
Is it easy to define a broker-based microservices architecture?
Yes. I provided an example.
Is it easy to define a public asynchronous API?
Yes. I provided an example.
Does it set the base ground to define an RPC (Remote Procedure Call) system over a message broker?
Yes. I provided a basic non-normative example.
Does it set the base ground to define a point-to-point RPC (Remote Procedure Call) API?
Yes. I provided a basic non-normative example.