Creating the AsyncAPI Domain specification

fmvilas commented 2 years ago

Note: I'm gonna call it "AsyncAPI Domain spec" for the lack of a better alternative. Let me know if you have a better one 🙏

TL;DR

channels in the good old AsyncAPI document will now refer only to channels exposed and owned by the application, like WebSocket or Socket.IO channels, and HTTP Server-Sent Events (I can't actually think of any other kind of channel an application can expose, maybe IPC?).
Local servers can only use the HTTP or WS protocol. Remote servers can be of any kind.
This proposal introduces a new root property in the application definition spec: domains.
This proposal creates a new spec. It's not that new since it's actually made of existing AsyncAPI objects. Just formalizing the common 3rd file everyone uses to share definitions across multiple applications.
Fun fact: the Domain spec should make it easier to adopt DDD (Domain-Driven Design).

Abstract

So far, we've always recommended people to extract server, channel, message, and schema definitions into a 3rd file they can reference from all the AsyncAPI Application definitions. This way, they keep them DRY.

It seems unavoidable to have this third file when defining broker-based architectures. Simply having 2 applications publishing and consuming from the same channels will make you want to split things out. Therefore, what I'm proposing here is to give an official name and structure to this 3rd file: the AsyncAPI Domain specification.

This proposal is motivated by previous conversations where we were discussing making the AsyncAPI document represent an application except when it doesn't have the operations keyword. In such a case, it would be defining a menu of available servers, channels, messages, etc. Not coupled to a specific application at all. It's precisely this kind of polymorphism that was driving me nuts. An apparently innocent keyword like operations was changing the whole meaning of the whole document. That's a big no-no to me.

Example

Global Company Domain

A domain for the whole company, to simplify things.

asyncapi-domain: 1.0.0 # Should we use the same version as asyncapi instead? E.g., 3.0.0.

# Slightly shorter than applications' info. Or maybe not, maybe we can just use the same Info Object.
info:
  title: Global Company Domain
  version: 1.0.0

# These are brokers in the domain.
brokers:
  production:
    url: mqtt://broker.mycompany.com
    protocol: mqtt
    protocolVersion: 3.1.1

# HTTP and WS servers
servers:
  myWebSocketApiServer:
    url: wss://mywebsocketapi.com
  dev8001:
    url: ws://localhost:8001

# Channels exposed in the domain that are not owned by a specific application.
# Usually, broker topics and queues.
channels:
  mqttPostLiked:
    address: post/liked
    message:
      $ref: '#/messages/postLiked'

# Messages used in the channels
messages:
  postLiked:
    payload:
      type: object
      properties:
        postId:
          type: number
        appInstanceId:
          type: string

components:
  schemas:
    ...
  ...

Usage from an application definition

asyncapi: 3.0.0

info:
  title: My WebSocket API
  version: 1.0.0

# An object holding different organizational domains.
# A domain can contain channels, messages, and servers.
# This application is not the owner of the domain assets.
domains:
  global:
    $ref: 'domain.asyncapi.yaml'

# Servers are only the server interfaces exposed by this or another application.
# Therefore, only HTTP and WebSocket interfaces are supported here now.
# Brokers belong to a domain because an application is not the owner of a broker.
# Fun fact: this is now matching OpenAPI servers except for the support of WS and bindings.
servers:
  local: # Local servers, i.e., server interfaces exposed by this application. Please bear with me, this syntax is not part of this proposal. Just needed a way to express these are server interfaces exposed by the application. There are other suggestions to use `servers` for local servers and `remotes` for remote servers. Both are fine for me.
    myWebSocketApiServer:
      $ref: '#/domains/global/servers/myWebSocketApiServer'
    dev:
      $ref: '#/domains/global/servers/dev8001'
  remote:
    mosquitto:
      url: 'mqtt://test.mosquitto.org:1883'

# Channels are now only the channels exposed by this application.
# These are not broker channels, those are defined in the domains.
# These can only be WS (and HTTP SSE?) channels.
channels:
  postLiked:
    address: postLiked
    message:
      $ref: '#/components/messages/postLiked'

# Operations this application is performing.
operations:
  onPostLiked:
    action: receive
    channel:
      $ref: '#/channels/postLiked'
  sendPostLikedToBroker: # Notice this operation is pointing to a domain channel.
    action: send
    channel:
      $ref: '#/domains/global/channels/postLiked'

components:
  messages:
    postLiked:
      payload:
        $ref: '#/components/schemas/postLiked'
  schemas:
    postLiked:
      type: object
        properties:
          postId:
            type: number

derberg commented 2 years ago

oh, new spec 👀

brokers would be only for this new spec, right?

I'm not 100% opinionated yet, but my thoughts so far are 1+ specs will complicate a lot. Because we already know users ignore that AsyncAPI is only for describing an application interface from user perspective. They do it simply because this way they can still use some of the tools that we produce, like docs generation.

This proposal is motivated by previous conversations where we were discussing making the AsyncAPI document represent an application except when it doesn't have the operations keyword. In such a case, it would be defining a menu of available servers, channels, messages, etc. Not coupled to a specific application at all. It's precisely this kind of polymorphism that was driving me nuts. An apparently innocent keyword like operations was changing the whole meaning of the whole document. That's a big no-no to me.

Do you mean that because of operations you can't imagine AsyncAPI 3.0 to be used to describe a broker?

fmvilas commented 2 years ago

brokers would be only for this new spec, right?

Yes, but that's only the key name. They're Server Objects.

I'm not 100% opinionated yet, but my thoughts so far are 1+ specs will complicate a lot. Because we already know users ignore that AsyncAPI is only for describing an application interface from user perspective. They do it simply because this way they can still use some of the tools that we produce, like docs generation.

Precisely. People are already doing it by misusing the spec. Let's give them a proper format for that. I don't think the complexity will grow a lot because this new spec is made of existing definitions in the spec. It's just another file with a new name to make it explicit that it's about a group of things and not about an application. Pretty much what we have right now in v3 if you omit the operations key. The difference is that having/not-having the operations key makes the change of meaning implicit and not obvious, and having this new file would make it explicit. We'll have to support these things either way.

Do you mean that because of operations you can't imagine AsyncAPI 3.0 to be used to describe a broker?

I can imagine it because it's actually happening 😄 I just don't think it's ideal and we can do better.

smoya commented 2 years ago

Local servers can only use the HTTP or WS protocol. Remote servers can be of any kind.

I agree is not a very common use case but what about the Event-Gateway, where the application (the event-gateway running app) exposes a Kafka proxy? The protocol cannot be only HTTP or WS. In this case, the app defines a local server with kafka protocol; it will act proxying messages to a broker with kafka protocol located in the new domain file.

fmvilas commented 2 years ago

An event gateway is not a common application IMHO. It's a server implementing any protocol and publishing and subscribing to all the channels. That would be defined using this new spec instead of the current Application spec.

derberg commented 2 years ago

More and more I'm thinking about, I think it will complicate things, and that we do not really need it.

Complexity by adding new spec

I just keep thinking about examples out there in the wild and the only one that comes to my mind is JSON Schema. So JSON Schema is for validation, but people (including us) misuse it by using it for code generation. Community introduced new spec for that but it is not adopted. What people do (including us)? we continue misuse JSON Schema as for majority of cases it is ok. We are not lazy, we are pragmatic.

Introducing new spec is not the same level of complexity as introducing new property in existing spec.

We tend to say AsyncAPI is for application, not a broker. Then we say WebSocket is different as it is an app. Maybe we should go back to the roots and the explanation behind AsyncAPI, and publish/subscribe.

I mean, the more I deep dive into the word of WebSocket, the more I'm convinced we are wrong with initial assumption. WebSocket has this core feature of sub protocols, and even official docs say that WebSocket when used only as transport in combination of subprotocol is called a MessageBroker.

You can write a server application that connects with WebSocket message broker, and then subscribes only to specific topics. You might want to describe your server application with AsyncAPI and then WebSocket server is remote (using naming from 3.0 spec work), just like in case of other broker centric architectures.

Frankly speaking, you do not need sub protocol in use for that. Wouldn't you like to write AsyncAPI file for your mobile app that uses websocket API? I see benefits.

The difference is that for:

WebSocket, as a user of the API I see only possible exchange, and have no idea who consumes or produces
Others broker centric, even if I do not care who consumes/produces, and only want to know what I can send or read from the broker, I'm not able to do it, I'm kinda forced to know what app is sending and reading...not sure if I me

Sorry for long context

What I'm trying to say is, AsyncAPI is already used by people to describe what is available on a broker, so let them do it, without hiding it, because even if we do additional spec, they will still do the hacks 🤷🏼 I think it is completely fine to say Operations are optional, basically all is optional except of Info and that spec could be used to describe any entity in the architecture 😄

derberg commented 2 years ago

Basing on recording of last 3.0 meeting https://www.youtube.com/watch?v=7CymWygvWwI

@fmvilas can you highlight why we need a new spec or even a new flag to indicate it is Application or a Broker.

Dumb question ahead: Why cannot spec say that it is for both and Operation is simply optional.

I ask this question in reference to my previous comment about Websocket that is a MessageBroker. It will be super confusing when you say: "so you describe WebSocket app that is a broker, but it is an app owning channels, so you use kind: blabla"

I really cannot imagine/understand why this is important to have an explicit flag. Even now I can have app described with AsyncAPI without operations, and some code generators will fail. So why spec need a flag or new spec, and not tools simply require certain objects?

Maybe we have a bad definition of an app 😄

fmvilas commented 2 years ago

It is not "it is Application or a Broker". It is about being an Application or simply a collection of resources (messages, channels, servers, etc.). I called the latter a Domain for the lack of a better term. Can also be called a Library, Menu, etc.

For Domain aka Library aka Menu, the purpose is to hold a set of reusable stuff. It's not a specific application.

can you highlight why we need a new spec or even a new flag to indicate it is Application or a Broker. Dumb question ahead: Why cannot spec say that it is for both and Operation is simply optional.

Yes, that's exactly what I proposed in the call. We just make operations optional BUT to make it more explicit that we're talking about one kind of document or another, we add a new field. The reason is to accommodate the spec for future growth where we may add/remove fields to/from one kind of document but not the other.

For instance, in the future, we may want to allow people to define reusable operations. This can be especially useful when you have multiple clients subscribing to the same channel. In such a case, the operations keyword would then be allowed on the Domain kind of document, making it indistinguishable from the Application one and forcing us to ship a new major version just for this flag.

I think changing the meaning of the whole document exclusively based on the presence/absence of a field that is completely unrelated is a weak design. So to answer your question: is it needed? no, but I think it's more resilient to future changes. Can we just rely on the operations field? Absolutely, but it has flaws.

Maybe we have a bad definition of an app 😄

I'm using the Application meaning from the spec:

An application is any kind of computer program or a group of them. It MUST be a producer, a consumer or both. An application MAY be a microservice, IoT device (sensor), mainframe process, etc. An application MAY be written in any number of different programming languages as long as they support the selected protocol. An application MUST also use a protocol supported by the server in order to connect and exchange messages.

fmvilas commented 2 years ago

Have a look at https://github.com/asyncapi/spec/pull/874. The need to clarify this stuff appeared again. @derberg

github-actions[bot] commented 1 year ago

This issue has been automatically marked as stale because it has not had recent activity :sleeping:

It will be closed in 120 days if no further activity occurs. To unstale this issue, add a comment with a detailed explanation.

There can be many reasons why some specific issue has no activity. The most probable cause is lack of time, not lack of interest. AsyncAPI Initiative is a Linux Foundation project not owned by a single for-profit company. It is a community-driven initiative ruled under open governance model.

Let us figure out together how to push this issue forward. Connect with us through one of many communication channels we established here.

Thank you for your patience :heart:

fmvilas commented 1 year ago

I'm closing this issue as, of now, we're not going to make it.

asyncapi / spec