search

asyncapi / shape-up-process

This repo contains pitches and the current cycle bets. More info about the Shape Up process: https://basecamp.com/shapeup
https://shapeup.asyncapi.io
11 stars 8 forks source link

Intent-driven API design for the Parser #82

Open fmvilas opened 3 years ago

fmvilas commented 3 years ago

On This Page

Summary

Problem Overview

Our current parser is tightly coupled to the AsyncAPI document structure.

Solution Overview

Create a new parser API that represents intents instead of the structure of the document.

Positive Side-Effects

  1. When tooling authors have to start supporting future versions of AsyncAPI, they will not have to change their code (unless it's for adding new features that didn't exist before).

Problem Details

Tight coupling to the document structure means that if we change it, we'll be creating breaking changes in the parser too and therefore having to release a new major version.

For instance, in our current version, we grab the title of our API from the info object:

asyncapi: 2.0.0
info:
  title: My API
asyncapi.info().title()

In the future, we decide to move title to the root of the document:

asyncapi: 3.0.0
title: My API
asyncapi.info().title()

The parser breaks because there's no such field title inside the info object on AsyncAPI v3. One could argue that we can make the title() function check the version of the document and therefore look for the title in a different place, depending on the version. However, the chain asyncapi -> info() -> title() suggests (and actually maps) to the structure of the document in v2.

Note this is just a simple example (and we most probably will not move the info object anywhere anytime soon) but it's highly probable that we reorganize things on the Channel Item Object. See the following example:

asyncapi: 2.0.0
channels:
  mychannel:
    description: My channel description
    subscribe:
      message:
        payload: ...
asyncapi.channel('mychannel').subscribe().message().payload()

This obtains the payload of the message people can subscribe to on the channel mychannel. There's a high chance we'll take the operation (subscribe) information out of this object somehow for the next version. Let's imagine the following situation:

asyncapi: 3.0.0
channels:
  mychannel:
    description: My channel description
    message:
      payload: ...
operations:
  subscribeToMyChannel:
    operation: subscribe
    channel: mychannel
asyncapi.channel('mychannel').subscribe().message().payload()

The chain of calls asyncapi -> channel('mychannel') -> subscribe() -> message() -> payload() doesn't make sense anymore. It will be a legacy thing from AsyncAPI v2. And we'd probably have to add a new chain of calls to match v3 structure but that actually does the same. Something like asyncapi -> operations('subscribeToMyChannel') -> message() -> payload().

In short, every single time we change the structure of the document we're breaking the parser and the semantics of this chain of calls.

Solution guidelines

First of all, I want to make it clear that I don't want you to implement my solution. I'd love for you to come up with a better solution and implement it.

In my opinion, the problem described above comes when we couple things tightly. It's beautiful and intuitive when you know the structure or the structure doesn't change very often. So how can we design this API so it doesn't change that often? Design based on intents. Make sure you have a look around all the tools we have using the current parser and try to guess what was the intent behind it.

A simple example:

asyncapi.channel('mychannel').subscribe().message().payload()

What's the intent of this call? The user wants to get the payload of the message when someone subscribes to the channel mychannel. This could be much shorter and intuitive for the user. For instance:

asyncapi.getMessageFor('mychannel', 'subscribe').payload()

// or

const message = asyncapi.getMessageFor('mychannel', 'subscribe')
console.log(message.payload())

In the chain of calls above, there's no implicit structure mapping with the AsyncAPI document 🎉 Now, even if we change the structure of the AsyncAPI document in future versions, this will still work the same way. People will only have to upgrade their parser. That's it.

As you can see, we're focusing on intents and concepts. There will always be channels, messages, operations, and payloads. The method getMessageFor(CHANNEL, OPERATION) expresses a clear intent and —independently of the version or structure of the document— the intent of the user will not change.

So my suggestion is that you go and define all the "concepts" we have on the spec and build a list of intents you see already happening on our tools plus others you think could be useful. A great place to look for inspiration is the Generator templates.

Boundaries

Don’t release v2 of the parser during the cycle

If you feel comfortable, go ahead and release v2-alpha, v2-beta, etc. but don't release v2 as we'll have to test it with the community properly during some time.

Please don't consider migrating the Generator templates part of this bet

It's ok to open pull requests on the Generator templates that would help us migrate them in the future. In the end, you have to test the new API somewhere. Just don't consider this work part of the bet. Don't get me wrong, I highly recommend you use the new parser on two or three templates, just don't consider this migration something you have to finish in this cycle.

Watch out for documentation

Current API documentation is far from perfect. We know and we'll dedicate time for it. While you have to change them to document the new API, don't go beyond that (like using other tools to generate docs, changing the structure of the docs, etc.)

Watch out for rabbit holes when listing all the possible intents

It's easy to get lost in trying to make this parser perfect on the first iteration. Don't worry about perfection now because we'll have to test this with the community for some time and we'll detect new intents as we go.

Long-Term Vision

OpenAPI 3.0.0 and the adoption problem

As some of you may have noticed, since Swagger 2.0 was donated by Smartbear to Linux Foundation, OpenAPI 3.0.0 has been released. It happened on the 26th of July 2017. Still, many people haven't migrated yet from Swagger 2.0 to OpenAPI 3.0.0. When I asked, many people told me:

It's great it's on v3 now but v2 is more than enough for me. v3 is not introducing enough value for me to switch. Also, the tooling is not yet updated so 🤷

However, v3 had a really good thing. They started moving things around the document to better suit the spec for future changes. Just like the example of channels and operations above. Moving them around doesn't provide much value for the end-user but it does for us and it's something you have to do from time to time, especially in the early stages (yes, we're still in the early stages 😄).

One of the reasons people didn't migrate to v3 quickly was that products weren't supporting it yet. It was a huge investment for the companies to migrate their tools to support v3 and it wasn't providing much new value to their users. The result: "yes, we'll do it but it's not a priority for us". This affected a lot the speed of adoption of OpenAPI 3.0.0. With this API design, companies should be able to quickly update their parsers and start supporting new versions. The only thing they'll have to invest time on is exactly what provides value for the users: the new features.

derberg commented 3 years ago

Would love to see API design before implementation 😄 best, with some snippets from current templates or react component, that show how the code looks now and how would it work now.

I personally see a need not only for getMessage that in return has payload as well but would be convenient to have getPayload too

jonaslagoni commented 3 years ago

I suggest we change the scope of the issue to exclude the implementation part of the issue, determining the API for the parser is the most important part of this bet and cannot be rushed. We have to get feedback, integrate it, and test it against breaking changes. We are currently on our 3rd (4th?) iteration of the API and are still not completely set on it and whether it will be the final iteration.

cc @derberg @smoya

derberg commented 3 years ago

From my point of view, the goal of this bet is to get a great API, not implement it, so tl;dr final implementation is not in scope

If you feel comfortable, go ahead and release v2-alpha, v2-beta, etc. but don't release v2 as we'll have to test it with the community properly during some time.

Working out API is time-consuming and complex and I think we all agree this takes time, so no worries. Focus on integrations, mocking and making it possible for others to play with the API.

I think it is safe to say that nobody expects perfect API design, especially that it is not really possible. Even if you make 10 iterations, in a month or two there we will see some areas for improvement 😄

now, about integration part where you try out the API, maybe we should start first with super simple templates, like nodejs-ws or template-for-templates where we do not have to much logic?

smoya commented 3 years ago

From my point of view, the goal of this bet is to get a great API, not implement it, so tl;dr final implementation is not in scope

As @derberg stated, I also consider implementation is not a requirement for this bet. I created scopes such as https://github.com/asyncapi/shape-up-process/issues/91, which perhaps was a mistake I made and they should not be there if we know we won't (most probably) work on it.

I think it makes sense to remove that scope for now. Instead, the focus should be kept in the areas it is now: Design + Validation, where validation includes coding a light version of the new API, mostly a mock but it could also be a very reduced but functional API that users can use (even with limitations).

now, about integration part where you try out the API, maybe we should start first with super simple templates, like nodejs-ws or template-for-templates where we do not have to much logic?

That's what we are doing right now (with mocks). However, we did not take the super simple ones but rather those that helped us define the intents as they have some particular behaviors. Anyway, I think picking one of the most used templates will be a good move in order to show a simple working example of the new API.

WDYT @jonaslagoni @derberg

jonaslagoni commented 3 years ago

Just to mention it somewhere and "keep track of it", mentioned in SIG meeting Tuesday, Apr 13, 2021.

In terms of the implementation (I know we agreed to keep it out of scope for this cycle, but this bet will be the starting point for the implementation), we should consider whether we can make a generic (code generation) approach, potentially collaboration project with OpenAPI, cc @MikeRalphson.

This could potentially long term save us some headaches if we generate the intent API (someway) however we need to keep in mind that we require changes in the parser for RFCs. This means we need to ensure that it does not add too much complexity to that process without having discussed it thoroughly.