[PROPOSAL] Build architecture for spec and client generation

VachaShah commented 1 year ago

What/Why

What are you proposing?

A build architecture for API spec generation and code generation for OpenSearch.

Description

Smithy models in this repo are used to generate Smithy and OpenAPI specs for the OpenSearch APIs. Smithy is an open source IDL which is protocol agnostic, extensible and supports code generation. (See https://github.com/opensearch-project/OpenSearch/issues/1590) Smithy models are converted to Smithy and OpenAPI specs during build. These specs can be hosted to view and edit and further be used for client and server generation.

What users have asked for this feature?

34

What problems are you trying to solve?

User can view OpenSearch API specs to get information about request, response, parameters etc. for an API.
API specs are updated when a change is made to the models.
Using the API specs, APIs can be generated in clients (https://github.com/opensearch-project/opensearch-clients/issues/19).
Using the API specs, server routes can be generated (https://github.com/opensearch-project/OpenSearch/issues/3090).

What is the developer experience going to be?

When a change is made to Smithy models in this repo, a workflow will update the API specs hosted.
When the specs are updated, each client gets updated with the newer API versions.

What is the user experience going to be?

Publish OpenSearch API specs

This proposal talks about hosting OpenSearch API specs on Github Pages. A Github workflow will be triggered when a change is made to the Smithy models. This workflow will build the models and if there is a change to the API specs, it will create a pull request to update the specs hosted in the repo and in turn update the specs on Github Pages. OpenAPI specs will be available to view using Swagger UI.

Similar process can be followed for hosting Smithy specs.

Other options explored

API specs can also be hosted on other programming language specific platforms like Maven, Ruby, npm but it makes it difficult for other language users to take benefits from the specs (see #34). The specs on Github Pages provide a dynamic UI to view the specs and Swagger editor can be used to edit and preview the specs in real time.

Client Generation

Using the above hosted API specs, client APIs can be generated using an open-api generator or a custom generator. Smithy specs can be used to generate APIs using smithy language generators. Client generators will reside in each client repo. Although what kind of generators can be used to generate clients is a separate proposal for each client (see https://github.com/opensearch-project/opensearch-java/issues/284, https://github.com/opensearch-project/opensearch-ruby/issues/139) which is out of scope for this proposal, this proposal talks about linking the process of how a change in API specs results in incremental API generation for clients.

A Github workflow in each client will be triggered when a change is detected in the API specs (a workflow in one repository can trigger an action in another repository see https://github.com/orgs/community/discussions/26323). The workflow will run the generator using the hosted API specs and a pull request will be created by the workflow to update the generated code.

TODO: Once a generator is ready for a client, implement CI for the client as mentioned above.

Server Route Generation

Similar CI workflow like client generation can be used for server route generation. The generators for server are out of scope of this proposal.

Why should it be built? Any reason not to?

Providing API specs for OpenSearch allows the users to get information regarding request-response contracts for the server.
With the hosted API specs, users can consume them to create clients outside the programming languages available.
Generating APIs from specs will allow to fill in gaps of missing APIs in clients, deliver new APIs to users quickly, help maintain consistency between server and client and generate a lot of boilerplate code automatically.

What will it take to execute?

Framework to view specs using Swagger UI.
CI to update specs when a change is made to the models.
CI to trigger API code updates on each client when specs are updated.
Generators for clients.

Any remaining open questions?

Enhancement: Host versioned API specs.
Should the specs live in the OpenSearch repo so they are a better representation of the server or should they live in opensearch-api-specification as a separate entity which represents both server and clients?
When generating client APIs, determine how the generated code regarding incremental changes for new features or missing APIs in clients will be handled. For instance, when the entire client is generated, will the client be generated during runtime?
How can we benefit from rest-api-spec to create Smithy/OpenAPI specs? - To benefit from the already existing rest-api-spec, a PR is out which converts the rest-api-spec to Smithy models using a translator.

VachaShah commented 1 year ago

Will raise a PR soon for publishing specs.

reta commented 1 year ago

We have a number of ongoing PoCs [1], [2], [3] to try out client generation for different programming languages and specs (Smithy, OpenAPI). We are still debating on what is the best approach here (should we use Smithy, OpenAPI or something else completely) but I think the one large key problems is not getting enough attention: crafting the Smithy / OpenAPI / ... specs.

As far as I can tell, all our PoCs craft them by hand, including this repo [4], this is huge effort. But we already have a large chunk of API specs here https://github.com/opensearch-project/OpenSearch/tree/main/rest-api-spec/src/main/resources/rest-api-spec/api, inherited from Elasticsearch. Shouldn't we consider to benefit from it and try to convert them into Smithy / OpenAPI specs?

[1] https://github.com/opensearch-project/opensearch-java/pull/336 [2] https://github.com/opensearch-project/opensearch-java/pull/318 [3] https://github.com/opensearch-project/opensearch-ruby/issues/139 [4] https://github.com/opensearch-project/opensearch-api-specification/tree/main/model

VachaShah commented 1 year ago

@reta Absolutely! I missed adding that in my proposal that we can explore on how rest-api-spec can be used to programmatically generate Smithy/OpenAPI specs. Currently, while writing models in this repo, I do look at rest-api-spec to get a better understanding of the API and figure out the missing pieces that are not reflected in rest-api-spec to complete the models.

Let me add it in this in the proposal too.

dblock commented 1 year ago

Late to this issue, but had some thoughts.

First, it may be easier to think of API specs being built, released, and follow semver that can match what they specify if that relationship is 1:1, or have their own version if it's 1:many. This way if a mistake was made in the specs for OpenSearch 2.6.0, it can be corrected, but we can distinguish API specs between before and after correction. I envision that this repo gets built and released, possibly just on GitHub. IMO this step is missing from the above proposal.

The GitHub pages hosted website can be a result of publishing a released version of the API specs and I like the idea of enabling it to be consumed by a Swagger UI.

dblock commented 7 months ago

We have this workflow in most clients now, closing.

opensearch-project / opensearch-api-specification