[PROPOSAL] Add support for API versioning

[dblock]

What/Why

What are you proposing?

Every version of OpenSearch has its own version of the API and therefore the API specs should reflect that.

What users have asked for this feature?

In https://github.com/opensearch-project/opensearch-clients/issues/58 we plan to auto-generate clients. Therefore we need a way to both version API specs and a way to know the exact API spec for a given version of server.

What problems are you trying to solve?

• Easily identify the exact spec for an exact version of OpenSearch.
• Ability to generate specs for flavors of deployments of OpenSearch, such as the Amazon managed service, which may have APIs added or removed.

What is the developer experience going to be?

I see several options.

a) The specs tag when APIs were added/removed/changed, see #80, and publish a single version of specs with the options/tags. b) The repo is branched in ways that follows OpenSearch and each branch publishes a version of the specs.

What is the user experience going to be?

I'd like to be able to say "where's the 2.7.0 version of OpenSearch specs" and consume it directly.

[wbeckler]

I would propose that the API spec include version ranges as a field for a method. A given client should be able to speak with multiple servers, and by knowing which server version we're talking to, or by specifying it in the request, a single client with a multi-version spec could handle that situation.

[reta]

a) The specs tag when APIs were added/removed/changed, see https://github.com/opensearch-project/opensearch-api-specification/issues/80, and publish a single version of specs with the options/tags.

I kind of like this approach: continuously evolving the spec. I would like to suggest a combined approach, keep one spec for a release line (1.x, 2.x, 3.x). Why is that: within the release line, the spec should evolve in a backward compatible way but not between major releases. I have difficulties expressing the API in spec that was introduced in 1.3.7, deprecated in 2.7.0 and removed in 3.0.0 (we could only express the latter essentially). The only option with single spec I see is when we always evolve the spec in backward compatible way.

For cases where there is a need to have connect to multi-version clusters, non issue in scope of same release line, but if the support of multiple major versions is needed, several clients could be generated easily (one for 2.x and one for 3.x fe).

What do you think?

[wbeckler]

I think if the spec can span as wide a range of versions as possible, it becomes more capable of generating a client that "just works." The challenge of migrating across major versions is the place where a multi-version client makes the biggest difference. The minor version changes, assuming semver is getting followed, introduce changes that are unlikely to lead to heartache of a user. If the client had a feature missing in a server, a lot of unlikely things would have to come together for an engineer to build an application that relied on those features because they exist in the client but didn't want to upgrade their server to enable the feature. I think I empathize with the more constrained engineer who is contending with major version changes and old codebases of working code, maybe because I've been there. That's who benefits from a more capable multi-major-version client.

[nhtruong]

@wbeckler A client that can talk to any server version is nice but it might be more troublesome than it's worth. It will drastically increase the complexity of the code generator, increase performance overheads for the client (from server version checking and guards for different versions of the server when a request is executed). On the other hand, there are very few use cases where an application has to talk to multiple OS clusters of different major versions.

[dblock]

You cannot evolve a spec in backwards compatible ways across major semver versions because of breaking changes in parameters. For example, imagine an API that takes a field called name of type String. If we change name to be Integer, we cannot express "name was added in 1.0 as String, but became Integer in 2.0". The easiest way to handle this is to branch specs the same way as the product, and make and tag releases.

[dblock]

@wbeckler A client that can talk to any server version is nice but it might be more troublesome than it's worth. It will drastically increase the complexity of the code generator, increase performance overheads for the client (from server version checking and guards for different versions of the server when a request is executed). On the other hand, there are very few use cases where an application has to talk to multiple OS clusters of different major versions.

That's actually not true.

1. Any live upgrade requires a client to be able to talk to at least 2 major versions of OpenSearch, albeit not at the same time.
2. Clients such as opensearch-hadoop want to be able to talk to multiple major versions of OpenSearch from a single installation of Hadoop.

[VachaShah]

Approaches for Versioning for API specs

The versioning implemented for specs will also be used for client generation, hence the purpose of this design is:

• Versioned specs depict the server APIs in a truthful manner.
• Provide a way to specify versions for the specs which can be easily consumed by the users.
• Versioned specs can be used to generate clients that can talk to server versions.

Approach 1

As the specs are evolving to be the true representation of the server, the OpenAPI specs can be divided according to major versions of the server in their own branch: 1.x branch for the 1.x server APIs, 2.x branch for the 2.x server APIs. This way the specs represent the various major versions of the server and are easy to consume by the users.

For minor versions of the APIs, the branched specs can extend the current OpenAPI vendor extension traits x-version-added, x-version-removed and x-version-deprecated to reflect in the published OpenAPI specs so that various minor versions of the APIs can exist in the same spec.

Pros

• Easier to read the specs and easier to consume by the users since they are branched according to server major versions.
• Easier to implement both in terms of spec generation and client generation.

Cons

• To generate clients that can talk across two major versions, client generators would have to use multiple specs for generation.

Approach 2

Publish a single spec for all versions of the server. Use the current OpenAPI vendor extensions to get the server versions of when the APIs were added, deprecated or removed. This approach works similar to approach 1 but spans across major versions.

Pros

• Easier to maintain a single spec file and look at the lifecycle of an API.

Cons

• For APIs that have a v1 and v2 across server major versions, this approach has problems since paths cannot be duplicated in OpenAPI.
• Client generation will become complicated.
• The spec may get a little messy with everything muddled in together.

Approach 3

OpenAPI specs are published for each minor version of the server.

Pros

• Easiest approach in terms of implementation.

Cons

• Generated clients get tied to server versions.
• Requires maintaining multiple spec versions.

With the above trade offs for approaches, to add versioning for the specs (approach 1), following items need to be addressed:

• [ ] Branch the api-spec repo according to server major versions.
• [ ] Generate specs for each branch and publish them on Github pages.
• [ ] Change the UI to publish x-version-* traits in the specs. The json file already has this information, the Swagger UI needs to be changed to reflect this information.

[reta]

Thanks @VachaShah , to me the Approach 1 presents the best balance of the usability and maintainability

[dblock]

With the publishing mechanism in #211 we can publish builds from multiple branches and thus track multiple versions of OpenSearch that aren't compatible.

[dblock]

https://github.com/opensearch-project/opensearch-api-specification/issues/120 is related