[PROPOSAL] Use API specification to generate clients

[dblock]

What kind of business use case are you trying to solve? What are your requirements?

There are 9 language clients for OpenSearch in this organization. Each client is significantly different from another in their implementation (e.g. different pagination support or transport layer), but implement the exact same API (e.g. creating an index). That API matrix is huge, including multiple versions of OpenSearch and plugins that bring their own API.

What is the problem? What is preventing you from meeting the requirements?

The maintenance of clients is a large burden. Every time a new API is added or changed, every client needs to be updated, along with tests.

What are you proposing? What do you suggest we do to solve the problem or improve the existing situation?

Automate the update process by auto-generating non-interesting parts of the client.

• https://github.com/opensearch-project/OpenSearch/issues/3090.
• Generate non-interesting parts of each client based on the API specification with templates
• Implement additional layers in the client such as pagination or Sigv4 signing
• Add support for consuming multiple versions of the API spec in clients
• Automate API updates via GHA, such as every time the specification changes, a PR is raised for each client

As an example, https://github.com/slack-ruby/slack-ruby-client is built this way. The API specification is scraped from Slack in https://github.com/slack-ruby/slack-api-ref and the client uses templates for about 75% of its code.

Which clients use a generator today?

• [x] opensearch-py - https://github.com/opensearch-project/opensearch-py/issues/714
• [x] opensearch-java - https://github.com/opensearch-project/opensearch-java/pull/366
• [ ] opensearch-ruby - https://github.com/opensearch-project/opensearch-ruby/issues/233
• [x] opensearch-php - https://github.com/opensearch-project/opensearch-php/issues/97
• [x] opensearch-net - https://github.com/opensearch-project/opensearch-net/issues/193
• [ ] opensearch-rs - https://github.com/opensearch-project/opensearch-rs/issues/192
• [ ] opensearch-go - https://github.com/opensearch-project/opensearch-go/issues/284
• [ ] opensearch-js - https://github.com/opensearch-project/opensearch-js/issues/789

[dblock]

Note that the Rust client is generated from spec, see https://github.com/dblock/opensearch-rs/tree/main/api_generator/rest_specs

[shyim]

The PHP client was generated also with the rest-spec. What I am unsure, it looks like we are building a second schema. So i am unsure if we should use the rest-sepcs.

https://github.com/opensearch-project/opensearch-api-specification

[dblock]

@shyim One reason to have opensearch-api-specification outside of OpenSearch core is that it also will/should contain schemas for various plugins, maybe as submodules (TBD). With everything I know I think I would invest in that repo, and delete everything else.

[dblock]

I added "Add support for consuming multiple versions of the API spec in clients" above. I am not sure it's the best idea, but it's an idea nevertheless. I imagine the spec is either targeting a specific version of OpenSearch or is annotated with specific versions of OpenSearch (e.g. API since version X). With extensions it's going to be an n x m matrix, too. A possible approach is that the client carries multiple sets of generated code under a versioned namespace (e.g. opensearch.2.client and opensearch.ext.anomaly_detection.8....) and the right version is loaded at runtime. If that's possible (definitely in Ruby) then we would simplify the spec maintenance and generation quite a bit I think as it would follow product branching.