search

opensearch-project / opensearch-py

Python Client for OpenSearch
https://opensearch.org/docs/latest/clients/python/
Apache License 2.0
350 stars 175 forks source link

[PROPOSAL]Merge Generated APIs into Existing Client Module #477

Closed saimedhi closed 1 year ago

saimedhi commented 1 year ago

What are you proposing?

Merge the generated APIs into the existing client module to progress towards complete client generation, ensuring alignment with the server.

How does the opensearch-py API Generator function?

  1. Run the generator as per the developer guide.
  2. The generator retrieves the OpenSearch API specification JSON data.
  3. API paths and methods are processed and grouped based on the x-operation-group field.
  4. API parameters, parts, methods, and documentation are updated and organized.
  5. For each API endpoint, instances of Module and API classes are initialized.
  6. Generator checks if corresponding module exists in the clients folder for each API's namespace.
  7. If the module is absent, creates a module for the corresponding namespace.
  8. In the module, a class is created based on the namespace, containing API definitions or actions.
  9. Ninja templates are utilized for generation, with overrides for compatibility.
  10. Format the code using instructions in developer guide.
  11. If a namespace/module already exists, then existing definitions in that module are replaced by the generated API definitions.

What problems are you trying to solve?

Consider 'indices' namespace, post-generation, existing APIs are replaced by generated ones. Some generated API may be different from the existing APIs. We need to explore solutions for handling divergent APIs, termed as 'Patch APIs.' Our goal is to structure the merging process to prevent breaking changes. We categorize APIs as 'generatable' or 'Patch'

image

Solutions:

Three approaches are discussed below to separate and structure generatable and patch APIs.

Current client structure:

image

Approach 1

Consider "indices" namespace, all the patch APIs are placed in a separate file "indices/patch.py". In this way while running generator, patch APIs wont be altered. And we generate all other APIs into "indices.py". Easy implementation and contributor-friendly.

image

Approach 2

Change client structure to one file per action/API within the namespace. Significant effort initially, both in changing the client structure and adapting the generator. The following PR shows more about this approach. https://github.com/opensearch-project/opensearch-py/pull/475/files

image

Approach 3

Modify specs of patch APIs within the generator to match existing APIs. Utilize overrides in ninja templates to overwrite generated APIs. Time-consuming but avoids major structural changes.

image

What is the developer experience going to be?

Developers must understand generator functionality before contributing. Decisions on contributions (API spec, generator, or patch) are essential.

Are there breaking changes to the User Experience?

Aims for no breaking changes; user experience remains consistent.

Why should it be built?

Facilitates easier addition of client support for new server-side APIs.

Please let me know your thoughts and suggestions.

dblock commented 1 year ago

This is quite thoughtful. If you didn't have to minimize breaking changes, what would this process look like? I would work backwards from there, with the hope that once the API generator is producing the entire API we can delete all patches and make one major version bump breaking backwards compatibility.