[FEATURE] Keep the OpenAPI specification vendor-neutral / vendor-agnostic

reta commented 3 months ago

Is your feature request related to a problem?

Coming from https://github.com/opensearch-project/opensearch-java/pull/1152#discussion_r1726911495 and https://github.com/opensearch-project/opensearch-api-specification/pull/510, I understand the bias towards supporting AWS, but I honestly believe the spec has to be vendor neutral / agnostic.

What solution would you like?

I believe it should not "pollute" the spec with x-distributions-excluded:. It should be feasible to keep the distribution specific details outside of the spec and allow the distribution specific spec to be produced if needed:

For example:

the vendor neutral spec: openapi-oss.yaml
the AWS managed spec: openapi-aws.yaml (would only contain new APIs, excluded APIs or APIs that are tailored)

The final spec for AWS: openapi-oss.yaml + openapi-aws.yaml

What alternatives have you considered?

Keep asking vendors to add their own distribution perferences

Do you have any additional context?

N/A

dblock commented 3 months ago

When I introduced this feature in https://github.com/opensearch-project/opensearch-api-specification/pull/510 I did not intend it to be biased towards supporting AWS in any way, but I did want to run the tests against Amazon Managed so I annotated some APIs.

As is, the spec welcomes other keywords, be it oracle-oci, aiven-managed or something platform-specific like community-freebsd in the short term. Therefore I believe the spec is vendor netural today. I think it's ok to mention vendors in the spec this way, but we can talk about it.

From the mechanics POV, what you are looking at is the source for the spec, not the spec. We currently publish the all-versions spec in a release that includes these x- options. But this repo is capable of producing a version-specific spec (e.g. 1.3.x, without APIs added in/after 2.0 for example or vice-versa) (see https://github.com/opensearch-project/opensearch-api-specification/blob/main/DEVELOPER_GUIDE.md#spec-merger), and that could be extended to produce a vendor-less (by stripping x-distribution options), or a vendor-specific spec (e.g. -amazon-managed.yaml).

Given the above, what's a better place to put the information that a particular distribution of OpenSearch supports or doesn't support an API or an API parameter?

reta commented 3 months ago

As is, the spec welcomes other keywords, be it oracle-oci, aiven-managed or something platform-specific like community-freebsd in the short term. Therefore I believe the spec is vendor netural today. I think it's ok to mention vendors in the spec this way, but we can talk about it.

I think it is an option but not sustainable way to manage that for many reasons:

it "pollutes" the spec with cloud vendor details (I can easily imagine such list offerings to grow unbound)
this is really a cloud vendor decision, and it should be also its responsibility to maintain

Given the above, what's a better place to put the information that a particular distribution of OpenSearch supports or doesn't support an API or an API parameter?

This is 100% cloud vendor decision: dev portal, Github,web site, ... (anywhere literally). What we could help with - tooling for easy reconciliation.

dblock commented 3 months ago

it "pollutes" the spec with cloud vendor details (I can easily imagine such list offerings to grow unbound)

I suppose I wouldn't be against it! As a positive, this gives incentive to cloud vendors not to deviate from spec and gives the community tooling to check that a vendor complies with the spec.

dbwiddis commented 3 months ago

it "pollutes" the spec with cloud vendor details (I can easily imagine such list offerings to grow unbound)

I suppose I wouldn't be against it! As a positive, this gives incentive to cloud vendors not to deviate from spec and gives the community tooling to check that a vendor complies with the spec.

To be a bit more specific/pedantic, I think it's less a "complies with the spec" as "allows a spec API".

I happened on this issue following a link trail with a completely unrelated issue, but it's very relevant. Specifically, the spec includes core and multiple different plugin APIs. Any given provider may choose to remove some plugins, or possibly remove access to a subset of the APIs. The "cloud vendor" certainly needs a way to annotate the spec with a "we do this"/"we don't do this" at various levels of detail (the entire API, or a limitation on which enum values of a parameter are legal, etc.).

What makes this more than a "cloud vendor" problem is the fact that front-ends / apps need to know what is supported. It's a poor app user / customer experience to have a menu where you can select an option that the API is spec'd to support, and it fails.

TLDR: We need a way for an app/front-end to query the spec (filtered by \<insert cloud vendor here>) to know whether to display options on it's UI. I don't know if that's part of this spec or part of some companion filtering program, but it's an obvious need.

reta commented 3 months ago

To be a bit more specific/pedantic, I think it's less a "complies with the spec" as "allows a spec API".

I believe it is out of scope for us to consider the custom deployment / tailoring / forking / etc: as a project, we provide single unified distribution (core / plugin / extension specs), what consumers could do with it - it is their choice.

TLDR: We need a way for an app/front-end to query the spec (filtered by ) to know whether to display options on it's UI. I don't know if that's part of this spec or part of some companion filtering program, but it's an obvious need.

The service (filtered by <insert cloud vendor here>) should provide the up to date spec for the UI / any 3rd party if the spec diverges. However, we could simplify the assemblage of such specs as per suggestion in description (this is just one of the ways to do it).

dblock commented 3 months ago

TLDR: We need a way for an app/front-end to query the spec (filtered by ) to know whether to display options on it's UI. I don't know if that's part of this spec or part of some companion filtering program, but it's an obvious need

However, we could simplify the assemblage of such specs as per suggestion in description (this is just one of the ways to do it).

This project contains OpenApiVersionExtractor that can generate a spec for a given version and distribution of OpenSearch using these annotations. The services should make the output of that available under https://service/openapi-spec.yaml or something like that. The merger tool needs a command line switch to take in the distribution like it takes in the version already (please contribute?).

We can move these annotations from the spec into some yaml that doesn't have to live in this repo (eventually), which I think would satisfy all the concerns above. Do note though that the spec is changing fast right now given that it's ~50% complete, I'd prefer not to do it right away.

reta commented 3 months ago

Do note though that the spec is changing fast right now given that it's ~50% complete, I'd prefer not to do it right away.

Getting rid of x-distribution-* would simplify the spec completion since we'll be laser focused on open source distribution only.

dbwiddis commented 3 months ago

We can move these annotations from the spec into some yaml that doesn't have to live in this repo (eventually), which I think would satisfy all the concerns above.

A possible solution would likely be a fork of this repo customized (by the provider) and kept up to date with merges/cherry-picks.

As long as the feature is available to use it should help satisfy the need.

opensearch-project / opensearch-api-specification