search

kubernetes-sigs / ingress2gateway

Convert Ingress resources to Gateway API resources
Apache License 2.0
324 stars 55 forks source link

Proposal: Add Conformance Tests For Conversion Logic #103

Open levikobi opened 7 months ago

levikobi commented 7 months ago

Currently, each provider configures their own test cases. This isn't ideal for two reasons:

  1. Implementors have to create entire test suites from scratch.
  2. Test coverage isn't aligned between different implementations across the same features.

Given that Gateway-API objects should look the same across different implementations across the core and extended features, we can construct a conformance suite. In conformance, we define a suite of tests for each feature (canary, mirroring, etc).

Since the Ingress manifests are different across different implementations, each implementation will have to configure a set of YAML files to be consumed by the conformance tests.

If a provider adds support for converting a core/extended feature, it must register itself for the associated suite of conformance tests.

This is just in the idea phase, and I don't have a design yet. But, if this makes sense for the community, I will gladly take it to the next step.

cc @robscott @LiorLieberman @mlavacca @dpasiukevich

LiorLieberman commented 6 months ago

This sounds interesting to me.

Trying to think about the proposed high level flow you had in mind?

  1. Provider claims it supports HTTPRequestPathRedirect
  2. Provider adds current yamls (CRDs and/or Ingresses) as an input to this conformance tests
  3. Expected Gateway API objects are shared between all the providers that subscribed to this feature
  4. ingress2gateway tool is invoked with this Provider to read and convert resources from the input files
  5. We compare diffs

Assuming this is more or less what you have in mind (do shout if it isnt) I am +1 overall. Though I think we will need to check if bullet 3 is viable - meaning if we can convert identically different providers resources to the same Gateway resource/s for a particular feature. (I think it should be the case but we need probably input from the wider community)

levikobi commented 6 months ago

This is at a high level what I had in mind @LiorLieberman . I agree with your comment on bullet 3, I am not sure about it either - it definitely needs verification.

mlavacca commented 6 months ago

I like this idea 👍 . For what concerns point 3, we may need some mechanism to customize the expected output from the provider (ImplementationSpecific options comes to my mind for example), as some providers can have different expected behaviours.

LiorLieberman commented 6 months ago

Thanks @mlavacca it is a good idea.

as some providers can have different expected behaviours.

Is this the case though? like do you have examples where this can happen? I just raised it because I was not sure if it is the case or not.

/cc @robscott for giving your input and/or cc'ing other implementors who can provide their input

k8s-triage-robot commented 3 months ago

The Kubernetes project currently lacks enough contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

You can:

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

levikobi commented 3 months ago

/remove-lifecycle stale

k8s-triage-robot commented 1 week ago

The Kubernetes project currently lacks enough contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

You can:

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale