Conformance: Allow skipping subtests

[briantkennedy]

What would you like to be added:

At the moment, it's possible to skip conformance tests. I would like the ability to skip individual subtests within a conformance test.

Why this is needed:

Many of the conformance test multiple behaviors. Controllers trying to achieve conformance may want to use a set of known passing tests in CI to prevent regression while aspiring to achieve a 100% pass rate on conformance. With the current skip mechanism, it's not possible to include passing subtests while excluding failing ones.

[howardjohn]

Interested in this as well

[sunjayBhatia]

+1

[youngnick]

Sounds reasonable to me, as long as using the "skip subtests" feature means that you can't be conformant (maybe it prohibits you from generating a conformance profile result?)

[howardjohn]

The profiles already report skipping top level tests so I think it is fine on that regard (assuming the implementation reports the subtest skipping as well, of course).

On Tue, Jul 25, 2023 at 11:52 PM Nick Young @.***> wrote:

Sounds reasonable to me, as long as using the "skip subtests" feature means that you can't be conformant (maybe it prohibits you from generating a conformance profile result?)

— Reply to this email directly, view it on GitHub https://github.com/kubernetes-sigs/gateway-api/issues/1996#issuecomment-1651083590, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAEYGXIMGHZUITCZGXHQS5DXSC5ENANCNFSM6AAAAAAXWFWA74 . You are receiving this because you commented.Message ID: @.***>

[shaneutt]

Sounds like a fine thing to add.

/triage accepted

It's not clear however that this is something that should hold up our GA release, so I'll remove it from the board.

[k8s-triage-robot]

This issue has not been updated in over 1 year, and should be re-triaged.

You can:

• Confirm that this issue is still relevant with /triage accepted (org members only)
• Close this issue with /close

For more details on the triage process, see https://www.kubernetes.dev/docs/guide/issue-triage/

/remove-triage accepted