Cumbersome experimental conformance reporting suite UX

[sunjayBhatia]

Currently in Contour we use these conformance suite options to ensure we run all tests by default, but intentionally have to exclude a few that either don't apply or don't pass. In our case, we exclude the Mesh feature and one of the extended support HTTPProxy tests.

Doing this in the experimental conformance suite currently is not as simple as that to get the full report on supported extended features out. Instead you need something like:

    cSuite, err := suite.NewExperimentalConformanceTestSuite(suite.ExperimentalConformanceOptions{
        Options: suite.Options{
                        ...
                        // Note this vs. using the EnableAllSupportedFeatures and ExemptFeatures field
            SupportedFeatures: suite.AllFeatures.Delete(suite.MeshCoreFeatures.UnsortedList()...),
            SkipTests: []string{
                tests.HTTPRouteRedirectPortAndScheme.ShortName,
            },
        },
        Implementation: conformance_v1alpha1.Implementation{
                         ...
        },
        ConformanceProfiles: sets.New(
            suite.HTTPConformanceProfileName,
            suite.TLSConformanceProfileName,
        ),
    })

It would be great if using the same suite.Options setup was possible between the two versions, especially if the current one is eventually replaced by the version that can report conformance results.

Notes:

• Logic for building up the experimental suite features is here: https://github.com/kubernetes-sigs/gateway-api/blob/6d0e6ccbccc73def46bce2e52ed50ca1e480a890/conformance/utils/suite/experimental_suite.go#L114-L147
• vs. standard logic here: https://github.com/kubernetes-sigs/gateway-api/blob/6d0e6ccbccc73def46bce2e52ed50ca1e480a890/conformance/utils/suite/suite.go#L92-L105
• Seems like we can simplify the experimental suite logic a bit and add in support for the exempt features etc.

[shaneutt]

Seems like a good area for improvement, thank you for trying out the experimental conformance profile test suite and providing some feedback!

/triage accepted

I don't think we need to necessarily worry about this for GA however, so I'm not intending to add it to that milestone for now but instead as something to follow up on post-GA:

/priority important-longterm

Ultimately the experimental suite should in time (as we graduate beyond experimental) replace the current test suite so in theory this gets resolved as a bi-product of normal progression? However if there's some more immediate need or something I've missed here please do let me know.

/cc @mlavacca

[mlavacca]

Thanks @sunjayBhatia, this would look like a good improvement in usability

Ultimately the experimental suite should in time (as we graduate beyond experimental) replace the current test suite so in theory this gets resolved as a bi-product of normal progression? However if there's some more immediate need or something I've missed here please do let me know.

+1, even if we should be careful not to lose track of this one, even because when the experimental will become the new standard, most likely we'll be deleting the standard one (or deprecating it), and we should take care of adding this improvement there.

[k8s-triage-robot]

This issue has not been updated in over 1 year, and should be re-triaged.

You can:

• Confirm that this issue is still relevant with /triage accepted (org members only)
• Close this issue with /close

For more details on the triage process, see https://www.kubernetes.dev/docs/guide/issue-triage/

/remove-triage accepted