search

kubernetes-sigs / ingress-controller-conformance

Repository for a compliance specification of ingress-controllers.
Apache License 2.0
43 stars 36 forks source link

Add ability to apply test filters and skips #86

Closed sunjayBhatia closed 2 years ago

sunjayBhatia commented 3 years ago

Currently it is not possible when running the Ingress conformance tests via Sonobuoy to filter or skip tests. Sonobuoy offers the --e2e-focus|skip flags that allow allow users to choose what tests to run and it would be great to be able to do so. A specific use case is for an Ingress controller that is partially conformant, still being able to run a subset of the test suite in CI for continuous validation.

In addition, for development, being able to filter and skip is quite useful, however cucumber tags are not really in use on all of the various features/scenarios: https://github.com/cucumber/godog#tags

It would be great if we could potentially plumb the cucumber filtering ability through so it can be used at the Sonobuoy level and even add some granular tagging to be able to filter tests

aledbf commented 3 years ago

In addition, for development, being able to filter and skip is quite useful, however cucumber tags are not really in use on all of the various features/scenarios: https://github.com/cucumber/godog#tags

Right now, the existing tags are defined at the feature level, and the tag flag is there.

Can you provide examples of what you want to do?

sunjayBhatia commented 3 years ago

Features aren't yet tagged distinctly, add at the very least a tag to each feature maybe more granular tags to individual scenarios so they can be selected to run (see "partially conformant" use case in original description)

At the moment the tag flag doesn't get used here, add that so running via Sonobuoy we can select a filter

sunjayBhatia commented 3 years ago

@aledbf does the above sound reasonable to you? I can make a PR if it is something that would be mergeable

fejta-bot commented 3 years ago

Issues go stale after 90d of inactivity. Mark the issue as fresh with /remove-lifecycle stale. Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-contributor-experience at kubernetes/community. /lifecycle stale

fejta-bot commented 3 years ago

Stale issues rot after 30d of inactivity. Mark the issue as fresh with /remove-lifecycle rotten. Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

Send feedback to sig-contributor-experience at kubernetes/community. /lifecycle rotten

fejta-bot commented 2 years ago

Rotten issues close after 30d of inactivity. Reopen the issue with /reopen. Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-contributor-experience at kubernetes/community. /close

k8s-ci-robot commented 2 years ago

@fejta-bot: Closing this issue.

In response to [this](https://github.com/kubernetes-sigs/ingress-controller-conformance/issues/86#issuecomment-877954566): >Rotten issues close after 30d of inactivity. >Reopen the issue with `/reopen`. >Mark the issue as fresh with `/remove-lifecycle rotten`. > >Send feedback to sig-contributor-experience at [kubernetes/community](https://github.com/kubernetes/community). >/close Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md). If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository.