search

kubernetes-sigs / ingress-controller-conformance

Repository for a compliance specification of ingress-controllers.
Apache License 2.0
43 stars 36 forks source link

Service APIs conformance divergence/future convergence #87

Closed sunjayBhatia closed 3 years ago

sunjayBhatia commented 3 years ago

Just wanted to open an issue here to cross-link https://github.com/kubernetes-sigs/service-apis/pull/516

It appears the Service APIs conformance suite is significantly diverging from this one, though https://github.com/kubernetes-sigs/service-apis/issues/20 has some comments indicating it shouldn't

Do the maintainers of this repo have any thoughts or feedback to give here? How would these two suites converge in the future given the starting point of Service APIs conformance and the state of Ingress conformance?

aledbf commented 3 years ago

How would these two suites converge in the future given the starting point of Service APIs conformance and the state of Ingress conformance?

My personal opinion is that there should be two different conformance suites. Both APIs are not comparable, and having one could be confusing, thinking both APIs are interchangeable or, even worse, it can be mixed. Also, Ingress is pretty simple in comparison to service-apis.

sunjayBhatia commented 3 years ago

Maybe a better way to phrase it is not necessarily in terms on content or shared tests, but maybe framework, reporting, etc.? Having to run multiple completely different types of test framework for "similar" high level conformance tests on the same/same type of component sounds like it will be messy

aledbf commented 3 years ago

@robscott ping

robscott commented 3 years ago

Thanks for raising this issue! I would like these test suites to share code wherever possible and maintain similar structures. Although my service-apis PR does not use godog, it does share code and general structure with this project. Both conformance test approaches fundamentally involve applying some manifests and then describing the expected behavior. I explored using godog as well, but as the tests became increasingly complex, I wanted to get some feedback on an approach that simply used go's testing framework.

That PR is still marked as a WIP and has many issues, but it was meant to be the start of a conversation. We also discussed potential approaches in more detail on the service-apis community call last week.

My theory is that writing tests with godog may become too complex with service-apis since we're trying to test and describe the relationships between multiple resources. I think BDD tests are awesome for certain use cases, but as I've started to dig into the complexity we'll need to cover, I've had a hard time finding a clean way to accomplish it with godog.

I also found the extra layer of abstraction added by godog resulted in some difficulty debugging test failures when they happened. There are certainly ways around this, but as we're just starting out with service-apis conformance tests, it seemed like optimizing for simplicity and debuggability at first would be worthwhile.

Something I mentioned on the call last week was that I do want to concentrate our testing logic in helpers wherever possible so if we choose to transition to godog it will be relatively straightforward. There are some awesome utilities here that would be good to share between repositories, we just may need to make some slight modifications to make them a bit more generic.

If you've got ideas for how we can be doing this better, I'd love the help! This is still very new. I intentionally created a PR early in the process to try to start a conversation and get feedback on my idea.

fejta-bot commented 3 years ago

Issues go stale after 90d of inactivity. Mark the issue as fresh with /remove-lifecycle stale. Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-contributor-experience at kubernetes/community. /lifecycle stale

fejta-bot commented 3 years ago

Stale issues rot after 30d of inactivity. Mark the issue as fresh with /remove-lifecycle rotten. Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

Send feedback to sig-contributor-experience at kubernetes/community. /lifecycle rotten

fejta-bot commented 3 years ago

Rotten issues close after 30d of inactivity. Reopen the issue with /reopen. Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-contributor-experience at kubernetes/community. /close

k8s-ci-robot commented 3 years ago

@fejta-bot: Closing this issue.

In response to [this](https://github.com/kubernetes-sigs/ingress-controller-conformance/issues/87#issuecomment-859136519): >Rotten issues close after 30d of inactivity. >Reopen the issue with `/reopen`. >Mark the issue as fresh with `/remove-lifecycle rotten`. > >Send feedback to sig-contributor-experience at [kubernetes/community](https://github.com/kubernetes/community). >/close Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md). If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository.