search

submariner-io / lighthouse

DNS service discovery across connected Kubernetes clusters.
https://submariner-io.github.io/architecture/service-discovery/
Apache License 2.0
101 stars 35 forks source link

Add GHA to run the MCS conformance tests #1630

Open tpantelis opened 2 months ago

tpantelis commented 2 months ago

This would identify and keep us aware of which aspects of the MCS spec where we may be non-compliant. Initially, we could just have the job run on-demand and eventually run on every PR once we're fully compliant.

dfarrell07 commented 2 months ago

There's likely a way to include these tests in our E2E suite.

tpantelis commented 2 months ago

There's likely a way to include these tests in our E2E suite.

I think these should be separate from the E2E tests. I propose we create a "conformance" dir under "test" (sibling to "e2e") with a _conformancetest.go file that imports sigs.k8s.io/mcs-api/conformance and invokes TestConformance. The GHA would run go test in "test/conformance". There's probably more details to work out but that's the general idea.

tpantelis commented 1 month ago

I tried this out and got it to run the conformance tests. The problem though is that LH imports v0.1.0 of sigs.k8s.io/mcs-api but it doesn't have the conformance tests and the current master branch has some incompatible changes to the MCS APIs so we can't import it in the top-level go.mod. So I added a separate go.mod in the test/conformance module to get around that but that causes issues with the Go linting job. I don't think think this approach is viable.

There is a new sigs.k8s.io/mcs-api release planned soon so we can wait for that although we'll have to do that in the next release due to MCS API changes. Either way, I think it would be better to clone the sigs.k8s.io/mcs-api repo with the desired commit in the GHA and run go test directly in the directory since the conformance tests are really tied to the MCS spec and not specifically to the API version so they'll likely evolve separately.