Conformance tests do not run any tests

[lentzi90]

/kind bug

What steps did you take and what happened:

As can be seen in the logs, the conformance tests do not run any tests as it is now.

 Will run 0 of 0 specs
Ran 0 of 0 Specs in 0.000 seconds
SUCCESS! -- 0 Passed | 0 Failed | 0 Pending | 0 Skipped
PASS 

It should definitely run some tests!

What did you expect to happen:

These tests should run.

Anything else you would like to add:

I think this is just a question of selecting tests correctly in the config or adjusting the make target or script.

/help /good-first-issue

[k8s-ci-robot]

@lentzi90: This request has been marked as suitable for new contributors.

Guidelines

Please ensure that the issue body includes answers to the following questions:

• Why are we solving this issue?
• To address this issue, are there any code changes? If there are code changes, what needs to be done in the code and what places can the assignee treat as reference points?
• Does this issue have zero to low barrier of entry?
• How can the assignee reach out to you for help?

For more details on the requirements of such an issue, please see here and ensure that they are met.

If this request no longer meets these requirements, the label can be removed by commenting with the /remove-good-first-issue command.

In response to [this](https://github.com/kubernetes-sigs/cluster-api-provider-openstack/issues/1800): >/kind bug > >**What steps did you take and what happened:** > >As can be seen in the [logs](https://prow.k8s.io/view/gs/kubernetes-jenkins/logs/periodic-cluster-api-provider-openstack-conformance-test-main-with-k8s-ci-artifacts/1737711524786475008#1:build-log.txt%3A1731), the conformance tests do not run any tests as it is now. > >``` > Will run 0 of 0 specs >Ran 0 of 0 Specs in 0.000 seconds >SUCCESS! -- 0 Passed | 0 Failed | 0 Pending | 0 Skipped >PASS >``` > >It should definitely run some tests! > >**What did you expect to happen:** > >[These tests should run](https://github.com/kubernetes-sigs/cluster-api-provider-openstack/blob/main/test/e2e/suites/conformance/conformance_test.go). > > >**Anything else you would like to add:** > >I think this is just a question of selecting tests correctly in the [config](https://github.com/kubernetes/test-infra/blob/73538ad97a1f454e05e6c31fcd74bdd7ac89efaf/config/jobs/kubernetes-sigs/cluster-api-provider-openstack/cluster-api-provider-openstack-periodics.yaml#L52-L94) or adjusting the make target or [script](https://github.com/kubernetes-sigs/cluster-api-provider-openstack/blob/main/scripts/ci-conformance.sh). > > >/help >/good-first-issue Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md). If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository.

[prashantrewar]

hey @lentzi90 , i would like to work on this, could you please guide me for this issue.

[lentzi90]

Sure! I think the issue is that ginkgo is skipping the tests. I'm not sure exactly why, but this is how it works:

1. The prow job is defined here: https://github.com/kubernetes/test-infra/blob/73538ad97a1f454e05e6c31fcd74bdd7ac89efaf/config/jobs/kubernetes-sigs/cluster-api-provider-openstack/cluster-api-provider-openstack-periodics.yaml#L52-L94 For other jobs we set some environment variables like here that could affect what tests are run. Maybe we need to add something?
2. The job runs this script: https://github.com/kubernetes-sigs/cluster-api-provider-openstack/blob/main/scripts/ci-conformance.sh
3. The script executes make test-conformance-fast: https://github.com/kubernetes-sigs/cluster-api-provider-openstack/blob/fdc5f492bfc958446e2de3f01ab8b119751fc35d/Makefile#L205-L206 This is where I suspect we have some issue. I think we need to add a -focus=conformance perhaps.

Is there anything in particular you are wondering?

[lentzi90]

@mdbooth any idea what the difference between conformance and conformance-fast is supposed to be? https://github.com/kubernetes-sigs/cluster-api-provider-openstack/blob/fdc5f492bfc958446e2de3f01ab8b119751fc35d/Makefile#L202-L206

I'm getting the feeling that the "fast" test is not meant to do much, but is it supposed to skip all the tests? :thinking:

[prashantrewar]

/assign

[mdbooth]

@mdbooth any idea what the difference between conformance and conformance-fast is supposed to be?

I'm getting the feeling that the "fast" test is not meant to do much, but is it supposed to skip all the tests? 🤔

I don't, I'm afraid: I've never been down this rabbit hole. However, I agree that skipping all the tests is clearly not useful. I'm guessing this is a copied pattern, though. Do other providers have these targets, and if so what do they do? I normally look at CAPA and CAPZ first for this kind of thing.

[lentzi90]

Good idea! CAPA has no fast conformance variant, CAPZ has, but it seems to be just higher parallelism and a possibility to use special config. From what I can tell both use the normal conformance test in CI. I suggest we try the same.

@prashantrewar could you change the script to do make test-conformance instead of make test-conformance-fast, and remove the test-conformance-fast target all together?

[prashantrewar]

Hey @lentzi90 I created a PR on this issue, could you please take a look