--ci-generate is not compatible with --repeat

[flx42]

Consider the following test:

import reframe as rfm
import reframe.utility.sanity as sn

@rfm.simple_test
class MyTest(rfm.RunOnlyRegressionTest):
    valid_systems = ['*']
    valid_prog_environs = ['*']

    executable = "/bin/true"

    @sanity_function
    def check_exit(self):
        return sn.assert_eq(self.job.exitcode, 0, msg="Exited with exit code {0}")

Using --ci-generate --repeat=2 yields the following:

$ reframe -c mytest.py --ci-generate=./sbatch.yaml --repeat=2
$ cat sbatch.yaml 
cache:
  key: ${CI_COMMIT_REF_SLUG}
  paths:
  - rfm-stage/${CI_COMMIT_SHORT_SHA}
stages:
- rfm-stage-0
MyTest_1:
  stage: rfm-stage-0
  script:
  - reframe --prefix=rfm-stage/${CI_COMMIT_SHORT_SHA}  -c mytest.py  --report-file=MyTest_1-report.json  --report-junit=MyTest_1-report.xml  -n /f425e3ea -r
  artifacts:
    paths:
    - MyTest_1-report.json
  needs: []
MyTest_0:
  stage: rfm-stage-0
  script:
  - reframe --prefix=rfm-stage/${CI_COMMIT_SHORT_SHA}  -c mytest.py  --report-file=MyTest_0-report.json  --report-junit=MyTest_0-report.xml  -n /63b97192 -r
  artifacts:
    paths:
    - MyTest_0-report.json
  needs: []

But if you run this manually, you will see that it won't work:

$ reframe -c mytest.py  -n /f425e3ea -r

[...]

[==========] Running 0 check(s)
[==========] Started on Thu Jun  8 14:56:23 2023 

[----------] start processing checks
[----------] all spawned checks have finished

[  PASSED  ] Ran 0/0 test case(s) from 0 check(s) (0 failure(s), 0 skipped, 0 aborted)
[==========] Finished on Thu Jun  8 14:56:23 2023 
Log file(s) saved in '/tmp/rfm-n340qjv0.log'

But even if you add --repeat=2 to the command, it still won't match any test, I suppose it's filtering tests before instancing new tests for each repeat?

[vkarak]

Indeed, the --repeat and the similar options work after the test filtering, that's why the reframe -c ../misc/rfm-support/issue-2915/test.py --repeat=2 -n /f425e3ea -l does not work. But that was by design, as the idea is to repeat the tests that you have selected.

However, I agree that it should work with the --ci-generate.

[vkarak]

I will convert this to a feature request as new functionality is required to support it and will open a PR with a note in the docs.