Automate exhaustive generation of small test cases

[roehrt]

Often a problem has a family of test cases which is small enough to be enumerated explicitly. Currently, enumerating these test cases individually is necessary, as illustrated below:

- small-1: small -c 1
- small-2: small -c 2
- small-3: small -c 3
...

Introducing a shorthand notation for this process would be awesome. Building upon the {seed} and lo..hi syntax, one approach could be:

- small: small -c {1..10}

Alternatively, for a specified set of tokens provided as a comma-separated list:

- small: small -c {some,important,tokens}

One alternative would be to allow generators to output multiple test cases in one invocation. However, I doubt the benefit in comparison to the cost of changing generator semantics. Also, at least I don't know of a clean way to separate the test cases.

[mpsijm]

Thanks for taking the time to write an issue! See also the discussion in https://github.com/RagnarGrootKoerkamp/BAPCtools/issues/271#issuecomment-1611211270 and onwards. A few weeks ago, we discussed this between the BAPCtools developers, and we decided that it's not worth the complexity right now:

Nonetheless, let's keep this issue open, since multiple people have asked for this.

[mzuenni]

For this we could substitute {count} with numbers 1..n when using the count key?

[RagnarGrootKoerkamp]

Oh adding the {count} variable sounds quite nice and like a natural extension indeed. This is restrictive enough to still encourage thinking about actually interesting/varied arguments for generators, but general enough to allow enumerating them.

Should it count from 0 instead? Not quite sure.

[mpsijm]

Supporting a {count} placeholder in generator invocations may be a rabbit hole for more feature requests regarding parameterized generator invocations. But if we're okay with that, I think it's a nice addition to help generating test cases exhaustively :slightly_smiling_face:

I think it should start counting from 1, based on the fact that our styleguide says to use 1-based indexing where possible. If you need anything else, you can modify it within the generator itself (e.g. do count - 1 or 2 ** count or something else).

[mzuenni]

I think this one is fine... else people might start to abuse {name} for this since it also contains the index.

[RagnarGrootKoerkamp]

{name} is always just testcase now I believe? But anyway, let's add this.

[mpsijm]

An update to generators.yaml that replaces {count} with 1..x (where x is determined by the count: x key) has been merged in #379. I think that's fixes this issue :slightly_smiling_face: