Generators: Add `testcase: count: X` for multiple testcase copies

[RagnarGrootKoerkamp]

Currently we do

- random: gen.py {seed:1}
- random: gen.py {seed:2}
- random: gen.py {seed:3}

which would be nice to replace with

- random:
    input: gen.py {seed}
    count: 3

(And maybe input: should be renamed to gen:?)

[mpsijm]

Duplicate of #135 and #162? (the fact that you've opened the same issue three times, probably means we should implement it :joy:)

Also relates to #272 (/ #220), for the naming of the input:/gen: key.

[RagnarGrootKoerkamp]

Alternative: Allow {seed:1,2,3,10-20}. It's shorter and can be done inline, and gives more control over the seed values being used. But also, the fact that count: N takes a separate line feels more obvious, and then tooling doesn't have to count the exact number of cases listed in the seed to determine the correct numbering. Seems much less brittle over all.

(Also, currently the part of {seed:...} after the : is just a string to change the hash of the entire command. Reinterpreting that as a list is somewhat ugly IMO)

[RagnarGrootKoerkamp]

Hmm; the problem with 'inline' numbering is that that wouldn't work if testcases aren't already numbered in the first place. For named cases you'd have to do random-1, random-2, or so.

[thorehusfeldt]

This is of course super useful for other generators than random ones.

Assume trees.py --i is a generator that produces all nonisomorphic trees on 7 vertices (indexed by k), I would love to do

- small:
    input: trees.py --k {seed}
    count: {2,4,5-12, 14}

instead of

- small trees.py --k 1
- small trees.py --k 2
- small trees.py --k 3
...

[RagnarGrootKoerkamp]

Ohh, interesting idea.

Some context Currently we allow {seed} and {seed:<any kind of salt>}, and the entire thing is replace by an integer hash of the entire generate command. (This way, gen.py {seed} 1 and gen.py {seed} 2 have distinct seeds, and gen.py {seed:1} and gen.py {seed:2} also have distinct seeds. This also means that {seed} is not a small incremental int typically. It's supposed to be used has seed a random number generator, but I still think it's good if {seed} is always pseudo-random, never 1, 2, 3, ....

Variables The above means that your example isn't the intended way of using {seed}. But I think we can do something cleaner:

- testcase:
    input: gen.py -n {n}
    n: [1,2,3, 10, 100, 1000]

Note that the value of n here is simply a list, and we generate one testcase for each element of the list.

Or maybe we should separate the variable n into a separate scope? (I don't think it's really needed though.)

- testcase:
    input: gen.py -n {n}
    vars:
      n: [1,2,3, 10, 100, 1000]

Expressions Since we already use python syntax, we could also allow eg input: gen.py -n {2**i}, but I don't think this is really necessary.

Multiple variables For multiple variables, we have to figure out how to make combinations:

- testcase:
    input: gen.py -n {n} -k {k}
    n: [1,2,3, 10, 100, 1000]
    k: [2,3,4,5,6,7]

This could mean either: a) cartesian product of the two lists, creating 6*6 = 36 testcases b) 'zipped' parameters, creating testcases with (n,k) in [(1,2), (2,3), (3, 4), (10, 5), (100, 6), (1000, 7)]. My initial preference is zipping: cartesian products quickly explode the number of testcases, and in practice I often want two parameters to grow in step with each other (e.g. the number of vertices and edges of a graph -- cartesian product wouldn't make sense there). This would imply an assertion that all lists have the same length.

Seeds Back to the original question, we could reuse the variable syntax and do

- testcase:
    input: gen.py {seed}
    seed: [1, 2, 3]

this would then mean {seed} is replaced by 1, 2, and 3. I think I can live with this since ideally {seed} should be used to seed an RNG. Alternatively we could special case this to mean that {seed} is replaced by the hash of gen.py {seed:1} and so on, which is backwards compatible with how we already do thing.

Counting the number of testcases I'd say that any 'unrecognised key' (ie not input solution visualizer in ans desc hint ...) is automatically assumed to be a variable and required to have list type. All these lists must have the same length (assuming we zip them), and that length is used as the count of testcases for this rule.

[RagnarGrootKoerkamp]

Hmm, one issue with the zipping is that for longer lists the correspondence between elements becomes unclear, especially if the numbers don't have the same length. We could do something like:

- testcase:
    input: gen.py -n {n} -k {k}
    vars: [{n: 1, k: 1}, {n: 2, k: 2}, ...]

but that's super verbose. Either way, I don't think that manually copying cases is that much of an issue, eg from last NWERC:

- manual-three: manual.py 3 1        
- manual-three: manual.py 301 100    
- manual-three: manual.py 299 100    
- manual-three: manual.py 1000 333   
- manual-three: manual.py 1000 334   
- manual-triangular: manual.py 14 5  
- manual-triangular: manual.py 16 5  
- manual-triangular: manual.py 15 4  
- manual-triangular: manual.py 5 2   
- manual-triangular: manual.py 818 38
- manual-triangular: manual.py 818 39
- manual-triangular: manual.py 818 40
...

[mpsijm]

My two cents:

• I agree that {seed} should always be pseudo-random.
• I prefer a separate key for all variables (or as you called it here, a "separate scope"). I don't really like the idea that "recognized keys" and "unrecognized keys" have different behaviour in the same scope.
• About the zipping, I think your last comment would work well for me. I assume you can also write this as follows?

• testcase: input: gen.py -n {n} -k {k} vars:
  • {n: 1, k: 1}
  • {n: 2, k: 2}
  • ...

    
    Or possibly even like this, if you want to be very verbose?

- testcase:
    input: gen.py -n {n} -k {k}
    vars:
      - n: 1
        k: 1
      - n: 2
        k: 2
      - ...

And then a final question: how does vars combine with count? My intuition would say that, for every (tuple of) value(s), the test case would be duplicated count times. So in the following example, you would get six test cases:

- testcase:
    input: gen.py -n {n} -k {k} -seed {seed}
    vars: [{n: 1, k: 1}, {n: 2, k: 2}]
    count: 3

Now that I think of it, count should probably be ignored (or even better, rejected) when the input does not contain {seed}.

[RagnarGrootKoerkamp]

• Yes, count: should require {seed}.
• I'm really not a fan of the dictionary syntax vars: [{n:1}, {n:2}, {n:3}, ..] is so verbose compared to vars: n: [1,2,3, ...], that just copying the entire line modifying the two numbers is simpler. I think for multiple vars something like vars: [(1,2), (2, 4), (3, 8), ...] would be cool. Could be vars: [n, k] and vals: [(1, 2), (2, 4), ...] maybe?

[mpsijm]

I think for multiple vars something like vars: [(1,2), (2, 4), (3, 8), ...]

Yes, that's nice! :smile: In that case, do we need to name the variables? We could also go for something like:

- testcase:
    input: gen.py -n {} -k {}
    vars: [(1, 2), (2, 4), ...]

Note that I usually write my Python generators with positional arguments, so that would look something like input: gen.py {} {}. I guess that's fine, if I put a comment documenting which variable is which :stuck_out_tongue:

[RagnarGrootKoerkamp]

Oops random misclick. Especially since arguments are usually positional it's nice to have them self-documenting and use {n} I think. But not naming them does make things simpler.

We could also say that gen.py {n} {k} is written but we substitute from left to right (positionally), but that's kind of a weird middle ground.