Shorthand for testcases/scripts that use the same custom oracle

[pdawyndt]

If all scripts of the same unit, and/or all testcases of the same script use the same custom oracle, it should be possible to define that oracle at the level of the unit or the script and use the inheritance principle.

Here's an example where all scripts in the unit use the same oracle.

- unit: 'Sum of three cubes'
  testcases:
    - stdin: '3'
      stdout: 
        value: |
          1
          1
          1
        evaluator: 'custom'
        language: 'python'
        name: 'sum_of_three_cubes'
        file: 'oracle.py'
    - stdin: '33'
      stdout:
        value: |
          8866128975287528
          -8778405442862239
          -2736111468807040
    - stdin: '42'
    - stdout:
        value: |
          -80538738812075974
          80435758145817515
          12602123297335631
        evaluator: 'custom'
        language: 'python'
        name: 'sum_of_three_cubes'
        file: 'oracle.py'

This repetition could be avoided by putting the oracle at the level of the unit:

- unit: 'Sum of three cubes'
  stdout: 
    evaluator: 'custom'
    language: 'python'
    name: 'sum_of_three_cubes'
    file: 'oracle.py'
  testcases:
    - stdin: '3'
      stdout: |
        1
        1
        1
    - stdin: '33'
      stdout: |
        8866128975287528
        -8778405442862239
        -2736111468807040
    - stdin: '42'
    - stdout: |
        -80538738812075974
        80435758145817515
        12602123297335631

[niknetniko]

An alternative is using YAML anchors and merge keys:

- unit: 'Sum of three cubes'
  definitions:
    sum_of_three: &sum_of_three
      oracle: 'custom_check'
      language: 'python'
      name: 'sum_of_three_cubes'
      file: 'oracle.py'
  testcases:
    - stdin: '3'
      stdout:
        <<: *sum_of_three
        data: |
          1
          1
          1
    - stdin: '33'
      stdout: |
        8866128975287528
        -8778405442862239
        -2736111468807040
    - stdin: '42'
    - stdout:
        <<: *sum_of_three
        data: |
          -80538738812075974
          80435758145817515
          12602123297335631

I think this has some advantages:

• It is very explicit that the oracle config comes from something else.
• It is opt-in for all cases (no implicit inheritance)
• It is very flexible

As a disadvantage:

• It is more verbose, as you need to specify it everywhere.

This requires minimal changes to our JSON Schema, and no changes to our implementation, as this happens in the YAML parser.

[niknetniko]

If we should decide to use this (I am in favour), I would propose to also remove the existing configuration inheritance mechanism and always use YAML merge tags.

Another disavantage is that YAML merge tags are technically not part of YAML 1.2 or later, only 1.1 (but our Python YAML parser only supports 1.1, so I'm not sure it is a big disadvantage: this construct is used in the wild, so it won't disappear quickly).