list of tracks with test generators

[petertseng]

Well, since the repo of https://github.com/exercism/discussions/issues/155 got closed, it's not possible to edit the issue text there anymore. It's not clear that there's a better home for this issue. I don't want to put it in https://github.com/orgs/exercism/teams/track-maintainers because then people not in the org can't see it, and I don't think this information should be withheld from people outside the org.

---

These are anything that use the canonical-data.json file from problem-specification and generate a test suite to be delivered to students of a given track.

If your track has these, I would be interested to hear about it.

I hope this can help tracks that don't have generators evaluate whether to have them, and allow tracks that already have generators to learn from each other.

Questions I would like to ask:

• How much additional code must you write to generate tests for each new exercise?
  • On one extreme, zero additional code is needed: A single generator can generate code for every single exercise.
  • On the other extreme, maximal additional code is needed: No code at all is shared between generators of any two exercises.
  • Where on this spectrum is your track currently?
  • Where on this spectrum would you like your track to be, ideally?
• Although all the inputs to an exercise are guaranteed to be under the input key, for exercises with more than one input, it's not certain what order they should go in. How do you deal with this?
  • Require a small bit of configuration per exercise that has multiple inputs, saying what order the inputs are in.
  • Accept the order that the inputs are defined in the textual representation of the JSON file (keeping in mind that https://www.json.org/ specifies they are unordered, but hoping that when they were written down in the JSON file, they have a consistent order?)
  • Your idea here
• Statically typed languages: How do you deal with the fact that you cannot determine what types the keys/values of a test case will have until you read the value at the property key? (In some JSON parsers in statically-typed languages, you must declare the types of all key/value pairs in a JSON object before you can parse it, but sometimes the key/value pairs present depend on property)
  • C# and Scala: Parse into a map/dictionary of string -> any type.
  • Go: Delay full parsing until property is read, then parse into different structs depending on what property is (https://github.com/exercism/go/pull/677).
• Are there any possible changes to the canonical JSON schema that would make generation easier?

This issue will be closed immediately, because there's no need to have it take up space in the list of open issues (there's no call to action). Of course, even after it is closed, please feel free to comment with any additional answers.

If as a result there are any proposed changes to the schema, an appropriate issue can be created for that.

To give us a head start, here is what I know of some languages' generators. Please forgive me for being greedy and filling in information for tracks that I am unfamiliar with. Please correct these or add any additional tracks I missed. In alphabetical order:

Bash

• https://github.com/guygastineau/bash-track-test-generator

C

• https://github.com/exercism/csharp/tree/master/generators
• per-exercise data at https://github.com/exercism/csharp/tree/master/generators/Exercises
  • simple example: leap https://github.com/exercism/csharp/blob/master/generators/Exercises/Generators/Leap.cs
  • allergies deals with multiple property: https://github.com/exercism/csharp/blob/master/generators/Exercises/Generators/Allergies.cs
• https://github.com/exercism/csharp/blob/master/generators/Input/CanonicalDataCase.cs#L7 is IDictionary<string, dynamic>, supporting values of any type.

CFML

• https://github.com/exercism/cfml/blob/master/tasks/GenerateTests.cfc ; see generateInput for handling of input.

Common Lisp

• https://github.com/exercism/common-lisp/blob/master/src/generate-tests.lisp

Dart

https://github.com/exercism/dart/blob/master/bin/create_exercise.dart

Erlang

https://github.com/exercism/erlang/tree/master/testgen - see https://github.com/exercism/erlang/tree/master/testgen/src for per-exercise configuration.

Factor

• https://github.com/catb0t/exercism.factor/blob/master/exercism/autogen-exercises/autogen-exercises.factor

Go

• An .meta/gen.go in each exercise directory defines the structure that the file is expected to have.
  • https://github.com/exercism/go/blob/master/exercises/leap/.meta/gen.go is a simple example for tests that have one property.
  • https://github.com/exercism/go/blob/master/exercises/clock/.meta/gen.go and https://github.com/exercism/go/blob/master/exercises/custom-set/.meta/gen.go examples for tests that have more than one property.
  • Parses property first in order to know how to parse the other key/value pairs.
• Common code at https://github.com/exercism/xgo/blob/master/gen/gen.go

JavaScript

• As I understand it, the deprecated track had a single generator: https://github.com/exercism/DEPRECATED.javascript/blob/master/exercises/custom-set/example-gen.js
• I am not aware of a generator in the new track (formerly the ecmascript track)

OCaml

• Common code in https://github.com/exercism/ocaml/tree/master/tools/test-generator/src
• a little additional for each exercise in https://github.com/exercism/ocaml/blob/master/tools/test-generator/src/ocaml_special_cases.ml
• templates for each exercise in https://github.com/exercism/ocaml/tree/master/tools/test-generator/templates/ocaml
  • leap is simple, only one property: https://github.com/exercism/ocaml/blob/master/tools/test-generator/templates/ocaml/leap/test.ml
  • run-length-encoding with multiple properties: https://github.com/exercism/ocaml/blob/master/tools/test-generator/templates/ocaml/run-length-encoding/test.ml

Perl 6

• Template in https://github.com/exercism/perl6/blob/master/templates/test.mustache
• Explanations in https://github.com/exercism/perl6/tree/master/bin
• YAML file for each exercise, then JSON file from x-common simply gets embedded into the test file
  • leap: https://github.com/exercism/perl6/blob/master/exercises/leap/.meta/exercise-data.yaml and https://github.com/exercism/perl6/blob/master/exercises/leap/leap.t
  • clock (multiple property): https://github.com/exercism/perl6/blob/master/exercises/clock/.meta/exercise-data.yaml and https://github.com/exercism/perl6/blob/master/exercises/clock/clock.t

Pharo (Smalltalk)

• Iterates over the (pre-checked out, problem-specifications) - finds the canonical-data.json, and creates a TestCase with methods named for each of the child "cases"
  • It mostly generates trouble free tests with little input, tests look mostly like this
  • test name is based on a concatenation of the nested "description" to avoid duplicate descriptions. The tests are numbered to give defined order (as per exercism requirement). This results in some very long test name identifiers. The work around would be to slim down descriptions in canonical data, to the detriment of others' use of them. A separate "identifier" field would be useful to separate concerns.
  • model to test is the exercise name with suffix "Calculator" (probably could infer this better - but works and makes sense most of the time )
  • Pharo is keyword syntax (like ObjectiveC) so the model method to call is "property" and subsequent "input" names as keyword parameters (this works mostly - some problems don't use great names)
  • the test assertion assumes "assert:equals:" against the expected dictionary, unless the expectation is a Dictionary with a single "error" value, then we generate an exception assertion
  • currently doesn't deal well with identity equality tests like for Clock (this was manually adjusted), and multi valued expectations (like Allergies - but I PR'd an updated spec, which looks to have gone through)
  • Regeneration is still on the todo list - for now, we periodically delete, regenerate and then diff - but should check the version number and do a partial generation
  • Meta information for the exercises (difficulty, where they should occur etc) is stored as a class method and the config.json is generated
  • Exercises (when loaded into the environment - Pharo has an IDE written in itself), store their submission id in a class instance variable, so we can post(patch) the submission back to exercism
  • When posting the solution, we add a testResults file so we can see the state of the solution
  • We have hooks to run a formatter and do extra lint checking, but this isn't enabled yet
  • code is viewable here - with main method at "#generateExerciseFrom: aFileSystemReference) (but its better to see in a smalltalk environment as per the project readme
  • would be handy to have some extra data flags to help steer generation: is it an exception test, is it an identity/equality test

Python

• CLI Entry Point: https://github.com/exercism/python/blob/master/bin/generate_tests.py
• The CLI program looks for an [exercise]/.meta/template.j2 file, and if one exists tests will be generated. Example: https://github.com/exercism/python/tree/master/exercises/word-count/.meta/template.j2
• If an [exercise]/.meta/additional_tests.json file exists the tests described in it will be added to those in the canonical-data.json. Example: https://github.com/exercism/python/tree/master/exercises/word-count/.meta/additional_tests.json
• The CLI will use the above to generate an [exercise]/[exercise]_test.py artifact, which is what's actually distributed to the students. Example: https://github.com/exercism/python/tree/master/exercises/word-count/acronym_test.py

Ruby

• A small script in the bin directory: https://github.com/exercism/ruby/blob/master/bin/generate
• A library in lib/generator providing common-case parsing: https://github.com/exercism/ruby/tree/master/lib/generator
• A .meta/generator in each exercise directory containing a *_case.rb file. If necessary a custom test_template.erb can be provided. Most tests use a common default template
  • Simple example with a custom template: https://github.com/exercism/ruby/blob/master/exercises/leap/.meta/generator/leap_case.rb and https://github.com/exercism/ruby/blob/master/exercises/leap/.meta/generator/test_template.erb
  • Example with more in generator: https://github.com/exercism/ruby/blob/master/exercises/bowling/.meta/generator/bowling_case.rb
  • But note that it does not need a custom template: https://github.com/exercism/ruby/tree/master/exercises/bowling/.meta/generator/test_template.erb is a 404. Most exercises do not need a custom template.

Rust

• https://github.com/exercism/rust/blob/master/util/exercise/src/cmd/generate.rs

Scala

• Common code in https://github.com/exercism/scala/tree/master/testgen/src/main/scala/testgen
• https://github.com/exercism/scala/blob/master/testgen/src/main/scala/testgen/CanonicalDataParser.scala#L12 allows any type in the parse result.
• Per-exercise portion in https://github.com/exercism/scala/tree/master/testgen/src/main/scala
  • Simple example: https://github.com/exercism/scala/blob/master/testgen/src/main/scala/BookStoreTestGenerator.scala
  • https://github.com/exercism/scala/blob/master/testgen/src/main/scala/AllergiesTestGenerator.scala deals with multiple properties. The list of inputs is defined per-property.

Vimscript

• https://github.com/exercism/vimscript/blob/master/lib/generate.vim

[NobbZ]

I was pushed to this list today and wanted to add erlang to the list.

How much additional code must you write to generate tests for each new exercise?

In the good case, there is not much to add, mostly mapping the named arguments from the JSON to an idiomatic order of positional arguments in the function call and spraying syntax generating functions around.

This is at least for the simpler exercises which do not require state keeping.

Of course we have to generate additional boilerplating code, if the test cases actually involve a sequence of calls of which we need to prepare state before and compare state after that.

Also sometimes when the test deals with complex data, we need to map that to more idiomatic ways of data keeping. Erlang doesn't have objects, and the use of maps is not idiomatic for most of those types, such that we have to translate them into something that is called a "record" in erlang speak, basically a tuple with syntactic sugar such that we can use some syntax to refer to fields by names.

Although all the inputs to an exercise are guaranteed to be under the input key, for exercises with more than one input, it's not certain what order they should go in. How do you deal with this?

We manually map them into an idiomatic positional argument schema, as outlined above.

Are there any possible changes to the canonical JSON schema that would make generation easier?

I have not yet seen those.

[Stargator]

Following up on @NobbZ's comment, Dart also has a test generator. It's in the bin/create_exercise.dart file.

How much additional code must you write to generate tests for each new exercise?

The test generator does not meet all of our needs. So sometimes it may not accurately define what the expected value's type should be. In those cases it just sees an empty list and doesn't know if that's suppose to be a List of objects, strings, or anything. But that's just changing the type in a handful of cases.

Although all the inputs to an exercise are guaranteed to be under the input key, for exercises with more than one input, it's not certain what order they should go in. How do you deal with this?

We generally take the first input and make it the first parameter.

Are there any possible changes to the canonical JSON schema that would make generation easier?

Unsure, as mentioned above specifying the type when the value is an empty collection (list, map, set, etc) could be useful, but types differ from one language to the other and I think that could make it more confusing for the maintainers of the test generators.

[petertseng]

Need to add https://github.com/exercism/python/blob/master/bin/generate_tests.py to this list

The discussions repo got reopened so now there are two editable versions of this issue, but I'll decide that this one will be the canonical one going forward since I already spent time on it after discussions repo was closed.

[yawpitch]

Need to add https://github.com/exercism/python/blob/master/bin/generate_tests.py to this list

The discussions repo got reopened so now there are two editable versions of this issue, but I'll decide that this one will be the canonical one going forward since I already spent time on it after discussions repo was closed.

Added the Python generator + details.