Create interface.md - Githubissues

iHiD commented 5 years ago

For everyone to tear to pieces :)

iHiD commented 5 years ago

rational-numbers has nearly 95000 [individual tests]

@SaschaMann That's useful to know, thanks. Again, I'd need to see how Julia outputs this to understand what the expectations are here for what a student would see.

NobbZ commented 5 years ago

Many unit test libraries do randomize execution order or might have randomized examples (as in property based testing) it's common that those libraries print a seed for reproduction of the testrun.

The JSON should contain a field to hold the used seed. null in case it's not used/not supported.

coriolinus commented 5 years ago

Does it work for everyone to just omit unused fields? I can see the benefit of a random_seed field at the top level, but it would just be boilerplate for me.

NobbZ commented 5 years ago

Thats why I have proposed null for "unused" or "not supported", this way you can just omit the field.

Also, while taking a shower I realised that Erlang and Elixir doe have some types that aren't represantable in JSON, as integers (unbound), atoms and tuples. Other languages have multireturn like Go, Python or Ruby.

How do we represent those in the "expected" and "actual" fields?

Or shall we just provide a string that represents the values in the languages syntax?

How shall we provide "expected exception, nothing happened" or the other way round? Shall we provide stacktraces on unexpected exceptions, crashes, panics (assuming we can easily access them in the formatter)?

For tests that are provided by the student, by either adding another file or by adding them inline in the module under test (seen this in erlang and rust), is there a way to mark them in the JSON output as "additional" or at least source file and line of definition (assuming its accessible for the test formatter)?

iHiD commented 5 years ago

I think the only points remaining on here are:

@tehsphinx and @SaschaMann's point about test names. @SaschaMann tells me this is looking sorted. @tehsphinx are you happy too?
@coriolinus's point about optional tests. As optional tests aren't part of Exercism's general spec right now, I'm not going to support them. However, it's something we might want to look at as part of the new project thinking about tracks so we can reevaluate then. For now I suggest just not including optional tests in the JSON or setting their status to optional then post-processing them out before exiting.
@nobbz's point about the placing (and wording) around the directory. We will duplicate whatever we do with https://github.com/exercism/automated-analysis/issues/77 and I will update this file to confirm that when that is decided.

Have I missed anything or can I merge this?

iHiD commented 5 years ago

Merging this. Thank you all.

exercism / automated-tests

Create interface.md #14