Open ee7 opened 3 years ago
I'd vote: it's mutable, except if the mutation would change the total set of errors in an exercise. Using your example, I'd expect a Rust test generator to come up with
enum Error {
ZeroIsRejectedNotANaturalNumber,
ClassificationIsOnlyPossibleForNaturalNumbers,
}
If a PR updates the text on all such errors, we now have
enum Error {
ZeroIsRejectedNotAPositiveInteger,
ClassificationIsOnlyPossibleForPositiveIntegers,
}
The set of errors, and the semantics, hasn't changed; only the enum's label. I think this is fine.
If, on the other hand, we rewrite things to enum Error
had some number of variants other than 2, then that's a breaking change; that should be added as a new test, because there are now new semantics in the exercise.
Make sense?
It may help to compare the implementations of one exercise involving errors across tracks to find out if they'd be breaking or not. I don't know if any tracks check for an exact error message for example.
Okay, I think we have an agreement that the error
property in the expected
property is allowed to be updated without introducing new test cases, but on certain conditions. @coriolinus phrased those conditions well. Would someone be interested in updating the documentation?
JavaScript often checks for an exact message; especially in older exercises. In newer exercises we usually check for class name, or better, provide it via the stub.
Example
In
perfect-numbers
we have this test case: https://github.com/exercism/problem-specifications/blob/8df6cb2ed8e77aa3ac0c02744cd9ddb83920f5f2/exercises/perfect-numbers/canonical-data.json#L121-L130And we notice (see https://github.com/exercism/problem-specifications/pull/1691) that we would like to make this change:
We seem to have decided that
description
is allowed to mutate when the "meaning is unchanged" (that is, we do not add areimplements
test case for e.g. a simpledescription
-only typo fix).The question is: should the contents of an
error
also be considered mutable when the "meaning is unchanged", or should that require a new test case?Discussion
My initial opinion is something like:
description
should be mutable if we don't change the "meaning" of the test.error
is similar to the "description" of the error. In the limiting case, it seems messy to add a new test case just to fix a typo in anerror
value.error
value should similarly be allowed when the "meaning is unchanged".Drawbacks:
tests.toml
didn't change.expected
doesn't change, but make a special case for anerror
-only change.