cue-lang / cue

The home of the CUE language! Validate and define text-based and dynamic configuration
https://cuelang.org
Apache License 2.0
5.14k stars 294 forks source link

cmd/vet: Allow negative testing of "all non-CUE files must fail to vet" as easily as the positive test flow #2272

Open jpluscplusm opened 1 year ago

jpluscplusm commented 1 year ago

Is your feature request related to a problem? Please describe.

A very common feature of CUE projects I work on is that we're integrating with some non-CUE-aware system. As such, I almost always end up with a library of fixtures/canned-files that represent "good" and "bad" input or output files - usually .yml or .json.

In order to assert that a particular CUE configuration validates each of the "good" files individually, cue vet has me covered:

$ cue vet <some-cue-package> path/to/library/of/good/files/*.yml
$ echo $?
0
$ # all is good

But testing that each "bad" file is rejected by the same CUE package requires this shell-ism:

$ fail=0
$ for FILE in path/to/library/of/bad/files/*.yml; do
  cue vet <some-cue-package> $FILE && fail=1
done; # do something if $fail>0

... or in a set -e environment:

$ for FILE in path/to/library/of/bad/files/*.yml; do
  ! cue vet <some-cue-package> $FILE
done

This is because cue vet's return code only indicates that at least one non-CUE file failed to unify, therefore it can't be used in this "negative" test scenario to not-validate more than one file at once.

This messy shell logic ends up living /everywhere/ that I use cue vet against some library of non-CUE fixtures.

Describe the solution you'd like

I'd like a flag for cue vet that tells it to modify its return code logic slightly, and only return "failure" if all non-CUE files fail to unify.

Or anything else that lets me use cue vet in the negative scenario as easily as the positive scenario.

Describe alternatives you've considered

I currently use the only viable alternative I've found: scripting around the negative test cases, in a loop, wherever this pattern is needed.

I suppose I /could/ use the CUE scripting layer to achieve something similar, but that both feels like a non-trivial change to make, and also (I think!) it'd be doing /essentially/ the same loop, under the hood.

Lastly, there's an argument to bring these non-CUE files into CUE, and arrange for a pseudo-test-suite of .cue files injecting positive and negative fixtures into the top level unification and testing them with success: ((negative_test_case & test) == _|_). But that feels, again, like a non-trivial change, and misses the point of testing against a canned set of unprocessed input/output files.

Additional context

n/a.

myitcv commented 1 year ago

In this round of issue gardening, we are not really touching feature requests. Instead choosing to prioritise getting the milestones for v0.6.0 and v0.7.0 in good shape. But we will revisit feature requests at a later date. Please shout, however, if you think this should be boosted!