cmd/cue: vet doesn't show multiple errors across multiple non-CUE files

jpluscplusm commented 1 year ago

What version of CUE are you using (`cue version`)?

$ cue version
cue version v0.6.0-alpha.2

go version go1.20.5
      -buildmode exe
       -compiler gc
       -trimpath true
     CGO_ENABLED 0
          GOARCH amd64
            GOOS linux
         GOAMD64 v1

Does this issue reproduce with the latest stable release?

Yes - 0.5.0 and 0.4.3

What did you do?

! exec cue vet --all-errors policy.cue dog1.yml dog2.yml
stderr dog1\.yml
stderr dog2\.yml
-- policy.cue --
species:   "dog"
adorable?: true
-- dog1.yml --
name: dog1
species: "dog"
adorable: false
-- dog2.yml --
name: dog2
species: "cat"
adorable: true

I'm vetting 2 YAML files against a CUE file containing 2 policy assertions.
Each YAML file violates a different policy.

I ran cue vet policy.cue file1.yml file2.yml, without --all-errors. I saw only a single file's error mentioned in the output.

I checked cue vet --help, which seems to contain 2 relevant portions:

In this [non-CUE files vetting] mode, each file will be verified against a CUE constraint.

This doesn't seem to be 100% accurate - only 1 file's errors are being reported.

It also documents this option:

...
  -E, --all-errors   print all available errors
...

I added this option, but cue vet's output remained unchanged.

What did you expect to see?

I expected the testscript repro to pass.

I expected the output from cue vet to show me each file with which there's a problem (even if it doesn't show every error that's present).

What did you see instead?

Only the first error encountered is displayed, even when I added in the --all-errors flag:

> ! exec cue vet --all-errors policy.cue dog1.yml dog2.yml
[stderr]
adorable: conflicting values true and false:
    ./dog1.yml:3:12
    ./policy.cue:2:12
[exit status 1]
> stderr dog1\.yml
> stderr dog2\.yml
FAIL: /tmp/testscript3983475236/multiple-errors.txtar/script.txtar:3: no match for `dog2\.yml` found in stderr
error running multiple-errors.txtar in /tmp/testscript3983475236/multiple-errors.txtar

jpluscplusm commented 1 year ago

I encountered this when using required fields (for the species field), but have removed them for the repro, above, as their use stopped me testing for a regression, i.e. on pre-0.6.0-alpha.x.

myitcv commented 1 year ago

Interestingly, the following does show multiple incomplete errors across the multiple files, hence the following passes:

! exec cue vet schema.cue name.yml age.yml
stderr name
stderr age
stderr address

-- schema.cue --
name: string
age:  int
address: string
-- age.yml --
age: 99
-- name.yml --
name: "cueckoo"

However, there is no mention of the .ymlfilenames involved:

> ! exec cue vet schema.cue name.yml age.yml
[stderr]
address: incomplete value string:
    ./schema.cue:3:10
name: incomplete value string:
    ./schema.cue:1:7
address: incomplete value string:
    ./schema.cue:3:10
age: incomplete value int:
    ./schema.cue:2:7
[exit status 1]

It's also unclear to me what the --all-errors could/should be doing here, because notice that there is more than one error per file reported, and I have not supplied the --all-errors flag.

So it strikes me there are a few possible bugs here not least:

--all-errors doing nothing, or some types of errors not respecting whether --all-errors is set or not (we could step back from this and ask whether the flag still serves a purpose). There is at least one bug here.
incomplete errors do appear to be reported for each file involved, whereas the original description from @jpluscplusm seems to show that other types or errors are not. The latter is a bug according to the docs of cue vet, per this issue's original description.

myitcv commented 1 year ago

Note that I have also just raised https://github.com/cue-lang/cue/issues/2521 because it's not clear to me that cue vet should behave the way it is documented with respect to multiple non-CUE files.

jpluscplusm commented 1 year ago

I've also noticed this very similar situation, where the error message only mentions type/value constraints, and not missing required fields:

! exec cue vet --all-errors .:example data.yml
stdout f1
stdout f2
stdout f3
-- example.cue --
package example

f1!: _
f2!: string
f3!: int & <10
-- data.yml --
f2: 123
f3: 200

I get this with cue 0.6.0:

> ! exec cue vet --all-errors .:example data.yml
[stderr]
f2: conflicting values 123 and string (mismatched types int and string):
    ./data.yml:1:6
    ./example.cue:4:6
f3: invalid value 200 (out of bound <10):
    ./example.cue:5:12
    ./data.yml:2:6
[exit status 1]
> stdout f1
FAIL: /tmp/testscript2993825173/foo.txtar/script.txtar:2: no match for `f1` found in stdout
error running foo.txtar in /tmp/testscript2993825173/foo.txtar

Happy to file a separate issue if this feels sufficiently different, given that it's intentionally only using a single data file.

jpluscplusm commented 7 months ago

Re-validated under CUE v0.8.0.

cue-lang / cue