chainguard-dev / malcontent

#supply #chain #attack #detection
Apache License 2.0
446 stars 31 forks source link

`make test` no longer runs sample tests #505

Closed tstromberg closed 1 month ago

tstromberg commented 1 month ago

As much as I'd love to believe all tests are passing in less than 2 seconds, I know it isn't true:

$ make test

cp -a test_data/. out/samples-75ee4b4f3e5c29484811e77ffa9d5d02eb75218f
go test github.com/chainguard-dev/malcontent/cmd/mal github.com/chainguard-dev/malcontent/pkg/action github.com/chainguard-dev/malcontent/pkg/compile github.com/chainguard-dev/malcontent/pkg/malcontent github.com/chainguard-dev/malcontent/pkg/profile github.com/chainguard-dev/malcontent/pkg/render github.com/chainguard-dev/malcontent/pkg/report github.com/chainguard-dev/malcontent/pkg/version github.com/chainguard-dev/malcontent/rules github.com/chainguard-dev/malcontent/third_party
?       github.com/chainguard-dev/malcontent/cmd/mal    [no test files]
?       github.com/chainguard-dev/malcontent/pkg/compile        [no test files]
?       github.com/chainguard-dev/malcontent/pkg/malcontent     [no test files]
?       github.com/chainguard-dev/malcontent/pkg/render [no test files]
?       github.com/chainguard-dev/malcontent/pkg/version        [no test files]
?       github.com/chainguard-dev/malcontent/rules      [no test files]
?       github.com/chainguard-dev/malcontent/third_party        [no test files]
ok      github.com/chainguard-dev/malcontent/pkg/action (cached)
ok      github.com/chainguard-dev/malcontent/pkg/profile        (cached)
ok      github.com/chainguard-dev/malcontent/pkg/report (cached)

Notice that samples_test does not exist there.

This means we are more likely to be introducing false-positives.

egibs commented 1 month ago

I'll have a fix for this in my next PR.

go test $(shell go list ./... | grep -Ev "samples|test_data")

should be

go test $(shell go list ./... | grep -Ev "^samples\$|test_data")
egibs commented 1 month ago

That said, the tests seem to be running in test_data even when filtering it out, so the tests can't find the actual samples.