Implement E2E Behavior Testing

[abhisek]

We need to build a framework for running behavior tests on generated vet binary to ensure we don't end up breaking cli contract. We need to explore if there are any cli testing frameworks. Alternatively we can just use something like RSpec and write custom helpers to wrap vet execution with params and package manifests (fixture files).

Key flows that need to be tested:

• Scan on directory
• Scan with lockfiles
• Scan with lockfiles and lockfile-as
• Query on JSON dump
• Filters
• Filter Suite
• Exception generator
• Exceptions
• Reporting modules

[insaaniManav]

https://github.com/intuit/judo - this exists for cli testing

[insaaniManav]

Here is a non exhaustive survey for CLI testing techniques

E2E testing using js frameworks

https://dev.to/valorsoftware/painless-cli-integration-testing-1bd7 https://medium.com/@zorrodg/integration-tests-on-node-js-cli-part-1-why-and-how-fa5b1ba552fe https://dev.to/florianrappl/how-we-wrote-our-cli-integration-tests-53i3

Testing using go os command

we can just run the cli using the os library in golang and then run assertions on it as go code

Judo

The judo framework - supports cli integration testing e2e and also doesn't require code, tests can be configured using simple yaml

Raw bash

The third and final option I can think of is just writing Raw bash , didnt find any guides anywhere but should work but again requires code

[abhisek]

@insaaniManav What are the dependencies for adopting Judo?

[insaaniManav]

The dependencies for judo are Node and NPM need to be installed it comes as an NPM package @abhisek

[abhisek]

@insaaniManav What do you think of https://pkg.go.dev/github.com/rogpeppe/go-internal/testscript ?

[insaaniManav]

@abhisek looks really good to me and the golang support also seems like a plus point , should we go ahead with this ?

[abhisek]

@insaaniManav Let us go ahead with testscript. The rationale is it does not add any new dependency to our development or CI environment and seems to meet our needs. While judo looks good, but I see there is no release since 2021 and the last commit is a year back. So it seems to me that we should avoid dependency on a new tool, as much as we can, especially if it is unmaintained.

But lets first port 1-2 scenario to testscript first and see if it works well and the developer experience of it. If it doesn't seem to work, we can revisit the tool decision.

We can probably store all the E2E scenarios in test/scenarios/scripts and have a Go test file refer that, something that I see in the original example.

func TestFoo(t *testing.T) {
    testscript.Run(t, testscript.Params{
        Dir: "testdata",
    })
}

[insaaniManav]

Perfect , i completely get your point on the decision let me begin work on this

[insaaniManav]

https://github.com/safedep/vet/pull/267 @abhisek draft PR for the basic structure , can you confirm this structure etc is okay ?