Added basic structure for e2e tests

[insaaniManav]

@abhisek I have also added a sample of what a judo test file would look like , I think it looks cleaner , is easier to manage and is far more documented , would you mind evaluating this ?

[abhisek]

@abhisek I have also added a sample of what a judo test file would look like , I think it looks cleaner , is easier to manage and is far more documented , would you mind evaluating this ?

@insaaniManav The YAML based approach is definitely more clean. We can remove the old experiments with testscript since we are not going with that approach. Also lets make it as simple as possible. It will be great if we can remove $VET_E2E_BINARY and just use vet in the YAML for command to exec.

[insaaniManav]

@abhisek I have updated and added more tests to this , I could not replicate 5 and 6 . 6 seemed like a combination of all of 2 tests and 5 seems like it needs a supplementary shell script to verify the result of the test , judo can only interact with stderr or stdout I think all the CLI test frameworks only interact with them. For tests like 5 where we need to test the artifacts produced maybe we can run a supplementary shell script afterwards to verify all the artifacts.

[abhisek]

@abhisek I have updated and added more tests to this , I could not replicate 5 and 6 . 6 seemed like a combination of all of 2 tests and 5 seems like it needs a supplementary shell script to verify the result of the test , judo can only interact with stderr or stdout I think all the CLI test frameworks only interact with them. For tests like 5 where we need to test the artifacts produced maybe we can run a supplementary shell script afterwards to verify all the artifacts.

@insaaniManav While the YAML based approach looks clean and easy to get started, I think we will end up restricting ourselves in "expressing an E2E Scenario" if we go by it because we are unable to express 5 which is a valid and important scenario for vet's ability to build a dependency graph. I would use the following framework for decision making:

• Benefit: Does judo/YAML significantly simplify writing E2E scenarios over Bash/shell script that are generic and more expressive?
• Risk: Does adopting judo/testscript puts us in risk of wasting time trying to figure out how to express a scenario (e.g. 5) than what can be done with Bash?
• Reward: What are we gaining? Simplicity of E2E tests? Time saving? Better performance? Is it worth compared to risk?

Unless the reward is significantly larger than risk, I think we should put this on hold and continue with Bash for E2E scripts. This may not be pretty but gets the job done for now. It seems to me cli E2E testing is not easy and there is not much matured tooling around this space.

What do you think?

[insaaniManav]

completely agreed the world needs more / a better e2e cli testing tool , we really would be restricting ourselves . its slightly cleaner but still not as powerful as bash will be