New E2E Test Framework Discussion

[sheidkamp]

Do you have a suggestion for code improvement or tracking existing technical debt? Please describe.

A space to track comments/thoughts/issues/learnings from onboarding/areas for improvement in the new E2E test framework

Describe the solution you'd like

No response

Additional Context

No response

[sheidkamp]

Like:

• The reusable YAML definitions. Also makes it easy to set up a demo and then just use the resources for the tests.
• The "debug test" button in vscode

Dislike:

• Lack of a stack trace on failures
• Inability to focus tests

Suggestions for future improvements:

• There are patterns developing that could/should get standardized into reusable code
  • The association of manifests and resources to tests
  • Looping over the manifests and expected resources at Test/Suite setup and teardown
  • Inclusion of standard/default resources( like a curl/nginx pod or a petstore deployment) handled by at a high level
    • Will need to be moved to a common place first. I did this in gloo with the curl pod as a first step.
  • An interface that includes what seem to be the common components of the test suites (ctx, testinstallation, testify suite) that can enable some of the above
• Look at the types of yaml resources we are creating and (if appropriate) come up with a standard naming convention and directory structure inside /testdata
• Documentation around things like "what is a feature?"