Test tool wishlist

[rdbruijne]

Patience

Please wait while we're testing your Patience...

Basics

• [ ] Decent config method for tests
• [ ] Main test tool to run test
• [ ] Output tool to sanitize output
• [ ] Success/fail is determined as a post-step
• [ ] Test & output tool can be a single .exe

Test attributes

• [ ] Must run as only processs?
• [ ] Test level
• [ ] Test arguments
• [ ] Retry count override
• [ ] Target platform tags

Main test tool

• [ ] Run specific/failed tests
• [ ] Able to run in gitlab, and locally
• [ ] Multiple tests in a process
• [ ] Ouput
  • [ ] Ability to attach arbitrary data to the output (pixel buffer, ...)
  • [ ] Structured output file (use output tool to view)
  • [ ] Terminal
  • [ ] Output basics in termial by default ([testIx / totalTest] path)
  • [ ] Terminal output should optionally be able to contain all info from output file
• [ ] Override config arguments via command line
• [ ] Default retry count

Output tool

C# tool separate from main tool, we like simple code

• [ ] Report per test, plus combined report
  • [ ] Combined report with options (eg. exclude succssful test)
• [ ] Convert output from test tool to mutiple outputs
• [ ] Things to be able to do: file links, commit links
• [ ] Output variants
  • [ ] Terminal glow -> render markdown in terminal
  • [ ] Markdown
  • [ ] HTML
• [ ] Reference files with accepted outputs
  • [ ] Patch several date we don't care about being different
  • [ ] Images
  • [ ] Text
  • [ ] Number ranges
  • [ ] ...
• [ ] Trendline: compare a lot of tests (line a month) to find flaky tests, etc...

Note: Rick will check qtest for useful options

[roenyroeny]

• [ ] Play this song during tests https://youtu.be/kWVFEVWJMz8