search

kudobuilder / kuttl

KUbernetes Test TooL (kuttl)
https://kuttl.dev
Apache License 2.0
688 stars 85 forks source link

[Enhancement] Output test case full path in final report #403

Open chipzoller opened 2 years ago

chipzoller commented 2 years ago

What would you like to be added:

Along with the result of the test (PASS | FAIL), print the full path to the test case if known through a TestSuite file.

Why is this needed:

With hundreds of possible tests being executed across a complex hierarchy of directories, knowing which test failed only by the name of the test case (i.e., the parent folder holding the kuttl test files) isn't often enough. For example, something like this is printed today:

        --- PASS: kuttl/harness/external-metrics (25.87s)
        --- PASS: kuttl/harness/nomutateDigest-verifyDigest-norequired (5.38s)
        --- PASS: kuttl/harness/mutateDigest-noverifyDigest-norequired (3.30s)
        --- PASS: kuttl/harness/keyless-nomutatedigest-noverifydigest-required (3.92s)

In reality, each of those test cases are six directories deep, and they may not all even be in the same parent directory.

This enhancement is to print, in the final report output, the full path to the test cases as known based upon the TestSuite file.

kensipe commented 2 years ago

Questions to work thru:

  1. do you want to the full path always or just on failure?
  2. do you want the full path from root / or drive?
  3. we allow for testsuites and tests to be pulled for url and/or tarball. These are exploded in a system temp folder. These could be not so friendly. Would it be best to show the url vs folder?
chipzoller commented 2 years ago
  1. Always
  2. Full relative path from what is specified in the TestSuite.
  3. Solution to 2 solves for this one; if it's a relative path then once untarred it still makes sense to users.