search

kubeshop / testkube

☸️ Kubernetes-native testing framework for test execution and orchestration
https://testkube.io
Other
1.27k stars 130 forks source link

retry a test case in test suites #3316

Open JinoArch opened 1 year ago

JinoArch commented 1 year ago

Is your feature request related to a problem? Please describe. retry a test case in test suite

Describe the solution you'd like Wonder if we have a retry feature for the tests that fails in a test suite? If not can we have that?

Describe alternatives you've considered Re-run it manually😅

JinoArch commented 1 year ago

Giving an example, There are 2 test runs test 0 and test 1. 1 failed and 1 passed. how can I re-run only the failed one?

running test [64018b67de8725a531714644]
🚚 Initializing...
🌍 Reading environment variables...
✅ Environment variables read successfully
RUNNER_ENDPOINT="testkube-minio-service-testkube:9000"
RUNNER_ACCESSKEYID="********"
RUNNER_SECRETACCESSKEY="*******"
RUNNER_LOCATION=""
RUNNER_TOKEN=""
RUNNER_BUCKET="testkube-artifacts"
RUNNER_SSL=false
RUNNER_SCRAPPERENABLED="true"
RUNNER_GITUSERNAME="JinoArch"
RUNNER_GITTOKEN="******"
RUNNER_DATADIR="/data"
📦 Fetching test content from git-dir...
✅ Test content fetched to path /data/repo/
📂 Fetching uploads from object store testkube-minio-service-testkube:9000...
📂 Placing files from buckets into /data/uploads/ []
📂 Getting the contents of bucket folders [test-jinos-terratest-suite]

📂 Setting up access to files in /data
🔬 Executing in directory /data: 
 $ chmod 
✅ Execution succeeded
✅ Access to files enabled
✅ Initialization successful
0xc00149f100
🌍 Preparing initial Ginkgo parameters
✅ Initial Ginkgo parameters prepared: map[GinkgoCompilers: GinkgoCover: GinkgoCoverProfile: GinkgoFailFast: GinkgoFailOnPending: GinkgoFlakeAttempts: GinkgoFocusFilter: GinkgoJsonReport: GinkgoJunitReport:--junit-report report.xml GinkgoKeepGoing:--keep-going GinkgoLabelFilter: GinkgoParallel:-p GinkgoParallelProcs: GinkgoRace: GinkgoRandomize:--randomize-all GinkgoRandomizeSuites:--randomize-suites GinkgoRecursive:-r GinkgoRepeat: GinkgoSkipFilter: GinkgoSkipPackage: GinkgoTeamCityReport: GinkgoTestPackage: GinkgoTimeout: GinkgoTrace:--trace GinkgoUntilItFails:]
🚚 Preparing test runner
🌍 Reading environment variables...
✅ Environment variables read successfully
RUNNER_ENDPOINT="testkube-minio-service-testkube:9000"
RUNNER_ACCESSKEYID="********"
RUNNER_SECRETACCESSKEY="********"
RUNNER_LOCATION=""
RUNNER_TOKEN=""
RUNNER_BUCKET="testkube-artifacts"
RUNNER_SSL=false
RUNNER_SCRAPPERENABLED="true"
RUNNER_GITUSERNAME="rr-jino-jose"
RUNNER_GITTOKEN="********"
RUNNER_DATADIR="/data"
running test [64018b67de8725a531714644]
🚚 Preparing for test run
📦 Fetching test content from git-dir...
✅ Test content fetched to path /tmp/git-checkout375033970/repo/
🌍 Setting Ginkgo parameters from variables
✅ Ginkgo parameters from variables set: map[GinkgoJunitReport:--junit-report report.xml GinkgoKeepGoing:--keep-going GinkgoParallel:-p GinkgoRandomize:--randomize-all GinkgoRandomizeSuites:--randomize-suites GinkgoRecursive:-r GinkgoTrace:--trace]
🌍 Building Ginkgo arguments from params
✅ Ginkgo arguments from params built: [--randomize-all -p --keep-going --trace -r --randomize-suites --junit-report report.xml /data/repo/internal/infra]
🌍 Building Ginkgo flags
✅ Ginkgo flags built: []
🔬 Executing in directory /data/repo: 
 $ ginkgo --randomize-all -p --keep-going --trace -r --randomize-suites --junit-report report.xml /data/repo/internal/infra
go: downloading github.com/sirupsen/logrus v1.9.0
go: downloading github.com/gruntwork-io/terratest v0.41.11
go: downloading github.com/onsi/gomega v1.27.1
go: downloading github.com/onsi/ginkgo/v2 v2.8.1
go: downloading golang.org/x/sys v0.5.0
go: downloading github.com/google/go-cmp v0.5.9
go: downloading github.com/go-logr/logr v1.2.3
go: downloading golang.org/x/net v0.7.0
go: downloading gopkg.in/yaml.v3 v3.0.1
go: downloading golang.org/x/text v0.7.0
go: downloading github.com/hashicorp/hcl/v2 v2.9.1
go: downloading github.com/hashicorp/go-multierror v1.1.1
go: downloading github.com/hashicorp/terraform-json v0.15.0
go: downloading github.com/jinzhu/copier v0.0.0-20190924061706-b57f9002281a
go: downloading github.com/stretchr/testify v1.8.0
go: downloading github.com/tmccombs/hcl2json v0.3.3
go: downloading github.com/zclconf/go-cty v1.12.1
go: downloading github.com/mattn/go-zglob v0.0.2-0.20190814121620-e3c945676326
go: downloading github.com/hashicorp/go-getter v1.6.1
go: downloading golang.org/x/crypto v0.5.0
go: downloading github.com/hashicorp/errwrap v1.0.0
go: downloading github.com/davecgh/go-spew v1.1.1
go: downloading github.com/pmezard/go-difflib v1.0.0
go: downloading github.com/agext/levenshtein v1.2.3
go: downloading github.com/apparentlymart/go-textseg/v13 v13.0.0
go: downloading github.com/mitchellh/go-wordwrap v1.0.1
go: downloading github.com/hashicorp/go-version v1.6.0
go: downloading cloud.google.com/go/storage v1.10.0
go: downloading github.com/aws/aws-sdk-go v1.40.56
go: downloading cloud.google.com/go v0.83.0
go: downloading github.com/bgentry/go-netrc v0.0.0-20140422174119-9fd32a8b3d3d
go: downloading github.com/hashicorp/go-cleanhttp v0.5.2
go: downloading github.com/hashicorp/go-safetemp v1.0.0
go: downloading github.com/klauspost/compress v1.13.0
go: downloading github.com/mitchellh/go-homedir v1.1.0
go: downloading github.com/mitchellh/go-testing-interface v1.0.0
go: downloading github.com/ulikunitz/xz v0.5.8
go: downloading golang.org/x/oauth2 v0.0.0-20220223155221-ee480838109b
go: downloading google.golang.org/api v0.47.0
go: downloading github.com/googleapis/gax-go/v2 v2.0.5
go: downloading google.golang.org/genproto v0.0.0-20210602131652-f16073e35f0c
go: downloading google.golang.org/grpc v1.38.0
go: downloading go.opencensus.io v0.23.0
go: downloading github.com/golang/snappy v0.0.3
go: downloading github.com/jmespath/go-jmespath v0.4.0
go: downloading github.com/golang/groupcache v0.0.0-20210331224755-41bb18bfe9da
go: downloading github.com/golang/protobuf v1.5.2
go: downloading google.golang.org/protobuf v1.28.1

Running Suite: K8s Test Suite - /data/repo/internal/infra
=========================================================
Random Seed: 1677822830 - will randomize all specs

Will run 2 of 2 specs
Running in parallel across 4 processes

------------------------------

• [FAILED] [0.128 seconds]
Infra
/data/repo/internal/infra/infra_test.go:18
  Infra Test
  /data/repo/internal/infra/infra_test.go:21
    [It] Return pass or fail
    /data/repo/internal/infra/infra_test.go:22

  Begin Captured StdOut/StdErr Output >>
    time="2023-03-03T05:54:22Z" level=info msg="Starting test 0"
    Infra Infra Test Return pass or fail 2023-03-03T05:54:22Z retry.go:91: terraform [init -upgrade=false]
    Infra Infra Test Return pass or fail 2023-03-03T05:54:22Z logger.go:66: Running command terraform with args [init -upgrade=false]
    Infra Infra Test Return pass or fail 2023-03-03T05:54:22Z logger.go:66: 
    Infra Infra Test Return pass or fail 2023-03-03T05:54:22Z logger.go:66: Initializing the backend...
    Infra Infra Test Return pass or fail 2023-03-03T05:54:22Z logger.go:66: 
    Infra Infra Test Return pass or fail 2023-03-03T05:54:22Z logger.go:66: Initializing provider plugins...
    Infra Infra Test Return pass or fail 2023-03-03T05:54:22Z logger.go:66: 
    Infra Infra Test Return pass or fail 2023-03-03T05:54:22Z logger.go:66: Terraform has been successfully initialized!
    Infra Infra Test Return pass or fail 2023-03-03T05:54:22Z logger.go:66: 
    Infra Infra Test Return pass or fail 2023-03-03T05:54:22Z logger.go:66: You may now begin working with Terraform. Try running "terraform plan" to see
    Infra Infra Test Return pass or fail 2023-03-03T05:54:22Z logger.go:66: any changes that are required for your infrastructure. All Terraform commands
    Infra Infra Test Return pass or fail 2023-03-03T05:54:22Z logger.go:66: should now work.
    Infra Infra Test Return pass or fail 2023-03-03T05:54:22Z logger.go:66: 
    Infra Infra Test Return pass or fail 2023-03-03T05:54:22Z logger.go:66: If you ever set or change modules or backend configuration for Terraform,
    Infra Infra Test Return pass or fail 2023-03-03T05:54:22Z logger.go:66: rerun this command to reinitialize your working directory. If you forget, other
    Infra Infra Test Return pass or fail 2023-03-03T05:54:22Z logger.go:66: commands will detect it and remind you to do so if necessary.
    Infra Infra Test Return pass or fail 2023-03-03T05:54:22Z retry.go:91: terraform [apply -input=false -auto-approve -lock=false]
    Infra Infra Test Return pass or fail 2023-03-03T05:54:22Z logger.go:66: Running command terraform with args [apply -input=false -auto-approve -lock=false]
    Infra Infra Test Return pass or fail 2023-03-03T05:54:22Z logger.go:66: 
    Infra Infra Test Return pass or fail 2023-03-03T05:54:22Z logger.go:66: No changes. Your infrastructure matches the configuration.
    Infra Infra Test Return pass or fail 2023-03-03T05:54:22Z logger.go:66: 
    Infra Infra Test Return pass or fail 2023-03-03T05:54:22Z logger.go:66: Terraform has compared your real infrastructure against your configuration
    Infra Infra Test Return pass or fail 2023-03-03T05:54:22Z logger.go:66: and found no differences, so no changes are needed.
    Infra Infra Test Return pass or fail 2023-03-03T05:54:22Z logger.go:66: 
    Infra Infra Test Return pass or fail 2023-03-03T05:54:22Z logger.go:66: Apply complete! Resources: 0 added, 0 changed, 0 destroyed.
    Infra Infra Test Return pass or fail 2023-03-03T05:54:22Z logger.go:66: 
    Infra Infra Test Return pass or fail 2023-03-03T05:54:22Z logger.go:66: Outputs:
    Infra Infra Test Return pass or fail 2023-03-03T05:54:22Z logger.go:66: 
    Infra Infra Test Return pass or fail 2023-03-03T05:54:22Z logger.go:66: hello_world = "false"
    Infra Infra Test Return pass or fail 2023-03-03T05:54:22Z retry.go:91: terraform [output -no-color -json hello_world]
    Infra Infra Test Return pass or fail 2023-03-03T05:54:22Z logger.go:66: Running command terraform with args [output -no-color -json hello_world]
    Infra Infra Test Return pass or fail 2023-03-03T05:54:22Z logger.go:66: "false"
    Infra Infra Test Return pass or fail 2023-03-03T05:54:22Z retry.go:91: terraform [destroy -auto-approve -input=false -lock=false]
    Infra Infra Test Return pass or fail 2023-03-03T05:54:22Z logger.go:66: Running command terraform with args [destroy -auto-approve -input=false -lock=false]
    Infra Infra Test Return pass or fail 2023-03-03T05:54:22Z logger.go:66: 
    Infra Infra Test Return pass or fail 2023-03-03T05:54:22Z logger.go:66: Changes to Outputs:
    Infra Infra Test Return pass or fail 2023-03-03T05:54:22Z logger.go:66:   - hello_world = "false" -> null
    Infra Infra Test Return pass or fail 2023-03-03T05:54:22Z logger.go:66: 
    Infra Infra Test Return pass or fail 2023-03-03T05:54:22Z logger.go:66: You can apply this plan to save these new output values to the Terraform
    Infra Infra Test Return pass or fail 2023-03-03T05:54:22Z logger.go:66: state, without changing any real infrastructure.
    Infra Infra Test Return pass or fail 2023-03-03T05:54:22Z logger.go:66: 
    Infra Infra Test Return pass or fail 2023-03-03T05:54:22Z logger.go:66: Destroy complete! Resources: 0 destroyed.
    Infra Infra Test Return pass or fail 2023-03-03T05:54:22Z logger.go:66: 
  << End Captured StdOut/StdErr Output

  Expected
      <string>: false
  to equal
      <string>: true
  In [It] at: /data/repo/internal/infra/infra_test.go:30

  Full Stack Trace
    github.com/rewards-devops/rewatest/internal/infra.glob..func3.1.1()
        /data/repo/internal/infra/infra_test.go:30 +0x2a5
------------------------------

• [0.139 seconds]
Infra Infra Test Return pass or fail
/data/repo/internal/infra/infra_test_1.go:14

  Begin Captured StdOut/StdErr Output >>

    time="2023-03-03T05:54:22Z" level=info msg="Starting test 1"
    Infra Infra Test Return pass or fail 2023-03-03T05:54:22Z retry.go:91: terraform [init -upgrade=false]
    Infra Infra Test Return pass or fail 2023-03-03T05:54:22Z logger.go:66: Running command terraform with args [init -upgrade=false]
    Infra Infra Test Return pass or fail 2023-03-03T05:54:22Z logger.go:66: 
    Infra Infra Test Return pass or fail 2023-03-03T05:54:22Z logger.go:66: Initializing the backend...
    Infra Infra Test Return pass or fail 2023-03-03T05:54:22Z logger.go:66: 
    Infra Infra Test Return pass or fail 2023-03-03T05:54:22Z logger.go:66: Initializing provider plugins...
    Infra Infra Test Return pass or fail 2023-03-03T05:54:22Z logger.go:66: 
    Infra Infra Test Return pass or fail 2023-03-03T05:54:22Z logger.go:66: Terraform has been successfully initialized!
    Infra Infra Test Return pass or fail 2023-03-03T05:54:22Z logger.go:66: 
    Infra Infra Test Return pass or fail 2023-03-03T05:54:22Z logger.go:66: You may now begin working with Terraform. Try running "terraform plan" to see
    Infra Infra Test Return pass or fail 2023-03-03T05:54:22Z logger.go:66: any changes that are required for your infrastructure. All Terraform commands
    Infra Infra Test Return pass or fail 2023-03-03T05:54:22Z logger.go:66: should now work.
    Infra Infra Test Return pass or fail 2023-03-03T05:54:22Z logger.go:66: 
    Infra Infra Test Return pass or fail 2023-03-03T05:54:22Z logger.go:66: If you ever set or change modules or backend configuration for Terraform,
    Infra Infra Test Return pass or fail 2023-03-03T05:54:22Z logger.go:66: rerun this command to reinitialize your working directory. If you forget, other
    Infra Infra Test Return pass or fail 2023-03-03T05:54:22Z logger.go:66: commands will detect it and remind you to do so if necessary.
    Infra Infra Test Return pass or fail 2023-03-03T05:54:22Z retry.go:91: terraform [apply -input=false -auto-approve -lock=false]
    Infra Infra Test Return pass or fail 2023-03-03T05:54:22Z logger.go:66: Running command terraform with args [apply -input=false -auto-approve -lock=false]
    Infra Infra Test Return pass or fail 2023-03-03T05:54:22Z logger.go:66: 
    Infra Infra Test Return pass or fail 2023-03-03T05:54:22Z logger.go:66: Changes to Outputs:
    Infra Infra Test Return pass or fail 2023-03-03T05:54:22Z logger.go:66:   + hello_world = "false"
    Infra Infra Test Return pass or fail 2023-03-03T05:54:22Z logger.go:66: 
    Infra Infra Test Return pass or fail 2023-03-03T05:54:22Z logger.go:66: You can apply this plan to save these new output values to the Terraform
    Infra Infra Test Return pass or fail 2023-03-03T05:54:22Z logger.go:66: state, without changing any real infrastructure.
    Infra Infra Test Return pass or fail 2023-03-03T05:54:22Z logger.go:66: 
    Infra Infra Test Return pass or fail 2023-03-03T05:54:22Z logger.go:66: Apply complete! Resources: 0 added, 0 changed, 0 destroyed.
    Infra Infra Test Return pass or fail 2023-03-03T05:54:22Z logger.go:66: 
    Infra Infra Test Return pass or fail 2023-03-03T05:54:22Z logger.go:66: Outputs:
    Infra Infra Test Return pass or fail 2023-03-03T05:54:22Z logger.go:66: 
    Infra Infra Test Return pass or fail 2023-03-03T05:54:22Z logger.go:66: hello_world = "false"
    Infra Infra Test Return pass or fail 2023-03-03T05:54:22Z retry.go:91: terraform [output -no-color -json hello_world]
    Infra Infra Test Return pass or fail 2023-03-03T05:54:22Z logger.go:66: Running command terraform with args [output -no-color -json hello_world]
    Infra Infra Test Return pass or fail 2023-03-03T05:54:22Z logger.go:66: "false"
    Infra Infra Test Return pass or fail 2023-03-03T05:54:22Z retry.go:91: terraform [destroy -auto-approve -input=false -lock=false]
    Infra Infra Test Return pass or fail 2023-03-03T05:54:22Z logger.go:66: Running command terraform with args [destroy -auto-approve -input=false -lock=false]
    Infra Infra Test Return pass or fail 2023-03-03T05:54:22Z logger.go:66: 
    Infra Infra Test Return pass or fail 2023-03-03T05:54:22Z logger.go:66: Changes to Outputs:
    Infra Infra Test Return pass or fail 2023-03-03T05:54:22Z logger.go:66:   - hello_world = "false" -> null
    Infra Infra Test Return pass or fail 2023-03-03T05:54:22Z logger.go:66: 
    Infra Infra Test Return pass or fail 2023-03-03T05:54:22Z logger.go:66: You can apply this plan to save these new output values to the Terraform
    Infra Infra Test Return pass or fail 2023-03-03T05:54:22Z logger.go:66: state, without changing any real infrastructure.
    Infra Infra Test Return pass or fail 2023-03-03T05:54:22Z logger.go:66: 
    Infra Infra Test Return pass or fail 2023-03-03T05:54:22Z logger.go:66: Destroy complete! Resources: 0 destroyed.
    Infra Infra Test Return pass or fail 2023-03-03T05:54:22Z logger.go:66: 
  << End Captured StdOut/StdErr Output
------------------------------

•

Summarizing 1 Failure:

  [FAIL] Infra Infra Test [It] Return pass or fail
  /data/repo/internal/infra/infra_test.go:30

Ran 2 of 2 Specs in 0.142 seconds
FAIL! -- 1 Passed | 
1 Failed | 
0 Pending | 
0 Skipped

You're using deprecated Ginkgo functionality:
=============================================
  --ginkgo.slow-spec-threshold is deprecated --slow-spec-threshold has been deprecated and will be removed in a future version of Ginkgo.  This feature has proved to be more noisy than useful.  You can use --poll-progress-after, instead, to get more actionable feedback about potentially slow specs and understand where they might be getting stuck.

To silence deprecations that can be silenced set the following environment variable:
  ACK_GINKGO_DEPRECATIONS=2.8.1

PASS

You're using deprecated Ginkgo functionality:
=============================================
  --ginkgo.slow-spec-threshold is deprecated --slow-spec-threshold has been deprecated and will be removed in a future version of Ginkgo.  This feature has proved to be more noisy than useful.  You can use --poll-progress-after, instead, to get more actionable feedback about potentially slow specs and understand where they might be getting stuck.

To silence deprecations that can be silenced set the following environment variable:
  ACK_GINKGO_DEPRECATIONS=2.8.1

--- FAIL: TestGinkgo (0.13s)
FAIL

You're using deprecated Ginkgo functionality:
=============================================
  --ginkgo.slow-spec-threshold is deprecated --slow-spec-threshold has been deprecated and will be removed in a future version of Ginkgo.  This feature has proved to be more noisy than useful.  You can use --poll-progress-after, instead, to get more actionable feedback about potentially slow specs and understand where they might be getting stuck.

To silence deprecations that can be silenced set the following environment variable:
  ACK_GINKGO_DEPRECATIONS=2.8.1

PASS

You're using deprecated Ginkgo functionality:
=============================================
  --ginkgo.slow-spec-threshold is deprecated --slow-spec-threshold has been deprecated and will be removed in a future version of Ginkgo.  This feature has proved to be more noisy than useful.  You can use --poll-progress-after, instead, to get more actionable feedback about potentially slow specs and understand where they might be getting stuck.

To silence deprecations that can be silenced set the following environment variable:
  ACK_GINKGO_DEPRECATIONS=2.8.1

PASS

Ginkgo ran 1 suite in 32.149440685s

Test Suite Failed

❌ Execution failed: process error: exit status 1
✅ Mapped Junit to Execution Results...
🗄️ Scraping artifacts [/data/repo/internal/infra/reports]
✅ Successfully scraped artifacts
process error: exit status 1
TheBrunoLopes commented 1 year ago

Hello @JinoArch! Would you like to see a button near the test case when you are looking at a test suite that allows it to re-run just that test case ? If so, do you want to change the status (pass/fail) of the test suite after ? Or/and would like to be able to define how many retries should be done in case of one test case fails ?

JinoArch commented 1 year ago

hello @TheBrunoLopes

Would you like to see a button near the test case when you are looking at a test suite that allows it to re-run just that test case ?

YES

If so, do you want to change the status (pass/fail) of the test suite after ? I would prefer status at last which says in Nth test it passed or similar

Or/and would like to be able to define how many retries should be done in case of one test case fails ? This should be controlled by user

aabedraba commented 1 year ago

In the new TestSuite format that coming in 1.13 we will be able to have parallelized tests in each steps.

This means that it's better to make the "re run" work on step-level rather than test level.

So each "execute" should be retriggered. 

apiVersion: tests.testkube.io/v3
kind: TestSuite
spec:
  description: Example simple tests orchestration
  steps:
    - execute:
        # test1 and test2 will be executed in parallel, 
        # but it will run a minimum of 50ms
        - test: test1
        - test: test2
        - delay: 1h
    - execute:
        - delay: 1m
    - execute:
        - testSuite: test-suite-1 # this will be supported in next iteration
        - test: test3
    - execute:
        - test: test4
windowsrefund commented 6 months ago

In the new TestSuite format that coming in 1.13 we will be able to have parallelized tests in each steps.

This means that it's better to make the "re run" work on step-level rather than test level.

So each "execute" should be retriggered.

apiVersion: tests.testkube.io/v3
kind: TestSuite
spec:
  description: Example simple tests orchestration
  steps:
    - execute:
        # test1 and test2 will be executed in parallel, 
        # but it will run a minimum of 50ms
        - test: test1
        - test: test2
        - delay: 1h
    - execute:
        - delay: 1m
    - execute:
        - testSuite: test-suite-1 # this will be supported in next iteration
        - test: test3
    - execute:
        - test: test4

This make no sense to me as it's not clear what the testSuite key/value is referring to. Would it not be more clear to simply add 2 components to each step allowing the user to specify the number of times to retry and wait period between each attempt? Something like retries: 3 and retry_wait: 30s

vsukhin commented 6 months ago

this is for @TheBrunoLopes to recheck

TheBrunoLopes commented 5 months ago

retries: 3 and retry_wait: 30s sounds good to me. @rangoo94 also had an idea of having something like "retry_until_success" would this be something useful ?

windowsrefund commented 5 months ago

I don't think a retry_until_success is a good idea as that's just a fancy way of allowing for an endless loop. Fundamentally, a thing should always pass or fail.