Test status not updated correctly

[VishwasSomasekhariah]

The final test status is not set to failed when thresholds are breached.

Steps to reproduce

• Set threshold on 'http_req_failed', in my case 'rate < 0.01'.
• Implement an assertion to check the response status and mark any requests other than 200 response code as a failed request.
• During the test execution kill the pod running the application that the load is being generated on.

Expected behavior

• All the redirected HTTP requests should now be marked as failed.
• 'http_req_failed' count should increase.
• If we see a failure rate of more than 1% then mark the test as failed.

Version / Cluster

• Testkube version: 1.12.3
• EKS Cluster
• Kubernetes version: 1.23

Screenshots You can see in the error message that some thresholds have failed but the status is still set to passed.

[vsukhin]

thank you @VishwasSomasekhariah

[vLia]

Thank you @VishwasSomasekhariah for reporting this issue. As I was trying to reproduce it, I created a failing threshold test using the string:

import http from 'k6/http';

export const options = {
  thresholds: {
    http_req_failed: ['rate<0.01'], // http errors should be less than 1%
    http_req_duration: ['p(95)<200'], // 95% of requests should be below 200ms
  },
};

export default function () {
  http.get('https://test-api.k6.io/public/crocodiles/101/');
}

I did indeed get the line marking one test step of the test as passed:

$ kubectl logs -f jobs/649030528e9505f6cba40858 -n testkube

time=\"2023-06-19T10:39:18Z\" level=error msg=\"thresholds on metrics 'http_req_failed' have been breached\"\n","outputType":"text/plain","errorMessage":"some thresholds have failed","steps":[{"name":"default: 1 iterations for each of 1 VUs (maxDuration: 10m0s, gracefulStop: 30s)","duration":"00m00.4s/10m0s","status":"passed"}]},"time":"2023-06-19T10:39:18.228908585Z"}

The status of the Testkube test was still failed:

$ testkube get tests | grep threshold

k6-test-threshold-failing | k6/script      | 2023-06-19 10:33:50 +0000 UTC | executor=k6-executor,          |          | failed | 649030528e9505f6cba40858

Could you please double-check if the status of the test shown with testkube get tests is passed?

[VishwasSomasekhariah]

@vLia, Thank you for getting back to me. What do you mean by the "test step of the test as passed"? even that should be marked as failed right.

[avinashkna4]

@vLia, that's bit misleading right ? We are totally relying on the webhook response for checking the status and it says "status":"passed".

Now one have to call test results API after webhook is received which shouldn't be the case right ?

Can we get the final test result as well in the webhook so that we don't have to call the result API again ?

Sample webhook response attached here for ref.

[vLia]

Hello @VishwasSomasekhariah and @avinashkna4, Thank you for looking into this more. Could you please share the whole response you are getting? This "status":"passed" is part of the Steps list. The whole structure is:

// ExecutionResult returned from executor
type ExecutionResult struct {
    Status *ExecutionStatus `json:"status"`
    // RAW Test execution output, depends of reporter used in particular tool
    Output string `json:"output,omitempty"`
    // output type depends of reporter used in particular tool
    OutputType string `json:"outputType,omitempty"`
    // error message when status is error, separate to output as output can be partial in case of error
    ErrorMessage string `json:"errorMessage,omitempty"`
    // execution steps (for collection of requests)
    Steps   []ExecutionStepResult   `json:"steps,omitempty"`
    Reports *ExecutionResultReports `json:"reports,omitempty"`
}

From here, in order to evaluate the test result, you should look into the Status field of the ExecutionResult instead of the individual steps in the list. While I do agree that the status in the Steps is misleading, unfortunately, the current implementation would require a more sophisticated parsing, as the output of scenarios used to get the steps does not contain any reference to the result of it,

  scenarios: (100.00%) 1 scenario, 1 max VUs, 10m30s max duration (incl. graceful stop):
           * default: 1 iterations for each of 1 VUs (maxDuration: 10m0s, gracefulStop: 30s)

There are other executors which do utilize this Steps field better, but it is more of a legacy feature, it is not utilised at all in the UI for example, and some of the newer executors don't even populate this field.

Now, we do have an issue focusing on making the processing of k6 results more refined: https://github.com/kubeshop/testkube/issues/1649. From the title, I am assuming you only need the final result of the whole test run, not the individual scenario statuses, which should already be available in the Status field of the ExecutionResult. Do you need the results of the individual scenarios, and if yes, is this a blocker right now? Please also let me know if the webhook response does not contain this higher Status.

[avinashkna4]

@vLia Thanks for the response. you are right, we are indeed getting the "executionResult":{"status":"failed"..... We will utilize this field for now.

[vLia]

Perfect, please let us know if anything else comes up. Closing this issue now.

[VishwasSomasekhariah]

Hi @vLia, you are correct that for our use case, we only worry about the final result of the whole test run. As @avinashkna4 mentioned, we can confirm that the Status field of the ExecutionResult is being set correctly as failed and we can make use of it for now. Thank you for the assist again.