dynatrace-oss / progressiveDelivery-masterclass

Apache License 2.0
4 stars 11 forks source link

KeptnTasks needs `retries` and `timeout` parameters specified #102

Closed agardnerIT closed 1 year ago

agardnerIT commented 1 year ago

Scenario: Rolling out V2 (intentionally slower version of the app) causes the load test to fail (as it should because hte app is too slow).

level=error msg="thresholds on metrics 'http_req_duration' have been crossed"

This means k6 fails and causes hte container to exit with a non-zero code.

Keptn interprets this as the container itself failing and thus retries the job. The default retries number is 10 (see this thread as it is currently undocumented).

This results in the container CrashLooping and actually causes 10 load tests to be executed.

However, this failure is actually "correct" and thus we should take the first failure as "genuine" and only run the LT once then stop.

Solution

https://github.com/dynatrace-oss/progressiveDelivery-masterclass/blob/35e93c4e624e1c984078a6c4e8a0ad1a63055d4a/gitops/manifests/demo-application/keptn.yaml#L6

Explicitly set retries to 1 and timeout to 5m0s under spec: in the `KeptnTaskDefinition:

apiVersion: lifecycle.keptn.sh/v1alpha3
kind: KeptnTask
metadata:
spec:
  ...
  retries: 0
  timeout: 5m0s
  ...
agardnerIT commented 1 year ago

Fixed by #93