search

argoproj / argo-rollouts

Progressive Delivery for Kubernetes
https://argo-rollouts.readthedocs.io/
Apache License 2.0
2.77k stars 867 forks source link

BlueGreen always abort on retrying once it got AnalysisRun fail #2824

Open httin opened 1 year ago

httin commented 1 year ago

Checklist:

Describe the bug Once AnalysisRuns got failed, the release is always aborted immediately after retrying. It is reproducible. If I delete the AnalysisRuns object, retrying works as expected.

Rollout Spec ```yaml apiVersion: argoproj.io/v1alpha1 kind: Rollout metadata: name: myservice-rollout spec: progressDeadlineSeconds: 2700 revisionHistoryLimit: 3 strategy: blueGreen: activeService: myservice-service autoPromotionEnabled: true prePromotionAnalysis: args: - name: latest-hash valueFrom: podTemplateHashValue: Latest templates: - templateName: myservice-functional-test - templateName: myservice-load-test - templateName: myservice-metrics previewService: myservice-service-green scaleDownDelayRevisionLimit: 1 scaleDownDelaySeconds: 300 workloadRef: apiVersion: apps/v1 kind: Deployment name: myservice-deploy ```

To Reproduce

Expected behavior

Screenshots

Version v1.5.0

Logs

Log for every retrying attempt ``` [argo-rollouts-bfff65d56-bk97r] time="2023-06-07T06:57:51Z" level=info msg="Start processing" resource=bluegreen/myservice-rollout [argo-rollouts-bfff65d56-bk97r] time="2023-06-07T06:57:51Z" level=info msg="Started syncing rollout" generation=3 namespace=bluegreen resourceVersion=937675920 rollout=myservice-rollout [argo-rollouts-bfff65d56-bk97r] time="2023-06-07T06:57:51Z" level=info msg="Trigger on-analysis-run-running result: [{[0].2jmj7l5rSw0yVb_vlWAYkK_YBwk [analysis-run-running] false}]" resource=bluegreen/myservice-rollout [argo-rollouts-bfff65d56-bk97r] time="2023-06-07T06:57:51Z" level=info msg="Trigger on-rollout-step-completed result: [{[0].2jmj7l5rSw0yVb_vlWAYkK_YBwk [rollout-step-completed] false}]" resource=bluegreen/myservice-rollout [argo-rollouts-bfff65d56-bk97r] time="2023-06-07T06:57:51Z" level=info msg="Trigger on-rollout-aborted result: [{[0].2jmj7l5rSw0yVb_vlWAYkK_YBwk [rollout-aborted] false}]" resource=bluegreen/myservice-rollout [argo-rollouts-bfff65d56-bk97r] time="2023-06-07T06:57:51Z" level=info msg="Trigger on-rollout-completed result: [{[0].2jmj7l5rSw0yVb_vlWAYkK_YBwk [rollout-completed] false}]" resource=bluegreen/myservice-rollout [argo-rollouts-bfff65d56-bk97r] time="2023-06-07T06:57:51Z" level=info msg="Trigger on-rollout-updated result: [{[0].2jmj7l5rSw0yVb_vlWAYkK_YBwk [rollout-updated] false}]" resource=bluegreen/myservice-rollout [argo-rollouts-bfff65d56-bk97r] time="2023-06-07T06:57:51Z" level=info msg="Trigger on-scaling-replica-set result: [{[0].2jmj7l5rSw0yVb_vlWAYkK_YBwk [scaling-replicaset] false}]" resource=bluegreen/myservice-rollout [argo-rollouts-bfff65d56-bk97r] time="2023-06-07T06:57:51Z" level=info msg="Trigger on-analysis-run-error result: [{[0].2jmj7l5rSw0yVb_vlWAYkK_YBwk [analysis-run-error] false}]" resource=bluegreen/myservice-rollout [argo-rollouts-bfff65d56-bk97r] time="2023-06-07T06:57:51Z" level=info msg="Trigger on-analysis-run-failed result: [{[0].2jmj7l5rSw0yVb_vlWAYkK_YBwk [analysis-run-failed] false}]" resource=bluegreen/myservice-rollout [argo-rollouts-bfff65d56-bk97r] time="2023-06-07T06:57:51Z" level=info msg="Processing completed" resource=bluegreen/myservice-rollout [argo-rollouts-bfff65d56-bk97r] time="2023-06-07T06:57:51Z" level=info msg="Patched conditions: {\"status\":{\"conditions\":[{\"lastTransitionTime\":\"2023-06-07T03:41:56Z\",\"lastUpdateTime\":\"2023-06-07T03:41:56Z\",\"message\":\"Rollout has minimum availability\",\"reason\":\"AvailableReason\",\"status\":\"True\",\"type\":\"Available\"},{\"lastTransitionTime\":\"2023-06-07T04:39:20Z\",\"lastUpdateTime\":\"2023-06-07T04:39:20Z\",\"message\":\"Rollout is not healthy\",\"reason\":\"RolloutHealthy\",\"status\":\"False\",\"type\":\"Healthy\"},{\"lastTransitionTime\":\"2023-06-07T04:39:20Z\",\"lastUpdateTime\":\"2023-06-07T04:39:20Z\",\"message\":\"RolloutCompleted\",\"reason\":\"RolloutCompleted\",\"status\":\"False\",\"type\":\"Completed\"},{\"lastTransitionTime\":\"2023-06-07T06:57:51Z\",\"lastUpdateTime\":\"2023-06-07T06:57:51Z\",\"message\":\"Retrying Rollout after abort\",\"reason\":\"RolloutRetry\",\"status\":\"Unknown\",\"type\":\"Progressing\"}],\"message\":\"more replicas need to be updated\",\"phase\":\"Progressing\"}}" generation=3 namespace=bluegreen resourceVersion=937675920 rollout=myservice-rollout [argo-rollouts-bfff65d56-bk97r] time="2023-06-07T06:57:51Z" level=info msg="Start processing" resource=bluegreen/myservice-rollout [argo-rollouts-bfff65d56-bk97r] time="2023-06-07T06:57:51Z" level=info msg="Trigger on-rollout-step-completed result: [{[0].2jmj7l5rSw0yVb_vlWAYkK_YBwk [rollout-step-completed] false}]" resource=bluegreen/myservice-rollout [argo-rollouts-bfff65d56-bk97r] time="2023-06-07T06:57:51Z" level=info msg="Trigger on-rollout-aborted result: [{[0].2jmj7l5rSw0yVb_vlWAYkK_YBwk [rollout-aborted] false}]" resource=bluegreen/myservice-rollout [argo-rollouts-bfff65d56-bk97r] time="2023-06-07T06:57:51Z" level=info msg="Trigger on-rollout-completed result: [{[0].2jmj7l5rSw0yVb_vlWAYkK_YBwk [rollout-completed] false}]" resource=bluegreen/myservice-rollout [argo-rollouts-bfff65d56-bk97r] time="2023-06-07T06:57:51Z" level=info msg="Trigger on-rollout-updated result: [{[0].2jmj7l5rSw0yVb_vlWAYkK_YBwk [rollout-updated] false}]" resource=bluegreen/myservice-rollout [argo-rollouts-bfff65d56-bk97r] time="2023-06-07T06:57:51Z" level=info msg="Trigger on-scaling-replica-set result: [{[0].2jmj7l5rSw0yVb_vlWAYkK_YBwk [scaling-replicaset] false}]" resource=bluegreen/myservice-rollout [argo-rollouts-bfff65d56-bk97r] time="2023-06-07T06:57:51Z" level=info msg="Trigger on-analysis-run-error result: [{[0].2jmj7l5rSw0yVb_vlWAYkK_YBwk [analysis-run-error] false}]" resource=bluegreen/myservice-rollout [argo-rollouts-bfff65d56-bk97r] time="2023-06-07T06:57:51Z" level=info msg="Trigger on-analysis-run-failed result: [{[0].2jmj7l5rSw0yVb_vlWAYkK_YBwk [analysis-run-failed] false}]" resource=bluegreen/myservice-rollout [argo-rollouts-bfff65d56-bk97r] time="2023-06-07T06:57:51Z" level=info msg="Trigger on-analysis-run-running result: [{[0].2jmj7l5rSw0yVb_vlWAYkK_YBwk [analysis-run-running] false}]" resource=bluegreen/myservice-rollout [argo-rollouts-bfff65d56-bk97r] time="2023-06-07T06:57:51Z" level=info msg="Processing completed" resource=bluegreen/myservice-rollout [argo-rollouts-bfff65d56-bk97r] time="2023-06-07T06:57:51Z" level=info msg="Removed 'scale-down-deadline' annotation from RS 'myservice-rollout-66dc89875b'" namespace=bluegreen rollout=myservice-rollout [argo-rollouts-bfff65d56-bk97r] time="2023-06-07T06:57:51Z" level=info msg="Reconciling stable ReplicaSet 'myservice-rollout-788bf6489b'" namespace=bluegreen rollout=myservice-rollout [argo-rollouts-bfff65d56-bk97r] time="2023-06-07T06:57:51Z" level=info msg="Enqueueing parent of bluegreen/myservice-rollout-66dc89875b: Rollout bluegreen/myservice-rollout" [argo-rollouts-bfff65d56-bk97r] time="2023-06-07T06:57:51Z" level=error msg="roCtx.reconcile err Operation cannot be fulfilled on replicasets.apps \"myservice-rollout-66dc89875b\": the object has been modified; please apply your changes to the latest version and try again" generation=3 namespace=bluegreen resourceVersion=937675920 rollout=myservice-rollout [argo-rollouts-bfff65d56-bk97r] time="2023-06-07T06:57:51Z" level=info msg="Reconciliation completed" generation=3 namespace=bluegreen resourceVersion=937675920 rollout=myservice-rollout time_ms=31.299924 [argo-rollouts-bfff65d56-bk97r] time="2023-06-07T06:57:51Z" level=error msg="rollout syncHandler error: Operation cannot be fulfilled on replicasets.apps \"myservice-rollout-66dc89875b\": the object has been modified; please apply your changes to the latest version and try again" namespace=bluegreen rollout=myservice-rollout [argo-rollouts-bfff65d56-bk97r] time="2023-06-07T06:57:51Z" level=info msg="rollout syncHandler queue retries: 4 : key \"bluegreen/myservice-rollout\"" namespace=bluegreen rollout=myservice-rollout [argo-rollouts-bfff65d56-bk97r] time="2023-06-07T06:57:51Z" level=error msg="Operation cannot be fulfilled on replicasets.apps \"myservice-rollout-66dc89875b\": the object has been modified; please apply your changes to the latest version and try again\n" error="" [argo-rollouts-bfff65d56-bk97r] time="2023-06-07T06:57:51Z" level=info msg="Started syncing rollout" generation=3 namespace=bluegreen resourceVersion=937675923 rollout=myservice-rollout [argo-rollouts-bfff65d56-bk97r] time="2023-06-07T06:57:51Z" level=info msg="Reconciling stable ReplicaSet 'myservice-rollout-788bf6489b'" namespace=bluegreen rollout=myservice-rollout [argo-rollouts-bfff65d56-bk97r] time="2023-06-07T06:57:51Z" level=info msg="Event(v1.ObjectReference{Kind:\"Rollout\", Namespace:\"bluegreen\", Name:\"myservice-rollout\", UID:\"2de7d458-5abf-4ce1-9d0e-caf8e66116d0\", APIVersion:\"argoproj.io/v1alpha1\", ResourceVersion:\"937675923\", FieldPath:\"\"}): type: 'Normal' reason: 'ScalingReplicaSet' Scaled up ReplicaSet myservice-rollout-66dc89875b (revision 2) from 0 to 2" [argo-rollouts-bfff65d56-bk97r] time="2023-06-07T06:57:51Z" level=info msg="Enqueueing parent of bluegreen/myservice-rollout-66dc89875b: Rollout bluegreen/myservice-rollout" [argo-rollouts-bfff65d56-bk97r] time="2023-06-07T06:57:51Z" level=info msg="Trigger on-scaling-replica-set result: [{[0].2jmj7l5rSw0yVb_vlWAYkK_YBwk [scaling-replicaset] false}]" [argo-rollouts-bfff65d56-bk97r] time="2023-06-07T06:57:51Z" level=info msg="Res When Condition hash: [0].2jmj7l5rSw0yVb_vlWAYkK_YBwk, Templates: [scaling-replicaset]" [argo-rollouts-bfff65d56-bk97r] time="2023-06-07T06:57:51Z" level=info msg="Enqueueing parent of bluegreen/myservice-rollout-66dc89875b: Rollout bluegreen/myservice-rollout" [argo-rollouts-bfff65d56-bk97r] time="2023-06-07T06:57:51Z" level=info msg="Enqueueing parent of bluegreen/myservice-rollout-66dc89875b: Rollout bluegreen/myservice-rollout" [argo-rollouts-bfff65d56-bk97r] time="2023-06-07T06:57:51Z" level=info msg="Trigger on-scaling-replica-set result: [{[0].2jmj7l5rSw0yVb_vlWAYkK_YBwk [scaling-replicaset] false}]" [argo-rollouts-bfff65d56-bk97r] time="2023-06-07T06:57:51Z" level=info msg="Res When Condition hash: [0].2jmj7l5rSw0yVb_vlWAYkK_YBwk, Templates: [scaling-replicaset]" [argo-rollouts-bfff65d56-bk97r] time="2023-06-07T06:57:51Z" level=error msg="notification error: notification service '' is not supported" [argo-rollouts-bfff65d56-bk97r] time="2023-06-07T06:57:51Z" level=error msg="Notifications failed to send for eventReason ScalingReplicaSet with error: notification service '' is not supported" event_reason=ScalingReplicaSet namespace=bluegreen rollout=myservice-rollout [argo-rollouts-bfff65d56-bk97r] time="2023-06-07T06:57:51Z" level=info msg="Scaled up ReplicaSet myservice-rollout-66dc89875b (revision 2) from 0 to 2" event_reason=ScalingReplicaSet namespace=bluegreen rollout=myservice-rollout [argo-rollouts-bfff65d56-bk97r] time="2023-06-07T06:57:51Z" level=info msg="New RS 'myservice-rollout-66dc89875b' is not ready to pause" namespace=bluegreen rollout=myservice-rollout [argo-rollouts-bfff65d56-bk97r] time="2023-06-07T06:57:51Z" level=info msg="skipping active service switch: New RS 'myservice-rollout-66dc89875b' is not fully saturated" namespace=bluegreen rollout=myservice-rollout [argo-rollouts-bfff65d56-bk97r] time="2023-06-07T06:57:51Z" level=info msg="Reconciling Pre Promotion Analysis" namespace=bluegreen rollout=myservice-rollout [argo-rollouts-bfff65d56-bk97r] time="2023-06-07T06:57:51Z" level=info msg="Event(v1.ObjectReference{Kind:\"Rollout\", Namespace:\"bluegreen\", Name:\"myservice-rollout\", UID:\"2de7d458-5abf-4ce1-9d0e-caf8e66116d0\", APIVersion:\"argoproj.io/v1alpha1\", ResourceVersion:\"937675923\", FieldPath:\"\"}): type: 'Warning' reason: 'RolloutAborted' Rollout aborted update to revision 2: Metric \"Cartographer5xxErrorRate\" assessed Failed due to failed (2) > failureLimit (1)" [argo-rollouts-bfff65d56-bk97r] time="2023-06-07T06:57:51Z" level=info msg="Trigger on-rollout-aborted result: [{[0].2jmj7l5rSw0yVb_vlWAYkK_YBwk [rollout-aborted] false}]" [argo-rollouts-bfff65d56-bk97r] time="2023-06-07T06:57:51Z" level=info msg="Res When Condition hash: [0].2jmj7l5rSw0yVb_vlWAYkK_YBwk, Templates: [rollout-aborted]" [argo-rollouts-bfff65d56-bk97r] time="2023-06-07T06:57:51Z" level=info msg="Trigger on-rollout-aborted result: [{[0].2jmj7l5rSw0yVb_vlWAYkK_YBwk [rollout-aborted] false}]" [argo-rollouts-bfff65d56-bk97r] time="2023-06-07T06:57:51Z" level=info msg="Res When Condition hash: [0].2jmj7l5rSw0yVb_vlWAYkK_YBwk, Templates: [rollout-aborted]" [argo-rollouts-bfff65d56-bk97r] time="2023-06-07T06:57:51Z" level=error msg="notification error: notification service '' is not supported" [argo-rollouts-bfff65d56-bk97r] time="2023-06-07T06:57:51Z" level=error msg="Notifications failed to send for eventReason RolloutAborted with error: notification service '' is not supported" event_reason=RolloutAborted namespace=bluegreen rollout=myservice-rollout [argo-rollouts-bfff65d56-bk97r] time="2023-06-07T06:57:51Z" level=warning msg="Rollout aborted update to revision 2: Metric \"Cartographer5xxErrorRate\" assessed Failed due to failed (2) > failureLimit (1)" event_reason=RolloutAborted namespace=bluegreen rollout=myservice-rollout [argo-rollouts-bfff65d56-bk97r] time="2023-06-07T06:57:51Z" level=info msg="Patched: {\"status\":{\"abort\":true,\"conditions\":[{\"lastTransitionTime\":\"2023-06-07T03:41:56Z\",\"lastUpdateTime\":\"2023-06-07T03:41:56Z\",\"message\":\"Rollout has minimum availability\",\"reason\":\"AvailableReason\",\"status\":\"True\",\"type\":\"Available\"},{\"lastTransitionTime\":\"2023-06-07T04:39:20Z\",\"lastUpdateTime\":\"2023-06-07T04:39:20Z\",\"message\":\"Rollout is not healthy\",\"reason\":\"RolloutHealthy\",\"status\":\"False\",\"type\":\"Healthy\"},{\"lastTransitionTime\":\"2023-06-07T04:39:20Z\",\"lastUpdateTime\":\"2023-06-07T04:39:20Z\",\"message\":\"RolloutCompleted\",\"reason\":\"RolloutCompleted\",\"status\":\"False\",\"type\":\"Completed\"},{\"lastTransitionTime\":\"2023-06-07T06:57:51Z\",\"lastUpdateTime\":\"2023-06-07T06:57:51Z\",\"message\":\"Rollout aborted update to revision 2: Metric \\\"Cartographer5xxErrorRate\\\" assessed Failed due to failed (2) \\u003e failureLimit (1)\",\"reason\":\"RolloutAborted\",\"status\":\"False\",\"type\":\"Progressing\"}],\"message\":\"RolloutAborted: Rollout aborted update to revision 2: Metric \\\"Cartographer5xxErrorRate\\\" assessed Failed due to failed (2) \\u003e failureLimit (1)\",\"phase\":\"Degraded\"}}" generation=3 namespace=bluegreen resourceVersion=937675923 rollout=myservice-rollout [argo-rollouts-bfff65d56-bk97r] time="2023-06-07T06:57:51Z" level=info msg="persisted to informer" generation=3 namespace=bluegreen resourceVersion=937675943 rollout=myservice-rollout [argo-rollouts-bfff65d56-bk97r] time="2023-06-07T06:57:51Z" level=info msg="Reconciliation completed" generation=3 namespace=bluegreen resourceVersion=937675923 rollout=myservice-rollout time_ms=327.578076 [argo-rollouts-bfff65d56-bk97r] time="2023-06-07T06:57:51Z" level=info msg="Start processing" resource=bluegreen/myservice-rollout [argo-rollouts-bfff65d56-bk97r] time="2023-06-07T06:57:51Z" level=info msg="Trigger on-rollout-aborted result: [{[0].2jmj7l5rSw0yVb_vlWAYkK_YBwk [rollout-aborted] false}]" resource=bluegreen/myservice-rollout [argo-rollouts-bfff65d56-bk97r] time="2023-06-07T06:57:51Z" level=info msg="Trigger on-rollout-completed result: [{[0].2jmj7l5rSw0yVb_vlWAYkK_YBwk [rollout-completed] false}]" resource=bluegreen/myservice-rollout [argo-rollouts-bfff65d56-bk97r] time="2023-06-07T06:57:51Z" level=info msg="Trigger on-rollout-updated result: [{[0].2jmj7l5rSw0yVb_vlWAYkK_YBwk [rollout-updated] false}]" resource=bluegreen/myservice-rollout [argo-rollouts-bfff65d56-bk97r] time="2023-06-07T06:57:51Z" level=info msg="Trigger on-scaling-replica-set result: [{[0].2jmj7l5rSw0yVb_vlWAYkK_YBwk [scaling-replicaset] false}]" resource=bluegreen/myservice-rollout [argo-rollouts-bfff65d56-bk97r] time="2023-06-07T06:57:51Z" level=info msg="Trigger on-analysis-run-error result: [{[0].2jmj7l5rSw0yVb_vlWAYkK_YBwk [analysis-run-error] false}]" resource=bluegreen/myservice-rollout [argo-rollouts-bfff65d56-bk97r] time="2023-06-07T06:57:51Z" level=info msg="Trigger on-analysis-run-failed result: [{[0].2jmj7l5rSw0yVb_vlWAYkK_YBwk [analysis-run-failed] false}]" resource=bluegreen/myservice-rollout [argo-rollouts-bfff65d56-bk97r] time="2023-06-07T06:57:51Z" level=info msg="Trigger on-analysis-run-running result: [{[0].2jmj7l5rSw0yVb_vlWAYkK_YBwk [analysis-run-running] false}]" resource=bluegreen/myservice-rollout [argo-rollouts-bfff65d56-bk97r] time="2023-06-07T06:57:51Z" level=info msg="Trigger on-rollout-step-completed result: [{[0].2jmj7l5rSw0yVb_vlWAYkK_YBwk [rollout-step-completed] false}]" resource=bluegreen/myservice-rollout [argo-rollouts-bfff65d56-bk97r] time="2023-06-07T06:57:51Z" level=info msg="Processing completed" resource=bluegreen/myservice-rollout [argo-rollouts-bfff65d56-bk97r] time="2023-06-07T06:57:51Z" level=info msg="Started syncing rollout" generation=3 namespace=bluegreen resourceVersion=937675943 rollout=myservice-rollout [argo-rollouts-bfff65d56-bk97r] time="2023-06-07T06:57:51Z" level=info msg="Reconciling stable ReplicaSet 'myservice-rollout-788bf6489b'" namespace=bluegreen rollout=myservice-rollout [argo-rollouts-bfff65d56-bk97r] time="2023-06-07T06:57:51Z" level=info msg="Scale down new rs 'myservice-rollout-66dc89875b' on abort (30s)" namespace=bluegreen rollout=myservice-rollout [argo-rollouts-bfff65d56-bk97r] time="2023-06-07T06:57:51Z" level=info msg="Set 'scale-down-deadline' annotation on 'myservice-rollout-66dc89875b' to 2023-06-07T06:58:21Z (30s)" namespace=bluegreen rollout=myservice-rollout [argo-rollouts-bfff65d56-bk97r] time="2023-06-07T06:57:51Z" level=info msg="skipping active service switch: New RS 'myservice-rollout-66dc89875b' is not fully saturated" namespace=bluegreen rollout=myservice-rollout [argo-rollouts-bfff65d56-bk97r] time="2023-06-07T06:57:51Z" level=info msg="Skipping analysis: isAborted: true, promoteFull: false, rollbackToScaleDownDelay: false, initialDeploy: false" namespace=bluegreen rollout=myservice-rollout [argo-rollouts-bfff65d56-bk97r] time="2023-06-07T06:57:51Z" level=info msg="Enqueueing parent of bluegreen/myservice-rollout-66dc89875b: Rollout bluegreen/myservice-rollout" [argo-rollouts-bfff65d56-bk97r] time="2023-06-07T06:57:51Z" level=info msg="Patched: {\"status\":{\"replicas\":4,\"updatedReplicas\":2}}" generation=3 namespace=bluegreen resourceVersion=937675943 rollout=myservice-rollout [argo-rollouts-bfff65d56-bk97r] time="2023-06-07T06:57:51Z" level=info msg="persisted to informer" generation=3 namespace=bluegreen resourceVersion=937675945 rollout=myservice-rollout [argo-rollouts-bfff65d56-bk97r] time="2023-06-07T06:57:51Z" level=info msg="Reconciliation completed" generation=3 namespace=bluegreen resourceVersion=937675943 rollout=myservice-rollout time_ms=26.639546 [argo-rollouts-bfff65d56-bk97r] time="2023-06-07T06:57:51Z" level=info msg="Started syncing rollout" generation=3 namespace=bluegreen resourceVersion=937675945 rollout=myservice-rollout [argo-rollouts-bfff65d56-bk97r] time="2023-06-07T06:57:51Z" level=info msg="Start processing" resource=bluegreen/myservice-rollout [argo-rollouts-bfff65d56-bk97r] time="2023-06-07T06:57:51Z" level=info msg="Trigger on-analysis-run-running result: [{[0].2jmj7l5rSw0yVb_vlWAYkK_YBwk [analysis-run-running] false}]" resource=bluegreen/myservice-rollout [argo-rollouts-bfff65d56-bk97r] time="2023-06-07T06:57:51Z" level=info msg="Trigger on-rollout-step-completed result: [{[0].2jmj7l5rSw0yVb_vlWAYkK_YBwk [rollout-step-completed] false}]" resource=bluegreen/myservice-rollout [argo-rollouts-bfff65d56-bk97r] time="2023-06-07T06:57:51Z" level=info msg="Trigger on-rollout-aborted result: [{[0].2jmj7l5rSw0yVb_vlWAYkK_YBwk [rollout-aborted] false}]" resource=bluegreen/myservice-rollout [argo-rollouts-bfff65d56-bk97r] time="2023-06-07T06:57:51Z" level=info msg="Trigger on-rollout-completed result: [{[0].2jmj7l5rSw0yVb_vlWAYkK_YBwk [rollout-completed] false}]" resource=bluegreen/myservice-rollout [argo-rollouts-bfff65d56-bk97r] time="2023-06-07T06:57:51Z" level=info msg="Trigger on-rollout-updated result: [{[0].2jmj7l5rSw0yVb_vlWAYkK_YBwk [rollout-updated] false}]" resource=bluegreen/myservice-rollout [argo-rollouts-bfff65d56-bk97r] time="2023-06-07T06:57:51Z" level=info msg="Trigger on-scaling-replica-set result: [{[0].2jmj7l5rSw0yVb_vlWAYkK_YBwk [scaling-replicaset] false}]" resource=bluegreen/myservice-rollout [argo-rollouts-bfff65d56-bk97r] time="2023-06-07T06:57:51Z" level=info msg="Trigger on-analysis-run-error result: [{[0].2jmj7l5rSw0yVb_vlWAYkK_YBwk [analysis-run-error] false}]" resource=bluegreen/myservice-rollout [argo-rollouts-bfff65d56-bk97r] time="2023-06-07T06:57:51Z" level=info msg="Trigger on-analysis-run-failed result: [{[0].2jmj7l5rSw0yVb_vlWAYkK_YBwk [analysis-run-failed] false}]" resource=bluegreen/myservice-rollout [argo-rollouts-bfff65d56-bk97r] time="2023-06-07T06:57:51Z" level=info msg="Processing completed" resource=bluegreen/myservice-rollout [argo-rollouts-bfff65d56-bk97r] time="2023-06-07T06:57:51Z" level=info msg="Reconciling stable ReplicaSet 'myservice-rollout-788bf6489b'" namespace=bluegreen rollout=myservice-rollout [argo-rollouts-bfff65d56-bk97r] time="2023-06-07T06:57:51Z" level=info msg="Scale down new rs 'myservice-rollout-66dc89875b' on abort (30s)" namespace=bluegreen rollout=myservice-rollout [argo-rollouts-bfff65d56-bk97r] time="2023-06-07T06:57:51Z" level=info msg="New rs 'myservice-rollout-66dc89875b' has scaledown deadline annotation: 2023-06-07T06:58:21Z" namespace=bluegreen rollout=myservice-rollout [argo-rollouts-bfff65d56-bk97r] time="2023-06-07T06:57:51Z" level=info msg="RS 'myservice-rollout-66dc89875b' has not reached the scaleDownTime" namespace=bluegreen rollout=myservice-rollout [argo-rollouts-bfff65d56-bk97r] time="2023-06-07T06:57:51Z" level=info msg="skipping active service switch: New RS 'myservice-rollout-66dc89875b' is not fully saturated" namespace=bluegreen rollout=myservice-rollout [argo-rollouts-bfff65d56-bk97r] time="2023-06-07T06:57:51Z" level=info msg="Skipping analysis: isAborted: true, promoteFull: false, rollbackToScaleDownDelay: true, initialDeploy: false" namespace=bluegreen rollout=myservice-rollout [argo-rollouts-bfff65d56-bk97r] time="2023-06-07T06:57:51Z" level=info msg="No status changes. Skipping patch" generation=3 namespace=bluegreen resourceVersion=937675945 rollout=myservice-rollout [argo-rollouts-bfff65d56-bk97r] time="2023-06-07T06:57:51Z" level=info msg="Reconciliation completed" generation=3 namespace=bluegreen resourceVersion=937675945 rollout=myservice-rollout time_ms=4.422574 [argo-rollouts-bfff65d56-bk97r] time="2023-06-07T06:57:51Z" level=info msg="Started syncing rollout" generation=3 namespace=bluegreen resourceVersion=937675945 rollout=myservice-rollout [argo-rollouts-bfff65d56-bk97r] time="2023-06-07T06:57:51Z" level=info msg="Reconciling stable ReplicaSet 'myservice-rollout-788bf6489b'" namespace=bluegreen rollout=myservice-rollout [argo-rollouts-bfff65d56-bk97r] time="2023-06-07T06:57:51Z" level=info msg="Scale down new rs 'myservice-rollout-66dc89875b' on abort (30s)" namespace=bluegreen rollout=myservice-rollout [argo-rollouts-bfff65d56-bk97r] time="2023-06-07T06:57:51Z" level=info msg="New rs 'myservice-rollout-66dc89875b' has scaledown deadline annotation: 2023-06-07T06:58:21Z" namespace=bluegreen rollout=myservice-rollout [argo-rollouts-bfff65d56-bk97r] time="2023-06-07T06:57:51Z" level=info msg="RS 'myservice-rollout-66dc89875b' has not reached the scaleDownTime" namespace=bluegreen rollout=myservice-rollout [argo-rollouts-bfff65d56-bk97r] time="2023-06-07T06:57:51Z" level=info msg="skipping active service switch: New RS 'myservice-rollout-66dc89875b' is not fully saturated" namespace=bluegreen rollout=myservice-rollout [argo-rollouts-bfff65d56-bk97r] time="2023-06-07T06:57:51Z" level=info msg="Skipping analysis: isAborted: true, promoteFull: false, rollbackToScaleDownDelay: true, initialDeploy: false" namespace=bluegreen rollout=myservice-rollout [argo-rollouts-bfff65d56-bk97r] time="2023-06-07T06:57:51Z" level=info msg="No status changes. Skipping patch" generation=3 namespace=bluegreen resourceVersion=937675945 rollout=myservice-rollout [argo-rollouts-bfff65d56-bk97r] time="2023-06-07T06:57:51Z" level=info msg="Reconciliation completed" generation=3 namespace=bluegreen resourceVersion=937675945 rollout=myservice-rollout time_ms=4.0290669999999995 ```

Message from the maintainers:

Impacted by this bug? Give it a 👍. We prioritize the issues with the most 👍.

github-actions[bot] commented 1 year ago

This issue is stale because it has been open 60 days with no activity.

amcamina commented 6 months ago

This is still happening to me

srinivasbandgi-ds commented 2 months ago

We are affected by this too. Using the standard istio 500 error rate metric as post-analysis, let it fail, get into aborted state. retry the rollout using the plugin, it spins up new revision pods, but fails immediately based on previous post-analysis run results, shouldn't it try checks again as part of retry ?