search

argoproj / argo-rollouts

Progressive Delivery for Kubernetes
https://argo-rollouts.readthedocs.io/
Apache License 2.0
2.8k stars 876 forks source link

Rollback fails for ConfigMaps #2675

Open abhi5296 opened 1 year ago

abhi5296 commented 1 year ago

Checklist:

Describe the bug

We're trying to implement automatic rollback for bad config but Rollback to previous versions causes older replica sets to scale up to 1 and then immediately back to 0 after 5-10 seconds.

To Reproduce

Expected behavior

When rolling back to a previous version of the config map, the older replica set should be scaled up to 1 and remain at 1. And newer replica set should be scaled to 0.

Version

v1.4.0

Logs

{"level":"info","msg":"Start processing","resource":"xxxxxx/app","time":"2023-03-06T04:35:29Z"}
{"generation":142,"level":"info","msg":"Started syncing rollout","namespace":"xxxxxx","resourceVersion":"605116944","rollout":"app","time":"2023-03-06T04:35:29Z"}
{"level":"info","msg":"Trigger on-analysis-run-failed result: [{[0].2jmj7l5rSw0yVb_vlWAYkK_YBwk  [analysis-run-failed] false}]","resource":"xxxxxx/app","time":"2023-03-06T04:35:29Z"}
{"level":"info","msg":"Trigger on-rollout-completed result: [{[0].2jmj7l5rSw0yVb_vlWAYkK_YBwk  [rollout-completed] false}]","resource":"xxxxxx/app","time":"2023-03-06T04:35:29Z"}
{"level":"info","msg":"Trigger on-analysis-run-error result: [{[0].2jmj7l5rSw0yVb_vlWAYkK_YBwk  [analysis-run-error] false}]","resource":"xxxxxx/app","time":"2023-03-06T04:35:29Z"}
{"level":"info","msg":"Processing completed","resource":"xxxxxx/app","time":"2023-03-06T04:35:29Z"}
{"level":"info","msg":"ComputePodTemplateHash hash changed (expected: 5c5df8ff4b, actual: c4dd98466)","namespace":"xxxxxx","rollout":"app","time":"2023-03-06T04:35:29Z"}
{"level":"info","msg":"Updating replica set 'app-c4dd98466' revision from 65 to 67","namespace":"xxxxxx","rollout":"app","time":"2023-03-06T04:35:29Z"}
{"level":"info","msg":"Pod template change detected (new: c4dd98466, old: 7c5f746dcb)","namespace":"xxxxxx","rollout":"app","time":"2023-03-06T04:35:29Z"}
{"level":"info","msg":"Enqueueing parent of xxxxxx/app-c4dd98466: Rollout xxxxxx/app","time":"2023-03-06T04:35:29Z"}
{"level":"info","msg":"Updating replica set 'app-c4dd98466' revision from 65 to 67","namespace":"xxxxxx","rollout":"app","time":"2023-03-06T04:35:29Z"}
{"generation":142,"level":"error","msg":"roCtx.reconcile err Operation cannot be fulfilled on replicasets.apps \"app-c4dd98466\": the object has been modified; please apply your changes to the latest version and try again","namespace":"xxxxxx","resourceVersion":"605116944","rollout":"app","time":"2023-03-06T04:35:29Z"}
{"generation":142,"level":"info","msg":"Reconciliation completed","namespace":"xxxxxx","resourceVersion":"605116944","rollout":"app","time":"2023-03-06T04:35:29Z","time_ms":212.640126}
{"level":"error","msg":"rollout syncHandler error: Operation cannot be fulfilled on replicasets.apps \"app-c4dd98466\": the object has been modified; please apply your changes to the latest version and try again","namespace":"xxxxxx","rollout":"app","time":"2023-03-06T04:35:29Z"}
{"level":"info","msg":"rollout syncHandler queue retries: 3 : key \"xxxxxx/app\"","namespace":"xxxxxx","rollout":"app","time":"2023-03-06T04:35:29Z"}
{"generation":142,"level":"info","msg":"Started syncing rollout","namespace":"xxxxxx","resourceVersion":"605116944","rollout":"app","time":"2023-03-06T04:35:29Z"}
{"level":"info","msg":"ComputePodTemplateHash hash changed (expected: 5c5df8ff4b, actual: c4dd98466)","namespace":"xxxxxx","rollout":"app","time":"2023-03-06T04:35:29Z"}
{"level":"info","msg":"Start processing","resource":"xxxxxx/app","time":"2023-03-06T04:35:29Z"}
{"level":"info","msg":"Trigger on-analysis-run-error result: [{[0].2jmj7l5rSw0yVb_vlWAYkK_YBwk  [analysis-run-error] false}]","resource":"xxxxxx/app","time":"2023-03-06T04:35:29Z"}
{"level":"info","msg":"Trigger on-rollout-completed result: [{[0].2jmj7l5rSw0yVb_vlWAYkK_YBwk  [rollout-completed] false}]","resource":"xxxxxx/app","time":"2023-03-06T04:35:29Z"}
{"level":"info","msg":"Trigger on-analysis-run-failed result: [{[0].2jmj7l5rSw0yVb_vlWAYkK_YBwk  [analysis-run-failed] false}]","resource":"xxxxxx/app","time":"2023-03-06T04:35:29Z"}
{"level":"info","msg":"Processing completed","resource":"xxxxxx/app","time":"2023-03-06T04:35:29Z"}
E0306 04:35:29.716442       1 controller.go:174] Operation cannot be fulfilled on replicasets.apps "app-c4dd98466": the object has been modified; please apply your changes to the latest version and try again
{"level":"debug","msg":"No configured destinations for trigger: on-rollout-updated","namespace":"xxxxxx","rollout":"app","time":"2023-03-06T04:35:29Z"}
{"level":"debug","msg":"Notification sent","namespace":"xxxxxx","rollout":"app","time":"2023-03-06T04:35:29Z","time_ms":0.11891399999999999}
{"event_reason":"RolloutUpdated","level":"info","msg":"Rollout updated to revision 67","namespace":"xxxxxx","rollout":"app","time":"2023-03-06T04:35:29Z"}
{"level":"info","msg":"Event(v1.ObjectReference{Kind:\"Rollout\", Namespace:\"xxxxxx\", Name:\"app\", UID:\"774c0ed8-61e9-447d-af0d-662a5f95d5d4\", APIVersion:\"argoproj.io/v1alpha1\", ResourceVersion:\"605116952\", FieldPath:\"\"}): type: 'Normal' reason: 'RolloutUpdated' Rollout updated to revision 67","time":"2023-03-06T04:35:29Z"}
{"level":"info","msg":"Pod template change detected (new: c4dd98466, old: 7c5f746dcb)","namespace":"xxxxxx","rollout":"app","time":"2023-03-06T04:35:29Z"}
{"level":"info","msg":"Pod template change detected (new: c4dd98466, old: 7c5f746dcb)","namespace":"xxxxxx","rollout":"app","time":"2023-03-06T04:35:29Z"}
{"level":"debug","msg":"No configured destinations for trigger: on-rollout-not-completed","namespace":"xxxxxx","rollout":"app","time":"2023-03-06T04:35:29Z"}
{"level":"debug","msg":"Notification sent","namespace":"xxxxxx","rollout":"app","time":"2023-03-06T04:35:29Z","time_ms":0.044606}
{"event_reason":"RolloutNotCompleted","level":"info","msg":"Rollout not completed, started update to revision 68 (c4dd98466)","namespace":"xxxxxx","rollout":"app","time":"2023-03-06T04:35:29Z"}
{"level":"info","msg":"Event(v1.ObjectReference{Kind:\"Rollout\", Namespace:\"xxxxxx\", Name:\"app\", UID:\"774c0ed8-61e9-447d-af0d-662a5f95d5d4\", APIVersion:\"argoproj.io/v1alpha1\", ResourceVersion:\"605116952\", FieldPath:\"\"}): type: 'Normal' reason: 'RolloutNotCompleted' Rollout not completed, started update to revision 68 (c4dd98466)","time":"2023-03-06T04:35:29Z"}
{"generation":142,"level":"info","msg":"Patched: {\"status\":{\"conditions\":[{\"lastTransitionTime\":\"2023-02-27T05:40:15Z\",\"lastUpdateTime\":\"2023-02-27T05:40:15Z\",\"message\":\"Rollout is paused\",\"reason\":\"RolloutPaused\",\"status\":\"False\",\"type\":\"Paused\"},{\"lastTransitionTime\":\"2023-03-05T16:11:03Z\",\"lastUpdateTime\":\"2023-03-05T16:11:03Z\",\"message\":\"Rollout has minimum availability\",\"reason\":\"AvailableReason\",\"status\":\"True\",\"type\":\"Available\"},{\"lastTransitionTime\":\"2023-03-06T04:35:29Z\",\"lastUpdateTime\":\"2023-03-06T04:35:29Z\",\"message\":\"Rollout is not healthy\",\"reason\":\"RolloutHealthy\",\"status\":\"False\",\"type\":\"Healthy\"},{\"lastTransitionTime\":\"2023-03-04T16:13:57Z\",\"lastUpdateTime\":\"2023-03-06T04:35:29Z\",\"message\":\"ReplicaSet \\\"app-c4dd98466\\\" is progressing.\",\"reason\":\"ReplicaSetUpdated\",\"status\":\"True\",\"type\":\"Progressing\"},{\"lastTransitionTime\":\"2023-03-06T04:35:29Z\",\"lastUpdateTime\":\"2023-03-06T04:35:29Z\",\"message\":\"RolloutCompleted\",\"reason\":\"RolloutCompleted\",\"status\":\"False\",\"type\":\"Completed\"}],\"currentPodHash\":\"c4dd98466\",\"currentStepIndex\":0,\"message\":\"more replicas need to be updated\",\"observedGeneration\":\"142\",\"phase\":\"Progressing\",\"updatedReplicas\":null}}","namespace":"xxxxxx","resourceVersion":"605116952","rollout":"app","time":"2023-03-06T04:35:29Z"}
{"generation":142,"level":"info","msg":"persisted to informer","namespace":"xxxxxx","resourceVersion":"605116957","rollout":"app","time":"2023-03-06T04:35:29Z"}
{"generation":142,"level":"info","msg":"Reconciliation completed","namespace":"xxxxxx","resourceVersion":"605116944","rollout":"app","time":"2023-03-06T04:35:29Z","time_ms":109.28402600000001}
{"generation":142,"level":"info","msg":"Started syncing rollout","namespace":"xxxxxx","resourceVersion":"605116957","rollout":"app","time":"2023-03-06T04:35:29Z"}
{"level":"info","msg":"ComputePodTemplateHash hash changed (expected: 5c5df8ff4b, actual: c4dd98466)","namespace":"xxxxxx","rollout":"app","time":"2023-03-06T04:35:29Z"}
{"level":"info","msg":"Start processing","resource":"xxxxxx/app","time":"2023-03-06T04:35:29Z"}
{"level":"info","msg":"Trigger on-rollout-completed result: [{[0].2jmj7l5rSw0yVb_vlWAYkK_YBwk  [rollout-completed] false}]","resource":"xxxxxx/app","time":"2023-03-06T04:35:29Z"}
{"level":"info","msg":"Trigger on-analysis-run-failed result: [{[0].2jmj7l5rSw0yVb_vlWAYkK_YBwk  [analysis-run-failed] false}]","resource":"xxxxxx/app","time":"2023-03-06T04:35:29Z"}
{"level":"info","msg":"Trigger on-analysis-run-error result: [{[0].2jmj7l5rSw0yVb_vlWAYkK_YBwk  [analysis-run-error] false}]","resource":"xxxxxx/app","time":"2023-03-06T04:35:29Z"}
{"level":"info","msg":"Processing completed","resource":"xxxxxx/app","time":"2023-03-06T04:35:29Z"}
{"level":"info","msg":"No TrafficRouting Reconcilers found","namespace":"xxxxxx","rollout":"app","time":"2023-03-06T04:35:29Z"}
{"level":"debug","msg":"No configured destinations for trigger: on-scaling-replica-set","namespace":"xxxxxx","rollout":"app","time":"2023-03-06T04:35:29Z"}
{"level":"debug","msg":"Notification sent","namespace":"xxxxxx","rollout":"app","time":"2023-03-06T04:35:29Z","time_ms":0.06810000000000001}
{"event_reason":"ScalingReplicaSet","level":"info","msg":"Scaled up ReplicaSet app-c4dd98466 (revision 67) from 0 to 1","namespace":"xxxxxx","rollout":"app","time":"2023-03-06T04:35:29Z"}
{"level":"info","msg":"Not finished reconciling new ReplicaSet 'app-c4dd98466'","namespace":"xxxxxx","rollout":"app","time":"2023-03-06T04:35:29Z"}
{"level":"info","msg":"Not finished reconciling ReplicaSets","namespace":"xxxxxx","rollout":"app","time":"2023-03-06T04:35:29Z"}
{"level":"info","msg":"Enqueueing parent of xxxxxx/app-c4dd98466: Rollout xxxxxx/app","time":"2023-03-06T04:35:29Z"}
{"level":"info","msg":"Event(v1.ObjectReference{Kind:\"Rollout\", Namespace:\"xxxxxx\", Name:\"app\", UID:\"774c0ed8-61e9-447d-af0d-662a5f95d5d4\", APIVersion:\"argoproj.io/v1alpha1\", ResourceVersion:\"605116957\", FieldPath:\"\"}): type: 'Normal' reason: 'ScalingReplicaSet' Scaled up ReplicaSet app-c4dd98466 (revision 67) from 0 to 1","time":"2023-03-06T04:35:29Z"}
{"level":"info","msg":"Timed out (false) [last progress check: 2023-03-06 04:35:29 +0000 UTC - now: 2023-03-06 04:35:29.774465411 +0000 UTC m=+1083615.577490147]","namespace":"xxxxxx","rollout":"app","time":"2023-03-06T04:35:29Z"}
{"generation":142,"level":"info","msg":"No status changes. Skipping patch","namespace":"xxxxxx","resourceVersion":"605116957","rollout":"app","time":"2023-03-06T04:35:29Z"}
{"level":"info","msg":"Queueing up rollout for a progress after 599s","namespace":"xxxxxx","rollout":"app","time":"2023-03-06T04:35:29Z"}
{"generation":142,"level":"info","msg":"Reconciliation completed","namespace":"xxxxxx","resourceVersion":"605116957","rollout":"app","time":"2023-03-06T04:35:29Z","time_ms":22.670316}
{"generation":142,"level":"info","msg":"Started syncing rollout","namespace":"xxxxxx","resourceVersion":"605116957","rollout":"app","time":"2023-03-06T04:35:29Z"}
{"level":"info","msg":"ComputePodTemplateHash hash changed (expected: 5c5df8ff4b, actual: c4dd98466)","namespace":"xxxxxx","rollout":"app","time":"2023-03-06T04:35:29Z"}
{"level":"info","msg":"No TrafficRouting Reconcilers found","namespace":"xxxxxx","rollout":"app","time":"2023-03-06T04:35:29Z"}
{"level":"info","msg":"Timed out (false) [last progress check: 2023-03-06 04:35:29 +0000 UTC - now: 2023-03-06 04:35:29.7799857 +0000 UTC m=+1083615.583010436]","namespace":"xxxxxx","rollout":"app","time":"2023-03-06T04:35:29Z"}
{"generation":142,"level":"info","msg":"No status changes. Skipping patch","namespace":"xxxxxx","resourceVersion":"605116957","rollout":"app","time":"2023-03-06T04:35:29Z"}
{"level":"info","msg":"Queueing up rollout for a progress after 599s","namespace":"xxxxxx","rollout":"app","time":"2023-03-06T04:35:29Z"}
{"generation":142,"level":"info","msg":"Reconciliation completed","namespace":"xxxxxx","resourceVersion":"605116957","rollout":"app","time":"2023-03-06T04:35:29Z","time_ms":5.425199}
{"generation":142,"level":"info","msg":"Started syncing rollout","namespace":"xxxxxx","resourceVersion":"605116957","rollout":"app","time":"2023-03-06T04:35:29Z"}
{"level":"info","msg":"ComputePodTemplateHash hash changed (expected: 5c5df8ff4b, actual: c4dd98466)","namespace":"xxxxxx","rollout":"app","time":"2023-03-06T04:35:29Z"}
{"level":"info","msg":"No TrafficRouting Reconcilers found","namespace":"xxxxxx","rollout":"app","time":"2023-03-06T04:35:29Z"}
{"level":"info","msg":"Timed out (false) [last progress check: 2023-03-06 04:35:29 +0000 UTC - now: 2023-03-06 04:35:29.785077984 +0000 UTC m=+1083615.588102726]","namespace":"xxxxxx","rollout":"app","time":"2023-03-06T04:35:29Z"}
{"generation":142,"level":"info","msg":"No status changes. Skipping patch","namespace":"xxxxxx","resourceVersion":"605116957","rollout":"app","time":"2023-03-06T04:35:29Z"}
{"level":"info","msg":"Queueing up rollout for a progress after 599s","namespace":"xxxxxx","rollout":"app","time":"2023-03-06T04:35:29Z"}
{"generation":142,"level":"info","msg":"Reconciliation completed","namespace":"xxxxxx","resourceVersion":"605116957","rollout":"app","time":"2023-03-06T04:35:29Z","time_ms":5.069825}
{"level":"info","msg":"Enqueueing parent of xxxxxx/app-c4dd98466: Rollout xxxxxx/app","time":"2023-03-06T04:35:29Z"}
{"generation":142,"level":"info","msg":"Started syncing rollout","namespace":"xxxxxx","resourceVersion":"605116957","rollout":"app","time":"2023-03-06T04:35:29Z"}
{"level":"info","msg":"ComputePodTemplateHash hash changed (expected: 5c5df8ff4b, actual: c4dd98466)","namespace":"xxxxxx","rollout":"app","time":"2023-03-06T04:35:29Z"}
{"level":"info","msg":"No TrafficRouting Reconcilers found","namespace":"xxxxxx","rollout":"app","time":"2023-03-06T04:35:29Z"}
{"level":"info","msg":"Timed out (false) [last progress check: 2023-03-06 04:35:29 +0000 UTC - now: 2023-03-06 04:35:29.833642089 +0000 UTC m=+1083615.636666814]","namespace":"xxxxxx","rollout":"app","time":"2023-03-06T04:35:29Z"}
{"generation":142,"level":"info","msg":"No status changes. Skipping patch","namespace":"xxxxxx","resourceVersion":"605116957","rollout":"app","time":"2023-03-06T04:35:29Z"}
{"level":"info","msg":"Queueing up rollout for a progress after 599s","namespace":"xxxxxx","rollout":"app","time":"2023-03-06T04:35:29Z"}
{"generation":142,"level":"info","msg":"Reconciliation completed","namespace":"xxxxxx","resourceVersion":"605116957","rollout":"app","time":"2023-03-06T04:35:29Z","time_ms":5.061815}
{"level":"info","msg":"Enqueueing parent of xxxxxx/app-c4dd98466: Rollout xxxxxx/app","time":"2023-03-06T04:35:29Z"}
{"generation":142,"level":"info","msg":"Started syncing rollout","namespace":"xxxxxx","resourceVersion":"605116957","rollout":"app","time":"2023-03-06T04:35:29Z"}
{"level":"info","msg":"ComputePodTemplateHash hash changed (expected: 5c5df8ff4b, actual: c4dd98466)","namespace":"xxxxxx","rollout":"app","time":"2023-03-06T04:35:29Z"}
{"level":"info","msg":"No TrafficRouting Reconcilers found","namespace":"xxxxxx","rollout":"app","time":"2023-03-06T04:35:29Z"}
{"level":"info","msg":"Start processing","resource":"xxxxxx/app","time":"2023-03-06T04:35:29Z"}
{"generation":142,"level":"info","msg":"Patched: {\"status\":{\"HPAReplicas\":2,\"conditions\":[{\"lastTransitionTime\":\"2023-02-27T05:40:15Z\",\"lastUpdateTime\":\"2023-02-27T05:40:15Z\",\"message\":\"Rollout is paused\",\"reason\":\"RolloutPaused\",\"status\":\"False\",\"type\":\"Paused\"},{\"lastTransitionTime\":\"2023-03-05T16:11:03Z\",\"lastUpdateTime\":\"2023-03-05T16:11:03Z\",\"message\":\"Rollout has minimum availability\",\"reason\":\"AvailableReason\",\"status\":\"True\",\"type\":\"Available\"},{\"lastTransitionTime\":\"2023-03-06T04:35:29Z\",\"lastUpdateTime\":\"2023-03-06T04:35:29Z\",\"message\":\"Rollout is not healthy\",\"reason\":\"RolloutHealthy\",\"status\":\"False\",\"type\":\"Healthy\"},{\"lastTransitionTime\":\"2023-03-06T04:35:29Z\",\"lastUpdateTime\":\"2023-03-06T04:35:29Z\",\"message\":\"RolloutCompleted\",\"reason\":\"RolloutCompleted\",\"status\":\"False\",\"type\":\"Completed\"},{\"lastTransitionTime\":\"2023-03-04T16:13:57Z\",\"lastUpdateTime\":\"2023-03-06T04:35:29Z\",\"message\":\"ReplicaSet \\\"app-c4dd98466\\\" is progressing.\",\"reason\":\"ReplicaSetUpdated\",\"status\":\"True\",\"type\":\"Progressing\"}],\"message\":\"old replicas are pending termination\",\"replicas\":2,\"updatedReplicas\":1}}","namespace":"xxxxxx","resourceVersion":"605116957","rollout":"app","time":"2023-03-06T04:35:29Z"}
{"level":"info","msg":"Trigger on-rollout-completed result: [{[0].2jmj7l5rSw0yVb_vlWAYkK_YBwk  [rollout-completed] false}]","resource":"xxxxxx/app","time":"2023-03-06T04:35:29Z"}
{"level":"info","msg":"Trigger on-analysis-run-failed result: [{[0].2jmj7l5rSw0yVb_vlWAYkK_YBwk  [analysis-run-failed] false}]","resource":"xxxxxx/app","time":"2023-03-06T04:35:29Z"}
{"level":"info","msg":"Trigger on-analysis-run-error result: [{[0].2jmj7l5rSw0yVb_vlWAYkK_YBwk  [analysis-run-error] false}]","resource":"xxxxxx/app","time":"2023-03-06T04:35:29Z"}
{"level":"info","msg":"Processing completed","resource":"xxxxxx/app","time":"2023-03-06T04:35:29Z"}
{"generation":142,"level":"info","msg":"persisted to informer","namespace":"xxxxxx","resourceVersion":"605116967","rollout":"app","time":"2023-03-06T04:35:29Z"}
{"generation":142,"level":"info","msg":"Reconciliation completed","namespace":"xxxxxx","resourceVersion":"605116957","rollout":"app","time":"2023-03-06T04:35:29Z","time_ms":32.157389}
{"generation":142,"level":"info","msg":"Started syncing rollout","namespace":"xxxxxx","resourceVersion":"605116967","rollout":"app","time":"2023-03-06T04:35:29Z"}
{"level":"info","msg":"ComputePodTemplateHash hash changed (expected: 5c5df8ff4b, actual: c4dd98466)","namespace":"xxxxxx","rollout":"app","time":"2023-03-06T04:35:29Z"}
{"level":"info","msg":"No TrafficRouting Reconcilers found","namespace":"xxxxxx","rollout":"app","time":"2023-03-06T04:35:29Z"}
{"level":"info","msg":"Timed out (false) [last progress check: 2023-03-06 04:35:29 +0000 UTC - now: 2023-03-06 04:35:29.879668681 +0000 UTC m=+1083615.682693406]","namespace":"xxxxxx","rollout":"app","time":"2023-03-06T04:35:29Z"}
{"generation":142,"level":"info","msg":"No status changes. Skipping patch","namespace":"xxxxxx","resourceVersion":"605116967","rollout":"app","time":"2023-03-06T04:35:29Z"}
{"level":"info","msg":"Queueing up rollout for a progress after 599s","namespace":"xxxxxx","rollout":"app","time":"2023-03-06T04:35:29Z"}
{"generation":142,"level":"info","msg":"Reconciliation completed","namespace":"xxxxxx","resourceVersion":"605116967","rollout":"app","time":"2023-03-06T04:35:29Z","time_ms":4.785313}
{"level":"info","msg":"Start processing","resource":"xxxxxx/app","time":"2023-03-06T04:35:32Z"}
{"level":"info","msg":"Trigger on-rollout-completed result: [{[0].2jmj7l5rSw0yVb_vlWAYkK_YBwk  [rollout-completed] false}]","resource":"xxxxxx/app","time":"2023-03-06T04:35:32Z"}
{"level":"info","msg":"Trigger on-analysis-run-failed result: [{[0].2jmj7l5rSw0yVb_vlWAYkK_YBwk  [analysis-run-failed] false}]","resource":"xxxxxx/app","time":"2023-03-06T04:35:32Z"}
{"level":"info","msg":"Trigger on-analysis-run-error result: [{[0].2jmj7l5rSw0yVb_vlWAYkK_YBwk  [analysis-run-error] false}]","resource":"xxxxxx/app","time":"2023-03-06T04:35:32Z"}
{"level":"info","msg":"Processing completed","resource":"xxxxxx/app","time":"2023-03-06T04:35:32Z"}
{"generation":143,"level":"info","msg":"Started syncing rollout","namespace":"xxxxxx","resourceVersion":"605117032","rollout":"app","time":"2023-03-06T04:35:32Z"}
{"level":"info","msg":"Updating replica set 'app-7c5f746dcb' revision from 66 to 68","namespace":"xxxxxx","rollout":"app","time":"2023-03-06T04:35:32Z"}
{"level":"info","msg":"Enqueueing parent of xxxxxx/app-7c5f746dcb: Rollout xxxxxx/app","time":"2023-03-06T04:35:32Z"}
{"level":"info","msg":"Pod template change detected (new: 7c5f746dcb, old: c4dd98466)","namespace":"xxxxxx","rollout":"app","time":"2023-03-06T04:35:32Z"}
{"level":"info","msg":"Updating replica set 'app-7c5f746dcb' revision from 66 to 68","namespace":"xxxxxx","rollout":"app","time":"2023-03-06T04:35:32Z"}
{"generation":143,"level":"error","msg":"roCtx.reconcile err Operation cannot be fulfilled on replicasets.apps \"app-7c5f746dcb\": the object has been modified; please apply your changes to the latest version and try again","namespace":"xxxxxx","resourceVersion":"605117032","rollout":"app","time":"2023-03-06T04:35:32Z"}
{"generation":143,"level":"info","msg":"Reconciliation completed","namespace":"xxxxxx","resourceVersion":"605117032","rollout":"app","time":"2023-03-06T04:35:32Z","time_ms":36.782386}
{"level":"error","msg":"rollout syncHandler error: Operation cannot be fulfilled on replicasets.apps \"app-7c5f746dcb\": the object has been modified; please apply your changes to the latest version and try again","namespace":"xxxxxx","rollout":"app","time":"2023-03-06T04:35:32Z"}
{"level":"info","msg":"rollout syncHandler queue retries: 3 : key \"xxxxxx/app\"","namespace":"xxxxxx","rollout":"app","time":"2023-03-06T04:35:32Z"}
E0306 04:35:32.624979       1 controller.go:174] Operation cannot be fulfilled on replicasets.apps "app-7c5f746dcb": the object has been modified; please apply your changes to the latest version and try again
{"generation":143,"level":"info","msg":"Started syncing rollout","namespace":"xxxxxx","resourceVersion":"605117032","rollout":"app","time":"2023-03-06T04:35:32Z"}
{"level":"debug","msg":"No configured destinations for trigger: on-rollout-updated","namespace":"xxxxxx","rollout":"app","time":"2023-03-06T04:35:32Z"}
{"level":"debug","msg":"Notification sent","namespace":"xxxxxx","rollout":"app","time":"2023-03-06T04:35:32Z","time_ms":0.06804600000000001}
{"event_reason":"RolloutUpdated","level":"info","msg":"Rollout updated to revision 68","namespace":"xxxxxx","rollout":"app","time":"2023-03-06T04:35:32Z"}
{"level":"info","msg":"Event(v1.ObjectReference{Kind:\"Rollout\", Namespace:\"xxxxxx\", Name:\"app\", UID:\"774c0ed8-61e9-447d-af0d-662a5f95d5d4\", APIVersion:\"argoproj.io/v1alpha1\", ResourceVersion:\"605117035\", FieldPath:\"\"}): type: 'Normal' reason: 'RolloutUpdated' Rollout updated to revision 68","time":"2023-03-06T04:35:32Z"}
{"level":"info","msg":"Pod template change detected (new: 7c5f746dcb, old: c4dd98466)","namespace":"xxxxxx","rollout":"app","time":"2023-03-06T04:35:32Z"}
{"level":"info","msg":"Pod template change detected (new: 7c5f746dcb, old: c4dd98466)","namespace":"xxxxxx","rollout":"app","time":"2023-03-06T04:35:32Z"}
{"level":"debug","msg":"No configured destinations for trigger: on-skip-steps","namespace":"xxxxxx","rollout":"app","time":"2023-03-06T04:35:32Z"}
{"level":"debug","msg":"Notification sent","namespace":"xxxxxx","rollout":"app","time":"2023-03-06T04:35:32Z","time_ms":0.052247}
{"event_reason":"SkipSteps","level":"info","msg":"Rollback to stable","namespace":"xxxxxx","rollout":"app","time":"2023-03-06T04:35:32Z"}
{"level":"info","msg":"Event(v1.ObjectReference{Kind:\"Rollout\", Namespace:\"xxxxxx\", Name:\"app\", UID:\"774c0ed8-61e9-447d-af0d-662a5f95d5d4\", APIVersion:\"argoproj.io/v1alpha1\", ResourceVersion:\"605117035\", FieldPath:\"\"}): type: 'Normal' reason: 'SkipSteps' Rollback to stable","time":"2023-03-06T04:35:32Z"}
{"level":"info","msg":"Start processing","resource":"xxxxxx/app","time":"2023-03-06T04:35:32Z"}
{"level":"info","msg":"Trigger on-analysis-run-failed result: [{[0].2jmj7l5rSw0yVb_vlWAYkK_YBwk  [analysis-run-failed] false}]","resource":"xxxxxx/app","time":"2023-03-06T04:35:32Z"}
{"level":"info","msg":"Trigger on-rollout-completed result: [{[0].2jmj7l5rSw0yVb_vlWAYkK_YBwk  [rollout-completed] false}]","resource":"xxxxxx/app","time":"2023-03-06T04:35:32Z"}
{"level":"info","msg":"Trigger on-analysis-run-error result: [{[0].2jmj7l5rSw0yVb_vlWAYkK_YBwk  [analysis-run-error] false}]","resource":"xxxxxx/app","time":"2023-03-06T04:35:32Z"}
{"level":"info","msg":"Processing completed","resource":"xxxxxx/app","time":"2023-03-06T04:35:32Z"}
{"generation":144,"level":"info","msg":"Patched: {\"status\":{\"conditions\":[{\"lastTransitionTime\":\"2023-02-27T05:40:15Z\",\"lastUpdateTime\":\"2023-02-27T05:40:15Z\",\"message\":\"Rollout is paused\",\"reason\":\"RolloutPaused\",\"status\":\"False\",\"type\":\"Paused\"},{\"lastTransitionTime\":\"2023-03-05T16:11:03Z\",\"lastUpdateTime\":\"2023-03-05T16:11:03Z\",\"message\":\"Rollout has minimum availability\",\"reason\":\"AvailableReason\",\"status\":\"True\",\"type\":\"Available\"},{\"lastTransitionTime\":\"2023-03-06T04:35:29Z\",\"lastUpdateTime\":\"2023-03-06T04:35:29Z\",\"message\":\"Rollout is not healthy\",\"reason\":\"RolloutHealthy\",\"status\":\"False\",\"type\":\"Healthy\"},{\"lastTransitionTime\":\"2023-03-04T16:13:57Z\",\"lastUpdateTime\":\"2023-03-06T04:35:32Z\",\"message\":\"ReplicaSet \\\"app-7c5f746dcb\\\" is progressing.\",\"reason\":\"ReplicaSetUpdated\",\"status\":\"True\",\"type\":\"Progressing\"},{\"lastTransitionTime\":\"2023-03-06T04:35:32Z\",\"lastUpdateTime\":\"2023-03-06T04:35:32Z\",\"message\":\"RolloutCompleted\",\"reason\":\"RolloutCompleted\",\"status\":\"True\",\"type\":\"Completed\"}],\"currentPodHash\":\"7c5f746dcb\",\"currentStepIndex\":5,\"observedGeneration\":\"144\"}}","namespace":"xxxxxx","resourceVersion":"605117035","rollout":"app","time":"2023-03-06T04:35:32Z"}
{"generation":144,"level":"info","msg":"persisted to informer","namespace":"xxxxxx","resourceVersion":"605117036","rollout":"app","time":"2023-03-06T04:35:32Z"}
{"generation":143,"level":"info","msg":"Reconciliation completed","namespace":"xxxxxx","resourceVersion":"605117032","rollout":"app","time":"2023-03-06T04:35:32Z","time_ms":57.465849999999996}
{"level":"info","msg":"Start processing","resource":"xxxxxx/app","time":"2023-03-06T04:35:32Z"}
{"level":"info","msg":"Trigger on-analysis-run-error result: [{[0].2jmj7l5rSw0yVb_vlWAYkK_YBwk  [analysis-run-error] false}]","resource":"xxxxxx/app","time":"2023-03-06T04:35:32Z"}
{"level":"info","msg":"Trigger on-analysis-run-failed result: [{[0].2jmj7l5rSw0yVb_vlWAYkK_YBwk  [analysis-run-failed] false}]","resource":"xxxxxx/app","time":"2023-03-06T04:35:32Z"}
{"level":"info","msg":"Trigger on-rollout-completed result: [{[0].2jmj7l5rSw0yVb_vlWAYkK_YBwk  [rollout-completed] false}]","resource":"xxxxxx/app","time":"2023-03-06T04:35:32Z"}
{"level":"info","msg":"Processing completed","resource":"xxxxxx/app","time":"2023-03-06T04:35:32Z"}
{"generation":144,"level":"info","msg":"Started syncing rollout","namespace":"xxxxxx","resourceVersion":"605117036","rollout":"app","time":"2023-03-06T04:35:32Z"}
{"level":"info","msg":"No TrafficRouting Reconcilers found","namespace":"xxxxxx","rollout":"app","time":"2023-03-06T04:35:32Z"}
{"level":"info","msg":"No StableRS exists to reconcile or matches newRS","namespace":"xxxxxx","rollout":"app","time":"2023-03-06T04:35:32Z"}
{"level":"info","msg":"Reconciling 1 old ReplicaSets (total pods: 1)","namespace":"xxxxxx","rollout":"app","time":"2023-03-06T04:35:32Z"}
{"level":"info","msg":"Found 0 available pods in old RS xxxxxx/app-c4dd98466","namespace":"xxxxxx","rollout":"app","time":"2023-03-06T04:35:32Z"}
{"level":"info","msg":"Enqueueing parent of xxxxxx/app-c4dd98466: Rollout xxxxxx/app","time":"2023-03-06T04:35:32Z"}
{"level":"debug","msg":"No configured destinations for trigger: on-scaling-replica-set","namespace":"xxxxxx","rollout":"app","time":"2023-03-06T04:35:32Z"}
{"level":"debug","msg":"Notification sent","namespace":"xxxxxx","rollout":"app","time":"2023-03-06T04:35:32Z","time_ms":0.089621}
{"event_reason":"ScalingReplicaSet","level":"info","msg":"Scaled down ReplicaSet app-c4dd98466 (revision 67) from 1 to 0","namespace":"xxxxxx","rollout":"app","time":"2023-03-06T04:35:32Z"}
{"level":"info","msg":"No Steps remain in the canary steps","namespace":"xxxxxx","rollout":"app","time":"2023-03-06T04:35:32Z"}
{"level":"info","msg":"Event(v1.ObjectReference{Kind:\"Rollout\", Namespace:\"xxxxxx\", Name:\"app\", UID:\"774c0ed8-61e9-447d-af0d-662a5f95d5d4\", APIVersion:\"argoproj.io/v1alpha1\", ResourceVersion:\"605117036\", FieldPath:\"\"}): type: 'Normal' reason: 'ScalingReplicaSet' Scaled down ReplicaSet app-c4dd98466 (revision 67) from 1 to 0","time":"2023-03-06T04:35:32Z"}
{"level":"info","msg":"Timed out (false) [last progress check: 2023-03-06 04:35:32 +0000 UTC - now: 2023-03-06 04:35:32.704640658 +0000 UTC m=+1083618.507665394]","namespace":"xxxxxx","rollout":"app","time":"2023-03-06T04:35:32Z"}
{"generation":144,"level":"info","msg":"No status changes. Skipping patch","namespace":"xxxxxx","resourceVersion":"605117036","rollout":"app","time":"2023-03-06T04:35:32Z"}
{"level":"info","msg":"Queueing up rollout for a progress after 599s","namespace":"xxxxxx","rollout":"app","time":"2023-03-06T04:35:32Z"}
{"generation":144,"level":"info","msg":"Reconciliation completed","namespace":"xxxxxx","resourceVersion":"605117036","rollout":"app","time":"2023-03-06T04:35:32Z","time_ms":22.436213}
{"generation":144,"level":"info","msg":"Started syncing rollout","namespace":"xxxxxx","resourceVersion":"605117036","rollout":"app","time":"2023-03-06T04:35:32Z"}
{"level":"info","msg":"No TrafficRouting Reconcilers found","namespace":"xxxxxx","rollout":"app","time":"2023-03-06T04:35:32Z"}
{"level":"info","msg":"No StableRS exists to reconcile or matches newRS","namespace":"xxxxxx","rollout":"app","time":"2023-03-06T04:35:32Z"}
{"level":"info","msg":"No Steps remain in the canary steps","namespace":"xxxxxx","rollout":"app","time":"2023-03-06T04:35:32Z"}
{"level":"info","msg":"Timed out (false) [last progress check: 2023-03-06 04:35:32 +0000 UTC - now: 2023-03-06 04:35:32.709848617 +0000 UTC m=+1083618.512873344]","namespace":"xxxxxx","rollout":"app","time":"2023-03-06T04:35:32Z"}
{"generation":144,"level":"info","msg":"No status changes. Skipping patch","namespace":"xxxxxx","resourceVersion":"605117036","rollout":"app","time":"2023-03-06T04:35:32Z"}
{"level":"info","msg":"Queueing up rollout for a progress after 599s","namespace":"xxxxxx","rollout":"app","time":"2023-03-06T04:35:32Z"}
{"generation":144,"level":"info","msg":"Reconciliation completed","namespace":"xxxxxx","resourceVersion":"605117036","rollout":"app","time":"2023-03-06T04:35:32Z","time_ms":5.1092569999999995}
{"level":"info","msg":"Enqueueing parent of xxxxxx/app-c4dd98466: Rollout xxxxxx/app","time":"2023-03-06T04:35:32Z"}
{"generation":144,"level":"info","msg":"Started syncing rollout","namespace":"xxxxxx","resourceVersion":"605117036","rollout":"app","time":"2023-03-06T04:35:32Z"}
{"level":"info","msg":"No TrafficRouting Reconcilers found","namespace":"xxxxxx","rollout":"app","time":"2023-03-06T04:35:32Z"}
{"level":"info","msg":"No StableRS exists to reconcile or matches newRS","namespace":"xxxxxx","rollout":"app","time":"2023-03-06T04:35:32Z"}
{"level":"info","msg":"No Steps remain in the canary steps","namespace":"xxxxxx","rollout":"app","time":"2023-03-06T04:35:32Z"}
{"level":"info","msg":"Timed out (false) [last progress check: 2023-03-06 04:35:32 +0000 UTC - now: 2023-03-06 04:35:32.740281562 +0000 UTC m=+1083618.543306287]","namespace":"xxxxxx","rollout":"app","time":"2023-03-06T04:35:32Z"}
{"generation":144,"level":"info","msg":"No status changes. Skipping patch","namespace":"xxxxxx","resourceVersion":"605117036","rollout":"app","time":"2023-03-06T04:35:32Z"}
{"level":"info","msg":"Queueing up rollout for a progress after 599s","namespace":"xxxxxx","rollout":"app","time":"2023-03-06T04:35:32Z"}
{"generation":144,"level":"info","msg":"Reconciliation completed","namespace":"xxxxxx","resourceVersion":"605117036","rollout":"app","time":"2023-03-06T04:35:32Z","time_ms":6.0225089999999994}
{"level":"info","msg":"Enqueueing parent of xxxxxx/app-c4dd98466: Rollout xxxxxx/app","time":"2023-03-06T04:35:32Z"}
{"generation":144,"level":"info","msg":"Started syncing rollout","namespace":"xxxxxx","resourceVersion":"605117036","rollout":"app","time":"2023-03-06T04:35:32Z"}
{"level":"info","msg":"No TrafficRouting Reconcilers found","namespace":"xxxxxx","rollout":"app","time":"2023-03-06T04:35:32Z"}
{"level":"info","msg":"No StableRS exists to reconcile or matches newRS","namespace":"xxxxxx","rollout":"app","time":"2023-03-06T04:35:32Z"}
{"level":"info","msg":"No Steps remain in the canary steps","namespace":"xxxxxx","rollout":"app","time":"2023-03-06T04:35:32Z"}
{"generation":144,"level":"info","msg":"Patched: {\"status\":{\"HPAReplicas\":1,\"conditions\":[{\"lastTransitionTime\":\"2023-02-27T05:40:15Z\",\"lastUpdateTime\":\"2023-02-27T05:40:15Z\",\"message\":\"Rollout is paused\",\"reason\":\"RolloutPaused\",\"status\":\"False\",\"type\":\"Paused\"},{\"lastTransitionTime\":\"2023-03-05T16:11:03Z\",\"lastUpdateTime\":\"2023-03-05T16:11:03Z\",\"message\":\"Rollout has minimum availability\",\"reason\":\"AvailableReason\",\"status\":\"True\",\"type\":\"Available\"},{\"lastTransitionTime\":\"2023-03-06T04:35:32Z\",\"lastUpdateTime\":\"2023-03-06T04:35:32Z\",\"message\":\"RolloutCompleted\",\"reason\":\"RolloutCompleted\",\"status\":\"True\",\"type\":\"Completed\"},{\"lastTransitionTime\":\"2023-03-06T04:35:32Z\",\"lastUpdateTime\":\"2023-03-06T04:35:32Z\",\"message\":\"Rollout is healthy\",\"reason\":\"RolloutHealthy\",\"status\":\"True\",\"type\":\"Healthy\"},{\"lastTransitionTime\":\"2023-03-04T16:13:57Z\",\"lastUpdateTime\":\"2023-03-06T04:35:32Z\",\"message\":\"ReplicaSet \\\"app-7c5f746dcb\\\" has successfully progressed.\",\"reason\":\"NewReplicaSetAvailable\",\"status\":\"True\",\"type\":\"Progressing\"}],\"message\":null,\"phase\":\"Healthy\",\"replicas\":1}}","namespace":"xxxxxx","resourceVersion":"605117036","rollout":"app","time":"2023-03-06T04:35:32Z"}
{"generation":144,"level":"info","msg":"persisted to informer","namespace":"xxxxxx","resourceVersion":"605117049","rollout":"app","time":"2023-03-06T04:35:32Z"}
{"generation":144,"level":"info","msg":"Reconciliation completed","namespace":"xxxxxx","resourceVersion":"605117036","rollout":"app","time":"2023-03-06T04:35:32Z","time_ms":36.404262}
{"level":"info","msg":"Start processing","resource":"xxxxxx/app","time":"2023-03-06T04:35:32Z"}
{"level":"info","msg":"Trigger on-analysis-run-failed result: [{[0].2jmj7l5rSw0yVb_vlWAYkK_YBwk  [analysis-run-failed] false}]","resource":"xxxxxx/app","time":"2023-03-06T04:35:32Z"}
{"level":"info","msg":"Trigger on-rollout-completed result: [{[0].2jmj7l5rSw0yVb_vlWAYkK_YBwk  [rollout-completed] false}]","resource":"xxxxxx/app","time":"2023-03-06T04:35:32Z"}
{"level":"info","msg":"Trigger on-analysis-run-error result: [{[0].2jmj7l5rSw0yVb_vlWAYkK_YBwk  [analysis-run-error] false}]","resource":"xxxxxx/app","time":"2023-03-06T04:35:32Z"}
{"level":"info","msg":"Processing completed","resource":"xxxxxx/app","time":"2023-03-06T04:35:32Z"}
{"generation":144,"level":"info","msg":"Started syncing rollout","namespace":"xxxxxx","resourceVersion":"605117049","rollout":"app","time":"2023-03-06T04:35:32Z"}
{"level":"info","msg":"No TrafficRouting Reconcilers found","namespace":"xxxxxx","rollout":"app","time":"2023-03-06T04:35:32Z"}
{"level":"info","msg":"No StableRS exists to reconcile or matches newRS","namespace":"xxxxxx","rollout":"app","time":"2023-03-06T04:35:32Z"}
{"level":"info","msg":"No Steps remain in the canary steps","namespace":"xxxxxx","rollout":"app","time":"2023-03-06T04:35:32Z"}
{"generation":144,"level":"info","msg":"No status changes. Skipping patch","namespace":"xxxxxx","resourceVersion":"605117049","rollout":"app","time":"2023-03-06T04:35:32Z"}
{"generation":144,"level":"info","msg":"Reconciliation completed","namespace":"xxxxxx","resourceVersion":"605117049","rollout":"app","time":"2023-03-06T04:35:32Z","time_ms":6.3885130000000006}

Message from the maintainers:

Impacted by this bug? Give it a 👍. We prioritize the issues with the most 👍.

github-actions[bot] commented 1 year ago

This issue is stale because it has been open 60 days with no activity.

Sineaggi commented 1 year ago

@abhi5296 is this still an issue in v1.5.0?

github-actions[bot] commented 1 year ago

This issue is stale because it has been open 60 days with no activity.