search

argoproj / argo-rollouts

Progressive Delivery for Kubernetes
https://argo-rollouts.readthedocs.io/
Apache License 2.0
2.76k stars 867 forks source link

Rollout stuck issue #3316

Open eugenepaniot opened 9 months ago

eugenepaniot commented 9 months ago

Checklist:

Describe the bug

We seem to have faced the issue described in:

https://github.com/argoproj/argo-rollouts/pull/3272 https://github.com/argoproj/argo-rollouts/pull/3257 https://github.com/argoproj/argo-rollouts/issues/3256

To mitigate the issue we've deployed master branch build from revision:

* 180b4bff (HEAD -> master, origin/master, origin/HEAD) chore(deps): bump github.com/evanphx/json-patch/v5 from 5.7.0 to 5.8.0 (#3309)

However it does not solve the issue.

To Reproduce

Expected behavior

Screenshots

Version

* 180b4bff (HEAD -> master, origin/master, origin/HEAD) chore(deps): bump github.com/evanphx/json-patch/v5 from 5.7.0 to 5.8.0 (#3309)

Logs

rollout controller:

# Paste the logs from the rollout controller

time="2024-01-18T13:05:34Z" level=error msg="roCtx.reconcile err error updating replicaset : Operation cannot be fulfilled on replicasets.apps \"application-internal-6fc65dcfc6\": the object has been modified; please apply your changes to the latest version and try again" generation=757 namespace=app-namespace resourceVersion=4332321110 rollout=application-internal

time="2024-01-18T13:05:34Z" level=info msg="Reconciliation completed" generation=757 namespace=app-namespace resourceVersion=4332321110 rollout=application-internal time_ms=38.83042

time="2024-01-18T13:05:34Z" level=error msg="rollout syncHandler error: error updating replicaset : Operation cannot be fulfilled on replicasets.apps \"application-internal-6fc65dcfc6\": the object has been modified; please apply your changes to the latest version and try again" namespace=app-namespace rollout=application-internal

time="2024-01-18T13:05:34Z" level=info msg="rollout syncHandler queue retries: 258 : key \"app-namespace/application-internal\"" namespace=app-namespace rollout=application-internal

time="2024-01-18T13:05:34Z" level=error msg="error updating replicaset : Operation cannot be fulfilled on replicasets.apps \"application-internal-6fc65dcfc6\": the object has been modified; please apply your changes to the latest version and try again\n" error="<nil>"

time="2024-01-18T13:05:44Z" level=info msg="Started syncing rollout" generation=757 namespace=app-namespace resourceVersion=4332321110 rollout=application-internal

argo rollouts:

$ kubectl  argo rollouts get rollout application-internal  -n app-namespace
Name:            application-internal
Namespace:       app-namespace
Status:          ◌ Progressing
Message:         more replicas need to be updated
Strategy:        Canary
  Step:          0/3
  SetWeight:     10
  ActualWeight:  0
Images:          eu.gcr.io/.../application-internal:6939469
                 eu.gcr.io/.../application-internal:afeee0f (stable)
Replicas:
  Desired:       4
  Current:       4
  Updated:       0
  Ready:         4
  Available:     4

NAME                                         KIND         STATUS         AGE   INFO
⟳ application-internal                            Rollout      ◌ Progressing  134d
├──# revision:717
│  └──⧉ application-internal-546856bb5c           ReplicaSet   • ScaledDown   52m   canary
├──# revision:716
│  └──⧉ application-internal-7fc96c849c           ReplicaSet   • ScaledDown   122m
├──# revision:715
│  ├──⧉ application-internal-575b7d5fc6           ReplicaSet   ✔ Healthy      161m
│  │  └──□ application-internal-575b7d5fc6-fxbzf  Pod          ✔ Running      161m  ready:2/2
│  └──α application-internal-575b7d5fc6-715-2     AnalysisRun  ✔ Successful   148m  ✔ 1
├──# revision:714
│  ├──⧉ application-internal-6fc65dcfc6           ReplicaSet   ✔ Healthy      177m  stable
│  │  ├──□ application-internal-6fc65dcfc6-fxgnb  Pod          ✔ Running      177m  ready:2/2
│  │  └──□ application-internal-6fc65dcfc6-ll8xg  Pod          ✔ Running      164m  ready:2/2
│  └──α application-internal-6fc65dcfc6-714-2     AnalysisRun  ✔ Successful   165m  ✔ 1
└──# revision:713
   └──⧉ application-internal-5f8bb4b544           ReplicaSet   • ScaledDown   179m

rs:

$ kubectl -n app-namespace  get rs | grep application-internal
application-internal-546856bb5c   0         0         0       52m  # rollout.argoproj.io/revision: "717"
application-internal-7fc96c849c   0         0         0       122m # rollout.argoproj.io/revision: "716"
application-internal-575b7d5fc6   1         1         1       161m # rollout.argoproj.io/revision: "715"
application-internal-6fc65dcfc6   2         2         2       177m # rollout.argoproj.io/revision: "714"
application-internal-5f8bb4b544   0         0         0       3h # rollout.argoproj.io/revision: "713"

happy to help with debugging, deploying a custom version, etc.

Workarond

We've restarted deploy/argo-rollouts. After controller restart:

$ kubectl  argo rollouts get rollout application-internal  -n app-namespace
Name:            application-internal
Namespace:       app-namespace
Status:          ◌ Progressing
Message:         more replicas need to be updated
Strategy:        Canary
  Step:          0/3
  SetWeight:     10
  ActualWeight:  0
Images:          eu.gcr.io/.../application-internal:1ba0f90 (canary)
                 eu.gcr.io/.../application-internal:afeee0f (stable)
Replicas:
  Desired:       4
  Current:       4
  Updated:       0
  Ready:         4
  Available:     4

NAME                                         KIND         STATUS         AGE   INFO
⟳ application-internal                            Rollout      ◌ Progressing  134d
├──# revision:717
│  └──⧉ application-internal-546856bb5c           ReplicaSet   ◌ Progressing  54m   canary
│     └──□ application-internal-546856bb5c-99xp7  Pod          ◌ Init:0/2     0s    ready:0/2
├──# revision:716
│  └──⧉ application-internal-7fc96c849c           ReplicaSet   • ScaledDown   123m
├──# revision:715
│  ├──⧉ application-internal-575b7d5fc6           ReplicaSet   • ScaledDown   163m
│  │  └──□ application-internal-575b7d5fc6-fxbzf  Pod          ◌ Terminating  163m  ready:2/2
│  └──α application-internal-575b7d5fc6-715-2     AnalysisRun  ✔ Successful   149m  ✔ 1
├──# revision:714
│  ├──⧉ application-internal-6fc65dcfc6           ReplicaSet   ✔ Healthy      178m  stable
│  │  ├──□ application-internal-6fc65dcfc6-fxgnb  Pod          ✔ Running      178m  ready:2/2
│  │  ├──□ application-internal-6fc65dcfc6-ll8xg  Pod          ✔ Running      166m  ready:2/2
│  │  ├──□ application-internal-6fc65dcfc6-dp8m8  Pod          ✔ Running      31s   ready:2/2
│  │  └──□ application-internal-6fc65dcfc6-jhcgr  Pod          ✔ Running      31s   ready:2/2
│  └──α application-internal-6fc65dcfc6-714-2     AnalysisRun  ✔ Successful   166m  ✔ 1
└──# revision:713
   └──⧉ application-internal-5f8bb4b544           ReplicaSet   • ScaledDown   3h1m

Message from the maintainers:

Impacted by this bug? Give it a 👍. We prioritize the issues with the most 👍.

eugenepaniot commented 9 months ago

Controller logs after restart (bottom -> top):


2024-01-18 13:30:30.357 
time="2024-01-18T13:30:30Z" level=info msg="Reconciliation completed" generation=757 namespace=app-namespace resourceVersion=4332565037 rollout=application-internal time_ms=33.285228000000004

2024-01-18 13:30:30.357 
time="2024-01-18T13:30:30Z" level=info msg="persisted to informer" generation=757 namespace=app-namespace resourceVersion=4332565046 rollout=application-internal
2024-01-18 13:30:30.354 
time="2024-01-18T13:30:30Z" level=info msg="Patched: {\"status\":{\"HPAReplicas\":4,\"availableReplicas\":4,\"readyReplicas\":4,\"replicas\":4}}" generation=757 namespace=app-namespace resourceVersion=4332565037 rollout=application-internal
2024-01-18 13:30:30.327 
time="2024-01-18T13:30:30Z" level=info msg="No Steps remain in the canary steps" namespace=app-namespace rollout=application-internal
2024-01-18 13:30:30.327 
time="2024-01-18T13:30:30Z" level=info msg="No TrafficRouting Reconcilers found" namespace=app-namespace rollout=application-internal
2024-01-18 13:30:30.325 
time="2024-01-18T13:30:30Z" level=info msg="Started syncing rollout" generation=757 namespace=app-namespace resourceVersion=4332565037 rollout=application-internal
2024-01-18 13:30:30.324 
time="2024-01-18T13:30:30Z" level=info msg="Reconciliation completed" generation=757 namespace=app-namespace resourceVersion=4332565037 rollout=application-internal time_ms=33.116942
2024-01-18 13:30:30.324 
time="2024-01-18T13:30:30Z" level=info msg="Queueing up rollout for a progress after 599s" namespace=app-namespace rollout=application-internal
2024-01-18 13:30:30.323 
time="2024-01-18T13:30:30Z" level=info msg="No status changes. Skipping patch" generation=757 namespace=app-namespace resourceVersion=4332565037 rollout=application-internal
2024-01-18 13:30:30.323 
time="2024-01-18T13:30:30Z" level=info msg="Timed out (false) [last progress check: 2024-01-18 13:30:30 +0000 UTC - now: 2024-01-18 13:30:30.323705864 +0000 UTC m=+723.090599918]" namespace=app-namespace rollout=application-internal
2024-01-18 13:30:30.323 
time="2024-01-18T13:30:30Z" level=info msg="Event(v1.ObjectReference{Kind:\"Rollout\", Namespace:\"app-namespace\", Name:\"application-internal\", UID:\"54546450-62d4-4bb2-92f6-17a049ec3243\", APIVersion:\"argoproj.io/v1alpha1\", ResourceVersion:\"4332565037\", FieldPath:\"\"}): type: 'Normal' reason: 'ScalingReplicaSet' Scaled up ReplicaSet application-internal-546856bb5c (revision 717) from 3 to 4"
2024-01-18 13:30:30.323 
time="2024-01-18T13:30:30Z" level=info msg="Not finished reconciling ReplicaSets" namespace=app-namespace rollout=application-internal
2024-01-18 13:30:30.323 
time="2024-01-18T13:30:30Z" level=info msg="Not finished reconciling new ReplicaSet 'application-internal-546856bb5c'" namespace=app-namespace rollout=application-internal
2024-01-18 13:30:30.322 
time="2024-01-18T13:30:30Z" level=info msg="Scaled up ReplicaSet application-internal-546856bb5c (revision 717) from 3 to 4" event_reason=ScalingReplicaSet namespace=app-namespace rollout=application-internal
2024-01-18 13:30:30.307 
time="2024-01-18T13:30:30Z" level=info msg="Enqueueing parent of app-namespace/application-internal-6fc65dcfc6: Rollout app-namespace/application-internal"
2024-01-18 13:30:30.298 
time="2024-01-18T13:30:30Z" level=info msg="No TrafficRouting Reconcilers found" namespace=app-namespace rollout=application-internal
2024-01-18 13:30:30.294 
time="2024-01-18T13:30:30Z" level=info msg="Started syncing rollout" generation=757 namespace=app-namespace resourceVersion=4332565037 rollout=application-internal
2024-01-18 13:30:30.294 
time="2024-01-18T13:30:30Z" level=info msg="Processing completed" resource=app-namespace/application-internal
2024-01-18 13:30:30.294 
time="2024-01-18T13:30:30Z" level=info msg="Trigger on-rollout-aborted result: [{[0].2jmj7l5rSw0yVb_vlWAYkK_YBwk  [rollout-aborted] false}]" resource=app-namespace/application-internal
2024-01-18 13:30:30.294 
time="2024-01-18T13:30:30Z" level=info msg="Trigger on-analysis-run-failed result: [{[0].2jmj7l5rSw0yVb_vlWAYkK_YBwk  [analysis-run-failed] false}]" resource=app-namespace/application-internal
2024-01-18 13:30:30.294 
time="2024-01-18T13:30:30Z" level=info msg="Trigger on-analysis-run-error result: [{[0].2jmj7l5rSw0yVb_vlWAYkK_YBwk  [analysis-run-error] false}]" resource=app-namespace/application-internal
2024-01-18 13:30:30.293 
time="2024-01-18T13:30:30Z" level=info msg="Start processing" resource=app-namespace/application-internal
2024-01-18 13:30:30.290 
time="2024-01-18T13:30:30Z" level=info msg="Reconciliation completed" generation=757 namespace=app-namespace resourceVersion=4332563912 rollout=application-internal time_ms=66.673889
2024-01-18 13:30:30.290 
time="2024-01-18T13:30:30Z" level=info msg="persisted to informer" generation=757 namespace=app-namespace resourceVersion=4332565037 rollout=application-internal
2024-01-18 13:30:30.288 
time="2024-01-18T13:30:30Z" level=info msg="Enqueueing parent of app-namespace/application-internal-6fc65dcfc6: Rollout app-namespace/application-internal"
2024-01-18 13:30:30.285 
time="2024-01-18T13:30:30Z" level=info msg="Patched: {\"status\":{\"availableReplicas\":5,\"conditions\":[{\"lastTransitionTime\":\"2024-01-18T10:36:03Z\",\"lastUpdateTime\":\"2024-01-18T10:36:03Z\",\"message\":\"Rollout is not healthy\",\"reason\":\"RolloutHealthy\",\"status\":\"False\",\"type\":\"Healthy\"},{\"lastTransitionTime\":\"2024-01-18T10:36:03Z\",\"lastUpdateTime\":\"2024-01-18T10:36:03Z\",\"message\":\"RolloutCompleted\",\"reason\":\"RolloutCompleted\",\"status\":\"False\",\"type\":\"Completed\"},{\"lastTransitionTime\":\"2024-01-18T13:19:14Z\",\"lastUpdateTime\":\"2024-01-18T13:19:14Z\",\"message\":\"Rollout has minimum availability\",\"reason\":\"AvailableReason\",\"status\":\"True\",\"type\":\"Available\"},{\"lastTransitionTime\":\"2024-01-18T13:29:34Z\",\"lastUpdateTime\":\"2024-01-18T13:29:34Z\",\"message\":\"Rollout is paused\",\"reason\":\"RolloutPaused\",\"status\":\"False\",\"type\":\"Paused\"},{\"lastTransitionTime\":\"2024-01-18T13:29:34Z\",\"lastUpdateTime\":\"2024-01-18T13:30:30Z\",\"message\":\"ReplicaSet \\\"application-internal-546856bb5c\\\" is progressing.\",\"reason\":\"ReplicaSetUpdated\",\"status\":\"True\",\"type\":\"Progressing\"}],\"readyReplicas\":5}}" generation=757 namespace=app-namespace resourceVersion=4332563912 rollout=application-internal
2024-01-18 13:30:30.258 
time="2024-01-18T13:30:30Z" level=info msg="Event(v1.ObjectReference{Kind:\"Rollout\", Namespace:\"app-namespace\", Name:\"application-internal\", UID:\"54546450-62d4-4bb2-92f6-17a049ec3243\", APIVersion:\"argoproj.io/v1alpha1\", ResourceVersion:\"4332563912\", FieldPath:\"\"}): type: 'Normal' reason: 'ScalingReplicaSet' Scaled down ReplicaSet application-internal-6fc65dcfc6 (revision 714) from 2 to 1"
2024-01-18 13:30:30.258 
time="2024-01-18T13:30:30Z" level=info msg="Not finished reconciling ReplicaSets" namespace=app-namespace rollout=application-internal
2024-01-18 13:30:30.258 
time="2024-01-18T13:30:30Z" level=info msg="Not finished reconciling stableRS" namespace=app-namespace rollout=application-internal
2024-01-18 13:30:30.258 
time="2024-01-18T13:30:30Z" level=info msg="Scaled down ReplicaSet application-internal-6fc65dcfc6 (revision 714) from 2 to 1" event_reason=ScalingReplicaSet namespace=app-namespace rollout=application-internal
2024-01-18 13:30:30.229 
time="2024-01-18T13:30:30Z" level=info msg="No TrafficRouting Reconcilers found" namespace=app-namespace rollout=application-internal
2024-01-18 13:30:30.225 
time="2024-01-18T13:30:30Z" level=info msg="Started syncing rollout" generation=757 namespace=app-namespace resourceVersion=4332563912 rollout=application-internal
2024-01-18 13:30:30.222 
time="2024-01-18T13:30:30Z" level=info msg="Enqueueing parent of app-namespace/application-internal-546856bb5c: Rollout app-namespace/application-internal"
2024-01-18 13:30:09.897 
time="2024-01-18T13:30:09Z" level=info msg="Reconciliation completed" generation=757 namespace=app-namespace resourceVersion=4332563912 rollout=application-internal time_ms=7.269118
2024-01-18 13:30:09.897 
time="2024-01-18T13:30:09Z" level=info msg="Queueing up rollout for a progress after 599s" namespace=app-namespace rollout=application-internal
2024-01-18 13:30:09.897 
time="2024-01-18T13:30:09Z" level=info msg="No status changes. Skipping patch" generation=757 namespace=app-namespace resourceVersion=4332563912 rollout=application-internal
2024-01-18 13:30:09.897 
time="2024-01-18T13:30:09Z" level=info msg="Timed out (false) [last progress check: 2024-01-18 13:30:09 +0000 UTC - now: 2024-01-18 13:30:09.897476783 +0000 UTC m=+702.664370858]" namespace=app-namespace rollout=application-internal
2024-01-18 13:30:09.896 
time="2024-01-18T13:30:09Z" level=info msg="No Steps remain in the canary steps" namespace=app-namespace rollout=application-internal
2024-01-18 13:30:09.896 
time="2024-01-18T13:30:09Z" level=info msg="No TrafficRouting Reconcilers found" namespace=app-namespace rollout=application-internal
2024-01-18 13:30:09.892 
time="2024-01-18T13:30:09Z" level=info msg="Started syncing rollout" generation=757 namespace=app-namespace resourceVersion=4332563912 rollout=application-internal
2024-01-18 13:30:09.890 
time="2024-01-18T13:30:09Z" level=info msg="Processing completed" resource=app-namespace/application-internal
2024-01-18 13:30:09.890 
time="2024-01-18T13:30:09Z" level=info msg="Trigger on-analysis-run-error result: [{[0].2jmj7l5rSw0yVb_vlWAYkK_YBwk  [analysis-run-error] false}]" resource=app-namespace/application-internal
2024-01-18 13:30:09.890 
time="2024-01-18T13:30:09Z" level=info msg="Trigger on-rollout-aborted result: [{[0].2jmj7l5rSw0yVb_vlWAYkK_YBwk  [rollout-aborted] false}]" resource=app-namespace/application-internal
2024-01-18 13:30:09.890 
time="2024-01-18T13:30:09Z" level=info msg="Trigger on-analysis-run-failed result: [{[0].2jmj7l5rSw0yVb_vlWAYkK_YBwk  [analysis-run-failed] false}]" resource=app-namespace/application-internal
2024-01-18 13:30:09.889 
time="2024-01-18T13:30:09Z" level=info msg="Start processing" resource=app-namespace/application-internal
2024-01-18 13:30:09.887 
time="2024-01-18T13:30:09Z" level=info msg="Reconciliation completed" generation=757 namespace=app-namespace resourceVersion=4332563900 rollout=application-internal time_ms=31.06391
2024-01-18 13:30:09.887 
time="2024-01-18T13:30:09Z" level=info msg="persisted to informer" generation=757 namespace=app-namespace resourceVersion=4332563912 rollout=application-internal
2024-01-18 13:30:09.884 
time="2024-01-18T13:30:09Z" level=info msg="Patched: {\"status\":{\"HPAReplicas\":5,\"replicas\":5,\"updatedReplicas\":3}}" generation=757 namespace=app-namespace resourceVersion=4332563900 rollout=application-internal
2024-01-18 13:30:09.859 
time="2024-01-18T13:30:09Z" level=info msg="No Steps remain in the canary steps" namespace=app-namespace rollout=application-internal
2024-01-18 13:30:09.859 
time="2024-01-18T13:30:09Z" level=info msg="No TrafficRouting Reconcilers found" namespace=app-namespace rollout=application-internal
2024-01-18 13:30:09.857 
time="2024-01-18T13:30:09Z" level=info msg="Started syncing rollout" generation=757 namespace=app-namespace resourceVersion=4332563900 rollout=application-internal
2024-01-18 13:30:09.854 
time="2024-01-18T13:30:09Z" level=info msg="Enqueueing parent of app-namespace/application-internal-546856bb5c: Rollout app-namespace/application-internal"
2024-01-18 13:30:09.846 
time="2024-01-18T13:30:09Z" level=info msg="Reconciliation completed" generation=757 namespace=app-namespace resourceVersion=4332563900 rollout=application-internal time_ms=7.237737999999999
2024-01-18 13:30:09.846 
time="2024-01-18T13:30:09Z" level=info msg="Queueing up rollout for a progress after 599s" namespace=app-namespace rollout=application-internal
2024-01-18 13:30:09.846 
time="2024-01-18T13:30:09Z" level=info msg="No status changes. Skipping patch" generation=757 namespace=app-namespace resourceVersion=4332563900 rollout=application-internal
2024-01-18 13:30:09.846 
time="2024-01-18T13:30:09Z" level=info msg="Timed out (false) [last progress check: 2024-01-18 13:30:09 +0000 UTC - now: 2024-01-18 13:30:09.846440433 +0000 UTC m=+702.613334493]" namespace=app-namespace rollout=application-internal
2024-01-18 13:30:09.845 
time="2024-01-18T13:30:09Z" level=info msg="No Steps remain in the canary steps" namespace=app-namespace rollout=application-internal
2024-01-18 13:30:09.845 
time="2024-01-18T13:30:09Z" level=info msg="No TrafficRouting Reconcilers found" namespace=app-namespace rollout=application-internal
2024-01-18 13:30:09.841 
time="2024-01-18T13:30:09Z" level=info msg="Started syncing rollout" generation=757 namespace=app-namespace resourceVersion=4332563900 rollout=application-internal
2024-01-18 13:30:09.838 
time="2024-01-18T13:30:09Z" level=info msg="Enqueueing parent of app-namespace/application-internal-546856bb5c: Rollout app-namespace/application-internal"
2024-01-18 13:30:09.769 
time="2024-01-18T13:30:09Z" level=info msg="Reconciliation completed" generation=757 namespace=app-namespace resourceVersion=4332563900 rollout=application-internal time_ms=4.354801999999999
2024-01-18 13:30:09.769 
time="2024-01-18T13:30:09Z" level=info msg="Queueing up rollout for a progress after 599s" namespace=app-namespace rollout=application-internal
2024-01-18 13:30:09.769 
time="2024-01-18T13:30:09Z" level=info msg="No status changes. Skipping patch" generation=757 namespace=app-namespace resourceVersion=4332563900 rollout=application-internal
2024-01-18 13:30:09.768 
time="2024-01-18T13:30:09Z" level=info msg="Timed out (false) [last progress check: 2024-01-18 13:30:09 +0000 UTC - now: 2024-01-18 13:30:09.768746267 +0000 UTC m=+702.535640329]" namespace=app-namespace rollout=application-internal
2024-01-18 13:30:09.768 
time="2024-01-18T13:30:09Z" level=info msg="No Steps remain in the canary steps" namespace=app-namespace rollout=application-internal
2024-01-18 13:30:09.768 
time="2024-01-18T13:30:09Z" level=info msg="No TrafficRouting Reconcilers found" namespace=app-namespace rollout=application-internal
2024-01-18 13:30:09.765 
time="2024-01-18T13:30:09Z" level=info msg="Started syncing rollout" generation=757 namespace=app-namespace resourceVersion=4332563900 rollout=application-internal
2024-01-18 13:30:09.764 
time="2024-01-18T13:30:09Z" level=info msg="Processing completed" resource=app-namespace/application-internal
2024-01-18 13:30:09.764 
time="2024-01-18T13:30:09Z" level=info msg="Trigger on-rollout-aborted result: [{[0].2jmj7l5rSw0yVb_vlWAYkK_YBwk  [rollout-aborted] false}]" resource=app-namespace/application-internal
2024-01-18 13:30:09.764 
time="2024-01-18T13:30:09Z" level=info msg="Trigger on-analysis-run-failed result: [{[0].2jmj7l5rSw0yVb_vlWAYkK_YBwk  [analysis-run-failed] false}]" resource=app-namespace/application-internal
2024-01-18 13:30:09.764 
time="2024-01-18T13:30:09Z" level=info msg="Trigger on-analysis-run-error result: [{[0].2jmj7l5rSw0yVb_vlWAYkK_YBwk  [analysis-run-error] false}]" resource=app-namespace/application-internal
2024-01-18 13:30:09.763 
time="2024-01-18T13:30:09Z" level=info msg="Start processing" resource=app-namespace/application-internal
2024-01-18 13:30:09.760 
time="2024-01-18T13:30:09Z" level=info msg="Reconciliation completed" generation=757 namespace=app-namespace resourceVersion=4332563892 rollout=application-internal time_ms=29.631196
2024-01-18 13:30:09.760 
time="2024-01-18T13:30:09Z" level=info msg="persisted to informer" generation=757 namespace=app-namespace resourceVersion=4332563900 rollout=application-internal
2024-01-18 13:30:09.758 
time="2024-01-18T13:30:09Z" level=info msg="Patched: {\"status\":{\"HPAReplicas\":4,\"availableReplicas\":4,\"readyReplicas\":4,\"replicas\":4}}" generation=757 namespace=app-namespace resourceVersion=4332563892 rollout=application-internal
2024-01-18 13:30:09.734 
time="2024-01-18T13:30:09Z" level=info msg="No Steps remain in the canary steps" namespace=app-namespace rollout=application-internal
2024-01-18 13:30:09.734 
time="2024-01-18T13:30:09Z" level=info msg="No TrafficRouting Reconcilers found" namespace=app-namespace rollout=application-internal
2024-01-18 13:30:09.732 
time="2024-01-18T13:30:09Z" level=info msg="Started syncing rollout" generation=757 namespace=app-namespace resourceVersion=4332563892 rollout=application-internal
2024-01-18 13:30:09.731 
time="2024-01-18T13:30:09Z" level=info msg="Reconciliation completed" generation=757 namespace=app-namespace resourceVersion=4332563892 rollout=application-internal time_ms=30.882494
2024-01-18 13:30:09.731 
time="2024-01-18T13:30:09Z" level=info msg="Queueing up rollout for a progress after 599s" namespace=app-namespace rollout=application-internal
2024-01-18 13:30:09.731 
time="2024-01-18T13:30:09Z" level=info msg="No status changes. Skipping patch" generation=757 namespace=app-namespace resourceVersion=4332563892 rollout=application-internal
2024-01-18 13:30:09.731 
time="2024-01-18T13:30:09Z" level=info msg="Timed out (false) [last progress check: 2024-01-18 13:30:09 +0000 UTC - now: 2024-01-18 13:30:09.730933061 +0000 UTC m=+702.497827107]" namespace=app-namespace rollout=application-internal
2024-01-18 13:30:09.730 
time="2024-01-18T13:30:09Z" level=info msg="Not finished reconciling ReplicaSets" namespace=app-namespace rollout=application-internal
2024-01-18 13:30:09.730 
time="2024-01-18T13:30:09Z" level=info msg="Event(v1.ObjectReference{Kind:\"Rollout\", Namespace:\"app-namespace\", Name:\"application-internal\", UID:\"54546450-62d4-4bb2-92f6-17a049ec3243\", APIVersion:\"argoproj.io/v1alpha1\", ResourceVersion:\"4332563892\", FieldPath:\"\"}): type: 'Normal' reason: 'ScalingReplicaSet' Scaled up ReplicaSet application-internal-546856bb5c (revision 717) from 2 to 3"
2024-01-18 13:30:09.730 
time="2024-01-18T13:30:09Z" level=info msg="Not finished reconciling new ReplicaSet 'application-internal-546856bb5c'" namespace=app-namespace rollout=application-internal
2024-01-18 13:30:09.730 
time="2024-01-18T13:30:09Z" level=info msg="Scaled up ReplicaSet application-internal-546856bb5c (revision 717) from 2 to 3" event_reason=ScalingReplicaSet namespace=app-namespace rollout=application-internal
2024-01-18 13:30:09.718 
time="2024-01-18T13:30:09Z" level=info msg="Enqueueing parent of app-namespace/application-internal-6fc65dcfc6: Rollout app-namespace/application-internal"
2024-01-18 13:30:09.706 
time="2024-01-18T13:30:09Z" level=info msg="No TrafficRouting Reconcilers found" namespace=app-namespace rollout=application-internal
2024-01-18 13:30:09.702 
time="2024-01-18T13:30:09Z" level=info msg="Enqueueing parent of app-namespace/application-internal-6fc65dcfc6: Rollout app-namespace/application-internal"
2024-01-18 13:30:09.701 
time="2024-01-18T13:30:09Z" level=info msg="Started syncing rollout" generation=757 namespace=app-namespace resourceVersion=4332563892 rollout=application-internal
2024-01-18 13:30:09.699 
time="2024-01-18T13:30:09Z" level=info msg="Processing completed" resource=app-namespace/application-internal
2024-01-18 13:30:09.699 
time="2024-01-18T13:30:09Z" level=info msg="Trigger on-rollout-aborted result: [{[0].2jmj7l5rSw0yVb_vlWAYkK_YBwk  [rollout-aborted] false}]" resource=app-namespace/application-internal
2024-01-18 13:30:09.699 
time="2024-01-18T13:30:09Z" level=info msg="Trigger on-analysis-run-failed result: [{[0].2jmj7l5rSw0yVb_vlWAYkK_YBwk  [analysis-run-failed] false}]" resource=app-namespace/application-internal
2024-01-18 13:30:09.699 
time="2024-01-18T13:30:09Z" level=info msg="Trigger on-analysis-run-error result: [{[0].2jmj7l5rSw0yVb_vlWAYkK_YBwk  [analysis-run-error] false}]" resource=app-namespace/application-internal
2024-01-18 13:30:09.698 
time="2024-01-18T13:30:09Z" level=info msg="Start processing" resource=app-namespace/application-internal
2024-01-18 13:30:09.696 
time="2024-01-18T13:30:09Z" level=info msg="Reconciliation completed" generation=757 namespace=app-namespace resourceVersion=4332562679 rollout=application-internal time_ms=58.785351
2024-01-18 13:30:09.696 
time="2024-01-18T13:30:09Z" level=info msg="persisted to informer" generation=757 namespace=app-namespace resourceVersion=4332563892 rollout=application-internal
2024-01-18 13:30:09.692 
time="2024-01-18T13:30:09Z" level=info msg="Patched: {\"status\":{\"availableReplicas\":5,\"conditions\":[{\"lastTransitionTime\":\"2024-01-18T10:36:03Z\",\"lastUpdateTime\":\"2024-01-18T10:36:03Z\",\"message\":\"Rollout is not healthy\",\"reason\":\"RolloutHealthy\",\"status\":\"False\",\"type\":\"Healthy\"},{\"lastTransitionTime\":\"2024-01-18T10:36:03Z\",\"lastUpdateTime\":\"2024-01-18T10:36:03Z\",\"message\":\"RolloutCompleted\",\"reason\":\"RolloutCompleted\",\"status\":\"False\",\"type\":\"Completed\"},{\"lastTransitionTime\":\"2024-01-18T13:19:14Z\",\"lastUpdateTime\":\"2024-01-18T13:19:14Z\",\"message\":\"Rollout has minimum availability\",\"reason\":\"AvailableReason\",\"status\":\"True\",\"type\":\"Available\"},{\"lastTransitionTime\":\"2024-01-18T13:29:34Z\",\"lastUpdateTime\":\"2024-01-18T13:29:34Z\",\"message\":\"Rollout is paused\",\"reason\":\"RolloutPaused\",\"status\":\"False\",\"type\":\"Paused\"},{\"lastTransitionTime\":\"2024-01-18T13:29:34Z\",\"lastUpdateTime\":\"2024-01-18T13:30:09Z\",\"message\":\"ReplicaSet \\\"application-internal-546856bb5c\\\" is progressing.\",\"reason\":\"ReplicaSetUpdated\",\"status\":\"True\",\"type\":\"Progressing\"}],\"readyReplicas\":5}}" generation=757 namespace=app-namespace resourceVersion=4332562679 rollout=application-internal
2024-01-18 13:30:09.665 
time="2024-01-18T13:30:09Z" level=info msg="Event(v1.ObjectReference{Kind:\"Rollout\", Namespace:\"app-namespace\", Name:\"application-internal\", UID:\"54546450-62d4-4bb2-92f6-17a049ec3243\", APIVersion:\"argoproj.io/v1alpha1\", ResourceVersion:\"4332562679\", FieldPath:\"\"}): type: 'Normal' reason: 'ScalingReplicaSet' Scaled down ReplicaSet application-internal-6fc65dcfc6 (revision 714) from 3 to 2"
2024-01-18 13:30:09.665 
time="2024-01-18T13:30:09Z" level=info msg="Not finished reconciling ReplicaSets" namespace=app-namespace rollout=application-internal
2024-01-18 13:30:09.665 
time="2024-01-18T13:30:09Z" level=info msg="Not finished reconciling stableRS" namespace=app-namespace rollout=application-internal
2024-01-18 13:30:09.665 
time="2024-01-18T13:30:09Z" level=info msg="Scaled down ReplicaSet application-internal-6fc65dcfc6 (revision 714) from 3 to 2" event_reason=ScalingReplicaSet namespace=app-namespace rollout=application-internal
2024-01-18 13:30:09.643 
time="2024-01-18T13:30:09Z" level=info msg="No TrafficRouting Reconcilers found" namespace=app-namespace rollout=application-internal
2024-01-18 13:30:09.639 
time="2024-01-18T13:30:09Z" level=info msg="Started syncing rollout" generation=757 namespace=app-namespace resourceVersion=4332562679 rollout=application-internal
2024-01-18 13:30:09.636 
time="2024-01-18T13:30:09Z" level=info msg="Enqueueing parent of app-namespace/application-internal-546856bb5c: Rollout app-namespace/application-internal"
2024-01-18 13:29:49.337 
time="2024-01-18T13:29:49Z" level=info msg="Reconciliation completed" generation=757 namespace=app-namespace resourceVersion=4332562679 rollout=application-internal time_ms=3.67861
2024-01-18 13:29:49.337 
time="2024-01-18T13:29:49Z" level=info msg="Queueing up rollout for a progress after 599s" namespace=app-namespace rollout=application-internal
2024-01-18 13:29:49.337 
time="2024-01-18T13:29:49Z" level=info msg="No status changes. Skipping patch" generation=757 namespace=app-namespace resourceVersion=4332562679 rollout=application-internal
2024-01-18 13:29:49.337 
time="2024-01-18T13:29:49Z" level=info msg="Timed out (false) [last progress check: 2024-01-18 13:29:49 +0000 UTC - now: 2024-01-18 13:29:49.337384394 +0000 UTC m=+682.104278449]" namespace=app-namespace rollout=application-internal
2024-01-18 13:29:49.336 
time="2024-01-18T13:29:49Z" level=info msg="No Steps remain in the canary steps" namespace=app-namespace rollout=application-internal
2024-01-18 13:29:49.336 
time="2024-01-18T13:29:49Z" level=info msg="No TrafficRouting Reconcilers found" namespace=app-namespace rollout=application-internal
2024-01-18 13:29:49.334 
time="2024-01-18T13:29:49Z" level=info msg="Started syncing rollout" generation=757 namespace=app-namespace resourceVersion=4332562679 rollout=application-internal
2024-01-18 13:29:49.334 
time="2024-01-18T13:29:49Z" level=info msg="Processing completed" resource=app-namespace/application-internal
2024-01-18 13:29:49.334 
time="2024-01-18T13:29:49Z" level=info msg="Trigger on-rollout-aborted result: [{[0].2jmj7l5rSw0yVb_vlWAYkK_YBwk  [rollout-aborted] false}]" resource=app-namespace/application-internal
2024-01-18 13:29:49.334 
time="2024-01-18T13:29:49Z" level=info msg="Trigger on-analysis-run-failed result: [{[0].2jmj7l5rSw0yVb_vlWAYkK_YBwk  [analysis-run-failed] false}]" resource=app-namespace/application-internal
2024-01-18 13:29:49.334 
time="2024-01-18T13:29:49Z" level=info msg="Trigger on-analysis-run-error result: [{[0].2jmj7l5rSw0yVb_vlWAYkK_YBwk  [analysis-run-error] false}]" resource=app-namespace/application-internal
2024-01-18 13:29:49.332 
time="2024-01-18T13:29:49Z" level=info msg="Start processing" resource=app-namespace/application-internal
2024-01-18 13:29:49.330 
time="2024-01-18T13:29:49Z" level=info msg="Reconciliation completed" generation=757 namespace=app-namespace resourceVersion=4332562666 rollout=application-internal time_ms=33.576164
2024-01-18 13:29:49.330 
time="2024-01-18T13:29:49Z" level=info msg="persisted to informer" generation=757 namespace=app-namespace resourceVersion=4332562679 rollout=application-internal
2024-01-18 13:29:49.328 
time="2024-01-18T13:29:49Z" level=info msg="Patched: {\"status\":{\"HPAReplicas\":5,\"replicas\":5,\"updatedReplicas\":2}}" generation=757 namespace=app-namespace resourceVersion=4332562666 rollout=application-internal
2024-01-18 13:29:49.302 
time="2024-01-18T13:29:49Z" level=info msg="No Steps remain in the canary steps" namespace=app-namespace rollout=application-internal
2024-01-18 13:29:49.302 
time="2024-01-18T13:29:49Z" level=info msg="No TrafficRouting Reconcilers found" namespace=app-namespace rollout=application-internal
2024-01-18 13:29:49.298 
time="2024-01-18T13:29:49Z" level=info msg="Started syncing rollout" generation=757 namespace=app-namespace resourceVersion=4332562666 rollout=application-internal
2024-01-18 13:29:49.295 
time="2024-01-18T13:29:49Z" level=info msg="Enqueueing parent of app-namespace/application-internal-546856bb5c: Rollout app-namespace/application-internal"
2024-01-18 13:29:49.282 
time="2024-01-18T13:29:49Z" level=info msg="Reconciliation completed" generation=757 namespace=app-namespace resourceVersion=4332562666 rollout=application-internal time_ms=3.842228
2024-01-18 13:29:49.282 
time="2024-01-18T13:29:49Z" level=info msg="Queueing up rollout for a progress after 599s" namespace=app-namespace rollout=application-internal
2024-01-18 13:29:49.282 
time="2024-01-18T13:29:49Z" level=info msg="No status changes. Skipping patch" generation=757 namespace=app-namespace resourceVersion=4332562666 rollout=application-internal
2024-01-18 13:29:49.282 
time="2024-01-18T13:29:49Z" level=info msg="Timed out (false) [last progress check: 2024-01-18 13:29:49 +0000 UTC - now: 2024-01-18 13:29:49.282415433 +0000 UTC m=+682.049309488]" namespace=app-namespace rollout=application-internal
2024-01-18 13:29:49.281 
time="2024-01-18T13:29:49Z" level=info msg="No Steps remain in the canary steps" namespace=app-namespace rollout=application-internal
2024-01-18 13:29:49.281 
time="2024-01-18T13:29:49Z" level=info msg="No TrafficRouting Reconcilers found" namespace=app-namespace rollout=application-internal
2024-01-18 13:29:49.279 
time="2024-01-18T13:29:49Z" level=info msg="Started syncing rollout" generation=757 namespace=app-namespace resourceVersion=4332562666 rollout=application-internal
2024-01-18 13:29:49.277 
time="2024-01-18T13:29:49Z" level=info msg="Enqueueing parent of app-namespace/application-internal-546856bb5c: Rollout app-namespace/application-internal"
2024-01-18 13:29:49.181 
time="2024-01-18T13:29:49Z" level=info msg="Reconciliation completed" generation=757 namespace=app-namespace resourceVersion=4332562666 rollout=application-internal time_ms=5.66521
2024-01-18 13:29:49.181 
time="2024-01-18T13:29:49Z" level=info msg="Queueing up rollout for a progress after 599s" namespace=app-namespace rollout=application-internal
2024-01-18 13:29:49.181 
time="2024-01-18T13:29:49Z" level=info msg="No status changes. Skipping patch" generation=757 namespace=app-namespace resourceVersion=4332562666 rollout=application-internal
2024-01-18 13:29:49.181 
time="2024-01-18T13:29:49Z" level=info msg="Timed out (false) [last progress check: 2024-01-18 13:29:49 +0000 UTC - now: 2024-01-18 13:29:49.18152355 +0000 UTC m=+681.948417618]" namespace=app-namespace rollout=application-internal
2024-01-18 13:29:49.180 
time="2024-01-18T13:29:49Z" level=info msg="No Steps remain in the canary steps" namespace=app-namespace rollout=application-internal
2024-01-18 13:29:49.180 
time="2024-01-18T13:29:49Z" level=info msg="No TrafficRouting Reconcilers found" namespace=app-namespace rollout=application-internal
2024-01-18 13:29:49.177 
time="2024-01-18T13:29:49Z" level=info msg="Started syncing rollout" generation=757 namespace=app-namespace resourceVersion=4332562666 rollout=application-internal
2024-01-18 13:29:49.175 
time="2024-01-18T13:29:49Z" level=info msg="Processing completed" resource=app-namespace/application-internal
2024-01-18 13:29:49.175 
time="2024-01-18T13:29:49Z" level=info msg="Trigger on-analysis-run-error result: [{[0].2jmj7l5rSw0yVb_vlWAYkK_YBwk  [analysis-run-error] false}]" resource=app-namespace/application-internal
2024-01-18 13:29:49.175 
time="2024-01-18T13:29:49Z" level=info msg="Trigger on-rollout-aborted result: [{[0].2jmj7l5rSw0yVb_vlWAYkK_YBwk  [rollout-aborted] false}]" resource=app-namespace/application-internal
2024-01-18 13:29:49.175 
time="2024-01-18T13:29:49Z" level=info msg="Trigger on-analysis-run-failed result: [{[0].2jmj7l5rSw0yVb_vlWAYkK_YBwk  [analysis-run-failed] false}]" resource=app-namespace/application-internal
2024-01-18 13:29:49.174 
time="2024-01-18T13:29:49Z" level=info msg="Start processing" resource=app-namespace/application-internal
2024-01-18 13:29:49.172 
time="2024-01-18T13:29:49Z" level=info msg="Reconciliation completed" generation=757 namespace=app-namespace resourceVersion=4332562654 rollout=application-internal time_ms=31.12714
2024-01-18 13:29:49.172 
time="2024-01-18T13:29:49Z" level=info msg="persisted to informer" generation=757 namespace=app-namespace resourceVersion=4332562666 rollout=application-internal
2024-01-18 13:29:49.169 
time="2024-01-18T13:29:49Z" level=info msg="Patched: {\"status\":{\"HPAReplicas\":4,\"availableReplicas\":4,\"readyReplicas\":4,\"replicas\":4}}" generation=757 namespace=app-namespace resourceVersion=4332562654 rollout=application-internal
2024-01-18 13:29:49.144 
time="2024-01-18T13:29:49Z" level=info msg="No Steps remain in the canary steps" namespace=app-namespace rollout=application-internal
2024-01-18 13:29:49.144 
time="2024-01-18T13:29:49Z" level=info msg="No TrafficRouting Reconcilers found" namespace=app-namespace rollout=application-internal
2024-01-18 13:29:49.142 
time="2024-01-18T13:29:49Z" level=info msg="Started syncing rollout" generation=757 namespace=app-namespace resourceVersion=4332562654 rollout=application-internal
2024-01-18 13:29:49.141 
time="2024-01-18T13:29:49Z" level=info msg="Reconciliation completed" generation=757 namespace=app-namespace resourceVersion=4332562654 rollout=application-internal time_ms=28.051906000000002
2024-01-18 13:29:49.141 
time="2024-01-18T13:29:49Z" level=info msg="Queueing up rollout for a progress after 599s" namespace=app-namespace rollout=application-internal
2024-01-18 13:29:49.141 
time="2024-01-18T13:29:49Z" level=info msg="No status changes. Skipping patch" generation=757 namespace=app-namespace resourceVersion=4332562654 rollout=application-internal
2024-01-18 13:29:49.140 
time="2024-01-18T13:29:49Z" level=info msg="Timed out (false) [last progress check: 2024-01-18 13:29:49 +0000 UTC - now: 2024-01-18 13:29:49.140766545 +0000 UTC m=+681.907660617]" namespace=app-namespace rollout=application-internal
2024-01-18 13:29:49.139 
time="2024-01-18T13:29:49Z" level=info msg="Event(v1.ObjectReference{Kind:\"Rollout\", Namespace:\"app-namespace\", Name:\"application-internal\", UID:\"54546450-62d4-4bb2-92f6-17a049ec3243\", APIVersion:\"argoproj.io/v1alpha1\", ResourceVersion:\"4332562654\", FieldPath:\"\"}): type: 'Normal' reason: 'ScalingReplicaSet' Scaled up ReplicaSet application-internal-546856bb5c (revision 717) from 1 to 2"
2024-01-18 13:29:49.139 
time="2024-01-18T13:29:49Z" level=info msg="Not finished reconciling ReplicaSets" namespace=app-namespace rollout=application-internal
2024-01-18 13:29:49.139 
time="2024-01-18T13:29:49Z" level=info msg="Not finished reconciling new ReplicaSet 'application-internal-546856bb5c'" namespace=app-namespace rollout=application-internal
2024-01-18 13:29:49.139 
time="2024-01-18T13:29:49Z" level=info msg="Scaled up ReplicaSet application-internal-546856bb5c (revision 717) from 1 to 2" event_reason=ScalingReplicaSet namespace=app-namespace rollout=application-internal
2024-01-18 13:29:49.127 
time="2024-01-18T13:29:49Z" level=info msg="Enqueueing parent of app-namespace/application-internal-6fc65dcfc6: Rollout app-namespace/application-internal"
2024-01-18 13:29:49.117 
time="2024-01-18T13:29:49Z" level=info msg="No TrafficRouting Reconcilers found" namespace=app-namespace rollout=application-internal
2024-01-18 13:29:49.114 
time="2024-01-18T13:29:49Z" level=info msg="Started syncing rollout" generation=757 namespace=app-namespace resourceVersion=4332562654 rollout=application-internal
2024-01-18 13:29:49.111 
time="2024-01-18T13:29:49Z" level=info msg="Enqueueing parent of app-namespace/application-internal-6fc65dcfc6: Rollout app-namespace/application-internal"
2024-01-18 13:29:49.088 
time="2024-01-18T13:29:49Z" level=info msg="Reconciliation completed" generation=757 namespace=app-namespace resourceVersion=4332562654 rollout=application-internal time_ms=33.672872
2024-01-18 13:29:49.088 
time="2024-01-18T13:29:49Z" level=info msg="Queueing up rollout for a progress after 599s" namespace=app-namespace rollout=application-internal
2024-01-18 13:29:49.088 
time="2024-01-18T13:29:49Z" level=info msg="No status changes. Skipping patch" generation=757 namespace=app-namespace resourceVersion=4332562654 rollout=application-internal
2024-01-18 13:29:49.088 
time="2024-01-18T13:29:49Z" level=info msg="Timed out (false) [last progress check: 2024-01-18 13:29:49 +0000 UTC - now: 2024-01-18 13:29:49.088369002 +0000 UTC m=+681.855263074]" namespace=app-namespace rollout=application-internal
2024-01-18 13:29:49.087 
time="2024-01-18T13:29:49Z" level=info msg="Event(v1.ObjectReference{Kind:\"Rollout\", Namespace:\"app-namespace\", Name:\"application-internal\", UID:\"54546450-62d4-4bb2-92f6-17a049ec3243\", APIVersion:\"argoproj.io/v1alpha1\", ResourceVersion:\"4332562654\", FieldPath:\"\"}): type: 'Normal' reason: 'ScalingReplicaSet' Scaled down ReplicaSet application-internal-6fc65dcfc6 (revision 714) from 4 to 3"
2024-01-18 13:29:49.087 
time="2024-01-18T13:29:49Z" level=info msg="Not finished reconciling ReplicaSets" namespace=app-namespace rollout=application-internal
2024-01-18 13:29:49.087 
time="2024-01-18T13:29:49Z" level=info msg="Not finished reconciling stableRS" namespace=app-namespace rollout=application-internal
2024-01-18 13:29:49.087 
time="2024-01-18T13:29:49Z" level=info msg="Scaled down ReplicaSet application-internal-6fc65dcfc6 (revision 714) from 4 to 3" event_reason=ScalingReplicaSet namespace=app-namespace rollout=application-internal
2024-01-18 13:29:49.060 
time="2024-01-18T13:29:49Z" level=info msg="No TrafficRouting Reconcilers found" namespace=app-namespace rollout=application-internal
2024-01-18 13:29:49.056 
time="2024-01-18T13:29:49Z" level=info msg="Started syncing rollout" generation=757 namespace=app-namespace resourceVersion=4332562654 rollout=application-internal
2024-01-18 13:29:49.056 
time="2024-01-18T13:29:49Z" level=info msg="Processing completed" resource=app-namespace/application-internal
2024-01-18 13:29:49.056 
time="2024-01-18T13:29:49Z" level=info msg="Trigger on-rollout-aborted result: [{[0].2jmj7l5rSw0yVb_vlWAYkK_YBwk  [rollout-aborted] false}]" resource=app-namespace/application-internal
2024-01-18 13:29:49.056 
time="2024-01-18T13:29:49Z" level=info msg="Trigger on-analysis-run-failed result: [{[0].2jmj7l5rSw0yVb_vlWAYkK_YBwk  [analysis-run-failed] false}]" resource=app-namespace/application-internal
2024-01-18 13:29:49.056 
time="2024-01-18T13:29:49Z" level=info msg="Trigger on-analysis-run-error result: [{[0].2jmj7l5rSw0yVb_vlWAYkK_YBwk  [analysis-run-error] false}]" resource=app-namespace/application-internal
2024-01-18 13:29:49.055 
time="2024-01-18T13:29:49Z" level=info msg="Start processing" resource=app-namespace/application-internal
2024-01-18 13:29:49.055 
time="2024-01-18T13:29:49Z" level=info msg="Reconciliation completed" generation=757 namespace=app-namespace resourceVersion=4332561918 rollout=application-internal time_ms=35.475315
2024-01-18 13:29:49.055 
time="2024-01-18T13:29:49Z" level=info msg="persisted to informer" generation=757 namespace=app-namespace resourceVersion=4332562654 rollout=application-internal
2024-01-18 13:29:49.048 
time="2024-01-18T13:29:49Z" level=info msg="Patched: {\"status\":{\"canary\":{\"currentStepAnalysisRunStatus\":null},\"conditions\":[{\"lastTransitionTime\":\"2024-01-18T10:36:03Z\",\"lastUpdateTime\":\"2024-01-18T10:36:03Z\",\"message\":\"Rollout is not healthy\",\"reason\":\"RolloutHealthy\",\"status\":\"False\",\"type\":\"Healthy\"},{\"lastTransitionTime\":\"2024-01-18T10:36:03Z\",\"lastUpdateTime\":\"2024-01-18T10:36:03Z\",\"message\":\"RolloutCompleted\",\"reason\":\"RolloutCompleted\",\"status\":\"False\",\"type\":\"Completed\"},{\"lastTransitionTime\":\"2024-01-18T13:19:14Z\",\"lastUpdateTime\":\"2024-01-18T13:19:14Z\",\"message\":\"Rollout has minimum availability\",\"reason\":\"AvailableReason\",\"status\":\"True\",\"type\":\"Available\"},{\"lastTransitionTime\":\"2024-01-18T13:29:34Z\",\"lastUpdateTime\":\"2024-01-18T13:29:34Z\",\"message\":\"Rollout is paused\",\"reason\":\"RolloutPaused\",\"status\":\"False\",\"type\":\"Paused\"},{\"lastTransitionTime\":\"2024-01-18T13:29:34Z\",\"lastUpdateTime\":\"2024-01-18T13:29:49Z\",\"message\":\"ReplicaSet \\\"application-internal-546856bb5c\\\" is progressing.\",\"reason\":\"ReplicaSetUpdated\",\"status\":\"True\",\"type\":\"Progressing\"}],\"currentStepIndex\":3}}" generation=757 namespace=app-namespace resourceVersion=4332561918 rollout=application-internal
2024-01-18 13:29:49.024 
time="2024-01-18T13:29:49Z" level=info msg="Event(v1.ObjectReference{Kind:\"Rollout\", Namespace:\"app-namespace\", Name:\"application-internal\", UID:\"54546450-62d4-4bb2-92f6-17a049ec3243\", APIVersion:\"argoproj.io/v1alpha1\", ResourceVersion:\"4332561918\", FieldPath:\"\"}): type: 'Normal' reason: 'RolloutStepCompleted' Rollout step 3/3 completed (analysis)"
2024-01-18 13:29:49.024 
time="2024-01-18T13:29:49Z" level=info msg="Rollout step 3/3 completed (analysis)" event_reason=RolloutStepCompleted namespace=app-namespace rollout=application-internal
2024-01-18 13:29:49.023 
time="2024-01-18T13:29:49Z" level=info msg="Event(v1.ObjectReference{Kind:\"Rollout\", Namespace:\"app-namespace\", Name:\"application-internal\", UID:\"54546450-62d4-4bb2-92f6-17a049ec3243\", APIVersion:\"argoproj.io/v1alpha1\", ResourceVersion:\"4332561918\", FieldPath:\"\"}): type: 'Normal' reason: 'AnalysisRunSuccessful' Step Analysis Run 'application-internal-546856bb5c-717-2' Status New: 'Successful' Previous: 'Running'"
2024-01-18 13:29:49.023 
time="2024-01-18T13:29:49Z" level=info msg="Step Analysis Run 'application-internal-546856bb5c-717-2' Status New: 'Successful' Previous: 'Running'" event_reason=AnalysisRunSuccessful namespace=app-namespace rollout=application-internal
2024-01-18 13:29:49.023 
time="2024-01-18T13:29:49Z" level=info msg="Reconciling analysis step (stepIndex: 2)" namespace=app-namespace rollout=application-internal
2024-01-18 13:29:49.023 
time="2024-01-18T13:29:49Z" level=info msg="No TrafficRouting Reconcilers found" namespace=app-namespace rollout=application-internal
2024-01-18 13:29:49.020 
time="2024-01-18T13:29:49Z" level=info msg="Started syncing rollout" generation=757 namespace=app-namespace resourceVersion=4332561918 rollout=application-internal
2024-01-18 13:29:49.018 
time="2024-01-18T13:29:49Z" level=info msg="Reconciliation completed" analysisrun=application-internal-546856bb5c-717-2 namespace=app-namespace time_ms=0.26737799999999995
2024-01-18 13:29:49.018 
time="2024-01-18T13:29:49Z" level=info msg="No status changes. Skipping patch" analysisrun=application-internal-546856bb5c-717-2 namespace=app-namespace
2024-01-18 13:29:49.018 
time="2024-01-18T13:29:49Z" level=info msg="Enqueueing parent of app-namespace/application-internal-546856bb5c-717-2: Rollout app-namespace/application-internal"
2024-01-18 13:29:49.018 
time="2024-01-18T13:29:49Z" level=info msg="Started syncing Analysis at (2024-01-18 13:29:49.018133913 +0000 UTC m=+681.785027974)" analysisrun=application-internal-546856bb5c-717-2 namespace=app-namespace
2024-01-18 13:29:49.018 
time="2024-01-18T13:29:49Z" level=info msg="Reconciliation completed" analysisrun=application-internal-546856bb5c-717-2 namespace=app-namespace time_ms=17.389265
2024-01-18 13:29:49.018 
time="2024-01-18T13:29:49Z" level=info msg="Patch status successfully" analysisrun=application-internal-546856bb5c-717-2 namespace=app-namespace
2024-01-18 13:29:49.007 
time="2024-01-18T13:29:49Z" level=info msg="Event(v1.ObjectReference{Kind:\"AnalysisRun\", Namespace:\"app-namespace\", Name:\"application-internal-546856bb5c-717-2\", UID:\"88049208-9b2f-4452-8943-f3d2e9507949\", APIVersion:\"argoproj.io/v1alpha1\", ResourceVersion:\"4332561914\", FieldPath:\"\"}): type: 'Normal' reason: 'AnalysisRunSuccessful' Analysis Completed. Result: Successful"
2024-01-18 13:29:49.007 
time="2024-01-18T13:29:49Z" level=info msg="Event(v1.ObjectReference{Kind:\"AnalysisRun\", Namespace:\"app-namespace\", Name:\"application-internal-546856bb5c-717-2\", UID:\"88049208-9b2f-4452-8943-f3d2e9507949\", APIVersion:\"argoproj.io/v1alpha1\", ResourceVersion:\"4332561914\", FieldPath:\"\"}): type: 'Normal' reason: 'MetricSuccessful' Metric 'kayenta' Completed. Result: Successful"
2024-01-18 13:29:49.007 
time="2024-01-18T13:29:49Z" level=info msg="Analysis Completed. Result: Successful" analysisrun=application-internal-546856bb5c-717-2 event_reason=AnalysisRunSuccessful namespace=app-namespace
2024-01-18 13:29:49.007 
time="2024-01-18T13:29:49Z" level=info msg="Metric 'kayenta' Completed. Result: Successful" analysisrun=application-internal-546856bb5c-717-2 event_reason=MetricSuccessful namespace=app-namespace
2024-01-18 13:29:49.007 
time="2024-01-18T13:29:49Z" level=info msg="Metric 'kayenta' transitioned from Running -> Successful" analysisrun=application-internal-546856bb5c-717-2 metric=kayenta namespace=app-namespace
2024-01-18 13:29:49.007 
time="2024-01-18T13:29:49Z" level=info msg="Measurement Completed. Result: Successful" analysisrun=application-internal-546856bb5c-717-2 metric=kayenta namespace=app-namespace
2024-01-18 13:29:49.001 
time="2024-01-18T13:29:49Z" level=info msg="Taking 1 Measurement(s)..." analysisrun=application-internal-546856bb5c-717-2 namespace=app-namespace
2024-01-18 13:29:49.001 
time="2024-01-18T13:29:49Z" level=info msg="Resuming in-progress measurement" analysisrun=application-internal-546856bb5c-717-2 metric=kayenta namespace=app-namespace
2024-01-18 13:29:49.000 
time="2024-01-18T13:29:49Z" level=info msg="Started syncing Analysis at (2024-01-18 13:29:49.000714205 +0000 UTC m=+681.767608355)" analysisrun=application-internal-546856bb5c-717-2 namespace=app-namespace
2024-01-18 13:29:34.474 
time="2024-01-18T13:29:34Z" level=info msg="Reconciliation completed" generation=757 namespace=app-namespace resourceVersion=4332561918 rollout=application-internal time_ms=3.7821119999999997
2024-01-18 13:29:34.474 
time="2024-01-18T13:29:34Z" level=info msg="No status changes. Skipping patch" generation=757 namespace=app-namespace resourceVersion=4332561918 rollout=application-internal
2024-01-18 13:29:34.473 
time="2024-01-18T13:29:34Z" level=info msg="Reconciling analysis step (stepIndex: 2)" namespace=app-namespace rollout=application-internal
2024-01-18 13:29:34.473 
time="2024-01-18T13:29:34Z" level=info msg="No TrafficRouting Reconcilers found" namespace=app-namespace rollout=application-internal
2024-01-18 13:29:34.471 
time="2024-01-18T13:29:34Z" level=info msg="Started syncing rollout" generation=757 namespace=app-namespace resourceVersion=4332561918 rollout=application-internal
2024-01-18 13:29:34.469 
time="2024-01-18T13:29:34Z" level=info msg="Processing completed" resource=app-namespace/application-internal
2024-01-18 13:29:34.469 
time="2024-01-18T13:29:34Z" level=info msg="Trigger on-rollout-aborted result: [{[0].2jmj7l5rSw0yVb_vlWAYkK_YBwk  [rollout-aborted] false}]" resource=app-namespace/application-internal
2024-01-18 13:29:34.469 
time="2024-01-18T13:29:34Z" level=info msg="Trigger on-analysis-run-failed result: [{[0].2jmj7l5rSw0yVb_vlWAYkK_YBwk  [analysis-run-failed] false}]" resource=app-namespace/application-internal
2024-01-18 13:29:34.469 
time="2024-01-18T13:29:34Z" level=info msg="Trigger on-analysis-run-error result: [{[0].2jmj7l5rSw0yVb_vlWAYkK_YBwk  [analysis-run-error] false}]" resource=app-namespace/application-internal
2024-01-18 13:29:34.468 
time="2024-01-18T13:29:34Z" level=info msg="Start processing" resource=app-namespace/application-internal
2024-01-18 13:29:34.466 
time="2024-01-18T13:29:34Z" level=info msg="Reconciliation completed" generation=757 namespace=app-namespace resourceVersion=4332561905 rollout=application-internal time_ms=33.328884
2024-01-18 13:29:34.466 
time="2024-01-18T13:29:34Z" level=info msg="persisted to informer" generation=757 namespace=app-namespace resourceVersion=4332561918 rollout=application-internal
2024-01-18 13:29:34.463 
time="2024-01-18T13:29:34Z" level=info msg="Patched: {\"status\":{\"canary\":{\"currentStepAnalysisRunStatus\":{\"status\":\"Running\"}}}}" generation=757 namespace=app-namespace resourceVersion=4332561905 rollout=application-internal
2024-01-18 13:29:34.436 
time="2024-01-18T13:29:34Z" level=info msg="Event(v1.ObjectReference{Kind:\"Rollout\", Namespace:\"app-namespace\", Name:\"application-internal\", UID:\"54546450-62d4-4bb2-92f6-17a049ec3243\", APIVersion:\"argoproj.io/v1alpha1\", ResourceVersion:\"4332561905\", FieldPath:\"\"}): type: 'Normal' reason: 'AnalysisRunRunning' Step Analysis Run 'application-internal-546856bb5c-717-2' Status New: 'Running' Previous: ''"
2024-01-18 13:29:34.436 
time="2024-01-18T13:29:34Z" level=info msg="Step Analysis Run 'application-internal-546856bb5c-717-2' Status New: 'Running' Previous: ''" event_reason=AnalysisRunRunning namespace=app-namespace rollout=application-internal
2024-01-18 13:29:34.436 
time="2024-01-18T13:29:34Z" level=info msg="Reconciling analysis step (stepIndex: 2)" namespace=app-namespace rollout=application-internal
2024-01-18 13:29:34.436 
time="2024-01-18T13:29:34Z" level=info msg="No TrafficRouting Reconcilers found" namespace=app-namespace rollout=application-internal
2024-01-18 13:29:34.434 
time="2024-01-18T13:29:34Z" level=info msg="Started syncing rollout" generation=757 namespace=app-namespace resourceVersion=4332561905 rollout=application-internal
2024-01-18 13:29:34.431 
time="2024-01-18T13:29:34Z" level=info msg="Reconciliation completed" analysisrun=application-internal-546856bb5c-717-2 namespace=app-namespace time_ms=0.38774000000000003
2024-01-18 13:29:34.431 
time="2024-01-18T13:29:34Z" level=info msg="No status changes. Skipping patch" analysisrun=application-internal-546856bb5c-717-2 namespace=app-namespace
2024-01-18 13:29:34.431 
time="2024-01-18T13:29:34Z" level=info msg="Enqueueing parent of app-namespace/application-internal-546856bb5c-717-2: Rollout app-namespace/application-internal"
2024-01-18 13:29:34.431 
time="2024-01-18T13:29:34Z" level=info msg="Enqueueing analysis after 14.568279967s" analysisrun=application-internal-546856bb5c-717-2 namespace=app-namespace
2024-01-18 13:29:34.431 
time="2024-01-18T13:29:34Z" level=info msg="Taking 0 Measurement(s)..." analysisrun=application-internal-546856bb5c-717-2 namespace=app-namespace
2024-01-18 13:29:34.431 
time="2024-01-18T13:29:34Z" level=info msg="Started syncing Analysis at (2024-01-18 13:29:34.431452065 +0000 UTC m=+667.198346126)" analysisrun=application-internal-546856bb5c-717-2 namespace=app-namespace
2024-01-18 13:29:34.431 
time="2024-01-18T13:29:34Z" level=info msg="Reconciliation completed" analysisrun=application-internal-546856bb5c-717-2 namespace=app-namespace time_ms=355.512182
2024-01-18 13:29:34.431 
time="2024-01-18T13:29:34Z" level=info msg="Patch status successfully" analysisrun=application-internal-546856bb5c-717-2 namespace=app-namespace
2024-01-18 13:29:34.418 
time="2024-01-18T13:29:34Z" level=info msg="Enqueueing analysis after 14.999901236s" analysisrun=application-internal-546856bb5c-717-2 namespace=app-namespace
2024-01-18 13:29:34.110 
time="2024-01-18T13:29:34Z" level=info msg="Reconciliation completed" generation=757 namespace=app-namespace resourceVersion=4332561905 rollout=application-internal time_ms=3.7816970000000003
2024-01-18 13:29:34.110 
time="2024-01-18T13:29:34Z" level=info msg="No status changes. Skipping patch" generation=757 namespace=app-namespace resourceVersion=4332561905 rollout=application-internal
2024-01-18 13:29:34.109 
time="2024-01-18T13:29:34Z" level=info msg="Reconciling analysis step (stepIndex: 2)" namespace=app-namespace rollout=application-internal
2024-01-18 13:29:34.109 
time="2024-01-18T13:29:34Z" level=info msg="No TrafficRouting Reconcilers found" namespace=app-namespace rollout=application-internal
2024-01-18 13:29:34.107 
time="2024-01-18T13:29:34Z" level=info msg="Started syncing rollout" generation=757 namespace=app-namespace resourceVersion=4332561905 rollout=application-internal
2024-01-18 13:29:34.106 
time="2024-01-18T13:29:34Z" level=info msg="Reconciliation completed" generation=757 namespace=app-namespace resourceVersion=4332561905 rollout=application-internal time_ms=3.4940499999999997
2024-01-18 13:29:34.106 
time="2024-01-18T13:29:34Z" level=info msg="No status changes. Skipping patch" generation=757 namespace=app-namespace resourceVersion=4332561905 rollout=application-internal
2024-01-18 13:29:34.106 
time="2024-01-18T13:29:34Z" level=info msg="Processing completed" resource=app-namespace/application-internal
2024-01-18 13:29:34.106 
time="2024-01-18T13:29:34Z" level=info msg="Trigger on-rollout-aborted result: [{[0].2jmj7l5rSw0yVb_vlWAYkK_YBwk  [rollout-aborted] false}]" resource=app-namespace/application-internal
2024-01-18 13:29:34.106 
time="2024-01-18T13:29:34Z" level=info msg="Trigger on-analysis-run-failed result: [{[0].2jmj7l5rSw0yVb_vlWAYkK_YBwk  [analysis-run-failed] false}]" resource=app-namespace/application-internal
2024-01-18 13:29:34.106 
time="2024-01-18T13:29:34Z" level=info msg="Trigger on-analysis-run-error result: [{[0].2jmj7l5rSw0yVb_vlWAYkK_YBwk  [analysis-run-error] false}]" resource=app-namespace/application-internal
2024-01-18 13:29:34.106 
time="2024-01-18T13:29:34Z" level=info msg="Reconciling analysis step (stepIndex: 2)" namespace=app-namespace rollout=application-internal
2024-01-18 13:29:34.106 
time="2024-01-18T13:29:34Z" level=info msg="No TrafficRouting Reconcilers found" namespace=app-namespace rollout=application-internal
2024-01-18 13:29:34.105 
time="2024-01-18T13:29:34Z" level=info msg="Start processing" resource=app-namespace/application-internal
2024-01-18 13:29:34.104 
time="2024-01-18T13:29:34Z" level=info msg="Started syncing rollout" generation=757 namespace=app-namespace resourceVersion=4332561905 rollout=application-internal
2024-01-18 13:29:34.103 
time="2024-01-18T13:29:34Z" level=info msg="Reconciliation completed" generation=757 namespace=app-namespace resourceVersion=4332561901 rollout=application-internal time_ms=65.843234
2024-01-18 13:29:34.103 
time="2024-01-18T13:29:34Z" level=info msg="persisted to informer" generation=757 namespace=app-namespace resourceVersion=4332561905 rollout=application-internal
2024-01-18 13:29:34.100 
time="2024-01-18T13:29:34Z" level=info msg="Patched: {\"status\":{\"canary\":{\"currentStepAnalysisRunStatus\":{\"name\":\"application-internal-546856bb5c-717-2\",\"status\":\"\"}}}}" generation=757 namespace=app-namespace resourceVersion=4332561901 rollout=application-internal
2024-01-18 13:29:34.076 
time="2024-01-18T13:29:34Z" level=info msg="Created AnalysisRun for step '2'" analysisrun=application-internal-546856bb5c-717-2 namespace=app-namespace rollout=application-internal
2024-01-18 13:29:34.076 
time="2024-01-18T13:29:34Z" level=info msg="Taking 1 Measurement(s)..." analysisrun=application-internal-546856bb5c-717-2 namespace=app-namespace
2024-01-18 13:29:34.076 
time="2024-01-18T13:29:34Z" level=info msg="Running initial measurement" analysisrun=application-internal-546856bb5c-717-2 metric=kayenta namespace=app-namespace
2024-01-18 13:29:34.075 
time="2024-01-18T13:29:34Z" level=info msg="Enqueueing parent of app-namespace/application-internal-546856bb5c-717-2: Rollout app-namespace/application-internal"
2024-01-18 13:29:34.075 
time="2024-01-18T13:29:34Z" level=info msg="Started syncing Analysis at (2024-01-18 13:29:34.075687372 +0000 UTC m=+666.842581557)" analysisrun=application-internal-546856bb5c-717-2 namespace=app-namespace
2024-01-18 13:29:34.068 
time="2024-01-18T13:29:34Z" level=info msg="Processing completed" resource=app-namespace/application-internal
2024-01-18 13:29:34.068 
time="2024-01-18T13:29:34Z" level=info msg="Trigger on-rollout-aborted result: [{[0].2jmj7l5rSw0yVb_vlWAYkK_YBwk  [rollout-aborted] false}]" resource=app-namespace/application-internal
2024-01-18 13:29:34.068 
time="2024-01-18T13:29:34Z" level=info msg="Trigger on-analysis-run-failed result: [{[0].2jmj7l5rSw0yVb_vlWAYkK_YBwk  [analysis-run-failed] false}]" resource=app-namespace/application-internal
2024-01-18 13:29:34.068 
time="2024-01-18T13:29:34Z" level=info msg="Trigger on-analysis-run-error result: [{[0].2jmj7l5rSw0yVb_vlWAYkK_YBwk  [analysis-run-error] false}]" resource=app-namespace/application-internal
2024-01-18 13:29:34.067 
time="2024-01-18T13:29:34Z" level=info msg="Start processing" resource=app-namespace/application-internal
2024-01-18 13:29:34.064 
time="2024-01-18T13:29:34Z" level=info msg="Reconciling analysis step (stepIndex: 2)" namespace=app-namespace rollout=application-internal
2024-01-18 13:29:34.064 
time="2024-01-18T13:29:34Z" level=info msg="No TrafficRouting Reconcilers found" namespace=app-namespace rollout=application-internal
2024-01-18 13:29:34.064 
time="2024-01-18T13:29:34Z" level=info msg="Patched conditions: {\"status\":{\"conditions\":[{\"lastTransitionTime\":\"2024-01-18T10:36:03Z\",\"lastUpdateTime\":\"2024-01-18T10:36:03Z\",\"message\":\"Rollout is not healthy\",\"reason\":\"RolloutHealthy\",\"status\":\"False\",\"type\":\"Healthy\"},{\"lastTransitionTime\":\"2024-01-18T10:36:03Z\",\"lastUpdateTime\":\"2024-01-18T10:36:03Z\",\"message\":\"RolloutCompleted\",\"reason\":\"RolloutCompleted\",\"status\":\"False\",\"type\":\"Completed\"},{\"lastTransitionTime\":\"2024-01-18T13:19:14Z\",\"lastUpdateTime\":\"2024-01-18T13:19:14Z\",\"message\":\"Rollout has minimum availability\",\"reason\":\"AvailableReason\",\"status\":\"True\",\"type\":\"Available\"},{\"lastTransitionTime\":\"2024-01-18T13:29:34Z\",\"lastUpdateTime\":\"2024-01-18T13:29:34Z\",\"message\":\"ReplicaSet \\\"application-internal-546856bb5c\\\" is progressing.\",\"reason\":\"ReplicaSetUpdated\",\"status\":\"True\",\"type\":\"Progressing\"},{\"lastTransitionTime\":\"2024-01-18T13:29:34Z\",\"lastUpdateTime\":\"2024-01-18T13:29:34Z\",\"message\":\"Rollout is paused\",\"reason\":\"RolloutPaused\",\"status\":\"False\",\"type\":\"Paused\"}]}}" generation=757 namespace=app-namespace resourceVersion=4332561901 rollout=application-internal
2024-01-18 13:29:34.038 
time="2024-01-18T13:29:34Z" level=info msg="Processing completed" resource=app-namespace/application-internal
2024-01-18 13:29:34.038 
time="2024-01-18T13:29:34Z" level=info msg="Trigger on-analysis-run-error result: [{[0].2jmj7l5rSw0yVb_vlWAYkK_YBwk  [analysis-run-error] false}]" resource=app-namespace/application-internal
2024-01-18 13:29:34.038 
time="2024-01-18T13:29:34Z" level=info msg="Trigger on-rollout-aborted result: [{[0].2jmj7l5rSw0yVb_vlWAYkK_YBwk  [rollout-aborted] false}]" resource=app-namespace/application-internal
2024-01-18 13:29:34.038 
time="2024-01-18T13:29:34Z" level=info msg="Trigger on-analysis-run-failed result: [{[0].2jmj7l5rSw0yVb_vlWAYkK_YBwk  [analysis-run-failed] false}]" resource=app-namespace/application-internal
2024-01-18 13:29:34.038 
time="2024-01-18T13:29:34Z" level=info msg="Started syncing rollout" generation=757 namespace=app-namespace resourceVersion=4332561901 rollout=application-internal
2024-01-18 13:29:34.037 
time="2024-01-18T13:29:34Z" level=info msg="Start processing" resource=app-namespace/application-internal
2024-01-18 13:29:34.037 
time="2024-01-18T13:29:34Z" level=info msg="Reconciliation completed" generation=757 namespace=app-namespace resourceVersion=4332528684 rollout=application-internal time_ms=36.192834
2024-01-18 13:29:34.037 
time="2024-01-18T13:29:34Z" level=info msg="persisted to informer" generation=757 namespace=app-namespace resourceVersion=4332561901 rollout=application-internal
2024-01-18 13:29:34.031 
time="2024-01-18T13:29:34Z" level=info msg="Patched: {\"status\":{\"conditions\":[{\"lastTransitionTime\":\"2024-01-18T10:36:03Z\",\"lastUpdateTime\":\"2024-01-18T10:36:03Z\",\"message\":\"Rollout is not healthy\",\"reason\":\"RolloutHealthy\",\"status\":\"False\",\"type\":\"Healthy\"},{\"lastTransitionTime\":\"2024-01-18T10:36:03Z\",\"lastUpdateTime\":\"2024-01-18T10:36:03Z\",\"message\":\"RolloutCompleted\",\"reason\":\"RolloutCompleted\",\"status\":\"False\",\"type\":\"Completed\"},{\"lastTransitionTime\":\"2024-01-18T13:19:14Z\",\"lastUpdateTime\":\"2024-01-18T13:19:14Z\",\"message\":\"Rollout has minimum availability\",\"reason\":\"AvailableReason\",\"status\":\"True\",\"type\":\"Available\"},{\"lastTransitionTime\":\"2024-01-18T13:19:35Z\",\"lastUpdateTime\":\"2024-01-18T13:19:35Z\",\"message\":\"Rollout is paused\",\"reason\":\"RolloutPaused\",\"status\":\"True\",\"type\":\"Paused\"},{\"lastTransitionTime\":\"2024-01-18T13:29:34Z\",\"lastUpdateTime\":\"2024-01-18T13:29:34Z\",\"message\":\"ReplicaSet \\\"application-internal-546856bb5c\\\" is progressing.\",\"reason\":\"ReplicaSetUpdated\",\"status\":\"True\",\"type\":\"Progressing\"}],\"controllerPause\":null,\"currentStepIndex\":2,\"message\":\"more replicas need to be updated\",\"pauseConditions\":null,\"phase\":\"Progressing\"}}" generation=757 namespace=app-namespace resourceVersion=4332528684 rollout=application-internal
2024-01-18 13:29:34.031 
time="2024-01-18T13:29:34Z" level=info msg="Rollout is resumed" event_reason=RolloutResumed namespace=app-namespace rollout=application-internal
2024-01-18 13:29:34.031 
time="2024-01-18T13:29:34Z" level=info msg="Event(v1.ObjectReference{Kind:\"Rollout\", Namespace:\"app-namespace\", Name:\"application-internal\", UID:\"54546450-62d4-4bb2-92f6-17a049ec3243\", APIVersion:\"argoproj.io/v1alpha1\", ResourceVersion:\"4332528684\", FieldPath:\"\"}): type: 'Normal' reason: 'RolloutResumed' Rollout is resumed"
2024-01-18 13:29:34.005 
time="2024-01-18T13:29:34Z" level=info msg="Event(v1.ObjectReference{Kind:\"Rollout\", Namespace:\"app-namespace\", Name:\"application-internal\", UID:\"54546450-62d4-4bb2-92f6-17a049ec3243\", APIVersion:\"argoproj.io/v1alpha1\", ResourceVersion:\"4332528684\", FieldPath:\"\"}): type: 'Normal' reason: 'RolloutStepCompleted' Rollout step 2/3 completed (pause: 10m)"
2024-01-18 13:29:34.004 
time="2024-01-18T13:29:34Z" level=info msg="Rollout step 2/3 completed (pause: 10m)" event_reason=RolloutStepCompleted namespace=app-namespace rollout=application-internal
2024-01-18 13:29:34.004 
time="2024-01-18T13:29:34Z" level=info msg="Rollout has waited the duration of the pause step" namespace=app-namespace rollout=application-internal
2024-01-18 13:29:34.004 
time="2024-01-18T13:29:34Z" level=info msg="Not finished reconciling Canary Pause" namespace=app-namespace rollout=application-internal
2024-01-18 13:29:34.004 
time="2024-01-18T13:29:34Z" level=info msg="Reconciling canary pause step (stepIndex: 1/3)" namespace=app-namespace rollout=application-internal
2024-01-18 13:29:34.004 
time="2024-01-18T13:29:34Z" level=info msg="No TrafficRouting Reconcilers found" namespace=app-namespace rollout=application-internal
2024-01-18 13:29:34.002 
time="2024-01-18T13:29:34Z" level=info msg="Started syncing rollout" generation=757 namespace=app-namespace resourceVersion=4332528684 rollout=application-internal
2024-01-18 13:19:35.057 
time="2024-01-18T13:19:35Z" level=info msg="Reconciliation completed" generation=757 namespace=app-namespace resourceVersion=4332528684 rollout=application-internal time_ms=3.540211
2024-01-18 13:19:35.057 
time="2024-01-18T13:19:35Z" level=info msg="No status changes. Skipping patch" generation=757 namespace=app-namespace resourceVersion=4332528684 rollout=application-internal
2024-01-18 13:19:35.057 
time="2024-01-18T13:19:35Z" level=info msg="Not finished reconciling Canary Pause" namespace=app-namespace rollout=application-internal
2024-01-18 13:19:35.057 
time="2024-01-18T13:19:35Z" level=info msg="Enqueueing Rollout in 9m58.942921491s seconds" namespace=app-namespace rollout=application-internal
2024-01-18 13:19:35.057 
time="2024-01-18T13:19:35Z" level=info msg="Reconciling canary pause step (stepIndex: 1/3)" namespace=app-namespace rollout=application-internal
2024-01-18 13:19:35.057 
time="2024-01-18T13:19:35Z" level=info msg="No TrafficRouting Reconcilers found" namespace=app-namespace rollout=application-internal
2024-01-18 13:19:35.055 
time="2024-01-18T13:19:35Z" level=info msg="Started syncing rollout" generation=757 namespace=app-namespace resourceVersion=4332528684 rollout=application-internal
2024-01-18 13:19:35.054 
time="2024-01-18T13:19:35Z" level=info msg="Processing completed" resource=app-namespace/application-internal
2024-01-18 13:19:35.054 
time="2024-01-18T13:19:35Z" level=info msg="Trigger on-rollout-aborted result: [{[0].2jmj7l5rSw0yVb_vlWAYkK_YBwk  [rollout-aborted] false}]" resource=app-namespace/application-internal
2024-01-18 13:19:35.054 
time="2024-01-18T13:19:35Z" level=info msg="Trigger on-analysis-run-failed result: [{[0].2jmj7l5rSw0yVb_vlWAYkK_YBwk  [analysis-run-failed] false}]" resource=app-namespace/application-internal
2024-01-18 13:19:35.054 
time="2024-01-18T13:19:35Z" level=info msg="Trigger on-analysis-run-error result: [{[0].2jmj7l5rSw0yVb_vlWAYkK_YBwk  [analysis-run-error] false}]" resource=app-namespace/application-internal
2024-01-18 13:19:35.053 
time="2024-01-18T13:19:35Z" level=info msg="Start processing" resource=app-namespace/application-internal
2024-01-18 13:19:35.053 
time="2024-01-18T13:19:35Z" level=info msg="Reconciliation completed" generation=757 namespace=app-namespace resourceVersion=4332528683 rollout=application-internal time_ms=32.764135
2024-01-18 13:19:35.053 
time="2024-01-18T13:19:35Z" level=info msg="persisted to informer" generation=757 namespace=app-namespace resourceVersion=4332528684 rollout=application-internal
2024-01-18 13:19:35.050 
time="2024-01-18T13:19:35Z" level=info msg="No status changes. Skipping patch" generation=757 namespace=app-namespace resourceVersion=4332528683 rollout=application-internal
2024-01-18 13:19:35.049 
time="2024-01-18T13:19:35Z" level=info msg="Not finished reconciling Canary Pause" namespace=app-namespace rollout=application-internal
2024-01-18 13:19:35.049 
time="2024-01-18T13:19:35Z" level=info msg="Enqueueing Rollout in 9m58.950816891s seconds" namespace=app-namespace rollout=application-internal
2024-01-18 13:19:35.049 
time="2024-01-18T13:19:35Z" level=info msg="Reconciling canary pause step (stepIndex: 1/3)" namespace=app-namespace rollout=application-internal
2024-01-18 13:19:35.049 
time="2024-01-18T13:19:35Z" level=info msg="No TrafficRouting Reconcilers found" namespace=app-namespace rollout=application-internal
2024-01-18 13:19:35.048 
time="2024-01-18T13:19:35Z" level=info msg="Patched conditions: {\"status\":{\"conditions\":[{\"lastTransitionTime\":\"2024-01-18T10:36:03Z\",\"lastUpdateTime\":\"2024-01-18T10:36:03Z\",\"message\":\"Rollout is not healthy\",\"reason\":\"RolloutHealthy\",\"status\":\"False\",\"type\":\"Healthy\"},{\"lastTransitionTime\":\"2024-01-18T10:36:03Z\",\"lastUpdateTime\":\"2024-01-18T10:36:03Z\",\"message\":\"RolloutCompleted\",\"reason\":\"RolloutCompleted\",\"status\":\"False\",\"type\":\"Completed\"},{\"lastTransitionTime\":\"2024-01-18T13:19:14Z\",\"lastUpdateTime\":\"2024-01-18T13:19:14Z\",\"message\":\"Rollout has minimum availability\",\"reason\":\"AvailableReason\",\"status\":\"True\",\"type\":\"Available\"},{\"lastTransitionTime\":\"2024-01-18T13:19:35Z\",\"lastUpdateTime\":\"2024-01-18T13:19:35Z\",\"message\":\"Rollout is paused\",\"reason\":\"RolloutPaused\",\"status\":\"Unknown\",\"type\":\"Progressing\"},{\"lastTransitionTime\":\"2024-01-18T13:19:35Z\",\"lastUpdateTime\":\"2024-01-18T13:19:35Z\",\"message\":\"Rollout is paused\",\"reason\":\"RolloutPaused\",\"status\":\"True\",\"type\":\"Paused\"}]}}" generation=757 namespace=app-namespace resourceVersion=4332528683 rollout=application-internal
2024-01-18 13:19:35.021 
time="2024-01-18T13:19:35Z" level=info msg="Started syncing rollout" generation=757 namespace=app-namespace resourceVersion=4332528683 rollout=application-internal
2024-01-18 13:19:35.020 
time="2024-01-18T13:19:35Z" level=info msg="Processing completed" resource=app-namespace/application-internal
2024-01-18 13:19:35.020 
time="2024-01-18T13:19:35Z" level=info msg="Trigger on-rollout-aborted result: [{[0].2jmj7l5rSw0yVb_vlWAYkK_YBwk  [rollout-aborted] false}]" resource=app-namespace/application-internal
2024-01-18 13:19:35.020 
time="2024-01-18T13:19:35Z" level=info msg="Trigger on-analysis-run-failed result: [{[0].2jmj7l5rSw0yVb_vlWAYkK_YBwk  [analysis-run-failed] false}]" resource=app-namespace/application-internal
2024-01-18 13:19:35.020 
time="2024-01-18T13:19:35Z" level=info msg="Trigger on-analysis-run-error result: [{[0].2jmj7l5rSw0yVb_vlWAYkK_YBwk  [analysis-run-error] false}]" resource=app-namespace/application-internal
2024-01-18 13:19:35.019 
time="2024-01-18T13:19:35Z" level=info msg="Start processing" resource=app-namespace/application-internal
2024-01-18 13:19:35.017 
time="2024-01-18T13:19:35Z" level=info msg="Reconciliation completed" generation=757 namespace=app-namespace resourceVersion=4332528681 rollout=application-internal time_ms=30.076992
2024-01-18 13:19:35.017 
time="2024-01-18T13:19:35Z" level=info msg="persisted to informer" generation=757 namespace=app-namespace resourceVersion=4332528683 rollout=application-internal
2024-01-18 13:19:35.014 
time="2024-01-18T13:19:35Z" level=info msg="Event(v1.ObjectReference{Kind:\"Rollout\", Namespace:\"app-namespace\", Name:\"application-internal\", UID:\"54546450-62d4-4bb2-92f6-17a049ec3243\", APIVersion:\"argoproj.io/v1alpha1\", ResourceVersion:\"4332528681\", FieldPath:\"\"}): type: 'Normal' reason: 'RolloutPaused' Rollout is paused (CanaryPauseStep)"
2024-01-18 13:19:35.014 
time="2024-01-18T13:19:35Z" level=info msg="Patched: {\"status\":{\"controllerPause\":true,\"message\":\"CanaryPauseStep\",\"pauseConditions\":[{\"reason\":\"CanaryPauseStep\",\"startTime\":\"2024-01-18T13:19:34Z\"}],\"phase\":\"Paused\"}}" generation=757 namespace=app-namespace resourceVersion=4332528681 rollout=application-internal
2024-01-18 13:19:35.014 
time="2024-01-18T13:19:35Z" level=info msg="Rollout is paused (CanaryPauseStep)" event_reason=RolloutPaused namespace=app-namespace rollout=application-internal
2024-01-18 13:19:34.990 
time="2024-01-18T13:19:34Z" level=info msg="Adding pause reason CanaryPauseStep with start time 2024-01-18T13:19:34Z" namespace=app-namespace rollout=application-internal
2024-01-18 13:19:34.990 
time="2024-01-18T13:19:34Z" level=info msg="Not finished reconciling Canary Pause" namespace=app-namespace rollout=application-internal
2024-01-18 13:19:34.990 
time="2024-01-18T13:19:34Z" level=info msg="Reconciling canary pause step (stepIndex: 1/3)" namespace=app-namespace rollout=application-internal
2024-01-18 13:19:34.990 
time="2024-01-18T13:19:34Z" level=info msg="No TrafficRouting Reconcilers found" namespace=app-namespace rollout=application-internal
2024-01-18 13:19:34.988 
time="2024-01-18T13:19:34Z" level=info msg="Started syncing rollout" generation=757 namespace=app-namespace resourceVersion=4332528681 rollout=application-internal
2024-01-18 13:19:34.986 
time="2024-01-18T13:19:34Z" level=info msg="Processing completed" resource=app-namespace/application-internal
2024-01-18 13:19:34.986 
time="2024-01-18T13:19:34Z" level=info msg="Trigger on-analysis-run-error result: [{[0].2jmj7l5rSw0yVb_vlWAYkK_YBwk  [analysis-run-error] false}]" resource=app-namespace/application-internal
2024-01-18 13:19:34.986 
time="2024-01-18T13:19:34Z" level=info msg="Trigger on-rollout-aborted result: [{[0].2jmj7l5rSw0yVb_vlWAYkK_YBwk  [rollout-aborted] false}]" resource=app-namespace/application-internal
2024-01-18 13:19:34.986 
time="2024-01-18T13:19:34Z" level=info msg="Trigger on-analysis-run-failed result: [{[0].2jmj7l5rSw0yVb_vlWAYkK_YBwk  [analysis-run-failed] false}]" resource=app-namespace/application-internal
2024-01-18 13:19:34.985 
time="2024-01-18T13:19:34Z" level=info msg="Start processing" resource=app-namespace/application-internal
2024-01-18 13:19:34.983 
time="2024-01-18T13:19:34Z" level=info msg="Reconciliation completed" generation=757 namespace=app-namespace resourceVersion=4332527495 rollout=application-internal time_ms=32.457277
2024-01-18 13:19:34.983 
time="2024-01-18T13:19:34Z" level=info msg="persisted to informer" generation=757 namespace=app-namespace resourceVersion=4332528681 rollout=application-internal
2024-01-18 13:19:34.980 
time="2024-01-18T13:19:34Z" level=info msg="Patched: {\"status\":{\"availableReplicas\":5,\"conditions\":[{\"lastTransitionTime\":\"2024-01-18T10:36:03Z\",\"lastUpdateTime\":\"2024-01-18T10:36:03Z\",\"message\":\"Rollout is not healthy\",\"reason\":\"RolloutHealthy\",\"status\":\"False\",\"type\":\"Healthy\"},{\"lastTransitionTime\":\"2024-01-18T10:36:03Z\",\"lastUpdateTime\":\"2024-01-18T10:36:03Z\",\"message\":\"RolloutCompleted\",\"reason\":\"RolloutCompleted\",\"status\":\"False\",\"type\":\"Completed\"},{\"lastTransitionTime\":\"2024-01-18T10:49:24Z\",\"lastUpdateTime\":\"2024-01-18T10:49:24Z\",\"message\":\"Rollout is paused\",\"reason\":\"RolloutPaused\",\"status\":\"False\",\"type\":\"Paused\"},{\"lastTransitionTime\":\"2024-01-18T13:19:14Z\",\"lastUpdateTime\":\"2024-01-18T13:19:14Z\",\"message\":\"Rollout has minimum availability\",\"reason\":\"AvailableReason\",\"status\":\"True\",\"type\":\"Available\"},{\"lastTransitionTime\":\"2024-01-18T10:49:24Z\",\"lastUpdateTime\":\"2024-01-18T13:19:34Z\",\"message\":\"ReplicaSet \\\"application-internal-546856bb5c\\\" is progressing.\",\"reason\":\"ReplicaSetUpdated\",\"status\":\"True\",\"type\":\"Progressing\"}],\"currentStepIndex\":1,\"readyReplicas\":5}}" generation=757 namespace=app-namespace resourceVersion=4332527495 rollout=application-internal
2024-01-18 13:19:34.954 
time="2024-01-18T13:19:34Z" level=info msg="Event(v1.ObjectReference{Kind:\"Rollout\", Namespace:\"app-namespace\", Name:\"application-internal\", UID:\"54546450-62d4-4bb2-92f6-17a049ec3243\", APIVersion:\"argoproj.io/v1alpha1\", ResourceVersion:\"4332527495\", FieldPath:\"\"}): type: 'Normal' reason: 'RolloutStepCompleted' Rollout step 1/3 completed (setWeight: 10)"
2024-01-18 13:19:34.954 
time="2024-01-18T13:19:34Z" level=info msg="Rollout step 1/3 completed (setWeight: 10)" event_reason=RolloutStepCompleted namespace=app-namespace rollout=application-internal
2024-01-18 13:19:34.953 
time="2024-01-18T13:19:34Z" level=info msg="No TrafficRouting Reconcilers found" namespace=app-namespace rollout=application-internal
2024-01-18 13:19:34.951 
time="2024-01-18T13:19:34Z" level=info msg="Started syncing rollout" generation=757 namespace=app-namespace resourceVersion=4332527495 rollout=application-internal
2024-01-18 13:19:34.949 
time="2024-01-18T13:19:34Z" level=info msg="Enqueueing parent of app-namespace/application-internal-546856bb5c: Rollout app-namespace/application-internal"
2024-01-18 13:19:14.645 
time="2024-01-18T13:19:14Z" level=info msg="Reconciliation completed" generation=757 namespace=app-namespace resourceVersion=4332527495 rollout=application-internal time_ms=4.096301
2024-01-18 13:19:14.645 
time="2024-01-18T13:19:14Z" level=info msg="Queueing up rollout for a progress after 599s" namespace=app-namespace rollout=application-internal
2024-01-18 13:19:14.645 
time="2024-01-18T13:19:14Z" level=info msg="No status changes. Skipping patch" generation=757 namespace=app-namespace resourceVersion=4332527495 rollout=application-internal
2024-01-18 13:19:14.645 
time="2024-01-18T13:19:14Z" level=info msg="Timed out (false) [last progress check: 2024-01-18 13:19:14 +0000 UTC - now: 2024-01-18 13:19:14.645243107 +0000 UTC m=+47.412137153]" namespace=app-namespace rollout=application-internal
2024-01-18 13:19:14.645 
time="2024-01-18T13:19:14Z" level=info msg="No TrafficRouting Reconcilers found" namespace=app-namespace rollout=application-internal
2024-01-18 13:19:14.645 
time="2024-01-18T13:19:14Z" level=info msg="Started syncing rollout" generation=757 namespace=app-namespace resourceVersion=4332527495 rollout=application-internal
2024-01-18 13:19:14.645 
time="2024-01-18T13:19:14Z" level=info msg="Processing completed" resource=app-namespace/application-internal
2024-01-18 13:19:14.645 
time="2024-01-18T13:19:14Z" level=info msg="Trigger on-rollout-aborted result: [{[0].2jmj7l5rSw0yVb_vlWAYkK_YBwk  [rollout-aborted] false}]" resource=app-namespace/application-internal
2024-01-18 13:19:14.645 
time="2024-01-18T13:19:14Z" level=info msg="Trigger on-analysis-run-failed result: [{[0].2jmj7l5rSw0yVb_vlWAYkK_YBwk  [analysis-run-failed] false}]" resource=app-namespace/application-internal
2024-01-18 13:19:14.641 
time="2024-01-18T13:19:14Z" level=info msg="Trigger on-analysis-run-error result: [{[0].2jmj7l5rSw0yVb_vlWAYkK_YBwk  [analysis-run-error] false}]" resource=app-namespace/application-internal
2024-01-18 13:19:14.639 
time="2024-01-18T13:19:14Z" level=info msg="Start processing" resource=app-namespace/application-internal
2024-01-18 13:19:14.638 
time="2024-01-18T13:19:14Z" level=info msg="Reconciliation completed" generation=757 namespace=app-namespace resourceVersion=4332527479 rollout=application-internal time_ms=31.419850999999998
2024-01-18 13:19:14.637 
time="2024-01-18T13:19:14Z" level=info msg="persisted to informer" generation=757 namespace=app-namespace resourceVersion=4332527495 rollout=application-internal
2024-01-18 13:19:14.635 
time="2024-01-18T13:19:14Z" level=info msg="Patched: {\"status\":{\"HPAReplicas\":5,\"replicas\":5,\"updatedReplicas\":1}}" generation=757 namespace=app-namespace resourceVersion=4332527479 rollout=application-internal
2024-01-18 13:19:14.609 
time="2024-01-18T13:19:14Z" level=info msg="No TrafficRouting Reconcilers found" namespace=app-namespace rollout=application-internal
2024-01-18 13:19:14.607 
time="2024-01-18T13:19:14Z" level=info msg="Started syncing rollout" generation=757 namespace=app-namespace resourceVersion=4332527479 rollout=application-internal
2024-01-18 13:19:14.605 
time="2024-01-18T13:19:14Z" level=info msg="Enqueueing parent of app-namespace/application-internal-546856bb5c: Rollout app-namespace/application-internal"
2024-01-18 13:19:14.596 
time="2024-01-18T13:19:14Z" level=info msg="Reconciliation completed" generation=757 namespace=app-namespace resourceVersion=4332527479 rollout=application-internal time_ms=4.866942
2024-01-18 13:19:14.596 
time="2024-01-18T13:19:14Z" level=info msg="Queueing up rollout for a progress after 599s" namespace=app-namespace rollout=application-internal
2024-01-18 13:19:14.596 
time="2024-01-18T13:19:14Z" level=info msg="No status changes. Skipping patch" generation=757 namespace=app-namespace resourceVersion=4332527479 rollout=application-internal
2024-01-18 13:19:14.596 
time="2024-01-18T13:19:14Z" level=info msg="Timed out (false) [last progress check: 2024-01-18 13:19:14 +0000 UTC - now: 2024-01-18 13:19:14.59621089 +0000 UTC m=+47.363104937]" namespace=app-namespace rollout=application-internal
2024-01-18 13:19:14.595 
time="2024-01-18T13:19:14Z" level=info msg="No TrafficRouting Reconcilers found" namespace=app-namespace rollout=application-internal
2024-01-18 13:19:14.592 
time="2024-01-18T13:19:14Z" level=info msg="Started syncing rollout" generation=757 namespace=app-namespace resourceVersion=4332527479 rollout=application-internal
2024-01-18 13:19:14.590 
time="2024-01-18T13:19:14Z" level=info msg="Enqueueing parent of app-namespace/application-internal-546856bb5c: Rollout app-namespace/application-internal"
2024-01-18 13:19:14.501 
time="2024-01-18T13:19:14Z" level=info msg="Reconciliation completed" generation=757 namespace=app-namespace resourceVersion=4332527479 rollout=application-internal time_ms=4.229297
2024-01-18 13:19:14.501 
time="2024-01-18T13:19:14Z" level=info msg="Queueing up rollout for a progress after 599s" namespace=app-namespace rollout=application-internal
2024-01-18 13:19:14.501 
time="2024-01-18T13:19:14Z" level=info msg="No status changes. Skipping patch" generation=757 namespace=app-namespace resourceVersion=4332527479 rollout=application-internal
2024-01-18 13:19:14.500 
time="2024-01-18T13:19:14Z" level=info msg="Timed out (false) [last progress check: 2024-01-18 13:19:14 +0000 UTC - now: 2024-01-18 13:19:14.500751256 +0000 UTC m=+47.267645313]" namespace=app-namespace rollout=application-internal
2024-01-18 13:19:14.500 
time="2024-01-18T13:19:14Z" level=info msg="No TrafficRouting Reconcilers found" namespace=app-namespace rollout=application-internal
2024-01-18 13:19:14.497 
time="2024-01-18T13:19:14Z" level=info msg="Started syncing rollout" generation=757 namespace=app-namespace resourceVersion=4332527479 rollout=application-internal
2024-01-18 13:19:14.496 
time="2024-01-18T13:19:14Z" level=info msg="Processing completed" resource=app-namespace/application-internal
2024-01-18 13:19:14.496 
time="2024-01-18T13:19:14Z" level=info msg="Trigger on-rollout-aborted result: [{[0].2jmj7l5rSw0yVb_vlWAYkK_YBwk  [rollout-aborted] false}]" resource=app-namespace/application-internal
2024-01-18 13:19:14.496 
time="2024-01-18T13:19:14Z" level=info msg="Trigger on-analysis-run-failed result: [{[0].2jmj7l5rSw0yVb_vlWAYkK_YBwk  [analysis-run-failed] false}]" resource=app-namespace/application-internal
2024-01-18 13:19:14.496 
time="2024-01-18T13:19:14Z" level=info msg="Trigger on-analysis-run-error result: [{[0].2jmj7l5rSw0yVb_vlWAYkK_YBwk  [analysis-run-error] false}]" resource=app-namespace/application-internal
2024-01-18 13:19:14.495 
time="2024-01-18T13:19:14Z" level=info msg="Start processing" resource=app-namespace/application-internal
2024-01-18 13:19:14.493 
time="2024-01-18T13:19:14Z" level=info msg="Reconciliation completed" generation=757 namespace=app-namespace resourceVersion=4332527470 rollout=application-internal time_ms=34.378069999999994
2024-01-18 13:19:14.493 
time="2024-01-18T13:19:14Z" level=info msg="persisted to informer" generation=757 namespace=app-namespace resourceVersion=4332527479 rollout=application-internal
2024-01-18 13:19:14.490 
time="2024-01-18T13:19:14Z" level=info msg="Patched: {\"status\":{\"HPAReplicas\":4,\"availableReplicas\":4,\"readyReplicas\":4,\"replicas\":4}}" generation=757 namespace=app-namespace resourceVersion=4332527470 rollout=application-internal
2024-01-18 13:19:14.462 
time="2024-01-18T13:19:14Z" level=info msg="No TrafficRouting Reconcilers found" namespace=app-namespace rollout=application-internal
2024-01-18 13:19:14.460 
time="2024-01-18T13:19:14Z" level=info msg="Started syncing rollout" generation=757 namespace=app-namespace resourceVersion=4332527470 rollout=application-internal
2024-01-18 13:19:14.458 
time="2024-01-18T13:19:14Z" level=info msg="Reconciliation completed" generation=757 namespace=app-namespace resourceVersion=4332527470 rollout=application-internal time_ms=36.834595
2024-01-18 13:19:14.458 
time="2024-01-18T13:19:14Z" level=info msg="Queueing up rollout for a progress after 599s" namespace=app-namespace rollout=application-internal
2024-01-18 13:19:14.458 
time="2024-01-18T13:19:14Z" level=info msg="No status changes. Skipping patch" generation=757 namespace=app-namespace resourceVersion=4332527470 rollout=application-internal
2024-01-18 13:19:14.458 
time="2024-01-18T13:19:14Z" level=info msg="Timed out (false) [last progress check: 2024-01-18 13:19:14 +0000 UTC - now: 2024-01-18 13:19:14.458447655 +0000 UTC m=+47.225341701]" namespace=app-namespace rollout=application-internal
2024-01-18 13:19:14.457 
time="2024-01-18T13:19:14Z" level=info msg="Event(v1.ObjectReference{Kind:\"Rollout\", Namespace:\"app-namespace\", Name:\"application-internal\", UID:\"54546450-62d4-4bb2-92f6-17a049ec3243\", APIVersion:\"argoproj.io/v1alpha1\", ResourceVersion:\"4332527470\", FieldPath:\"\"}): type: 'Normal' reason: 'ScalingReplicaSet' Scaled up ReplicaSet application-internal-546856bb5c (revision 717) from 0 to 1"
2024-01-18 13:19:14.457 
time="2024-01-18T13:19:14Z" level=info msg="Not finished reconciling ReplicaSets" namespace=app-namespace rollout=application-internal
2024-01-18 13:19:14.457 
time="2024-01-18T13:19:14Z" level=info msg="Not finished reconciling new ReplicaSet 'application-internal-546856bb5c'" namespace=app-namespace rollout=application-internal
2024-01-18 13:19:14.457 
time="2024-01-18T13:19:14Z" level=info msg="Scaled up ReplicaSet application-internal-546856bb5c (revision 717) from 0 to 1" event_reason=ScalingReplicaSet namespace=app-namespace rollout=application-internal
2024-01-18 13:19:14.435 
time="2024-01-18T13:19:14Z" level=info msg="Enqueueing parent of app-namespace/application-internal-575b7d5fc6: Rollout app-namespace/application-internal"
2024-01-18 13:19:14.433 
time="2024-01-18T13:19:14Z" level=info msg="No TrafficRouting Reconcilers found" namespace=app-namespace rollout=application-internal
2024-01-18 13:19:14.430 
time="2024-01-18T13:19:14Z" level=info msg="Processing completed" resource=app-namespace/application-internal
2024-01-18 13:19:14.430 
time="2024-01-18T13:19:14Z" level=info msg="Trigger on-rollout-aborted result: [{[0].2jmj7l5rSw0yVb_vlWAYkK_YBwk  [rollout-aborted] false}]" resource=app-namespace/application-internal
2024-01-18 13:19:14.430 
time="2024-01-18T13:19:14Z" level=info msg="Trigger on-analysis-run-failed result: [{[0].2jmj7l5rSw0yVb_vlWAYkK_YBwk  [analysis-run-failed] false}]" resource=app-namespace/application-internal
2024-01-18 13:19:14.430 
time="2024-01-18T13:19:14Z" level=info msg="Trigger on-analysis-run-error result: [{[0].2jmj7l5rSw0yVb_vlWAYkK_YBwk  [analysis-run-error] false}]" resource=app-namespace/application-internal
2024-01-18 13:19:14.429 
time="2024-01-18T13:19:14Z" level=info msg="Started syncing rollout" generation=757 namespace=app-namespace resourceVersion=4332527470 rollout=application-internal
2024-01-18 13:19:14.427 
time="2024-01-18T13:19:14Z" level=info msg="Start processing" resource=app-namespace/application-internal
2024-01-18 13:19:14.421 
time="2024-01-18T13:19:14Z" level=info msg="Reconciliation completed" generation=757 namespace=app-namespace resourceVersion=4332527452 rollout=application-internal time_ms=66.490968
2024-01-18 13:19:14.421 
time="2024-01-18T13:19:14Z" level=info msg="persisted to informer" generation=757 namespace=app-namespace resourceVersion=4332527470 rollout=application-internal
2024-01-18 13:19:14.419 
time="2024-01-18T13:19:14Z" level=info msg="Patched: {\"status\":{\"availableReplicas\":5,\"conditions\":[{\"lastTransitionTime\":\"2024-01-18T10:36:03Z\",\"lastUpdateTime\":\"2024-01-18T10:36:03Z\",\"message\":\"Rollout is not healthy\",\"reason\":\"RolloutHealthy\",\"status\":\"False\",\"type\":\"Healthy\"},{\"lastTransitionTime\":\"2024-01-18T10:36:03Z\",\"lastUpdateTime\":\"2024-01-18T10:36:03Z\",\"message\":\"RolloutCompleted\",\"reason\":\"RolloutCompleted\",\"status\":\"False\",\"type\":\"Completed\"},{\"lastTransitionTime\":\"2024-01-18T10:49:24Z\",\"lastUpdateTime\":\"2024-01-18T10:49:24Z\",\"message\":\"Rollout is paused\",\"reason\":\"RolloutPaused\",\"status\":\"False\",\"type\":\"Paused\"},{\"lastTransitionTime\":\"2024-01-18T13:19:14Z\",\"lastUpdateTime\":\"2024-01-18T13:19:14Z\",\"message\":\"Rollout has minimum availability\",\"reason\":\"AvailableReason\",\"status\":\"True\",\"type\":\"Available\"},{\"lastTransitionTime\":\"2024-01-18T10:49:24Z\",\"lastUpdateTime\":\"2024-01-18T13:19:14Z\",\"message\":\"ReplicaSet \\\"application-internal-546856bb5c\\\" is progressing.\",\"reason\":\"ReplicaSetUpdated\",\"status\":\"True\",\"type\":\"Progressing\"}],\"readyReplicas\":5}}" generation=757 namespace=app-namespace resourceVersion=4332527452 rollout=application-internal
2024-01-18 13:19:14.419 
time="2024-01-18T13:19:14Z" level=info msg="Enqueueing parent of app-namespace/application-internal-575b7d5fc6: Rollout app-namespace/application-internal"
2024-01-18 13:19:14.387 
time="2024-01-18T13:19:14Z" level=info msg="Event(v1.ObjectReference{Kind:\"Rollout\", Namespace:\"app-namespace\", Name:\"application-internal\", UID:\"54546450-62d4-4bb2-92f6-17a049ec3243\", APIVersion:\"argoproj.io/v1alpha1\", ResourceVersion:\"4332527452\", FieldPath:\"\"}): type: 'Normal' reason: 'ScalingReplicaSet' Scaled down ReplicaSet application-internal-575b7d5fc6 (revision 715) from 1 to 0"
2024-01-18 13:19:14.387 
time="2024-01-18T13:19:14Z" level=info msg="Not finished reconciling ReplicaSets" namespace=app-namespace rollout=application-internal
2024-01-18 13:19:14.387 
time="2024-01-18T13:19:14Z" level=info msg="Not finished reconciling old ReplicaSets" namespace=app-namespace rollout=application-internal
2024-01-18 13:19:14.387 
time="2024-01-18T13:19:14Z" level=info msg="Scaled down old RSes" namespace=app-namespace rollout=application-internal
2024-01-18 13:19:14.387 
time="2024-01-18T13:19:14Z" level=info msg="Scaled down ReplicaSet application-internal-575b7d5fc6 (revision 715) from 1 to 0" event_reason=ScalingReplicaSet namespace=app-namespace rollout=application-internal
2024-01-18 13:19:14.358 
time="2024-01-18T13:19:14Z" level=info msg="Found 5 available pods, scaling down old RSes (minAvailable: 4, maxScaleDown: 1)" namespace=app-namespace rollout=application-internal
2024-01-18 13:19:14.358 
time="2024-01-18T13:19:14Z" level=info msg="Found 1 available pods in old RS app-namespace/application-internal-575b7d5fc6" namespace=app-namespace rollout=application-internal
2024-01-18 13:19:14.358 
time="2024-01-18T13:19:14Z" level=info msg="Reconciling 1 old ReplicaSets (total pods: 1)" namespace=app-namespace rollout=application-internal
2024-01-18 13:19:14.358 
time="2024-01-18T13:19:14Z" level=info msg="No TrafficRouting Reconcilers found" namespace=app-namespace rollout=application-internal
2024-01-18 13:19:14.356 
time="2024-01-18T13:19:14Z" level=info msg="Started syncing rollout" generation=757 namespace=app-namespace resourceVersion=4332527452 rollout=application-internal
2024-01-18 13:19:14.354 
time="2024-01-18T13:19:14Z" level=info msg="Enqueueing parent of app-namespace/application-internal-6fc65dcfc6: Rollout app-namespace/application-internal"
2024-01-18 13:19:14.197 
time="2024-01-18T13:19:14Z" level=info msg="Reconciliation completed" generation=757 namespace=app-namespace resourceVersion=4332527452 rollout=application-internal time_ms=4.852804
2024-01-18 13:19:14.197 
time="2024-01-18T13:19:14Z" level=info msg="Queueing up rollout for a progress after 599s" namespace=app-namespace rollout=application-internal
2024-01-18 13:19:14.197 
time="2024-01-18T13:19:14Z" level=info msg="No status changes. Skipping patch" generation=757 namespace=app-namespace resourceVersion=4332527452 rollout=application-internal
2024-01-18 13:19:14.196 
time="2024-01-18T13:19:14Z" level=info msg="Timed out (false) [last progress check: 2024-01-18 13:19:14 +0000 UTC - now: 2024-01-18 13:19:14.196772686 +0000 UTC m=+46.963666732]" namespace=app-namespace rollout=application-internal
2024-01-18 13:19:14.196 
time="2024-01-18T13:19:14Z" level=info msg="Found 1 available pods in old RS app-namespace/application-internal-575b7d5fc6" namespace=app-namespace rollout=application-internal
2024-01-18 13:19:14.196 
time="2024-01-18T13:19:14Z" level=info msg="Reconciling 1 old ReplicaSets (total pods: 1)" namespace=app-namespace rollout=application-internal
2024-01-18 13:19:14.196 
time="2024-01-18T13:19:14Z" level=info msg="No TrafficRouting Reconcilers found" namespace=app-namespace rollout=application-internal
2024-01-18 13:19:14.193 
time="2024-01-18T13:19:14Z" level=info msg="Started syncing rollout" generation=757 namespace=app-namespace resourceVersion=4332527452 rollout=application-internal
2024-01-18 13:19:14.192 
time="2024-01-18T13:19:14Z" level=info msg="Processing completed" resource=app-namespace/application-internal
2024-01-18 13:19:14.192 
time="2024-01-18T13:19:14Z" level=info msg="Trigger on-analysis-run-error result: [{[0].2jmj7l5rSw0yVb_vlWAYkK_YBwk  [analysis-run-error] false}]" resource=app-namespace/application-internal
2024-01-18 13:19:14.192 
time="2024-01-18T13:19:14Z" level=info msg="Trigger on-rollout-aborted result: [{[0].2jmj7l5rSw0yVb_vlWAYkK_YBwk  [rollout-aborted] false}]" resource=app-namespace/application-internal
2024-01-18 13:19:14.192 
time="2024-01-18T13:19:14Z" level=info msg="Trigger on-analysis-run-failed result: [{[0].2jmj7l5rSw0yVb_vlWAYkK_YBwk  [analysis-run-failed] false}]" resource=app-namespace/application-internal
2024-01-18 13:19:14.191 
time="2024-01-18T13:19:14Z" level=info msg="Start processing" resource=app-namespace/application-internal
2024-01-18 13:19:14.191 
time="2024-01-18T13:19:14Z" level=info msg="Reconciliation completed" generation=757 namespace=app-namespace resourceVersion=4332525746 rollout=application-internal time_ms=37.918701999999996
2024-01-18 13:19:14.191 
time="2024-01-18T13:19:14Z" level=info msg="persisted to informer" generation=757 namespace=app-namespace resourceVersion=4332527452 rollout=application-internal
2024-01-18 13:19:14.187 
time="2024-01-18T13:19:14Z" level=info msg="Patched: {\"status\":{\"availableReplicas\":4,\"conditions\":[{\"lastTransitionTime\":\"2024-01-18T10:36:03Z\",\"lastUpdateTime\":\"2024-01-18T10:36:03Z\",\"message\":\"Rollout is not healthy\",\"reason\":\"RolloutHealthy\",\"status\":\"False\",\"type\":\"Healthy\"},{\"lastTransitionTime\":\"2024-01-18T10:36:03Z\",\"lastUpdateTime\":\"2024-01-18T10:36:03Z\",\"message\":\"RolloutCompleted\",\"reason\":\"RolloutCompleted\",\"status\":\"False\",\"type\":\"Completed\"},{\"lastTransitionTime\":\"2024-01-18T10:49:24Z\",\"lastUpdateTime\":\"2024-01-18T10:49:24Z\",\"message\":\"Rollout is paused\",\"reason\":\"RolloutPaused\",\"status\":\"False\",\"type\":\"Paused\"},{\"lastTransitionTime\":\"2024-01-18T10:49:24Z\",\"lastUpdateTime\":\"2024-01-18T13:19:14Z\",\"message\":\"ReplicaSet \\\"application-internal-546856bb5c\\\" is progressing.\",\"reason\":\"ReplicaSetUpdated\",\"status\":\"True\",\"type\":\"Progressing\"},{\"lastTransitionTime\":\"2024-01-18T13:19:14Z\",\"lastUpdateTime\":\"2024-01-18T13:19:14Z\",\"message\":\"Rollout has minimum availability\",\"reason\":\"AvailableReason\",\"status\":\"True\",\"type\":\"Available\"}],\"readyReplicas\":4}}" generation=757 namespace=app-namespace resourceVersion=4332525746 rollout=application-internal
2024-01-18 13:19:14.156 
time="2024-01-18T13:19:14Z" level=info msg="Found 1 available pods in old RS app-namespace/application-internal-575b7d5fc6" namespace=app-namespace rollout=application-internal
2024-01-18 13:19:14.156 
time="2024-01-18T13:19:14Z" level=info msg="Reconciling 1 old ReplicaSets (total pods: 1)" namespace=app-namespace rollout=application-internal
2024-01-18 13:19:14.156 
time="2024-01-18T13:19:14Z" level=info msg="No TrafficRouting Reconcilers found" namespace=app-namespace rollout=application-internal
2024-01-18 13:19:14.154 
time="2024-01-18T13:19:14Z" level=info msg="Started syncing rollout" generation=757 namespace=app-namespace resourceVersion=4332525746 rollout=application-internal
2024-01-18 13:19:14.152 
time="2024-01-18T13:19:14Z" level=info msg="Enqueueing parent of app-namespace/application-internal-6fc65dcfc6: Rollout app-namespace/application-internal"
2024-01-18 13:18:43.851 
time="2024-01-18T13:18:43Z" level=info msg="Reconciliation completed" generation=757 namespace=app-namespace resourceVersion=4332525746 rollout=application-internal time_ms=3.6966400000000004
2024-01-18 13:18:43.851 
time="2024-01-18T13:18:43Z" level=info msg="Queueing up rollout for a progress after 599s" namespace=app-namespace rollout=application-internal
2024-01-18 13:18:43.851 
time="2024-01-18T13:18:43Z" level=info msg="No status changes. Skipping patch" generation=757 namespace=app-namespace resourceVersion=4332525746 rollout=application-internal
2024-01-18 13:18:43.850 
time="2024-01-18T13:18:43Z" level=info msg="Timed out (false) [last progress check: 2024-01-18 13:18:43 +0000 UTC - now: 2024-01-18 13:18:43.850827589 +0000 UTC m=+16.617721636]" namespace=app-namespace rollout=application-internal
2024-01-18 13:18:43.850 
time="2024-01-18T13:18:43Z" level=info msg="Found 1 available pods in old RS app-namespace/application-internal-575b7d5fc6" namespace=app-namespace rollout=application-internal
2024-01-18 13:18:43.850 
time="2024-01-18T13:18:43Z" level=info msg="Reconciling 1 old ReplicaSets (total pods: 1)" namespace=app-namespace rollout=application-internal
2024-01-18 13:18:43.850 
time="2024-01-18T13:18:43Z" level=info msg="No TrafficRouting Reconcilers found" namespace=app-namespace rollout=application-internal
2024-01-18 13:18:43.848 
time="2024-01-18T13:18:43Z" level=info msg="Started syncing rollout" generation=757 namespace=app-namespace resourceVersion=4332525746 rollout=application-internal
2024-01-18 13:18:43.847 
time="2024-01-18T13:18:43Z" level=info msg="Processing completed" resource=app-namespace/application-internal
2024-01-18 13:18:43.847 
time="2024-01-18T13:18:43Z" level=info msg="Trigger on-rollout-aborted result: [{[0].2jmj7l5rSw0yVb_vlWAYkK_YBwk  [rollout-aborted] false}]" resource=app-namespace/application-internal
2024-01-18 13:18:43.846 
time="2024-01-18T13:18:43Z" level=info msg="Trigger on-analysis-run-failed result: [{[0].2jmj7l5rSw0yVb_vlWAYkK_YBwk  [analysis-run-failed] false}]" resource=app-namespace/application-internal
2024-01-18 13:18:43.846 
time="2024-01-18T13:18:43Z" level=info msg="Trigger on-analysis-run-error result: [{[0].2jmj7l5rSw0yVb_vlWAYkK_YBwk  [analysis-run-error] false}]" resource=app-namespace/application-internal
2024-01-18 13:18:43.845 
time="2024-01-18T13:18:43Z" level=info msg="Start processing" resource=app-namespace/application-internal
2024-01-18 13:18:43.843 
time="2024-01-18T13:18:43Z" level=info msg="Reconciliation completed" generation=757 namespace=app-namespace resourceVersion=4332525723 rollout=application-internal time_ms=35.801431
2024-01-18 13:18:43.843 
time="2024-01-18T13:18:43Z" level=info msg="persisted to informer" generation=757 namespace=app-namespace resourceVersion=4332525746 rollout=application-internal
2024-01-18 13:18:43.840 
time="2024-01-18T13:18:43Z" level=info msg="Patched: {\"status\":{\"HPAReplicas\":5,\"replicas\":5}}" generation=757 namespace=app-namespace resourceVersion=4332525723 rollout=application-internal
2024-01-18 13:18:43.815 
time="2024-01-18T13:18:43Z" level=info msg="Timed out (false) [last progress check: 2024-01-18 13:18:43 +0000 UTC - now: 2024-01-18 13:18:43.814986071 +0000 UTC m=+16.581880124]" namespace=app-namespace rollout=application-internal
2024-01-18 13:18:43.814 
time="2024-01-18T13:18:43Z" level=info msg="Found 1 available pods in old RS app-namespace/application-internal-575b7d5fc6" namespace=app-namespace rollout=application-internal
2024-01-18 13:18:43.814 
time="2024-01-18T13:18:43Z" level=info msg="Reconciling 1 old ReplicaSets (total pods: 1)" namespace=app-namespace rollout=application-internal
2024-01-18 13:18:43.814 
time="2024-01-18T13:18:43Z" level=info msg="No TrafficRouting Reconcilers found" namespace=app-namespace rollout=application-internal
2024-01-18 13:18:43.808 
time="2024-01-18T13:18:43Z" level=info msg="Started syncing rollout" generation=757 namespace=app-namespace resourceVersion=4332525723 rollout=application-internal
2024-01-18 13:18:43.806 
time="2024-01-18T13:18:43Z" level=info msg="Enqueueing parent of app-namespace/application-internal-6fc65dcfc6: Rollout app-namespace/application-internal"
2024-01-18 13:18:43.798 
time="2024-01-18T13:18:43Z" level=info msg="Reconciliation completed" generation=757 namespace=app-namespace resourceVersion=4332525723 rollout=application-internal time_ms=8.392495
2024-01-18 13:18:43.798 
time="2024-01-18T13:18:43Z" level=info msg="Queueing up rollout for a progress after 599s" namespace=app-namespace rollout=application-internal
2024-01-18 13:18:43.798 
time="2024-01-18T13:18:43Z" level=info msg="No status changes. Skipping patch" generation=757 namespace=app-namespace resourceVersion=4332525723 rollout=application-internal
2024-01-18 13:18:43.798 
time="2024-01-18T13:18:43Z" level=info msg="Timed out (false) [last progress check: 2024-01-18 13:18:43 +0000 UTC - now: 2024-01-18 13:18:43.798097226 +0000 UTC m=+16.564991302]" namespace=app-namespace rollout=application-internal
2024-01-18 13:18:43.796 
time="2024-01-18T13:18:43Z" level=info msg="Found 1 available pods in old RS app-namespace/application-internal-575b7d5fc6" namespace=app-namespace rollout=application-internal
2024-01-18 13:18:43.796 
time="2024-01-18T13:18:43Z" level=info msg="Reconciling 1 old ReplicaSets (total pods: 1)" namespace=app-namespace rollout=application-internal
2024-01-18 13:18:43.796 
time="2024-01-18T13:18:43Z" level=info msg="No TrafficRouting Reconcilers found" namespace=app-namespace rollout=application-internal
2024-01-18 13:18:43.792 
time="2024-01-18T13:18:43Z" level=info msg="Started syncing rollout" generation=757 namespace=app-namespace resourceVersion=4332525723 rollout=application-internal
2024-01-18 13:18:43.789 
time="2024-01-18T13:18:43Z" level=info msg="Enqueueing parent of app-namespace/application-internal-6fc65dcfc6: Rollout app-namespace/application-internal"
2024-01-18 13:18:43.571 
time="2024-01-18T13:18:43Z" level=info msg="Reconciliation completed" generation=757 namespace=app-namespace resourceVersion=4332525723 rollout=application-internal time_ms=7.259841000000001
2024-01-18 13:18:43.571 
time="2024-01-18T13:18:43Z" level=info msg="Queueing up rollout for a progress after 599s" namespace=app-namespace rollout=application-internal
2024-01-18 13:18:43.571 
time="2024-01-18T13:18:43Z" level=info msg="No status changes. Skipping patch" generation=757 namespace=app-namespace resourceVersion=4332525723 rollout=application-internal
2024-01-18 13:18:43.571 
time="2024-01-18T13:18:43Z" level=info msg="Timed out (false) [last progress check: 2024-01-18 13:18:43 +0000 UTC - now: 2024-01-18 13:18:43.571170622 +0000 UTC m=+16.338064705]" namespace=app-namespace rollout=application-internal
2024-01-18 13:18:43.570 
time="2024-01-18T13:18:43Z" level=info msg="Found 1 available pods in old RS app-namespace/application-internal-575b7d5fc6" namespace=app-namespace rollout=application-internal
2024-01-18 13:18:43.570 
time="2024-01-18T13:18:43Z" level=info msg="Reconciling 1 old ReplicaSets (total pods: 1)" namespace=app-namespace rollout=application-internal
2024-01-18 13:18:43.570 
time="2024-01-18T13:18:43Z" level=info msg="No TrafficRouting Reconcilers found" namespace=app-namespace rollout=application-internal
2024-01-18 13:18:43.566 
time="2024-01-18T13:18:43Z" level=info msg="Started syncing rollout" generation=757 namespace=app-namespace resourceVersion=4332525723 rollout=application-internal
2024-01-18 13:18:43.564 
time="2024-01-18T13:18:43Z" level=info msg="Processing completed" resource=app-namespace/application-internal
2024-01-18 13:18:43.564 
time="2024-01-18T13:18:43Z" level=info msg="Trigger on-rollout-aborted result: [{[0].2jmj7l5rSw0yVb_vlWAYkK_YBwk  [rollout-aborted] false}]" resource=app-namespace/application-internal
2024-01-18 13:18:43.564 
time="2024-01-18T13:18:43Z" level=info msg="Trigger on-analysis-run-failed result: [{[0].2jmj7l5rSw0yVb_vlWAYkK_YBwk  [analysis-run-failed] false}]" resource=app-namespace/application-internal
2024-01-18 13:18:43.564 
time="2024-01-18T13:18:43Z" level=info msg="Trigger on-analysis-run-error result: [{[0].2jmj7l5rSw0yVb_vlWAYkK_YBwk  [analysis-run-error] false}]" resource=app-namespace/application-internal
2024-01-18 13:18:43.562 
time="2024-01-18T13:18:43Z" level=info msg="Start processing" resource=app-namespace/application-internal
2024-01-18 13:18:43.559 
time="2024-01-18T13:18:43Z" level=info msg="Reconciliation completed" generation=757 namespace=app-namespace resourceVersion=4332321110 rollout=application-internal time_ms=69.43661900000001
2024-01-18 13:18:43.559 
time="2024-01-18T13:18:43Z" level=info msg="persisted to informer" generation=757 namespace=app-namespace resourceVersion=4332525723 rollout=application-internal
2024-01-18 13:18:43.556 
time="2024-01-18T13:18:43Z" level=info msg="Patched: {\"status\":{\"HPAReplicas\":3,\"availableReplicas\":3,\"conditions\":[{\"lastTransitionTime\":\"2024-01-18T10:36:03Z\",\"lastUpdateTime\":\"2024-01-18T10:36:03Z\",\"message\":\"Rollout is not healthy\",\"reason\":\"RolloutHealthy\",\"status\":\"False\",\"type\":\"Healthy\"},{\"lastTransitionTime\":\"2024-01-18T10:36:03Z\",\"lastUpdateTime\":\"2024-01-18T10:36:03Z\",\"message\":\"RolloutCompleted\",\"reason\":\"RolloutCompleted\",\"status\":\"False\",\"type\":\"Completed\"},{\"lastTransitionTime\":\"2024-01-18T10:49:24Z\",\"lastUpdateTime\":\"2024-01-18T10:49:24Z\",\"message\":\"Rollout is paused\",\"reason\":\"RolloutPaused\",\"status\":\"False\",\"type\":\"Paused\"},{\"lastTransitionTime\":\"2024-01-18T10:49:24Z\",\"lastUpdateTime\":\"2024-01-18T13:18:43Z\",\"message\":\"ReplicaSet \\\"application-internal-546856bb5c\\\" is progressing.\",\"reason\":\"ReplicaSetUpdated\",\"status\":\"True\",\"type\":\"Progressing\"},{\"lastTransitionTime\":\"2024-01-18T13:18:43Z\",\"lastUpdateTime\":\"2024-01-18T13:18:43Z\",\"message\":\"Rollout does not have minimum availability\",\"reason\":\"AvailableReason\",\"status\":\"False\",\"type\":\"Available\"}],\"readyReplicas\":3,\"replicas\":3}}" generation=757 namespace=app-namespace resourceVersion=4332321110 rollout=application-internal
2024-01-18 13:18:43.528 
time="2024-01-18T13:18:43Z" level=info msg="Event(v1.ObjectReference{Kind:\"Rollout\", Namespace:\"app-namespace\", Name:\"application-internal\", UID:\"54546450-62d4-4bb2-92f6-17a049ec3243\", APIVersion:\"argoproj.io/v1alpha1\", ResourceVersion:\"4332321110\", FieldPath:\"\"}): type: 'Normal' reason: 'ScalingReplicaSet' Scaled up ReplicaSet application-internal-6fc65dcfc6 (revision 714) from 2 to 4"
2024-01-18 13:18:43.528 
time="2024-01-18T13:18:43Z" level=info msg="Not finished reconciling ReplicaSets" namespace=app-namespace rollout=application-internal
2024-01-18 13:18:43.528 
time="2024-01-18T13:18:43Z" level=info msg="Not finished reconciling stableRS" namespace=app-namespace rollout=application-internal
2024-01-18 13:18:43.528 
time="2024-01-18T13:18:43Z" level=info msg="Scaled up ReplicaSet application-internal-6fc65dcfc6 (revision 714) from 2 to 4" event_reason=ScalingReplicaSet namespace=app-namespace rollout=application-internal
2024-01-18 13:18:43.495 
time="2024-01-18T13:18:43Z" level=info msg="No TrafficRouting Reconcilers found" namespace=app-namespace rollout=application-internal
2024-01-18 13:18:43.492 
time="2024-01-18T13:18:43Z" level=info msg="Started syncing rollout" generation=757 namespace=app-namespace resourceVersion=4332321110 rollout=application-internal
2024-01-18 13:18:43.482 
time="2024-01-18T13:18:43Z" level=info msg="Reconciliation completed" analysisrun=application-internal-6fc65dcfc6-714-2 namespace=app-namespace time_ms=3.419999
2024-01-18 13:18:43.482 
time="2024-01-18T13:18:43Z" level=info msg="No status changes. Skipping patch" analysisrun=application-internal-6fc65dcfc6-714-2 namespace=app-namespace
2024-01-18 13:18:43.481 
time="2024-01-18T13:18:43Z" level=info msg="Reconciliation completed" analysisrun=application-internal-575b7d5fc6-715-2 namespace=app-namespace time_ms=2.607729
2024-01-18 13:18:43.481 
time="2024-01-18T13:18:43Z" level=info msg="No status changes. Skipping patch" analysisrun=application-internal-575b7d5fc6-715-2 namespace=app-namespace
2024-01-18 13:18:43.479 
time="2024-01-18T13:18:43Z" level=info msg="Started syncing Analysis at (2024-01-18 13:18:43.478550445 +0000 UTC m=+16.245444593)" analysisrun=application-internal-575b7d5fc6-715-2 namespace=app-namespace
2024-01-18 13:18:43.479 
time="2024-01-18T13:18:43Z" level=info msg="Processing completed" resource=app-namespace/application-internal
2024-01-18 13:18:43.479 
time="2024-01-18T13:18:43Z" level=info msg="Trigger on-analysis-run-error result: [{[0].2jmj7l5rSw0yVb_vlWAYkK_YBwk  [analysis-run-error] false}]" resource=app-namespace/application-internal
2024-01-18 13:18:43.479 
time="2024-01-18T13:18:43Z" level=info msg="Trigger on-rollout-aborted result: [{[0].2jmj7l5rSw0yVb_vlWAYkK_YBwk  [rollout-aborted] false}]" resource=app-namespace/application-internal
2024-01-18 13:18:43.479 
time="2024-01-18T13:18:43Z" level=info msg="Trigger on-analysis-run-failed result: [{[0].2jmj7l5rSw0yVb_vlWAYkK_YBwk  [analysis-run-failed] false}]" resource=app-namespace/application-internal
2024-01-18 13:18:43.478 
time="2024-01-18T13:18:43Z" level=info msg="Started syncing Analysis at (2024-01-18 13:18:43.47857547 +0000 UTC m=+16.245469540)" analysisrun=application-internal-6fc65dcfc6-714-2 namespace=app-namespace
2024-01-18 13:18:43.476 
time="2024-01-18T13:18:43Z" level=info msg="Start processing" resource=app-namespace/application-internal
2024-01-18 13:18:43.387 
time="2024-01-18T13:18:43Z" level=info msg="Enqueueing parent of app-namespace/application-internal-7fc96c849c: Rollout app-namespace/application-internal"
2024-01-18 13:18:43.386 
time="2024-01-18T13:18:43Z" level=info msg="Enqueueing parent of app-namespace/application-internal-5f8bb4b544: Rollout app-namespace/application-internal"
2024-01-18 13:18:43.385 
time="2024-01-18T13:18:43Z" level=info msg="Enqueueing parent of app-namespace/application-internal-6fc65dcfc6: Rollout app-namespace/application-internal"
2024-01-18 13:18:43.384 
time="2024-01-18T13:18:43Z" level=info msg="Enqueueing parent of app-namespace/application-internal-575b7d5fc6: Rollout app-namespace/application-internal"
2024-01-18 13:18:43.381 
time="2024-01-18T13:18:43Z" level=info msg="Enqueueing parent of app-namespace/application-internal-546856bb5c: Rollout app-namespace/application-internal"
2024-01-18 13:18:42.930 
time="2024-01-18T13:18:42Z" level=info msg="Event(v1.ObjectReference{Kind:\"Rollout\", Namespace:\"app-namespace\", Name:\"application-internal\", UID:\"54546450-62d4-4bb2-92f6-17a049ec3243\", APIVersion:\"argoproj.io/v1alpha1\", ResourceVersion:\"4332321110\", FieldPath:\"\"}): type: 'Normal' reason: 'RolloutAddedToInformer' Rollout resource added to informer: app-namespace/application-internal"
2024-01-18 13:18:42.930 
time="2024-01-18T13:18:42Z" level=info msg="Rollout resource added to informer: app-namespace/application-internal" event_reason=RolloutAddedToInformer namespace=app-namespace rollout=application-internal
2024-01-18 13:18:42.895 
time="2024-01-18T13:18:42Z" level=info msg="Enqueueing parent of app-namespace/application-internal-6fc65dcfc6-714-2: Rollout app-namespace/application-internal"
2024-01-18 13:18:42.895 
time="2024-01-18T13:18:42Z" level=info msg="Enqueueing parent of app-namespace/application-internal-575b7d5fc6-715-2: Rollout app-namespace/application-internal"

events:

59m         Normal    SuccessfulCreate         replicaset/application-internal-6fc65dcfc6                  Created pod: application-internal-6fc65dcfc6-jhcgr
59m         Normal    SuccessfulCreate         replicaset/application-internal-6fc65dcfc6                  Created pod: application-internal-6fc65dcfc6-dp8m8
48m         Normal    SuccessfulDelete         replicaset/application-internal-6fc65dcfc6                  Deleted pod: application-internal-6fc65dcfc6-dp8m8
48m         Normal    SuccessfulDelete         replicaset/application-internal-6fc65dcfc6                  Deleted pod: application-internal-6fc65dcfc6-jhcgr
47m         Normal    SuccessfulDelete         replicaset/application-internal-6fc65dcfc6                  Deleted pod: application-internal-6fc65dcfc6-fxgnb
47m         Normal    SuccessfulDelete         replicaset/application-internal-6fc65dcfc6                  Deleted pod: application-internal-6fc65dcfc6-ll8xg
59m         Normal    RolloutAddedToInformer   rollout/application-internal                                Rollout resource added to informer: app-namespace/application-internal
59m         Normal    ScalingReplicaSet        rollout/application-internal                                Scaled up ReplicaSet application-internal-6fc65dcfc6 (revision 714) from 2 to 4
59m         Normal    ScalingReplicaSet        rollout/application-internal                                Scaled down ReplicaSet application-internal-575b7d5fc6 (revision 715) from 1 to 0
59m         Normal    ScalingReplicaSet        rollout/application-internal                                Scaled up ReplicaSet application-internal-546856bb5c (revision 717) from 0 to 1
58m         Normal    RolloutStepCompleted     rollout/application-internal                                Rollout step 1/3 completed (setWeight: 10)
58m         Normal    RolloutPaused            rollout/application-internal                                Rollout is paused (CanaryPauseStep)
48m         Normal    RolloutStepCompleted     rollout/application-internal                                Rollout step 2/3 completed (pause: 10m)
48m         Normal    RolloutResumed           rollout/application-internal                                Rollout is resumed
48m         Normal    AnalysisRunRunning       rollout/application-internal                                Step Analysis Run 'application-internal-546856bb5c-717-2' Status New: 'Running' Previous: ''
48m         Normal    AnalysisRunSuccessful    rollout/application-internal                                Step Analysis Run 'application-internal-546856bb5c-717-2' Status New: 'Successful' Previous: 'Running'
48m         Normal    RolloutStepCompleted     rollout/application-internal                                Rollout step 3/3 completed (analysis)
zachaller commented 9 months ago

Are you able to semi reliably reproduce this? One of the big pain points around this bug is that it's been a bit elusive to reproduce.

eugenepaniot commented 9 months ago

@zachaller, unfortunately, I couldn't reliably reproduce it, but I'll keep a close eye on it and try to gather more information. Additionally, for experimental purposes, I've set the --rollout-resync=60. Though I don't have enough data to confirm if it's related or if it has helped. I have observed that after this change, we still got roCtx.reconcile err error updating replicaset: Operation cannot be fulfilled on replicasets.apps... error, but there are no hung rollouts anymore. I will keep you updated as I gather more information. Thanks!

eugenepaniot commented 9 months ago

I am pretty much sure rollout-resync=60 helped, since right now I have similar setup (another app):

1. revision:124 (stable)
2. revision:125 (canary) // lasts for 10m until AnalysisRun 
3. revision:126 (new canary deployed) // seemed to be deployed during revision:125 rollout
3.1 revision:125 started to ScaledDown as expected as we could not see before (in report)

I will keep you updated as I gather more information. Thanks!

zachaller commented 9 months ago

Just for a bit of context as well we actually see the conflict on replicasets a lot but we are running like 8,000 rollout resources, so I don't think the observation of that log is always leads to a stuck rollout. We have also seen stuck rollouts but not any where close to the rate that we see the conflict. Those conflicts should not in theory cause an issue if the rollouts controller does what it suppose to and retry's the reconcile but something is going wrong at some point it's probably some race condition which makes it hard.

I want to add a bit more context to the logs we don't do a great job at logging the whole error function call chain and the function that that log line is comming from is called in various spots.

eugenepaniot commented 9 months ago

Yes, sure. We'd be happy to deploy a custom version to gather more information about this issue.

zachaller commented 9 months ago

@eugenepaniot you mind running 1.6.5 or master I added some logging

eugenepaniot commented 9 months ago

@zachaller, thank you. I have deployed the recent 1.6.5 version. Since we still have --rollout-resync=60 enabled, and we haven't faced any issues since then (or just ignoring'em). However, there are logs attached from recent deployments that might potentially reflect this issue:

argo-rollouts-9f7668599-wkb6j argo-rollouts time="2024-01-29T15:28:08Z" level=info msg="Reconciliation completed" generation=8966 namespace=app-namespace resourceVersion=4387395617 rollout=application time_ms=55.852259000000004
argo-rollouts-9f7668599-wkb6j argo-rollouts time="2024-01-29T15:28:08Z" level=info msg="Started syncing rollout" generation=8966 namespace=app-namespace resourceVersion=4387395628 rollout=application
argo-rollouts-9f7668599-wkb6j argo-rollouts time="2024-01-29T15:28:08Z" level=info msg="Enqueueing parent of app-namespace/application-56c85f4cf7: Rollout app-namespace/application"
argo-rollouts-9f7668599-wkb6j argo-rollouts time="2024-01-29T15:28:08Z" level=info msg="Start processing" resource=app-namespace/application
argo-rollouts-9f7668599-wkb6j argo-rollouts time="2024-01-29T15:28:08Z" level=info msg="No TrafficRouting Reconcilers found" namespace=app-namespace rollout=application
argo-rollouts-9f7668599-wkb6j argo-rollouts time="2024-01-29T15:28:08Z" level=info msg="Trigger on-analysis-run-error result: [{[0].2jmj7l5rSw0yVb_vlWAYkK_YBwk  [analysis-run-error] false}]" resource=app-namespace/application
argo-rollouts-9f7668599-wkb6j argo-rollouts time="2024-01-29T15:28:08Z" level=info msg="Trigger on-analysis-run-failed result: [{[0].2jmj7l5rSw0yVb_vlWAYkK_YBwk  [analysis-run-failed] false}]" resource=app-namespace/application
argo-rollouts-9f7668599-wkb6j argo-rollouts time="2024-01-29T15:28:08Z" level=info msg="Trigger on-rollout-aborted result: [{[0].2jmj7l5rSw0yVb_vlWAYkK_YBwk  [rollout-aborted] false}]" resource=app-namespace/application
argo-rollouts-9f7668599-wkb6j argo-rollouts time="2024-01-29T15:28:08Z" level=info msg="Processing completed" resource=app-namespace/application
argo-rollouts-9f7668599-wkb6j argo-rollouts time="2024-01-29T15:28:08Z" level=info msg="Enqueueing parent of app-namespace/application-56c85f4cf7: Rollout app-namespace/application"
argo-rollouts-9f7668599-wkb6j argo-rollouts time="2024-01-29T15:28:08Z" level=info msg="Enqueueing parent of app-namespace/application-56cb4869f5: Rollout app-namespace/application"

argo-rollouts-9f7668599-wkb6j argo-rollouts time="2024-01-29T15:28:08Z" level=error msg="roCtx.reconcile err failed to scaleReplicaSetAndRecordEvent in reconcileNewReplicaSet: failed to scaleReplicaSet in scaleReplicaSetAndRecordEvent: error updating replicaset application-56cb4869f5: Operation cannot be fulfilled on replicasets.apps \"application-56cb4869f5\": the object has been modified; please apply your changes to the latest version and try again" generation=8966 namespace=app-namespace resourceVersion=4387395628 rollout=application
argo-rollouts-9f7668599-wkb6j argo-rollouts time="2024-01-29T15:28:08Z" level=info msg="Reconciliation completed" generation=8966 namespace=app-namespace resourceVersion=4387395628 rollout=application time_ms=36.475529
argo-rollouts-9f7668599-wkb6j argo-rollouts time="2024-01-29T15:28:08Z" level=error msg="rollout syncHandler error: failed to scaleReplicaSetAndRecordEvent in reconcileNewReplicaSet: failed to scaleReplicaSet in scaleReplicaSetAndRecordEvent: error updating replicaset application-56cb4869f5: Operation cannot be fulfilled on replicasets.apps \"application-56cb4869f5\": the object has been modified; please apply your changes to the latest version and try again" namespace=app-namespace rollout=application
argo-rollouts-9f7668599-wkb6j argo-rollouts time="2024-01-29T15:28:08Z" level=info msg="rollout syncHandler queue retries: 5 : key \"app-namespace/application\"" namespace=app-namespace rollout=application
argo-rollouts-9f7668599-wkb6j argo-rollouts time="2024-01-29T15:28:08Z" level=error msg="failed to scaleReplicaSetAndRecordEvent in reconcileNewReplicaSet: failed to scaleReplicaSet in scaleReplicaSetAndRecordEvent: error updating replicaset application-56cb4869f5: Operation cannot be fulfilled on replicasets.apps \"application-56cb4869f5\": the object has been modified; please apply your changes to the latest version and try again\n" error="<nil>"
argo-rollouts-9f7668599-wkb6j argo-rollouts time="2024-01-29T15:28:08Z" level=info msg="Started syncing rollout" generation=8966 namespace=app-namespace resourceVersion=4387395628 rollout=application
argo-rollouts-9f7668599-wkb6j argo-rollouts time="2024-01-29T15:28:08Z" level=info msg="No TrafficRouting Reconcilers found" namespace=app-namespace rollout=application
argo-rollouts-9f7668599-wkb6j argo-rollouts time="2024-01-29T15:28:08Z" level=info msg="Enqueueing parent of app-namespace/application-56cb4869f5: Rollout app-namespace/application"
argo-rollouts-9f7668599-wkb6j argo-rollouts time="2024-01-29T15:28:08Z" level=debug msg="No configured destinations for trigger: on-scaling-replica-set" namespace=app-namespace rollout=application
argo-rollouts-9f7668599-wkb6j argo-rollouts time="2024-01-29T15:28:08Z" level=debug msg="Notification sent" namespace=app-namespace rollout=application time_ms=0.10429000000000001
argo-rollouts-9f7668599-wkb6j argo-rollouts time="2024-01-29T15:28:08Z" level=debug msg="No configured destinations for trigger: on-scaling-replica-set" namespace=app-namespace rollout=application
argo-rollouts-9f7668599-wkb6j argo-rollouts time="2024-01-29T15:28:08Z" level=debug msg="Notification sent" namespace=app-namespace rollout=application time_ms=0.09786
argo-rollouts-9f7668599-wkb6j argo-rollouts time="2024-01-29T15:28:08Z" level=info msg="Scaled down ReplicaSet application-56c85f4cf7 (revision 776) from 25 to 24" event_reason=ScalingReplicaSet namespace=app-namespace rollout=application
argo-rollouts-9f7668599-wkb6j argo-rollouts time="2024-01-29T15:28:08Z" level=info msg="Event(v1.ObjectReference{Kind:\"Rollout\", Namespace:\"app-namespace\", Name:\"application\", UID:\"56f200a0-d3ce-423e-a2dc-bbad86f75f02\", APIVersion:\"argoproj.io/v1alpha1\", ResourceVersion:\"4387395628\", FieldPath:\"\"}): type: 'Normal' reason: 'ScalingReplicaSet' Scaled down ReplicaSet application-56c85f4cf7 (revision 776) from 25 to 24"
argo-rollouts-9f7668599-wkb6j argo-rollouts time="2024-01-29T15:28:08Z" level=info msg="Not finished reconciling stableRS" namespace=app-namespace rollout=application
argo-rollouts-9f7668599-wkb6j argo-rollouts time="2024-01-29T15:28:08Z" level=info msg="Not finished reconciling ReplicaSets" namespace=app-namespace rollout=application
argo-rollouts-9f7668599-wkb6j argo-rollouts time="2024-01-29T15:28:08Z" level=info msg="Patched: {\"status\":{\"HPAReplicas\":60,\"replicas\":60}}" generation=8966 namespace=app-namespace resourceVersion=4387395628 rollout=application
argo-rollouts-9f7668599-wkb6j argo-rollouts time="2024-01-29T15:28:08Z" level=info msg="Enqueueing parent of app-namespace/application-56c85f4cf7: Rollout app-namespace/application"
argo-rollouts-9f7668599-wkb6j argo-rollouts time="2024-01-29T15:28:08Z" level=info msg="persisted to informer" generation=8966 namespace=app-namespace resourceVersion=4387395652 rollout=application
argo-rollouts-9f7668599-wkb6j argo-rollouts time="2024-01-29T15:28:08Z" level=info msg="Reconciliation completed" generation=8966 namespace=app-namespace resourceVersion=4387395628 rollout=application time_ms=61.087029
argo-rollouts-9f7668599-wkb6j argo-rollouts time="2024-01-29T15:28:08Z" level=info msg="Enqueueing parent of app-namespace/application-56cb4869f5: Rollout app-namespace/application"
argo-rollouts-9f7668599-wkb6j argo-rollouts time="2024-01-29T15:28:08Z" level=info msg="Started syncing rollout" generation=8966 namespace=app-namespace resourceVersion=4387395652 rollout=application
argo-rollouts-9f7668599-wkb6j argo-rollouts time="2024-01-29T15:28:08Z" level=info msg="Start processing" resource=app-namespace/application
argo-rollouts-9f7668599-wkb6j argo-rollouts time="2024-01-29T15:28:08Z" level=info msg="No TrafficRouting Reconcilers found" namespace=app-namespace rollout=application
argo-rollouts-9f7668599-wkb6j argo-rollouts time="2024-01-29T15:28:08Z" level=info msg="Trigger on-analysis-run-error result: [{[0].2jmj7l5rSw0yVb_vlWAYkK_YBwk  [analysis-run-error] false}]" resource=app-namespace/application

...

argo-rollouts-9f7668599-wkb6j argo-rollouts time="2024-01-29T15:28:08Z" level=info msg="Trigger on-analysis-run-failed result: [{[0].2jmj7l5rSw0yVb_vlWAYkK_YBwk  [analysis-run-failed] false}]" resource=app-namespace/application
argo-rollouts-9f7668599-wkb6j argo-rollouts time="2024-01-29T15:28:08Z" level=info msg="Processing completed" resource=app-namespace/application
argo-rollouts-9f7668599-wkb6j argo-rollouts time="2024-01-29T15:28:08Z" level=info msg="No TrafficRouting Reconcilers found" namespace=app-namespace rollout=application
argo-rollouts-9f7668599-wkb6j argo-rollouts time="2024-01-29T15:28:08Z" level=info msg="Enqueueing parent of app-namespace/application-56c85f4cf7: Rollout app-namespace/application"
argo-rollouts-9f7668599-wkb6j argo-rollouts time="2024-01-29T15:28:08Z" level=debug msg="No configured destinations for trigger: on-scaling-replica-set" namespace=app-namespace rollout=application
argo-rollouts-9f7668599-wkb6j argo-rollouts time="2024-01-29T15:28:08Z" level=debug msg="Notification sent" namespace=app-namespace rollout=application time_ms=0.08191
argo-rollouts-9f7668599-wkb6j argo-rollouts time="2024-01-29T15:28:08Z" level=debug msg="No configured destinations for trigger: on-scaling-replica-set" namespace=app-namespace rollout=application
argo-rollouts-9f7668599-wkb6j argo-rollouts time="2024-01-29T15:28:08Z" level=debug msg="Notification sent" namespace=app-namespace rollout=application time_ms=0.02699
argo-rollouts-9f7668599-wkb6j argo-rollouts time="2024-01-29T15:28:08Z" level=info msg="Scaled up ReplicaSet application-56cb4869f5 (revision 777) from 33 to 34" event_reason=ScalingReplicaSet namespace=app-namespace r
ollout=application
argo-rollouts-9f7668599-wkb6j argo-rollouts time="2024-01-29T15:28:08Z" level=info msg="Not finished reconciling new ReplicaSet 'application-56cb4869f5'" namespace=app-namespace rollout=application
argo-rollouts-9f7668599-wkb6j argo-rollouts time="2024-01-29T15:28:08Z" level=info msg="Not finished reconciling ReplicaSets" namespace=app-namespace rollout=application
argo-rollouts-9f7668599-wkb6j argo-rollouts time="2024-01-29T15:28:08Z" level=info msg="Event(v1.ObjectReference{Kind:\"Rollout\", Namespace:\"app-namespace\", Name:\"application\", UID:\"56f200a0-d3ce-423e-a2dc-bbad86
f75f02\", APIVersion:\"argoproj.io/v1alpha1\", ResourceVersion:\"4387395554\", FieldPath:\"\"}): type: 'Normal' reason: 'ScalingReplicaSet' Scaled up ReplicaSet application-56cb4869f5 (revision 777) from 33 to 34"
argo-rollouts-9f7668599-wkb6j argo-rollouts time="2024-01-29T15:28:08Z" level=info msg="Timed out (false) [last progress check: 2024-01-29 15:28:08 +0000 UTC - now: 2024-01-29 15:28:08.293178698 +0000 UTC m=+
264109.831373744]" namespace=app-namespace rollout=application
argo-rollouts-9f7668599-wkb6j argo-rollouts time="2024-01-29T15:28:08Z" level=info msg="No status changes. Skipping patch" generation=8966 namespace=app-namespace resourceVersion=4387395554 rollout=application
argo-rollouts-9f7668599-wkb6j argo-rollouts time="2024-01-29T15:28:08Z" level=info msg="Queueing up rollout for a progress after 599s" namespace=app-namespace rollout=application
argo-rollouts-9f7668599-wkb6j argo-rollouts time="2024-01-29T15:28:08Z" level=info msg="Reconciliation completed" generation=8966 namespace=app-namespace resourceVersion=4387395554 rollout=application time_ms=28.160159
argo-rollouts-9f7668599-wkb6j argo-rollouts time="2024-01-29T15:28:08Z" level=info msg="Started syncing rollout" generation=8966 namespace=app-namespace resourceVersion=4387395554 rollout=application
argo-rollouts-9f7668599-wkb6j argo-rollouts time="2024-01-29T15:28:08Z" level=info msg="No TrafficRouting Reconcilers found" namespace=app-namespace rollout=application
argo-rollouts-9f7668599-wkb6j argo-rollouts time="2024-01-29T15:28:08Z" level=info msg="No Steps remain in the canary steps" namespace=app-namespace rollout=application
argo-rollouts-9f7668599-wkb6j argo-rollouts time="2024-01-29T15:28:08Z" level=info msg="Patched: {\"status\":{\"HPAReplicas\":61,\"availableReplicas\":49,\"readyReplicas\":49,\"replicas\":61}}" generation=8966 namespace=app-namespace resourceVersion=4387395554 rollout=application
argo-rollouts-9f7668599-wkb6j argo-rollouts time="2024-01-29T15:28:08Z" level=info msg="persisted to informer" generation=8966 namespace=app-namespace resourceVersion=4387395563 rollout=application
argo-rollouts-9f7668599-wkb6j argo-rollouts time="2024-01-29T15:28:08Z" level=info msg="Reconciliation completed" generation=8966 namespace=app-namespace resourceVersion=4387395554 rollout=application time_ms=30.81137
argo-rollouts-9f7668599-wkb6j argo-rollouts time="2024-01-29T15:28:08Z" level=info msg="Start processing" resource=app-namespace/application
argo-rollouts-9f7668599-wkb6j argo-rollouts time="2024-01-29T15:28:08Z" level=info msg="Trigger on-analysis-run-error result: [{[0].2jmj7l5rSw0yVb_vlWAYkK_YBwk  [analysis-run-error] false}]" resource=app-namespace/application
argo-rollouts-9f7668599-wkb6j argo-rollouts time="2024-01-29T15:28:08Z" level=info msg="Trigger on-analysis-run-failed result: [{[0].2jmj7l5rSw0yVb_vlWAYkK_YBwk  [analysis-run-failed] false}]" resource=app-namespace/application
argo-rollouts-9f7668599-wkb6j argo-rollouts time="2024-01-29T15:28:08Z" level=info msg="Trigger on-rollout-aborted result: [{[0].2jmj7l5rSw0yVb_vlWAYkK_YBwk  [rollout-aborted] false}]" resource=app-namespace/application
argo-rollouts-9f7668599-wkb6j argo-rollouts time="2024-01-29T15:28:08Z" level=info msg="Processing completed" resource=app-namespace/application
argo-rollouts-9f7668599-wkb6j argo-rollouts time="2024-01-29T15:28:08Z" level=info msg="Started syncing rollout" generation=8966 namespace=app-namespace resourceVersion=4387395563 rollout=application
argo-rollouts-9f7668599-wkb6j argo-rollouts time="2024-01-29T15:28:08Z" level=info msg="No TrafficRouting Reconcilers found" namespace=app-namespace rollout=application
argo-rollouts-9f7668599-wkb6j argo-rollouts time="2024-01-29T15:28:08Z" level=info msg="No Steps remain in the canary steps" namespace=app-namespace rollout=application
argo-rollouts-9f7668599-wkb6j argo-rollouts time="2024-01-29T15:28:08Z" level=info msg="Timed out (false) [last progress check: 2024-01-29 15:28:08 +0000 UTC - now: 2024-01-29 15:28:08.334515377 +0000 UTC m=+264109.872710434]" namespace=app-namespace rollout=application
argo-rollouts-9f7668599-wkb6j argo-rollouts time="2024-01-29T15:28:08Z" level=info msg="No status changes. Skipping patch" generation=8966 namespace=app-namespace resourceVersion=4387395563 rollout=application
argo-rollouts-9f7668599-wkb6j argo-rollouts time="2024-01-29T15:28:08Z" level=info msg="Queueing up rollout for a progress after 599s" namespace=app-namespace rollout=application
argo-rollouts-9f7668599-wkb6j argo-rollouts time="2024-01-29T15:28:08Z" level=info msg="Reconciliation completed" generation=8966 namespace=app-namespace resourceVersion=4387395563 rollout=application time_ms=5.6810690
00000001
argo-rollouts-9f7668599-wkb6j argo-rollouts time="2024-01-29T15:28:08Z" level=info msg="Enqueueing parent of app-namespace/application-56cb4869f5: Rollout app-namespace/application"
argo-rollouts-9f7668599-wkb6j argo-rollouts time="2024-01-29T15:28:08Z" level=info msg="Started syncing rollout" generation=8966 namespace=app-namespace resourceVersion=4387395563 rollout=application
argo-rollouts-9f7668599-wkb6j argo-rollouts time="2024-01-29T15:28:08Z" level=info msg="No TrafficRouting Reconcilers found" namespace=app-namespace rollout=application
argo-rollouts-9f7668599-wkb6j argo-rollouts time="2024-01-29T15:28:08Z" level=info msg="No Steps remain in the canary steps" namespace=app-namespace rollout=application
argo-rollouts-9f7668599-wkb6j argo-rollouts time="2024-01-29T15:28:08Z" level=info msg="Timed out (false) [last progress check: 2024-01-29 15:28:08 +0000 UTC - now: 2024-01-29 15:28:08.421341405 +0000 UTC m=+
264109.959536462]" namespace=app-namespace rollout=application

...

argo-rollouts-9f7668599-wkb6j argo-rollouts time="2024-01-29T15:28:08Z" level=info msg="No status changes. Skipping patch" generation=8966 namespace=app-namespace resourceVersion=4387395563 rollout=application
argo-rollouts-9f7668599-wkb6j argo-rollouts time="2024-01-29T15:28:08Z" level=info msg="Queueing up rollout for a progress after 599s" namespace=app-namespace rollout=application
argo-rollouts-9f7668599-wkb6j argo-rollouts time="2024-01-29T15:28:08Z" level=info msg="Reconciliation completed" generation=8966 namespace=app-namespace resourceVersion=4387395563 rollout=application time_ms=11.08803
argo-rollouts-9f7668599-wkb6j argo-rollouts time="2024-01-29T15:28:08Z" level=info msg="Enqueueing parent of app-namespace/application-56cb4869f5: Rollout app-namespace/application"
argo-rollouts-9f7668599-wkb6j argo-rollouts time="2024-01-29T15:28:08Z" level=info msg="Started syncing rollout" generation=8966 namespace=app-namespace resourceVersion=4387395563 rollout=application
argo-rollouts-9f7668599-wkb6j argo-rollouts time="2024-01-29T15:28:08Z" level=info msg="No TrafficRouting Reconcilers found" namespace=app-namespace rollout=application
argo-rollouts-9f7668599-wkb6j argo-rollouts time="2024-01-29T15:28:08Z" level=info msg="No Steps remain in the canary steps" namespace=app-namespace rollout=application
argo-rollouts-9f7668599-wkb6j argo-rollouts time="2024-01-29T15:28:08Z" level=info msg="Enqueueing parent of app-namespace/application-56cb4869f5: Rollout app-namespace/application"
argo-rollouts-9f7668599-wkb6j argo-rollouts time="2024-01-29T15:28:08Z" level=info msg="Patched: {\"status\":{\"HPAReplicas\":62,\"replicas\":62,\"updatedReplicas\":34}}" generation=8966 namespace=app-namespace re
sourceVersion=4387395563 rollout=application
argo-rollouts-9f7668599-wkb6j argo-rollouts time="2024-01-29T15:28:08Z" level=info msg="persisted to informer" generation=8966 namespace=app-namespace resourceVersion=4387395584 rollout=application
argo-rollouts-9f7668599-wkb6j argo-rollouts time="2024-01-29T15:28:08Z" level=info msg="Reconciliation completed" generation=8966 namespace=app-namespace resourceVersion=4387395563 rollout=application time_ms=36.01666
argo-rollouts-9f7668599-wkb6j argo-rollouts time="2024-01-29T15:28:08Z" level=info msg="Started syncing rollout" generation=8966 namespace=app-namespace resourceVersion=4387395584 rollout=application
argo-rollouts-9f7668599-wkb6j argo-rollouts time="2024-01-29T15:28:08Z" level=info msg="Start processing" resource=app-namespace/application
argo-rollouts-9f7668599-wkb6j argo-rollouts time="2024-01-29T15:28:08Z" level=info msg="Trigger on-analysis-run-error result: [{[0].2jmj7l5rSw0yVb_vlWAYkK_YBwk  [analysis-run-error] false}]" resource=app-namespace
/application
argo-rollouts-9f7668599-wkb6j argo-rollouts time="2024-01-29T15:28:08Z" level=info msg="Trigger on-analysis-run-failed result: [{[0].2jmj7l5rSw0yVb_vlWAYkK_YBwk  [analysis-run-failed] false}]" resource=app-namespace/application
argo-rollouts-9f7668599-wkb6j argo-rollouts time="2024-01-29T15:28:08Z" level=info msg="Trigger on-rollout-aborted result: [{[0].2jmj7l5rSw0yVb_vlWAYkK_YBwk  [rollout-aborted] false}]" resource=app-namespace/application

argo-rollouts-9f7668599-wkb6j argo-rollouts time="2024-01-29T15:28:08Z" level=info msg="Processing completed" resource=app-namespace/application
argo-rollouts-9f7668599-wkb6j argo-rollouts time="2024-01-29T15:28:08Z" level=info msg="No TrafficRouting Reconcilers found" namespace=app-namespace rollout=application
argo-rollouts-9f7668599-wkb6j argo-rollouts time="2024-01-29T15:28:08Z" level=info msg="Enqueueing parent of app-namespace/application-56c85f4cf7: Rollout app-namespace/application"
argo-rollouts-9f7668599-wkb6j argo-rollouts time="2024-01-29T15:28:08Z" level=debug msg="No configured destinations for trigger: on-scaling-replica-set" namespace=app-namespace rollout=application
argo-rollouts-9f7668599-wkb6j argo-rollouts time="2024-01-29T15:28:08Z" level=debug msg="Notification sent" namespace=app-namespace rollout=application time_ms=0.09036
argo-rollouts-9f7668599-wkb6j argo-rollouts time="2024-01-29T15:28:08Z" level=info msg="Event(v1.ObjectReference{Kind:\"Rollout\", Namespace:\"app-namespace\", Name:\"application\", UID:\"56f200a0-d3ce-423e-a2dc-bbad86
f75f02\", APIVersion:\"argoproj.io/v1alpha1\", ResourceVersion:\"4387395584\", FieldPath:\"\"}): type: 'Normal' reason: 'ScalingReplicaSet' Scaled down ReplicaSet application-56c85f4cf7 (revision 776) from 28 to 2
7"
argo-rollouts-9f7668599-wkb6j argo-rollouts time="2024-01-29T15:28:08Z" level=debug msg="No configured destinations for trigger: on-scaling-replica-set" namespace=app-namespace rollout=application
argo-rollouts-9f7668599-wkb6j argo-rollouts time="2024-01-29T15:28:08Z" level=debug msg="Notification sent" namespace=app-namespace rollout=application time_ms=0.04672
argo-rollouts-9f7668599-wkb6j argo-rollouts time="2024-01-29T15:28:08Z" level=info msg="Scaled down ReplicaSet application-56c85f4cf7 (revision 776) from 28 to 27" event_reason=ScalingReplicaSet namespace=app-namespace
 rollout=application
argo-rollouts-9f7668599-wkb6j argo-rollouts time="2024-01-29T15:28:08Z" level=info msg="Not finished reconciling stableRS" namespace=app-namespace rollout=application
argo-rollouts-9f7668599-wkb6j argo-rollouts time="2024-01-29T15:28:08Z" level=info msg="Not finished reconciling ReplicaSets" namespace=app-namespace rollout=application
argo-rollouts-9f7668599-wkb6j argo-rollouts time="2024-01-29T15:28:08Z" level=info msg="Enqueueing parent of app-namespace/application-56c85f4cf7: Rollout app-namespace/application"
argo-rollouts-9f7668599-wkb6j argo-rollouts time="2024-01-29T15:28:08Z" level=info msg="Patched: {\"status\":{\"availableReplicas\":50,\"readyReplicas\":50}}" generation=8966 namespace=app-namespace resourceVersio
n=4387395584 rollout=application
argo-rollouts-9f7668599-wkb6j argo-rollouts time="2024-01-29T15:28:08Z" level=info msg="persisted to informer" generation=8966 namespace=app-namespace resourceVersion=4387395589 rollout=application
argo-rollouts-9f7668599-wkb6j argo-rollouts time="2024-01-29T15:28:08Z" level=info msg="Reconciliation completed" generation=8966 namespace=app-namespace resourceVersion=4387395584 rollout=application time_ms=63.935588
argo-rollouts-9f7668599-wkb6j argo-rollouts time="2024-01-29T15:28:08Z" level=info msg="Started syncing rollout" generation=8966 namespace=app-namespace resourceVersion=4387395589 rollout=application
argo-rollouts-9f7668599-wkb6j argo-rollouts time="2024-01-29T15:28:08Z" level=info msg="Start processing" resource=app-namespace/application
argo-rollouts-9f7668599-wkb6j argo-rollouts time="2024-01-29T15:28:08Z" level=info msg="Trigger on-analysis-run-error result: [{[0].2jmj7l5rSw0yVb_vlWAYkK_YBwk  [analysis-run-error] false}]" resource=app-namespace
/application
argo-rollouts-9f7668599-wkb6j argo-rollouts time="2024-01-29T15:28:08Z" level=info msg="Trigger on-analysis-run-failed result: [{[0].2jmj7l5rSw0yVb_vlWAYkK_YBwk  [analysis-run-failed] false}]" resource=app-namespace/application
argo-rollouts-9f7668599-wkb6j argo-rollouts time="2024-01-29T15:28:08Z" level=info msg="Trigger on-rollout-aborted result: [{[0].2jmj7l5rSw0yVb_vlWAYkK_YBwk  [rollout-aborted] false}]" resource=app-namespace/application
argo-rollouts-9f7668599-wkb6j argo-rollouts time="2024-01-29T15:28:08Z" level=info msg="Processing completed" resource=app-namespace/application

...

argo-rollouts-9f7668599-wkb6j argo-rollouts time="2024-01-29T15:28:08Z" level=info msg="No TrafficRouting Reconcilers found" namespace=app-namespace rollout=application
argo-rollouts-9f7668599-wkb6j argo-rollouts time="2024-01-29T15:28:08Z" level=info msg="Enqueueing parent of app-namespace/application-56cb4869f5: Rollout app-namespace/application"
argo-rollouts-9f7668599-wkb6j argo-rollouts time="2024-01-29T15:28:08Z" level=info msg="Enqueueing parent of app-namespace/application-56c85f4cf7: Rollout app-namespace/application"
argo-rollouts-9f7668599-wkb6j argo-rollouts time="2024-01-29T15:28:08Z" level=error msg="roCtx.reconcile err failed to scaleReplicaSetAndRecordEvent in reconcileNewReplicaSet: failed to scaleReplicaSet in sca
leReplicaSetAndRecordEvent: error updating replicaset application-56cb4869f5: Operation cannot be fulfilled on replicasets.apps \"application-56cb4869f5\": the object has been modified; please apply your changes to the
 latest version and try again" generation=8966 namespace=app-namespace resourceVersion=4387395617 rollout=application
argo-rollouts-9f7668599-wkb6j argo-rollouts time="2024-01-29T15:28:08Z" level=info msg="Reconciliation completed" generation=8966 namespace=app-namespace resourceVersion=4387395617 rollout=application time_ms=39.023489
000000005
argo-rollouts-9f7668599-wkb6j argo-rollouts time="2024-01-29T15:28:08Z" level=error msg="rollout syncHandler error: failed to scaleReplicaSetAndRecordEvent in reconcileNewReplicaSet: failed to scaleReplicaSet
 in scaleReplicaSetAndRecordEvent: error updating replicaset application-56cb4869f5: Operation cannot be fulfilled on replicasets.apps \"application-56cb4869f5\": the object has been modified; please apply your changes
 to the latest version and try again" namespace=app-namespace rollout=application
argo-rollouts-9f7668599-wkb6j argo-rollouts time="2024-01-29T15:28:08Z" level=info msg="rollout syncHandler queue retries: 4 : key \"app-namespace/application\"" namespace=app-namespace rollout=application
argo-rollouts-9f7668599-wkb6j argo-rollouts time="2024-01-29T15:28:08Z" level=error msg="failed to scaleReplicaSetAndRecordEvent in reconcileNewReplicaSet: failed to scaleReplicaSet in scaleReplicaSetAndRecor
dEvent: error updating replicaset application-56cb4869f5: Operation cannot be fulfilled on replicasets.apps \"application-56cb4869f5\": the object has been modified; please apply your changes to the latest version and
try again\n" error="<nil>"
argo-rollouts-9f7668599-wkb6j argo-rollouts time="2024-01-29T15:28:08Z" level=info msg="Started syncing rollout" generation=8966 namespace=app-namespace resourceVersion=4387395617 rollout=application
argo-rollouts-9f7668599-wkb6j argo-rollouts time="2024-01-29T15:28:08Z" level=info msg="No TrafficRouting Reconcilers found" namespace=app-namespace rollout=application
argo-rollouts-9f7668599-wkb6j argo-rollouts time="2024-01-29T15:28:08Z" level=debug msg="No configured destinations for trigger: on-scaling-replica-set" namespace=app-namespace rollout=application
argo-rollouts-9f7668599-wkb6j argo-rollouts time="2024-01-29T15:28:08Z" level=debug msg="Notification sent" namespace=app-namespace rollout=application time_ms=0.08063000000000001
argo-rollouts-9f7668599-wkb6j argo-rollouts time="2024-01-29T15:28:08Z" level=debug msg="No configured destinations for trigger: on-scaling-replica-set" namespace=app-namespace rollout=application
argo-rollouts-9f7668599-wkb6j argo-rollouts time="2024-01-29T15:28:08Z" level=debug msg="Notification sent" namespace=app-namespace rollout=application time_ms=0.091921
argo-rollouts-9f7668599-wkb6j argo-rollouts time="2024-01-29T15:28:08Z" level=info msg="Scaled down ReplicaSet application-56c85f4cf7 (revision 776) from 26 to 25" event_reason=ScalingReplicaSet namespace=app-namespace
 rollout=application
argo-rollouts-9f7668599-wkb6j argo-rollouts time="2024-01-29T15:28:08Z" level=info msg="Not finished reconciling stableRS" namespace=app-namespace rollout=application
argo-rollouts-9f7668599-wkb6j argo-rollouts time="2024-01-29T15:28:08Z" level=info msg="Not finished reconciling ReplicaSets" namespace=app-namespace rollout=application
argo-rollouts-9f7668599-wkb6j argo-rollouts time="2024-01-29T15:28:08Z" level=info msg="Event(v1.ObjectReference{Kind:\"Rollout\", Namespace:\"app-namespace\", Name:\"application\", UID:\"56f200a0-d3ce-423e-a2dc-bbad86
f75f02\", APIVersion:\"argoproj.io/v1alpha1\", ResourceVersion:\"4387395617\", FieldPath:\"\"}): type: 'Normal' reason: 'ScalingReplicaSet' Scaled down ReplicaSet application-56c85f4cf7 (revision 776) from 26 to 2
5"
argo-rollouts-9f7668599-wkb6j argo-rollouts time="2024-01-29T15:28:08Z" level=info msg="Patched: {\"status\":{\"HPAReplicas\":61,\"replicas\":61}}" generation=8966 namespace=app-namespace resourceVersion=438739561
7 rollout=application
argo-rollouts-9f7668599-wkb6j argo-rollouts time="2024-01-29T15:28:08Z" level=info msg="persisted to informer" generation=8966 namespace=app-namespace resourceVersion=4387395628 rollout=application
argo-rollouts-9f7668599-wkb6j argo-rollouts time="2024-01-29T15:28:08Z" level=info msg="Reconciliation completed" generation=8966 namespace=app-namespace resourceVersion=4387395617 rollout=application time_ms=55.852259
000000004
argo-rollouts-9f7668599-wkb6j argo-rollouts time="2024-01-29T15:28:08Z" level=info msg="Started syncing rollout" generation=8966 namespace=app-namespace resourceVersion=4387395628 rollout=application
argo-rollouts-9f7668599-wkb6j argo-rollouts time="2024-01-29T15:28:08Z" level=info msg="Enqueueing parent of app-namespace/application-56c85f4cf7: Rollout app-namespace/application"
argo-rollouts-9f7668599-wkb6j argo-rollouts time="2024-01-29T15:28:08Z" level=info msg="Start processing" resource=app-namespace/application
argo-rollouts-9f7668599-wkb6j argo-rollouts time="2024-01-29T15:28:08Z" level=info msg="No TrafficRouting Reconcilers found" namespace=app-namespace rollout=application
argo-rollouts-9f7668599-wkb6j argo-rollouts time="2024-01-29T15:28:08Z" level=info msg="Trigger on-analysis-run-error result: [{[0].2jmj7l5rSw0yVb_vlWAYkK_YBwk  [analysis-run-error] false}]" resource=app-namespace/application
argo-rollouts-9f7668599-wkb6j argo-rollouts time="2024-01-29T15:28:08Z" level=info msg="Trigger on-analysis-run-failed result: [{[0].2jmj7l5rSw0yVb_vlWAYkK_YBwk  [analysis-run-failed] false}]" resource=app-namespace/application
argo-rollouts-9f7668599-wkb6j argo-rollouts time="2024-01-29T15:28:08Z" level=info msg="Trigger on-rollout-aborted result: [{[0].2jmj7l5rSw0yVb_vlWAYkK_YBwk  [rollout-aborted] false}]" resource=app-namespace/application

...

argo-rollouts-9f7668599-wkb6j argo-rollouts time="2024-01-29T15:28:08Z" level=info msg="Processing completed" resource=app-namespace/application
argo-rollouts-9f7668599-wkb6j argo-rollouts time="2024-01-29T15:28:08Z" level=info msg="Enqueueing parent of app-namespace/application-56c85f4cf7: Rollout app-namespace/application"
argo-rollouts-9f7668599-wkb6j argo-rollouts time="2024-01-29T15:28:08Z" level=info msg="Enqueueing parent of app-namespace/application-56cb4869f5: Rollout app-namespace/application"
argo-rollouts-9f7668599-wkb6j argo-rollouts time="2024-01-29T15:28:08Z" level=error msg="roCtx.reconcile err failed to scaleReplicaSetAndRecordEvent in reconcileNewReplicaSet: failed to scaleReplicaSet in scaleReplicaSetAndRecordEvent: error updating replicaset application-56cb4869f5: Operation cannot be fulfilled on replicasets.apps \"application-56cb4869f5\": the object has been modified; please apply your changes to the latest version and try again" generation=8966 namespace=app-namespace resourceVersion=4387395628 rollout=application
argo-rollouts-9f7668599-wkb6j argo-rollouts time="2024-01-29T15:28:08Z" level=info msg="Reconciliation completed" generation=8966 namespace=app-namespace resourceVersion=4387395628 rollout=application time_ms=36.475529
argo-rollouts-9f7668599-wkb6j argo-rollouts time="2024-01-29T15:28:08Z" level=error msg="rollout syncHandler error: failed to scaleReplicaSetAndRecordEvent in reconcileNewReplicaSet: failed to scaleReplicaSet
 in scaleReplicaSetAndRecordEvent: error updating replicaset application-56cb4869f5: Operation cannot be fulfilled on replicasets.apps \"application-56cb4869f5\": the object has been modified; please apply your changes
 to the latest version and try again" namespace=app-namespace rollout=application
argo-rollouts-9f7668599-wkb6j argo-rollouts time="2024-01-29T15:28:08Z" level=info msg="rollout syncHandler queue retries: 5 : key \"app-namespace/application\"" namespace=app-namespace rollout=application
argo-rollouts-9f7668599-wkb6j argo-rollouts time="2024-01-29T15:28:08Z" level=error msg="failed to scaleReplicaSetAndRecordEvent in reconcileNewReplicaSet: failed to scaleReplicaSet in scaleReplicaSetAndRecor
dEvent: error updating replicaset application-56cb4869f5: Operation cannot be fulfilled on replicasets.apps \"application-56cb4869f5\": the object has been modified; please apply your changes to the latest version and
try again\n" error="<nil>"
argo-rollouts-9f7668599-wkb6j argo-rollouts time="2024-01-29T15:28:08Z" level=info msg="Started syncing rollout" generation=8966 namespace=app-namespace resourceVersion=4387395628 rollout=application
argo-rollouts-9f7668599-wkb6j argo-rollouts time="2024-01-29T15:28:08Z" level=info msg="No TrafficRouting Reconcilers found" namespace=app-namespace rollout=application
argo-rollouts-9f7668599-wkb6j argo-rollouts time="2024-01-29T15:28:08Z" level=info msg="Enqueueing parent of app-namespace/application-56cb4869f5: Rollout app-namespace/application"
argo-rollouts-9f7668599-wkb6j argo-rollouts time="2024-01-29T15:28:08Z" level=debug msg="No configured destinations for trigger: on-scaling-replica-set" namespace=app-namespace rollout=application
argo-rollouts-9f7668599-wkb6j argo-rollouts time="2024-01-29T15:28:08Z" level=debug msg="Notification sent" namespace=app-namespace rollout=application time_ms=0.10429000000000001
argo-rollouts-9f7668599-wkb6j argo-rollouts time="2024-01-29T15:28:08Z" level=debug msg="No configured destinations for trigger: on-scaling-replica-set" namespace=app-namespace rollout=application
argo-rollouts-9f7668599-wkb6j argo-rollouts time="2024-01-29T15:28:08Z" level=debug msg="Notification sent" namespace=app-namespace rollout=application time_ms=0.09786
argo-rollouts-9f7668599-wkb6j argo-rollouts time="2024-01-29T15:28:08Z" level=info msg="Scaled down ReplicaSet application-56c85f4cf7 (revision 776) from 25 to 24" event_reason=ScalingReplicaSet namespace=app-namespace
 rollout=application
argo-rollouts-9f7668599-wkb6j argo-rollouts time="2024-01-29T15:28:08Z" level=info msg="Event(v1.ObjectReference{Kind:\"Rollout\", Namespace:\"app-namespace\", Name:\"application\", UID:\"56f200a0-d3ce-423e-a2dc-bbad86
f75f02\", APIVersion:\"argoproj.io/v1alpha1\", ResourceVersion:\"4387395628\", FieldPath:\"\"}): type: 'Normal' reason: 'ScalingReplicaSet' Scaled down ReplicaSet application-56c85f4cf7 (revision 776) from 25 to 2
4"
argo-rollouts-9f7668599-wkb6j argo-rollouts time="2024-01-29T15:28:08Z" level=info msg="Not finished reconciling stableRS" namespace=app-namespace rollout=application
argo-rollouts-9f7668599-wkb6j argo-rollouts time="2024-01-29T15:28:08Z" level=info msg="Not finished reconciling ReplicaSets" namespace=app-namespace rollout=application
argo-rollouts-9f7668599-wkb6j argo-rollouts time="2024-01-29T15:28:08Z" level=info msg="Patched: {\"status\":{\"HPAReplicas\":60,\"replicas\":60}}" generation=8966 namespace=app-namespace resourceVersion=438739562
8 rollout=application
argo-rollouts-9f7668599-wkb6j argo-rollouts time="2024-01-29T15:28:08Z" level=info msg="Enqueueing parent of app-namespace/application-56c85f4cf7: Rollout app-namespace/application"
argo-rollouts-9f7668599-wkb6j argo-rollouts time="2024-01-29T15:28:08Z" level=info msg="persisted to informer" generation=8966 namespace=app-namespace resourceVersion=4387395652 rollout=application
argo-rollouts-9f7668599-wkb6j argo-rollouts time="2024-01-29T15:28:08Z" level=info msg="Reconciliation completed" generation=8966 namespace=app-namespace resourceVersion=4387395628 rollout=application time_ms=61.087029
argo-rollouts-9f7668599-wkb6j argo-rollouts time="2024-01-29T15:28:08Z" level=info msg="Enqueueing parent of app-namespace/application-56cb4869f5: Rollout app-namespace/application"
argo-rollouts-9f7668599-wkb6j argo-rollouts time="2024-01-29T15:28:08Z" level=info msg="Started syncing rollout" generation=8966 namespace=app-namespace resourceVersion=4387395652 rollout=application
argo-rollouts-9f7668599-wkb6j argo-rollouts time="2024-01-29T15:28:08Z" level=info msg="Start processing" resource=app-namespace/application
argo-rollouts-9f7668599-wkb6j argo-rollouts time="2024-01-29T15:28:08Z" level=info msg="No TrafficRouting Reconcilers found" namespace=app-namespace rollout=application
argo-rollouts-9f7668599-wkb6j argo-rollouts time="2024-01-29T15:28:08Z" level=info msg="Trigger on-analysis-run-error result: [{[0].2jmj7l5rSw0yVb_vlWAYkK_YBwk  [analysis-run-error] false}]" resource=app-namespace
/application
argo-rollouts-9f7668599-wkb6j argo-rollouts time="2024-01-29T15:28:08Z" level=info msg="Trigger on-analysis-run-failed result: [{[0].2jmj7l5rSw0yVb_vlWAYkK_YBwk  [analysis-run-failed] false}]" resource=app-namespace/application
argo-rollouts-9f7668599-wkb6j argo-rollouts time="2024-01-29T15:28:08Z" level=info msg="Trigger on-rollout-aborted result: [{[0].2jmj7l5rSw0yVb_vlWAYkK_YBwk  [rollout-aborted] false}]" resource=app-namespace/application
argo-rollouts-9f7668599-wkb6j argo-rollouts time="2024-01-29T15:28:08Z" level=info msg="Processing completed" resource=app-namespace/application
argo-rollouts-9f7668599-wkb6j argo-rollouts time="2024-01-29T15:28:08Z" level=info msg="Enqueueing parent of app-namespace/application-56c85f4cf7: Rollout app-namespace/application"
argo-rollouts-9f7668599-wkb6j argo-rollouts time="2024-01-29T15:28:08Z" level=error msg="roCtx.reconcile err failed to scaleReplicaSetAndRecordEvent in reconcileCanaryStableReplicaSet:L failed to scaleReplica
Set in scaleReplicaSetAndRecordEvent: error updating replicaset application-56c85f4cf7: Operation cannot be fulfilled on replicasets.apps \"application-56c85f4cf7\": the object has been modified; please apply your chan
ges to the latest version and try again" generation=8966 namespace=app-namespace resourceVersion=4387395652 rollout=application
argo-rollouts-9f7668599-wkb6j argo-rollouts time="2024-01-29T15:28:08Z" level=info msg="Reconciliation completed" generation=8966 namespace=app-namespace resourceVersion=4387395652 rollout=application time_ms=35.519069
argo-rollouts-9f7668599-wkb6j argo-rollouts time="2024-01-29T15:28:08Z" level=error msg="rollout syncHandler error: failed to scaleReplicaSetAndRecordEvent in reconcileCanaryStableReplicaSet:L failed to scale
ReplicaSet in scaleReplicaSetAndRecordEvent: error updating replicaset application-56c85f4cf7: Operation cannot be fulfilled on replicasets.apps \"application-56c85f4cf7\": the object has been modified; please apply yo
ur changes to the latest version and try again" namespace=app-namespace rollout=application

...

argo-rollouts-9f7668599-wkb6j argo-rollouts time="2024-01-29T15:28:52Z" level=info msg="Enqueueing parent of app-namespace/application-56cb4869f5: Rollout app-namespace/application"
argo-rollouts-9f7668599-wkb6j argo-rollouts time="2024-01-29T15:28:52Z" level=error msg="roCtx.reconcile err error updating replicaset in syncEphemeralMetadata: Operation cannot be fulfilled on replicasets.ap
ps \"application-56cb4869f5\": the object has been modified; please apply your changes to the latest version and try again" generation=8966 namespace=app-namespace resourceVersion=4387398384 rollout=application
argo-rollouts-9f7668599-wkb6j argo-rollouts time="2024-01-29T15:28:52Z" level=info msg="Reconciliation completed" generation=8966 namespace=app-namespace resourceVersion=4387398384 rollout=application time_ms=2160.3069
15
argo-rollouts-9f7668599-wkb6j argo-rollouts time="2024-01-29T15:28:52Z" level=error msg="rollout syncHandler error: error updating replicaset in syncEphemeralMetadata: Operation cannot be fulfilled on replicasets.apps \"application-56cb4869f5\": the object has been modified; please apply your changes to the latest version and try again" namespace=app-namespace rollout=application
argo-rollouts-9f7668599-wkb6j argo-rollouts time="2024-01-29T15:28:52Z" level=info msg="rollout syncHandler queue retries: 52 : key \"app-namespace/application\"" namespace=app-namespace rollout=application
argo-rollouts-9f7668599-wkb6j argo-rollouts time="2024-01-29T15:28:52Z" level=error msg="error updating replicaset in syncEphemeralMetadata: Operation cannot be fulfilled on replicasets.apps \"application-56cb4869f5\": the object has been modified; please apply your changes to the latest version and try again\n" error="<nil>"
argo-rollouts-9f7668599-wkb6j argo-rollouts time="2024-01-29T15:28:52Z" level=info msg="Started syncing rollout" generation=8966 namespace=app-namespace resourceVersion=4387398384 rollout=application
argo-rollouts-9f7668599-wkb6j argo-rollouts time="2024-01-29T15:28:52Z" level=info msg="synced ephemeral metadata &PodTemplateMetadata{Labels:map[string]string{canary: stable,},Annotations:map[string]string{},} to ReplicaSet application-56cb4869f5" namespace=app-namespace rollout=application
argo-rollouts-9f7668599-wkb6j argo-rollouts time="2024-01-29T15:28:52Z" level=info msg="Enqueueing parent of app-namespace/application-56cb4869f5: Rollout app-namespace/application"
argo-rollouts-9f7668599-wkb6j argo-rollouts time="2024-01-29T15:28:52Z" level=info msg="synced ephemeral metadata nil to Pod application-56c85f4cf7-2rglx" namespace=app-namespace rollout=application
argo-rollouts-9f7668599-wkb6j argo-rollouts time="2024-01-29T15:28:52Z" level=info msg="synced ephemeral metadata nil to Pod application-56c85f4cf7-4z4sk" namespace=app-namespace rollout=application
argo-rollouts-9f7668599-wkb6j argo-rollouts time="2024-01-29T15:28:52Z" level=info msg="synced ephemeral metadata nil to Pod application-56c85f4cf7-8sx72" namespace=app-namespace rollout=application
argo-rollouts-9f7668599-wkb6j argo-rollouts time="2024-01-29T15:28:52Z" level=info msg="synced ephemeral metadata nil to Pod application-56c85f4cf7-cpdkf" namespace=app-namespace rollout=application
argo-rollouts-9f7668599-wkb6j argo-rollouts time="2024-01-29T15:28:53Z" level=info msg="synced ephemeral metadata nil to Pod application-56c85f4cf7-cz6h4" namespace=app-namespace rollout=application
argo-rollouts-9f7668599-wkb6j argo-rollouts time="2024-01-29T15:28:53Z" level=info msg="synced ephemeral metadata nil to Pod application-56c85f4cf7-drwdg" namespace=app-namespace rollout=application
argo-rollouts-9f7668599-wkb6j argo-rollouts time="2024-01-29T15:28:53Z" level=info msg="synced ephemeral metadata nil to Pod application-56c85f4cf7-f9w9n" namespace=app-namespace rollout=application
argo-rollouts-9f7668599-wkb6j argo-rollouts time="2024-01-29T15:28:53Z" level=info msg="synced ephemeral metadata nil to Pod application-56c85f4cf7-fcj8p" namespace=app-namespace rollout=application
argo-rollouts-9f7668599-wkb6j argo-rollouts time="2024-01-29T15:28:53Z" level=info msg="synced ephemeral metadata nil to Pod application-56c85f4cf7-fdcrm" namespace=app-namespace rollout=application
argo-rollouts-9f7668599-wkb6j argo-rollouts time="2024-01-29T15:28:53Z" level=info msg="synced ephemeral metadata nil to Pod application-56c85f4cf7-g8krx" namespace=app-namespace rollout=application
argo-rollouts-9f7668599-wkb6j argo-rollouts time="2024-01-29T15:28:53Z" level=info msg="synced ephemeral metadata nil to Pod application-56c85f4cf7-kjcdt" namespace=app-namespace rollout=application
argo-rollouts-9f7668599-wkb6j argo-rollouts time="2024-01-29T15:28:53Z" level=info msg="synced ephemeral metadata nil to Pod application-56c85f4cf7-kk26n" namespace=app-namespace rollout=application
argo-rollouts-9f7668599-wkb6j argo-rollouts time="2024-01-29T15:28:53Z" level=info msg="synced ephemeral metadata nil to Pod application-56c85f4cf7-ld299" namespace=app-namespace rollout=application
argo-rollouts-9f7668599-wkb6j argo-rollouts time="2024-01-29T15:28:53Z" level=info msg="synced ephemeral metadata nil to Pod application-56c85f4cf7-lrpb7" namespace=app-namespace rollout=application
argo-rollouts-9f7668599-wkb6j argo-rollouts time="2024-01-29T15:28:53Z" level=error msg="roCtx.reconcile err Operation cannot be fulfilled on pods \"application-56c85f4cf7-n48gk\": the object has been modified; pl
ease apply your changes to the latest version and try again" generation=8966 namespace=app-namespace resourceVersion=4387398384 rollout=application
argo-rollouts-9f7668599-wkb6j argo-rollouts time="2024-01-29T15:28:53Z" level=info msg="Reconciliation completed" generation=8966 namespace=app-namespace resourceVersion=4387398384 rollout=application time_ms=813.65643
3
argo-rollouts-9f7668599-wkb6j argo-rollouts time="2024-01-29T15:28:53Z" level=error msg="rollout syncHandler error: Operation cannot be fulfilled on pods \"application-56c85f4cf7-n48gk\": the object has been modif
ied; please apply your changes to the latest version and try again" namespace=app-namespace rollout=application
argo-rollouts-9f7668599-wkb6j argo-rollouts time="2024-01-29T15:28:53Z" level=info msg="rollout syncHandler queue retries: 54 : key \"app-namespace/application\"" namespace=app-namespace rollout=application
argo-rollouts-9f7668599-wkb6j argo-rollouts time="2024-01-29T15:28:53Z" level=error msg="Operation cannot be fulfilled on pods \"application-56c85f4cf7-n48gk\": the object has been modified; please apply your chan
ges to the latest version and try again\n" error="<nil>"
argo-rollouts-9f7668599-wkb6j argo-rollouts time="2024-01-29T15:29:01Z" level=info msg="Started syncing rollout" generation=8966 namespace=app-namespace resourceVersion=4387398384 rollout=application
argo-rollouts-9f7668599-wkb6j argo-rollouts time="2024-01-29T15:29:01Z" level=info msg="synced ephemeral metadata nil to Pod application-56c85f4cf7-2rglx" namespace=app-namespace rollout=application
argo-rollouts-9f7668599-wkb6j argo-rollouts time="2024-01-29T15:29:01Z" level=info msg="synced ephemeral metadata nil to Pod application-56c85f4cf7-4z4sk" namespace=app-namespace rollout=application
argo-rollouts-9f7668599-wkb6j argo-rollouts time="2024-01-29T15:29:01Z" level=info msg="synced ephemeral metadata nil to Pod application-56c85f4cf7-8sx72" namespace=app-namespace rollout=application

Few words about application setup:

spec:
  replicas: 55
  restartAt: "2023-12-22T09:49:46Z"
  revisionHistoryLimit: 3
  rollbackWindow:
    revisions: 3
  selector:
    matchLabels:
      app: application
      app.kubernetes.io/instance: application
  strategy:
    canary:
      canaryMetadata:
        labels:
          canary: canary
      maxSurge: 25%
      maxUnavailable: 0
      stableMetadata:
        labels:
          canary: stable
      steps:
      - setWeight: 10
      - pause:
          duration: 10m
      - analysis:
          analysisRunMetadata:
            labels:
              app: application
              app.kubernetes.io/instance: application
              app.kubernetes.io/managed-by: Helm
              app.kubernetes.io/name: application
              app.kubernetes.io/part-of: application
          args:
          - name: stable-hash
            valueFrom:
              podTemplateHashValue: Stable
          - name: canary-hash
            valueFrom:
              podTemplateHashValue: Latest
          dryRun:
          - metricName: .*
          templates:
          - templateName: application

The application also has HPA enabled.

Please let me know if you need more information. Thanks.

toyamagu-2021 commented 9 months ago

I'm facing with similar issue where my Rollouts with HPA stack on Progressing status permanently in v1.6.2. (This occurs randomly, and I've not tested v1.6.5. yet) In my case, we have some strange Rollouts status as following:

status:
  HPAReplicas: 13
  availableReplicas: 13
  blueGreen: {}
  canary: {} 
  currentPodHash: 5f9cf5d68b
  currentStepHash: d68d4f7d8
  currentStepIndex: 2
  message: old replicas are pending termination
  observedGeneration: '8654'
  phase: Progressing
  readyReplicas: 11
  replicas: 11
  stableRS: 5f9cf5d68b
  updatedReplicas: 10

Logs

time="2024-01-30T17:16:05Z" level=error msg="rollout syncHandler error: Operation cannot be fulfilled on replicasets.apps \"app-6b76b6dd98\": the object has been modified; please apply your changes to the latest version and try again" namespace=apps rollout=app
time="2024-01-30T17:16:05Z" level=info msg="rollout syncHandler queue retries: 1865 : key \"apps/app\"" namespace=apps rollout=app

Rollouts is fully promoted and new version released, but stacks on Progressing status because of inconsistent replicaset.count. Intuitively, I suspect HPA should relevent with this issue. Unlike #3256, the pod template has not been changed.

Additional info

int128 commented 9 months ago

I updated the controller to v1.6.5 and this issue still occurs. https://github.com/argoproj/argo-rollouts/issues/3080#issuecomment-1835809731

For now, we run a job to restart the controller every hour, but we still get ProgressDeadlineExceeded: ReplicaSet ... has timed out progressing errors.

andrii-korotkov-verkada commented 9 months ago

I've seen the issue with 1.6.4 where the newest replicaset have a relatively small number of current and desired replicas (but more than minimum from KEDA HPA), while an older replicaset was getting desired replicas to the max, but was stuck at 0 replicas. It actually used to work for some time after doing a rollout with scaling as expected, but something broke later causing the behavior above. Doing a new rollout mitigated the issue, but it feels like playing a lottery.

It seems like one of two things:

Maybe the warning about "the object has been modified; please apply your changes to the latest version and try again" is just a symptom of propagating desired replicas to the older replica set.

andrii-korotkov-verkada commented 9 months ago

Looking at the Event logs, there are RolloutNotCompleted followed by RolloutUpdated. But there's no RolloutCompleted for the app in question. Argo UI still shows spinning sync circle after several hours too. ScalingReplicaSet events were there for the new replica set, but they were scaling to the small number of replicas (wrongly) until the new rollout began.

zachaller commented 8 months ago

V1.6.6 was released it fixes one reproducible version of a stuck rollout people might want to give it a try.

jchenbl commented 8 months ago

hi guys we also experienced some issues that canary rollout stucked, we couldn't reproduce steadily but what I found when I added a couple more logs in argo rollouts, it seems like there is race condition betweem the syncReplicasOnly and rolloutCanary that they are updating the c.newRS at the same time (and potentially together with reconcileEphemeralMetadata which is syncing c.newRS to kubernetes at the same time), so the value in c.newRS could be different than the actual replicaSet in the kubernetes.

kaushikjampala-cb commented 8 months ago

We are on v1.6.6 and were seeing the same errors listed above. We added the --rollout-resync=60 and errors subsided initially but gradually came back. We tried to upgrade version to deal with ReplicaSet not found errors but have had to rollback due to these errors.

zachaller commented 8 months ago

hi guys we also experienced some issues that canary rollout stucked, we couldn't reproduce steadily but what I found when I added a couple more logs in argo rollouts, it seems like there is race condition betweem the syncReplicasOnly and rolloutCanary that they are updating the c.newRS at the same time (and potentially together with reconcileEphemeralMetadata which is syncing c.newRS to kubernetes at the same time), so the value in c.newRS could be different than the actual replicaSet in the kubernetes.

Yup I have also found this there a few places where we clobber the in memory state of the replicaset. I do suspect this is somewhat the root cause. The one thing that bothers me though is why the retry reconcile does not correct it and instead it gets stuck. I think the issues has been there a while but it seems to have gotten worse with recent versions. This is going to become one of my top priorities to figure out.

gavin-db commented 8 months ago

Hey @zachaller. Curious if there is an update on this? We have been seeing the same issue on 1.6.3. We are currently resolving via controller restarts, but have had to pause our upgrade to 1.6. Might be able to share some logs next time it happens if there is anything in particular you are looking for.

Hronom commented 8 months ago

Just got in trap with this on 1.6.6 version of Argo Rollouts:

{"generation":272,"level":"error","msg":"roCtx.reconcile err error updating replicaset in syncEphemeralMetadata: Operation cannot be fulfilled on replicasets.apps \"xxx-111111\": the object has been modified; please apply your changes to the latest version and try again","namespace":"xxx-111111","resourceVersion":"243975074","rollout":"xxx-111111","time":"2024-03-07T00:52:07Z"}
{"level":"error","msg":"rollout syncHandler error: error updating replicaset in syncEphemeralMetadata: Operation cannot be fulfilled on replicasets.apps \"xxx-111111\": the object has been modified; please apply your changes to the latest version and try again","namespace":"xxx-111111","rollout":"xxx-111111","time":"2024-03-07T00:52:07Z"}
time="2024-03-07T00:52:07Z" level=error msg="error updating replicaset in syncEphemeralMetadata: Operation cannot be fulfilled on replicasets.apps \"xxx-111111\": the object has been modified; please apply your changes to the latest version and try again\n" error="<nil>"

The only what helps is restart of argo-rollouts

eugenepaniot commented 7 months ago

We tried version 1.6.6 but decided to revert back to 1.6.5 since we faced issues more often than it used to be.

triplewy commented 7 months ago

Any updates on this? We are facing the same issue with v1.6.6 consistently with argo rollouts and HPA.

Vinaum8 commented 7 months ago

Me too!

Vinaum8 commented 7 months ago

We tried version 1.6.6 but decided to revert back to 1.6.5 since we faced issues more often than it used to be.

Have you had these problems since 2018?

Hronom commented 6 months ago

@zachaller did you able to check it? What version more or less stable? We are on latest 1.6.6 and it's pretty frequent

zachaller commented 6 months ago

I have not yet but it is on my todo and is getting some priority at Intuit, we don't see it a whole lot across our 8k rollout resources though which effects my priority a bit however that said it is starting to get some traction internally.

zeemy23 commented 5 months ago

+1 we're seeing issues that require restart on 1.6.6

Sineaggi commented 5 months ago

We've also have to restart argo-rollouts to get a few rollouts un-stuck.

jeromeguiard commented 5 months ago

+1 I am trying a cronJob to restart it daily to see if it helps to reduce the frequency of the issue

njtman commented 4 months ago

Has anyone tried replicating with v1.7.0 yet? It includes PR #3559 which may address the stuck rollouts issue.

andrii-korotkov-verkada commented 4 months ago

Has anyone tried replicating with v1.7.0 yet? It includes PR #3559 which may address the stuck rollouts issue.

I've tried 1.7.0-rc1, though it has been before this PR and also had this issue https://github.com/argoproj/argo-rollouts/issues/3605.

jccastillocano commented 4 months ago

Hi all! We've also seen this error still in 1.7.1 provisioned by argo-rollouts 2.37.0 chart, see logs below

time="2024-06-26T10:08:32Z" level=info msg="Pod template change detected (new: 587d96649f, old: 547f9fc495)" namespace=canary-analysis rollout=podinfo
time="2024-06-26T10:08:32Z" level=info msg="Assuming 587d96649f for new replicaset pod hash" namespace=canary-analysis rollout=podinfo
time="2024-06-26T10:08:32Z" level=info msg="Pod template change detected (new: 587d96649f, old: 547f9fc495)" namespace=canary-analysis rollout=podinfo
time="2024-06-26T10:08:32Z" level=info msg="Patched: {\"status\":{\"abort\":null,\"abortedAt\":null,\"canary\":{\"currentStepAnalysisRunStatus\":null},\"currentPodHash\":\"587d96649f\",\"workloadObservedGeneration\":\"333\"}}" generation=229 namespace=canary-analysis res>
time="2024-06-26T10:08:32Z" level=info msg="persisted to informer" generation=229 namespace=canary-analysis resourceVersion=392564479 rollout=podinfo
time="2024-06-26T10:08:32Z" level=info msg="Reconciliation completed" generation=229 namespace=canary-analysis resourceVersion=392564477 rollout=podinfo time_ms=18.695665
time="2024-06-26T10:08:32Z" level=info msg="Started syncing rollout" generation=229 namespace=canary-analysis resourceVersion=392564479 rollout=podinfo
time="2024-06-26T10:08:32Z" level=info msg="Patched conditions: {\"status\":{\"conditions\":[{\"lastTransitionTime\":\"2024-06-11T20:58:09Z\",\"lastUpdateTime\":\"2024-06-11T20:58:09Z\",\"message\":\"Rollout has minimum availability\",\"reason\":\"AvailableReason\",\"sta>
time="2024-06-26T10:08:32Z" level=info msg="Updating replica set 'podinfo-587d96649f' revision from 0 to 282" namespace=canary-analysis rollout=podinfo
time="2024-06-26T10:08:32Z" level=info msg="Enqueueing parent of canary-analysis/podinfo-587d96649f: Rollout canary-analysis/podinfo"
time="2024-06-26T10:08:32Z" level=info msg="Created ReplicaSet podinfo-587d96649f" namespace=canary-analysis rollout=podinfo
time="2024-06-26T10:08:32Z" level=info msg="Enqueueing parent of canary-analysis/podinfo-587d96649f: Rollout canary-analysis/podinfo"
time="2024-06-26T10:08:32Z" level=error msg="Error: updating rollout revision" error="Operation cannot be fulfilled on rollouts.argoproj.io \"podinfo\": the object has been modified; please apply your changes to the latest version and try again" namespace=canary-analysis>
time="2024-06-26T10:08:32Z" level=error msg="roCtx.reconcile err failed to getAllReplicaSetsAndSyncRevision in rolloutCanary create true: Operation cannot be fulfilled on rollouts.argoproj.io \"podinfo\": the object has been modified; please apply your changes to the lat>
time="2024-06-26T10:08:32Z" level=info msg="Reconciliation completed" generation=229 namespace=canary-analysis resourceVersion=392564479 rollout=podinfo time_ms=65.382286
time="2024-06-26T10:08:32Z" level=error msg="rollout syncHandler error: failed to getAllReplicaSetsAndSyncRevision in rolloutCanary create true: Operation cannot be fulfilled on rollouts.argoproj.io \"podinfo\": the object has been modified; please apply your changes to >
time="2024-06-26T10:08:32Z" level=info msg="rollout syncHandler queue retries: 5 : key \"canary-analysis/podinfo\"" namespace=canary-analysis rollout=podinfo
time="2024-06-26T10:08:32Z" level=error msg="failed to getAllReplicaSetsAndSyncRevision in rolloutCanary create true: Operation cannot be fulfilled on rollouts.argoproj.io \"podinfo\": the object has been modified; please apply your changes to the latest version and try >
time="2024-06-26T10:08:32Z" level=info msg="Started syncing rollout" generation=229 namespace=canary-analysis resourceVersion=392564480 rollout=podinfo
time="2024-06-26T10:08:32Z" level=info msg="Rollout updated to revision 282" event_reason=RolloutUpdated namespace=canary-analysis rollout=podinfo
zachaller commented 4 months ago

@jccastillocano conflicts are generally not an issue and a normal k8s pattern, the conflict here is on the rollout resource. The patch in the issue is just on conflicts related to replicasets. This is to insure that we can always update the scale of the replicaset and avoids conflict loops. We have not seen any loops on conflicts on Rollout resources.

crenshaw-dev commented 1 month ago

This issue included information on a lot of different bugs. The Intuit team encountered one bug today, so I'll share the details of that particular problem:

When a Rollout starts, Argo kicks off a timeout counter. We require that, before that timer expires, the number replicas running in the new ReplicaSet is equal to the sum of all replicas running in all ReplicaSets (i.e. everything is scaled to where it should be scaled).

Argo relies on a cache of all ReplicaSet objects to calculate how many replicas each is running. That cache is maintained by handling an event stream from the Kubernetes API.

If an event fails to be delivered, Argo Rollouts can retain a stale replica count for an old ReplicaSet. In this case, it retained a replica count of 1 for the old ReplicaSet. So Argo Rollouts believed 4 Pods were running instead of the desired 3. There is no internal mechanism to clear this stale data.

Once the progressing deadline is exceeded due to the stale data, we mark the Rollout as Degraded. It will stay that way until the stale data is cleared.

Workarounds include:

  • Restart the Rollouts controller. This will re-list all ReplicaSets and get up-to-date data.
  • Delete old ReplicaSets (starting with the oldest and moving to the newest non-active ReplicaSet) until the Rollout becomes healthy. By deleting the stale ReplicaSet, you force a re-calculation of running replicas excluding the stale ReplicaSet. This strategy can be problematic. If you delete the second most-recent ReplicaSet, you complicate the rollback process if an issue is encountered with the newest revision.

We could build a mechanism into Argo Rollouts to re-list ReplicaSets occasionally and clear stale data. That would be a significant refactor.

Update: we've found that making an insignificant edit on old ReplicaSets instead of deleting them is a safer and equally effective way to clear the issue. I just added annotation test: test.

nebojsa-prodana commented 3 weeks ago

Update: we've found that making an insignificant edit on old ReplicaSets instead of deleting them is a safer and equally effective way to clear the issue. I just added annotation test: test.

Can this be implemented in argo-rollouts? For example, can an annotation change to old RS be made on a cron, or can it be hidden behind a flag that needs to be enabled by the users?

Alternatively, would lowering revisionHistoryLimit or setting it to 0 make this problem go away?