Open DanTulovsky opened 3 years ago
Additionally, the 'Steps' currently all show as complete. So I have the new Revision 7, the previous upgrade at Revision 6 and the previous stable at Revision 5.
What seems to be happening now is that all pods are being moved from Revision 6 to Revision 7 very quickly, without pausing. Revision 5 pods are not being touched yet.
Strategy: Canary
Step: 14/14
SetWeight: 100
ActualWeight: 100
Did you happen to go back to the previously deployed version or all these pod specs different from each other?
No... it was going forward..
On Mon, Jul 26, 2021, at 4:31 PM, Jesse Suen wrote:
Did you happen to go back to the previously deployed version?
— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/argoproj/argo-rollouts/issues/1372#issuecomment-887006338, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAD4PP3OZQNYNLOQPBXCTPLTZXA2LANCNFSM5BAVRREA.
I can reliable reproduce a similar issues with replicaset using the rollout + template below. Getting the following error:
ERRO[2021-08-25T12:46:30-07:00] roCtx.reconcile err Operation cannot be fulfilled on replicasets.apps "rollout-analysis-step-6d9cf96448": the object has been modified; please apply your changes to the latest version and try again generation=2 namespace=default resourceVersion=497932 rollout=rollout-analysis-step
apiVersion: argoproj.io/v1alpha1
kind: AnalysisTemplate
metadata:
name: success-rate-qal-usw2
spec:
args:
- name: analysis-interval
value: "300s"
- name: analysis-runs
value: "36"
- name: failure-limit
value: "4"
- name: inconclusive-limit
value: "4"
- name: secret-url
valueFrom:
secretKeyRef:
name: example-secret
key: secretUrl
metrics:
- name: webmetric
successCondition: result == 'It worked!'
count: "{{args.analysis-runs}}"
interval: "{{args.analysis-interval}}"
failureLimit: "{{args.failure-limit}}"
provider:
web:
# placeholders are resolved when an AnalysisRun is created
url: "{{args.secret-url}}"
jsonPath: "{$.message}"
---
apiVersion: argoproj.io/v1alpha1
kind: Rollout
metadata:
name: rollout-analysis-step
spec:
replicas: 4
revisionHistoryLimit: 2
selector:
matchLabels:
app: rollout-analysis-step
template:
metadata:
labels:
app: rollout-analysis-step
spec:
containers:
- name: rollouts-demo
image: argoproj/rollouts-demo:blue
imagePullPolicy: Always
ports:
- containerPort: 8080
strategy:
canary:
steps:
- setWeight: 25
# An AnalysisTemplate is referenced at the second step, which starts an AnalysisRun after
# the setWeight step. The rollout will not progress to the following step until the
# AnalysisRun is complete. A failure/error of the analysis will cause the rollout's update to
# abort, and set the canary weight to zero.
- analysis:
templates:
- templateName: success-rate-qal-usw2
Issue is reproducible locally with k3s and rollout controller version: https://github.com/argoproj/argo-rollouts/commit/a601a0c18e657ae81a0269b229767dcbe9bbf56a
This issue is stale because it has been open 60 days with no activity.
I faced same issue in our deployments. argo rollout status being "more replicas need to be updated" and in controller logs "Skip scale down of older RS 'service-123asd': still referenced" say this error
Hit this error too. The rollout stuck at progressing waiting for more replicas
Summary
I had a slow rollout progressing through about 8 steps, with a 30 min wait between each. Before it was finished, I pushed another change. This change create a new Revision, but then it got stuck. No pods were upgraded to the new revision, the previous revision and stable versions stopped progressing.
I unstuck it by running
promote
via the CLI tool. I ran it once, and the UI showed that it started the step over again. I then had to promote it through the entire workflow. No changes were being made to the newest revision. Once I went through the entire Rollout like this, the new revision started to upgrade.Diagnostics
1.0.2
Logs for when in broken state:
Message from the maintainers:
Impacted by this bug? Give it a 👍. We prioritize the issues with the most 👍.