Open DanTulovsky opened 3 years ago
DanTulovsky
commented
3 years ago Additionally, the 'Steps' currently all show as complete. So I have the new Revision 7, the previous upgrade at Revision 6 and the previous stable at Revision 5.
What seems to be happening now is that all pods are being moved from Revision 6 to Revision 7 very quickly, without pausing. Revision 5 pods are not being touched yet.
Strategy: Canary
Step: 14/14
SetWeight: 100
ActualWeight: 100
jessesuen
commented
3 years ago Did you happen to go back to the previously deployed version or all these pod specs different from each other?
DanTulovsky
commented
3 years ago No... it was going forward..
On Mon, Jul 26, 2021, at 4:31 PM, Jesse Suen wrote:
Did you happen to go back to the previously deployed version?
— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/argoproj/argo-rollouts/issues/1372#issuecomment-887006338, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAD4PP3OZQNYNLOQPBXCTPLTZXA2LANCNFSM5BAVRREA.
alexmt
commented
3 years ago I can reliable reproduce a similar issues with replicaset using the rollout + template below. Getting the following error:
ERRO[2021-08-25T12:46:30-07:00] roCtx.reconcile err Operation cannot be fulfilled on replicasets.apps "rollout-analysis-step-6d9cf96448": the object has been modified; please apply your changes to the latest version and try again generation=2 namespace=default resourceVersion=497932 rollout=rollout-analysis-stepapiVersion: argoproj.io/v1alpha1
kind: AnalysisTemplate
metadata:
name: success-rate-qal-usw2
spec:
args:
- name: analysis-interval
value: "300s"
- name: analysis-runs
value: "36"
- name: failure-limit
value: "4"
- name: inconclusive-limit
value: "4"
- name: secret-url
valueFrom:
secretKeyRef:
name: example-secret
key: secretUrl
metrics:
- name: webmetric
successCondition: result == 'It worked!'
count: "{{args.analysis-runs}}"
interval: "{{args.analysis-interval}}"
failureLimit: "{{args.failure-limit}}"
provider:
web:
# placeholders are resolved when an AnalysisRun is created
url: "{{args.secret-url}}"
jsonPath: "{$.message}"
---
apiVersion: argoproj.io/v1alpha1
kind: Rollout
metadata:
name: rollout-analysis-step
spec:
replicas: 4
revisionHistoryLimit: 2
selector:
matchLabels:
app: rollout-analysis-step
template:
metadata:
labels:
app: rollout-analysis-step
spec:
containers:
- name: rollouts-demo
image: argoproj/rollouts-demo:blue
imagePullPolicy: Always
ports:
- containerPort: 8080
strategy:
canary:
steps:
- setWeight: 25
# An AnalysisTemplate is referenced at the second step, which starts an AnalysisRun after
# the setWeight step. The rollout will not progress to the following step until the
# AnalysisRun is complete. A failure/error of the analysis will cause the rollout's update to
# abort, and set the canary weight to zero.
- analysis:
templates:
- templateName: success-rate-qal-usw2Issue is reproducible locally with k3s and rollout controller version: https://github.com/argoproj/argo-rollouts/commit/a601a0c18e657ae81a0269b229767dcbe9bbf56a
github-actions[bot]
commented
1 year ago This issue is stale because it has been open 60 days with no activity.
jayadeep-saaslabs
commented
7 months ago I faced same issue in our deployments. argo rollout status being "more replicas need to be updated" and in controller logs "Skip scale down of older RS 'service-123asd': still referenced" say this error
ju187
commented
7 months ago Hit this error too. The rollout stuck at progressing waiting for more replicas
Summary
I had a slow rollout progressing through about 8 steps, with a 30 min wait between each. Before it was finished, I pushed another change. This change create a new Revision, but then it got stuck. No pods were upgraded to the new revision, the previous revision and stable versions stopped progressing.
I unstuck it by running
promotevia the CLI tool. I ran it once, and the UI showed that it started the step over again. I then had to promote it through the entire workflow. No changes were being made to the newest revision. Once I went through the entire Rollout like this, the new revision started to upgrade.Diagnostics
1.0.2
Logs for when in broken state:
Message from the maintainers:
Impacted by this bug? Give it a 👍. We prioritize the issues with the most 👍.