argoproj-labs / rollouts-plugin-trafficrouter-openshift

Apache License 2.0
1 stars 4 forks source link

route weights don't get reset to original values. #14

Open nsalfos opened 3 weeks ago

nsalfos commented 3 weeks ago

Hi Folks, i was able to get the plugin running after building it myself (the latest release on your repo doesn't work properly apparently) now, when testing the rollout the first time it worked great, and i was able to see traffic shifting and also the weights being changed as i promoted the rollout. but when deleting the rollout and creating one again the route weights remain whatever the last value was before completing the promotion.

even after deleting the route and creating it again with stable: 100 / canary: 0 now is not changing the weights. so this only worked once for me and not sure why.

I saw another issue similar to this one, i which you tried steps to reproduce but it worked fine for you. Is it strictly necesary to have "Red Hat OpenShift GitOps" operator installed for this to work as expected??

last time i saw "new weights" being set on was on the first try with working plugin as metnioned ``` > k logs -n argo-rollouts argo-rollouts-754c744cf6-m9tfh | grep "New weights" time="2024-10-22T17:06:38Z" level=info msg="New weights: &TrafficWeights{Canary:WeightDestination{Weight:0,ServiceName:argo-rollouts-canary-service,PodTemplateHash:687d76d795,},Stable:WeightDestination{Weight:100,ServiceName:argo-rollouts-stable-service,PodTemplateHash:,},Additional:[]WeightDestination{},Verified:nil,}" namespace=nsalfos-triage rollout=rollouts-demo time="2024-10-22T17:06:39Z" level=info msg="New weights: &TrafficWeights{Canary:WeightDestination{Weight:0,ServiceName:argo-rollouts-canary-service,PodTemplateHash:687d76d795,},Stable:WeightDestination{Weight:100,ServiceName:argo-rollouts-stable-service,PodTemplateHash:687d76d795,},Additional:[]WeightDestination{},Verified:nil,}" namespace=nsalfos-triage rollout=rollouts-demo time="2024-10-22T17:07:01Z" level=info msg="New weights: &TrafficWeights{Canary:WeightDestination{Weight:0,ServiceName:argo-rollouts-canary-service,PodTemplateHash:6cf78c66c5,},Stable:WeightDestination{Weight:100,ServiceName:argo-rollouts-stable-service,PodTemplateHash:687d76d795,},Additional:[]WeightDestination{},Verified:nil,}" namespace=nsalfos-triage rollout=rollouts-demo time="2024-10-22T17:07:02Z" level=info msg="New weights: &TrafficWeights{Canary:WeightDestination{Weight:25,ServiceName:argo-rollouts-canary-service,PodTemplateHash:6cf78c66c5,},Stable:WeightDestination{Weight:75,ServiceName:argo-rollouts-stable-service,PodTemplateHash:687d76d795,},Additional:[]WeightDestination{},Verified:nil,}" namespace=nsalfos-triage rollout=rollouts-demo time="2024-10-22T17:09:30Z" level=info msg="New weights: &TrafficWeights{Canary:WeightDestination{Weight:75,ServiceName:argo-rollouts-canary-service,PodTemplateHash:6cf78c66c5,},Stable:WeightDestination{Weight:25,ServiceName:argo-rollouts-stable-service,PodTemplateHash:687d76d795,},Additional:[]WeightDestination{},Verified:nil,}" namespace=nsalfos-triage rollout=rollouts-demo ```
Full scenario showing steps
Step 1 - no rollout, route state after deletion of previous rollouts ``` > k argo rollouts get rollout rollouts-demo --watch Error: rollout.argoproj.io "rollouts-demo" not found > k describe route rollouts-demo-route Name: rollouts-demo-route Namespace: nsalfos-triage Labels: Annotations: openshift.io/host.generated: true API Version: route.openshift.io/v1 Kind: Route Metadata: Creation Timestamp: 2024-10-22T09:06:48Z Resource Version: 1310740253 UID: ca882b51-823a-4cd3-9b7a-1bf1762a7879 Spec: Alternate Backends: Kind: Service Name: argo-rollouts-canary-service Weight: 75 Host: Port: Target Port: http Tls: Insecure Edge Termination Policy: Redirect Termination: edge To: Kind: Service Name: argo-rollouts-stable-service Weight: 25 Wildcard Policy: None Status: Ingress: Conditions: Last Transition Time: 2024-10-22T09:06:48Z Status: True Type: Admitted Host: Router Name: router Wildcard Policy: None Conditions: Last Transition Time: 2024-10-22T09:06:48Z Status: True Type: Admitted Host: Router Canonical Hostname: Router Name: default Wildcard Policy: None Events: ```
Step 2- new rollout ``` > oc create -f ocp-route-plugin/rollout-cr.yaml rollout.argoproj.io/rollouts-demo created > k argo rollouts get rollout rollouts-demo && k get route rollouts-demo-route Name: rollouts-demo Namespace: nsalfos-triage Status: ✔ Healthy Strategy: Canary Step: 4/4 SetWeight: 100 ActualWeight: 100 Images: argoproj/rollouts-demo:blue (stable) Replicas: Desired: 4 Current: 4 Updated: 4 Ready: 4 Available: 4 NAME KIND STATUS AGE INFO ⟳ rollouts-demo Rollout ✔ Healthy 57s └──# revision:1 └──⧉ rollouts-demo-687d76d795 ReplicaSet ✔ Healthy 57s stable ├──□ rollouts-demo-687d76d795-2xrbs Pod ✔ Running 57s ready:1/1 ├──□ rollouts-demo-687d76d795-4zfkg Pod ✔ Running 57s ready:1/1 ├──□ rollouts-demo-687d76d795-kd5nr Pod ✔ Running 57s ready:1/1 └──□ rollouts-demo-687d76d795-tpfgd Pod ✔ Running 57s ready:1/1 NAME HOST/PORT PATH SERVICES PORT TERMINATION WILDCARD rollouts-demo-route argo-rollouts-stable-service(25%),argo-rollouts-canary-service(75%) http edge/Redirect None ```
Step 3 - changing image no changes to route. ``` > k argo rollouts get rollout rollouts-demo && k get route rollouts-demo-route Name: rollouts-demo Namespace: nsalfos-triage Status: ॥ Paused Message: CanaryPauseStep Strategy: Canary Step: 1/4 SetWeight: 25 ActualWeight: 25 Images: argoproj/rollouts-demo:blue (stable) argoproj/rollouts-demo:yellow (canary) Replicas: Desired: 4 Current: 4 Updated: 1 Ready: 4 Available: 4 NAME KIND STATUS AGE INFO ⟳ rollouts-demo Rollout ॥ Paused 2m56s ├──# revision:2 │ └──⧉ rollouts-demo-6cf78c66c5 ReplicaSet ✔ Healthy 3s canary │ └──□ rollouts-demo-6cf78c66c5-tfnsv Pod ✔ Running 3s ready:1/1 └──# revision:1 └──⧉ rollouts-demo-687d76d795 ReplicaSet ✔ Healthy 2m56s stable ├──□ rollouts-demo-687d76d795-2xrbs Pod ✔ Running 2m56s ready:1/1 ├──□ rollouts-demo-687d76d795-4zfkg Pod ◌ Terminating 2m56s ready:1/1 ├──□ rollouts-demo-687d76d795-kd5nr Pod ✔ Running 2m56s ready:1/1 └──□ rollouts-demo-687d76d795-tpfgd Pod ✔ Running 2m56s ready:1/1 NAME HOST/PORT PATH SERVICES PORT TERMINATION WILDCARD rollouts-demo-route argo-rollouts-stable-service(25%),argo-rollouts-canary-service(75%) http edge/Redirect None ```
Step 4 - fully promoting ``` > k argo rollouts get rollout rollouts-demo && k get route rollouts-demo-route Name: rollouts-demo Namespace: nsalfos-triage Status: ✔ Healthy Strategy: Canary Step: 4/4 SetWeight: 100 ActualWeight: 100 Images: argoproj/rollouts-demo:yellow (stable) Replicas: Desired: 4 Current: 4 Updated: 4 Ready: 4 Available: 4 NAME KIND STATUS AGE INFO ⟳ rollouts-demo Rollout ✔ Healthy 17m ├──# revision:2 │ └──⧉ rollouts-demo-6cf78c66c5 ReplicaSet ✔ Healthy 14m stable │ ├──□ rollouts-demo-6cf78c66c5-tfnsv Pod ✔ Running 14m ready:1/1 │ ├──□ rollouts-demo-6cf78c66c5-cfswh Pod ✔ Running 12m ready:1/1 │ ├──□ rollouts-demo-6cf78c66c5-xhqv8 Pod ✔ Running 12m ready:1/1 │ └──□ rollouts-demo-6cf78c66c5-pb2rc Pod ✔ Running 7s ready:1/1 └──# revision:1 └──⧉ rollouts-demo-687d76d795 ReplicaSet • ScaledDown 17m └──□ rollouts-demo-687d76d795-kd5nr Pod ◌ Terminating 17m ready:1/1 NAME HOST/PORT PATH SERVICES PORT TERMINATION WILDCARD rollouts-demo-route argo-rollouts-stable-service(25%),argo-rollouts-canary-service(75%) http edge/Redirect None ```
New rollout, new route ``` > oc create -f ocp-route-plugin/route.yaml route.route.openshift.io/rollouts-demo-route created > k argo rollouts get rollout rollouts-demo; k get route rollouts-demo-route Error: rollout.argoproj.io "rollouts-demo" not found NAME HOST/PORT PATH SERVICES PORT TERMINATION WILDCARD rollouts-demo-route argo-rollouts-stable-service(100%),argo-rollouts-canary-service(0%) http edge/Redirect None > oc create -f ocp-route-plugin/rollout-cr.yaml rollout.argoproj.io/rollouts-demo created > k argo rollouts get rollout rollouts-demo; k get route rollouts-demo-route Name: rollouts-demo Namespace: nsalfos-triage Status: ✔ Healthy Strategy: Canary Step: 4/4 SetWeight: 100 ActualWeight: 100 Images: argoproj/rollouts-demo:blue (stable) Replicas: Desired: 4 Current: 4 Updated: 4 Ready: 4 Available: 4 NAME KIND STATUS AGE INFO ⟳ rollouts-demo Rollout ✔ Healthy 3s └──# revision:1 └──⧉ rollouts-demo-687d76d795 ReplicaSet ✔ Healthy 3s stable ├──□ rollouts-demo-687d76d795-cbhrh Pod ✔ Running 3s ready:1/1 ├──□ rollouts-demo-687d76d795-kh4g6 Pod ✔ Running 3s ready:1/1 ├──□ rollouts-demo-687d76d795-v5hwb Pod ✔ Running 3s ready:1/1 └──□ rollouts-demo-687d76d795-v8k5x Pod ✔ Running 3s ready:1/1 NAME HOST/PORT PATH SERVICES PORT TERMINATION WILDCARD rollouts-demo-route argo-rollouts-stable-service(100%),argo-rollouts-canary-service(0%) http edge/Redirect None > kubectl argo rollouts set image rollouts-demo \ rollouts-demo=argoproj/rollouts-demo:yellow rollout "rollouts-demo" image updated > k argo rollouts get rollout rollouts-demo; k get route rollouts-demo-route Name: rollouts-demo Namespace: nsalfos-triage Status: ॥ Paused Message: CanaryPauseStep Strategy: Canary Step: 1/4 SetWeight: 25 ActualWeight: 25 Images: argoproj/rollouts-demo:blue (stable) argoproj/rollouts-demo:yellow (canary) Replicas: Desired: 4 Current: 4 Updated: 1 Ready: 4 Available: 4 NAME KIND STATUS AGE INFO ⟳ rollouts-demo Rollout ॥ Paused 18s ├──# revision:2 │ └──⧉ rollouts-demo-6cf78c66c5 ReplicaSet ✔ Healthy 4s canary │ └──□ rollouts-demo-6cf78c66c5-k2pxz Pod ✔ Running 3s ready:1/1 └──# revision:1 └──⧉ rollouts-demo-687d76d795 ReplicaSet ✔ Healthy 18s stable ├──□ rollouts-demo-687d76d795-cbhrh Pod ✔ Running 18s ready:1/1 ├──□ rollouts-demo-687d76d795-kh4g6 Pod ✔ Running 18s ready:1/1 ├──□ rollouts-demo-687d76d795-v5hwb Pod ✔ Running 18s ready:1/1 └──□ rollouts-demo-687d76d795-v8k5x Pod ◌ Terminating 18s ready:1/1 NAME HOST/PORT PATH SERVICES PORT TERMINATION WILDCARD rollouts-demo-route argo-rollouts-stable-service(100%),argo-rollouts-canary-service(0%) http edge/Redirect None > k argo rollouts promote rollouts-demo rollout 'rollouts-demo' promoted > k argo rollouts get rollout rollouts-demo; k get route rollouts-demo-route Name: rollouts-demo Namespace: nsalfos-triage Status: ॥ Paused Message: CanaryPauseStep Strategy: Canary Step: 3/4 SetWeight: 75 ActualWeight: 75 Images: argoproj/rollouts-demo:blue (stable) argoproj/rollouts-demo:yellow (canary) Replicas: Desired: 4 Current: 4 Updated: 3 Ready: 4 Available: 4 NAME KIND STATUS AGE INFO ⟳ rollouts-demo Rollout ॥ Paused 6m42s ├──# revision:2 │ └──⧉ rollouts-demo-6cf78c66c5 ReplicaSet ✔ Healthy 6m28s canary │ ├──□ rollouts-demo-6cf78c66c5-k2pxz Pod ✔ Running 6m27s ready:1/1 │ ├──□ rollouts-demo-6cf78c66c5-bwpmq Pod ✔ Running 3s ready:1/1 │ └──□ rollouts-demo-6cf78c66c5-rvt2f Pod ✔ Running 3s ready:1/1 └──# revision:1 └──⧉ rollouts-demo-687d76d795 ReplicaSet ✔ Healthy 6m42s stable ├──□ rollouts-demo-687d76d795-cbhrh Pod ✔ Running 6m42s ready:1/1 ├──□ rollouts-demo-687d76d795-kh4g6 Pod ◌ Terminating 6m42s ready:1/1 └──□ rollouts-demo-687d76d795-v5hwb Pod ◌ Terminating 6m42s ready:1/1 NAME HOST/PORT PATH SERVICES PORT TERMINATION WILDCARD rollouts-demo-route argo-rollouts-stable-service(100%),argo-rollouts-canary-service(0%) http edge/Redirect None > k argo rollouts get rollout rollouts-demo; k get route rollouts-demo-route Name: rollouts-demo Namespace: nsalfos-triage Status: ✔ Healthy Strategy: Canary Step: 4/4 SetWeight: 100 ActualWeight: 100 Images: argoproj/rollouts-demo:yellow (stable) Replicas: Desired: 4 Current: 4 Updated: 4 Ready: 4 Available: 4 NAME KIND STATUS AGE INFO ⟳ rollouts-demo Rollout ✔ Healthy 6m52s ├──# revision:2 │ └──⧉ rollouts-demo-6cf78c66c5 ReplicaSet ✔ Healthy 6m38s stable │ ├──□ rollouts-demo-6cf78c66c5-k2pxz Pod ✔ Running 6m37s ready:1/1 │ ├──□ rollouts-demo-6cf78c66c5-bwpmq Pod ✔ Running 13s ready:1/1 │ ├──□ rollouts-demo-6cf78c66c5-rvt2f Pod ✔ Running 13s ready:1/1 │ └──□ rollouts-demo-6cf78c66c5-pp5gp Pod ✔ Running 2s ready:1/1 └──# revision:1 └──⧉ rollouts-demo-687d76d795 ReplicaSet • ScaledDown 6m52s └──□ rollouts-demo-687d76d795-cbhrh Pod ◌ Terminating 6m52s ready:1/1 NAME HOST/PORT PATH SERVICES PORT TERMINATION WILDCARD rollouts-demo-route argo-rollouts-stable-service(100%),argo-rollouts-canary-service(0%) http edge/Redirect None ```
nsalfos commented 1 week ago

anyone around?