Open schlags opened 3 months ago
Unless I'm mistaken, this fix sounds like it's addressing a similar type of case where the rollout controller setting traffic to 100% isn't allowing enough time for it to verify the weight behind the scenes with the ALB.
If this was happening for the default behavior of the end of a rollout, I feel like it tracks that this issue where the default behavior of a preempted rollout would be needing a similar fix.
Checklist:
Describe the bug
We are using the ArgoCD Image Updater to set image parameters for our application.
We are using the ALB ingress class, with the
pingPong
strategyWe are running Rollouts v1.6.6
We have readiness-gate-injection enabled on the pods in our namespace where this takes place
We have observed, very frequently, that if a new image is set for the rollout while it is progressing, it causes 503 responses to clients that were previously shaped to the canary at that time.
It typically goes like the following.
Assume
image A
is the stable,image B
is currently canarying, andimage C
is set whileimage B
is still progressing.scaling down intermediate RS
is written// It is safe to scale the intermediate RS down, since no traffic is directed to it.
but the following disproves that a bitI am wondering if we are missing something in our traffic shaping configuration to address this downtime on preemption in rollouts.
To Reproduce
Expected behavior
All ingress rules should be confirmed to be reset to the state at the beginning of the canary before a new replicaset has gets set for the service selector. (Because otherwise, the ingress would send that percentage of traffic to a service/target group with no assigned IP's - hence the 503 response).
Towards the end of the log output below, you can see that we observe the selector swapping on the service to the new replicaset before restoring the desiredWeight to 0. I believe this is a bug for this reason, but open to ideas on how I'm approaching this incorrectly if this is the intended behavior.
Shouldn't the ingress be set to desired weight 0 before the service selector moves to a replicaset that hasn't even scaled up yet?
Screenshots
Version
v1.6.6
Logs
Message from the maintainers:
Impacted by this bug? Give it a 👍. We prioritize the issues with the most 👍.