implement a more A/B testing style rolling upgrade using Ingress or Traefik?

[jstrachan]

Right now the Rolling Updates in Deployments is based purely on pod numbers.

I wonder how hard it'd be for folks to configure a Microservice to use a more sophisticated A/B style rolling update; such as a %age of the traffic. So that the new version gets X% of the traffic and rather than doing a rolling update of the pods, we do a rolling update of the %age of traffic? The underlying old and new ReplicaSets could then be autoscaled based on HTTP traffic; then we just do a rolling update of the traffic %age?

So I'm wondering if the CI / CD flow could perform a regular Deployment change; where the old and new RS are auto-scaled based on HTTP traffic; but that rather than using a regular Kubernetes Service, we introduce a Kubernetes Service for Traefik which then uses weighting rules to route between the 2 deployments.

BTW am not sure if Ingress will ever have the ability to specify routing %ages / limits of a single host to multiple service implementations? It'd be nice to be able to do all this via just pure Ingress :). e.g. adding the weightings to the Ingress as an annotation and have a Traefik implementation of ingress that watches the Ingress?

[rawlingsj]

I've not had a look yet but I've just had a notification form https://github.com/nginxinc/kubernetes-ingress/issues/21#issuecomment-222837139 which shows how to override the nginx ingress controller config using a configmap. I wonder if there's something there where we could add weighings.

[jstrachan]

@rawlingsj ah thanks! It'd be really nice if we could just annotate the Ingress really! After re-reading the ingress docs it looks like one day that might happen. But until then, putting the nginx config into a ConfigMap would be a nice simple solution ;)

[jstrachan]

though I don't see yet how to use that nginx ConfigMap to specify weightings between a single host and multiple service back ends...

[jstrachan]

BTW I'm not even sure if an Ingress resource can be used to route the same host/path to multiple back end services either

[jimmidyson]

How about having a service target pods managed by 2 separate ReplicaSets with a common label subset? So you get the weighting by virtue of the ratio of pods managed by each RS?

[jstrachan]

@jimmidyson Agreed. If you're happy to use the ratio of pods for the old & new RS implementations then the current Deployment / Rolling Update works a treat!

The main issue though is if folks are only running a couple of pods and want to try, say, 1% of traffic to the new version (maybe only for a certain subset of users or whatever) then the current service-with-label selector approach doesn't work that well as its too course grained.

I guess TL;DR; this issue is about having more fine grained weighting/load balancing between different versions of a service than just the pod counts

[jimmidyson]

See https://github.com/kubernetes/kubernetes/issues/25485

[jimmidyson]

And this comment https://github.com/kubernetes/kubernetes/issues/16891#issuecomment-176879744

[jstrachan]

Thanks for those links @jimmidyson!

Given the differences in algorithms, configurations and load balancer implementations its looking like it may be a while before there's a LCD set of metadata we can add to Ingress. So its maybe simpler for now to look at using a Traefik service for the fine grained A/B load balancing weightings; where the configuration could be specified in a ConfigMap?

[jstrachan]

ah it looks like weightings and A/B load balancing across services are coming to OpenShift soon:

• https://github.com/openshift/origin/pull/8854
• https://github.com/openshift/origin/pull/9070

[bgrant0607]

Also provided by: https://www.amalgam8.io/

[jstrachan]

@bgrant0607 thanks for the heads up, i'll take a look!

[bgrant0607]

Another relevant issue: https://github.com/kubernetes/kubernetes/issues/17686

[errordeveloper]

Alos provided by Istio.

[errordeveloper]

I've had some thoughts on this, and concluded that extending the deployment object may be a little too complicated, I'd rather consider allowing for DeploymentStrategy to refer to a CRD-based controller that takes care of managing the scale of old vs new replicasets in any way user might like. But as @jstrachan pointed out above, using the ratio of pods is quite a limited method, it's too naive to try managing how incoming traffic would be split, although it could work fine for worker type of applications that don't actually serve clients as such, e.g. you have 5 pods that continuously get data from another service and do some work, and you just want to have like 1 out of 5 pods use modified code or config and see how it behaves.