基于 ingress-nginx 实现灰度发布和蓝绿发布

本文描述如何基于 https://github.com/kubernetes/ingress-nginx 实现灰度发布和蓝绿发布功能：

灰度发布：
- 基于Request Header的流量切分
- 基于Cookie的流量切分
- 基于Query Param的流量切分
蓝绿发布：
- 根据权重进行流量切分

本文基于 controller-0.32.0 版本

修改后的代码仓库地址：https://github.com/penglongli/ingress-nginx/tree/controller-0.32.0
提供编译后的镜像使用：docker pull pelin/nginx-ingress-controller:0.32.0

概述

ingress-nginx 目前其实已经做了灰度发布、蓝绿发布，但是在使用方式上需要建多个 ingress 的方式来实现。我们目前希望能够在一个 Ingress 中即实现上述功能，则需要做一部分的开发工作。

目标

此处的目标是实现与阿里云 相同的方式，阿里云的使用方式参考：Kubernetes 集群中通过 Ingress 实现灰度发布和蓝绿发布

灰度发布

在 Ingress 建立的时候，使用如下 annotation 来实现灰度发布：

nginx.ingress.kubernetes.io/service-match: |
  # 请求匹配到 cookie 存在 c=test，则转发到 new-nginx
  # 如：curl -H 'Cookie: c=test' canary.example.com
  new-nginx: cookie("c", "test")
  # 请求匹配到 header 存在 h=test，则转发到 new-nginx
  # 如：curl -H 'h: test' canary.example.com
  new-nginx: header("h", "test")
  # 请求匹配到 查询参数 存在 q=test，则转发到 new-nginx
  # 如：curl http://canary.example.com?q=test
  new-nginx: query("q", "test")

蓝绿发布

在 Ingress 建立时，使用如下 annotation 来实现灰度发布：

nginx.ingress.kubernetes.io/service-weight: |
  new-nginx: 20, old-nginx: 80

则会出现 20% 的请求量被分配到 new-ingress。（nginx-ingress此处的请求权重分配非绝对分配，并且分配不均匀）

对比阿里云

阿里云的实现更多的是对 balancer.lua 的修改，有兴趣可以看一下其修改的 LUA 模块。

ingress-nginx 原理

后端

开启 syncIngress 同步任务[https://github.com/penglongli/ingress-nginx/blob/controller-0.32.0/internal/ingress/controller/nginx.go#L310]
初始化同步任务[https://github.com/penglongli/ingress-nginx/blob/controller-0.32.0/internal/ingress/controller/nginx.go#L139]
获取并组装 Ingress 信息[https://github.com/penglongli/ingress-nginx/blob/controller-0.32.0/internal/ingress/controller/controller.go#L135]
- 此步骤会将 Ingress 的信息拆解开，并通过 kubernetes API 获取 Service 对应的 Endpoint 地址，将其组装进来
判断本次是否需要 Reload Nginx 配置文件[https://github.com/penglongli/ingress-nginx/blob/controller-0.32.0/internal/ingress/controller/controller.go#L146]

LUA 模块

Nginx 配置里的动态 Upstream [https://github.com/penglongli/ingress-nginx/blob/controller-0.32.0/rootfs/etc/nginx/template/nginx.tmpl#L443]
LUA 动态 Upstream [https://github.com/penglongli/ingress-nginx/blob/controller-0.32.0/rootfs/etc/nginx/lua/balancer.lua#L274]
- 后端会开启一个 10246 端口，LUA 模块会请求 http://127.0.0.1:10246/configuration/backends 获取配置(此为获取 HTTP Backend 配置)

实现过程

代码修改提交：https://github.com/penglongli/ingress-nginx/commit/ad38db951224a1d2696c7d6d2298832b09becdaf

灰度

controller.go：

func (n *NGINXController) getBackendServers(ingresses []*ingress.Ingress) ([]*ingress.Backend, []*ingress.Server) {
    for _, ing := range ingresses {
        ..............................
        ..............................

        // 解析 nginx.ingress.kubernetes.io/service-match 注解，抽出灰度发布的策略，组装成 GrayStrategy 对象
        svcStrategy := make(map[string]*ingress.GrayStrategy)
        if serviceMatch := ing.Annotations["nginx.ingress.kubernetes.io/service-match"]; strings.TrimSpace(serviceMatch) != "" {
            for _, annotation := range strings.Split(serviceMatch, "\n") {
                splitServiceMatch(annotation, svcStrategy)
            }
        }

        for _, rule := range ing.Spec.Rules {
            ..............................
            ..............................

            // 设置每个 Rule(如：canary.example.com)下灰度策略的：灰度 Backend、灰度 Service
            var graySvcSlice []string
            for _, path := range rule.HTTP.Paths {
                if strategy := svcStrategy[path.Backend.ServiceName]; strategy != nil {
                    graySvcSlice = append(graySvcSlice, path.Backend.ServiceName)

                    strategy.Service = path.Backend.ServiceName
                    strategy.Backend = upstreamName(ing.Namespace, path.Backend.ServiceName, path.Backend.ServicePort)
                }
            }

            for pathIndex, path := range rule.HTTP.Paths {
                ..............................
                ..............................

                for _, loc := range server.Locations {
                    // 为第一个 Location 设置灰度策略
                    if pathIndex == 0 {
                        for _, svc := range graySvcSlice {
                            loc.GrayStrategies = append(loc.GrayStrategies, svcStrategy[svc])
                        }
                    }
                }
            }
        }
    }

    return aUpstreams, aServers
}

nginx.tmpl 模板

为每个域名（如：canary.example.com）判断是否需要灰度策略，并通过 generateTmplate() 将策略生成到 nginx.conf 配置中。
优先级：Cookie > Header > Query

{{ range $k, $v := $location.GrayStrategies }}
    {{ $condition := (buildGrayIfCondition $v) }}

    # 判断是否存在 Query 灰度策略
    {{ if (ne $condition.QueryCondition "" )}}
    if ({{ $condition.QueryCondition }}) {
        set $service_name {{ $v.Service }};
        set $proxy_upstream_name {{ $v.Backend }};
    }
    {{ end }}

    # 判断是否存在 Header 灰度策略
    {{ if (ne $condition.HeaderCondition "" )}}
    if ({{ $condition.HeaderCondition }}) {
        set $service_name {{ $v.Service }};
        set $proxy_upstream_name {{ $v.Backend }};
    }
    {{ end }}

    # 判断是否存在 Cookie 灰度策略
    {{ if (ne $condition.CookieCondition "" )}}
    if ({{ $condition.CookieCondition }}) {
        set $service_name {{ $v.Service }};
        set $proxy_upstream_name {{ $v.Backend }};
    }
    {{ end }}

{{ end }}

权重

controller.go

// createUpstreams creates the NGINX upstreams (Endpoints) for each Service
// referenced in Ingress rules.
func (n *NGINXController) createUpstreams(data []*ingress.Ingress, du *ingress.Backend) map[string]*ingress.Backend {
    upstreams := make(map[string]*ingress.Backend)
    upstreams[defUpstreamName] = du

    for _, ing := range data {
        anns := ing.ParsedAnnotations

        var defBackend string

        // 解析 nginx.ingress.kubernetes.io/service-weight 权重策略注解
        serviceWeight := splitServiceWeight(anns)
        for _, rule := range ing.Spec.Rules {
            if rule.HTTP == nil {
                continue
            }

            for i, path := range rule.HTTP.Paths {
                // 为所有非 第一个 的 Path 设置权重参数
                // 为什么“非第一个”？因为第一个会被映射到 nginx.conf 文件中的 $serviceName
                if i != 0 {
                    weight := serviceWeight[path.Backend.ServiceName]
                    if weight > 0 && weight < 100 {
                        upstreams[name].NoServer = true
                        upstreams[name].TrafficShapingPolicy = ingress.TrafficShapingPolicy{
                            Weight:        weight,
                            Header:        anns.Canary.Header,
                            HeaderValue:   anns.Canary.HeaderValue,
                            HeaderPattern: anns.Canary.HeaderPattern,
                            Cookie:        anns.Canary.Cookie,
                        }
                    }
                }
            }

            // 为第一个 Path 设置 alternativeService（此 Service 即为“替代 Service”，用于权重策略）
            if len(rule.HTTP.Paths) >= 2 {
                path := rule.HTTP.Paths[0]
                name := upstreamName(ing.Namespace, path.Backend.ServiceName, path.Backend.ServicePort)
                for i := 1; i < len(rule.HTTP.Paths); i++ {
                    if serviceWeight[path.Backend.ServiceName] != 0 {
                        upstreams[name].AlternativeBackends = append(
                            upstreams[name].AlternativeBackends,
                            upstreamName(ing.Namespace, rule.HTTP.Paths[i].Backend.ServiceName, rule.HTTP.Paths[i].Backend.ServicePort),
                        )
                    }
                }
            }
        }
    }

    return upstreams
}

调试

在 nginx-ingress-controller 服务启动后，修改代码，使用如下脚本来编译运行：

#!/bin/bash

# 编译
make build

# 拷贝 nginx-ingress-controller
docker ps | grep "/usr/bin/dumb-ini" | awk '{print $1}' | xargs -I {} docker cp bin/amd64/nginx-ingress-controller {}:/nginx-ingress-controller

# 拷贝 nginx.tmpl
docker ps | grep "/usr/bin/dumb-ini" | awk '{print $1}' | xargs -I {} docker cp rootfs/etc/nginx/template/nginx.tmpl {}:/etc/nginx/template/

# 重启 nginx-ingress-controller 容器
docker ps | grep "/usr/bin/dumb-ini" | awk '{print $1}' | xargs docker restart

使用

部署服务

首先，部署两个服务：test-python、test-nginx

# test-python 服务
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: test-python
  labels:
    app: python
  namespace: kube-system
spec:
  replicas: 1
  selector:
    matchLabels:
      app: python
  template:
    metadata:
      labels:
        app: python
    spec:
      containers:
      - name: centos
        image: centos:7
        command: ["python"]
        args:
          - "-m"
          - "SimpleHTTPServer"
          - "8080"
---
apiVersion: v1
kind: Service
metadata:
  name: test-python
  namespace: kube-system
spec:
  ports:
  - port: 8080
    protocol: TCP
    targetPort: 8080
  selector:
    app: python
  type: ClusterIP

# test-nginx 服务
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: test-nginx
  labels:
    app: nginx
  namespace: kube-system
spec:
  replicas: 2
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
      - name: nginx
        image: nginx:1.18.0
        imagePullPolicy: IfNotPresent
---
apiVersion: v1
kind: Service
metadata:
  name: test-nginx
  namespace: kube-system
spec:
  ports:
  - port: 8080
    protocol: TCP
    targetPort: 80
  selector:
    app: nginx
  type: ClusterIP

创建 Ingress

权重

---
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  annotations:
    nginx.ingress.kubernetes.io/service-weight: |
      test-python: 50, test-nginx: 50
  labels:
    app: demo
  name: demo-ingress
  namespace: kube-system
spec:
  rules:
  - host: canary.example.com
    http:
      paths:
      - backend:
          serviceName: test-python
          servicePort: 8080
        path: /
      - backend:
          serviceName: test-nginx
          servicePort: 8080
        path: /

请求域名判断是否能够按照 50% 分配到两个应用

灰度

---
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  annotations:
    nginx.ingress.kubernetes.io/service-match: |
      test-nginx: cookie("c", "test")
  labels:
    app: demo
  name: demo-ingress
  namespace: kube-system
spec:
  rules:
  - host: canary.example.com
    http:
      paths:
      - backend:
          serviceName: test-python
          servicePort: 8080
        path: /
      - backend:
          serviceName: test-nginx
          servicePort: 8080
        path: /

使用 curl http://canary.example.com -H 'Cookie: c=test' 确定请求是否能够一直落在 test-nginx 上

penglongli / blog