Cannot sync https gateway when change ingress host+cert

[zzjin]

Updated: It seems related to fallbackHttps feature:

apiVersion: v1
data:
  cert: |
    automaticHttps: false
    fallbackForInvalidSecret: true # <- change to false, problem resolved.
    acmeIssuer:
    - email: cloud@sealos.io
      name: letsencrypt
    renewBeforeDays: 1
    credentialConfig:
    - domains:
        - '*.example.com'
        - 'example.com'
      tlsSecret: sealos-system/wildcard-cert
kind: ConfigMap
metadata:
  name: higress-https
  namespace: higress-system

---

Higress cannot sync/update ingress https route, when change one ingress's host and tls name.

Base ingress:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: network-hgrzpprwzkyc
  annotations:
    kubernetes.io/ingress.class: nginx
    nginx.ingress.kubernetes.io/ssl-redirect: 'false'
    nginx.ingress.kubernetes.io/backend-protocol: HTTP
spec:
  rules:
    - host: xyfpethumfvc.example.com
      http:
        paths:
          - pathType: Prefix
            path: /
            backend:
              service:
                name: hello-world
                port:
                  number: 80
  tls:
    - hosts:
        - xyfpethumfvc.example.com
      secretName: wildcard-cert

Change to:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: network-hgrzpprwzkyc
  annotations:
    kubernetes.io/ingress.class: nginx
    nginx.ingress.kubernetes.io/ssl-redirect: 'false'
    nginx.ingress.kubernetes.io/backend-protocol: HTTP
spec:
  rules:
    - host: test.custom.com
      http:
        paths:
          - pathType: Prefix
            path: /
            backend:
              service:
                name: hello-world
                port:
                  number: 80
  tls:
    - hosts:
        - test.custom.com
      secretName: network-hgrzpprwzkyc
---
apiVersion: cert-manager.io/v1
kind: Issuer
metadata:
  name: network-hgrzpprwzkyc
spec:
  acme:
    server: https://acme-v02.api.letsencrypt.org/directory
    email: admin@sealos.io
    privateKeySecretRef:
      name: letsencrypt-prod
    solvers:
      - http01:
          ingress:
            class: nginx
            serviceType: ClusterIP
---
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: network-hgrzpprwzkyc
spec:
  secretName: network-hgrzpprwzkyc
  dnsNames:
    - test.custom.com
  issuerRef:
    name: network-hgrzpprwzkyc
    kind: Issuer

And apply. higress can generate right http gateway rds, but cannot do https.

Restart higress-controller, https gateway gens right.

command curl localhost:15014/debug/configz to higress-controller shows:

    {
        "kind": "Gateway",
        "apiVersion": "networking.istio.io/v1alpha3",
        "metadata": {
            "name": "89442f11352e0fad206069b57c830d75",
            "namespace": "higress-system",
            "resourceVersion": "2024-08-21 09:32:10.6211316 +0000 UTC m=+1401.924797401",
            "creationTimestamp": null,
            "annotations": {
                "ResourceVersion": ""
            }
        },
        "spec": {
            "selector": {
                "higress": "higress-system-higress-gateway"
            },
            "servers": [
                {
                    "hosts": [
                        "test.custom.com"
                    ],
                    "port": {
                        "name": "http-80-ingress",
                        "number": 80,
                        "protocol": "HTTP"
                    }
                }
            ]
        }
    },

after restart higress-controller, config shows:

{
        "kind": "Gateway",
        "apiVersion": "networking.istio.io/v1alpha3",
        "metadata": {
            "name": "fde34efa8fc3871cbf369a0208c52baa",
            "namespace": "higress-system",
            "resourceVersion": "2024-08-21 09:41:39.64467156 +0000 UTC m=+1970.948337361",
            "creationTimestamp": "2024-08-21T09:41:39Z"
        },
        "spec": {
            "selector": {
                "higress": "higress-system-higress-gateway"
            },
            "servers": [
                {
                    "hosts": [
                        "test.custom.com"
                    ],
                    "port": {
                        "name": "http-80-ingress",
                        "number": 80,
                        "protocol": "HTTP"
                    }
                },
                {
                    "hosts": [
                        "test.custom.com"
                    ],
                    "port": {
                        "name": "https-443-ingress",
                        "number": 443,
                        "protocol": "HTTPS"
                    },
                    "tls": {
                        "credentialName": "kubernetes-ingress://Kubernetes/ns-admin/network-hgrzpprwzkyc",
                        "mode": "SIMPLE"
                    }
                }
            ]
        }
    },

---

higress version: 1.4.2 kubernetes version: 1.27.1

[zijiren233]

After testing, it was found that the issue was not related to ns/certname. The problem was resolved by changing fallbackForInvalidSecret from true to false

[johnlanni]

cc @2456868764

[2456868764]

当 fallbackForInvalidSecret 为 True时候，会查找当前 Ingress 的 namespace 下 ， secretName 对应的 secret 是否存在，如果不存在，再到 higress-https configmap 根据 hostName查找。所以当 Ingress 变更 hostName 和 secretName 时候，对应secret 是否已经存在？cert-manager 用 letsencrypt 生成 secret 有秒级别延时。

[zijiren233]

对应的secret是异步申请的，创建ingress的时候并不存在，而是一段时间后才被创建出来，secret生成后对应的Gateway中并没有增加https的内容，重启controller后恢复

[2456868764]

复现情况用 cert-manager selfsign 模式：

1. create cluster issuer

apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: k8s-ca-issuer
spec:
  ca:
    secretName: ca-key-pair

2. 测试 test.yaml 如下：

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: network-hgrzpprwzkyc
  annotations:
    nginx.ingress.kubernetes.io/ssl-redirect: 'false'
    nginx.ingress.kubernetes.io/backend-protocol: HTTP
spec:
  ingressClassName: higress
  rules:
    - host: test.custom.com
      http:
        paths:
          - pathType: Prefix
            path: /
            backend:
              service:
                name: httpbin
                port:
                  number: 80
  tls:
    - hosts:
        - test.custom.com
      secretName: network-hgrzpprwzkyc
---
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: network-hgrzpprwzkyc
spec:
  secretName: network-hgrzpprwzkyc
  dnsNames:
    - test.custom.com
  issuerRef:
    name: k8s-ca-issuer ### 指定为自签名发行人
    kind: ClusterIssuer

一起 apply

kubectl apply -f test.yaml -n ns-admin

3. 调试 controller，增加日志输出：发现 ingress apply 时刻是找不着这个secret： network-hgrzpprwzkyc

   

4. 重新打开 127.0.0.1:8888/debug/configz 接口刷新一下， 发现就可以发现 secretName

   

5.这里代码逻辑如下: