More than one mcrt gets created

[fproulx-dfuse]

I have a support ticket on this Case 19503956. Looks like in some case more than one mcrt resource get created. One of them ends up working, but then I have some stale extra stuff.

---

kubectl apply -f demo.yaml

apiVersion: v1
kind: Namespace
metadata:
  name: demo-ingress-with-managed-cert
---
apiVersion: networking.gke.io/v1beta1
kind: ManagedCertificate
metadata:
  name: example-certificate
  namespace: demo-ingress-with-managed-cert
spec:
  domains:
    - demo.example
---
apiVersion: v1
kind: Service
metadata:
  name: example-nodeport-service
  namespace: demo-ingress-with-managed-cert
spec:
  type: NodePort
  selector:
    app: nginx
  ports:
    - protocol: TCP
      port: 80
      targetPort: 80
---
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: example-ingress
  namespace: demo-ingress-with-managed-cert
  annotations:
    networking.gke.io/managed-certificates: example-certificate
spec:
  rules:
  - host: demo.example
    http:
      paths:
      - backend:
          serviceName: example-nodeport-service
          servicePort: 80
---
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: nginx
  namespace: demo-ingress-with-managed-cert
spec:
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
      - image: nginx
        name: nginx
        ports:
        - containerPort: 80
          name: nginx
          protocol: TCP

[Cidan]

Came here to report this exact same issue. @fproulx-eoscanada are you using a regional cluster? By any chance, is the number of certs created the same as the number of zones your cluster is configured to use?

My gut here says this is related to regional clusters and the number of zones in them, but I haven't confirmed this theory. I do have two production clusters running, both regional in us-central1, both with three zone's configured, and I get 3 certs issued each time.

What's worse is, deleting the certs won't help, because GKE will just hammer away and remake them.

[fproulx-dfuse]

@Cidan yes regional. hmmm I have 2 zones in my regional cluster, but I sometimes get 3 certs.... actually there are certain cases where I simply do not appear to get this issue. For instance, when my operator (metacontroller-based) managed a namespace those work fine (it creates the ManagedCert resource and adds annotation ....). Looks like thiss happens when doing kubectl apply mostly

[fproulx-dfuse]

I'm also in us-central1

[fredrikf]

Reproduced in GKE us-east4 (3 zones) using regional clusters despite actually choosing only 1 zone.

Result: 3 certs (mcrt-*) for each ManagedCertificate resource.

Reproduced in GKE us-central1 (4 zones) using regional clusters despite actually choosing only 1 zone.

Result: 3 (not 4) certs (mcrt-*) for each ManagedCertificate resource.

Verifying GKE us-east4-a (1 zone) zonal cluster.

Result (correct): 1 certs (mcrt-*) for each ManagedCertificate resource.

[adrianchifor]

As an update, a fix has been implemented and should be available in 1.12.8-gke.8 once it's out :)

[matti]

What about 1.13 ? I'm seeing this in 1.13.6-gke.0

[matti]

in 1.13 I requested 2 certs, so I got 6 and only one works.

[matti]

googleapi: Error 400: Master version "1.12.8-gke.8" is unsupported., badRequest

1.12.8-gke.6 is the latest.

Also my second certificate eventually provisoned - now I have 3 active certs and 3 stale "PROVISIONING" certs.

[adrianchifor]

1.12.8-gke.8 is in test, should be released in the next few weeks. The fix is going to be included in any 1.13 version that comes after.

[krzykwas]

The fix in GKE will be in version > 1.12.9-gke.3 which may begin rolling out at the earliest on 24.06.

[erinmez]

Hi, is there an update on when this will be rolled out?

[matti]

Currently GKE 1.13 has unpatched CVE (https://github.com/kubernetes/kubernetes/issues/78308) for the last 30 days... soo no rush. even EKS has patched that.

[erinmez]

I can confirm that 1.13.7-gke.8 resolves this issue for us. Thanks a lot!

[krzykwas]

This issue is fixed in GKE 1.12.9-gke.7+

[bkw]

I am seeing this with 1.13.7-gke.8.

Shall I create a new issue?

Update: It only seems to be happening when the ingress and managed certificate have been created using kubectl apply. Using kubectl create instead for both seems to be working.

[gromez]

I am seeing this with 1.13.7-gke.8.

Shall I create a new issue?

Update: It only seems to be happening when the ingress and managed certificate have been created using kubectl apply. Using kubectl create instead for both seems to be working.

I have a similar issue on 1.12.9-gke.13, I was on 1.12.7-gke.25 before upgrade. Duplicated certificates must be deleted manually?

Are you using create even if the ingress and certificates already exists?