search

GoogleCloudPlatform / gke-managed-certs

Managed Certificates for Kubernetes clusters using GCLB
Apache License 2.0
246 stars 32 forks source link

Multiple Certs not working: Only picks up the first managedcert in the list #35

Closed domparry closed 4 years ago

domparry commented 5 years ago

The following config for my ingress create a LB with only the first cert. If I swap them around, I get the other one:

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  annotations:
    kubernetes.io/ingress.global-static-ip-name: om-static-ip
    networking.gke.io/managed-certificates: om-ssl-google-managed,om-no-www-ssl-google-managed
  name: om-prod-ssl
  namespace: default
spec:
  rules:
  - host: www.temp-om.simply.co.za
    http:
      paths:
      - backend:
          serviceName: om-tenandsix-prod
          servicePort: 8080
  - host: temp-om.simply.co.za
    http:
      paths:
      - backend:
          serviceName: om-tenandsix-prod
          servicePort: 8080

The resulting annotations copied from the ingress on cloud console:


ingress.gcp.kubernetes.io/pre-shared-cert: mcrt-7d7ac878-3a4f-4fe7-b23d-483813bb6ac0
ingress.kubernetes.io/backends: {"k8s-be-30009--4d15a37c4c5becdc":"HEALTHY","k8s-be-31353--4d15a37c4c5becdc":"HEALTHY"}
ingress.kubernetes.io/forwarding-rule: k8s-fw-default-om-prod-ssl--4d15a37c4c5becdc
ingress.kubernetes.io/https-forwarding-rule: k8s-fws-default-om-prod-ssl--4d15a37c4c5becdc
ingress.kubernetes.io/https-target-proxy: k8s-tps-default-om-prod-ssl--4d15a37c4c5becdc
ingress.kubernetes.io/ssl-cert: mcrt-7d7ac878-3a4f-4fe7-b23d-483813bb6ac0
ingress.kubernetes.io/target-proxy: k8s-tp-default-om-prod-ssl--4d15a37c4c5becdc
ingress.kubernetes.io/url-map: k8s-um-default-om-prod-ssl--4d15a37c4c5becdc
kubernetes.io/ingress.global-static-ip-name: om-static-ip
networking.gke.io/managed-certificates: om-ssl-google-managed,om-no-www-ssl-google-managed```
domparry commented 4 years ago

This has suddenly started working now...

vrobert78 commented 4 years ago

Hi,

we have currently the same problem. We have opened a case at the GCP support.

GKE Version: v1.14.10-gke.17

Here our configuration:

kind: Ingress
apiVersion: networking.k8s.io/v1beta1
metadata:
  name: images-recognition
  namespace: images-recognition
  annotations:
    kubernetes.io/ingress.allow-http: "false"
    kubernetes.io/ingress.global-static-ip-name: ingress-images-recognition
    networking.gke.io/managed-certificates: cert-imgreco,cert-imgreco-ancien-neuf,cert-imgreco-exterieur-elements,cert-imgreco-interieur-elements,cert-imgreco-interieur-matieres,cert-imgreco-interieur-pieces,cert-imgreco-visuels
spec:
  backend:
    serviceName: images-recognition
    servicePort: 5000
  rules:
    - host: imgreco.ouestfrance-immo.com
      http:
        paths:
        - path: /*
          backend:
            serviceName: images-recognition
            servicePort: 5000
    - host: imgreco-ancien-neuf.ouestfrance-immo.com
      http:
        paths:
        - path: /*
          backend:
            serviceName: images-recognition-ancien-neuf
            servicePort: 5000
    - host: imgreco-exterieur-elements.ouestfrance-immo.com
      http:
        paths:
        - path: /*
          backend:
            serviceName: images-recognition-exterieur-elements
            servicePort: 5000
    - host: imgreco-interieur-elements.ouestfrance-immo.com
      http:
        paths:
        - path: /*
          backend:
            serviceName: images-recognition-interieur-elements
            servicePort: 5000
    - host: imgreco-interieur-matieres.ouestfrance-immo.com
      http:
        paths:
        - path: /*
          backend:
            serviceName: images-recognition-interieur-matieres
            servicePort: 5000
    - host: imgreco-interieur-pieces.ouestfrance-immo.com
      http:
        paths:
        - path: /*
          backend:
            serviceName: images-recognition-interieur-pieces
            servicePort: 5000
    - host: imgreco-visuels.ouestfrance-immo.com
      http:
        paths:
        - path: /*
          backend:
            serviceName: images-recognition-visuels
            servicePort: 5000

Here the annotations:

ingress.kubernetes.io/https-forwarding-rule:       k8s-fws-images-recognition-images-recognition--bd24109445b008c0
  ingress.kubernetes.io/backends:                    {"k8s1-bd241094-images-recog-images-recognition-exter-50-e31b7b75":"HEALTHY","k8s1-bd241094-images-recog-images-recognition-inter-50-5999c0d4":"HEALTHY","k8s1-bd241094-images-recog-images-recognition-inter-50-ebf20c03":"HEALTHY","k8s1-bd241094-images-recog-images-recognition-inter-50-fcba0df3":"HEALTHY","k8s1-bd241094-images-recogni-images-recognition-anc-50-41a97672":"HEALTHY","k8s1-bd241094-images-recognit-images-recognition-v-500-715a016e":"HEALTHY","k8s1-bd241094-images-recognition-images-recognitio-500-8ce5ddaa":"HEALTHY"}
  ingress.kubernetes.io/https-target-proxy:          k8s-tps-images-recognition-images-recognition--bd24109445b008c0
  ingress.kubernetes.io/ssl-cert:                    mcrt-1aa80f1a-174e-4f5f-9b94-a40d777d2a92
  ingress.kubernetes.io/url-map:                     k8s-um-images-recognition-images-recognition--bd24109445b008cf
  kubectl.kubernetes.io/last-applied-configuration:  {"apiVersion":"networking.k8s.io/v1beta1","kind":"Ingress","metadata":{"annotations":{"kubernetes.io/ingress.allow-http":"false","kubernetes.io/ingress.global-static-ip-name":"ingress-images-recognition","networking.gke.io/managed-certificates":"cert-imgreco, cert-imgreco-ancien-neuf, cert-imgreco-exterieur-elements, cert-imgreco-interieur-elements, cert-imgreco-interieur-matieres, cert-imgreco-interieur-pieces, cert-imgreco-visuels"},"name":"images-recognition","namespace":"images-recognition"},"spec":{"backend":{"serviceName":"images-recognition","servicePort":5000},"rules":[{"host":"imgreco.ouestfrance-immo.com","http":{"paths":[{"backend":{"serviceName":"images-recognition","servicePort":5000},"path":"/*"}]}},{"host":"imgreco-ancien-neuf.ouestfrance-immo.com","http":{"paths":[{"backend":{"serviceName":"images-recognition-ancien-neuf","servicePort":5000},"path":"/*"}]}},{"host":"imgreco-exterieur-elements.ouestfrance-immo.com","http":{"paths":[{"backend":{"serviceName":"images-recognition-exterieur-elements","servicePort":5000},"path":"/*"}]}},{"host":"imgreco-interieur-elements.ouestfrance-immo.com","http":{"paths":[{"backend":{"serviceName":"images-recognition-interieur-elements","servicePort":5000},"path":"/*"}]}},{"host":"imgreco-interieur-matieres.ouestfrance-immo.com","http":{"paths":[{"backend":{"serviceName":"images-recognition-interieur-matieres","servicePort":5000},"path":"/*"}]}},{"host":"imgreco-interieur-pieces.ouestfrance-immo.com","http":{"paths":[{"backend":{"serviceName":"images-recognition-interieur-pieces","servicePort":5000},"path":"/*"}]}},{"host":"imgreco-visuels.ouestfrance-immo.com","http":{"paths":[{"backend":{"serviceName":"images-recognition-visuels","servicePort":5000},"path":"/*"}]}}]}}

  kubernetes.io/ingress.allow-http:             false
  kubernetes.io/ingress.global-static-ip-name:  ingress-images-recognition
  networking.gke.io/managed-certificates:       cert-imgreco,cert-imgreco-ancien-neuf,cert-imgreco-exterieur-elements,cert-imgreco-interieur-elements,cert-imgreco-interieur-matieres,cert-imgreco-interieur-pieces,cert-imgreco-visuels
  ingress.gcp.kubernetes.io/pre-shared-cert:    mcrt-1aa80f1a-174e-4f5f-9b94-a40d777d2a92

For us, only the last one is took into account.

Any idea ?

vrobert78 commented 4 years ago

@domparry Could you reopen the case ?

domparry commented 4 years ago

We're now on 1.15.8-gke.3, and it works well with the following:

metadata:
  annotations:
    kubernetes.io/ingress.global-static-ip-name: om-static-ip
    networking.gke.io/managed-certificates: om-collections-no-www-ssl-google-managed,om-group-www-ssl-google-managed,om-group-no-www-ssl-google-managed,om-admin-no-www-ssl-google-managed,om-admin-www-ssl-google-managed,om-home-ssl-google-managed,om-home-no-www-ssl-google-managed,om-callcentre-no-www-ssl-google-managed,om-postoffice-no-www-ssl-google-managed,om-app-no-www-ssl-google-managed
  name: om-prod-ssl
  namespace: default

Which results in the following annotations:


ingress.gcp.kubernetes.io/pre-shared-cert: mcrt-118fd68b-4134-4694-968f-a19b26695427,mcrt-19374f70-94c9-4128-b540-d09a48311af1,mcrt-3389cdda-f3f8-45a0-89f3-7bd6c042d713,mcrt-3547e7ac-5b1e-4739-a743-ef6f247fa348,mcrt-3b1faa81-0deb-498a-a339-56249fbd83bf,mcrt-701e17a8-a3fe-4ec2-afc8-cf740878bc30,mcrt-75b73c73-0e27-4420-a691-5ccb13a9cbff,mcrt-91396b87-1137-414d-936c-02a297727fe0,mcrt-9668cbe6-b532-42bf-8741-c4b368741a29,mcrt-ba63e6ef-65e9-4266-8f46-29925759710d
ingress.kubernetes.io/backends: {"k8s-be-30009--4d15a37c4c5becdc":"HEALTHY","k8s-be-31029--4d15a37c4c5becdc":"HEALTHY","k8s-be-31353--4d15a37c4c5becdc":"HEALTHY","k8s-be-31438--4d15a37c4c5becdc":"HEALTHY","k8s-be-31522--4d15a37c4c5becdc":"HEALTHY","k8s-be-32031--4d15a37c4c5becdc":"HEALTHY","k8s-be-32676--4d15a37c4c5becdc":"HEALTHY"}
ingress.kubernetes.io/forwarding-rule: k8s-fw-default-om-prod-ssl--4d15a37c4c5becdc
ingress.kubernetes.io/https-forwarding-rule: k8s-fws-default-om-prod-ssl--4d15a37c4c5becdc
ingress.kubernetes.io/https-target-proxy: k8s-tps-default-om-prod-ssl--4d15a37c4c5becdc
ingress.kubernetes.io/ssl-cert: mcrt-118fd68b-4134-4694-968f-a19b26695427,mcrt-19374f70-94c9-4128-b540-d09a48311af1,mcrt-3389cdda-f3f8-45a0-89f3-7bd6c042d713,mcrt-3547e7ac-5b1e-4739-a743-ef6f247fa348,mcrt-3b1faa81-0deb-498a-a339-56249fbd83bf,mcrt-701e17a8-a3fe-4ec2-afc8-cf740878bc30,mcrt-75b73c73-0e27-4420-a691-5ccb13a9cbff,mcrt-91396b87-1137-414d-936c-02a297727fe0,mcrt-9668cbe6-b532-42bf-8741-c4b368741a29,mcrt-ba63e6ef-65e9-4266-8f46-29925759710d
ingress.kubernetes.io/target-proxy: k8s-tp-default-om-prod-ssl--4d15a37c4c5becdc
ingress.kubernetes.io/url-map: k8s-um-default-om-prod-ssl--4d15a37c4c5becdc
kubernetes.io/ingress.global-static-ip-name: om-static-ip
networking.gke.io/managed-certificates: om-collections-no-www-ssl-google-managed,om-group-www-ssl-google-managed,om-group-no-www-ssl-google-managed,om-admin-no-www-ssl-google-managed,om-admin-www-ssl-google-managed,om-home-ssl-google-managed,om-home-no-www-ssl-google-managed,om-callcentre-no-www-ssl-google-managed,om-postoffice-no-www-ssl-google-managed,om-app-no-www-ssl-google-managed```

On a different cluster however, I've opted to use a wildcard cert which works really well.
vrobert78 commented 4 years ago

You say that you use a Google Managed Wilcard Cert ? I was thinking it isn't possible to do wildcard, are you sure ? Perhaps, you use a normal wilcard cert, not autogenerated ?

domparry commented 4 years ago

Sorry @vrobert78 , I meant a multi domain cert, not a wildcard cert. It's defined like this:

gcloud beta compute ssl-certificates create cert-name --project=projectId --domains domain1.co.za,domain2.co.za,domain3.co.za

used like this:

metadata:
  annotations:
    ingress.kubernetes.io/ssl-cert: cert-name
vrobert78 commented 4 years ago

Ok. I understand.

We tested to create one, but unfortunately it's not supported yet in 1.14.10-gke.17.

vrobert78 commented 4 years ago

Hi, we solved the problem thanks the the Google Support.

We had to delete the certs in errors, but not by deleting the managedcertificates.networking.gke.io. Instead, the resource to be deleted is mcrt.

You have to do a: kubectl delete mcrt xxx, wait 2 minutes, then recreate the cert by reapplying your yml.

krzykwas commented 4 years ago

It was caused by a bug, which is already fixed in 0.4.2/GKE GKE 1.16.8-gke.3, sorry. See #45 for more info. I'm closing this issue and let's continue the discussion there.