search

kubernetes-sigs / external-dns

Configure external DNS servers (AWS Route53, Google CloudDNS and others) for Kubernetes Ingresses and Services
Apache License 2.0
7.61k stars 2.55k forks source link

External DNS refuses to create CNAME record #4185

Closed bo0tzz closed 8 months ago

bo0tzz commented 8 months ago

What happened: When creating an Ingress with the external-dns.alpha.kubernetes.io/target annotation, External DNS sometimes refuses to create a CNAME record.

What you expected to happen: A CNAME to the annotated target is created.

How to reproduce it (as minimally and precisely as possible): I am not able to find a minimal reproduction for this case. This ingress is failing to create the CNAME:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: emby-external
  namespace: media
  uid: 0d8c0cbb-9c15-4ad4-8d00-f916299a6930
  resourceVersion: '221261565'
  generation: 1
  creationTimestamp: '2024-01-16T11:15:01Z'
  labels:
    app.kubernetes.io/instance: emby
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/name: emby
    helm.sh/chart: app-template-0.2.2
    helm.toolkit.fluxcd.io/name: emby
    helm.toolkit.fluxcd.io/namespace: media
  annotations:
    auth.kube.bo0tzz.me/enabled: 'true'
    cert-manager.io/cluster-issuer: letsencrypt-prod
    external-dns.alpha.kubernetes.io/target: bo0tzz.me
    kubernetes.io/ingress.class: nginx
    meta.helm.sh/release-name: emby
    meta.helm.sh/release-namespace: media
    nginx.ingress.kubernetes.io/auth-response-headers: >-
      Set-Cookie,X-authentik-username,X-authentik-groups,X-authentik-email,X-authentik-name,X-authentik-uid
    nginx.ingress.kubernetes.io/auth-signin: /outpost.goauthentik.io/start?rd=$escaped_request_uri
    nginx.ingress.kubernetes.io/auth-snippet: proxy_set_header X-Forwarded-Host $http_host;
    nginx.ingress.kubernetes.io/auth-url: >-
      http://ak-outpost-authentik-embedded-outpost.auth.svc.cluster.local:9000/outpost.goauthentik.io/auth/nginx
    policies.kyverno.io/last-applied-patches: >
      auth-external.apply-ingress-auth-annotations.kyverno.io: added
      /metadata/annotations/nginx.ingress.kubernetes.io~1auth-response-headers
  managedFields:
    - manager: helm-controller
      operation: Update
      apiVersion: networking.k8s.io/v1
      time: '2024-01-16T11:15:01Z'
      fieldsType: FieldsV1
      fieldsV1:
        f:metadata:
          f:annotations:
            .: {}
            f:auth.kube.bo0tzz.me/enabled: {}
            f:cert-manager.io/cluster-issuer: {}
            f:external-dns.alpha.kubernetes.io/target: {}
            f:kubernetes.io/ingress.class: {}
            f:meta.helm.sh/release-name: {}
            f:meta.helm.sh/release-namespace: {}
          f:labels:
            .: {}
            f:app.kubernetes.io/instance: {}
            f:app.kubernetes.io/managed-by: {}
            f:app.kubernetes.io/name: {}
            f:helm.sh/chart: {}
            f:helm.toolkit.fluxcd.io/name: {}
            f:helm.toolkit.fluxcd.io/namespace: {}
        f:spec:
          f:rules: {}
          f:tls: {}
    - manager: nginx-ingress-controller
      operation: Update
      apiVersion: networking.k8s.io/v1
      time: '2024-01-16T11:15:46Z'
      fieldsType: FieldsV1
      fieldsV1:
        f:status:
          f:loadBalancer:
            f:ingress: {}
      subresource: status
  selfLink: /apis/networking.k8s.io/v1/namespaces/media/ingresses/emby-external
status:
  loadBalancer:
    ingress:
      - ip: 192.168.4.100
spec:
  tls:
    - hosts:
        - emby-external.bo0tzz.me
      secretName: emby-external-web-tls
  rules:
    - host: emby-external.bo0tzz.me
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: emby
                port:
                  number: 8096

This ingress works as expected:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: emby-test
  namespace: media
  uid: 8de245b0-a1c6-431a-a5b0-60607166fc89
  resourceVersion: '221264420'
  generation: 1
  creationTimestamp: '2024-01-16T11:24:20Z'
  annotations:
    auth.kube.bo0tzz.me/enabled: 'true'
    cert-manager.io/cluster-issuer: letsencrypt-prod
    external-dns.alpha.kubernetes.io/target: bo0tzz.me
    kubectl.kubernetes.io/last-applied-configuration: >
      {"apiVersion":"networking.k8s.io/v1","kind":"Ingress","metadata":{"annotations":{"auth.kube.bo0tzz.me/enabled":"true","cert-manager.io/cluster-issuer":"letsencrypt-prod","external-dns.alpha.kubernetes.io/target":"bo0tzz.me","kubernetes.io/ingress.class":"nginx"},"name":"emby-test","namespace":"media"},"spec":{"rules":[{"host":"emby-test.bo0tzz.me","http":{"paths":[{"backend":{"service":{"name":"emby","port":{"number":8096}}},"path":"/","pathType":"Prefix"}]}}],"tls":[{"hosts":["emby-test.bo0tzz.me"],"secretName":"emby-test-web-tls"}]}}
    kubernetes.io/ingress.class: nginx
    nginx.ingress.kubernetes.io/auth-response-headers: >-
      Set-Cookie,X-authentik-username,X-authentik-groups,X-authentik-email,X-authentik-name,X-authentik-uid
    nginx.ingress.kubernetes.io/auth-signin: /outpost.goauthentik.io/start?rd=$escaped_request_uri
    nginx.ingress.kubernetes.io/auth-snippet: proxy_set_header X-Forwarded-Host $http_host;
    nginx.ingress.kubernetes.io/auth-url: >-
      http://ak-outpost-authentik-embedded-outpost.auth.svc.cluster.local:9000/outpost.goauthentik.io/auth/nginx
    policies.kyverno.io/last-applied-patches: >
      auth-external.apply-ingress-auth-annotations.kyverno.io: added
      /metadata/annotations/nginx.ingress.kubernetes.io~1auth-snippet
  managedFields:
    - manager: kubectl-client-side-apply
      operation: Update
      apiVersion: networking.k8s.io/v1
      time: '2024-01-16T11:24:20Z'
      fieldsType: FieldsV1
      fieldsV1:
        f:metadata:
          f:annotations:
            .: {}
            f:auth.kube.bo0tzz.me/enabled: {}
            f:cert-manager.io/cluster-issuer: {}
            f:external-dns.alpha.kubernetes.io/target: {}
            f:kubectl.kubernetes.io/last-applied-configuration: {}
            f:kubernetes.io/ingress.class: {}
        f:spec:
          f:rules: {}
          f:tls: {}
    - manager: nginx-ingress-controller
      operation: Update
      apiVersion: networking.k8s.io/v1
      time: '2024-01-16T11:24:46Z'
      fieldsType: FieldsV1
      fieldsV1:
        f:status:
          f:loadBalancer:
            f:ingress: {}
      subresource: status
  selfLink: /apis/networking.k8s.io/v1/namespaces/media/ingresses/emby-test
status:
  loadBalancer:
    ingress:
      - ip: 192.168.4.100
spec:
  tls:
    - hosts:
        - emby-test.bo0tzz.me
      secretName: emby-test-web-tls
  rules:
    - host: emby-test.bo0tzz.me
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: emby
                port:
                  number: 8096

Anything else we need to know?:

The following is printed to the logs:

time="2024-01-16T11:24:27Z" level=info msg="Changing record." action=CREATE record=emby-test.bo0tzz.me ttl=1 type=CNAME zone=6f6e43f8581a7957412c960a384c760a
time="2024-01-16T11:24:28Z" level=info msg="Changing record." action=CREATE record=emby-test.bo0tzz.me ttl=1 type=TXT zone=6f6e43f8581a7957412c960a384c760a
time="2024-01-16T11:24:29Z" level=info msg="Changing record." action=CREATE record=cname-emby-test.bo0tzz.me ttl=1 type=TXT zone=6f6e43f8581a7957412c960a384c760a
time="2024-01-16T11:25:25Z" level=info msg="Domain emby-external.bo0tzz.me. contains conflicting record type candidates; discarding CNAME record"

I believe this is happening since #3747, but I don't know where external-dns is finding a conflicting record candidate from.

Environment:

bo0tzz commented 8 months ago

Apologies, it turns out this was user error. The authentication tooling I'm using creates an extra ingress object at https://host/outpost.goauthentik.io, which didn't have the appropriate external-dns.alpha.kubernetes.io/target annotation and was thus causing the conflict.