search

kubernetes-sigs / external-dns

Configure external DNS servers (AWS Route53, Google CloudDNS and others) for Kubernetes Ingresses and Services
Apache License 2.0
7.72k stars 2.57k forks source link

DNS records will be not removed after ingress was deleted #882

Closed de1m closed 4 years ago

de1m commented 5 years ago

I've found the issue #442, but I get the same error. My external-dns deploy:

        args:
        - --source=ingress
        - --source=service
        - --log-level=debug
        - --provider=rfc2136
        - --rfc2136-host=192.168.40.62
        - --rfc2136-port=53
        - --rfc2136-zone=aditocloud.de
        - --rfc2136-tsig-secret=1xyBcSWjTXB1+imMeotnlCxCaUB4bl7XXzP9zJtJaf1qSPQdDCEM2ujStkolUnnyhPqwBmzTB7qt5Yw78+TXBQ==
        - --rfc2136-tsig-secret-alg=hmac-md5
        - --rfc2136-tsig-keyname=adito-de-trans
        - --rfc2136-tsig-axfr
        - --registry=txt
        - --txt-owner-id=extdns
        - --policy=sync
        - --interval=20s

Ingress example:

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: k8s-exmp-new.aditocloud.de
  namespace: control
  annotations:
    external-dns.alpha.kubernetes.io/target: heku.aditocloud.de
    external-dns.alpha.kubernetes.io/ttl: "10"
spec:
  rules:
  - host: k8s-exmp-2.aditocloud.de
    http:
      paths:
      - path: /
        backend:
          serviceName: nginx
          servicePort: 8080
      - path: /jclient
        backend:
          serviceName: nginx
          servicePort: 7779

The external-dns create the dns record

time="2019-01-30T14:48:56Z" level=debug msg=ApplyChanges
time="2019-01-30T14:48:56Z" level=debug msg="AddRecord.ep=k8s-exmp-2.aditocloud.de 10 IN CNAME heku.aditocloud.de []"
time="2019-01-30T14:48:56Z" level=debug msg="Adding RR: k8s-exmp-2.aditocloud.de 10 CNAME heku.aditocloud.de"
time="2019-01-30T14:48:56Z" level=debug msg=SendMessage
time="2019-01-30T14:48:56Z" level=debug msg=SendMessage.success
time="2019-01-30T14:48:56Z" level=debug msg="AddRecord.ep=k8s-exmp-2.aditocloud.de 0 IN TXT \"heritage=external-dns,external-dns/owner=extdns,external-dns/resource=ingress/control/k8s-exmp-new.aditocloud.de\" []"
time="2019-01-30T14:48:56Z" level=debug msg="Adding RR: k8s-exmp-2.aditocloud.de 0 TXT \"heritage=external-dns,external-dns/owner=extdns,external-dns/resource=ingress/control/k8s-exmp-new.aditocloud.de\""
time="2019-01-30T14:48:56Z" level=debug msg=SendMessage
time="2019-01-30T14:48:56Z" level=debug msg=SendMessage.success

but not delete this after deleting the ingress.

; <<>> DiG 9.10.3-P4-Ubuntu <<>> @192.168.40.62 aditocloud.de AXFR
; (1 server found)
;; global options: +cmd
aditocloud.de.          600     IN      SOA     a.ns.aditocloud.de. hostmaster.aditocloud.de. 2018112305 14400 3600 604800 180
aditocloud.de.          600     IN      CAA     0 issue "letsencrypt.org"
aditocloud.de.          600     IN      NS      a.ns.aditocloud.de.
aditocloud.de.          600     IN      NS      b.ns.aditocloud.de.
aditocloud.de.          600     IN      NS      c.ns.aditocloud.de.
aditocloud.de.          600     IN      NS      d.ns.aditocloud.de.
aditocloud.de.          600     IN      MX      10 smtp.aditocloud.de.
aditocloud.de.          600     IN      MX      20 mail.aditocloud.de.
aditocloud.de.          600     IN      A       94.130.25.247
aditocloud.de.          600     IN      A       94.130.25.248
aditocloud.de.          600     IN      A       94.130.25.249
heku.aditocloud.de.     600     IN      A       94.130.25.247
heku.aditocloud.de.     600     IN      A       94.130.25.248
heku.aditocloud.de.     600     IN      A       94.130.25.249
k8s-exmp-2.aditocloud.de. 10    IN      CNAME   heku.aditocloud.de.
mail.aditocloud.de.     600     IN      A       212.114.197.221
a.ns.aditocloud.de.     600     IN      A       94.130.25.247
b.ns.aditocloud.de.     600     IN      A       94.130.25.248
c.ns.aditocloud.de.     600     IN      A       94.130.25.249
d.ns.aditocloud.de.     600     IN      A       212.114.19.217
smtp.aditocloud.de.     600     IN      A       212.114.19.221
aditocloud.de.          600     IN      SOA     a.ns.aditocloud.de. hostmaster.aditocloud.de. 2018112305 14400 3600 604800 180
;; Query time: 4 msec
;; SERVER: 192.168.40.62#53(192.168.40.62)
;; WHEN: Wed Jan 30 15:55:03 CET 2019
;; XFR size: 22 records (messages 1, bytes 490)

Can me help someone?

Ps: I use the BIND DNS 9

njuettner commented 5 years ago

What happens if you update the record? From what I've seen in the code it deletes the record and add's a new one. Does this work?

I just wanna narrow down the issue.

de1m commented 5 years ago

So, I've created the ingress (like above) then I've saw, that the cname record was created (k8s-exmp-2.aditocloud.de). After this I changed the Ingress and apply the changes. The external-dns created a second record with "k8s-exmp-3.aditocloud.de". My external-dns docker image is "bitnami/external-dns:0.5.10" and I've testet also "registry.opensource.zalan.do/teapot/external-dns:latest"

fejta-bot commented 5 years ago

Issues go stale after 90d of inactivity. Mark the issue as fresh with /remove-lifecycle stale. Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta. /lifecycle stale

fejta-bot commented 5 years ago

Stale issues rot after 30d of inactivity. Mark the issue as fresh with /remove-lifecycle rotten. Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta. /lifecycle rotten

de1m commented 5 years ago

/remove-lifecycle rotten

Symbianx commented 5 years ago

~We also have this issue, we were using the powerdns provider and want to change for the RFC2136 one, but this issue is a show stopper.~

We were missing the EXTERNAL_DNS_RFC2136_TSIG_AXFR=true environment variable in external-dns. Now it works perfectly.

bjschafer commented 5 years ago

We are also experiencing this issue. We're trying to integrate with Active Directory DNS, which doesn't support TSIG, so enabling it isn't an option for us. I have tried it with a variety of variations of the TSIG parameters as suggested here and elsewhere, but it doesn't seem to matter.

Here's the parameters to the container:

- args:
 - --log-level=info
 - --policy=sync
 - --provider=rfc2136
 - --registry=txt
 - --interval=1m
 - --source=service
 - --source=ingress
 - --rfc2136-host=ad.example.com
 - --rfc2136-port=53
 - --rfc2136-zone=k8stmp.example.com
 - --rfc2136-insecure

I do not see anything in the container's logs indicating it trying to delete records:

[~]$ kl external-dns-58c6d9fbf-plk8g | grep '\.k8stmp'
time="2019-08-28T21:26:52Z" level=info msg="Adding RR: tst-zbx.k8stmp.example.com 0 A 10.128.26.119"
time="2019-08-28T21:26:52Z" level=info msg="Adding RR: tst-zbx.k8stmp.example.com 0 A 10.128.26.120"
time="2019-08-28T21:26:52Z" level=info msg="Adding RR: tst-zbx.k8stmp.example.com 0 A 10.128.26.121"
time="2019-08-28T21:26:52Z" level=info msg="Adding RR: tst-zbx.k8stmp.example.com 0 A 10.142.67.252"
time="2019-08-28T21:26:52Z" level=info msg="Adding RR: tst-zbx.k8stmp.example.com 0 A 10.142.67.253"
time="2019-08-28T21:26:52Z" level=info msg="Adding RR: tst-zbx.k8stmp.example.com 0 A 10.142.67.254"
time="2019-08-28T21:26:52Z" level=info msg="Adding RR: tst-zbx.k8stmp.example.com 0 TXT \"heritage=external-dns,external-dns/owner=default,external-dns/resource=ingress/zabbix/tst-zbx\""

Here's the startup message:

time="2019-08-28T21:15:50Z" level=info msg="config: {Master: KubeConfig: RequestTimeout:30s IstioIngressGatewayServices:[istio-system/istio-ingressgateway] ContourLoadBalancerService:heptio-contour/contour Sources:[service ingress] Namespace: AnnotationFilter: FQDNTemplate: CombineFQDNAndAnnotation:false IgnoreHostnameAnnotation:false Compatibility: PublishInternal:false PublishHostIP:false ConnectorSourceServer:localhost:8080 Provider:rfc2136 GoogleProject: DomainFilter:[] ExcludeDomains:[] ZoneIDFilter:[] AlibabaCloudConfigFile:/etc/kubernetes/alibaba-cloud.json AlibabaCloudZoneType: AWSZoneType: AWSZoneTagFilter:[] AWSAssumeRole: AWSBatchChangeSize:1000 AWSBatchChangeInterval:1s AWSEvaluateTargetHealth:true AWSAPIRetries:3 AWSPreferCNAME:false AzureConfigFile:/etc/kubernetes/azure.json AzureResourceGroup: CloudflareProxied:false CloudflareZonesPerPage:50 CoreDNSPrefix:/skydns/ RcodezeroTXTEncrypt:false InfobloxGridHost: InfobloxWapiPort:443 InfobloxWapiUsername:admin InfobloxWapiPassword: InfobloxWapiVersion:2.3.1 InfobloxSSLVerify:true InfobloxView: InfobloxMaxResults:0 DynCustomerName: DynUsername: DynPassword: DynMinTTLSeconds:0 OCIConfigFile:/etc/kubernetes/oci.yaml InMemoryZones:[] PDNSServer:http://localhost:8081 PDNSAPIKey: PDNSTLSEnabled:false TLSCA: TLSClientCert: TLSClientCertKey: Policy:sync Registry:txt TXTOwnerID:default TXTPrefix: Interval:1m0s Once:false DryRun:false LogFormat:text MetricsAddress::7979 LogLevel:info TXTCacheInterval:0s ExoscaleEndpoint:https://api.exoscale.ch/dns ExoscaleAPIKey: ExoscaleAPISecret: CRDSourceAPIVersion:externaldns.k8s.io/v1alpha1 CRDSourceKind:DNSEndpoint ServiceTypeFilter:[] CFAPIEndpoint: CFUsername: CFPassword: RFC2136Host:ad.example.com RFC2136Port:53 RFC2136Zone:k8stmp.example.com RFC2136Insecure:true RFC2136TSIGKeyName: RFC2136TSIGSecret: RFC2136TSIGSecretAlg: RFC2136TAXFR:false NS1Endpoint: NS1IgnoreSSL:false TransIPAccountName: TransIPPrivateKeyFile:}"
linki commented 5 years ago

Could you run it with --log-level=debug and check the additional logs. Specifically look for the DESIRED CHANGE log lines.

bjschafer commented 5 years ago

I've set --log-level=debug but am not seeing any DESIRED CHANGE lines.

[~]$ kl external-dns-84b68dfc58-vfq2l | grep 'DESIRED CHANGE'
[~]$
dbason commented 5 years ago

@bjschafer I had the same issue. I've added the --rfc2136-tsig-axfr flag and it seems to be deleting the entries correctly now. I think the issue is that axfr is turned on by default if you're using tsig security but if you're not you need to explicitly turn it on.

Update The deletion works if you're using A records. If you use a CNAME via external-dns.alpha.kubernetes.io/target it will never work. This is because if you have a CNAME you can't have any other data (including TXT records). I believe that because of this external-dns doesn't believe it has ownership of the record and so won't clean it up afterwards.

bjschafer commented 5 years ago

@dbason good call on --rfc2136-tsig-axfr - that does indeed allow deletion. We're not currently using CNAMEs (nor do we plan to), so I think this works for us. Thanks a ton!

The only issue remaining is how to restrict zone transfers on the Windows DNS side, since external-dns isn't actually a DNS server (the MMC snapin doesn't like it, anyways). But that's neither here nor there :)

I think I'll try and update the RFC2136 documentation to clarify some of what we've learned and send in a PR.

fejta-bot commented 4 years ago

Issues go stale after 90d of inactivity. Mark the issue as fresh with /remove-lifecycle stale. Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta. /lifecycle stale

fejta-bot commented 4 years ago

Stale issues rot after 30d of inactivity. Mark the issue as fresh with /remove-lifecycle rotten. Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta. /lifecycle rotten

fejta-bot commented 4 years ago

Rotten issues close after 30d of inactivity. Reopen the issue with /reopen. Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta. /close

k8s-ci-robot commented 4 years ago

@fejta-bot: Closing this issue.

In response to [this](https://github.com/kubernetes-sigs/external-dns/issues/882#issuecomment-581160701): >Rotten issues close after 30d of inactivity. >Reopen the issue with `/reopen`. >Mark the issue as fresh with `/remove-lifecycle rotten`. > >Send feedback to sig-testing, kubernetes/test-infra and/or [fejta](https://github.com/fejta). >/close Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md). If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository.