Closed peedrr closed 2 years ago
Having chatted with @jpetazzo (creator of Linode's tutorial mentioned above), it does look like the reason might be because of external-dns's logic to create a CNAME if the target looks like an ELB. Setting the --txt-prefix flag, as per #262, gets around the issue of the TXT not being created.
I believe that the logic of detecting an ELB should be changed. If the LoadBalancer has an IP address and not a domain name, this logic/change from A Record to CNAME does not seem necessary. Furthermore, applying the --txt-prefix
is prone to user error; if for some reason this prefix was not applied in the future, or applied incorrectly (e.g. the prefix changed), external-dns would lose track of the domains it is managing.
I think I'm hitting a similar issue after upgrading my LKE cluster from 1.20 to 1.22. I'm using traefik 2.5.6
and external-dns 0.10.2
.
When I enable debug logs I see:
time="2022-01-18T15:07:57Z" level=debug msg="Endpoints generated from service: default/traefik: [mydomain.com 0 IN A x.x.x.x [] mydomain.com 0 IN CNAME x-x-x-x.ip.linodeusercontent.com []]"
time="2022-01-18T15:07:57Z" level=info msg="Creating record." action=Create record= type=CNAME zoneID=xxxxxx zoneName=mydomain.com
time="2022-01-18T15:07:57Z" level=error msg="Failed to Create record: [400] [name] Invalid hostname " action=Create record= type=CNAME zoneID=xxxxxx zoneName=mydomain.com
time="2022-01-18T15:07:57Z" level=info msg="Creating record." action=Create record=external-dns type=TXT zoneID=xxxxxx zoneName=mydomain.com
Note the empty record=
there. A TXT
record is repeatedly created for as long as I let it run, but neither the A
nor CNAME
are created. That both the IP and Hostname are enumerated by external-dns
is interesting to me, I'm curious what others might see here.
I noticed the new domain on my new nodebalancer and eventually found https://www.linode.com/blog/linode/notice-changes-to-members-linode-com-and-nodebalancer-linode-com/ -- My cluster had been up for over a year so I hadn't noticed earlier. Seems like it could be related.
Prior to this I'd run into https://github.com/kubernetes-sigs/external-dns/issues/961 -- I've verified the RBAC rules appear to be correct, and --txt-prefix
doesn't seem to fix it (as you can see above i've set the value to external-dns.
). Everything is running in default
namespace and I've tried explicitly limiting to --source=service
(to rule out crd availability and such).
same behaviour for me than @peedrr described it.. is there a fix planned?
For me too, same behaviour than @peedrr described it.
Is there an annotation to force an A record rather than a CNAME record? You can't use a CNAME at the base of a domain and external-dns has no way to know if a record will be the base or not. A user hint seems the only way out.
I hope I'm not muddying the waters in case my issues are unrelated, but I'm not seeing that it's a unique issue just yet.
In my case, I'm running traefik ingress, with these service
annotations:
annotations:
external-dns.alpha.kubernetes.io/hostname: mydomain.com.
meta.helm.sh/release-name: traefik
meta.helm.sh/release-namespace: default
In case its relevant, the service is also configured with:
.spec.externalTrafficPolicy: Local
Before upgrading my clusters (1.20)/nodebalancer, external-dns
seemed to work as expected with k8s.gcr.io/external-dns/external-dns:v0.7.4
(upgraded to v0.10.2
, didn't work, reverted, still didn't work).
Runtime configs:
- --log-level=debug
- --log-format=text
- --interval=1m
- --source=ingress
- --source=service
- --policy=sync
- --registry=txt
- --txt-owner-id=<my-id>
- --txt-prefix=external-dns.
- --domain-filter=mydomain.com
- --provider=linode
... passing LINODE_TOKEN
via env var from a secret, which appears to work because repeat TXT
ownership records are created.
You can't use a CNAME at the base of a domain and external-dns has no way to know if a record will be the base or not.
At least in my case I never intended to create a CNAME
record at the root domain (only that an A
record be created at the given hostname
), and as far as I can tell the Linode provider didn't do this before the nodebalancer changes/cluster upgrades.
Is there an annotation to force an A record rather than a CNAME record?
Not that I could find: https://github.com/kubernetes-sigs/external-dns/blob/master/source/source.go#L40
The root of this issue, as mentioned before, is that you need to add a prefix or suffix to the TXT record so that it has a unique hostname. In the official Helm chart, this is accomplished via something like:
spec:
values:
registry: txt
txtOwnerId: ""
txtPrefix: ""
txtSuffix: "_externaldns"
After hitting the issue described, the above fixed my issues on LKE 1.22 (for services, at least -- haven't tested with ingress like @mars64 was mentioning).
The same principals from the AWS FAQ entry you mentioned apply. More discussion is also in this StackOverflow.
I do agree that the auto-detection logic should probably be changed, though.
After looking into this more, the behavior has definitely changed since the rollout of the new subdomain for nodebalancers.
Using --source=service
as an example: when external-dns runs, it creates Endpoint
s for both the hostname
(CNAME) and the ip
(A) returned in the status.loadBalancer.ingress
section of your kube service. Then, it uses a simple string comparison to determine which is "less". This is where the behavior has changed in which type of record you get.
Now, a target like "23-92-23-89.ip.linodeusercontent.com"
is considered less than a target like "23.92.23.89"
. Therefore, a CNAME gets created.
Previously, the hostname
of the svc would've been something like "nb-23-92-23-89.dallas.nodebalancer.linode.com"
which is considered to be more than "23.92.23.89"
. Therefore, an A record gets created.
Go playground to demonstrate it.
Really weird behavior. Still looking more into it, but hopefully this clarifies why it's happening.
EDIT: it's because string comparisons like greater/less than compare alphabetically (lexicographically). Since -
is unicode 45 and .
is unicode 46, 23-
is "alphabetically" before 23.
// FIXME We really need to define under which circumstances a list Targets // is considered 'less' than another.
Indeed. This seems like behavior that should be configurable (to choose the service IP or choose the hostname (CNAME)).
I think this might be related. I've recently found that when adding the service.beta.kubernetes.io.do-loadbalancer-hostname annotation to external-dns's service, it causes external-dns to create CNAMEs rather than A records.
Perhaps that because this setting replaces status.LoadBalancer.ingress.ip="1.2.3.4" with status.LoadBalancer.ingress.hostname="example.com" and it doesn't have an IP address to which to point A records.
That's a digitalocean specific setting, but I suspect there are equivalents for other providers?
Is that behavior expected, and likely an unavoidable consequence?
What happened:
Deployed external-dns on Linode Kubernetes Engine using bitnami's helm chart and referencing the instructions from Linode's recent Helm/Linode Kubernetes Engine tutorials.
Annotated a service with a LoadBalancer
external-dns created a CNAME Alias, failed to create a TXT record as it had the same hostname as the CNAME record, and didn't attempt to create an A Record. Logs confirm this:
time="2022-01-01T14:45:11Z" level=info msg="Creating record." action=Create > record=juicy type=TXT zoneID=1770931 zoneName=mydomain.com
time="2022-01-01T14:45:11Z" level=error msg="Failed to Create record: [400] [name] Record conflict - CNAMES must be unique" action=Create record=juicy type=TXT zoneID=1770931 zoneName=mydomain.com
helm upgrade --install nginx bitnami/nginx
helm upgrade --install external-dns bitnami/external-dns \
--namespace external-dns --create-namespace \ --set provider=linode \
--set linode.apiToken=$LINODE_API_TOKEN
kubectl annotate service nginx \
external-dns.alpha.kubernetes.io/nginx=nginx.mydomain.com