Open tobiasehlert opened 1 month ago
@tobiasehlert Thanks for reporting this. I had no idea on the number of nameservers limitation.
Here are steps to fix it, I will try to address this later, but if you want to shoot a PR, don't hesitate please.
Certainly. Let's recap the solution to your DNS configuration issue with kube-hetzner:
Problem: Your autoscaler nodes have 4 nameserver entries (2 IPv4 + 2 IPv6) in their /etc/resolv.conf, which exceeds Kubernetes' limit of 3 total nameservers.
Solution: Modify the DNS configuration for autoscaler nodes to include only 3 nameserver entries total, combining both IPv4 and IPv6 addresses.
Steps to implement:
a. Review your kube-hetzner configuration, focusing on autoscaler node provisioning.
b. Locate the part of the configuration (likely in cloudinit_config or user-data scripts) that sets up DNS for autoscaler nodes.
c. Modify this configuration to limit nameservers to 3 entries. For example:
nameserver 185.12.64.2
nameserver 185.12.64.1
nameserver 2a01:4ff:ff00::add:2
d. Ensure this modified configuration is applied when new autoscaler nodes are created.
e. After making changes, you may need to recreate existing autoscaler nodes to apply the new configuration.
Expected outcome: Once implemented, your autoscaler nodes should have the same number of nameserver entries as your worker nodes, resolving the "Nameserver limits were exceeded" error.
Verification: After applying changes, check the /etc/resolv.conf on new or recreated autoscaler nodes to confirm it contains only 3 nameserver entries.
Remember, the key is to ensure that your DNS configuration provides adequate resolution for both IPv4 and IPv6 as needed by your cluster, while staying within the 3-nameserver limit imposed by Kubernetes.
hi @mysticaltech,
I've had some progress on the cloud-init stuff, but want to check with you on some things..
It seems like it takes a while and then the autoscaler node actually gets updated and has other DNS settings than right after start. I guess that the K8s stuff start before the networking things are fully settled yet, but I'm not sure on that part.
Anyhow, it looks to me that there is an edit of the /etc/resolv.conf file through _cloudinit_write_filescommon (in addition to modifying /etc/NetworkManager/conf.d/dns.conf) with this:
write_files:
- content: |
nameserver 185.12.64.1
nameserver 185.12.64.2
nameserver 2a01:4ff:ff00::add:1
path: /etc/resolv.conf
permissions: '0644'
I read some in the documentation of cloud-init and even there are some ways forward..
If using option 1, it would look something like this:
manage_resolv_conf: true
resolv_conf:
nameservers:
- 185.12.64.1
- 185.12.64.2
- 2a01:4ff:ff00::add:1
What would you prefer? I've changed some things and you can take a look at it and see what you think.. link to my fork: https://github.com/kube-hetzner/terraform-hcloud-kube-hetzner/compare/master...tobiasehlert:terraform-hcloud-kube-hetzner:fix-dns-configuration-with-cloud-init
Description
This is the error returned by multiple pods on my autoscaler nodes:
Apparently k8s only support 3 entries for DNS servers (source #689).
When looking on both worker and autoscaler nodes, here is how the /etc/resolv.conf looks on two servers for comparison.
So my initial though is that that there is something missing when booting a autoscaler node that gets bootstraped with cloudinit_config, but what I haven't figured out yet.
Kube.tf file
Screenshots
No response
Platform
Linux