DNS resolution fails after changing hostname

[vishwanathjadhav]

Summary

Trying to change the hostname of the nodes

What Should Happen Instead?

After changing the hostname DNS service did not work.

Reproduction Steps

Steps to reproduce on Debian-11:

1. Install Master
    1. Enable dns, ingress, storage, helm3
    2. wait for addons and corresponding pods to be in a running state.
    3. microk8s stop
    4. hostnamectl set-hostname "mksmaster"
    5. Add hostname entries in /etc/hosts, confirm properly set.
    6.  microk8s start
2. Follow similar steps while setting other nodes(4-node cluster)
3. Now create the cluster(4-node cluster)
4. Exec to ingress controller on all the nodes individually
    1. microk8s kubectl exec -it nginx-ingress-microk8s-controller-wf9rx -n ingress -- /bin/bash
6.  check nslookup to  nameserver inside /etc/resolve.conf "nslookup 10.152.183.10":
    1. It only works on the ingress controller scheduled on the master node. But it does not work on the other  Nginx-ingress-microk8s-controller scheduled on other nodes.

As a result deployed APP is failing

Introspection Report

yes

Can you suggest a fix?

Expecting a solution

Are you interested in contributing with a fix?

no

[vishwanathjadhav]

@ktsakalozos - please help me on the same.

[vishwanathjadhav]

Checked routes, the routes added for the vxlan.calico interfaces of the other nodes are failing. i.e. ping to it's IP address fails. This could be the real root cause of the issue.

root@przmk8snodemaster:~# route -n
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
0.0.0.0         192.69.36.1      0.0.0.0         UG    0      0        0 eth0
10.1.12.128     10.1.12.128     255.255.255.192 UG    0      0        0 vxlan.calico
10.1.145.128    0.0.0.0         255.255.255.192 U     0      0        0 *
10.1.145.130    0.0.0.0         255.255.255.255 UH    0      0        0 cali2af005c6b1d
10.1.145.132    0.0.0.0         255.255.255.255 UH    0      0        0 cali1cc062d3237
10.1.145.134    0.0.0.0         255.255.255.255 UH    0      0        0 cali287d0d7cba1
192.69.36.0      0.0.0.0         255.255.252.0   U     0      0        0 eth0
172.17.0.0      0.0.0.0         255.255.0.0     U     0      0        0 docker0
root@przmk8snodemaster:~# ping 10.1.145.128
ping: socket: Address family not supported by protocol
PING 10.1.145.128 (10.1.145.128) 56(84) bytes of data.
64 bytes from 10.1.145.128: icmp_seq=1 ttl=64 time=0.040 ms

--- 10.1.145.128 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 0.040/0.040/0.040/0.000 ms

root@przmk8snodemaster:~# ping 10.1.12.128
ping: socket: Address family not supported by protocol
PING 10.1.12.128 (10.1.12.128) 56(84) bytes of data.

--- 10.1.12.128 ping statistics ---
4 packets transmitted, 0 received, 100% packet loss, time 3110ms

To resolve this, deleted calico and Nginx pods but the issue still persists

[ktsakalozos]

Hi @vishwanathjadhav, did you try to disable and re-enable the dns and ingress addons (killing the respective pods would also work I suspect). Could you also try to restart every node just to make sure there are no leftover iptable rules?

[vishwanathjadhav]

@ktsakalozos - Yes, I have tried restarting, nodes, and pods (from namespace Kube-system and ingress). When restarting DNS it shows the following errors in /var/log/syslog

Jul 10 14:57:21 microk8s microk8s.daemon-kubelite[44298]: E0710 14:57:21.080269   44298 kubelet.go:2419] "Error getting node" err="node \"przmk8snodemaster\" not found"
Jul 10 14:57:21 microk8s microk8s.daemon-kubelite[44298]: E0710 14:57:21.180822   44298 kubelet.go:2419] "Error getting node" err="node \"przmk8snodemaster\" not found"

[vishwanathjadhav]

@ktsakalozos - Any updates for me?

[vishwanathjadhav]

@ktsakalozos - Did you get a chance to look into it? Am I missing something during the process which is preventing me from changing the hostname?

[ktsakalozos]

@vishwanathjadhav, in general changing the hostname of a node in a k8s cluster is not supported. Nodes register with their hostnames (see microk8s kubectl get no), so if you change their hostnames is not recommended. One question is why do you want to change the hostname of the node after installing MicroK8s? Why don't you change the hostname and then install MicroK8s?

[vishwanathjadhav]

@ktsakalozos - Hey, Thanks for the update. Sorry for the delayed response. My requirement is to create multiple instances of pre-cooked/contextualized machines (VMC/AWS instances). So that I don't have to install it every time. There are few more setup steps because I am doing it in an on-Prem offline environment. Here I have to keep all the required container images pre-imported before starting add-ons. So it is becoming too much work to create multiple instances(multinode setups).

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.