search

canonical / microk8s

MicroK8s is a small, fast, single-package Kubernetes for datacenters and the edge.
https://microk8s.io
Apache License 2.0
8.45k stars 770 forks source link

Hostname change fails on MicroK8S(environment Debian:11.6, MicroK8S:1.26.1) #3753

Closed vishwanathjadhav closed 8 months ago

vishwanathjadhav commented 1 year ago

Need help on the following issue. I have Installed MicroK8S(1.26.1) on Debian-11.6. I tried to change the hostname of the node after installing the MicroK8S. Followed the following steps to change the hostname.

  1. Enabled Default Addons
  2. sed -i "s/$(hostname)/node-master/g" /etc/hosts
  3. hostnamectl set-hostname node-master
  4. reboot the system
  5. kubectl delete node hostmaster (i.e. old-hostname, hostmaster, the old hostname which is marked as not-ready)
  6. Enable the RBAC addon (Important step, After enabling RBAC the microk8s setup becomes unstable.) Configmaps are still referring to old hostnames

The syslog is filled with the following error messages:

Feb 14 11:54:30 node-master microk8s.daemon-kubelite[828]: E0214 11:54:30.948997     828 controller.go:146] failed to ensure lease exists, will retry in 7s, error: leases.coordination.k8s.io "node-master" is forbidden: User "system:node:hostmaster" cannot get resource "leases" in API group "coordination.k8s.io" in the namespace "kube-node-lease": can only access node lease with the same name as the requesting node
Feb 14 11:54:31 node-master microk8s.daemon-kubelite[828]: W0214 11:54:31.769946     828 reflector.go:424] object-"container-registry"/"kube-root-ca.crt": failed to list *v1.ConfigMap: configmaps "kube-root-ca.crt" is forbidden: User "system:node:hostmaster" cannot list resource "configmaps" in API group "" in the namespace "container-registry": no relationship found between node 'hostmaster' and this object
Feb 14 11:54:31 node-master microk8s.daemon-kubelite[828]: E0214 11:54:31.769990     828 reflector.go:140] object-"container-registry"/"kube-root-ca.crt": Failed to watch *v1.ConfigMap: failed to list *v1.ConfigMap: configmaps "kube-root-ca.crt" is forbidden: User "system:node:hostmaster" cannot list resource "configmaps" in API group "" in the namespace "container-registry": no relationship found between node 'hostmaster' and this object
Feb 14 11:54:32 node-master systemd-timesyncd[598]: Timed out waiting for reply from 38.17.55.111:123 (3.debian.pool.ntp.org).
Feb 14 11:54:35 node-master microk8s.daemon-kubelite[828]: E0214 11:54:35.385448     828 desired_state_of_world_populator.go:312] "Error processing volume" err="error processing PVC container-registry/registry-claim: failed to fetch PVC from API server: persistentvolumeclaims \"registry-claim\" is forbidden: User \"system:node:hostmaster\" cannot get resource \"persistentvolumeclaims\" in API group \"\" in the namespace \"container-registry\": no relationship found between node 'hostmaster' and this object" pod="container-registry/registry-77c7575667-42l6x" volumeName="registry-data"
Feb 14 11:54:35 node-master microk8s.daemon-kubelite[828]: E0214 11:54:35.664720     828 kubelet.go:1821] "Unable to attach or mount volumes for pod; skipping pod" err="unmounted volumes=[registry-data kube-api-access-fzk9q], unattached volumes=[registry-data kube-api-access-fzk9q]: error processing PVC container-registry/registry-claim: failed to fetch PVC from API server: persistentvolumeclaims \"registry-claim\" is forbidden: User \"system:node:hostmaster\" cannot get resource \"persistentvolumeclaims\" in API group \"\" in the namespace \"container-registry\": no relationship found between node 'hostmaster' and this object" pod="container-registry/registry-77c7575667-42l6x"
Feb 14 11:54:35 node-master microk8s.daemon-kubelite[828]: E0214 11:54:35.664823     828 pod_workers.go:965] "Error syncing pod, skipping" err="unmounted volumes=[registry-data kube-api-access-fzk9q], unattached volumes=[registry-data kube-api-access-fzk9q]: error processing PVC container-registry/registry-claim: failed to fetch PVC from API server: persistentvolumeclaims \"registry-claim\" is forbidden: User \"system:node:hostmaster\" cannot get resource \"persistentvolumeclaims\" in API group \"\" in the namespace \"container-registry\": no relationship found between node 'hostmaster' and this object" pod="container-registry/registry-77c7575667-42l6x" podUID=90e2cbd3-3b11-40f8-86ff-8e92704e153f
Feb 14 11:54:37 node-master microk8s.daemon-kubelite[828]: W0214 11:54:37.692498     828 reflector.go:424] object-"kube-system"/"coredns": failed to list *v1.ConfigMap: configmaps "coredns" is forbidden: User "system:node:hostmaster" cannot list resource "configmaps" in API group "" in the namespace "kube-system": no relationship found between node 'hostmaster' and this object
Feb 14 11:54:37 node-master microk8s.daemon-kubelite[828]: E0214 11:54:37.692570     828 reflector.go:140] object-"kube-system"/"coredns": Failed to watch *v1.ConfigMap: failed to list *v1.ConfigMap: configmaps "coredns" is forbidden: User "system:node:hostmaster" cannot list resource "configmaps" in API group "" in the namespace "kube-system": no relationship found between node 'hostmaster' and this object
Feb 14 11:54:37 node-master microk8s.daemon-kubelite[828]: E0214 11:54:37.986141     828 controller.go:146] failed to ensure lease exists, will retry in 7s, error: leases.coordination.k8s.io "node-master" is forbidden: User "system:node:hostmaster" cannot get resource "leases" in API group "coordination.k8s.io" in the namespace "kube-node-lease": can only access node lease with the same name as the requesting node
Feb 14 11:54:38 node-master microk8s.daemon-kubelite[828]: W0214 11:54:38.694869     828 reflector.go:424] object-"kube-system"/"calico-config": failed to list *v1.ConfigMap: configmaps "calico-config" is forbidden: User "system:node:hostmaster" cannot list resource "configmaps" in API group "" in the namespace "kube-system": no relationship found between node 'hostmaster' and this object
Feb 14 11:54:38 node-master microk8s.daemon-kubelite[828]: E0214 11:54:38.694947     828 reflector.go:140] object-"kube-system"/"calico-config": Failed to watch *v1.ConfigMap: failed to list *v1.ConfigMap: configmaps "calico-config" is forbidden: User "system:node:hostmaster" cannot list resource "configmaps" in API group "" in the namespace "kube-system": no relationship found between node 'hostmaster' and this object
Feb 14 11:54:42 node-master systemd-timesyncd[598]: Timed out waiting for reply from 208.67.75.242:123 (3.debian.pool.ntp.org).
Feb 14 11:54:44 node-master microk8s.daemon-kubelite[828]: E0214 11:54:44.987959     828 controller.go:146] failed to ensure lease exists, will retry in 7s, error: leases.coordination.k8s.io "node-master" is forbidden: User "system:node:hostmaster" cannot get resource "leases" in API group "coordination.k8s.io" in the namespace "kube-node-lease": can only access node lease with the same name as the requesting node
Feb 14 11:54:47 node-master microk8s.daemon-kubelite[828]: E0214 11:54:47.274585     828 desired_state_of_world_populator.go:312] "Error processing volume" err="error processing PVC container-registry/registry-claim: failed to fetch PVC from API server: persistentvolumeclaims \"registry-claim\" is forbidden: User \"system:node:hostmaster\" cannot get resource \"persistentvolumeclaims\" in API group \"\" in the namespace \"container-registry\": no relationship found between node 'hostmaster' and this object" pod="container-registry/registry-77c7575667-42l6x" volumeName="registry-data"
Feb 14 11:54:47 node-master microk8s.daemon-kubelite[828]: E0214 11:54:47.528463     828 kubelet.go:1821] "Unable to attach or mount volumes for pod; skipping pod" err="unmounted volumes=[kube-api-access-fzk9q registry-data], unattached volumes=[kube-api-access-fzk9q registry-data]: error processing PVC container-registry/registry-claim: failed to fetch PVC from API server: persistentvolumeclaims \"registry-claim\" is forbidden: User \"system:node:hostmaster\" cannot get resource \"persistentvolumeclaims\" in API group \"\" in the namespace \"container-registry\": no relationship found between node 'hostmaster' and this object" pod="container-registry/registry-77c7575667-42l6x"
Feb 14 11:54:47 node-master microk8s.daemon-kubelite[828]: E0214 11:54:47.528545     828 pod_workers.go:965] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-fzk9q registry-data], unattached volumes=[kube-api-access-fzk9q registry-data]: error processing PVC container-registry/registry-claim: failed to fetch PVC from API server: persistentvolumeclaims \"registry-claim\" is forbidden: User \"system:node:hostmaster\" cannot get resource \"persistentvolumeclaims\" in API group \"\" in the namespace \"container-registry\": no relationship found between node 'hostmaster' and this object" pod="container-registry/registry-77c7575667-42l6x" podUID=90e2cbd3-3b11-40f8-86ff-8e92704e153f
Feb 14 11:54:47 node-master microk8s.daemon-kubelite[828]: E0214 11:54:47.653240     828 resource_quota_controller.go:417] unable to retrieve the complete list of server APIs: metrics.k8s.io/v1beta1: the server is currently unable to handle the request
Feb 14 11:54:49 node-master microk8s.daemon-kubelite[828]: E0214 11:54:49.293978     828 kubelet.go:1821] "Unable to attach or mount volumes for pod; skipping pod" err="unmounted volumes=[kube-api-access-qghbl], unattached volumes=[tmp-dir kube-api-access-qghbl]: timed out waiting for the condition" pod="kube-system/metrics-server-6f754f88d-mmxhw"
Feb 14 11:54:49 node-master microk8s.daemon-kubelite[828]: E0214 11:54:49.294034     828 kubelet.go:1821] "Unable to attach or mount volumes for pod; skipping pod" err="unmounted volumes=[kube-api-access-42bjj], unattached volumes=[kube-api-access-42bjj]: timed out waiting for the condition" pod="kube-system/calico-kube-controllers-79568db7f8-4ccs4"
Feb 14 11:54:49 node-master microk8s.daemon-kubelite[828]: E0214 11:54:49.294075     828 kubelet.go:1821] "Unable to attach or mount volumes for pod; skipping pod" err="unmounted volumes=[kube-api-access-nnbbb], unattached volumes=[cni-log-dir cni-bin-dir kube-api-access-nnbbb var-lib-calico var-run-calico lib-modules xtables-lock policysync cni-net-dir host-local-net-dir]: timed out waiting for the condition" pod="kube-system/calico-node-vhshj"
Feb 14 11:54:49 node-master microk8s.daemon-kubelite[828]: E0214 11:54:49.294123     828 pod_workers.go:965] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-nnbbb], unattached volumes=[cni-log-dir cni-bin-dir kube-api-access-nnbbb var-lib-calico var-run-calico lib-modules xtables-lock policysync cni-net-dir host-local-net-dir]: timed out waiting for the condition" pod="kube-system/calico-node-vhshj" podUID=184939c0-34c0-4ca3-a176-395c9d02b6ed
Feb 14 11:54:49 node-master microk8s.daemon-kubelite[828]: E0214 11:54:49.294081     828 pod_workers.go:965] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-42bjj], unattached volumes=[kube-api-access-42bjj]: timed out waiting for the condition" pod="kube-system/calico-kube-controllers-79568db7f8-4ccs4" podUID=3b93e849-2889-4ea3-b155-dca66d29ae32
Feb 14 11:54:49 node-master microk8s.daemon-kubelite[828]: E0214 11:54:49.294054     828 pod_workers.go:965] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-qghbl], unattached volumes=[tmp-dir kube-api-access-qghbl]: timed out waiting for the condition" pod="kube-system/metrics-server-6f754f88d-mmxhw" podUID=bb298398-7a03-402e-952f-886a41016e6a
Feb 14 11:54:49 node-master microk8s.daemon-kubelite[828]: E0214 11:54:49.294297     828 kubelet.go:1821] "Unable to attach or mount volumes for pod; skipping pod" err="unmounted volumes=[kube-api-access-5rn6b], unattached volumes=[kube-api-access-5rn6b pv-volume]: timed out waiting for the condition" pod="kube-system/hostpath-provisioner-69cd9ff5b8-cf44z"
Feb 14 11:54:49 node-master microk8s.daemon-kubelite[828]: E0214 11:54:49.294413     828 pod_workers.go:965] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-5rn6b], unattached volumes=[kube-api-access-5rn6b pv-volume]: timed out waiting for the condition" pod="kube-system/hostpath-provisioner-69cd9ff5b8-cf44z" podUID=a0a8ca6d-5787-4e12-9811-e903b1dbe9d3
Feb 14 11:54:49 node-master microk8s.daemon-kubelite[828]: E0214 11:54:49.294354     828 kubelet.go:1821] "Unable to attach or mount volumes for pod; skipping pod" err="unmounted volumes=[kube-api-access-zwfkb], unattached volumes=[kube-api-access-zwfkb tmp-volume]: timed out waiting for the condition" pod="kube-system/dashboard-metrics-scraper-7bc864c59-2bwr2"
Feb 14 11:54:49 node-master microk8s.daemon-kubelite[828]: E0214 11:54:49.294445     828 kubelet.go:1821] "Unable to attach or mount volumes for pod; skipping pod" err="unmounted volumes=[kubernetes-dashboard-certs kube-api-access-xxcdh], unattached volumes=[kubernetes-dashboard-certs tmp-volume kube-api-access-xxcdh]: timed out waiting for the condition" pod="kube-system/kubernetes-dashboard-dc96f9fc-qnrgx"
Feb 14 11:54:49 node-master microk8s.daemon-kubelite[828]: E0214 11:54:49.294458     828 kubelet.go:1821] "Unable to attach or mount volumes for pod; skipping pod" err="unmounted volumes=[config-volume kube-api-access-lwjs2], unattached volumes=[config-volume kube-api-access-lwjs2]: timed out waiting for the condition" pod="kube-system/coredns-6f5f9b5d74-n8bcj"
Feb 14 11:54:49 node-master microk8s.daemon-kubelite[828]: E0214 11:54:49.294497     828 pod_workers.go:965] "Error syncing pod, skipping" err="unmounted volumes=[config-volume kube-api-access-lwjs2], unattached volumes=[config-volume kube-api-access-lwjs2]: timed out waiting for the condition" pod="kube-system/coredns-6f5f9b5d74-n8bcj" podUID=7cd69261-cf57-49c0-898b-44d43397b174
Feb 14 11:54:49 node-master microk8s.daemon-kubelite[828]: E0214 11:54:49.294475     828 pod_workers.go:965] "Error syncing pod, skipping" err="unmounted volumes=[kubernetes-dashboard-certs kube-api-access-xxcdh], unattached volumes=[kubernetes-dashboard-certs tmp-volume kube-api-access-xxcdh]: timed out waiting for the condition" pod="kube-system/kubernetes-dashboard-dc96f9fc-qnrgx" podUID=0d3d2d57-c2cf-4af8-9c59-ec12f151cb9a
Feb 14 11:54:49 node-master microk8s.daemon-kubelite[828]: E0214 11:54:49.294451     828 pod_workers.go:965] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-zwfkb], unattached volumes=[kube-api-access-zwfkb tmp-volume]: timed out waiting for the condition" pod="kube-system/dashboard-metrics-scraper-7bc864c59-2bwr2" podUID=de724959-6b19-413d-9505-ba47da3af872
Feb 14 11:54:49 node-master microk8s.daemon-kubelite[828]: E0214 11:54:49.294528     828 kubelet.go:1821] "Unable to attach or mount volumes for pod; skipping pod" err="unmounted volumes=[kube-api-access-7vbz9], unattached volumes=[kube-api-access-7vbz9]: timed out waiting for the condition" pod="ingress/nginx-ingress-microk8s-controller-bckxp"
Feb 14 11:54:49 node-master microk8s.daemon-kubelite[828]: E0214 11:54:49.294556     828 pod_workers.go:965] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-7vbz9], unattached volumes=[kube-api-access-7vbz9]: timed out waiting for the condition" pod="ingress/nginx-ingress-microk8s-controller-bckxp" podUID=e0524248-a12b-42d5-ace1-d789d78bf9ec
Feb 14 11:54:50 node-master microk8s.daemon-kubelite[828]: W0214 11:54:50.883430     828 garbagecollector.go:752] failed to discover some groups: map[metrics.k8s.io/v1beta1:the server is currently unable to handle the request]
Feb 14 11:54:51 node-master microk8s.daemon-kubelite[828]: W0214 11:54:51.868966     828 handler_proxy.go:106] no RequestInfo found in the context
Feb 14 11:54:51 node-master microk8s.daemon-kubelite[828]: E0214 11:54:51.869037     828 controller.go:113] loading OpenAPI spec for "v1beta1.metrics.k8s.io" failed with: Error, could not get list of group versions for APIService
Feb 14 11:54:51 node-master microk8s.daemon-kubelite[828]: I0214 11:54:51.869049     828 controller.go:126] OpenAPI AggregationController: action for item v1beta1.metrics.k8s.io: Rate Limited Requeue.
Feb 14 11:54:51 node-master microk8s.daemon-kubelite[828]: W0214 11:54:51.878025     828 handler_proxy.go:106] no RequestInfo found in the context
Feb 14 11:54:51 node-master microk8s.daemon-kubelite[828]: E0214 11:54:51.878625     828 controller.go:116] loading OpenAPI spec for "v1beta1.metrics.k8s.io" failed with: failed to retrieve openAPI spec, http error: ResponseCode: 503, Body: service unavailable
Feb 14 11:54:51 node-master microk8s.daemon-kubelite[828]: , Header: map[Content-Type:[text/plain; charset=utf-8] X-Content-Type-Options:[nosniff]]

microk8s status:

microk8s is not running. Use microk8s inspect for a deeper inspection.

Do I need to regenerate the configmaps(error message: "node-master" is forbidden: User system:node:hostmaster") to fix the error? Is there a way to fix the error?

neoaggelos commented 1 year ago

Hi @vishwanathjadhav,

In general, it is quite problematic to change the hostname of the node after deploying MicroK8s. If possible, I would suggest starting from a clean install instead:

# remove everything
sudo snap remove microk8s --purge

# start anew, it should use the new hostname
sudo snap install microk8s --classic [...]

Alternatively, and I strongly advise against doing any of the things below if you must preserve this installation, there are a number of steps you have to follow. The first is to refresh all the server certificates with

sudo microk8s refresh-certs

Then, go into /var/snap/microk8s/current/credentials/known_tokens.csv and replace instances of your old host name with the new one. Make sure to restart MicroK8s afterwards:

sudo snap restart microk8s

This should get some things back up. There may be more things to look out for, but that should get you a working kubernetes control plane.

stale[bot] commented 9 months ago

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.