search

siderolabs / talos

Talos Linux is a modern Linux distribution built for Kubernetes.
https://www.talos.dev
Mozilla Public License 2.0
6.96k stars 565 forks source link

Talos Install by PXE, losing important containers #7377

Closed marrgieD closed 1 year ago

marrgieD commented 1 year ago

Bug Report

Description

I use CentOS7 to installing talos by PXE. It seems like that my enviroment lose some container. Most importantly, there is no container for kube-apiserver, which makes me can do nothing. 

[xxx@localhost talosyaml]$ talosctl containers -k
NODE              NAMESPACE   ID                                                                                          IMAGE                                             PID    STATUS
192.168.242.188   k8s.io      kube-system/kube-controller-manager-talos-0sr-1c0                                           registry.k8s.io/pause:3.6                         3374   SANDBOX_READY
192.168.242.188   k8s.io      └─ kube-system/kube-controller-manager-talos-0sr-1c0:kube-controller-manager:27eb31032e33   registry.k8s.io/kube-controller-manager:v1.27.1   0      CONTAINER_EXITED
192.168.242.188   k8s.io      kube-system/kube-scheduler-talos-0sr-1c0                                                    registry.k8s.io/pause:3.6                         3381   SANDBOX_READY
192.168.242.188   k8s.io      └─ kube-system/kube-scheduler-talos-0sr-1c0:kube-scheduler:cf2d8b2e033f                     registry.k8s.io/kube-scheduler:v1.27.1            0      CONTAINER_EXITED
[xxx@localhost talosyaml]$ talosctl images
ghcr.io/siderolabs/flannel:v0.21.4
ghcr.io/siderolabs/install-cni:v1.4.0-1-g9b07505
docker.io/coredns/coredns:1.10.1
gcr.io/etcd-development/etcd:v3.5.8
registry.k8s.io/kube-apiserver:v1.27.1
registry.k8s.io/kube-controller-manager:v1.27.1
registry.k8s.io/kube-scheduler:v1.27.1
registry.k8s.io/kube-proxy:v1.27.1
ghcr.io/siderolabs/kubelet:v1.27.1
ghcr.io/siderolabs/installer:v1.4.0
registry.k8s.io/pause:3.6

I cannot get node

[xxx@localhost talosyaml]$ kubectl get node
The connection to the server 192.168.242.188:6443 was refused - did you specify the right host or port?

on talos machine always circularly the following two warnings:

user: warning: [2023-06-14T09:03:31.2973559712]: [talos] kubernetes endpoint watch error {"component":"controller-runtime","controller":"k8s.EndpointController",“error":"failed to list *v1.Endpoints: Get x5c"https://192.168.242.188:6443/api/v1/namespaces/default/endpoints?fieldSelector=metadata.name%3Dkubernetes&limit=500&resourceVersion=0\x5c": dial tcp 192.168.242.188:6443: conect:connection refuse"}
user: warning:[2023-06-14T09:03:36.7748739712]: [talos] controller failed {"component":"controller-runtimer","controller":"k8s.KubeletStaticPodControllerter","error":"error refreshing pod status: error fetching pod status: an error on the server (\x5c"Authorization error (user=apiserver-kubelet-client, verb=get,resource=nodes,subresource=proxy)\ x5c") has prevented the request from succeeding}

And it is not healthy constantly, sometimes unhealthy, and the state always switch to another one for a short time, about several minutes

Logs

[xxx@localhost talosyaml] talosctl logs kubelet  
192.168.242.188: {"ts":1686800682156.1724,"caller":"cache/reflector.go:533","msg":"vendor/k8s.io/client-go/informers/factory.go:150: failed to list *v1.RuntimeClass: Get \"https://localhost:6443/apis/node.k8s.io/v1/runtimeclasses?limit=500&resourceVersion=0\": dial tcp [::1]:6443: connect: connection refused\n","v":0}
192.168.242.188: {"ts":1686800682156.209,"caller":"cache/reflector.go:148","msg":"vendor/k8s.io/client-go/informers/factory.go:150: Failed to watch *v1.RuntimeClass: failed to list *v1.RuntimeClass: Get \"https://localhost:6443/apis/node.k8s.io/v1/runtimeclasses?limit=500&resourceVersion=0\": dial tcp [::1]:6443: connect: connection refused\n"}
192.168.242.188: {"ts":1686800682572.3499,"caller":"certificate/transport.go:123","msg":"No valid client certificate is found but the server is not responsive. A restart may be necessary to retrieve new initial credentials.","lastCertificateAvailabilityTime":1686800682572.3606,"shutdownThreshold":"5m0s"}
192.168.242.188: {"ts":1686800682882.5461,"caller":"certificate/transport.go:123","msg":"No valid client certificate is found but the server is not responsive. A restart may be necessary to retrieve new initial credentials.","lastCertificateAvailabilityTime":1686800682882.5557,"shutdownThreshold":"5m0s"}
192.168.242.188: {"ts":1686800683571.4993,"caller":"certificate/transport.go:123","msg":"No valid client certificate is found but the server is not responsive. A restart may be necessary to retrieve new initial credentials.","lastCertificateAvailabilityTime":1686800683571.5056,"shutdownThreshold":"5m0s"}
192.168.242.188: {"ts":1686800683786.7485,"caller":"lease/controller.go:146","msg":"Failed to ensure lease exists, will retry","interval":"7s","err":"Get \"https://localhost:6443/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/talos-0sr-1c0?timeout=10s\": dial tcp [::1]:6443: connect: connection refused"}
192.168.242.188: {"ts":1686800684098.6763,"caller":"record/event.go:289","msg":"Unable to write event: '&v1.Event{TypeMeta:v1.TypeMeta{Kind:\"\", APIVersion:\"\"}, ObjectMeta:v1.ObjectMeta{Name:\"talos-0sr-1c0.176877f833c37cf0\", GenerateName:\"\", Namespace:\"default\", SelfLink:\"\", UID:\"\", ResourceVersion:\"\", Generation:0, CreationTimestamp:time.Date(1, time.January, 1, 0, 0, 0, 0, time.UTC), DeletionTimestamp:<nil>, DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string(nil), Annotations:map[string]string(nil), OwnerReferences:[]v1.OwnerReference(nil), Finalizers:[]string(nil), ManagedFields:[]v1.ManagedFieldsEntry(nil)}, InvolvedObject:v1.ObjectReference{Kind:\"Node\", Namespace:\"\", Name:\"talos-0sr-1c0\", UID:\"talos-0sr-1c0\", APIVersion:\"\", ResourceVersion:\"\", FieldPath:\"\"}, Reason:\"NodeHasSufficientPID\", Message:\"Node talos-0sr-1c0 status is now: NodeHasSufficientPID\", Source:v1.EventSource{Component:\"kubelet\", Host:\"talos-0sr-1c0\"}, FirstTimestamp:time.Date(2023, time.June, 14, 8, 6, 8, 354295024, time.Local), LastTimestamp:time.Date(2023, time.June, 14, 8, 6, 28, 670925733, time.Local), Count:10, Type:\"Normal\", EventTime:time.Date(1, time.January, 1, 0, 0, 0, 0, time.UTC), Series:(*v1.EventSeries)(nil), Action:\"\", Related:(*v1.ObjectReference)(nil), ReportingController:\"\", ReportingInstance:\"\"}': 'Patch \"https://localhost:6443/api/v1/namespaces/default/events/talos-0sr-1c0.176877f833c37cf0\": dial tcp [::1]:6443: connect: connection refused'(may retry after sleeping)\n"}
192.168.242.188: {"ts":1686800684113.6108,"caller":"kubelet/kubelet_node_status.go:70","msg":"Attempting to register node","v":0,"node":{"name":"talos-0sr-1c0"}}
192.168.242.188: {"ts":1686800684119.7327,"caller":"kubelet/kubelet_node_status.go:92","msg":"Unable to register node with API server","node":{"name":"talos-0sr-1c0"},"err":"Post \"https://localhost:6443/api/v1/nodes\": dial tcp [::1]:6443: connect: connection refused"}
192.168.242.188: {"ts":1686800684569.721,"caller":"certificate/transport.go:123","msg":"No valid client certificate is found but the server is not responsive. A restart may be necessary to retrieve new initial credentials.","lastCertificateAvailabilityTime":1686800684569.73,"shutdownThreshold":"5m0s"}
192.168.242.188: {"ts":1686800685081.303,"caller":"webhook/webhook.go:223","msg":"Failed to make webhook authorizer request: Post \"https://localhost:6443/apis/authorization.k8s.io/v1/subjectaccessreviews\": dial tcp [::1]:6443: connect: connection refused\n"}
192.168.242.188: {"ts":1686800685082.9692,"caller":"server/server.go:324","msg":"Authorization error","user":"apiserver-kubelet-client","verb":"get","resource":"nodes","subresource":"proxy","err":"Post \"https://localhost:6443/apis/authorization.k8s.io/v1/subjectaccessreviews\": dial tcp [::1]:6443: connect: connection refused"}
192.168.242.188: {"ts":1686800685278.8682,"caller":"status/status_manager.go:809","msg":"Failed to get status for pod","v":0,"podUID":"d936553f85a956933111e63a1eed3d5e","pod":{"name":"kube-apiserver-talos-0sr-1c0","namespace":"kube-system"},"err":"Get \"https://localhost:6443/api/v1/namespaces/kube-system/pods/kube-apiserver-talos-0sr-1c0\": dial tcp [::1]:6443: connect: connection refused"}
192.168.242.188: {"ts":1686800685279.452,"caller":"status/status_manager.go:809","msg":"Failed to get status for pod","v":0,"podUID":"235afef6d6002298a84128703f0db3ef","pod":{"name":"kube-controller-manager-talos-0sr-1c0","namespace":"kube-system"},"err":"Get \"https://localhost:6443/api/v1/namespaces/kube-system/pods/kube-controller-manager-talos-0sr-1c0\": dial tcp [::1]:6443: connect: connection refused"}
192.168.242.188: {"ts":1686800685279.664,"caller":"status/status_manager.go:809","msg":"Failed to get status for pod","v":0,"podUID":"74c0dfb1fbba92866cdad369dbd7ec91","pod":{"name":"kube-scheduler-talos-0sr-1c0","namespace":"kube-system"},"err":"Get \"https://localhost:6443/api/v1/namespaces/kube-system/pods/kube-scheduler-talos-0sr-1c0\": dial tcp [::1]:6443: connect: connection refused"}
192.168.242.188: {"ts":1686800685570.436,"caller":"certificate/transport.go:123","msg":"No valid client certificate is found but the server is not responsive. A restart may be necessary to retrieve new initial credentials.","lastCertificateAvailabilityTime":1686800685570.4453,"shutdownThreshold":"5m0s"}
192.168.242.188: {"ts":1686800686569.6677,"caller":"certificate/transport.go:123","msg":"No valid client certificate is found but the server is not responsive. A restart may be necessary to retrieve new initial credentials.","lastCertificateAvailabilityTime":1686800686569.6802,"shutdownThreshold":"5m0s"}
192.168.242.188: {"ts":1686800687571.779,"caller":"certificate/transport.go:123","msg":"No valid client certificate is found but the server is not responsive. A restart may be necessary to retrieve new initial credentials.","lastCertificateAvailabilityTime":1686800687571.791,"shutdownThreshold":"5m0s"}
192.168.242.188: {"ts":1686800688569.9294,"caller":"certificate/transport.go:123","msg":"No valid client certificate is found but the server is not responsive. A restart may be necessary to retrieve new initial credentials.","lastCertificateAvailabilityTime":1686800688569.9373,"shutdownThreshold":"5m0s"}
192.168.242.188: {"ts":1686800689569.7117,"caller":"certificate/transport.go:123","msg":"No valid client certificate is found but the server is not responsive. A restart may be necessary to retrieve new initial credentials.","lastCertificateAvailabilityTime":1686800689569.7192,"shutdownThreshold":"5m0s"}
192.168.242.188: {"ts":1686800690570.3015,"caller":"certificate/transport.go:123","msg":"No valid client certificate is found but the server is not responsive. A restart may be necessary to retrieve new initial credentials.","lastCertificateAvailabilityTime":1686800690570.3093,"shutdownThreshold":"5m0s"}
192.168.242.188: {"ts":1686800690721.814,"caller":"eviction/eviction_manager.go:262","msg":"Eviction manager: failed to get summary stats","err":"failed to get node info: node \"talos-0sr-1c0\" not found"}
192.168.242.188: {"ts":1686800690793.2747,"caller":"lease/controller.go:146","msg":"Failed to ensure lease exists, will retry","interval":"7s","err":"Get \"https://localhost:6443/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/talos-0sr-1c0?timeout=10s\": dial tcp [::1]:6443: connect: connection refused"}
192.168.242.188: {"ts":1686800691156.2214,"caller":"kubelet/kubelet_node_status.go:70","msg":"Attempting to register node","v":0,"node":{"name":"talos-0sr-1c0"}}
192.168.242.188: {"ts":1686800691166.6245,"caller":"kubelet/kubelet_node_status.go:92","msg":"Unable to register node with API server","node":{"name":"talos-0sr-1c0"},"err":"Post \"https://localhost:6443/api/v1/nodes\": dial tcp [::1]:6443: connect: connection refused"}
192.168.242.188: {"ts":1686800691578.8225,"caller":"certificate/transport.go:123","msg":"No valid client certificate is found but the server is not responsive. A restart may be necessary to retrieve new initial credentials.","lastCertificateAvailabilityTime":1686800691580.3967,"shutdownThreshold":"5m0s"}
192.168.242.188: {"ts":1686800692570.9922,"caller":"certificate/transport.go:123","msg":"No valid client certificate is found but the server is not responsive. A restart may be necessary to retrieve new initial credentials.","lastCertificateAvailabilityTime":1686800692571.003,"shutdownThreshold":"5m0s"}
192.168.242.188: {"ts":1686800692883.2576,"caller":"certificate/transport.go:123","msg":"No valid client certificate is found but the server is not responsive. A restart may be necessary to retrieve new initial credentials.","lastCertificateAvailabilityTime":1686800692883.2676,"shutdownThreshold":"5m0s"}
192.168.242.188: {"ts":1686800693571.3633,"caller":"certificate/transport.go:123","msg":"No valid client certificate is found but the server is not responsive. A restart may be necessary to retrieve new initial credentials.","lastCertificateAvailabilityTime":1686800693571.3735,"shutdownThreshold":"5m0s"}
192.168.242.188: {"ts":1686800694114.1138,"caller":"record/event.go:289","msg":"Unable to write event: '&v1.Event{TypeMeta:v1.TypeMeta{Kind:\"\", APIVersion:\"\"}, ObjectMeta:v1.ObjectMeta{Name:\"talos-0sr-1c0.176877f833c37cf0\", GenerateName:\"\", Namespace:\"default\", SelfLink:\"\", UID:\"\", ResourceVersion:\"\", Generation:0, CreationTimestamp:time.Date(1, time.January, 1, 0, 0, 0, 0, time.UTC), DeletionTimestamp:<nil>, DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string(nil), Annotations:map[string]string(nil), OwnerReferences:[]v1.OwnerReference(nil), Finalizers:[]string(nil), ManagedFields:[]v1.ManagedFieldsEntry(nil)}, InvolvedObject:v1.ObjectReference{Kind:\"Node\", Namespace:\"\", Name:\"talos-0sr-1c0\", UID:\"talos-0sr-1c0\", APIVersion:\"\", ResourceVersion:\"\", FieldPath:\"\"}, Reason:\"NodeHasSufficientPID\", Message:\"Node talos-0sr-1c0 status is now: NodeHasSufficientPID\", Source:v1.EventSource{Component:\"kubelet\", Host:\"talos-0sr-1c0\"}, FirstTimestamp:time.Date(2023, time.June, 14, 8, 6, 8, 354295024, time.Local), LastTimestamp:time.Date(2023, time.June, 14, 8, 6, 28, 670925733, time.Local), Count:10, Type:\"Normal\", EventTime:time.Date(1, time.January, 1, 0, 0, 0, 0, time.UTC), Series:(*v1.EventSeries)(nil), Action:\"\", Related:(*v1.ObjectReference)(nil), ReportingController:\"\", ReportingInstance:\"\"}': 'Patch \"https://localhost:6443/api/v1/namespaces/default/events/talos-0sr-1c0.176877f833c37cf0\": dial tcp [::1]:6443: connect: connection refused'(may retry after sleeping)\n"}
192.168.242.188: {"ts":1686800694569.7632,"caller":"certificate/transport.go:123","msg":"No valid client certificate is found but the server is not responsive. A restart may be necessary to retrieve new initial credentials.","lastCertificateAvailabilityTime":1686800694569.773,"shutdownThreshold":"5m0s"}
192.168.242.188: {"ts":1686800695279.1274,"caller":"status/status_manager.go:809","msg":"Failed to get status for pod","v":0,"podUID":"d936553f85a956933111e63a1eed3d5e","pod":{"name":"kube-apiserver-talos-0sr-1c0","namespace":"kube-system"},"err":"Get \"https://localhost:6443/api/v1/namespaces/kube-system/pods/kube-apiserver-talos-0sr-1c0\": dial tcp [::1]:6443: connect: connection refused"}
192.168.242.188: {"ts":1686800695280.4475,"caller":"status/status_manager.go:809","msg":"Failed to get status for pod","v":0,"podUID":"235afef6d6002298a84128703f0db3ef","pod":{"name":"kube-controller-manager-talos-0sr-1c0","namespace":"kube-system"},"err":"Get \"https://localhost:6443/api/v1/namespaces/kube-system/pods/kube-controller-manager-talos-0sr-1c0\": dial tcp [::1]:6443: connect: connection refused"}
192.168.242.188: {"ts":1686800695280.7898,"caller":"status/status_manager.go:809","msg":"Failed to get status for pod","v":0,"podUID":"74c0dfb1fbba92866cdad369dbd7ec91","pod":{"name":"kube-scheduler-talos-0sr-1c0","namespace":"kube-system"},"err":"Get \"https://localhost:6443/api/v1/namespaces/kube-system/pods/kube-scheduler-talos-0sr-1c0\": dial tcp [::1]:6443: connect: connection refused"}
192.168.242.188: {"ts":1686800695570.058,"caller":"certificate/transport.go:123","msg":"No valid client certificate is found but the server is not responsive. A restart may be necessary to retrieve new initial credentials.","lastCertificateAvailabilityTime":1686800695570.065,"shutdownThreshold":"5m0s"}
192.168.242.188: {"ts":1686800696569.5984,"caller":"certificate/transport.go:123","msg":"No valid client certificate is found but the server is not responsive. A restart may be necessary to retrieve new initial credentials.","lastCertificateAvailabilityTime":1686800696569.611,"shutdownThreshold":"5m0s"}
192.168.242.188: {"ts":1686800697569.655,"caller":"certificate/transport.go:123","msg":"No valid client certificate is found but the server is not responsive. A restart may be necessary to retrieve new initial credentials.","lastCertificateAvailabilityTime":1686800697569.6619,"shutdownThreshold":"5m0s"}
192.168.242.188: {"ts":1686800697798.6013,"caller":"lease/controller.go:146","msg":"Failed to ensure lease exists, will retry","interval":"7s","err":"Get \"https://localhost:6443/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/talos-0sr-1c0?timeout=10s\": dial tcp [::1]:6443: connect: connection refused"}
192.168.242.188: {"ts":1686800698191.38,"caller":"kubelet/kubelet_node_status.go:70","msg":"Attempting to register node","v":0,"node":{"name":"talos-0sr-1c0"}}
192.168.242.188: {"ts":1686800698201.5662,"caller":"kubelet/kubelet_node_status.go:92","msg":"Unable to register node with API server","node":{"name":"talos-0sr-1c0"},"err":"Post \"https://localhost:6443/api/v1/nodes\": dial tcp [::1]:6443: connect: connection refused"}
192.168.242.188: {"ts":1686800698570.1309,"caller":"certificate/transport.go:123","msg":"No valid client certificate is found but the server is not responsive. A restart may be necessary to retrieve new initial credentials.","lastCertificateAvailabilityTime":1686800698570.1438,"shutdownThreshold":"5m0s"}
192.168.242.188: {"ts":1686800699571.9082,"caller":"certificate/transport.go:123","msg":"No valid client certificate is found but the server is not responsive. A restart may be necessary to retrieve new initial credentials.","lastCertificateAvailabilityTime":1686800699571.9153,"shutdownThreshold":"5m0s"}
192.168.242.188: {"ts":1686800700502.0852,"caller":"certificate/certificate_manager.go:562","msg":"kubernetes.io/kube-apiserver-client-kubelet: Failed while requesting a signed certificate from the control plane: cannot create certificate signing request: Post \"https://localhost:6443/apis/certificates.k8s.io/v1/certificatesigningrequests\": dial tcp [::1]:6443: connect: connection refused\n"}
192.168.242.188: {"ts":1686800700571.1812,"caller":"certificate/transport.go:123","msg":"No valid client certificate is found but the server is not responsive. A restart may be necessary to retrieve new initial credentials.","lastCertificateAvailabilityTime":1686800700571.188,"shutdownThreshold":"5m0s"}
192.168.242.188: {"ts":1686800700724.729,"caller":"eviction/eviction_manager.go:262","msg":"Eviction manager: failed to get summary stats","err":"failed to get node info: node \"talos-0sr-1c0\" not found"}
192.168.242.188: {"ts":1686800700849.8787,"caller":"webhook/webhook.go:223","msg":"Failed to make webhook authorizer request: Post \"https://localhost:6443/apis/authorization.k8s.io/v1/subjectaccessreviews\": dial tcp [::1]:6443: connect: connection refused\n"}
192.168.242.188: {"ts":1686800700850.5422,"caller":"server/server.go:324","msg":"Authorization error","user":"apiserver-kubelet-client","verb":"get","resource":"nodes","subresource":"proxy","err":"Post \"https://localhost:6443/apis/authorization.k8s.io/v1/subjectaccessreviews\": dial tcp [::1]:6443: connect: connection refused"}
192.168.242.188: {"ts":1686800701571.6729,"caller":"certificate/transport.go:123","msg":"No valid client certificate is found but the server is not responsive. A restart may be necessary to retrieve new initial credentials.","lastCertificateAvailabilityTime":1686800701571.681,"shutdownThreshold":"5m0s"}
192.168.242.188: {"ts":1686800702401.457,"caller":"cache/reflector.go:533","msg":"vendor/k8s.io/client-go/informers/factory.go:150: failed to list *v1.Node: Get \"https://localhost:6443/api/v1/nodes?fieldSelector=metadata.name%3Dtalos-0sr-1c0&limit=500&resourceVersion=0\": dial tcp [::1]:6443: connect: connection refused\n","v":0}
192.168.242.188: {"ts":1686800702402.0266,"caller":"cache/reflector.go:148","msg":"vendor/k8s.io/client-go/informers/factory.go:150: Failed to watch *v1.Node: failed to list *v1.Node: Get \"https://localhost:6443/api/v1/nodes?fieldSelector=metadata.name%3Dtalos-0sr-1c0&limit=500&resourceVersion=0\": dial tcp [::1]:6443: connect: connection refused\n"}
192.168.242.188: {"ts":1686800702570.1902,"caller":"certificate/transport.go:123","msg":"No valid client certificate is found but the server is not responsive. A restart may be necessary to retrieve new initial credentials.","lastCertificateAvailabilityTime":1686800702570.2021,"shutdownThreshold":"5m0s"}
192.168.242.188: {"ts":1686800702884.2832,"caller":"certificate/transport.go:123","msg":"No valid client certificate is found but the server is not responsive. A restart may be necessary to retrieve new initial credentials.","lastCertificateAvailabilityTime":1686800702884.2927,"shutdownThreshold":"5m0s"}
192.168.242.188: {"ts":1686800703570.1594,"caller":"certificate/transport.go:123","msg":"No valid client certificate is found but the server is not responsive. A restart may be necessary to retrieve new initial credentials.","lastCertificateAvailabilityTime":1686800703570.168,"shutdownThreshold":"5m0s"}
192.168.242.188: {"ts":1686800704134.7026,"caller":"record/event.go:289","msg":"Unable to write event: '&v1.Event{TypeMeta:v1.TypeMeta{Kind:\"\", APIVersion:\"\"}, ObjectMeta:v1.ObjectMeta{Name:\"talos-0sr-1c0.176877f833c37cf0\", GenerateName:\"\", Namespace:\"default\", SelfLink:\"\", UID:\"\", ResourceVersion:\"\", Generation:0, CreationTimestamp:time.Date(1, time.January, 1, 0, 0, 0, 0, time.UTC), DeletionTimestamp:<nil>, DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string(nil), Annotations:map[string]string(nil), OwnerReferences:[]v1.OwnerReference(nil), Finalizers:[]string(nil), ManagedFields:[]v1.ManagedFieldsEntry(nil)}, InvolvedObject:v1.ObjectReference{Kind:\"Node\", Namespace:\"\", Name:\"talos-0sr-1c0\", UID:\"talos-0sr-1c0\", APIVersion:\"\", ResourceVersion:\"\", FieldPath:\"\"}, Reason:\"NodeHasSufficientPID\", Message:\"Node talos-0sr-1c0 status is now: NodeHasSufficientPID\", Source:v1.EventSource{Component:\"kubelet\", Host:\"talos-0sr-1c0\"}, FirstTimestamp:time.Date(2023, time.June, 14, 8, 6, 8, 354295024, time.Local), LastTimestamp:time.Date(2023, time.June, 14, 8, 6, 28, 670925733, time.Local), Count:10, Type:\"Normal\", EventTime:time.Date(1, time.January, 1, 0, 0, 0, 0, time.UTC), Series:(*v1.EventSeries)(nil), Action:\"\", Related:(*v1.ObjectReference)(nil), ReportingController:\"\", ReportingInstance:\"\"}': 'Patch \"https://localhost:6443/api/v1/namespaces/default/events/talos-0sr-1c0.176877f833c37cf0\": dial tcp [::1]:6443: connect: connection refused'(may retry after sleeping)\n"}
192.168.242.188: {"ts":1686800704569.885,"caller":"certificate/transport.go:123","msg":"No valid client certificate is found but the server is not responsive. A restart may be necessary to retrieve new initial credentials.","lastCertificateAvailabilityTime":1686800704569.893,"shutdownThreshold":"5m0s"}
192.168.242.188: {"ts":1686800704607.0688,"caller":"cache/reflector.go:533","msg":"vendor/k8s.io/client-go/informers/factory.go:150: failed to list *v1.Service: Get \"https://localhost:6443/api/v1/services?limit=500&resourceVersion=0\": dial tcp [::1]:6443: connect: connection refused\n","v":0}
192.168.242.188: {"ts":1686800704607.229,"caller":"cache/reflector.go:148","msg":"vendor/k8s.io/client-go/informers/factory.go:150: Failed to watch *v1.Service: failed to list *v1.Service: Get \"https://localhost:6443/api/v1/services?limit=500&resourceVersion=0\": dial tcp [::1]:6443: connect: connection refused\n"}
192.168.242.188: {"ts":1686800704802.3083,"caller":"lease/controller.go:146","msg":"Failed to ensure lease exists, will retry","interval":"7s","err":"Get \"https://localhost:6443/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/talos-0sr-1c0?timeout=10s\": dial tcp [::1]:6443: connect: connection refused"}
192.168.242.188: {"ts":1686800705280.074,"caller":"status/status_manager.go:809","msg":"Failed to get status for pod","v":0,"podUID":"d936553f85a956933111e63a1eed3d5e","pod":{"name":"kube-apiserver-talos-0sr-1c0","namespace":"kube-system"},"err":"Get \"https://localhost:6443/api/v1/namespaces/kube-system/pods/kube-apiserver-talos-0sr-1c0\": dial tcp [::1]:6443: connect: connection refused"}
192.168.242.188: {"ts":1686800705289.139,"caller":"status/status_manager.go:809","msg":"Failed to get status for pod","v":0,"podUID":"235afef6d6002298a84128703f0db3ef","pod":{"name":"kube-controller-manager-talos-0sr-1c0","namespace":"kube-system"},"err":"Get \"https://localhost:6443/api/v1/namespaces/kube-system/pods/kube-controller-manager-talos-0sr-1c0\": dial tcp [::1]:6443: connect: connection refused"}
192.168.242.188: {"ts":1686800705289.366,"caller":"status/status_manager.go:809","msg":"Failed to get status for pod","v":0,"podUID":"74c0dfb1fbba92866cdad369dbd7ec91","pod":{"name":"kube-scheduler-talos-0sr-1c0","namespace":"kube-system"},"err":"Get \"https://localhost:6443/api/v1/namespaces/kube-system/pods/kube-scheduler-talos-0sr-1c0\": dial tcp [::1]:6443: connect: connection refused"}
192.168.242.188: {"ts":1686800705310.0413,"caller":"kubelet/kubelet_node_status.go:70","msg":"Attempting to register node","v":0,"node":{"name":"talos-0sr-1c0"}}
192.168.242.188: {"ts":1686800705317.2166,"caller":"kubelet/kubelet_node_status.go:92","msg":"Unable to register node with API server","node":{"name":"talos-0sr-1c0"},"err":"Post \"https://localhost:6443/api/v1/nodes\": dial tcp [::1]:6443: connect: connection refused"}
192.168.242.188: {"ts":1686800705573.0054,"caller":"certificate/transport.go:123","msg":"No valid client certificate is found but the server is not responsive. A restart may be necessary to retrieve new initial credentials.","lastCertificateAvailabilityTime":1686800705573.0122,"shutdownThreshold":"5m0s"}
192.168.242.188: {"ts":1686800706570.5098,"caller":"certificate/transport.go:123","msg":"No valid client certificate is found but the server is not responsive. A restart may be necessary to retrieve new initial credentials.","lastCertificateAvailabilityTime":1686800706570.5178,"shutdownThreshold":"5m0s"}
192.168.242.188: {"ts":1686800707569.8828,"caller":"certificate/transport.go:123","msg":"No valid client certificate is found but the server is not responsive. A restart may be necessary to retrieve new initial credentials.","lastCertificateAvailabilityTime":1686800707569.9001,"shutdownThreshold":"5m0s"}
192.168.242.188: {"ts":1686800708570.1487,"caller":"certificate/transport.go:123","msg":"No valid client certificate is found but the server is not responsive. A restart may be necessary to retrieve new initial credentials.","lastCertificateAvailabilityTime":1686800708570.1602,"shutdownThreshold":"5m0s"}
192.168.242.188: {"ts":1686800709579.3604,"caller":"certificate/transport.go:123","msg":"No valid client certificate is found but the server is not responsive. A restart may be necessary to retrieve new initial credentials.","lastCertificateAvailabilityTime":1686800709579.3706,"shutdownThreshold":"5m0s"}
192.168.242.188: {"ts":1686800710569.6658,"caller":"certificate/transport.go:123","msg":"No valid client certificate is found but the server is not responsive. A restart may be necessary to retrieve new initial credentials.","lastCertificateAvailabilityTime":1686800710569.6758,"shutdownThreshold":"5m0s"}
192.168.242.188: {"ts":1686800710732.7202,"caller":"eviction/eviction_manager.go:262","msg":"Eviction manager: failed to get summary stats","err":"failed to get node info: node \"talos-0sr-1c0\" not found"}
192.168.242.188: {"ts":1686800711571.5132,"caller":"certificate/transport.go:123","msg":"No valid client certificate is found but the server is not responsive. A restart may be necessary to retrieve new initial credentials.","lastCertificateAvailabilityTime":1686800711573.4817,"shutdownThreshold":"5m0s"}
192.168.242.188: {"ts":1686800711820.6387,"caller":"lease/controller.go:146","msg":"Failed to ensure lease exists, will retry","interval":"7s","err":"Get \"https://localhost:6443/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/talos-0sr-1c0?timeout=10s\": dial tcp [::1]:6443: connect: connection refused"}
192.168.242.188: {"ts":1686800712586.6858,"caller":"certificate/transport.go:123","msg":"No valid client certificate is found but the server is not responsive. A restart may be necessary to retrieve new initial credentials.","lastCertificateAvailabilityTime":1686800712588.7812,"shutdownThreshold":"5m0s"}
192.168.242.188: {"ts":1686800712938.8962,"caller":"certificate/transport.go:123","msg":"No valid client certificate is found but the server is not responsive. A restart may be necessary to retrieve new initial credentials.","lastCertificateAvailabilityTime":1686800712938.9053,"shutdownThreshold":"5m0s"}
192.168.242.188: {"ts":1686800713129.143,"caller":"kubelet/kubelet_node_status.go:70","msg":"Attempting to register node","v":0,"node":{"name":"talos-0sr-1c0"}}
192.168.242.188: {"ts":1686800713147.323,"caller":"kubelet/kubelet_node_status.go:92","msg":"Unable to register node with API server","node":{"name":"talos-0sr-1c0"},"err":"Post \"https://localhost:6443/api/v1/nodes\": dial tcp [::1]:6443: connect: connection refused"}
192.168.242.188: {"ts":1686800713572.008,"caller":"certificate/transport.go:123","msg":"No valid client certificate is found but the server is not responsive. A restart may be necessary to retrieve new initial credentials.","lastCertificateAvailabilityTime":1686800713572.313,"shutdownThreshold":"5m0s"}
192.168.242.188: {"ts":1686800714154.8135,"caller":"record/event.go:289","msg":"Unable to write event: '&v1.Event{TypeMeta:v1.TypeMeta{Kind:\"\", APIVersion:\"\"}, ObjectMeta:v1.ObjectMeta{Name:\"talos-0sr-1c0.176877f833c37cf0\", GenerateName:\"\", Namespace:\"default\", SelfLink:\"\", UID:\"\", ResourceVersion:\"\", Generation:0, CreationTimestamp:time.Date(1, time.January, 1, 0, 0, 0, 0, time.UTC), DeletionTimestamp:<nil>, DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string(nil), Annotations:map[string]string(nil), OwnerReferences:[]v1.OwnerReference(nil), Finalizers:[]string(nil), ManagedFields:[]v1.ManagedFieldsEntry(nil)}, InvolvedObject:v1.ObjectReference{Kind:\"Node\", Namespace:\"\", Name:\"talos-0sr-1c0\", UID:\"talos-0sr-1c0\", APIVersion:\"\", ResourceVersion:\"\", FieldPath:\"\"}, Reason:\"NodeHasSufficientPID\", Message:\"Node talos-0sr-1c0 status is now: NodeHasSufficientPID\", Source:v1.EventSource{Component:\"kubelet\", Host:\"talos-0sr-1c0\"}, FirstTimestamp:time.Date(2023, time.June, 14, 8, 6, 8, 354295024, time.Local), LastTimestamp:time.Date(2023, time.June, 14, 8, 6, 28, 670925733, time.Local), Count:10, Type:\"Normal\", EventTime:time.Date(1, time.January, 1, 0, 0, 0, 0, time.UTC), Series:(*v1.EventSeries)(nil), Action:\"\", Related:(*v1.ObjectReference)(nil), ReportingController:\"\", ReportingInstance:\"\"}': 'Patch \"https://localhost:6443/api/v1/namespaces/default/events/talos-0sr-1c0.176877f833c37cf0\": dial tcp [::1]:6443: connect: connection refused'(may retry after sleeping)\n"}
192.168.242.188: {"ts":1686800714571.2986,"caller":"certificate/transport.go:123","msg":"No valid client certificate is found but the server is not responsive. A restart may be necessary to retrieve new initial credentials.","lastCertificateAvailabilityTime":1686800714571.3062,"shutdownThreshold":"5m0s"}
192.168.242.188: {"ts":1686800715278.426,"caller":"status/status_manager.go:809","msg":"Failed to get status for pod","v":0,"podUID":"d936553f85a956933111e63a1eed3d5e","pod":{"name":"kube-apiserver-talos-0sr-1c0","namespace":"kube-system"},"err":"Get \"https://localhost:6443/api/v1/namespaces/kube-system/pods/kube-apiserver-talos-0sr-1c0\": dial tcp [::1]:6443: connect: connection refused"}
192.168.242.188: {"ts":1686800715279.4375,"caller":"status/status_manager.go:809","msg":"Failed to get status for pod","v":0,"podUID":"235afef6d6002298a84128703f0db3ef","pod":{"name":"kube-controller-manager-talos-0sr-1c0","namespace":"kube-system"},"err":"Get \"https://localhost:6443/api/v1/namespaces/kube-system/pods/kube-controller-manager-talos-0sr-1c0\": dial tcp [::1]:6443: connect: connection refused"}
192.168.242.188: {"ts":1686800715280.0266,"caller":"status/status_manager.go:809","msg":"Failed to get status for pod","v":0,"podUID":"74c0dfb1fbba92866cdad369dbd7ec91","pod":{"name":"kube-scheduler-talos-0sr-1c0","namespace":"kube-system"},"err":"Get \"https://localhost:6443/api/v1/namespaces/kube-system/pods/kube-scheduler-talos-0sr-1c0\": dial tcp [::1]:6443: connect: connection refused"}
192.168.242.188: {"ts":1686800715570.5566,"caller":"certificate/transport.go:123","msg":"No valid client certificate is found but the server is not responsive. A restart may be necessary to retrieve new initial credentials.","lastCertificateAvailabilityTime":1686800715570.5686,"shutdownThreshold":"5m0s"}
192.168.242.188: {"ts":1686800716283.5115,"caller":"webhook/webhook.go:223","msg":"Failed to make webhook authorizer request: Post \"https://localhost:6443/apis/authorization.k8s.io/v1/subjectaccessreviews\": dial tcp [::1]:6443: connect: connection refused\n"}
192.168.242.188: {"ts":1686800716284.9224,"caller":"server/server.go:324","msg":"Authorization error","user":"apiserver-kubelet-client","verb":"get","resource":"nodes","subresource":"proxy","err":"Post \"https://localhost:6443/apis/authorization.k8s.io/v1/subjectaccessreviews\": dial tcp [::1]:6443: connect: connection refused"}
192.168.242.188: {"ts":1686800716570.4805,"caller":"certificate/transport.go:123","msg":"No valid client certificate is found but the server is not responsive. A restart may be necessary to retrieve new initial credentials.","lastCertificateAvailabilityTime":1686800716570.4893,"shutdownThreshold":"5m0s"}
192.168.242.188: {"ts":1686800717571.4182,"caller":"certificate/transport.go:123","msg":"No valid client certificate is found but the server is not responsive. A restart may be necessary to retrieve new initial credentials.","lastCertificateAvailabilityTime":1686800717571.428,"shutdownThreshold":"5m0s"}
192.168.242.188: {"ts":1686800718569.4734,"caller":"certificate/transport.go:123","msg":"No valid client certificate is found but the server is not responsive. A restart may be necessary to retrieve new initial credentials.","lastCertificateAvailabilityTime":1686800718569.481,"shutdownThreshold":"5m0s"}
192.168.242.188: {"ts":1686800718826.3774,"caller":"lease/controller.go:146","msg":"Failed to ensure lease exists, will retry","interval":"7s","err":"Get \"https://localhost:6443/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/talos-0sr-1c0?timeout=10s\": dial tcp [::1]:6443: connect: connection refused"}
192.168.242.188: {"ts":1686800719569.5728,"caller":"certificate/transport.go:123","msg":"No valid client certificate is found but the server is not responsive. A restart may be necessary to retrieve new initial credentials.","lastCertificateAvailabilityTime":1686800719569.581,"shutdownThreshold":"5m0s"}
192.168.242.188: {"ts":1686800720360.9485,"caller":"kubelet/kubelet_node_status.go:70","msg":"Attempting to register node","v":0,"node":{"name":"talos-0sr-1c0"}}
192.168.242.188: {"ts":1686800720372.943,"caller":"kubelet/kubelet_node_status.go:92","msg":"Unable to register node with API server","node":{"name":"talos-0sr-1c0"},"err":"Post \"https://localhost:6443/api/v1/nodes\": dial tcp [::1]:6443: connect: connection refused"}
192.168.242.188: {"ts":1686800720569.5952,"caller":"certificate/transport.go:123","msg":"No valid client certificate is found but the server is not responsive. A restart may be necessary to retrieve new initial credentials.","lastCertificateAvailabilityTime":1686800720569.6045,"shutdownThreshold":"5m0s"}
192.168.242.188: {"ts":1686800720737.6748,"caller":"eviction/eviction_manager.go:262","msg":"Eviction manager: failed to get summary stats","err":"failed to get node info: node \"talos-0sr-1c0\" not found"}
192.168.242.188: {"ts":1686800721569.5718,"caller":"certificate/transport.go:123","msg":"No valid client certificate is found but the server is not responsive. A restart may be necessary to retrieve new initial credentials.","lastCertificateAvailabilityTime":1686800721569.5798,"shutdownThreshold":"5m0s"}
192.168.242.188: {"ts":1686800722569.5598,"caller":"certificate/transport.go:123","msg":"No valid client certificate is found but the server is not responsive. A restart may be necessary to retrieve new initial credentials.","lastCertificateAvailabilityTime":1686800722569.5674,"shutdownThreshold":"5m0s"}
192.168.242.188: {"ts":1686800722939.5947,"caller":"certificate/transport.go:123","msg":"No valid client certificate is found but the server is not responsive. A restart may be necessary to retrieve new initial credentials.","lastCertificateAvailabilityTime":1686800722939.6045,"shutdownThreshold":"5m0s"}
192.168.242.188: {"ts":1686800723569.5479,"caller":"certificate/transport.go:123","msg":"No valid client certificate is found but the server is not responsive. A restart may be necessary to retrieve new initial credentials.","lastCertificateAvailabilityTime":1686800723569.5557,"shutdownThreshold":"5m0s"}
192.168.242.188: {"ts":1686800724170.9695,"caller":"record/event.go:289","msg":"Unable to write event: '&v1.Event{TypeMeta:v1.TypeMeta{Kind:\"\", APIVersion:\"\"}, ObjectMeta:v1.ObjectMeta{Name:\"talos-0sr-1c0.176877f833c37cf0\", GenerateName:\"\", Namespace:\"default\", SelfLink:\"\", UID:\"\", ResourceVersion:\"\", Generation:0, CreationTimestamp:time.Date(1, time.January, 1, 0, 0, 0, 0, time.UTC), DeletionTimestamp:<nil>, DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string(nil), Annotations:map[string]string(nil), OwnerReferences:[]v1.OwnerReference(nil), Finalizers:[]string(nil), ManagedFields:[]v1.ManagedFieldsEntry(nil)}, InvolvedObject:v1.ObjectReference{Kind:\"Node\", Namespace:\"\", Name:\"talos-0sr-1c0\", UID:\"talos-0sr-1c0\", APIVersion:\"\", ResourceVersion:\"\", FieldPath:\"\"}, Reason:\"NodeHasSufficientPID\", Message:\"Node talos-0sr-1c0 status is now: NodeHasSufficientPID\", Source:v1.EventSource{Component:\"kubelet\", Host:\"talos-0sr-1c0\"}, FirstTimestamp:time.Date(2023, time.June, 14, 8, 6, 8, 354295024, time.Local), LastTimestamp:time.Date(2023, time.June, 14, 8, 6, 28, 670925733, time.Local), Count:10, Type:\"Normal\", EventTime:time.Date(1, time.January, 1, 0, 0, 0, 0, time.UTC), Series:(*v1.EventSeries)(nil), Action:\"\", Related:(*v1.ObjectReference)(nil), ReportingController:\"\", ReportingInstance:\"\"}': 'Patch \"https://localhost:6443/api/v1/namespaces/default/events/talos-0sr-1c0.176877f833c37cf0\": dial tcp [::1]:6443: connect: connection refused'(may retry after sleeping)\n"}
192.168.242.188: {"ts":1686800724569.4834,"caller":"certificate/transport.go:123","msg":"No valid client certificate is found but the server is not responsive. A restart may be necessary to retrieve new initial credentials.","lastCertificateAvailabilityTime":1686800724569.491,"shutdownThreshold":"5m0s"}
192.168.242.188: {"ts":1686800725280.5708,"caller":"status/status_manager.go:809","msg":"Failed to get status for pod","v":0,"podUID":"235afef6d6002298a84128703f0db3ef","pod":{"name":"kube-controller-manager-talos-0sr-1c0","namespace":"kube-system"},"err":"Get \"https://localhost:6443/api/v1/namespaces/kube-system/pods/kube-controller-manager-talos-0sr-1c0\": dial tcp [::1]:6443: connect: connection refused"}
192.168.242.188: {"ts":1686800725283.1047,"caller":"status/status_manager.go:809","msg":"Failed to get status for pod","v":0,"podUID":"74c0dfb1fbba92866cdad369dbd7ec91","pod":{"name":"kube-scheduler-talos-0sr-1c0","namespace":"kube-system"},"err":"Get \"https://localhost:6443/api/v1/namespaces/kube-system/pods/kube-scheduler-talos-0sr-1c0\": dial tcp [::1]:6443: connect: connection refused"}
192.168.242.188: {"ts":1686800725283.6138,"caller":"status/status_manager.go:809","msg":"Failed to get status for pod","v":0,"podUID":"d936553f85a956933111e63a1eed3d5e","pod":{"name":"kube-apiserver-talos-0sr-1c0","namespace":"kube-system"},"err":"Get \"https://localhost:6443/api/v1/namespaces/kube-system/pods/kube-apiserver-talos-0sr-1c0\": dial tcp [::1]:6443: connect: connection refused"}
192.168.242.188: {"ts":1686800725573.8196,"caller":"certificate/transport.go:123","msg":"No valid client certificate is found but the server is not responsive. A restart may be necessary to retrieve new initial credentials.","lastCertificateAvailabilityTime":1686800725573.8271,"shutdownThreshold":"5m0s"}
192.168.242.188: {"ts":1686800725833.625,"caller":"lease/controller.go:146","msg":"Failed to ensure lease exists, will retry","interval":"7s","err":"Get \"https://localhost:6443/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/talos-0sr-1c0?timeout=10s\": dial tcp [::1]:6443: connect: connection refused"}
192.168.242.188: {"ts":1686800726570.3672,"caller":"certificate/transport.go:123","msg":"No valid client certificate is found but the server is not responsive. A restart may be necessary to retrieve new initial credentials.","lastCertificateAvailabilityTime":1686800726570.3752,"shutdownThreshold":"5m0s"}
192.168.242.188: {"ts":1686800727395.9695,"caller":"kubelet/kubelet_node_status.go:70","msg":"Attempting to register node","v":0,"node":{"name":"talos-0sr-1c0"}}
192.168.242.188: {"ts":1686800727403.8074,"caller":"kubelet/kubelet_node_status.go:92","msg":"Unable to register node with API server","node":{"name":"talos-0sr-1c0"},"err":"Post \"https://localhost:6443/api/v1/nodes\": dial tcp [::1]:6443: connect: connection refused"}
192.168.242.188: {"ts":1686800727570.269,"caller":"certificate/transport.go:123","msg":"No valid client certificate is found but the server is not responsive. A restart may be necessary to retrieve new initial credentials.","lastCertificateAvailabilityTime":1686800727570.2766,"shutdownThreshold":"5m0s"}
192.168.242.188: {"ts":1686800727682.184,"caller":"cache/reflector.go:533","msg":"vendor/k8s.io/client-go/informers/factory.go:150: failed to list *v1.CSIDriver: Get \"https://localhost:6443/apis/storage.k8s.io/v1/csidrivers?limit=500&resourceVersion=0\": dial tcp [::1]:6443: connect: connection refused\n","v":0}
192.168.242.188: {"ts":1686800727682.7988,"caller":"cache/reflector.go:148","msg":"vendor/k8s.io/client-go/informers/factory.go:150: Failed to watch *v1.CSIDriver: failed to list *v1.CSIDriver: Get \"https://localhost:6443/apis/storage.k8s.io/v1/csidrivers?limit=500&resourceVersion=0\": dial tcp [::1]:6443: connect: connection refused\n"}

Environment

frezbo commented 1 year ago

Please use k8s 1.27.2 or later, 1.27.1 had a bug with kubelet not starting static pods correctly.

marrgieD commented 1 year ago

Please use k8s 1.27.2 or later, 1.27.1 had a bug with kubelet not starting static pods correctly.

But my colleague use 1.27.1 and succeed, his bare metal is on ESXI and mine on VMWare, so I thin 1.27.1 is ok as well. “not starting static pods correctly” is it cannot bootstrap automatically? I usetalosctl bootstrap to make etcd start. And is 1.27.2 support talos 1.4.0?

marrgieD commented 1 year ago

Please use k8s 1.27.2 or later, 1.27.1 had a bug with kubelet not starting static pods correctly.

[xxx@localhost talosnew]$ talosctl  -n 192.168.242.188  get staticpods
NODE              NAMESPACE   TYPE        ID                        VERSION
192.168.242.188   k8s         StaticPod   kube-apiserver            1
192.168.242.188   k8s         StaticPod   kube-controller-manager   1
192.168.242.188   k8s         StaticPod   kube-scheduler            1

It seems this is ok

frezbo commented 1 year ago

the 1.27.1 bug is intermittent, please upgrade to something above that

smira commented 1 year ago

please also make sure you have even disk/RAM: https://www.talos.dev/v1.4/introduction/system-requirements/#recommended