search

kubernetes / kubeadm

Aggregator for issues filed against kubeadm
Apache License 2.0
3.75k stars 715 forks source link

kubelet.go:2267] node "master" not found #2166

Closed liangguohun closed 4 years ago

liangguohun commented 4 years ago

Versions

kubeadm v1.18.2 linux/amd64 kernal 3.10.0-514.el7.x86_64 local docker Version: 19.03.11 API version: 1.40 Go version: go1.13.10

What happened?

when i vi /etc/kubernetes/manifests/kube-apiserver.yaml
add this line
- --service-node-port-range=1-65535

What you expected to happen?

the master can't find itself ,it can run well before.

6月 07 13:20:54 master systemd[1]: kubelet.service: main process exited, code=exited, status=255/n/a
6月 07 13:20:54 master systemd[1]: Unit kubelet.service entered failed state.
6月 07 13:20:54 master systemd[1]: kubelet.service failed.
6月 07 13:21:04 master systemd[1]: kubelet.service holdoff time over, scheduling restart.
6月 07 13:21:04 master systemd[1]: Started kubelet: The Kubernetes Node Agent.
6月 07 13:21:04 master systemd[1]: Starting kubelet: The Kubernetes Node Agent...
6月 07 13:21:05 master kubelet[1518]: Flag --cgroup-driver has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
6月 07 13:21:05 master kubelet[1518]: Flag --cgroup-driver has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
6月 07 13:21:05 master kubelet[1518]: I0607 13:21:05.018299    1518 server.go:417] Version: v1.18.2
6月 07 13:21:05 master kubelet[1518]: I0607 13:21:05.018602    1518 plugins.go:100] No cloud provider specified.
6月 07 13:21:05 master kubelet[1518]: I0607 13:21:05.018627    1518 server.go:837] Client rotation is on, will bootstrap in background
6月 07 13:21:05 master kubelet[1518]: I0607 13:21:05.020587    1518 certificate_store.go:130] Loading cert/key pair from "/var/lib/kubelet/pki/kubelet-client-current.pem".
6月 07 13:21:05 master kubelet[1518]: I0607 13:21:05.111771    1518 server.go:646] --cgroups-per-qos enabled, but --cgroup-root was not specified.  defaulting to /
6月 07 13:21:05 master kubelet[1518]: I0607 13:21:05.112178    1518 container_manager_linux.go:266] container manager verified user specified cgroup-root exists: []
6月 07 13:21:05 master kubelet[1518]: I0607 13:21:05.112215    1518 container_manager_linux.go:271] Creating Container Manager object based on Node Config: {RuntimeCgroupsName: SystemCgroupsName: KubeletCgroupsName: ContainerRuntime:docker CgroupsPerQOS:true CgroupRoot:/ CgroupDriver:cgroupfs KubeletRootDir:/var/lib/kubelet ProtectKernelDefaults:false NodeAllocatableConfig:{KubeReservedCgroupName: SystemReservedCgroupName: ReservedSystemCPUs: EnforceNodeAllocatable:map[pods:{}] KubeReserved:map[] SystemReserved:map[] HardEvictionThresholds:[{Signal:imagefs.available Operator:LessThan Value:{Quantity:<nil> Percentage:0.15} GracePeriod:0s MinReclaim:<nil>} {Signal:memory.available Operator:LessThan Value:{Quantity:100Mi Percentage:0} GracePeriod:0s MinReclaim:<nil>} {Signal:nodefs.available Operator:LessThan Value:{Quantity:<nil> Percentage:0.1} GracePeriod:0s MinReclaim:<nil>} {Signal:nodefs.inodesFree Operator:LessThan Value:{Quantity:<nil> Percentage:0.05} GracePeriod:0s MinReclaim:<nil>}]} QOSReserved:map[] ExperimentalCPUManagerPolicy:none ExperimentalCPUManagerReconcilePeriod:10s ExperimentalPodPidsLimit:-1 EnforceCPULimits:true CPUCFSQuotaPeriod:100ms ExperimentalTopologyManagerPolicy:none}
6月 07 13:21:05 master kubelet[1518]: I0607 13:21:05.112350    1518 topology_manager.go:126] [topologymanager] Creating topology manager with none policy
6月 07 13:21:05 master kubelet[1518]: I0607 13:21:05.112362    1518 container_manager_linux.go:301] [topologymanager] Initializing Topology Manager with none policy
6月 07 13:21:05 master kubelet[1518]: I0607 13:21:05.112368    1518 container_manager_linux.go:306] Creating device plugin manager: true
6月 07 13:21:05 master kubelet[1518]: I0607 13:21:05.112497    1518 client.go:75] Connecting to docker on unix:///var/run/docker.sock
6月 07 13:21:05 master kubelet[1518]: I0607 13:21:05.112516    1518 client.go:92] Start docker client with request timeout=2m0s
6月 07 13:21:05 master kubelet[1518]: W0607 13:21:05.125216    1518 docker_service.go:561] Hairpin mode set to "promiscuous-bridge" but kubenet is not enabled, falling back to "hairpin-veth"
6月 07 13:21:05 master kubelet[1518]: I0607 13:21:05.125279    1518 docker_service.go:238] Hairpin mode set to "hairpin-veth"
6月 07 13:21:05 master kubelet[1518]: I0607 13:21:05.168940    1518 docker_service.go:253] Docker cri networking managed by cni
6月 07 13:21:05 master kubelet[1518]: I0607 13:21:05.180340    1518 docker_service.go:258] Docker Info: &{ID:QEG2:6LNA:OQSL:IQIV:26VS:F3OG:YGHZ:PQL3:Q4OT:RDXP:SVWI:EJDD Containers:137 ContainersRunning:7 ContainersPaused:0 ContainersStopped:130 Images:17 Driver:overlay2 DriverStatus:[[Backing Filesystem xfs] [Supports d_type true] [Native Overlay Diff false]] SystemStatus:[] Plugins:{Volume:[local] Network:[bridge host ipvlan macvlan null overlay] Authorization:[] Log:[awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog]} MemoryLimit:true SwapLimit:true KernelMemory:true KernelMemoryTCP:true CPUCfsPeriod:true CPUCfsQuota:true CPUShares:true CPUSet:true PidsLimit:true IPv4Forwarding:true BridgeNfIptables:true BridgeNfIP6tables:true Debug:false NFd:59 OomKillDisable:true NGoroutines:64 SystemTime:2020-06-07T13:21:05.169795159+08:00 LoggingDriver:json-file CgroupDriver:cgroupfs NEventsListener:0 KernelVersion:3.10.0-514.el7.x86_64 OperatingSystem:CentOS Linux 7 (Core) OSType:linux Architecture:x86_64 IndexServerAddress:https://index.docker.io/v1/ RegistryConfig:0xc0006780e0 NCPU:8 MemTotal:8186183680 GenericResources:[] DockerRootDir:/var/lib/docker HTTPProxy: HTTPSProxy: NoProxy: Name:master Labels:[] ExperimentalBuild:false ServerVersion:19.03.10 ClusterStore: ClusterAdvertise: Runtimes:map[runc:{Path:runc Args:[]}] DefaultRuntime:runc Swarm:{NodeID: NodeAddr: LocalNodeState:inactive ControlAvailable:false Error: RemoteManagers:[] Nodes:0 Managers:0 Cluster:<nil> Warnings:[]} LiveRestoreEnabled:false Isolation: InitBinary:docker-init ContainerdCommit:{ID:7ad184331fa3e55e52b890ea95e65ba581ae3429 Expected:7ad184331fa3e55e52b890ea95e65ba581ae3429} RuncCommit:{ID:dc9208a3303feef5b3839f4323d9beb36df0a9dd Expected:dc9208a3303feef5b3839f4323d9beb36df0a9dd} InitCommit:{ID:fec3683 Expected:fec3683} SecurityOptions:[name=seccomp,profile=default] ProductLicense: Warnings:[]}
6月 07 13:21:05 master kubelet[1518]: I0607 13:21:05.180481    1518 docker_service.go:271] Setting cgroupDriver to cgroupfs
6月 07 13:21:05 master kubelet[1518]: I0607 13:21:05.195282    1518 remote_runtime.go:59] parsed scheme: ""
6月 07 13:21:05 master kubelet[1518]: I0607 13:21:05.195304    1518 remote_runtime.go:59] scheme "" not registered, fallback to default scheme
6月 07 13:21:05 master kubelet[1518]: I0607 13:21:05.195351    1518 passthrough.go:48] ccResolverWrapper: sending update to cc: {[{/var/run/dockershim.sock  <nil> 0 <nil>}] <nil> <nil>}
6月 07 13:21:05 master kubelet[1518]: I0607 13:21:05.195368    1518 clientconn.go:933] ClientConn switching balancer to "pick_first"
6月 07 13:21:05 master kubelet[1518]: I0607 13:21:05.195434    1518 remote_image.go:50] parsed scheme: ""
6月 07 13:21:05 master kubelet[1518]: I0607 13:21:05.195442    1518 remote_image.go:50] scheme "" not registered, fallback to default scheme
6月 07 13:21:05 master kubelet[1518]: I0607 13:21:05.195456    1518 passthrough.go:48] ccResolverWrapper: sending update to cc: {[{/var/run/dockershim.sock  <nil> 0 <nil>}] <nil> <nil>}
6月 07 13:21:05 master kubelet[1518]: I0607 13:21:05.195462    1518 clientconn.go:933] ClientConn switching balancer to "pick_first"
6月 07 13:21:05 master kubelet[1518]: I0607 13:21:05.195497    1518 kubelet.go:292] Adding pod path: /etc/kubernetes/manifests
6月 07 13:21:05 master kubelet[1518]: I0607 13:21:05.195526    1518 kubelet.go:317] Watching apiserver
6月 07 13:21:05 master kubelet[1518]: E0607 13:21:05.200595    1518 reflector.go:178] k8s.io/kubernetes/pkg/kubelet/kubelet.go:526: Failed to list *v1.Node: Get https://192.168.1.4:6443/api/v1/nodes?fieldSelector=metadata.name%3Dmaster&limit=500&resourceVersion=0: dial tcp 192.168.1.4:6443: connect: connection refused
6月 07 13:21:05 master kubelet[1518]: E0607 13:21:05.201001    1518 reflector.go:178] k8s.io/kubernetes/pkg/kubelet/kubelet.go:517: Failed to list *v1.Service: Get https://192.168.1.4:6443/api/v1/services?limit=500&resourceVersion=0: dial tcp 192.168.1.4:6443: connect: connection refused
6月 07 13:21:05 master kubelet[1518]: E0607 13:21:05.201239    1518 reflector.go:178] k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:46: Failed to list *v1.Pod: Get https://192.168.1.4:6443/api/v1/pods?fieldSelector=spec.nodeName%3Dmaster&limit=500&resourceVersion=0: dial tcp 192.168.1.4:6443: connect: connection refused
6月 07 13:21:05 master kubelet[1518]: E0607 13:21:05.200966    1518 reflector.go:178] k8s.io/kubernetes/pkg/kubelet/kubelet.go:526: Failed to list *v1.Node: Get https://192.168.1.4:6443/api/v1/nodes?fieldSelector=metadata.name%3Dmaster&limit=500&resourceVersion=0: dial tcp 192.168.1.4:6443: connect: connection refused
6月 07 13:21:05 master kubelet[1518]: E0607 13:21:05.208851    1518 reflector.go:178] k8s.io/kubernetes/pkg/kubelet/kubelet.go:517: Failed to list *v1.Service: Get https://192.168.1.4:6443/api/v1/services?limit=500&resourceVersion=0: dial tcp 192.168.1.4:6443: connect: connection refused
6月 07 13:21:05 master kubelet[1518]: E0607 13:21:05.210246    1518 reflector.go:178] k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:46: Failed to list *v1.Pod: Get https://192.168.1.4:6443/api/v1/pods?fieldSelector=spec.nodeName%3Dmaster&limit=500&resourceVersion=0: dial tcp 192.168.1.4:6443: connect: connection refused
6月 07 13:21:07 master kubelet[1518]: E0607 13:21:07.063536    1518 reflector.go:178] k8s.io/kubernetes/pkg/kubelet/kubelet.go:517: Failed to list *v1.Service: Get https://192.168.1.4:6443/api/v1/services?limit=500&resourceVersion=0: dial tcp 192.168.1.4:6443: connect: connection refused
6月 07 13:21:07 master kubelet[1518]: E0607 13:21:07.597821    1518 reflector.go:178] k8s.io/kubernetes/pkg/kubelet/kubelet.go:526: Failed to list *v1.Node: Get https://192.168.1.4:6443/api/v1/nodes?fieldSelector=metadata.name%3Dmaster&limit=500&resourceVersion=0: dial tcp 192.168.1.4:6443: connect: connection refused
6月 07 13:21:08 master kubelet[1518]: E0607 13:21:08.139462    1518 reflector.go:178] k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:46: Failed to list *v1.Pod: Get https://192.168.1.4:6443/api/v1/pods?fieldSelector=spec.nodeName%3Dmaster&limit=500&resourceVersion=0: dial tcp 192.168.1.4:6443: connect: connection refused
^C
[root@master ~]# journalctl -f -u kubelet
-- Logs begin at 日 2020-06-07 02:54:16 CST. --
6月 07 13:22:26 master kubelet[2632]: E0607 13:22:26.768146    2632 kubelet.go:2267] node "master" not found
6月 07 13:22:26 master kubelet[2632]: W0607 13:22:26.782026    2632 status_manager.go:556] Failed to get status for pod "kube-apiserver-master_kube-system(4556025ee7b4ba5d7df1be8a853f2fcf)": Get https://192.168.1.4:6443/api/v1/namespaces/kube-system/pods/kube-apiserver-master: dial tcp 192.168.1.4:6443: connect: connection refused
6月 07 13:22:26 master kubelet[2632]: E0607 13:22:26.861770    2632 reflector.go:178] k8s.io/client-go/informers/factory.go:135: Failed to list *v1.CSIDriver: Get https://192.168.1.4:6443/apis/storage.k8s.io/v1/csidrivers?limit=500&resourceVersion=0: dial tcp 192.168.1.4:6443: connect: connection refused
6月 07 13:22:26 master kubelet[2632]: E0607 13:22:26.868639    2632 kubelet.go:2267] node "master" not found
6月 07 13:22:26 master kubelet[2632]: E0607 13:22:26.969429    2632 kubelet.go:2267] node "master" not found
6月 07 13:22:26 master kubelet[2632]: E0607 13:22:26.975482    2632 csi_plugin.go:271] Failed to initialize CSINodeInfo: error updating CSINode annotation: timed out waiting for the condition; caused by: Get https://192.168.1.4:6443/apis/storage.k8s.io/v1/csinodes/master: dial tcp 192.168.1.4:6443: connect: connection refused
6月 07 13:22:26 master kubelet[2632]: F0607 13:22:26.975515    2632 csi_plugin.go:285] Failed to initialize CSINodeInfo after retrying
6月 07 13:22:26 master systemd[1]: kubelet.service: main process exited, code=exited, status=255/n/a
6月 07 13:22:26 master systemd[1]: Unit kubelet.service entered failed state.
6月 07 13:22:26 master systemd[1]: kubelet.service failed.

How to reproduce it (as minimally and precisely as possible)?

test etcd is well just one master in local network er. i have a old nfs system from the v1.13 kubeadm used, and create the same pvc but running in pending and run the svc then change the port limit

/usr/bin/kubelet \
    --bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.conf \
    --kubeconfig=/etc/kubernetes/kubelet.conf \
    --config=/var/lib/kubelet/config.yaml \
    --resolv-conf=/run/systemd/resolve/resolv.conf \
    --cgroup-driver=cgroupfs

the bootstrap-kubelet.conf  resolv.conf not exists.

/etc/kubernetes/kubelet.conf

apiVersion: v1
clusters:
- cluster:
    certificate-authority-data: LS0tLS1C...........
    server: https://192.168.1.4:6443
  name: kubernetes
contexts:
- context:
    cluster: kubernetes
    user: system:node:master
  name: system:node:master@kubernetes
current-context: system:node:master@kubernetes
kind: Config
preferences: {}
users:
- name: system:node:master
  user:
    client-certificate: /var/lib/kubelet/pki/kubelet-client-current.pem
    client-key: /var/lib/kubelet/pki/kubelet-client-current.pem

/var/lib/kubelet/config.yaml

apiVersion: v1
clusters:
- cluster:
    certificate-authority-data: LS0tLS1CRUdJTiBDRVJU.......
    server: https://192.168.1.4:6443
  name: kubernetes
contexts:
- context:
    cluster: kubernetes
    user: system:node:master
  name: system:node:master@kubernetes
current-context: system:node:master@kubernetes
kind: Config
preferences: {}
users:
- name: system:node:master
  user:
    client-certificate: /var/lib/kubelet/pki/kubelet-client-current.pem
    client-key: /var/lib/kubelet/pki/kubelet-client-current.pem
[root@master kubeadm_v1.18.2]# cat /var/lib/kubelet/config.yaml
apiVersion: kubelet.config.k8s.io/v1beta1
authentication:
  anonymous:
    enabled: false
  webhook:
    cacheTTL: 0s
    enabled: true
  x509:
    clientCAFile: /etc/kubernetes/pki/ca.crt
authorization:
  mode: Webhook
  webhook:
    cacheAuthorizedTTL: 0s
    cacheUnauthorizedTTL: 0s
clusterDNS:
- 10.96.0.10
clusterDomain: cluster.local
cpuManagerReconcilePeriod: 0s
evictionPressureTransitionPeriod: 0s
fileCheckFrequency: 0s
healthzBindAddress: 127.0.0.1
healthzPort: 10248
httpCheckFrequency: 0s
imageMinimumGCAge: 0s
kind: KubeletConfiguration
nodeStatusReportFrequency: 0s
nodeStatusUpdateFrequency: 0s
rotateCertificates: true
runtimeRequestTimeout: 0s
staticPodPath: /etc/kubernetes/manifests
streamingConnectionIdleTimeout: 0s
syncFrequency: 0s
volumeStatsAggPeriod: 0s

Anything else we need to know?

how to make it stable

[root@master kubeadm_v1.18.2]# iptables -S
-P INPUT ACCEPT
-P FORWARD DROP
-P OUTPUT ACCEPT
-N DOCKER
-N DOCKER-ISOLATION-STAGE-1
-N DOCKER-ISOLATION-STAGE-2
-N DOCKER-USER
-N KUBE-FIREWALL
-N KUBE-KUBELET-CANARY
-A INPUT -j KUBE-FIREWALL
-A FORWARD -j DOCKER-USER
-A FORWARD -j DOCKER-ISOLATION-STAGE-1
-A FORWARD -o docker0 -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A FORWARD -o docker0 -j DOCKER
-A FORWARD -i docker0 ! -o docker0 -j ACCEPT
-A FORWARD -i docker0 -o docker0 -j ACCEPT
-A OUTPUT -j KUBE-FIREWALL
-A DOCKER-ISOLATION-STAGE-1 -i docker0 ! -o docker0 -j DOCKER-ISOLATION-STAGE-2
-A DOCKER-ISOLATION-STAGE-1 -j RETURN
-A DOCKER-ISOLATION-STAGE-2 -o docker0 -j DROP
-A DOCKER-ISOLATION-STAGE-2 -j RETURN
-A DOCKER-USER -j RETURN
-A KUBE-FIREWALL -m comment --comment "kubernetes firewall for dropping marked packets" -m mark --mark 0x8000/0x8000 -j DROP
liangguohun commented 4 years ago

i found that in my master the docker images always be remove by kubelet

master kubelet[49597]: 49597 eviction_manager.go:570] eviction manager: cannot evict a critical pod etcd-master_kube-system(7b00a94b9db45f4e2fda9672380546e4)
master kubelet[49597]: 49597 eviction_manager.go:570] eviction manager: cannot evict a critical pod calico-node-9pvgn_kube-system(e9d2b6a2-ded9-4109-8b65-fa8f6ad8f4f1)
master kubelet[49597]: 49597 eviction_manager.go:570] eviction manager: cannot evict a critical pod kube-proxy-qhbqd_kube-system(3d14d18a-9995-43a4-b46d-5f7a7a0402bd)
master kubelet[49597]: 49597 eviction_manager.go:391] eviction manager: unable to evict any pods from the node
master kubelet[49597]: 49597 eviction_manager.go:339] eviction manager: attempting to reclaim ephemeral-storage
master kubelet[49597]: 49597 container_gc.go:85] attempting to delete unused containers
master kubelet[49597]: 49597 image_gc_manager.go:322] attempting to delete unused images
master kubelet[49597]: 49597 image_gc_manager.go:376] [imageGCManager]: Removing image "sha256:86262685d9abb35698a4e03ed13f9ded5b97c6c85b466285e4f367e5232eeee4" to free 36937728 bytes
master kubelet[49597]: 49597 remote_image.go:135] RemoveImage "sha256:86262685d9abb35698a4e03ed13f9ded5b97c6c85b466285e4f367e5232eeee4" from image service failed: rpc error: code = Unknown desc = Error response from daemon: readlink /var/lib/docker/overlay2/l: invalid argument
master kubelet[49597]: 49597 kuberuntime_image.go:120] Remove image "sha256:86262685d9abb35698a4e03ed13f9ded5b97c6c85b466285e4f367e5232eeee4" failed: rpc error: code = Unknown desc = Error response from daemon: readlink /var/lib/docker/overlay2/l: invalid argument
master kubelet[49597]: 49597 image_gc_manager.go:376] [imageGCManager]: Removing image "sha256:58477a771fb42465076dc6a43ee1615db9c85c3805a6818063f5d55eb80f2577" to free 361463901 bytes
master kubelet[49597]: 49597 remote_image.go:135] RemoveImage "sha256:58477a771fb42465076dc6a43ee1615db9c85c3805a6818063f5d55eb80f2577" from image service failed: rpc error: code = Unknown desc = Error response from daemon: conflict: unable to remove repository reference "mongo:4.2.0-bionic" (must force) - container f02d51a13579 is using its referenced image 58477a771fb4
master kubelet[49597]: 49597 kuberuntime_image.go:120] Remove image "sha256:58477a771fb42465076dc6a43ee1615db9c85c3805a6818063f5d55eb80f2577" failed: rpc error: code = Unknown desc = Error response from daemon: conflict: unable to remove repository reference "mongo:4.2.0-bionic" (must force) - container f02d51a13579 is using its referenced image 58477a771fb4
master kubelet[49597]: 49597 image_gc_manager.go:376] [imageGCManager]: Removing image "sha256:5d3600fd5cf27c3a789e7d4a3ce9448086d73a4247a118d8cdcdbabb1aff4ff7" to free 538907195 bytes
master kubelet[49597]: 49597 remote_image.go:135] RemoveImage "sha256:5d3600fd5cf27c3a789e7d4a3ce9448086d73a4247a118d8cdcdbabb1aff4ff7" from image service failed: rpc error: code = Unknown desc = Error response from daemon: conflict: unable to remove repository reference "sonatype/nexus3:3.15.2" (must force) - container 10bfe5ad14bb is using its referenced image 5d3600fd5cf2
master kubelet[49597]: 49597 kuberuntime_image.go:120] Remove image "sha256:5d3600fd5cf27c3a789e7d4a3ce9448086d73a4247a118d8cdcdbabb1aff4ff7" failed: rpc error: code = Unknown desc = Error response from daemon: conflict: unable to remove repository reference "sonatype/nexus3:3.15.2" (must force) - container 10bfe5ad14bb is using its referenced image 5d3600fd5cf2
master kubelet[49597]: 49597 image_gc_manager.go:376] [imageGCManager]: Removing image "sha256:347b3fbd4cb3f2a70cbc3d89ac9fa9a170acb4b3e0185d6c2eb6a92cd0759367" to free 52430272 bytes
master kubelet[49597]: 49597 remote_image.go:135] RemoveImage "sha256:347b3fbd4cb3f2a70cbc3d89ac9fa9a170acb4b3e0185d6c2eb6a92cd0759367" from image service failed: rpc error: code = Unknown desc = Error response from daemon: conflict: unable to remove repository reference "gitlab/gitlab-runner-helper:x86_64-a8a019e0" (must force) - container a19520a0583a is using its referenced image 347b3fbd4cb3
master kubelet[49597]: 49597 kuberuntime_image.go:120] Remove image "sha256:347b3fbd4cb3f2a70cbc3d89ac9fa9a170acb4b3e0185d6c2eb6a92cd0759367" failed: rpc error: code = Unknown desc = Error response from daemon: conflict: unable to remove repository reference "gitlab/gitlab-runner-helper:x86_64-a8a019e0" (must force) - container a19520a0583a is using its referenced image 347b3fbd4cb3
master kubelet[49597]: 49597 image_gc_manager.go:376] [imageGCManager]: Removing image "sha256:9b576cf06cb5ae39d775759ff420e2287e46a6753dad4ffffc2d849a9cb3548d" to free 275178062 bytes
master kubelet[49597]: 49597 remote_image.go:135] RemoveImage "sha256:9b576cf06cb5ae39d775759ff420e2287e46a6753dad4ffffc2d849a9cb3548d" from image service failed: rpc error: code = Unknown desc = Error response from daemon: conflict: unable to remove repository reference "192.168.1.4:5000/maven:3.6.2-ibmjava-8-alpine" (must force) - container aab938f2ea30 is using its referenced image 9b576cf06cb5
master kubelet[49597]: 49597 kuberuntime_image.go:120] Remove image "sha256:9b576cf06cb5ae39d775759ff420e2287e46a6753dad4ffffc2d849a9cb3548d" failed: rpc error: code = Unknown desc = Error response from daemon: conflict: unable to remove repository reference "192.168.1.4:5000/maven:3.6.2-ibmjava-8-alpine" (must force) - container aab938f2ea30 is using its referenced image 9b576cf06cb5
master kubelet[49597]: 49597 eviction_manager.go:425] eviction manager: unexpected error when attempting to reduce ephemeral-storage pressure: wanted to free 9223372036854775807 bytes, but freed 0 bytes space with errors in image deletion: [rpc error: code = Unknown desc = Error response from daemon: readlink /var/lib/docker/overlay2/l: invalid argument, rpc error: code = Unknown desc = Error response from daemon: conflict: unable to remove repository reference "mongo:4.2.0-bionic" (must force) - container f02d51a13579 is using its referenced image 58477a771fb4, rpc error: code = Unknown desc = Error response from daemon: conflict: unable to remove repository reference "sonatype/nexus3:3.15.2" (must force) - container 10bfe5ad14bb is using its referenced image 5d3600fd5cf2, rpc error: code = Unknown desc = Error response from daemon: conflict: unable to remove repository reference "gitlab/gitlab-runner-helper:x86_64-a8a019e0" (must force) - container a19520a0583a is using its referenced image 347b3fbd4cb3, rpc error: code = Unknown desc = Error response from daemon: conflict: unable to remove repository reference "192.168.1.4:5000/maven:3.6.2-ibmjava-8-alpine" (must force) - container aab938f2ea30 is using its referenced image 9b576cf06cb5]
master kubelet[49597]: 49597 eviction_manager.go:350] eviction manager: must evict pod(s) to reclaim ephemeral-storage
master kubelet[49597]: 49597 eviction_manager.go:368] eviction manager: pods ranked for eviction: kube-controller-manager-master_kube-system(bde38af668115eac9d0a0ed7d36ade15), kube-scheduler-master_kube-system(155707e0c19147c8dc5e997f089c0ad1), kube-apiserver-master_kube-system(4556025ee7b4ba5d7df1be8a853f2fcf), etcd-master_kube-system(7b00a94b9db45f4e2fda9672380546e4), calico-node-9pvgn_kube-system(e9d2b6a2-ded9-4109-8b65-fa8f6ad8f4f1), kube-proxy-qhbqd_kube-system(3d14d18a-9995-43a4-b46d-5f7a7a0402bd)
master kubelet[49597]: 49597 eviction_manager.go:570] eviction manager: cannot evict a critical pod kube-controller-manager-master_kube-system(bde38af668115eac9d0a0ed7d36ade15)
master kubelet[49597]: 49597 eviction_manager.go:570] eviction manager: cannot evict a critical pod kube-scheduler-master_kube-system(155707e0c19147c8dc5e997f089c0ad1)
master kubelet[49597]: 49597 eviction_manager.go:570] eviction manager: cannot evict a critical pod kube-apiserver-master_kube-system(4556025ee7b4ba5d7df1be8a853f2fcf)
master kubelet[49597]: 49597 eviction_manager.go:570] eviction manager: cannot evict a critical pod etcd-master_kube-system(7b00a94b9db45f4e2fda9672380546e4)
master kubelet[49597]: 49597 eviction_manager.go:570] eviction manager: cannot evict a critical pod calico-node-9pvgn_kube-system(e9d2b6a2-ded9-4109-8b65-fa8f6ad8f4f1)
master kubelet[49597]: 49597 eviction_manager.go:570] eviction manager: cannot evict a critical pod kube-proxy-qhbqd_kube-system(3d14d18a-9995-43a4-b46d-5f7a7a0402bd)
master kubelet[49597]: 49597 eviction_manager.go:391] eviction manager: unable to evict any pods from the node
liangguohun commented 4 years ago

i know is Eviction Policy

Huangsh17 commented 3 years ago

How to solve it？