search

kubernetes / minikube

Run Kubernetes locally
https://minikube.sigs.k8s.io/
Apache License 2.0
29.14k stars 4.86k forks source link

passing --memory=8g --driver virtualbox does not change memory for vm #9254

Closed woodcockjosh closed 3 years ago

woodcockjosh commented 3 years ago

Steps to reproduce the issue:

  1. minikube start -p my-cluster --memory 8g --driver virtualbox

Full output of failed command: 

🔥  Creating virtualbox VM (CPUs=2, Memory=6000MB, Disk=20000MB) ..

Full output of minikube start command used, if not already included:

Spacees-MacBook-Pro:k8s josh$ minikube start -p my-cluster --memory=16gb
😄  [my-cluster] minikube v1.12.3 on Darwin 10.15.6
✨  Using the virtualbox driver based on user configuration
👍  Starting control plane node my-cluster in cluster my-cluster
🔥  Creating virtualbox VM (CPUs=2, Memory=6000MB, Disk=20000MB) ...
🐳  Preparing Kubernetes v1.16.6 on Docker 19.03.12 ...
🔎  Verifying Kubernetes components...
🌟  Enabled addons: default-storageclass, storage-provisioner
🏄  Done! kubectl is now configured to use "my-cluster"

Optional: Full output of minikube logs command:

==> Docker <== -- Logs begin at Tue 2020-09-15 11:39:53 UTC, end at Tue 2020-09-15 11:44:01 UTC. -- Sep 15 11:40:06 my-cluster dockerd[2393]: time="2020-09-15T11:40:06.748034681Z" level=info msg="loading plugin "io.containerd.runtime.v1.linux"..." type=io.containerd.runtime.v1 Sep 15 11:40:06 my-cluster dockerd[2393]: time="2020-09-15T11:40:06.748100440Z" level=info msg="loading plugin "io.containerd.runtime.v2.task"..." type=io.containerd.runtime.v2 Sep 15 11:40:06 my-cluster dockerd[2393]: time="2020-09-15T11:40:06.748162792Z" level=info msg="loading plugin "io.containerd.monitor.v1.cgroups"..." type=io.containerd.monitor.v1 Sep 15 11:40:06 my-cluster dockerd[2393]: time="2020-09-15T11:40:06.748426192Z" level=info msg="loading plugin "io.containerd.service.v1.tasks-service"..." type=io.containerd.service.v1 Sep 15 11:40:06 my-cluster dockerd[2393]: time="2020-09-15T11:40:06.748448606Z" level=info msg="loading plugin "io.containerd.internal.v1.restart"..." type=io.containerd.internal.v1 Sep 15 11:40:06 my-cluster dockerd[2393]: time="2020-09-15T11:40:06.748505247Z" level=info msg="loading plugin "io.containerd.grpc.v1.containers"..." type=io.containerd.grpc.v1 Sep 15 11:40:06 my-cluster dockerd[2393]: time="2020-09-15T11:40:06.748514790Z" level=info msg="loading plugin "io.containerd.grpc.v1.content"..." type=io.containerd.grpc.v1 Sep 15 11:40:06 my-cluster dockerd[2393]: time="2020-09-15T11:40:06.748521010Z" level=info msg="loading plugin "io.containerd.grpc.v1.diff"..." type=io.containerd.grpc.v1 Sep 15 11:40:06 my-cluster dockerd[2393]: time="2020-09-15T11:40:06.748526935Z" level=info msg="loading plugin "io.containerd.grpc.v1.events"..." type=io.containerd.grpc.v1 Sep 15 11:40:06 my-cluster dockerd[2393]: time="2020-09-15T11:40:06.748532506Z" level=info msg="loading plugin "io.containerd.grpc.v1.healthcheck"..." type=io.containerd.grpc.v1 Sep 15 11:40:06 my-cluster dockerd[2393]: time="2020-09-15T11:40:06.748538679Z" level=info msg="loading plugin "io.containerd.grpc.v1.images"..." type=io.containerd.grpc.v1 Sep 15 11:40:06 my-cluster dockerd[2393]: time="2020-09-15T11:40:06.748544393Z" level=info msg="loading plugin "io.containerd.grpc.v1.leases"..." type=io.containerd.grpc.v1 Sep 15 11:40:06 my-cluster dockerd[2393]: time="2020-09-15T11:40:06.748550072Z" level=info msg="loading plugin "io.containerd.grpc.v1.namespaces"..." type=io.containerd.grpc.v1 Sep 15 11:40:06 my-cluster dockerd[2393]: time="2020-09-15T11:40:06.748557265Z" level=info msg="loading plugin "io.containerd.internal.v1.opt"..." type=io.containerd.internal.v1 Sep 15 11:40:06 my-cluster dockerd[2393]: time="2020-09-15T11:40:06.748585192Z" level=info msg="loading plugin "io.containerd.grpc.v1.snapshots"..." type=io.containerd.grpc.v1 Sep 15 11:40:06 my-cluster dockerd[2393]: time="2020-09-15T11:40:06.748592789Z" level=info msg="loading plugin "io.containerd.grpc.v1.tasks"..." type=io.containerd.grpc.v1 Sep 15 11:40:06 my-cluster dockerd[2393]: time="2020-09-15T11:40:06.748600137Z" level=info msg="loading plugin "io.containerd.grpc.v1.version"..." type=io.containerd.grpc.v1 Sep 15 11:40:06 my-cluster dockerd[2393]: time="2020-09-15T11:40:06.748606741Z" level=info msg="loading plugin "io.containerd.grpc.v1.introspection"..." type=io.containerd.grpc.v1 Sep 15 11:40:06 my-cluster dockerd[2393]: time="2020-09-15T11:40:06.748666181Z" level=info msg=serving... address="/var/run/docker/containerd/containerd-debug.sock" Sep 15 11:40:06 my-cluster dockerd[2393]: time="2020-09-15T11:40:06.748698960Z" level=info msg=serving... address="/var/run/docker/containerd/containerd.sock" Sep 15 11:40:06 my-cluster dockerd[2393]: time="2020-09-15T11:40:06.748705326Z" level=info msg="containerd successfully booted in 0.005297s" Sep 15 11:40:06 my-cluster dockerd[2393]: time="2020-09-15T11:40:06.755113587Z" level=info msg="parsed scheme: \"unix\"" module=grpc Sep 15 11:40:06 my-cluster dockerd[2393]: time="2020-09-15T11:40:06.755134278Z" level=info msg="scheme \"unix\" not registered, fallback to default scheme" module=grpc Sep 15 11:40:06 my-cluster dockerd[2393]: time="2020-09-15T11:40:06.755146020Z" level=info msg="ccResolverWrapper: sending update to cc: {[{unix:///var/run/docker/containerd/containerd.sock 0 }] }" module=grpc Sep 15 11:40:06 my-cluster dockerd[2393]: time="2020-09-15T11:40:06.755152738Z" level=info msg="ClientConn switching balancer to \"pick_first\"" module=grpc Sep 15 11:40:06 my-cluster dockerd[2393]: time="2020-09-15T11:40:06.755825885Z" level=info msg="parsed scheme: \"unix\"" module=grpc Sep 15 11:40:06 my-cluster dockerd[2393]: time="2020-09-15T11:40:06.755845685Z" level=info msg="scheme \"unix\" not registered, fallback to default scheme" module=grpc Sep 15 11:40:06 my-cluster dockerd[2393]: time="2020-09-15T11:40:06.755857107Z" level=info msg="ccResolverWrapper: sending update to cc: {[{unix:///var/run/docker/containerd/containerd.sock 0 }] }" module=grpc Sep 15 11:40:06 my-cluster dockerd[2393]: time="2020-09-15T11:40:06.755864011Z" level=info msg="ClientConn switching balancer to \"pick_first\"" module=grpc Sep 15 11:40:06 my-cluster dockerd[2393]: time="2020-09-15T11:40:06.773557864Z" level=warning msg="Your kernel does not support cgroup blkio weight" Sep 15 11:40:06 my-cluster dockerd[2393]: time="2020-09-15T11:40:06.773579429Z" level=warning msg="Your kernel does not support cgroup blkio weight_device" Sep 15 11:40:06 my-cluster dockerd[2393]: time="2020-09-15T11:40:06.773585950Z" level=warning msg="Your kernel does not support cgroup blkio throttle.read_bps_device" Sep 15 11:40:06 my-cluster dockerd[2393]: time="2020-09-15T11:40:06.773589936Z" level=warning msg="Your kernel does not support cgroup blkio throttle.write_bps_device" Sep 15 11:40:06 my-cluster dockerd[2393]: time="2020-09-15T11:40:06.773593649Z" level=warning msg="Your kernel does not support cgroup blkio throttle.read_iops_device" Sep 15 11:40:06 my-cluster dockerd[2393]: time="2020-09-15T11:40:06.773597195Z" level=warning msg="Your kernel does not support cgroup blkio throttle.write_iops_device" Sep 15 11:40:06 my-cluster dockerd[2393]: time="2020-09-15T11:40:06.773750888Z" level=info msg="Loading containers: start." Sep 15 11:40:06 my-cluster dockerd[2393]: time="2020-09-15T11:40:06.820522319Z" level=info msg="Default bridge (docker0) is assigned with an IP address 172.17.0.0/16. Daemon option --bip can be used to set a preferred IP address" Sep 15 11:40:06 my-cluster dockerd[2393]: time="2020-09-15T11:40:06.853561145Z" level=info msg="Loading containers: done." Sep 15 11:40:06 my-cluster dockerd[2393]: time="2020-09-15T11:40:06.866026536Z" level=info msg="Docker daemon" commit=48a66213fe graphdriver(s)=overlay2 version=19.03.12 Sep 15 11:40:06 my-cluster dockerd[2393]: time="2020-09-15T11:40:06.871002234Z" level=info msg="Daemon has completed initialization" Sep 15 11:40:06 my-cluster dockerd[2393]: time="2020-09-15T11:40:06.882919881Z" level=info msg="API listen on /var/run/docker.sock" Sep 15 11:40:06 my-cluster dockerd[2393]: time="2020-09-15T11:40:06.882966315Z" level=info msg="API listen on [::]:2376" Sep 15 11:40:06 my-cluster systemd[1]: Started Docker Application Container Engine. Sep 15 11:40:36 my-cluster dockerd[2393]: time="2020-09-15T11:40:36.239615549Z" level=info msg="shim containerd-shim started" address="/containerd-shim/moby/4e50dd45488ef1ab58a916461996fcdd0ff6679daf3b2a22fbf2b6dcbe174733/shim.sock" debug=false pid=3695 Sep 15 11:40:36 my-cluster dockerd[2393]: time="2020-09-15T11:40:36.240552242Z" level=info msg="shim containerd-shim started" address="/containerd-shim/moby/79463fad5715ef3bc483310c2d0e0fd6f9d16f060e2d64501dd6f9b55ccf0fe2/shim.sock" debug=false pid=3700 Sep 15 11:40:36 my-cluster dockerd[2393]: time="2020-09-15T11:40:36.240731987Z" level=info msg="shim containerd-shim started" address="/containerd-shim/moby/1fa499eeeeadc85b3dc30dbce4db17e24480168116209cb793fee550ca5d8d9a/shim.sock" debug=false pid=3696 Sep 15 11:40:36 my-cluster dockerd[2393]: time="2020-09-15T11:40:36.243484770Z" level=info msg="shim containerd-shim started" address="/containerd-shim/moby/27d6b92ef63c36cec8f7fe7e209152c6aa0a6e09170d774fbd7dd2d855264d48/shim.sock" debug=false pid=3719 Sep 15 11:40:36 my-cluster dockerd[2393]: time="2020-09-15T11:40:36.435717294Z" level=info msg="shim containerd-shim started" address="/containerd-shim/moby/c2281b58d19efd5a96efa05ba2642e4842b857c6187715a99aafb921143660a7/shim.sock" debug=false pid=3840 Sep 15 11:40:36 my-cluster dockerd[2393]: time="2020-09-15T11:40:36.518909207Z" level=info msg="shim containerd-shim started" address="/containerd-shim/moby/4cffca0b8f2730e208adb3fd1f1666644fa650022b2f9417dad47c4561039aa0/shim.sock" debug=false pid=3874 Sep 15 11:40:36 my-cluster dockerd[2393]: time="2020-09-15T11:40:36.520290459Z" level=info msg="shim containerd-shim started" address="/containerd-shim/moby/74707db23bd95cd881d7f8fcaddc9e35bf3ba2c1f31d94df49eb02f59b5c1779/shim.sock" debug=false pid=3875 Sep 15 11:40:36 my-cluster dockerd[2393]: time="2020-09-15T11:40:36.527961444Z" level=info msg="shim containerd-shim started" address="/containerd-shim/moby/b0131d3d02f76d7226a922cee699b7a225c8d2040800edb5fb39305b34973687/shim.sock" debug=false pid=3893 Sep 15 11:40:50 my-cluster dockerd[2393]: time="2020-09-15T11:40:50.019666044Z" level=info msg="shim containerd-shim started" address="/containerd-shim/moby/ec26dc1ce55043ede7943b37664b5038e10330eb6fdc379015a7d31065e240dc/shim.sock" debug=false pid=4291 Sep 15 11:40:50 my-cluster dockerd[2393]: time="2020-09-15T11:40:50.058769105Z" level=info msg="shim containerd-shim started" address="/containerd-shim/moby/4d43bf348abfe8b2b77f1ce0f427eafedf9def82e40e90b74dc728a24069765c/shim.sock" debug=false pid=4305 Sep 15 11:40:50 my-cluster dockerd[2393]: time="2020-09-15T11:40:50.087751187Z" level=info msg="shim containerd-shim started" address="/containerd-shim/moby/df6acc1c47ed6081ae2149ca59536424ebd4d45da0b75fb663fd45e875601525/shim.sock" debug=false pid=4320 Sep 15 11:40:50 my-cluster dockerd[2393]: time="2020-09-15T11:40:50.262022055Z" level=info msg="shim containerd-shim started" address="/containerd-shim/moby/5d70a2da1b6e45c01636bd55414e8677682bf93863d7284713f30824e581ed84/shim.sock" debug=false pid=4407 Sep 15 11:40:50 my-cluster dockerd[2393]: time="2020-09-15T11:40:50.320701818Z" level=info msg="shim containerd-shim started" address="/containerd-shim/moby/ddc0c0687de9a5d22ab35ade03ee4169de0c2d6b39c928d023079cc56b29f447/shim.sock" debug=false pid=4435 Sep 15 11:40:50 my-cluster dockerd[2393]: time="2020-09-15T11:40:50.429547079Z" level=info msg="shim containerd-shim started" address="/containerd-shim/moby/728b9615bf80abc69b2ea802561666a003d5ad951c0bc4bfbf3dd85ec7e7dc10/shim.sock" debug=false pid=4501 Sep 15 11:41:20 my-cluster dockerd[2393]: time="2020-09-15T11:41:20.485335233Z" level=info msg="shim reaped" id=5d70a2da1b6e45c01636bd55414e8677682bf93863d7284713f30824e581ed84 Sep 15 11:41:20 my-cluster dockerd[2393]: time="2020-09-15T11:41:20.495507640Z" level=info msg="ignoring event" module=libcontainerd namespace=moby topic=/tasks/delete type="*events.TaskDelete" Sep 15 11:41:20 my-cluster dockerd[2393]: time="2020-09-15T11:41:20.759256327Z" level=info msg="shim containerd-shim started" address="/containerd-shim/moby/70b7e36e263cb24aa3244ab9a2608566a166db56864129716b4b844af580276f/shim.sock" debug=false pid=4744 ==> container status <== CONTAINER IMAGE CREATED STATE NAME ATTEMPT POD ID 70b7e36e263cb ac5e2ed5acc50 2 minutes ago Running storage-provisioner 1 ec26dc1ce5504 728b9615bf80a bf261d1579144 3 minutes ago Running coredns 0 df6acc1c47ed6 ddc0c0687de9a 284f0d3c94206 3 minutes ago Running kube-proxy 0 4d43bf348abfe 5d70a2da1b6e4 ac5e2ed5acc50 3 minutes ago Exited storage-provisioner 0 ec26dc1ce5504 b0131d3d02f76 cd48205a40f00 3 minutes ago Running kube-controller-manager 0 1fa499eeeeadc 4cffca0b8f273 b2756210eeabf 3 minutes ago Running etcd 0 27d6b92ef63c3 74707db23bd95 5732fe50f6f52 3 minutes ago Running kube-apiserver 0 4e50dd45488ef c2281b58d19ef 6bed756ced733 3 minutes ago Running kube-scheduler 0 79463fad5715e ==> coredns [728b9615bf80] <== .:53 2020-09-15T11:40:55.548Z [INFO] plugin/reload: Running configuration MD5 = f64cb9b977c7dfca58c4fab108535a76 2020-09-15T11:40:55.548Z [INFO] CoreDNS-1.6.2 2020-09-15T11:40:55.548Z [INFO] linux/amd64, go1.12.8, 795a3eb CoreDNS-1.6.2 linux/amd64, go1.12.8, 795a3eb 2020-09-15T11:40:56.951Z [INFO] plugin/ready: Still waiting on: "kubernetes" 2020-09-15T11:41:06.951Z [INFO] plugin/ready: Still waiting on: "kubernetes" 2020-09-15T11:41:16.951Z [INFO] plugin/ready: Still waiting on: "kubernetes" I0915 11:41:20.549032 1 trace.go:82] Trace[240545644]: "Reflector pkg/mod/k8s.io/client-go@v11.0.0+incompatible/tools/cache/reflector.go:94 ListAndWatch" (started: 2020-09-15 11:40:50.54815617 +0000 UTC m=+0.016585572) (total time: 30.000854001s): Trace[240545644]: [30.000854001s] [30.000854001s] END E0915 11:41:20.549050 1 reflector.go:126] pkg/mod/k8s.io/client-go@v11.0.0+incompatible/tools/cache/reflector.go:94: Failed to list *v1.Endpoints: Get https://10.96.0.1:443/api/v1/endpoints?limit=500&resourceVersion=0: dial tcp 10.96.0.1:443: i/o timeout E0915 11:41:20.549050 1 reflector.go:126] pkg/mod/k8s.io/client-go@v11.0.0+incompatible/tools/cache/reflector.go:94: Failed to list *v1.Endpoints: Get https://10.96.0.1:443/api/v1/endpoints?limit=500&resourceVersion=0: dial tcp 10.96.0.1:443: i/o timeout E0915 11:41:20.549050 1 reflector.go:126] pkg/mod/k8s.io/client-go@v11.0.0+incompatible/tools/cache/reflector.go:94: Failed to list *v1.Endpoints: Get https://10.96.0.1:443/api/v1/endpoints?limit=500&resourceVersion=0: dial tcp 10.96.0.1:443: i/o timeout I0915 11:41:20.549219 1 trace.go:82] Trace[187105362]: "Reflector pkg/mod/k8s.io/client-go@v11.0.0+incompatible/tools/cache/reflector.go:94 ListAndWatch" (started: 2020-09-15 11:40:50.547954289 +0000 UTC m=+0.016383681) (total time: 30.001255402s): Trace[187105362]: [30.001255402s] [30.001255402s] END E0915 11:41:20.549225 1 reflector.go:126] pkg/mod/k8s.io/client-go@v11.0.0+incompatible/tools/cache/reflector.go:94: Failed to list *v1.Service: Get https://10.96.0.1:443/api/v1/services?limit=500&resourceVersion=0: dial tcp 10.96.0.1:443: i/o timeout E0915 11:41:20.549225 1 reflector.go:126] pkg/mod/k8s.io/client-go@v11.0.0+incompatible/tools/cache/reflector.go:94: Failed to list *v1.Service: Get https://10.96.0.1:443/api/v1/services?limit=500&resourceVersion=0: dial tcp 10.96.0.1:443: i/o timeout E0915 11:41:20.549225 1 reflector.go:126] pkg/mod/k8s.io/client-go@v11.0.0+incompatible/tools/cache/reflector.go:94: Failed to list *v1.Service: Get https://10.96.0.1:443/api/v1/services?limit=500&resourceVersion=0: dial tcp 10.96.0.1:443: i/o timeout I0915 11:41:20.549256 1 trace.go:82] Trace[1143163359]: "Reflector pkg/mod/k8s.io/client-go@v11.0.0+incompatible/tools/cache/reflector.go:94 ListAndWatch" (started: 2020-09-15 11:40:50.547555436 +0000 UTC m=+0.015984828) (total time: 30.001682051s): Trace[1143163359]: [30.001682051s] [30.001682051s] END E0915 11:41:20.549268 1 reflector.go:126] pkg/mod/k8s.io/client-go@v11.0.0+incompatible/tools/cache/reflector.go:94: Failed to list *v1.Namespace: Get https://10.96.0.1:443/api/v1/namespaces?limit=500&resourceVersion=0: dial tcp 10.96.0.1:443: i/o timeout E0915 11:41:20.549268 1 reflector.go:126] pkg/mod/k8s.io/client-go@v11.0.0+incompatible/tools/cache/reflector.go:94: Failed to list *v1.Namespace: Get https://10.96.0.1:443/api/v1/namespaces?limit=500&resourceVersion=0: dial tcp 10.96.0.1:443: i/o timeout E0915 11:41:20.549268 1 reflector.go:126] pkg/mod/k8s.io/client-go@v11.0.0+incompatible/tools/cache/reflector.go:94: Failed to list *v1.Namespace: Get https://10.96.0.1:443/api/v1/namespaces?limit=500&resourceVersion=0: dial tcp 10.96.0.1:443: i/o timeout E0915 11:41:20.549050 1 reflector.go:126] pkg/mod/k8s.io/client-go@v11.0.0+incompatible/tools/cache/reflector.go:94: Failed to list *v1.Endpoints: Get https://10.96.0.1:443/api/v1/endpoints?limit=500&resourceVersion=0: dial tcp 10.96.0.1:443: i/o timeout E0915 11:41:20.549225 1 reflector.go:126] pkg/mod/k8s.io/client-go@v11.0.0+incompatible/tools/cache/reflector.go:94: Failed to list *v1.Service: Get https://10.96.0.1:443/api/v1/services?limit=500&resourceVersion=0: dial tcp 10.96.0.1:443: i/o timeout E0915 11:41:20.549268 1 reflector.go:126] pkg/mod/k8s.io/client-go@v11.0.0+incompatible/tools/cache/reflector.go:94: Failed to list *v1.Namespace: Get https://10.96.0.1:443/api/v1/namespaces?limit=500&resourceVersion=0: dial tcp 10.96.0.1:443: i/o timeout ==> describe nodes <== Name: my-cluster Roles: master Labels: beta.kubernetes.io/arch=amd64 beta.kubernetes.io/os=linux kubernetes.io/arch=amd64 kubernetes.io/hostname=my-cluster kubernetes.io/os=linux minikube.k8s.io/commit=2243b4b97c131e3244c5f014faedca0d846599f5 minikube.k8s.io/name=my-cluster minikube.k8s.io/updated_at=2020_09_15T06_40_44_0700 minikube.k8s.io/version=v1.12.3 node-role.kubernetes.io/master= Annotations: kubeadm.alpha.kubernetes.io/cri-socket: /var/run/dockershim.sock node.alpha.kubernetes.io/ttl: 0 volumes.kubernetes.io/controller-managed-attach-detach: true CreationTimestamp: Tue, 15 Sep 2020 11:40:40 +0000 Taints: Unschedulable: false Conditions: Type Status LastHeartbeatTime LastTransitionTime Reason Message ---- ------ ----------------- ------------------ ------ ------- MemoryPressure False Tue, 15 Sep 2020 11:43:41 +0000 Tue, 15 Sep 2020 11:40:36 +0000 KubeletHasSufficientMemory kubelet has sufficient memory available DiskPressure False Tue, 15 Sep 2020 11:43:41 +0000 Tue, 15 Sep 2020 11:40:36 +0000 KubeletHasNoDiskPressure kubelet has no disk pressure PIDPressure False Tue, 15 Sep 2020 11:43:41 +0000 Tue, 15 Sep 2020 11:40:36 +0000 KubeletHasSufficientPID kubelet has sufficient PID available Ready True Tue, 15 Sep 2020 11:43:41 +0000 Tue, 15 Sep 2020 11:40:36 +0000 KubeletReady kubelet is posting ready status Addresses: InternalIP: 192.168.99.103 Hostname: my-cluster Capacity: cpu: 2 ephemeral-storage: 17784752Ki hugepages-2Mi: 0 memory: 5952056Ki pods: 110 Allocatable: cpu: 2 ephemeral-storage: 17784752Ki hugepages-2Mi: 0 memory: 5952056Ki pods: 110 System Info: Machine ID: eef140315bc143428eba1dbe8798272d System UUID: b8547f89-1ec9-de49-af03-1b8eb1bb39db Boot ID: 3399abfb-2b97-41cb-b73b-338dc2d3b380 Kernel Version: 4.19.114 OS Image: Buildroot 2019.02.11 Operating System: linux Architecture: amd64 Container Runtime Version: docker://19.3.12 Kubelet Version: v1.16.6 Kube-Proxy Version: v1.16.6 Non-terminated Pods: (7 in total) Namespace Name CPU Requests CPU Limits Memory Requests Memory Limits AGE --------- ---- ------------ ---------- --------------- ------------- --- kube-system coredns-5644d7b6d9-882h8 100m (5%) 0 (0%) 70Mi (1%) 170Mi (2%) 3m13s kube-system etcd-my-cluster 0 (0%) 0 (0%) 0 (0%) 0 (0%) 2m22s kube-system kube-apiserver-my-cluster 250m (12%) 0 (0%) 0 (0%) 0 (0%) 2m23s kube-system kube-controller-manager-my-cluster 200m (10%) 0 (0%) 0 (0%) 0 (0%) 2m22s kube-system kube-proxy-jbmx2 0 (0%) 0 (0%) 0 (0%) 0 (0%) 3m13s kube-system kube-scheduler-my-cluster 100m (5%) 0 (0%) 0 (0%) 0 (0%) 2m21s kube-system storage-provisioner 0 (0%) 0 (0%) 0 (0%) 0 (0%) 3m18s Allocated resources: (Total limits may be over 100 percent, i.e., overcommitted.) Resource Requests Limits -------- -------- ------ cpu 650m (32%) 0 (0%) memory 70Mi (1%) 170Mi (2%) ephemeral-storage 0 (0%) 0 (0%) Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal NodeHasSufficientMemory 3m27s (x8 over 3m27s) kubelet, my-cluster Node my-cluster status is now: NodeHasSufficientMemory Normal NodeHasNoDiskPressure 3m27s (x8 over 3m27s) kubelet, my-cluster Node my-cluster status is now: NodeHasNoDiskPressure Normal NodeHasSufficientPID 3m27s (x7 over 3m27s) kubelet, my-cluster Node my-cluster status is now: NodeHasSufficientPID Normal Starting 3m12s kube-proxy, my-cluster Starting kube-proxy. ==> dmesg <== [ +0.000064] 00:00:00.002558 main 5.2.42 r137960 started. Verbose level = 0 [ +0.422367] NFSD: the nfsdcld client tracking upcall will be removed in 3.10. Please transition to using nfsdcltrack. [ +4.608438] hpet1: lost 286 rtc interrupts [Sep15 11:40] hpet1: lost 319 rtc interrupts [ +3.449943] systemd-fstab-generator[2373]: Ignoring "noauto" for root device [ +0.065259] systemd-fstab-generator[2383]: Ignoring "noauto" for root device [ +0.991276] systemd-fstab-generator[2567]: Ignoring "noauto" for root device [ +0.494939] hpet_rtc_timer_reinit: 66 callbacks suppressed [ +0.000009] hpet1: lost 318 rtc interrupts [ +5.001770] hpet1: lost 318 rtc interrupts [ +5.000745] hpet1: lost 318 rtc interrupts [ +4.999975] hpet1: lost 318 rtc interrupts [ +3.577921] systemd-fstab-generator[3020]: Ignoring "noauto" for root device [ +1.000549] systemd-fstab-generator[3224]: Ignoring "noauto" for root device [ +0.422334] hpet1: lost 318 rtc interrupts [ +5.001242] hpet1: lost 318 rtc interrupts [ +5.001362] hpet_rtc_timer_reinit: 24 callbacks suppressed [ +0.000009] hpet1: lost 318 rtc interrupts [ +5.000995] hpet1: lost 318 rtc interrupts [ +5.000707] hpet1: lost 318 rtc interrupts [ +10.002072] hpet_rtc_timer_reinit: 34 callbacks suppressed [ +0.000010] hpet1: lost 318 rtc interrupts [Sep15 11:41] hpet1: lost 318 rtc interrupts [ +5.001249] hpet1: lost 318 rtc interrupts [ +5.000375] hpet1: lost 318 rtc interrupts [ +5.002064] hpet1: lost 319 rtc interrupts [ +8.828591] kauditd_printk_skb: 4 callbacks suppressed [ +1.173510] hpet1: lost 318 rtc interrupts [ +5.000403] hpet1: lost 318 rtc interrupts [ +5.002117] hpet1: lost 318 rtc interrupts [ +5.000466] hpet1: lost 318 rtc interrupts [ +5.001932] hpet1: lost 318 rtc interrupts [ +5.000411] hpet1: lost 318 rtc interrupts [ +4.272595] NFSD: Unable to end grace period: -110 [ +0.729112] hpet1: lost 318 rtc interrupts [Sep15 11:42] hpet1: lost 318 rtc interrupts [ +5.001100] hpet1: lost 318 rtc interrupts [ +5.001164] hpet1: lost 318 rtc interrupts [ +5.001057] hpet1: lost 318 rtc interrupts [ +5.001634] hpet1: lost 319 rtc interrupts [ +5.001793] hpet1: lost 318 rtc interrupts [ +5.000431] hpet1: lost 318 rtc interrupts [ +5.001279] hpet1: lost 318 rtc interrupts [ +5.000962] hpet1: lost 318 rtc interrupts [ +5.001525] hpet1: lost 318 rtc interrupts [ +5.000492] hpet1: lost 318 rtc interrupts [ +5.000656] hpet1: lost 318 rtc interrupts [Sep15 11:43] hpet1: lost 318 rtc interrupts [ +5.001366] hpet1: lost 318 rtc interrupts [ +5.001901] hpet1: lost 318 rtc interrupts [ +5.001179] hpet1: lost 318 rtc interrupts [ +5.000750] hpet1: lost 318 rtc interrupts [ +5.001570] hpet1: lost 319 rtc interrupts [ +5.002095] hpet1: lost 318 rtc interrupts [ +5.000633] hpet1: lost 319 rtc interrupts [ +5.001608] hpet1: lost 318 rtc interrupts [ +5.001417] hpet1: lost 318 rtc interrupts [ +5.000525] hpet1: lost 318 rtc interrupts [ +5.001720] hpet1: lost 318 rtc interrupts [Sep15 11:44] hpet1: lost 318 rtc interrupts ==> etcd [4cffca0b8f27] <== 2020-09-15 11:40:37.140828 I | etcdmain: etcd Version: 3.3.15 2020-09-15 11:40:37.140870 I | etcdmain: Git SHA: 94745a4ee 2020-09-15 11:40:37.140873 I | etcdmain: Go Version: go1.12.9 2020-09-15 11:40:37.140875 I | etcdmain: Go OS/Arch: linux/amd64 2020-09-15 11:40:37.140878 I | etcdmain: setting maximum number of CPUs to 2, total number of available CPUs is 2 2020-09-15 11:40:37.141045 I | embed: peerTLS: cert = /var/lib/minikube/certs/etcd/peer.crt, key = /var/lib/minikube/certs/etcd/peer.key, ca = , trusted-ca = /var/lib/minikube/certs/etcd/ca.crt, client-cert-auth = true, crl-file = 2020-09-15 11:40:37.141765 I | embed: listening for peers on https://192.168.99.103:2380 2020-09-15 11:40:37.141799 I | embed: listening for client requests on 127.0.0.1:2379 2020-09-15 11:40:37.141812 I | embed: listening for client requests on 192.168.99.103:2379 2020-09-15 11:40:37.143397 I | etcdserver: name = my-cluster 2020-09-15 11:40:37.143438 I | etcdserver: data dir = /var/lib/minikube/etcd 2020-09-15 11:40:37.143442 I | etcdserver: member dir = /var/lib/minikube/etcd/member 2020-09-15 11:40:37.143447 I | etcdserver: heartbeat = 100ms 2020-09-15 11:40:37.143449 I | etcdserver: election = 1000ms 2020-09-15 11:40:37.143451 I | etcdserver: snapshot count = 10000 2020-09-15 11:40:37.143456 I | etcdserver: advertise client URLs = https://192.168.99.103:2379 2020-09-15 11:40:37.143458 I | etcdserver: initial advertise peer URLs = https://192.168.99.103:2380 2020-09-15 11:40:37.143462 I | etcdserver: initial cluster = my-cluster=https://192.168.99.103:2380 2020-09-15 11:40:37.242410 I | etcdserver: starting member bd07d860304161f8 in cluster 6ffb9f662ec1e33b 2020-09-15 11:40:37.242427 I | raft: bd07d860304161f8 became follower at term 0 2020-09-15 11:40:37.279287 I | raft: newRaft bd07d860304161f8 [peers: [], term: 0, commit: 0, applied: 0, lastindex: 0, lastterm: 0] 2020-09-15 11:40:37.288518 I | raft: bd07d860304161f8 became follower at term 1 2020-09-15 11:40:37.522454 W | auth: simple token is not cryptographically signed 2020-09-15 11:40:37.648871 I | etcdserver: starting server... [version: 3.3.15, cluster version: to_be_decided] 2020-09-15 11:40:37.652181 I | embed: ClientTLS: cert = /var/lib/minikube/certs/etcd/server.crt, key = /var/lib/minikube/certs/etcd/server.key, ca = , trusted-ca = /var/lib/minikube/certs/etcd/ca.crt, client-cert-auth = true, crl-file = 2020-09-15 11:40:37.652596 I | embed: listening for metrics on http://192.168.99.103:2381 2020-09-15 11:40:37.653061 I | etcdserver: bd07d860304161f8 as single-node; fast-forwarding 9 ticks (election ticks 10) 2020-09-15 11:40:37.653098 W | raft: bd07d860304161f8 cannot campaign at term 1 since there are still 1 pending configuration changes to apply 2020-09-15 11:40:37.654164 I | embed: listening for metrics on http://127.0.0.1:2381 2020-09-15 11:40:37.654640 I | etcdserver/membership: added member bd07d860304161f8 [https://192.168.99.103:2380] to cluster 6ffb9f662ec1e33b 2020-09-15 11:40:38.589183 I | raft: bd07d860304161f8 is starting a new election at term 1 2020-09-15 11:40:38.589279 I | raft: bd07d860304161f8 became candidate at term 2 2020-09-15 11:40:38.589293 I | raft: bd07d860304161f8 received MsgVoteResp from bd07d860304161f8 at term 2 2020-09-15 11:40:38.589304 I | raft: bd07d860304161f8 became leader at term 2 2020-09-15 11:40:38.589310 I | raft: raft.node: bd07d860304161f8 elected leader bd07d860304161f8 at term 2 2020-09-15 11:40:38.589739 I | etcdserver: published {Name:my-cluster ClientURLs:[https://192.168.99.103:2379]} to cluster 6ffb9f662ec1e33b 2020-09-15 11:40:38.589916 I | embed: ready to serve client requests 2020-09-15 11:40:38.591113 I | embed: serving client requests on 192.168.99.103:2379 2020-09-15 11:40:38.591189 I | etcdserver: setting up the initial cluster version to 3.3 2020-09-15 11:40:38.591424 I | embed: ready to serve client requests 2020-09-15 11:40:38.591638 N | etcdserver/membership: set the initial cluster version to 3.3 2020-09-15 11:40:38.591678 I | etcdserver/api: enabled capabilities for version 3.3 2020-09-15 11:40:38.592343 I | embed: serving client requests on 127.0.0.1:2379 ==> kernel <== 11:44:04 up 4 min, 0 users, load average: 0.27, 0.50, 0.26 Linux my-cluster 4.19.114 #1 SMP Mon Aug 3 12:35:22 PDT 2020 x86_64 GNU/Linux PRETTY_NAME="Buildroot 2019.02.11" ==> kube-apiserver [74707db23bd9] <== I0915 11:40:39.077571 1 endpoint.go:68] ccResolverWrapper: sending new addresses to cc: [{https://127.0.0.1:2379 0 }] I0915 11:40:39.092407 1 client.go:357] parsed scheme: "endpoint" I0915 11:40:39.092623 1 endpoint.go:68] ccResolverWrapper: sending new addresses to cc: [{https://127.0.0.1:2379 0 }] I0915 11:40:39.104757 1 client.go:357] parsed scheme: "endpoint" I0915 11:40:39.104866 1 endpoint.go:68] ccResolverWrapper: sending new addresses to cc: [{https://127.0.0.1:2379 0 }] I0915 11:40:39.112499 1 client.go:357] parsed scheme: "endpoint" I0915 11:40:39.112585 1 endpoint.go:68] ccResolverWrapper: sending new addresses to cc: [{https://127.0.0.1:2379 0 }] I0915 11:40:39.119883 1 client.go:357] parsed scheme: "endpoint" I0915 11:40:39.120012 1 endpoint.go:68] ccResolverWrapper: sending new addresses to cc: [{https://127.0.0.1:2379 0 }] W0915 11:40:39.210148 1 genericapiserver.go:404] Skipping API batch/v2alpha1 because it has no resources. W0915 11:40:39.222540 1 genericapiserver.go:404] Skipping API node.k8s.io/v1alpha1 because it has no resources. W0915 11:40:39.234616 1 genericapiserver.go:404] Skipping API rbac.authorization.k8s.io/v1alpha1 because it has no resources. W0915 11:40:39.236801 1 genericapiserver.go:404] Skipping API scheduling.k8s.io/v1alpha1 because it has no resources. W0915 11:40:39.244106 1 genericapiserver.go:404] Skipping API storage.k8s.io/v1alpha1 because it has no resources. W0915 11:40:39.256537 1 genericapiserver.go:404] Skipping API apps/v1beta2 because it has no resources. W0915 11:40:39.256559 1 genericapiserver.go:404] Skipping API apps/v1beta1 because it has no resources. I0915 11:40:39.263135 1 plugins.go:158] Loaded 11 mutating admission controller(s) successfully in the following order: NamespaceLifecycle,LimitRanger,ServiceAccount,NodeRestriction,TaintNodesByCondition,Priority,DefaultTolerationSeconds,DefaultStorageClass,StorageObjectInUseProtection,MutatingAdmissionWebhook,RuntimeClass. I0915 11:40:39.263142 1 plugins.go:161] Loaded 7 validating admission controller(s) successfully in the following order: LimitRanger,ServiceAccount,Priority,PersistentVolumeClaimResize,ValidatingAdmissionWebhook,RuntimeClass,ResourceQuota. I0915 11:40:39.264459 1 client.go:357] parsed scheme: "endpoint" I0915 11:40:39.264483 1 endpoint.go:68] ccResolverWrapper: sending new addresses to cc: [{https://127.0.0.1:2379 0 }] I0915 11:40:39.270344 1 client.go:357] parsed scheme: "endpoint" I0915 11:40:39.270410 1 endpoint.go:68] ccResolverWrapper: sending new addresses to cc: [{https://127.0.0.1:2379 0 }] I0915 11:40:40.798141 1 secure_serving.go:123] Serving securely on [::]:8443 I0915 11:40:40.799822 1 crd_finalizer.go:274] Starting CRDFinalizer I0915 11:40:40.799961 1 autoregister_controller.go:140] Starting autoregister controller I0915 11:40:40.799976 1 cache.go:32] Waiting for caches to sync for autoregister controller I0915 11:40:40.799989 1 controller.go:81] Starting OpenAPI AggregationController I0915 11:40:40.803864 1 controller.go:85] Starting OpenAPI controller I0915 11:40:40.803950 1 customresource_discovery_controller.go:208] Starting DiscoveryController I0915 11:40:40.803963 1 naming_controller.go:288] Starting NamingConditionController I0915 11:40:40.803971 1 establishing_controller.go:73] Starting EstablishingController I0915 11:40:40.803978 1 nonstructuralschema_controller.go:191] Starting NonStructuralSchemaConditionController I0915 11:40:40.803986 1 apiapproval_controller.go:185] Starting KubernetesAPIApprovalPolicyConformantConditionController I0915 11:40:40.803995 1 crdregistration_controller.go:111] Starting crd-autoregister controller I0915 11:40:40.803999 1 shared_informer.go:197] Waiting for caches to sync for crd-autoregister I0915 11:40:40.805544 1 apiservice_controller.go:94] Starting APIServiceRegistrationController I0915 11:40:40.805629 1 cache.go:32] Waiting for caches to sync for APIServiceRegistrationController controller I0915 11:40:40.805731 1 available_controller.go:383] Starting AvailableConditionController I0915 11:40:40.805767 1 cache.go:32] Waiting for caches to sync for AvailableConditionController controller E0915 11:40:40.867110 1 controller.go:154] Unable to remove old endpoints from kubernetes service: StorageError: key not found, Code: 1, Key: /registry/masterleases/192.168.99.103, ResourceVersion: 0, AdditionalErrorMsg: I0915 11:40:40.900041 1 cache.go:39] Caches are synced for autoregister controller I0915 11:40:40.905454 1 shared_informer.go:204] Caches are synced for crd-autoregister I0915 11:40:40.905874 1 cache.go:39] Caches are synced for AvailableConditionController controller I0915 11:40:40.906006 1 cache.go:39] Caches are synced for APIServiceRegistrationController controller I0915 11:40:41.798428 1 controller.go:107] OpenAPI AggregationController: Processing item I0915 11:40:41.798687 1 controller.go:130] OpenAPI AggregationController: action for item : Nothing (removed from the queue). I0915 11:40:41.798960 1 controller.go:130] OpenAPI AggregationController: action for item k8s_internal_local_delegation_chain_0000000000: Nothing (removed from the queue). I0915 11:40:41.804000 1 storage_scheduling.go:139] created PriorityClass system-node-critical with value 2000001000 I0915 11:40:41.809580 1 storage_scheduling.go:139] created PriorityClass system-cluster-critical with value 2000000000 I0915 11:40:41.809633 1 storage_scheduling.go:148] all system priority classes are created successfully or already exist. I0915 11:40:42.048746 1 controller.go:606] quota admission added evaluator for: roles.rbac.authorization.k8s.io I0915 11:40:42.070619 1 controller.go:606] quota admission added evaluator for: rolebindings.rbac.authorization.k8s.io W0915 11:40:42.136410 1 lease.go:222] Resetting endpoints for master service "kubernetes" to [192.168.99.103] I0915 11:40:42.136883 1 controller.go:606] quota admission added evaluator for: endpoints I0915 11:40:43.095961 1 controller.go:606] quota admission added evaluator for: serviceaccounts I0915 11:40:43.660608 1 controller.go:606] quota admission added evaluator for: deployments.apps I0915 11:40:44.005661 1 controller.go:606] quota admission added evaluator for: daemonsets.apps I0915 11:40:44.161416 1 controller.go:606] quota admission added evaluator for: leases.coordination.k8s.io I0915 11:40:49.641327 1 controller.go:606] quota admission added evaluator for: replicasets.apps I0915 11:40:49.654949 1 controller.go:606] quota admission added evaluator for: controllerrevisions.apps ==> kube-controller-manager [b0131d3d02f7] <== I0915 11:40:48.635097 1 controllermanager.go:534] Started "replicationcontroller" I0915 11:40:48.635340 1 replica_set.go:182] Starting replicationcontroller controller I0915 11:40:48.635450 1 shared_informer.go:197] Waiting for caches to sync for ReplicationController I0915 11:40:48.885551 1 controllermanager.go:534] Started "job" I0915 11:40:48.885886 1 job_controller.go:143] Starting job controller I0915 11:40:48.886007 1 shared_informer.go:197] Waiting for caches to sync for job I0915 11:40:49.036313 1 controllermanager.go:534] Started "csrsigning" W0915 11:40:49.036425 1 controllermanager.go:513] "endpointslice" is disabled I0915 11:40:49.036352 1 certificate_controller.go:113] Starting certificate controller I0915 11:40:49.036545 1 shared_informer.go:197] Waiting for caches to sync for certificate I0915 11:40:49.293561 1 controllermanager.go:534] Started "namespace" I0915 11:40:49.293801 1 namespace_controller.go:186] Starting namespace controller I0915 11:40:49.293980 1 shared_informer.go:197] Waiting for caches to sync for namespace I0915 11:40:49.537018 1 controllermanager.go:534] Started "daemonset" I0915 11:40:49.537150 1 core.go:211] Will not configure cloud provider routes for allocate-node-cidrs: false, configure-cloud-routes: true. W0915 11:40:49.537348 1 controllermanager.go:526] Skipping "route" I0915 11:40:49.537783 1 shared_informer.go:197] Waiting for caches to sync for resource quota I0915 11:40:49.537948 1 daemon_controller.go:267] Starting daemon sets controller I0915 11:40:49.538089 1 shared_informer.go:197] Waiting for caches to sync for daemon sets I0915 11:40:49.548094 1 shared_informer.go:197] Waiting for caches to sync for garbage collector W0915 11:40:49.559832 1 actual_state_of_world.go:506] Failed to update statusUpdateNeeded field in actual state of world: Failed to set statusUpdateNeeded to needed true, because nodeName="my-cluster" does not exist I0915 11:40:49.561988 1 shared_informer.go:204] Caches are synced for HPA I0915 11:40:49.585499 1 shared_informer.go:204] Caches are synced for certificate I0915 11:40:49.585599 1 shared_informer.go:204] Caches are synced for GC I0915 11:40:49.585913 1 shared_informer.go:204] Caches are synced for PVC protection I0915 11:40:49.585924 1 shared_informer.go:204] Caches are synced for ClusterRoleAggregator I0915 11:40:49.585993 1 shared_informer.go:204] Caches are synced for service account I0915 11:40:49.586000 1 shared_informer.go:204] Caches are synced for taint I0915 11:40:49.590512 1 node_lifecycle_controller.go:1208] Initializing eviction metric for zone: W0915 11:40:49.590682 1 node_lifecycle_controller.go:903] Missing timestamp for Node my-cluster. Assuming now as a timestamp. I0915 11:40:49.590805 1 node_lifecycle_controller.go:1108] Controller detected that zone is now in state Normal. I0915 11:40:49.590982 1 event.go:274] Event(v1.ObjectReference{Kind:"Node", Namespace:"", Name:"my-cluster", UID:"3941175c-a965-4de3-bcd4-59f7eb25742d", APIVersion:"", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'RegisteredNode' Node my-cluster event: Registered Node my-cluster in Controller I0915 11:40:49.591416 1 taint_manager.go:186] Starting NoExecuteTaintManager I0915 11:40:49.599598 1 shared_informer.go:204] Caches are synced for namespace I0915 11:40:49.604049 1 shared_informer.go:204] Caches are synced for ReplicaSet I0915 11:40:49.635416 1 shared_informer.go:204] Caches are synced for deployment I0915 11:40:49.635652 1 shared_informer.go:204] Caches are synced for ReplicationController I0915 11:40:49.635748 1 shared_informer.go:204] Caches are synced for stateful set I0915 11:40:49.636208 1 shared_informer.go:204] Caches are synced for TTL I0915 11:40:49.636903 1 shared_informer.go:204] Caches are synced for certificate I0915 11:40:49.647875 1 shared_informer.go:204] Caches are synced for bootstrap_signer I0915 11:40:49.648227 1 shared_informer.go:204] Caches are synced for daemon sets I0915 11:40:49.653588 1 log.go:172] [INFO] signed certificate with serial number 95765727313505264710504695642140317020828184822 E0915 11:40:49.670151 1 clusterroleaggregation_controller.go:180] admin failed with : Operation cannot be fulfilled on clusterroles.rbac.authorization.k8s.io "admin": the object has been modified; please apply your changes to the latest version and try again I0915 11:40:49.670663 1 event.go:274] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"kube-system", Name:"coredns", UID:"7895388d-56cf-4a72-88de-93fe6cdea487", APIVersion:"apps/v1", ResourceVersion:"220", FieldPath:""}): type: 'Normal' reason: 'ScalingReplicaSet' Scaled up replica set coredns-5644d7b6d9 to 1 I0915 11:40:49.683440 1 event.go:274] Event(v1.ObjectReference{Kind:"DaemonSet", Namespace:"kube-system", Name:"kube-proxy", UID:"b96bd66c-778d-4067-8212-488276ddb5b0", APIVersion:"apps/v1", ResourceVersion:"209", FieldPath:""}): type: 'Normal' reason: 'SuccessfulCreate' Created pod: kube-proxy-jbmx2 I0915 11:40:49.697247 1 event.go:274] Event(v1.ObjectReference{Kind:"ReplicaSet", Namespace:"kube-system", Name:"coredns-5644d7b6d9", UID:"9458c351-348a-4f7d-b6e2-5f8672692717", APIVersion:"apps/v1", ResourceVersion:"316", FieldPath:""}): type: 'Normal' reason: 'SuccessfulCreate' Created pod: coredns-5644d7b6d9-882h8 I0915 11:40:49.886317 1 shared_informer.go:204] Caches are synced for job I0915 11:40:49.937010 1 shared_informer.go:204] Caches are synced for expand I0915 11:40:49.937103 1 shared_informer.go:204] Caches are synced for PV protection I0915 11:40:49.937314 1 shared_informer.go:204] Caches are synced for attach detach I0915 11:40:49.987476 1 shared_informer.go:204] Caches are synced for persistent volume I0915 11:40:50.035717 1 shared_informer.go:204] Caches are synced for endpoint I0915 11:40:50.038426 1 shared_informer.go:204] Caches are synced for resource quota I0915 11:40:50.038600 1 shared_informer.go:204] Caches are synced for resource quota I0915 11:40:50.057516 1 shared_informer.go:204] Caches are synced for disruption I0915 11:40:50.057528 1 disruption.go:338] Sending events to api server. I0915 11:40:50.140259 1 shared_informer.go:204] Caches are synced for garbage collector I0915 11:40:50.140271 1 garbagecollector.go:139] Garbage collector: all resource monitors have synced. Proceeding to collect garbage I0915 11:40:50.148408 1 shared_informer.go:204] Caches are synced for garbage collector ==> kube-proxy [ddc0c0687de9] <== W0915 11:40:50.557718 1 server_others.go:330] Flag proxy-mode="" unknown, assuming iptables proxy I0915 11:40:50.562834 1 node.go:135] Successfully retrieved node IP: 192.168.99.103 I0915 11:40:50.562856 1 server_others.go:150] Using iptables Proxier. W0915 11:40:50.562905 1 proxier.go:282] clusterCIDR not specified, unable to distinguish between internal and external traffic I0915 11:40:50.563071 1 server.go:529] Version: v1.16.6 I0915 11:40:50.563326 1 conntrack.go:100] Set sysctl 'net/netfilter/nf_conntrack_max' to 131072 I0915 11:40:50.563339 1 conntrack.go:52] Setting nf_conntrack_max to 131072 I0915 11:40:50.563367 1 conntrack.go:100] Set sysctl 'net/netfilter/nf_conntrack_tcp_timeout_established' to 86400 I0915 11:40:50.563388 1 conntrack.go:100] Set sysctl 'net/netfilter/nf_conntrack_tcp_timeout_close_wait' to 3600 I0915 11:40:50.563757 1 config.go:313] Starting service config controller I0915 11:40:50.563772 1 shared_informer.go:197] Waiting for caches to sync for service config I0915 11:40:50.564087 1 config.go:131] Starting endpoints config controller I0915 11:40:50.564104 1 shared_informer.go:197] Waiting for caches to sync for endpoints config I0915 11:40:50.664247 1 shared_informer.go:204] Caches are synced for endpoints config I0915 11:40:50.664391 1 shared_informer.go:204] Caches are synced for service config ==> kube-scheduler [c2281b58d19e] <== I0915 11:40:36.911935 1 serving.go:319] Generated self-signed cert in-memory W0915 11:40:40.876122 1 authentication.go:262] Unable to get configmap/extension-apiserver-authentication in kube-system. Usually fixed by 'kubectl create rolebinding -n kube-system ROLEBINDING_NAME --role=extension-apiserver-authentication-reader --serviceaccount=YOUR_NS:YOUR_SA' W0915 11:40:40.876145 1 authentication.go:199] Error looking up in-cluster authentication configuration: configmaps "extension-apiserver-authentication" is forbidden: User "system:kube-scheduler" cannot get resource "configmaps" in API group "" in the namespace "kube-system" W0915 11:40:40.876152 1 authentication.go:200] Continuing without authentication configuration. This may treat all requests as anonymous. W0915 11:40:40.876156 1 authentication.go:201] To require authentication configuration lookup to succeed, set --authentication-tolerate-lookup-failure=false I0915 11:40:40.880587 1 server.go:148] Version: v1.16.6 I0915 11:40:40.884086 1 defaults.go:91] TaintNodesByCondition is enabled, PodToleratesNodeTaints predicate is mandatory W0915 11:40:40.895347 1 authorization.go:47] Authorization is disabled W0915 11:40:40.895368 1 authentication.go:79] Authentication is disabled I0915 11:40:40.895376 1 deprecated_insecure_serving.go:51] Serving healthz insecurely on [::]:10251 I0915 11:40:40.896215 1 secure_serving.go:123] Serving securely on 127.0.0.1:10259 E0915 11:40:40.898790 1 reflector.go:123] k8s.io/kubernetes/cmd/kube-scheduler/app/server.go:250: Failed to list *v1.Pod: pods is forbidden: User "system:kube-scheduler" cannot list resource "pods" in API group "" at the cluster scope E0915 11:40:40.898847 1 reflector.go:123] k8s.io/client-go/informers/factory.go:134: Failed to list *v1.StorageClass: storageclasses.storage.k8s.io is forbidden: User "system:kube-scheduler" cannot list resource "storageclasses" in API group "storage.k8s.io" at the cluster scope E0915 11:40:40.899150 1 reflector.go:123] k8s.io/client-go/informers/factory.go:134: Failed to list *v1.Service: services is forbidden: User "system:kube-scheduler" cannot list resource "services" in API group "" at the cluster scope E0915 11:40:40.899174 1 reflector.go:123] k8s.io/client-go/informers/factory.go:134: Failed to list *v1beta1.CSINode: csinodes.storage.k8s.io is forbidden: User "system:kube-scheduler" cannot list resource "csinodes" in API group "storage.k8s.io" at the cluster scope E0915 11:40:40.899519 1 reflector.go:123] k8s.io/client-go/informers/factory.go:134: Failed to list *v1.ReplicaSet: replicasets.apps is forbidden: User "system:kube-scheduler" cannot list resource "replicasets" in API group "apps" at the cluster scope E0915 11:40:40.899563 1 reflector.go:123] k8s.io/client-go/informers/factory.go:134: Failed to list *v1beta1.PodDisruptionBudget: poddisruptionbudgets.policy is forbidden: User "system:kube-scheduler" cannot list resource "poddisruptionbudgets" in API group "policy" at the cluster scope E0915 11:40:40.900043 1 reflector.go:123] k8s.io/client-go/informers/factory.go:134: Failed to list *v1.ReplicationController: replicationcontrollers is forbidden: User "system:kube-scheduler" cannot list resource "replicationcontrollers" in API group "" at the cluster scope E0915 11:40:40.900111 1 reflector.go:123] k8s.io/client-go/informers/factory.go:134: Failed to list *v1.PersistentVolume: persistentvolumes is forbidden: User "system:kube-scheduler" cannot list resource "persistentvolumes" in API group "" at the cluster scope E0915 11:40:40.900371 1 reflector.go:123] k8s.io/client-go/informers/factory.go:134: Failed to list *v1.PersistentVolumeClaim: persistentvolumeclaims is forbidden: User "system:kube-scheduler" cannot list resource "persistentvolumeclaims" in API group "" at the cluster scope E0915 11:40:40.901670 1 reflector.go:123] k8s.io/client-go/informers/factory.go:134: Failed to list *v1.StatefulSet: statefulsets.apps is forbidden: User "system:kube-scheduler" cannot list resource "statefulsets" in API group "apps" at the cluster scope E0915 11:40:40.901714 1 reflector.go:123] k8s.io/client-go/informers/factory.go:134: Failed to list *v1.Node: nodes is forbidden: User "system:kube-scheduler" cannot list resource "nodes" in API group "" at the cluster scope E0915 11:40:41.900637 1 reflector.go:123] k8s.io/kubernetes/cmd/kube-scheduler/app/server.go:250: Failed to list *v1.Pod: pods is forbidden: User "system:kube-scheduler" cannot list resource "pods" in API group "" at the cluster scope E0915 11:40:41.902873 1 reflector.go:123] k8s.io/client-go/informers/factory.go:134: Failed to list *v1.StorageClass: storageclasses.storage.k8s.io is forbidden: User "system:kube-scheduler" cannot list resource "storageclasses" in API group "storage.k8s.io" at the cluster scope E0915 11:40:41.904324 1 reflector.go:123] k8s.io/client-go/informers/factory.go:134: Failed to list *v1.Service: services is forbidden: User "system:kube-scheduler" cannot list resource "services" in API group "" at the cluster scope E0915 11:40:41.905519 1 reflector.go:123] k8s.io/client-go/informers/factory.go:134: Failed to list *v1beta1.CSINode: csinodes.storage.k8s.io is forbidden: User "system:kube-scheduler" cannot list resource "csinodes" in API group "storage.k8s.io" at the cluster scope E0915 11:40:41.906559 1 reflector.go:123] k8s.io/client-go/informers/factory.go:134: Failed to list *v1.ReplicaSet: replicasets.apps is forbidden: User "system:kube-scheduler" cannot list resource "replicasets" in API group "apps" at the cluster scope E0915 11:40:41.907748 1 reflector.go:123] k8s.io/client-go/informers/factory.go:134: Failed to list *v1beta1.PodDisruptionBudget: poddisruptionbudgets.policy is forbidden: User "system:kube-scheduler" cannot list resource "poddisruptionbudgets" in API group "policy" at the cluster scope E0915 11:40:41.909846 1 reflector.go:123] k8s.io/client-go/informers/factory.go:134: Failed to list *v1.ReplicationController: replicationcontrollers is forbidden: User "system:kube-scheduler" cannot list resource "replicationcontrollers" in API group "" at the cluster scope E0915 11:40:41.911063 1 reflector.go:123] k8s.io/client-go/informers/factory.go:134: Failed to list *v1.PersistentVolume: persistentvolumes is forbidden: User "system:kube-scheduler" cannot list resource "persistentvolumes" in API group "" at the cluster scope E0915 11:40:41.913028 1 reflector.go:123] k8s.io/client-go/informers/factory.go:134: Failed to list *v1.PersistentVolumeClaim: persistentvolumeclaims is forbidden: User "system:kube-scheduler" cannot list resource "persistentvolumeclaims" in API group "" at the cluster scope E0915 11:40:41.914368 1 reflector.go:123] k8s.io/client-go/informers/factory.go:134: Failed to list *v1.StatefulSet: statefulsets.apps is forbidden: User "system:kube-scheduler" cannot list resource "statefulsets" in API group "apps" at the cluster scope E0915 11:40:41.915403 1 reflector.go:123] k8s.io/client-go/informers/factory.go:134: Failed to list *v1.Node: nodes is forbidden: User "system:kube-scheduler" cannot list resource "nodes" in API group "" at the cluster scope E0915 11:40:44.529742 1 factory.go:585] pod is already present in the activeQ ==> kubelet <== -- Logs begin at Tue 2020-09-15 11:39:53 UTC, end at Tue 2020-09-15 11:44:06 UTC. -- Sep 15 11:40:38 my-cluster kubelet[3257]: E0915 11:40:38.255152 3257 kubelet.go:2267] node "my-cluster" not found Sep 15 11:40:38 my-cluster kubelet[3257]: E0915 11:40:38.355993 3257 kubelet.go:2267] node "my-cluster" not found Sep 15 11:40:38 my-cluster kubelet[3257]: E0915 11:40:38.457383 3257 kubelet.go:2267] node "my-cluster" not found Sep 15 11:40:38 my-cluster kubelet[3257]: E0915 11:40:38.558706 3257 kubelet.go:2267] node "my-cluster" not found Sep 15 11:40:38 my-cluster kubelet[3257]: E0915 11:40:38.658910 3257 kubelet.go:2267] node "my-cluster" not found Sep 15 11:40:38 my-cluster kubelet[3257]: E0915 11:40:38.759355 3257 kubelet.go:2267] node "my-cluster" not found Sep 15 11:40:38 my-cluster kubelet[3257]: E0915 11:40:38.859555 3257 kubelet.go:2267] node "my-cluster" not found Sep 15 11:40:38 my-cluster kubelet[3257]: E0915 11:40:38.959844 3257 kubelet.go:2267] node "my-cluster" not found Sep 15 11:40:39 my-cluster kubelet[3257]: E0915 11:40:39.060464 3257 kubelet.go:2267] node "my-cluster" not found Sep 15 11:40:39 my-cluster kubelet[3257]: E0915 11:40:39.161469 3257 kubelet.go:2267] node "my-cluster" not found Sep 15 11:40:39 my-cluster kubelet[3257]: E0915 11:40:39.261691 3257 kubelet.go:2267] node "my-cluster" not found Sep 15 11:40:39 my-cluster kubelet[3257]: E0915 11:40:39.362319 3257 kubelet.go:2267] node "my-cluster" not found Sep 15 11:40:39 my-cluster kubelet[3257]: E0915 11:40:39.462664 3257 kubelet.go:2267] node "my-cluster" not found Sep 15 11:40:39 my-cluster kubelet[3257]: E0915 11:40:39.563374 3257 kubelet.go:2267] node "my-cluster" not found Sep 15 11:40:39 my-cluster kubelet[3257]: E0915 11:40:39.664246 3257 kubelet.go:2267] node "my-cluster" not found Sep 15 11:40:39 my-cluster kubelet[3257]: E0915 11:40:39.765410 3257 kubelet.go:2267] node "my-cluster" not found Sep 15 11:40:39 my-cluster kubelet[3257]: E0915 11:40:39.865786 3257 kubelet.go:2267] node "my-cluster" not found Sep 15 11:40:39 my-cluster kubelet[3257]: E0915 11:40:39.966065 3257 kubelet.go:2267] node "my-cluster" not found Sep 15 11:40:40 my-cluster kubelet[3257]: E0915 11:40:40.066412 3257 kubelet.go:2267] node "my-cluster" not found Sep 15 11:40:40 my-cluster kubelet[3257]: E0915 11:40:40.167589 3257 kubelet.go:2267] node "my-cluster" not found Sep 15 11:40:40 my-cluster kubelet[3257]: E0915 11:40:40.267895 3257 kubelet.go:2267] node "my-cluster" not found Sep 15 11:40:40 my-cluster kubelet[3257]: E0915 11:40:40.368527 3257 kubelet.go:2267] node "my-cluster" not found Sep 15 11:40:40 my-cluster kubelet[3257]: E0915 11:40:40.468858 3257 kubelet.go:2267] node "my-cluster" not found Sep 15 11:40:40 my-cluster kubelet[3257]: E0915 11:40:40.569158 3257 kubelet.go:2267] node "my-cluster" not found Sep 15 11:40:40 my-cluster kubelet[3257]: E0915 11:40:40.669639 3257 kubelet.go:2267] node "my-cluster" not found Sep 15 11:40:40 my-cluster kubelet[3257]: E0915 11:40:40.769896 3257 kubelet.go:2267] node "my-cluster" not found Sep 15 11:40:40 my-cluster kubelet[3257]: E0915 11:40:40.870412 3257 kubelet.go:2267] node "my-cluster" not found Sep 15 11:40:40 my-cluster kubelet[3257]: E0915 11:40:40.888291 3257 controller.go:220] failed to get node "my-cluster" when trying to set owner ref to the node lease: nodes "my-cluster" not found Sep 15 11:40:40 my-cluster kubelet[3257]: E0915 11:40:40.956784 3257 controller.go:135] failed to ensure node lease exists, will retry in 3.2s, error: namespaces "kube-node-lease" not found Sep 15 11:40:40 my-cluster kubelet[3257]: I0915 11:40:40.963491 3257 kubelet_node_status.go:75] Successfully registered node my-cluster Sep 15 11:40:40 my-cluster kubelet[3257]: I0915 11:40:40.972285 3257 reconciler.go:154] Reconciler: start to sync state Sep 15 11:40:41 my-cluster kubelet[3257]: E0915 11:40:41.012512 3257 event.go:256] Server rejected event '&v1.Event{TypeMeta:v1.TypeMeta{Kind:"", APIVersion:""}, ObjectMeta:v1.ObjectMeta{Name:"my-cluster.1634f22b3c511483", GenerateName:"", Namespace:"default", SelfLink:"", UID:"", ResourceVersion:"", Generation:0, CreationTimestamp:v1.Time{Time:time.Time{wall:0x0, ext:0, loc:(*time.Location)(nil)}}, DeletionTimestamp:(*v1.Time)(nil), DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string(nil), Annotations:map[string]string(nil), OwnerReferences:[]v1.OwnerReference(nil), Finalizers:[]string(nil), ClusterName:"", ManagedFields:[]v1.ManagedFieldsEntry(nil)}, InvolvedObject:v1.ObjectReference{Kind:"Node", Namespace:"", Name:"my-cluster", UID:"my-cluster", APIVersion:"", ResourceVersion:"", FieldPath:""}, Reason:"Starting", Message:"Starting kubelet.", Source:v1.EventSource{Component:"kubelet", Host:"my-cluster"}, FirstTimestamp:v1.Time{Time:time.Time{wall:0xbfd047ecc7ead683, ext:5228955607, loc:(*time.Location)(0x6e0d080)}}, LastTimestamp:v1.Time{Time:time.Time{wall:0xbfd047ecc7ead683, ext:5228955607, loc:(*time.Location)(0x6e0d080)}}, Count:1, Type:"Normal", EventTime:v1.MicroTime{Time:time.Time{wall:0x0, ext:0, loc:(*time.Location)(nil)}}, Series:(*v1.EventSeries)(nil), Action:"", Related:(*v1.ObjectReference)(nil), ReportingController:"", ReportingInstance:""}': 'namespaces "default" not found' (will not retry!) Sep 15 11:40:41 my-cluster kubelet[3257]: E0915 11:40:41.065637 3257 event.go:256] Server rejected event '&v1.Event{TypeMeta:v1.TypeMeta{Kind:"", APIVersion:""}, ObjectMeta:v1.ObjectMeta{Name:"my-cluster.1634f22b41c58cd9", GenerateName:"", Namespace:"default", SelfLink:"", UID:"", ResourceVersion:"", Generation:0, CreationTimestamp:v1.Time{Time:time.Time{wall:0x0, ext:0, loc:(*time.Location)(nil)}}, DeletionTimestamp:(*v1.Time)(nil), DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string(nil), Annotations:map[string]string(nil), OwnerReferences:[]v1.OwnerReference(nil), Finalizers:[]string(nil), ClusterName:"", ManagedFields:[]v1.ManagedFieldsEntry(nil)}, InvolvedObject:v1.ObjectReference{Kind:"Node", Namespace:"", Name:"my-cluster", UID:"my-cluster", APIVersion:"", ResourceVersion:"", FieldPath:""}, Reason:"NodeHasSufficientPID", Message:"Node my-cluster status is now: NodeHasSufficientPID", Source:v1.EventSource{Component:"kubelet", Host:"my-cluster"}, FirstTimestamp:v1.Time{Time:time.Time{wall:0xbfd047eccd5f4ed9, ext:5320474663, loc:(*time.Location)(0x6e0d080)}}, LastTimestamp:v1.Time{Time:time.Time{wall:0xbfd047eccd5f4ed9, ext:5320474663, loc:(*time.Location)(0x6e0d080)}}, Count:1, Type:"Normal", EventTime:v1.MicroTime{Time:time.Time{wall:0x0, ext:0, loc:(*time.Location)(nil)}}, Series:(*v1.EventSeries)(nil), Action:"", Related:(*v1.ObjectReference)(nil), ReportingController:"", ReportingInstance:""}': 'namespaces "default" not found' (will not retry!) Sep 15 11:40:41 my-cluster kubelet[3257]: E0915 11:40:41.118618 3257 event.go:256] Server rejected event '&v1.Event{TypeMeta:v1.TypeMeta{Kind:"", APIVersion:""}, ObjectMeta:v1.ObjectMeta{Name:"my-cluster.1634f22b41c56d25", GenerateName:"", Namespace:"default", SelfLink:"", UID:"", ResourceVersion:"", Generation:0, CreationTimestamp:v1.Time{Time:time.Time{wall:0x0, ext:0, loc:(*time.Location)(nil)}}, DeletionTimestamp:(*v1.Time)(nil), DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string(nil), Annotations:map[string]string(nil), OwnerReferences:[]v1.OwnerReference(nil), Finalizers:[]string(nil), ClusterName:"", ManagedFields:[]v1.ManagedFieldsEntry(nil)}, InvolvedObject:v1.ObjectReference{Kind:"Node", Namespace:"", Name:"my-cluster", UID:"my-cluster", APIVersion:"", ResourceVersion:"", FieldPath:""}, Reason:"NodeHasSufficientMemory", Message:"Node my-cluster status is now: NodeHasSufficientMemory", Source:v1.EventSource{Component:"kubelet", Host:"my-cluster"}, FirstTimestamp:v1.Time{Time:time.Time{wall:0xbfd047eccd5f2f25, ext:5320466548, loc:(*time.Location)(0x6e0d080)}}, LastTimestamp:v1.Time{Time:time.Time{wall:0xbfd047eccd5f2f25, ext:5320466548, loc:(*time.Location)(0x6e0d080)}}, Count:1, Type:"Normal", EventTime:v1.MicroTime{Time:time.Time{wall:0x0, ext:0, loc:(*time.Location)(nil)}}, Series:(*v1.EventSeries)(nil), Action:"", Related:(*v1.ObjectReference)(nil), ReportingController:"", ReportingInstance:""}': 'namespaces "default" not found' (will not retry!) Sep 15 11:40:41 my-cluster kubelet[3257]: E0915 11:40:41.171693 3257 event.go:256] Server rejected event '&v1.Event{TypeMeta:v1.TypeMeta{Kind:"", APIVersion:""}, ObjectMeta:v1.ObjectMeta{Name:"my-cluster.1634f22b41c5846b", GenerateName:"", Namespace:"default", SelfLink:"", UID:"", ResourceVersion:"", Generation:0, CreationTimestamp:v1.Time{Time:time.Time{wall:0x0, ext:0, loc:(*time.Location)(nil)}}, DeletionTimestamp:(*v1.Time)(nil), DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string(nil), Annotations:map[string]string(nil), OwnerReferences:[]v1.OwnerReference(nil), Finalizers:[]string(nil), ClusterName:"", ManagedFields:[]v1.ManagedFieldsEntry(nil)}, InvolvedObject:v1.ObjectReference{Kind:"Node", Namespace:"", Name:"my-cluster", UID:"my-cluster", APIVersion:"", ResourceVersion:"", FieldPath:""}, Reason:"NodeHasNoDiskPressure", Message:"Node my-cluster status is now: NodeHasNoDiskPressure", Source:v1.EventSource{Component:"kubelet", Host:"my-cluster"}, FirstTimestamp:v1.Time{Time:time.Time{wall:0xbfd047eccd5f466b, ext:5320472505, loc:(*time.Location)(0x6e0d080)}}, LastTimestamp:v1.Time{Time:time.Time{wall:0xbfd047eccd5f466b, ext:5320472505, loc:(*time.Location)(0x6e0d080)}}, Count:1, Type:"Normal", EventTime:v1.MicroTime{Time:time.Time{wall:0x0, ext:0, loc:(*time.Location)(nil)}}, Series:(*v1.EventSeries)(nil), Action:"", Related:(*v1.ObjectReference)(nil), ReportingController:"", ReportingInstance:""}': 'namespaces "default" not found' (will not retry!) Sep 15 11:40:41 my-cluster kubelet[3257]: E0915 11:40:41.224530 3257 event.go:256] Server rejected event '&v1.Event{TypeMeta:v1.TypeMeta{Kind:"", APIVersion:""}, ObjectMeta:v1.ObjectMeta{Name:"my-cluster.1634f22b4374fac4", GenerateName:"", Namespace:"default", SelfLink:"", UID:"", ResourceVersion:"", Generation:0, CreationTimestamp:v1.Time{Time:time.Time{wall:0x0, ext:0, loc:(*time.Location)(nil)}}, DeletionTimestamp:(*v1.Time)(nil), DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string(nil), Annotations:map[string]string(nil), OwnerReferences:[]v1.OwnerReference(nil), Finalizers:[]string(nil), ClusterName:"", ManagedFields:[]v1.ManagedFieldsEntry(nil)}, InvolvedObject:v1.ObjectReference{Kind:"Node", Namespace:"", Name:"my-cluster", UID:"my-cluster", APIVersion:"", ResourceVersion:"", FieldPath:""}, Reason:"NodeAllocatableEnforced", Message:"Updated Node Allocatable limit across pods", Source:v1.EventSource{Component:"kubelet", Host:"my-cluster"}, FirstTimestamp:v1.Time{Time:time.Time{wall:0xbfd047eccf0ebcc4, ext:5348748820, loc:(*time.Location)(0x6e0d080)}}, LastTimestamp:v1.Time{Time:time.Time{wall:0xbfd047eccf0ebcc4, ext:5348748820, loc:(*time.Location)(0x6e0d080)}}, Count:1, Type:"Normal", EventTime:v1.MicroTime{Time:time.Time{wall:0x0, ext:0, loc:(*time.Location)(nil)}}, Series:(*v1.EventSeries)(nil), Action:"", Related:(*v1.ObjectReference)(nil), ReportingController:"", ReportingInstance:""}': 'namespaces "default" not found' (will not retry!) Sep 15 11:40:41 my-cluster kubelet[3257]: E0915 11:40:41.280080 3257 event.go:256] Server rejected event '&v1.Event{TypeMeta:v1.TypeMeta{Kind:"", APIVersion:""}, ObjectMeta:v1.ObjectMeta{Name:"my-cluster.1634f22b41c56d25", GenerateName:"", Namespace:"default", SelfLink:"", UID:"", ResourceVersion:"", Generation:0, CreationTimestamp:v1.Time{Time:time.Time{wall:0x0, ext:0, loc:(*time.Location)(nil)}}, DeletionTimestamp:(*v1.Time)(nil), DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string(nil), Annotations:map[string]string(nil), OwnerReferences:[]v1.OwnerReference(nil), Finalizers:[]string(nil), ClusterName:"", ManagedFields:[]v1.ManagedFieldsEntry(nil)}, InvolvedObject:v1.ObjectReference{Kind:"Node", Namespace:"", Name:"my-cluster", UID:"my-cluster", APIVersion:"", ResourceVersion:"", FieldPath:""}, Reason:"NodeHasSufficientMemory", Message:"Node my-cluster status is now: NodeHasSufficientMemory", Source:v1.EventSource{Component:"kubelet", Host:"my-cluster"}, FirstTimestamp:v1.Time{Time:time.Time{wall:0xbfd047eccd5f2f25, ext:5320466548, loc:(*time.Location)(0x6e0d080)}}, LastTimestamp:v1.Time{Time:time.Time{wall:0xbfd047eccf5ecc60, ext:5353995699, loc:(*time.Location)(0x6e0d080)}}, Count:2, Type:"Normal", EventTime:v1.MicroTime{Time:time.Time{wall:0x0, ext:0, loc:(*time.Location)(nil)}}, Series:(*v1.EventSeries)(nil), Action:"", Related:(*v1.ObjectReference)(nil), ReportingController:"", ReportingInstance:""}': 'namespaces "default" not found' (will not retry!) Sep 15 11:40:41 my-cluster kubelet[3257]: E0915 11:40:41.334826 3257 event.go:256] Server rejected event '&v1.Event{TypeMeta:v1.TypeMeta{Kind:"", APIVersion:""}, ObjectMeta:v1.ObjectMeta{Name:"my-cluster.1634f22b41c5846b", GenerateName:"", Namespace:"default", SelfLink:"", UID:"", ResourceVersion:"", Generation:0, CreationTimestamp:v1.Time{Time:time.Time{wall:0x0, ext:0, loc:(*time.Location)(nil)}}, DeletionTimestamp:(*v1.Time)(nil), DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string(nil), Annotations:map[string]string(nil), OwnerReferences:[]v1.OwnerReference(nil), Finalizers:[]string(nil), ClusterName:"", ManagedFields:[]v1.ManagedFieldsEntry(nil)}, InvolvedObject:v1.ObjectReference{Kind:"Node", Namespace:"", Name:"my-cluster", UID:"my-cluster", APIVersion:"", ResourceVersion:"", FieldPath:""}, Reason:"NodeHasNoDiskPressure", Message:"Node my-cluster status is now: NodeHasNoDiskPressure", Source:v1.EventSource{Component:"kubelet", Host:"my-cluster"}, FirstTimestamp:v1.Time{Time:time.Time{wall:0xbfd047eccd5f466b, ext:5320472505, loc:(*time.Location)(0x6e0d080)}}, LastTimestamp:v1.Time{Time:time.Time{wall:0xbfd047eccf5ed712, ext:5353998436, loc:(*time.Location)(0x6e0d080)}}, Count:2, Type:"Normal", EventTime:v1.MicroTime{Time:time.Time{wall:0x0, ext:0, loc:(*time.Location)(nil)}}, Series:(*v1.EventSeries)(nil), Action:"", Related:(*v1.ObjectReference)(nil), ReportingController:"", ReportingInstance:""}': 'namespaces "default" not found' (will not retry!) Sep 15 11:40:41 my-cluster kubelet[3257]: E0915 11:40:41.390490 3257 event.go:256] Server rejected event '&v1.Event{TypeMeta:v1.TypeMeta{Kind:"", APIVersion:""}, ObjectMeta:v1.ObjectMeta{Name:"my-cluster.1634f22b41c58cd9", GenerateName:"", Namespace:"default", SelfLink:"", UID:"", ResourceVersion:"", Generation:0, CreationTimestamp:v1.Time{Time:time.Time{wall:0x0, ext:0, loc:(*time.Location)(nil)}}, DeletionTimestamp:(*v1.Time)(nil), DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string(nil), Annotations:map[string]string(nil), OwnerReferences:[]v1.OwnerReference(nil), Finalizers:[]string(nil), ClusterName:"", ManagedFields:[]v1.ManagedFieldsEntry(nil)}, InvolvedObject:v1.ObjectReference{Kind:"Node", Namespace:"", Name:"my-cluster", UID:"my-cluster", APIVersion:"", ResourceVersion:"", FieldPath:""}, Reason:"NodeHasSufficientPID", Message:"Node my-cluster status is now: NodeHasSufficientPID", Source:v1.EventSource{Component:"kubelet", Host:"my-cluster"}, FirstTimestamp:v1.Time{Time:time.Time{wall:0xbfd047eccd5f4ed9, ext:5320474663, loc:(*time.Location)(0x6e0d080)}}, LastTimestamp:v1.Time{Time:time.Time{wall:0xbfd047eccf5ee07a, ext:5354000839, loc:(*time.Location)(0x6e0d080)}}, Count:2, Type:"Normal", EventTime:v1.MicroTime{Time:time.Time{wall:0x0, ext:0, loc:(*time.Location)(nil)}}, Series:(*v1.EventSeries)(nil), Action:"", Related:(*v1.ObjectReference)(nil), ReportingController:"", ReportingInstance:""}': 'namespaces "default" not found' (will not retry!) Sep 15 11:40:41 my-cluster kubelet[3257]: E0915 11:40:41.466623 3257 event.go:256] Server rejected event '&v1.Event{TypeMeta:v1.TypeMeta{Kind:"", APIVersion:""}, ObjectMeta:v1.ObjectMeta{Name:"my-cluster.1634f22b41c56d25", GenerateName:"", Namespace:"default", SelfLink:"", UID:"", ResourceVersion:"", Generation:0, CreationTimestamp:v1.Time{Time:time.Time{wall:0x0, ext:0, loc:(*time.Location)(nil)}}, DeletionTimestamp:(*v1.Time)(nil), DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string(nil), Annotations:map[string]string(nil), OwnerReferences:[]v1.OwnerReference(nil), Finalizers:[]string(nil), ClusterName:"", ManagedFields:[]v1.ManagedFieldsEntry(nil)}, InvolvedObject:v1.ObjectReference{Kind:"Node", Namespace:"", Name:"my-cluster", UID:"my-cluster", APIVersion:"", ResourceVersion:"", FieldPath:""}, Reason:"NodeHasSufficientMemory", Message:"Node my-cluster status is now: NodeHasSufficientMemory", Source:v1.EventSource{Component:"kubelet", Host:"my-cluster"}, FirstTimestamp:v1.Time{Time:time.Time{wall:0xbfd047eccd5f2f25, ext:5320466548, loc:(*time.Location)(0x6e0d080)}}, LastTimestamp:v1.Time{Time:time.Time{wall:0xbfd047ecdb9d04c2, ext:5559399961, loc:(*time.Location)(0x6e0d080)}}, Count:3, Type:"Normal", EventTime:v1.MicroTime{Time:time.Time{wall:0x0, ext:0, loc:(*time.Location)(nil)}}, Series:(*v1.EventSeries)(nil), Action:"", Related:(*v1.ObjectReference)(nil), ReportingController:"", ReportingInstance:""}': 'namespaces "default" not found' (will not retry!) Sep 15 11:40:41 my-cluster kubelet[3257]: E0915 11:40:41.867293 3257 event.go:256] Server rejected event '&v1.Event{TypeMeta:v1.TypeMeta{Kind:"", APIVersion:""}, ObjectMeta:v1.ObjectMeta{Name:"my-cluster.1634f22b41c5846b", GenerateName:"", Namespace:"default", SelfLink:"", UID:"", ResourceVersion:"", Generation:0, CreationTimestamp:v1.Time{Time:time.Time{wall:0x0, ext:0, loc:(*time.Location)(nil)}}, DeletionTimestamp:(*v1.Time)(nil), DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string(nil), Annotations:map[string]string(nil), OwnerReferences:[]v1.OwnerReference(nil), Finalizers:[]string(nil), ClusterName:"", ManagedFields:[]v1.ManagedFieldsEntry(nil)}, InvolvedObject:v1.ObjectReference{Kind:"Node", Namespace:"", Name:"my-cluster", UID:"my-cluster", APIVersion:"", ResourceVersion:"", FieldPath:""}, Reason:"NodeHasNoDiskPressure", Message:"Node my-cluster status is now: NodeHasNoDiskPressure", Source:v1.EventSource{Component:"kubelet", Host:"my-cluster"}, FirstTimestamp:v1.Time{Time:time.Time{wall:0xbfd047eccd5f466b, ext:5320472505, loc:(*time.Location)(0x6e0d080)}}, LastTimestamp:v1.Time{Time:time.Time{wall:0xbfd047ecdb9d1679, ext:5559404496, loc:(*time.Location)(0x6e0d080)}}, Count:3, Type:"Normal", EventTime:v1.MicroTime{Time:time.Time{wall:0x0, ext:0, loc:(*time.Location)(nil)}}, Series:(*v1.EventSeries)(nil), Action:"", Related:(*v1.ObjectReference)(nil), ReportingController:"", ReportingInstance:""}': 'namespaces "default" not found' (will not retry!) Sep 15 11:40:49 my-cluster kubelet[3257]: I0915 11:40:49.724156 3257 reconciler.go:207] operationExecutor.VerifyControllerAttachedVolume started for volume "storage-provisioner-token-ngjmz" (UniqueName: "kubernetes.io/secret/f87c81f2-8901-40fa-90de-d71fbee02b6c-storage-provisioner-token-ngjmz") pod "storage-provisioner" (UID: "f87c81f2-8901-40fa-90de-d71fbee02b6c") Sep 15 11:40:49 my-cluster kubelet[3257]: I0915 11:40:49.724184 3257 reconciler.go:207] operationExecutor.VerifyControllerAttachedVolume started for volume "tmp" (UniqueName: "kubernetes.io/host-path/f87c81f2-8901-40fa-90de-d71fbee02b6c-tmp") pod "storage-provisioner" (UID: "f87c81f2-8901-40fa-90de-d71fbee02b6c") Sep 15 11:40:49 my-cluster kubelet[3257]: I0915 11:40:49.824462 3257 reconciler.go:207] operationExecutor.VerifyControllerAttachedVolume started for volume "lib-modules" (UniqueName: "kubernetes.io/host-path/f5c34f1d-90d6-4891-b27a-423cb906ca55-lib-modules") pod "kube-proxy-jbmx2" (UID: "f5c34f1d-90d6-4891-b27a-423cb906ca55") Sep 15 11:40:49 my-cluster kubelet[3257]: I0915 11:40:49.824632 3257 reconciler.go:207] operationExecutor.VerifyControllerAttachedVolume started for volume "xtables-lock" (UniqueName: "kubernetes.io/host-path/f5c34f1d-90d6-4891-b27a-423cb906ca55-xtables-lock") pod "kube-proxy-jbmx2" (UID: "f5c34f1d-90d6-4891-b27a-423cb906ca55") Sep 15 11:40:49 my-cluster kubelet[3257]: I0915 11:40:49.824715 3257 reconciler.go:207] operationExecutor.VerifyControllerAttachedVolume started for volume "kube-proxy" (UniqueName: "kubernetes.io/configmap/f5c34f1d-90d6-4891-b27a-423cb906ca55-kube-proxy") pod "kube-proxy-jbmx2" (UID: "f5c34f1d-90d6-4891-b27a-423cb906ca55") Sep 15 11:40:49 my-cluster kubelet[3257]: I0915 11:40:49.824779 3257 reconciler.go:207] operationExecutor.VerifyControllerAttachedVolume started for volume "kube-proxy-token-kzds8" (UniqueName: "kubernetes.io/secret/f5c34f1d-90d6-4891-b27a-423cb906ca55-kube-proxy-token-kzds8") pod "kube-proxy-jbmx2" (UID: "f5c34f1d-90d6-4891-b27a-423cb906ca55") Sep 15 11:40:49 my-cluster kubelet[3257]: I0915 11:40:49.824842 3257 reconciler.go:207] operationExecutor.VerifyControllerAttachedVolume started for volume "config-volume" (UniqueName: "kubernetes.io/configmap/638b50ff-95de-4f10-a60c-7b13bc70ff30-config-volume") pod "coredns-5644d7b6d9-882h8" (UID: "638b50ff-95de-4f10-a60c-7b13bc70ff30") Sep 15 11:40:49 my-cluster kubelet[3257]: I0915 11:40:49.824973 3257 reconciler.go:207] operationExecutor.VerifyControllerAttachedVolume started for volume "coredns-token-q4rvk" (UniqueName: "kubernetes.io/secret/638b50ff-95de-4f10-a60c-7b13bc70ff30-coredns-token-q4rvk") pod "coredns-5644d7b6d9-882h8" (UID: "638b50ff-95de-4f10-a60c-7b13bc70ff30") Sep 15 11:40:49 my-cluster kubelet[3257]: I0915 11:40:49.967809 3257 transport.go:132] certificate rotation detected, shutting down client connections to start using new credentials Sep 15 11:40:49 my-cluster kubelet[3257]: W0915 11:40:49.968365 3257 reflector.go:299] object-"kube-system"/"storage-provisioner-token-ngjmz": watch of *v1.Secret ended with: very short watch: object-"kube-system"/"storage-provisioner-token-ngjmz": Unexpected watch close - watch lasted less than a second and no items received Sep 15 11:40:49 my-cluster kubelet[3257]: W0915 11:40:49.968445 3257 reflector.go:299] object-"kube-system"/"kube-proxy": watch of *v1.ConfigMap ended with: very short watch: object-"kube-system"/"kube-proxy": Unexpected watch close - watch lasted less than a second and no items received Sep 15 11:40:49 my-cluster kubelet[3257]: W0915 11:40:49.968810 3257 reflector.go:299] object-"kube-system"/"coredns-token-q4rvk": watch of *v1.Secret ended with: very short watch: object-"kube-system"/"coredns-token-q4rvk": Unexpected watch close - watch lasted less than a second and no items received Sep 15 11:40:49 my-cluster kubelet[3257]: W0915 11:40:49.969321 3257 reflector.go:299] object-"kube-system"/"coredns": watch of *v1.ConfigMap ended with: very short watch: object-"kube-system"/"coredns": Unexpected watch close - watch lasted less than a second and no items received Sep 15 11:40:49 my-cluster kubelet[3257]: W0915 11:40:49.969388 3257 reflector.go:299] object-"kube-system"/"kube-proxy-token-kzds8": watch of *v1.Secret ended with: very short watch: object-"kube-system"/"kube-proxy-token-kzds8": Unexpected watch close - watch lasted less than a second and no items received Sep 15 11:40:50 my-cluster kubelet[3257]: W0915 11:40:50.281661 3257 pod_container_deletor.go:75] Container "4d43bf348abfe8b2b77f1ce0f427eafedf9def82e40e90b74dc728a24069765c" not found in pod's containers Sep 15 11:40:50 my-cluster kubelet[3257]: W0915 11:40:50.400972 3257 docker_sandbox.go:394] failed to read pod IP from plugin/docker: Couldn't find network status for kube-system/coredns-5644d7b6d9-882h8 through plugin: invalid network status for Sep 15 11:40:50 my-cluster kubelet[3257]: W0915 11:40:50.449972 3257 docker_sandbox.go:394] failed to read pod IP from plugin/docker: Couldn't find network status for kube-system/coredns-5644d7b6d9-882h8 through plugin: invalid network status for Sep 15 11:40:50 my-cluster kubelet[3257]: W0915 11:40:50.543022 3257 pod_container_deletor.go:75] Container "df6acc1c47ed6081ae2149ca59536424ebd4d45da0b75fb663fd45e875601525" not found in pod's containers Sep 15 11:40:51 my-cluster kubelet[3257]: W0915 11:40:51.549247 3257 docker_sandbox.go:394] failed to read pod IP from plugin/docker: Couldn't find network status for kube-system/coredns-5644d7b6d9-882h8 through plugin: invalid network status for ==> storage-provisioner [5d70a2da1b6e] <== F0915 11:41:20.443776 1 main.go:39] error getting server version: Get "https://10.96.0.1:443/version?timeout=32s": dial tcp 10.96.0.1:443: i/o timeout ==> storage-provisioner [70b7e36e263c] <== I0915 11:41:20.865601 1 leaderelection.go:242] attempting to acquire leader lease kube-system/k8s.io-minikube-hostpath... I0915 11:41:20.872810 1 leaderelection.go:252] successfully acquired lease kube-system/k8s.io-minikube-hostpath I0915 11:41:20.873068 1 event.go:281] Event(v1.ObjectReference{Kind:"Endpoints", Namespace:"kube-system", Name:"k8s.io-minikube-hostpath", UID:"58a15527-487c-4d8d-a4cb-f7401997a9da", APIVersion:"v1", ResourceVersion:"382", FieldPath:""}): type: 'Normal' reason: 'LeaderElection' my-cluster_6e859847-140d-4bdc-9a92-1f08ae2db160 became leader I0915 11:41:20.873662 1 controller.go:799] Starting provisioner controller k8s.io/minikube-hostpath_my-cluster_6e859847-140d-4bdc-9a92-1f08ae2db160! I0915 11:41:20.974383 1 controller.go:848] Started provisioner controller k8s.io/minikube-hostpath_my-cluster_6e859847-140d-4bdc-9a92-1f08ae2db160!
woodcockjosh commented 3 years ago

I have ran the command again and it appears to be working now. I'm not sure if this is intermittent or if this was an issue with a previous version.

tstromberg commented 3 years ago

I noticed your output mentions v1.12.3: I believe this bug was fixed in v1.13: https://github.com/kubernetes/minikube/pull/9033

Have you seen this with the latest release?

woodcockjosh commented 3 years ago

yeah I think when I upgraded it got fixed