search

OverC / meta-overc

OverC + cubes distribution layer
MIT License
13 stars 28 forks source link

Description of boot process #292

Closed outbackdingo closed 2 years ago

outbackdingo commented 4 years ago

can someone provide me an architectural overview of the overc boot process ? x86_64 uses grub, and boots what / where / how is the rest of the stack initialized to spawn all the containers my guess would be via some init script / systemd process and lxc ? i dont need a dev doc as i understand it, just a description of the process.

simply put, ive so far worked out the x86_64 so now im on to making this work/boot on an ARMv8 device also via u-boot

zeddii commented 4 years ago

I have written this up in detail in the past, I just don't know where it landed. I don't have access to those old write ups, but will start a dump of info here, and then I can get it into the READMEs. Here's some bare bones information:

Leaving the bootloader steps out of the way for now, the heart of the boot starts after control has been handed to cube-essential. That hand off can be via bootloader -> initramfs/initrd, or whatever the target dictates.

Essential only has one container runtime: pflask. Which is super tiny and basic, and has enough functionality to launch oci containers that complete the rest of the boot process. Essential is hardly ever upgraded and should be considered like firmware, hence the small size, low attack surface and minimal services.

Essential has a basic service that starts those core pflask container(s). One of which is dom0 the other is typically the vrf. Those are system containers and supply the services for the finalization of the boot.

VRF does the routing, and dom0 has the runc container runtime (which it grafts to essential for some namespace magic). Dom0 has a list of containers to start and a service that starts them (via runc).

What is started by which container is a property of how they are installed. The cube-installer sets this for the system/default containers and the others are set by the user when they are installed (see the "autostart" cube-cfg/cube-ctl calls). What devices are available to a container, what privileges it has, what it starts, etc, are all properties of the container itself or the configuration as set when the container was installed.

outbackdingo commented 4 years ago

ok, great... better understanding now. as for essential, so being "firmware" there really isnt a need for it to get a tty is there, to prevent "local" access in a deployment scenerio .... or move it to a later tty so its not visible

zeddii commented 4 years ago

essential having a tty can be configured (but I don't recall it being changed recently), and it has been argued both ways if it should or shouldn't have one by default. The active tty/vt is presented normally as dom0 or cube-server/desktop to give the user the feeling that the plumbing of the system doesn't need to be used for day to day activities.

outbackdingo commented 4 years ago

ok to continue this "thread" ... Dom0 has a list of containers to start and a service that starts them (via runc). .... where is this "list" located, and does it include specified parameters / privs per container?

objective: deploy 2-3 or 4 cube-k8s nodes with different hostnames on a single overc server.... 1 master and 3 workers..... i can bring up k8s one cube-k8s no worries, and install weave, so far so good. ssh into cube-k8s run: mount -t devtmpfs none /dev mount devpts /dev/pts -t devpts rm /opt/cni/bin cp -a /usr/libexec/cni/ /opt/cni/bin vi /etc/hosts (add 192.168.1.42 cube-k8s) swapoff -a kubeadm init --cri-socket /var/run/dockershim.sock --ignore-preflight-errors=ALL

results in W0809 16:12:40.581320 919 configset.go:348] WARNING: kubeadm cannot validate component configs for API groups [kubelet.config.k8s.io kubeproxy.config.k8s.io] [init] Using Kubernetes version: v1.18.6 [preflight] Running pre-flight checks [WARNING Service-Docker]: docker service is not enabled, please run 'systemctl enable docker.service' [WARNING IsDockerSystemdCheck]: detected "cgroupfs" as the Docker cgroup driver. The recommended driver is "systemd". Please follow the guide at https://kubernetes.io/docs/setup/cri/ [WARNING SystemVerification]: missing optional cgroups: hugetlb [WARNING KubeletVersion]: the kubelet version is higher than the control plane version. This is not a supported version skew and may lead to a malfunctional cluster. Kubelet version: "1.19.0-rc.3.31+bdc575e10c35a3-dirty" Control plane version: "1.18.6" [preflight] Pulling images required for setting up a Kubernetes cluster [preflight] This might take a minute or two, depending on the speed of your internet connection [preflight] You can also perform this action in beforehand using 'kubeadm config images pull' [certs] Using certificateDir folder "/etc/kubernetes/pki" [certs] Generating "ca" certificate and key [certs] Generating "apiserver" certificate and key [certs] apiserver serving cert is signed for DNS names [cube-k8s kubernetes kubernetes.default kubernetes.default.svc kubernetes.default.svc.cluster.local] and IPs [10.96.0.1 192.168.122.124] [certs] Generating "apiserver-kubelet-client" certificate and key [certs] Generating "front-proxy-ca" certificate and key [certs] Generating "front-proxy-client" certificate and key [certs] Generating "etcd/ca" certificate and key [certs] Generating "etcd/server" certificate and key [certs] etcd/server serving cert is signed for DNS names [cube-k8s localhost] and IPs [192.168.122.124 127.0.0.1 ::1] [certs] Generating "etcd/peer" certificate and key [certs] etcd/peer serving cert is signed for DNS names [cube-k8s localhost] and IPs [192.168.122.124 127.0.0.1 ::1] [certs] Generating "etcd/healthcheck-client" certificate and key [certs] Generating "apiserver-etcd-client" certificate and key [certs] Generating "sa" key and public key [kubeconfig] Using kubeconfig folder "/etc/kubernetes" [kubeconfig] Writing "admin.conf" kubeconfig file [kubeconfig] Writing "kubelet.conf" kubeconfig file [kubeconfig] Writing "controller-manager.conf" kubeconfig file [kubeconfig] Writing "scheduler.conf" kubeconfig file [kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env" [kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml" [kubelet-start] Starting the kubelet [control-plane] Using manifest folder "/etc/kubernetes/manifests" [control-plane] Creating static Pod manifest for "kube-apiserver" [control-plane] Creating static Pod manifest for "kube-controller-manager" [control-plane] Creating static Pod manifest for "kube-scheduler" [etcd] Creating static Pod manifest for local etcd in "/etc/kubernetes/manifests" [wait-control-plane] Waiting for the kubelet to boot up the control plane as static Pods from directory "/etc/kubernetes/manifests". This can take up to 4m0s [apiclient] All control plane components are healthy after 29.503100 seconds [upload-config] Storing the configuration used in ConfigMap "kubeadm-config" in the "kube-system" Namespace [kubelet] Creating a ConfigMap "kubelet-config-1.18" in namespace kube-system with the configuration for the kubelets in the cluster [upload-certs] Skipping phase. Please see --upload-certs [mark-control-plane] Marking the node cube-k8s as control-plane by adding the label "node-role.kubernetes.io/master=''" [mark-control-plane] Marking the node cube-k8s as control-plane by adding the taints [node-role.kubernetes.io/master:NoSchedule] [bootstrap-token] Using token: gv4gre.ymeijp4bcrza2e2l [bootstrap-token] Configuring bootstrap tokens, cluster-info ConfigMap, RBAC Roles [bootstrap-token] configured RBAC rules to allow Node Bootstrap tokens to get nodes [bootstrap-token] configured RBAC rules to allow Node Bootstrap tokens to post CSRs in order for nodes to get long term certificate credentials [bootstrap-token] configured RBAC rules to allow the csrapprover controller automatically approve CSRs from a Node Bootstrap Token [bootstrap-token] configured RBAC rules to allow certificate rotation for all node client certificates in the cluster [bootstrap-token] Creating the "cluster-info" ConfigMap in the "kube-public" namespace [kubelet-finalize] Updating "/etc/kubernetes/kubelet.conf" to point to a rotatable kubelet client certificate and key [addons] Applied essential addon: CoreDNS [addons] Applied essential addon: kube-proxy

Your Kubernetes control-plane has initialized successfully!

To start using your cluster, you need to run the following as a regular user:

mkdir -p $HOME/.kube sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config sudo chown $(id -u):$(id -g) $HOME/.kube/config

You should now deploy a pod network to the cluster. Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at: https://kubernetes.io/docs/concepts/cluster-administration/addons/

Then you can join any number of worker nodes by running the following on each as root:

kubeadm join 192.168.122.124:6443 --token gv4gre.ymeijp4bcrza2e2l \ --discovery-token-ca-cert-hash sha256:6a79f91c3a7e85a3532ad40301a7e25cbaf7fdbcb8483085f1a3078bc1d18e00

results as expected on cube-k8s

kubectl get node NAME STATUS ROLES AGE VERSION cube-k8s NotReady master 116s v1.19.0-rc.3.31+bdc575e10c35a3-dirty

root@cube-k8s:~# kubectl get pods --all-namespaces NAMESPACE NAME READY STATUS RESTARTS AGE kube-system coredns-6f5c7bbdfb-5bs8w 0/1 Pending 0 118s kube-system coredns-6f5c7bbdfb-svdr5 0/1 Pending 0 118s kube-system etcd-cube-k8s 1/1 Running 0 119s kube-system kube-apiserver-cube-k8s 1/1 Running 0 119s kube-system kube-controller-manager-cube-k8s 1/1 Running 0 119s kube-system kube-proxy-bs424 1/1 Running 0 118s kube-system kube-scheduler-cube-k8s 1/1 Running 0 119s

install weave ..... kubectl apply -f "https://cloud.weave.works/k8s/net?k8s-version=$(kubectl version | base64 | tr -d '\n')"

results as expected ...

root@cube-k8s:~# kubectl get node NAME STATUS ROLES AGE VERSION cube-k8s Ready master 5m29s v1.19.0-rc.3.31+bdc575e10c35a3-dirty

root@cube-k8s:~# kubectl get pods --all-namespaces NAMESPACE NAME READY STATUS RESTARTS AGE kube-system coredns-6f5c7bbdfb-5bs8w 1/1 Running 0 5m20s kube-system coredns-6f5c7bbdfb-svdr5 1/1 Running 0 5m20s kube-system etcd-cube-k8s 1/1 Running 0 5m21s kube-system kube-apiserver-cube-k8s 1/1 Running 0 5m21s kube-system kube-controller-manager-cube-k8s 1/1 Running 0 5m21s kube-system kube-proxy-bs424 1/1 Running 0 5m20s kube-system kube-scheduler-cube-k8s 1/1 Running 0 5m21s kube-system weave-net-j6p62 2/2 Running 0 95s

then scp cube-k8s-node-genericx86-64-20200809085758.rootfs.tar.bz2

scp build/tmp/deploy/images/genericx86-64/cube-k8s-node-genericx86-64-20200809085758.rootfs.tar.bz2 root@192.168.122.124:/root/ root@192.168.122.124's password: cube-k8s-node-genericx86-64-20200809085758.rootfs.tar.bz2 100% 741MB 616.1MB/s 00:01

then to dom0 root@cube-k8s:~# scp cube-k8s-node-genericx86-64-20200809085758.rootfs.tar.bz2 root@192.168.42.3:/root/ root@192.168.42.3's password: cube-k8s-node-genericx86-64-20200809085758.rootfs.tar.bz2 100% 741MB 391.1MB/s 00:01

root@cube-k8s:~# ssh root@192.168.42.3 root@192.168.42.3's password: Last login: Sun Aug 9 15:54:13 2020 from 192.168.42.1

root@cube-dom0:~# cube-ctl add -n cube-node1 cube-k8s-node-genericx86-64-20200809085758.rootfs.tar.bz2 [INFO] Installing container cube-node1 to /opt/container//cube-node1 [INFO] Extracting rootfs..... [INFO] Succeeded [INFO] Performing OCI configuration ...

root@cube-dom0:~# cube-ctl status name type status attributes addresses

cube-builder runc running -- 192.168.42.240 cube-desktop runc running -- 192.168.42.33 cube-k8s runc running netprime 192.168.42.1,192.168.122.124 cube-node1 runc available -- -- cube-server runc running -- 192.168.42.178 cube-vrf pflask running vrf 192.168.42.4 dom0 pflask running -- 192.168.42.3

we now have cube-node1 available, so as cube-k8s is set for cube.container.system we do the same for cube-node1

cube-cfg -n cube-node1 set cube.container.system:true

then

root@cube-dom0:~# cube-ctl start cube-node1 root@cube-dom0:~# cube-ctl status name type status attributes addresses

cube-builder runc running -- 192.168.42.240 cube-desktop runc running -- 192.168.42.33 cube-k8s runc running netprime 192.168.42.1,192.168.122.124 cube-node1 runc running dirty 192.168.42.133 cube-server runc running -- 192.168.42.178 cube-vrf pflask running vrf 192.168.42.4 dom0 pflask running -- 192.168.42.3

ssh into cube-node1 run mount -t devtmpfs none /dev mount devpts /dev/pts -t devpts rm /opt/cni/bin cp -a /usr/libexec/cni/ /opt/cni/bin swapoff -a edit /etc/hosts (add 192.168.42.1 cube-k8s, and 192.168.42.xxx cube-node1)

then run

kubeadm join 192.168.122.124:6443 --token gv4gre.ymeijp4bcrza2e2l \

--discovery-token-ca-cert-hash sha256:6a79f91c3a7e85a3532ad40301a7e25cbaf7fdbcb8483085f1a3078bc1d18e00 --cri-socket /var/run/dockershim.sock

[preflight] Running pre-flight checks [WARNING Service-Docker]: docker service is not enabled, please run 'systemctl enable docker.service' [WARNING IsDockerSystemdCheck]: detected "cgroupfs" as the Docker cgroup driver. The recommended driver is "systemd". Please follow the guide at https://kubernetes.io/docs/setup/cri/ [WARNING SystemVerification]: missing optional cgroups: hugetlb [preflight] Reading configuration from the cluster... [preflight] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -oyaml' [kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml" [kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env" [kubelet-start] Starting the kubelet [kubelet-start] Waiting for the kubelet to perform the TLS Bootstrap...

This node has joined the cluster:

Run 'kubectl get nodes' on the control-plane to see this node join the cluster.

on cube-k8s

kubectl get pods --all-namespaces NAMESPACE NAME READY STATUS RESTARTS AGE kube-system coredns-6f5c7bbdfb-5bs8w 1/1 Running 0 26m kube-system coredns-6f5c7bbdfb-svdr5 1/1 Running 0 26m kube-system etcd-cube-k8s 1/1 Running 0 26m kube-system kube-apiserver-cube-k8s 1/1 Running 0 26m kube-system kube-controller-manager-cube-k8s 1/1 Running 0 26m kube-system kube-proxy-2vg6r 0/1 RunContainerError 1 69s kube-system kube-proxy-bs424 1/1 Running 0 26m kube-system kube-scheduler-cube-k8s 1/1 Running 0 26m kube-system weave-net-5986m 0/2 RunContainerError 2 69s kube-system weave-net-j6p62 2/2 Running 0 22m

now we start to fail....

in journalctl -f i noticed this Aug 09 16:40:27 cube-node1 dockerd[268]: time="2020-08-09T16:40:27.618336948Z" level=error msg="Handler for POST /v1.40/containers/b0e116e123f723d8a1a9f68aedf760326365aaceb6485bc6599b641ceb0c0b05/start returned error: OCI runtime create failed: container_linux.go:345: starting container process caused \"apply caps: operation not permitted\": unknown"

so if cube-k8s can run weave, why cannot an injected cube-k8s as cube-node ??

Im would have thought this to be simple... :) any insight ?

ive added journalctl -f logs from cube-node1

root@cube-node1:~# journalctl -f -- Logs begin at Sun 2020-08-09 16:30:07 UTC. -- Aug 09 16:40:27 cube-node1 systemd[128]: var-lib-docker-overlay2-723261dbf88bb0d6a3697ad13105b57b566eb5d198d96ac33e3c393a55a058d0-merged.mount: Succeeded. Aug 09 16:40:27 cube-node1 dockerd[268]: time="2020-08-09T16:40:27.618309036Z" level=error msg="b0e116e123f723d8a1a9f68aedf760326365aaceb6485bc6599b641ceb0c0b05 cleanup: failed to delete container from containerd: no such container" Aug 09 16:40:27 cube-node1 dockerd[268]: time="2020-08-09T16:40:27.618336948Z" level=error msg="Handler for POST /v1.40/containers/b0e116e123f723d8a1a9f68aedf760326365aaceb6485bc6599b641ceb0c0b05/start returned error: OCI runtime create failed: container_linux.go:345: starting container process caused \"apply caps: operation not permitted\": unknown" Aug 09 16:40:27 cube-node1 kubelet[703]: E0809 16:40:27.619262 703 remote_runtime.go:248] StartContainer "b0e116e123f723d8a1a9f68aedf760326365aaceb6485bc6599b641ceb0c0b05" from runtime service failed: rpc error: code = Unknown desc = failed to start container "b0e116e123f723d8a1a9f68aedf760326365aaceb6485bc6599b641ceb0c0b05": Error response from daemon: OCI runtime create failed: container_linux.go:345: starting container process caused "apply caps: operation not permitted": unknown Aug 09 16:40:27 cube-node1 kubelet[703]: E0809 16:40:27.619353 703 kuberuntime_manager.go:798] container &Container{Name:kube-proxy,Image:k8s.gcr.io/kube-proxy:v1.18.6,Command:[/usr/local/bin/kube-proxy --config=/var/lib/kube-proxy/config.conf --hostname-override=$(NODE_NAME)],Args:[],WorkingDir:,Ports:[]ContainerPort{},Env:[]EnvVar{EnvVar{Name:NODE_NAME,Value:,ValueFrom:&EnvVarSource{FieldRef:&ObjectFieldSelector{APIVersion:v1,FieldPath:spec.nodeName,},ResourceFieldRef:nil,ConfigMapKeyRef:nil,SecretKeyRef:nil,},},},Resources:ResourceRequirements{Limits:ResourceList{},Requests:ResourceList{},},VolumeMounts:[]VolumeMount{VolumeMount{Name:kube-proxy,ReadOnly:false,MountPath:/var/lib/kube-proxy,SubPath:,MountPropagation:nil,SubPathExpr:,},VolumeMount{Name:xtables-lock,ReadOnly:false,MountPath:/run/xtables.lock,SubPath:,MountPropagation:nil,SubPathExpr:,},VolumeMount{Name:lib-modules,ReadOnly:true,MountPath:/lib/modules,SubPath:,MountPropagation:nil,SubPathExpr:,},VolumeMount{Name:kube-proxy-token-mjsh6,ReadOnly:true,MountPath:/var/run/secrets/kubernetes.io/serviceaccount,SubPath:,MountPropagation:nil,SubPathExpr:,},},LivenessProbe:nil,ReadinessProbe:nil,Lifecycle:nil,TerminationMessagePath:/dev/termination-log,ImagePullPolicy:IfNotPresent,SecurityContext:&SecurityContext{Capabilities:nil,Privileged:true,SELinuxOptions:nil,RunAsUser:nil,RunAsNonRoot:nil,ReadOnlyRootFilesystem:nil,AllowPrivilegeEscalation:nil,RunAsGroup:nil,ProcMount:nil,WindowsOptions:nil,SeccompProfile:nil,},Stdin:false,StdinOnce:false,TTY:false,EnvFrom:[]EnvFromSource{},TerminationMessagePolicy:File,VolumeDevices:[]VolumeDevice{},StartupProbe:nil,} start failed in pod kube-proxy-2vg6r_kube-system(d4463fe1-d07b-4377-ad89-a9993bff619f): RunContainerError: failed to start container "b0e116e123f723d8a1a9f68aedf760326365aaceb6485bc6599b641ceb0c0b05": Error response from daemon: OCI runtime create failed: container_linux.go:345: starting container process caused "apply caps: operation not permitted": unknown Aug 09 16:40:27 cube-node1 kubelet[703]: E0809 16:40:27.619379 703 pod_workers.go:191] Error syncing pod d4463fe1-d07b-4377-ad89-a9993bff619f ("kube-proxy-2vg6r_kube-system(d4463fe1-d07b-4377-ad89-a9993bff619f)"), skipping: failed to "StartContainer" for "kube-proxy" with RunContainerError: "failed to start container \"b0e116e123f723d8a1a9f68aedf760326365aaceb6485bc6599b641ceb0c0b05\": Error response from daemon: OCI runtime create failed: container_linux.go:345: starting container process caused \"apply caps: operation not permitted\": unknown" Aug 09 16:40:29 cube-node1 kubelet[703]: W0809 16:40:29.777753 703 cni.go:239] Unable to update cni config: no networks found in /etc/cni/net.d Aug 09 16:40:31 cube-node1 kubelet[703]: E0809 16:40:31.329269 703 kubelet.go:2100] Container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:docker: network plugin is not ready: cni config uninitialized Aug 09 16:40:34 cube-node1 kubelet[703]: W0809 16:40:34.778305 703 cni.go:239] Unable to update cni config: no networks found in /etc/cni/net.d Aug 09 16:40:36 cube-node1 kubelet[703]: E0809 16:40:36.340841 703 kubelet.go:2100] Container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:docker: network plugin is not ready: cni config uninitialized Aug 09 16:40:39 cube-node1 kubelet[703]: I0809 16:40:39.061491 703 topology_manager.go:219] [topologymanager] RemoveContainer - Container ID: aae9b8182ea6643f992adb97995861ec448e94be17ebf3be49c34b50aafe049d Aug 09 16:40:39 cube-node1 kubelet[703]: I0809 16:40:39.061681 703 topology_manager.go:219] [topologymanager] RemoveContainer - Container ID: 8718abc3dd84e67f277051063dfe04558d316d53e8d9d7c9806882e9f8cd53c3 Aug 09 16:40:39 cube-node1 dockerd[268]: time="2020-08-09T16:40:39.076929124Z" level=warning msg="Your kernel does not support CPU cfs period or the cgroup is not mounted. Period discarded." Aug 09 16:40:39 cube-node1 systemd[1]: var-lib-docker-overlay2-dc6e918ee74eb4765069e9865dcf45dd92cb3bbfe8eb48d98f5150998d4a04b7\x2dinit-merged.mount: Succeeded. Aug 09 16:40:39 cube-node1 systemd[128]: var-lib-docker-overlay2-dc6e918ee74eb4765069e9865dcf45dd92cb3bbfe8eb48d98f5150998d4a04b7\x2dinit-merged.mount: Succeeded. Aug 09 16:40:39 cube-node1 dockerd[278]: time="2020-08-09T16:40:39.179032160Z" level=info msg="shim containerd-shim started" address="/containerd-shim/moby/a8ddf12c8890b5a382fb9c20435cc756a04841d375cabf39683533ab0f3af967/shim.sock" debug=false pid=1698 Aug 09 16:40:39 cube-node1 dockerd[278]: time="2020-08-09T16:40:39.220599412Z" level=info msg="shim reaped" id=a8ddf12c8890b5a382fb9c20435cc756a04841d375cabf39683533ab0f3af967 Aug 09 16:40:39 cube-node1 dockerd[268]: time="2020-08-09T16:40:39.230820645Z" level=error msg="stream copy error: reading from a closed fifo" Aug 09 16:40:39 cube-node1 dockerd[268]: time="2020-08-09T16:40:39.230849052Z" level=error msg="stream copy error: reading from a closed fifo" Aug 09 16:40:39 cube-node1 systemd[1]: var-lib-docker-overlay2-dc6e918ee74eb4765069e9865dcf45dd92cb3bbfe8eb48d98f5150998d4a04b7-merged.mount: Succeeded. Aug 09 16:40:39 cube-node1 systemd[128]: var-lib-docker-overlay2-dc6e918ee74eb4765069e9865dcf45dd92cb3bbfe8eb48d98f5150998d4a04b7-merged.mount: Succeeded. Aug 09 16:40:39 cube-node1 dockerd[268]: time="2020-08-09T16:40:39.268142289Z" level=error msg="a8ddf12c8890b5a382fb9c20435cc756a04841d375cabf39683533ab0f3af967 cleanup: failed to delete container from containerd: no such container" Aug 09 16:40:39 cube-node1 dockerd[268]: time="2020-08-09T16:40:39.268189226Z" level=error msg="Handler for POST /v1.40/containers/a8ddf12c8890b5a382fb9c20435cc756a04841d375cabf39683533ab0f3af967/start returned error: OCI runtime create failed: container_linux.go:345: starting container process caused \"apply caps: operation not permitted\": unknown" Aug 09 16:40:39 cube-node1 kubelet[703]: E0809 16:40:39.270257 703 remote_runtime.go:248] StartContainer "a8ddf12c8890b5a382fb9c20435cc756a04841d375cabf39683533ab0f3af967" from runtime service failed: rpc error: code = Unknown desc = failed to start container "a8ddf12c8890b5a382fb9c20435cc756a04841d375cabf39683533ab0f3af967": Error response from daemon: OCI runtime create failed: container_linux.go:345: starting container process caused "apply caps: operation not permitted": unknown Aug 09 16:40:39 cube-node1 kubelet[703]: E0809 16:40:39.270404 703 kuberuntime_manager.go:798] container &Container{Name:weave,Image:docker.io/weaveworks/weave-kube:2.7.0,Command:[/home/weave/launch.sh],Args:[],WorkingDir:,Ports:[]ContainerPort{},Env:[]EnvVar{EnvVar{Name:HOSTNAME,Value:,ValueFrom:&EnvVarSource{FieldRef:&ObjectFieldSelector{APIVersion:v1,FieldPath:spec.nodeName,},ResourceFieldRef:nil,ConfigMapKeyRef:nil,SecretKeyRef:nil,},},},Resources:ResourceRequirements{Limits:ResourceList{},Requests:ResourceList{cpu: {{50 -3} {} 50m DecimalSI},memory: {{104857600 0} {} 100Mi BinarySI},},},VolumeMounts:[]VolumeMount{VolumeMount{Name:weavedb,ReadOnly:false,MountPath:/weavedb,SubPath:,MountPropagation:nil,SubPathExpr:,},VolumeMount{Name:cni-bin,ReadOnly:false,MountPath:/host/opt,SubPath:,MountPropagation:nil,SubPathExpr:,},VolumeMount{Name:cni-bin2,ReadOnly:false,MountPath:/host/home,SubPath:,MountPropagation:nil,SubPathExpr:,},VolumeMount{Name:cni-conf,ReadOnly:false,MountPath:/host/etc,SubPath:,MountPropagation:nil,SubPathExpr:,},VolumeMount{Name:dbus,ReadOnly:false,MountPath:/host/var/lib/dbus,SubPath:,MountPropagation:nil,SubPathExpr:,},VolumeMount{Name:lib-modules,ReadOnly:false,MountPath:/lib/modules,SubPath:,MountPropagation:nil,SubPathExpr:,},VolumeMount{Name:xtables-lock,ReadOnly:false,MountPath:/run/xtables.lock,SubPath:,MountPropagation:nil,SubPathExpr:,},VolumeMount{Name:weave-net-token-dpw9l,ReadOnly:true,MountPath:/var/run/secrets/kubernetes.io/serviceaccount,SubPath:,MountPropagation:nil,SubPathExpr:,},},LivenessProbe:nil,ReadinessProbe:&Probe{Handler:Handler{Exec:nil,HTTPGet:&HTTPGetAction{Path:/status,Port:{0 6784 },Host:127.0.0.1,Scheme:HTTP,HTTPHeaders:[]HTTPHeader{},},TCPSocket:nil,},InitialDelaySeconds:0,TimeoutSeconds:1,PeriodSeconds:10,SuccessThreshold:1,FailureThreshold:3,},Lifecycle:nil,TerminationMessagePath:/dev/termination-log,ImagePullPolicy:IfNotPresent,SecurityContext:&SecurityContext{Capabilities:nil,Privileged:true,SELinuxOptions:nil,RunAsUser:nil,RunAsNonRoot:nil,ReadOnlyRootFilesystem:nil,AllowPrivilegeEscalation:nil,RunAsGroup:nil,ProcMount:nil,WindowsOptions:nil,SeccompProfile:nil,},Stdin:false,StdinOnce:false,TTY:false,EnvFrom:[]EnvFromSource{},TerminationMessagePolicy:File,VolumeDevices:[]VolumeDevice{},StartupProbe:nil,} start failed in pod weave-net-5986m_kube-system(9801ab59-a02b-461a-b8eb-293863bdf7da): RunContainerError: failed to start container "a8ddf12c8890b5a382fb9c20435cc756a04841d375cabf39683533ab0f3af967": Error response from daemon: OCI runtime create failed: container_linux.go:345: starting container process caused "apply caps: operation not permitted": unknown Aug 09 16:40:39 cube-node1 dockerd[268]: time="2020-08-09T16:40:39.276339893Z" level=warning msg="Your kernel does not support CPU cfs period or the cgroup is not mounted. Period discarded." Aug 09 16:40:39 cube-node1 systemd[1]: var-lib-docker-overlay2-800e736633aae2932251143f089dabdb07d6beace907911f329236e81e512b11\x2dinit-merged.mount: Succeeded. Aug 09 16:40:39 cube-node1 systemd[128]: var-lib-docker-overlay2-800e736633aae2932251143f089dabdb07d6beace907911f329236e81e512b11\x2dinit-merged.mount: Succeeded. Aug 09 16:40:39 cube-node1 systemd[1]: var-lib-docker-overlay2-800e736633aae2932251143f089dabdb07d6beace907911f329236e81e512b11-merged.mount: Succeeded. Aug 09 16:40:39 cube-node1 systemd[128]: var-lib-docker-overlay2-800e736633aae2932251143f089dabdb07d6beace907911f329236e81e512b11-merged.mount: Succeeded. Aug 09 16:40:39 cube-node1 dockerd[278]: time="2020-08-09T16:40:39.362116124Z" level=info msg="shim containerd-shim started" address="/containerd-shim/moby/1ed1f13a580ff0399a049d12e9050e64cb1482f45d838fd96f28d11582f61597/shim.sock" debug=false pid=1719 Aug 09 16:40:39 cube-node1 dockerd[278]: time="2020-08-09T16:40:39.404703792Z" level=info msg="shim reaped" id=1ed1f13a580ff0399a049d12e9050e64cb1482f45d838fd96f28d11582f61597 Aug 09 16:40:39 cube-node1 dockerd[268]: time="2020-08-09T16:40:39.414864213Z" level=error msg="stream copy error: reading from a closed fifo" Aug 09 16:40:39 cube-node1 dockerd[268]: time="2020-08-09T16:40:39.414990907Z" level=error msg="stream copy error: reading from a closed fifo" Aug 09 16:40:39 cube-node1 systemd[128]: var-lib-docker-overlay2-800e736633aae2932251143f089dabdb07d6beace907911f329236e81e512b11-merged.mount: Succeeded. Aug 09 16:40:39 cube-node1 systemd[1]: var-lib-docker-overlay2-800e736633aae2932251143f089dabdb07d6beace907911f329236e81e512b11-merged.mount: Succeeded. Aug 09 16:40:39 cube-node1 dockerd[268]: time="2020-08-09T16:40:39.445768659Z" level=error msg="1ed1f13a580ff0399a049d12e9050e64cb1482f45d838fd96f28d11582f61597 cleanup: failed to delete container from containerd: no such container" Aug 09 16:40:39 cube-node1 dockerd[268]: time="2020-08-09T16:40:39.445992690Z" level=error msg="Handler for POST /v1.40/containers/1ed1f13a580ff0399a049d12e9050e64cb1482f45d838fd96f28d11582f61597/start returned error: OCI runtime create failed: container_linux.go:345: starting container process caused \"apply caps: operation not permitted\": unknown" Aug 09 16:40:39 cube-node1 kubelet[703]: E0809 16:40:39.447575 703 remote_runtime.go:248] StartContainer "1ed1f13a580ff0399a049d12e9050e64cb1482f45d838fd96f28d11582f61597" from runtime service failed: rpc error: code = Unknown desc = failed to start container "1ed1f13a580ff0399a049d12e9050e64cb1482f45d838fd96f28d11582f61597": Error response from daemon: OCI runtime create failed: container_linux.go:345: starting container process caused "apply caps: operation not permitted": unknown Aug 09 16:40:39 cube-node1 kubelet[703]: E0809 16:40:39.447720 703 kuberuntime_manager.go:798] container &Container{Name:weave-npc,Image:docker.io/weaveworks/weave-npc:2.7.0,Command:[],Args:[],WorkingDir:,Ports:[]ContainerPort{},Env:[]EnvVar{EnvVar{Name:HOSTNAME,Value:,ValueFrom:&EnvVarSource{FieldRef:&ObjectFieldSelector{APIVersion:v1,FieldPath:spec.nodeName,},ResourceFieldRef:nil,ConfigMapKeyRef:nil,SecretKeyRef:nil,},},},Resources:ResourceRequirements{Limits:ResourceList{},Requests:ResourceList{cpu: {{50 -3} {} 50m DecimalSI},memory: {{104857600 0} {} 100Mi BinarySI},},},VolumeMounts:[]VolumeMount{VolumeMount{Name:xtables-lock,ReadOnly:false,MountPath:/run/xtables.lock,SubPath:,MountPropagation:nil,SubPathExpr:,},VolumeMount{Name:weave-net-token-dpw9l,ReadOnly:true,MountPath:/var/run/secrets/kubernetes.io/serviceaccount,SubPath:,MountPropagation:nil,SubPathExpr:,},},LivenessProbe:nil,ReadinessProbe:nil,Lifecycle:nil,TerminationMessagePath:/dev/termination-log,ImagePullPolicy:IfNotPresent,SecurityContext:&SecurityContext{Capabilities:nil,Privileged:*true,SELinuxOptions:nil,RunAsUser:nil,RunAsNonRoot:nil,ReadOnlyRootFilesystem:nil,AllowPrivilegeEscalation:nil,RunAsGroup:nil,ProcMount:nil,WindowsOptions:nil,SeccompProfile:nil,},Stdin:false,StdinOnce:false,TTY:false,EnvFrom:[]EnvFromSource{},TerminationMessagePolicy:File,VolumeDevices:[]VolumeDevice{},StartupProbe:nil,} start failed in pod weave-net-5986m_kube-system(9801ab59-a02b-461a-b8eb-293863bdf7da): RunContainerError: failed to start container "1ed1f13a580ff0399a049d12e9050e64cb1482f45d838fd96f28d11582f61597": Error response from daemon: OCI runtime create failed: container_linux.go:345: starting container process caused "apply caps: operation not permitted": unknown Aug 09 16:40:39 cube-node1 kubelet[703]: E0809 16:40:39.447759 703 pod_workers.go:191] Error syncing pod 9801ab59-a02b-461a-b8eb-293863bdf7da ("weave-net-5986m_kube-system(9801ab59-a02b-461a-b8eb-293863bdf7da)"), skipping: [failed to "StartContainer" for "weave" with RunContainerError: "failed to start container \"a8ddf12c8890b5a382fb9c20435cc756a04841d375cabf39683533ab0f3af967\": Error response from daemon: OCI runtime create failed: container_linux.go:345: starting container process caused \"apply caps: operation not permitted\": unknown", failed to "StartContainer" for "weave-npc" with RunContainerError: "failed to start container \"1ed1f13a580ff0399a049d12e9050e64cb1482f45d838fd96f28d11582f61597\": Error response from daemon: OCI runtime create failed: container_linux.go:345: starting container process caused \"apply caps: operation not permitted\": unknown"] Aug 09 16:40:39 cube-node1 kubelet[703]: W0809 16:40:39.778651 703 cni.go:239] Unable to update cni config: no networks found in /etc/cni/net.d Aug 09 16:40:39 cube-node1 systemd-journald[27]: Forwarding to syslog missed 116 messages. Aug 09 16:40:40 cube-node1 kubelet[703]: I0809 16:40:40.061396 703 topology_manager.go:219] [topologymanager] RemoveContainer - Container ID: b0e116e123f723d8a1a9f68aedf760326365aaceb6485bc6599b641ceb0c0b05 Aug 09 16:40:40 cube-node1 kubelet[703]: E0809 16:40:40.062318 703 pod_workers.go:191] Error syncing pod d4463fe1-d07b-4377-ad89-a9993bff619f ("kube-proxy-2vg6r_kube-system(d4463fe1-d07b-4377-ad89-a9993bff619f)"), skipping: failed to "StartContainer" for "kube-proxy" with CrashLoopBackOff: "back-off 40s restarting failed container=kube-proxy pod=kube-proxy-2vg6r_kube-system(d4463fe1-d07b-4377-ad89-a9993bff619f)" Aug 09 16:40:41 cube-node1 kubelet[703]: E0809 16:40:41.349806 703 kubelet.go:2100] Container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:docker: network plugin is not ready: cni config uninitialized Aug 09 16:40:41 cube-node1 kubelet[703]: W0809 16:40:41.632743 703 container.go:549] Failed to update stats for container "/docker/ebad596b4218122f78a06502a91a8c8f023a274117b23b3f26ade40ef8bf837d": unable to determine device info for dir: /var/lib/docker/overlay2/56bffad9cdf4e79d74de0217515c0280a6cbf3c647bf1a0cd74d71fa2b996ba2/diff: could not find device with major: 0, minor: 41 in cached partitions map, continuing to push stats Aug 09 16:40:44 cube-node1 kubelet[703]: W0809 16:40:44.359241 703 container.go:549] Failed to update stats for container "/docker/80f77a517e3d60b21ae99ede09a47cd98bed2bb9a9ca4e2d0a0126c6d8722d6c": unable to determine device info for dir: /var/lib/docker/overlay2/4b58bde0611b7ec04e859c8a2a37acf40bd441b63d9ec93af83f8a9c3b5ea468/diff: could not find device with major: 0, minor: 41 in cached partitions map, continuing to push stats Aug 09 16:40:44 cube-node1 kubelet[703]: W0809 16:40:44.779108 703 cni.go:239] Unable to update cni config: no networks found in /etc/cni/net.d Aug 09 16:40:46 cube-node1 kubelet[703]: E0809 16:40:46.359579 703 kubelet.go:2100] Container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:docker: network plugin is not ready: cni config uninitialized Aug 09 16:40:49 cube-node1 kubelet[703]: W0809 16:40:49.779272 703 cni.go:239] Unable to update cni config: no networks found in /etc/cni/net.d Aug 09 16:40:51 cube-node1 kubelet[703]: I0809 16:40:51.061513 703 topology_manager.go:219] [topologymanager] RemoveContainer - Container ID: b0e116e123f723d8a1a9f68aedf760326365aaceb6485bc6599b641ceb0c0b05 Aug 09 16:40:51 cube-node1 kubelet[703]: E0809 16:40:51.061973 703 pod_workers.go:191] Error syncing pod d4463fe1-d07b-4377-ad89-a9993bff619f ("kube-proxy-2vg6r_kube-system(d4463fe1-d07b-4377-ad89-a9993bff619f)"), skipping: failed to "StartContainer" for "kube-proxy" with CrashLoopBackOff: "back-off 40s restarting failed container=kube-proxy pod=kube-proxy-2vg6r_kube-system(d4463fe1-d07b-4377-ad89-a9993bff619f)" Aug 09 16:40:51 cube-node1 kubelet[703]: E0809 16:40:51.368370 703 kubelet.go:2100] Container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:docker: network plugin is not ready: cni config uninitialized Aug 09 16:40:52 cube-node1 kubelet[703]: I0809 16:40:52.061674 703 topology_manager.go:219] [topologymanager] RemoveContainer - Container ID: a8ddf12c8890b5a382fb9c20435cc756a04841d375cabf39683533ab0f3af967 Aug 09 16:40:52 cube-node1 kubelet[703]: I0809 16:40:52.061967 703 topology_manager.go:219] [topologymanager] RemoveContainer - Container ID: 1ed1f13a580ff0399a049d12e9050e64cb1482f45d838fd96f28d11582f61597 Aug 09 16:40:52 cube-node1 kubelet[703]: E0809 16:40:52.063257 703 pod_workers.go:191] Error syncing pod 9801ab59-a02b-461a-b8eb-293863bdf7da ("weave-net-5986m_kube-system(9801ab59-a02b-461a-b8eb-293863bdf7da)"), skipping: [failed to "StartContainer" for "weave" with CrashLoopBackOff: "back-off 40s restarting failed container=weave pod=weave-net-5986m_kube-system(9801ab59-a02b-461a-b8eb-293863bdf7da)", failed to "StartContainer" for "weave-npc" with CrashLoopBackOff: "back-off 40s restarting failed container=weave-npc pod=weave-net-5986m_kube-system(9801ab59-a02b-461a-b8eb-293863bdf7da)"] Aug 09 16:40:54 cube-node1 kubelet[703]: W0809 16:40:54.779594 703 cni.go:239] Unable to update cni config: no networks found in /etc/cni/net.d Aug 09 16:40:56 cube-node1 kubelet[703]: E0809 16:40:56.375561 703 kubelet.go:2100] Container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:docker: network plugin is not ready: cni config uninitialized Aug 09 16:40:59 cube-node1 kubelet[703]: W0809 16:40:59.780586 703 cni.go:239] Unable to update cni config: no networks found in /etc/cni/net.d Aug 09 16:41:01 cube-node1 kubelet[703]: E0809 16:41:01.383873 703 kubelet.go:2100] Container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:docker: network plugin is not ready: cni config uninitialized Aug 09 16:41:03 cube-node1 kubelet[703]: I0809 16:41:03.061567 703 topology_manager.go:219] [topologymanager] RemoveContainer - Container ID: b0e116e123f723d8a1a9f68aedf760326365aaceb6485bc6599b641ceb0c0b05 Aug 09 16:41:03 cube-node1 kubelet[703]: E0809 16:41:03.062112 703 pod_workers.go:191] Error syncing pod d4463fe1-d07b-4377-ad89-a9993bff619f ("kube-proxy-2vg6r_kube-system(d4463fe1-d07b-4377-ad89-a9993bff619f)"), skipping: failed to "StartContainer" for "kube-proxy" with CrashLoopBackOff: "back-off 40s restarting failed container=kube-proxy pod=kube-proxy-2vg6r_kube-system(d4463fe1-d07b-4377-ad89-a9993bff619f)" Aug 09 16:41:04 cube-node1 kubelet[703]: W0809 16:41:04.780843 703 cni.go:239] Unable to update cni config: no networks found in /etc/cni/net.d Aug 09 16:41:06 cube-node1 kubelet[703]: I0809 16:41:06.061463 703 topology_manager.go:219] [topologymanager] RemoveContainer - Container ID: a8ddf12c8890b5a382fb9c20435cc756a04841d375cabf39683533ab0f3af967 Aug 09 16:41:06 cube-node1 kubelet[703]: I0809 16:41:06.062321 703 topology_manager.go:219] [topologymanager] RemoveContainer - Container ID: 1ed1f13a580ff0399a049d12e9050e64cb1482f45d838fd96f28d11582f61597 Aug 09 16:41:06 cube-node1 kubelet[703]: E0809 16:41:06.063381 703 pod_workers.go:191] Error syncing pod 9801ab59-a02b-461a-b8eb-293863bdf7da ("weave-net-5986m_kube-system(9801ab59-a02b-461a-b8eb-293863bdf7da)"), skipping: [failed to "StartContainer" for "weave" with CrashLoopBackOff: "back-off 40s restarting failed container=weave pod=weave-net-5986m_kube-system(9801ab59-a02b-461a-b8eb-293863bdf7da)", failed to "StartContainer" for "weave-npc" with CrashLoopBackOff: "back-off 40s restarting failed container=weave-npc pod=weave-net-5986m_kube-system(9801ab59-a02b-461a-b8eb-293863bdf7da)"] Aug 09 16:41:06 cube-node1 kubelet[703]: E0809 16:41:06.392292 703 kubelet.go:2100] Container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:docker: network plugin is not ready: cni config uninitialized Aug 09 16:41:09 cube-node1 kubelet[703]: W0809 16:41:09.781248 703 cni.go:239] Unable to update cni config: no networks found in /etc/cni/net.d Aug 09 16:41:09 cube-node1 systemd-journald[27]: Forwarding to syslog missed 26 messages. Aug 09 16:41:11 cube-node1 kubelet[703]: E0809 16:41:11.401022 703 kubelet.go:2100] Container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:docker: network plugin is not ready: cni config uninitialized Aug 09 16:41:14 cube-node1 kubelet[703]: I0809 16:41:14.061547 703 topology_manager.go:219] [topologymanager] RemoveContainer - Container ID: b0e116e123f723d8a1a9f68aedf760326365aaceb6485bc6599b641ceb0c0b05 Aug 09 16:41:14 cube-node1 dockerd[268]: time="2020-08-09T16:41:14.074677085Z" level=warning msg="Your kernel does not support CPU cfs period or the cgroup is not mounted. Period discarded." Aug 09 16:41:14 cube-node1 systemd[1]: var-lib-docker-overlay2-2676ce8f3d47bb70931fac0f3fdf1f42e51577340da13cef331effadfc53d344\x2dinit-merged.mount: Succeeded. Aug 09 16:41:14 cube-node1 systemd[128]: var-lib-docker-overlay2-2676ce8f3d47bb70931fac0f3fdf1f42e51577340da13cef331effadfc53d344\x2dinit-merged.mount: Succeeded. Aug 09 16:41:14 cube-node1 dockerd[278]: time="2020-08-09T16:41:14.150961707Z" level=info msg="shim containerd-shim started" address="/containerd-shim/moby/a81746b73514748eb615313910ea47d0d296f68888666d59803148a1404f5fc4/shim.sock" debug=false pid=1868 Aug 09 16:41:14 cube-node1 systemd[1]: run-docker-runtime\x2drunc-moby-a81746b73514748eb615313910ea47d0d296f68888666d59803148a1404f5fc4-runc.CJOYQv.mount: Succeeded. Aug 09 16:41:14 cube-node1 systemd[128]: run-docker-runtime\x2drunc-moby-a81746b73514748eb615313910ea47d0d296f68888666d59803148a1404f5fc4-runc.CJOYQv.mount: Succeeded. Aug 09 16:41:14 cube-node1 dockerd[278]: time="2020-08-09T16:41:14.184956961Z" level=info msg="shim reaped" id=a81746b73514748eb615313910ea47d0d296f68888666d59803148a1404f5fc4 Aug 09 16:41:14 cube-node1 dockerd[268]: time="2020-08-09T16:41:14.195283288Z" level=error msg="stream copy error: reading from a closed fifo" Aug 09 16:41:14 cube-node1 dockerd[268]: time="2020-08-09T16:41:14.195311706Z" level=error msg="stream copy error: reading from a closed fifo" Aug 09 16:41:14 cube-node1 systemd[1]: var-lib-docker-overlay2-2676ce8f3d47bb70931fac0f3fdf1f42e51577340da13cef331effadfc53d344-merged.mount: Succeeded. Aug 09 16:41:14 cube-node1 systemd[128]: var-lib-docker-overlay2-2676ce8f3d47bb70931fac0f3fdf1f42e51577340da13cef331effadfc53d344-merged.mount: Succeeded. Aug 09 16:41:14 cube-node1 dockerd[268]: time="2020-08-09T16:41:14.225654970Z" level=error msg="a81746b73514748eb615313910ea47d0d296f68888666d59803148a1404f5fc4 cleanup: failed to delete container from containerd: no such container" Aug 09 16:41:14 cube-node1 dockerd[268]: time="2020-08-09T16:41:14.225695881Z" level=error msg="Handler for POST /v1.40/containers/a81746b73514748eb615313910ea47d0d296f68888666d59803148a1404f5fc4/start returned error: OCI runtime create failed: container_linux.go:345: starting container process caused \"apply caps: operation not permitted\": unknown" Aug 09 16:41:14 cube-node1 kubelet[703]: E0809 16:41:14.226634 703 remote_runtime.go:248] StartContainer "a81746b73514748eb615313910ea47d0d296f68888666d59803148a1404f5fc4" from runtime service failed: rpc error: code = Unknown desc = failed to start container "a81746b73514748eb615313910ea47d0d296f68888666d59803148a1404f5fc4": Error response from daemon: OCI runtime create failed: container_linux.go:345: starting container process caused "apply caps: operation not permitted": unknown Aug 09 16:41:14 cube-node1 kubelet[703]: E0809 16:41:14.226729 703 kuberuntime_manager.go:798] container &Container{Name:kube-proxy,Image:k8s.gcr.io/kube-proxy:v1.18.6,Command:[/usr/local/bin/kube-proxy --config=/var/lib/kube-proxy/config.conf --hostname-override=$(NODE_NAME)],Args:[],WorkingDir:,Ports:[]ContainerPort{},Env:[]EnvVar{EnvVar{Name:NODE_NAME,Value:,ValueFrom:&EnvVarSource{FieldRef:&ObjectFieldSelector{APIVersion:v1,FieldPath:spec.nodeName,},ResourceFieldRef:nil,ConfigMapKeyRef:nil,SecretKeyRef:nil,},},},Resources:ResourceRequirements{Limits:ResourceList{},Requests:ResourceList{},},VolumeMounts:[]VolumeMount{VolumeMount{Name:kube-proxy,ReadOnly:false,MountPath:/var/lib/kube-proxy,SubPath:,MountPropagation:nil,SubPathExpr:,},VolumeMount{Name:xtables-lock,ReadOnly:false,MountPath:/run/xtables.lock,SubPath:,MountPropagation:nil,SubPathExpr:,},VolumeMount{Name:lib-modules,ReadOnly:true,MountPath:/lib/modules,SubPath:,MountPropagation:nil,SubPathExpr:,},VolumeMount{Name:kube-proxy-token-mjsh6,ReadOnly:true,MountPath:/var/run/secrets/kubernetes.io/serviceaccount,SubPath:,MountPropagation:nil,SubPathExpr:,},},LivenessProbe:nil,ReadinessProbe:nil,Lifecycle:nil,TerminationMessagePath:/dev/termination-log,ImagePullPolicy:IfNotPresent,SecurityContext:&SecurityContext{Capabilities:nil,Privileged:*true,SELinuxOptions:nil,RunAsUser:nil,RunAsNonRoot:nil,ReadOnlyRootFilesystem:nil,AllowPrivilegeEscalation:nil,RunAsGroup:nil,ProcMount:nil,WindowsOptions:nil,SeccompProfile:nil,},Stdin:false,StdinOnce:false,TTY:false,EnvFrom:[]EnvFromSource{},TerminationMessagePolicy:File,VolumeDevices:[]VolumeDevice{},StartupProbe:nil,} start failed in pod kube-proxy-2vg6r_kube-system(d4463fe1-d07b-4377-ad89-a9993bff619f): RunContainerError: failed to start container "a81746b73514748eb615313910ea47d0d296f68888666d59803148a1404f5fc4": Error response from daemon: OCI runtime create failed: container_linux.go:345: starting container process caused "apply caps: operation not permitted": unknown Aug 09 16:41:14 cube-node1 kubelet[703]: E0809 16:41:14.226752 703 pod_workers.go:191] Error syncing pod d4463fe1-d07b-4377-ad89-a9993bff619f ("kube-proxy-2vg6r_kube-system(d4463fe1-d07b-4377-ad89-a9993bff619f)"), skipping: failed to "StartContainer" for "kube-proxy" with RunContainerError: "failed to start container \"a81746b73514748eb615313910ea47d0d296f68888666d59803148a1404f5fc4\": Error response from daemon: OCI runtime create failed: container_linux.go:345: starting container process caused \"apply caps: operation not permitted\": unknown" Aug 09 16:41:14 cube-node1 kubelet[703]: W0809 16:41:14.781618 703 cni.go:239] Unable to update cni config: no networks found in /etc/cni/net.d Aug 09 16:41:16 cube-node1 kubelet[703]: E0809 16:41:16.410977 703 kubelet.go:2100] Container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:docker: network plugin is not ready: cni config uninitialized Aug 09 16:41:19 cube-node1 kubelet[703]: I0809 16:41:19.061445 703 topology_manager.go:219] [topologymanager] RemoveContainer - Container ID: a8ddf12c8890b5a382fb9c20435cc756a04841d375cabf39683533ab0f3af967 Aug 09 16:41:19 cube-node1 kubelet[703]: I0809 16:41:19.061582 703 topology_manager.go:219] [topologymanager] RemoveContainer - Container ID: 1ed1f13a580ff0399a049d12e9050e64cb1482f45d838fd96f28d11582f61597 Aug 09 16:41:19 cube-node1 kubelet[703]: E0809 16:41:19.062085 703 pod_workers.go:191] Error syncing pod 9801ab59-a02b-461a-b8eb-293863bdf7da ("weave-net-5986m_kube-system(9801ab59-a02b-461a-b8eb-293863bdf7da)"), skipping: [failed to "StartContainer" for "weave" with CrashLoopBackOff: "back-off 40s restarting failed container=weave pod=weave-net-5986m_kube-system(9801ab59-a02b-461a-b8eb-293863bdf7da)", failed to "StartContainer" for "weave-npc" with CrashLoopBackOff: "back-off 40s restarting failed container=weave-npc pod=weave-net-5986m_kube-system(9801ab59-a02b-461a-b8eb-293863bdf7da)"] Aug 09 16:41:19 cube-node1 kubelet[703]: W0809 16:41:19.781987 703 cni.go:239] Unable to update cni config: no networks found in /etc/cni/net.d Aug 09 16:41:21 cube-node1 kubelet[703]: E0809 16:41:21.419683 703 kubelet.go:2100] Container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:docker: network plugin is not ready: cni config uninitialized Aug 09 16:41:24 cube-node1 kubelet[703]: W0809 16:41:24.782240 703 cni.go:239] Unable to update cni config: no networks found in /etc/cni/net.d Aug 09 16:41:26 cube-node1 kubelet[703]: I0809 16:41:26.193246 703 topology_manager.go:219] [topologymanager] RemoveContainer - Container ID: aae9b8182ea6643f992adb97995861ec448e94be17ebf3be49c34b50aafe049d Aug 09 16:41:26 cube-node1 kubelet[703]: I0809 16:41:26.213919 703 topology_manager.go:219] [topologymanager] RemoveContainer - Container ID: 1e9b495d3a2af0d7501a44dc2ab015a7f7991f71887824d4063be47e64c90055 Aug 09 16:41:26 cube-node1 kubelet[703]: I0809 16:41:26.231878 703 topology_manager.go:219] [topologymanager] RemoveContainer - Container ID: b0e116e123f723d8a1a9f68aedf760326365aaceb6485bc6599b641ceb0c0b05 Aug 09 16:41:26 cube-node1 kubelet[703]: I0809 16:41:26.250266 703 topology_manager.go:219] [topologymanager] RemoveContainer - Container ID: 8718abc3dd84e67f277051063dfe04558d316d53e8d9d7c9806882e9f8cd53c3 Aug 09 16:41:26 cube-node1 kubelet[703]: E0809 16:41:26.429729 703 kubelet.go:2100] Container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:docker: network plugin is not ready: cni config uninitialized Aug 09 16:41:26 cube-node1 kubelet[703]: W0809 16:41:26.782232 703 pod_container_deletor.go:79] Container "b0e116e123f723d8a1a9f68aedf760326365aaceb6485bc6599b641ceb0c0b05" not found in pod's containers Aug 09 16:41:26 cube-node1 kubelet[703]: W0809 16:41:26.782299 703 pod_container_deletor.go:79] Container "1e9b495d3a2af0d7501a44dc2ab015a7f7991f71887824d4063be47e64c90055" not found in pod's containers Aug 09 16:41:26 cube-node1 kubelet[703]: I0809 16:41:26.782713 703 topology_manager.go:219] [topologymanager] RemoveContainer - Container ID: a81746b73514748eb615313910ea47d0d296f68888666d59803148a1404f5fc4 Aug 09 16:41:26 cube-node1 kubelet[703]: E0809 16:41:26.783585 703 pod_workers.go:191] Error syncing pod d4463fe1-d07b-4377-ad89-a9993bff619f ("kube-proxy-2vg6r_kube-system(d4463fe1-d07b-4377-ad89-a9993bff619f)"), skipping: failed to "StartContainer" for "kube-proxy" with CrashLoopBackOff: "back-off 1m20s restarting failed container=kube-proxy pod=kube-proxy-2vg6r_kube-system(d4463fe1-d07b-4377-ad89-a9993bff619f)" Aug 09 16:41:26 cube-node1 kubelet[703]: W0809 16:41:26.795422 703 pod_container_deletor.go:79] Container "8718abc3dd84e67f277051063dfe04558d316d53e8d9d7c9806882e9f8cd53c3" not found in pod's containers Aug 09 16:41:26 cube-node1 kubelet[703]: I0809 16:41:26.795732 703 topology_manager.go:219] [topologymanager] RemoveContainer - Container ID: a8ddf12c8890b5a382fb9c20435cc756a04841d375cabf39683533ab0f3af967 Aug 09 16:41:26 cube-node1 kubelet[703]: W0809 16:41:26.795763 703 pod_container_deletor.go:79] Container "aae9b8182ea6643f992adb97995861ec448e94be17ebf3be49c34b50aafe049d" not found in pod's containers Aug 09 16:41:26 cube-node1 kubelet[703]: I0809 16:41:26.795838 703 topology_manager.go:219] [topologymanager] RemoveContainer - Container ID: 1ed1f13a580ff0399a049d12e9050e64cb1482f45d838fd96f28d11582f61597 Aug 09 16:41:26 cube-node1 dockerd[268]: time="2020-08-09T16:41:26.803689746Z" level=warning msg="Your kernel does not support CPU cfs period or the cgroup is not mounted. Period discarded." Aug 09 16:41:26 cube-node1 systemd[1]: var-lib-docker-overlay2-cc1aade1b5186360858c642c77b5a8d923e67e43c1f2d857a0455c72ff6f892f\x2dinit-merged.mount: Succeeded. Aug 09 16:41:26 cube-node1 systemd[128]: var-lib-docker-overlay2-cc1aade1b5186360858c642c77b5a8d923e67e43c1f2d857a0455c72ff6f892f\x2dinit-merged.mount: Succeeded. Aug 09 16:41:26 cube-node1 dockerd[278]: time="2020-08-09T16:41:26.880559191Z" level=info msg="shim containerd-shim started" address="/containerd-shim/moby/ea55242559317f04426e22cf70bf84c37a896f68287defd0b0495e9a9f99bca9/shim.sock" debug=false pid=1960 Aug 09 16:41:26 cube-node1 dockerd[278]: time="2020-08-09T16:41:26.912717551Z" level=info msg="shim reaped" id=ea55242559317f04426e22cf70bf84c37a896f68287defd0b0495e9a9f99bca9 Aug 09 16:41:26 cube-node1 dockerd[268]: time="2020-08-09T16:41:26.922973672Z" level=error msg="stream copy error: reading from a closed fifo" Aug 09 16:41:26 cube-node1 dockerd[268]: time="2020-08-09T16:41:26.924457483Z" level=error msg="stream copy error: reading from a closed fifo" Aug 09 16:41:26 cube-node1 systemd[1]: var-lib-docker-overlay2-cc1aade1b5186360858c642c77b5a8d923e67e43c1f2d857a0455c72ff6f892f-merged.mount: Succeeded. Aug 09 16:41:26 cube-node1 systemd[128]: var-lib-docker-overlay2-cc1aade1b5186360858c642c77b5a8d923e67e43c1f2d857a0455c72ff6f892f-merged.mount: Succeeded. Aug 09 16:41:26 cube-node1 dockerd[268]: time="2020-08-09T16:41:26.947404366Z" level=error msg="ea55242559317f04426e22cf70bf84c37a896f68287defd0b0495e9a9f99bca9 cleanup: failed to delete container from containerd: no such container" Aug 09 16:41:26 cube-node1 dockerd[268]: time="2020-08-09T16:41:26.947426534Z" level=error msg="Handler for POST /v1.40/containers/ea55242559317f04426e22cf70bf84c37a896f68287defd0b0495e9a9f99bca9/start returned error: OCI runtime create failed: container_linux.go:345: starting container process caused \"apply caps: operation not permitted\": unknown" Aug 09 16:41:26 cube-node1 kubelet[703]: E0809 16:41:26.948436 703 remote_runtime.go:248] StartContainer "ea55242559317f04426e22cf70bf84c37a896f68287defd0b0495e9a9f99bca9" from runtime service failed: rpc error: code = Unknown desc = failed to start container "ea55242559317f04426e22cf70bf84c37a896f68287defd0b0495e9a9f99bca9": Error response from daemon: OCI runtime create failed: container_linux.go:345: starting container process caused "apply caps: operation not permitted": unknown Aug 09 16:41:26 cube-node1 kubelet[703]: E0809 16:41:26.948556 703 kuberuntime_manager.go:798] container &Container{Name:weave,Image:docker.io/weaveworks/weave-kube:2.7.0,Command:[/home/weave/launch.sh],Args:[],WorkingDir:,Ports:[]ContainerPort{},Env:[]EnvVar{EnvVar{Name:HOSTNAME,Value:,ValueFrom:&EnvVarSource{FieldRef:&ObjectFieldSelector{APIVersion:v1,FieldPath:spec.nodeName,},ResourceFieldRef:nil,ConfigMapKeyRef:nil,SecretKeyRef:nil,},},},Resources:ResourceRequirements{Limits:ResourceList{},Requests:ResourceList{cpu: {{50 -3} {} 50m DecimalSI},memory: {{104857600 0} {} 100Mi BinarySI},},},VolumeMounts:[]VolumeMount{VolumeMount{Name:weavedb,ReadOnly:false,MountPath:/weavedb,SubPath:,MountPropagation:nil,SubPathExpr:,},VolumeMount{Name:cni-bin,ReadOnly:false,MountPath:/host/opt,SubPath:,MountPropagation:nil,SubPathExpr:,},VolumeMount{Name:cni-bin2,ReadOnly:false,MountPath:/host/home,SubPath:,MountPropagation:nil,SubPathExpr:,},VolumeMount{Name:cni-conf,ReadOnly:false,MountPath:/host/etc,SubPath:,MountPropagation:nil,SubPathExpr:,},VolumeMount{Name:dbus,ReadOnly:false,MountPath:/host/var/lib/dbus,SubPath:,MountPropagation:nil,SubPathExpr:,},VolumeMount{Name:lib-modules,ReadOnly:false,MountPath:/lib/modules,SubPath:,MountPropagation:nil,SubPathExpr:,},VolumeMount{Name:xtables-lock,ReadOnly:false,MountPath:/run/xtables.lock,SubPath:,MountPropagation:nil,SubPathExpr:,},VolumeMount{Name:weave-net-token-dpw9l,ReadOnly:true,MountPath:/var/run/secrets/kubernetes.io/serviceaccount,SubPath:,MountPropagation:nil,SubPathExpr:,},},LivenessProbe:nil,ReadinessProbe:&Probe{Handler:Handler{Exec:nil,HTTPGet:&HTTPGetAction{Path:/status,Port:{0 6784 },Host:127.0.0.1,Scheme:HTTP,HTTPHeaders:[]HTTPHeader{},},TCPSocket:nil,},InitialDelaySeconds:0,TimeoutSeconds:1,PeriodSeconds:10,SuccessThreshold:1,FailureThreshold:3,},Lifecycle:nil,TerminationMessagePath:/dev/termination-log,ImagePullPolicy:IfNotPresent,SecurityContext:&SecurityContext{Capabilities:nil,Privileged:true,SELinuxOptions:nil,RunAsUser:nil,RunAsNonRoot:nil,ReadOnlyRootFilesystem:nil,AllowPrivilegeEscalation:nil,RunAsGroup:nil,ProcMount:nil,WindowsOptions:nil,SeccompProfile:nil,},Stdin:false,StdinOnce:false,TTY:false,EnvFrom:[]EnvFromSource{},TerminationMessagePolicy:File,VolumeDevices:[]VolumeDevice{},StartupProbe:nil,} start failed in pod weave-net-5986m_kube-system(9801ab59-a02b-461a-b8eb-293863bdf7da): RunContainerError: failed to start container "ea55242559317f04426e22cf70bf84c37a896f68287defd0b0495e9a9f99bca9": Error response from daemon: OCI runtime create failed: container_linux.go:345: starting container process caused "apply caps: operation not permitted": unknown Aug 09 16:41:26 cube-node1 dockerd[268]: time="2020-08-09T16:41:26.952826859Z" level=warning msg="Your kernel does not support CPU cfs period or the cgroup is not mounted. Period discarded." Aug 09 16:41:26 cube-node1 systemd[1]: var-lib-docker-overlay2-7da3da08fc7a60c0535fbaa62abded66066b5ec3b7853eb4e6eee2dba1bfb677\x2dinit-merged.mount: Succeeded. Aug 09 16:41:26 cube-node1 systemd[128]: var-lib-docker-overlay2-7da3da08fc7a60c0535fbaa62abded66066b5ec3b7853eb4e6eee2dba1bfb677\x2dinit-merged.mount: Succeeded. Aug 09 16:41:27 cube-node1 dockerd[278]: time="2020-08-09T16:41:27.011662830Z" level=info msg="shim containerd-shim started" address="/containerd-shim/moby/76baed69e682c202e31f190446abdfdec1b79345732bd5c9709130d26616e66b/shim.sock" debug=false pid=1983 Aug 09 16:41:27 cube-node1 dockerd[278]: time="2020-08-09T16:41:27.051894357Z" level=info msg="shim reaped" id=76baed69e682c202e31f190446abdfdec1b79345732bd5c9709130d26616e66b Aug 09 16:41:27 cube-node1 dockerd[268]: time="2020-08-09T16:41:27.062025696Z" level=error msg="stream copy error: reading from a closed fifo" Aug 09 16:41:27 cube-node1 dockerd[268]: time="2020-08-09T16:41:27.062078618Z" level=error msg="stream copy error: reading from a closed fifo" Aug 09 16:41:27 cube-node1 systemd[1]: var-lib-docker-overlay2-7da3da08fc7a60c0535fbaa62abded66066b5ec3b7853eb4e6eee2dba1bfb677-merged.mount: Succeeded. Aug 09 16:41:27 cube-node1 systemd[128]: var-lib-docker-overlay2-7da3da08fc7a60c0535fbaa62abded66066b5ec3b7853eb4e6eee2dba1bfb677-merged.mount: Succeeded. Aug 09 16:41:27 cube-node1 dockerd[268]: time="2020-08-09T16:41:27.088475368Z" level=error msg="76baed69e682c202e31f190446abdfdec1b79345732bd5c9709130d26616e66b cleanup: failed to delete container from containerd: no such container" Aug 09 16:41:27 cube-node1 dockerd[268]: time="2020-08-09T16:41:27.088518969Z" level=error msg="Handler for POST /v1.40/containers/76baed69e682c202e31f190446abdfdec1b79345732bd5c9709130d26616e66b/start returned error: OCI runtime create failed: container_linux.go:345: starting container process caused \"apply caps: operation not permitted\": unknown" Aug 09 16:41:27 cube-node1 kubelet[703]: E0809 16:41:27.089627 703 remote_runtime.go:248] StartContainer "76baed69e682c202e31f190446abdfdec1b79345732bd5c9709130d26616e66b" from runtime service failed: rpc error: code = Unknown desc = failed to start container "76baed69e682c202e31f190446abdfdec1b79345732bd5c9709130d26616e66b": Error response from daemon: OCI runtime create failed: container_linux.go:345: starting container process caused "apply caps: operation not permitted": unknown Aug 09 16:41:27 cube-node1 kubelet[703]: E0809 16:41:27.089735 703 kuberuntime_manager.go:798] container &Container{Name:weave-npc,Image:docker.io/weaveworks/weave-npc:2.7.0,Command:[],Args:[],WorkingDir:,Ports:[]ContainerPort{},Env:[]EnvVar{EnvVar{Name:HOSTNAME,Value:,ValueFrom:&EnvVarSource{FieldRef:&ObjectFieldSelector{APIVersion:v1,FieldPath:spec.nodeName,},ResourceFieldRef:nil,ConfigMapKeyRef:nil,SecretKeyRef:nil,},},},Resources:ResourceRequirements{Limits:ResourceList{},Requests:ResourceList{cpu: {{50 -3} {} 50m DecimalSI},memory: {{104857600 0} {} 100Mi BinarySI},},},VolumeMounts:[]VolumeMount{VolumeMount{Name:xtables-lock,ReadOnly:false,MountPath:/run/xtables.lock,SubPath:,MountPropagation:nil,SubPathExpr:,},VolumeMount{Name:weave-net-token-dpw9l,ReadOnly:true,MountPath:/var/run/secrets/kubernetes.io/serviceaccount,SubPath:,MountPropagation:nil,SubPathExpr:,},},LivenessProbe:nil,ReadinessProbe:nil,Lifecycle:nil,TerminationMessagePath:/dev/termination-log,ImagePullPolicy:IfNotPresent,SecurityContext:&SecurityContext{Capabilities:nil,Privileged:true,SELinuxOptions:nil,RunAsUser:nil,RunAsNonRoot:nil,ReadOnlyRootFilesystem:nil,AllowPrivilegeEscalation:nil,RunAsGroup:nil,ProcMount:nil,WindowsOptions:nil,SeccompProfile:nil,},Stdin:false,StdinOnce:false,TTY:false,EnvFrom:[]EnvFromSource{},TerminationMessagePolicy:File,VolumeDevices:[]VolumeDevice{},StartupProbe:nil,} start failed in pod weave-net-5986m_kube-system(9801ab59-a02b-461a-b8eb-293863bdf7da): RunContainerError: failed to start container "76baed69e682c202e31f190446abdfdec1b79345732bd5c9709130d26616e66b": Error response from daemon: OCI runtime create failed: container_linux.go:345: starting container process caused "apply caps: operation not permitted": unknown Aug 09 16:41:27 cube-node1 kubelet[703]: E0809 16:41:27.089771 703 pod_workers.go:191] Error syncing pod 9801ab59-a02b-461a-b8eb-293863bdf7da ("weave-net-5986m_kube-system(9801ab59-a02b-461a-b8eb-293863bdf7da)"), skipping: [failed to "StartContainer" for "weave" with RunContainerError: "failed to start container \"ea55242559317f04426e22cf70bf84c37a896f68287defd0b0495e9a9f99bca9\": Error response from daemon: OCI runtime create failed: container_linux.go:345: starting container process caused \"apply caps: operation not permitted\": unknown", failed to "StartContainer" for "weave-npc" with RunContainerError: "failed to start container \"76baed69e682c202e31f190446abdfdec1b79345732bd5c9709130d26616e66b\": Error response from daemon: OCI runtime create failed: container_linux.go:345: starting container process caused \"apply caps: operation not permitted\": unknown"] Aug 09 16:41:27 cube-node1 kubelet[703]: I0809 16:41:27.820599 703 topology_manager.go:219] [topologymanager] RemoveContainer - Container ID: a81746b73514748eb615313910ea47d0d296f68888666d59803148a1404f5fc4 Aug 09 16:41:27 cube-node1 kubelet[703]: I0809 16:41:27.820858 703 topology_manager.go:219] [topologymanager] RemoveContainer - Container ID: ea55242559317f04426e22cf70bf84c37a896f68287defd0b0495e9a9f99bca9 Aug 09 16:41:27 cube-node1 kubelet[703]: E0809 16:41:27.820933 703 pod_workers.go:191] Error syncing pod d4463fe1-d07b-4377-ad89-a9993bff619f ("kube-proxy-2vg6r_kube-system(d4463fe1-d07b-4377-ad89-a9993bff619f)"), skipping: failed to "StartContainer" for "kube-proxy" with CrashLoopBackOff: "back-off 1m20s restarting failed container=kube-proxy pod=kube-proxy-2vg6r_kube-system(d4463fe1-d07b-4377-ad89-a9993bff619f)" Aug 09 16:41:27 cube-node1 kubelet[703]: I0809 16:41:27.820972 703 topology_manager.go:219] [topologymanager] RemoveContainer - Container ID: 76baed69e682c202e31f190446abdfdec1b79345732bd5c9709130d26616e66b Aug 09 16:41:27 cube-node1 kubelet[703]: E0809 16:41:27.821727 703 pod_workers.go:191] Error syncing pod 9801ab59-a02b-461a-b8eb-293863bdf7da ("weave-net-5986m_kube-system(9801ab59-a02b-461a-b8eb-293863bdf7da)"), skipping: [failed to "StartContainer" for "weave" with CrashLoopBackOff: "back-off 1m20s restarting failed container=weave pod=weave-net-5986m_kube-system(9801ab59-a02b-461a-b8eb-293863bdf7da)", failed to "StartContainer" for "weave-npc" with CrashLoopBackOff: "back-off 1m20s restarting failed container=weave-npc pod=weave-net-5986m_kube-system(9801ab59-a02b-461a-b8eb-293863bdf7da)"] Aug 09 16:41:29 cube-node1 kubelet[703]: W0809 16:41:29.782545 703 cni.go:239] Unable to update cni config: no networks found in /etc/cni/net.d Aug 09 16:41:31 cube-node1 kubelet[703]: E0809 16:41:31.436949 703 kubelet.go:2100] Container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:docker: network plugin is not ready: cni config uninitialized Aug 09 16:41:34 cube-node1 kubelet[703]: W0809 16:41:34.782756 703 cni.go:239] Unable to update cni config: no networks found in /etc/cni/net.d Aug 09 16:41:36 cube-node1 kubelet[703]: E0809 16:41:36.444857 703 kubelet.go:2100] Container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:docker: network plugin is not ready: cni config uninitialized Aug 09 16:41:39 cube-node1 kubelet[703]: I0809 16:41:39.061504 703 topology_manager.go:219] [topologymanager] RemoveContainer - Container ID: ea55242559317f04426e22cf70bf84c37a896f68287defd0b0495e9a9f99bca9 Aug 09 16:41:39 cube-node1 kubelet[703]: I0809 16:41:39.061660 703 topology_manager.go:219] [topologymanager] RemoveContainer - Container ID: 76baed69e682c202e31f190446abdfdec1b79345732bd5c9709130d26616e66b Aug 09 16:41:39 cube-node1 kubelet[703]: E0809 16:41:39.062305 703 pod_workers.go:191] Error syncing pod 9801ab59-a02b-461a-b8eb-293863bdf7da ("weave-net-5986m_kube-system(9801ab59-a02b-461a-b8eb-293863bdf7da)"), skipping: [failed to "StartContainer" for "weave" with CrashLoopBackOff: "back-off 1m20s restarting failed container=weave pod=weave-net-5986m_kube-system(9801ab59-a02b-461a-b8eb-293863bdf7da)", failed to "StartContainer" for "weave-npc" with CrashLoopBackOff: "back-off 1m20s restarting failed container=weave-npc pod=weave-net-5986m_kube-system(9801ab59-a02b-461a-b8eb-293863bdf7da)"] Aug 09 16:41:39 cube-node1 kubelet[703]: W0809 16:41:39.783438 703 cni.go:239] Unable to update cni config: no networks found in /etc/cni/net.d Aug 09 16:41:39 cube-node1 systemd-journald[27]: Forwarding to syslog missed 79 messages. Aug 09 16:41:41 cube-node1 kubelet[703]: E0809 16:41:41.453639 703 kubelet.go:2100] Container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:docker: network plugin is not ready: cni config uninitialized Aug 09 16:41:42 cube-node1 kubelet[703]: I0809 16:41:42.061384 703 topology_manager.go:219] [topologymanager] RemoveContainer - Container ID: a81746b73514748eb615313910ea47d0d296f68888666d59803148a1404f5fc4 Aug 09 16:41:42 cube-node1 kubelet[703]: E0809 16:41:42.061720 703 pod_workers.go:191] Error syncing pod d4463fe1-d07b-4377-ad89-a9993bff619f ("kube-proxy-2vg6r_kube-system(d4463fe1-d07b-4377-ad89-a9993bff619f)"), skipping: failed to "StartContainer" for "kube-proxy" with CrashLoopBackOff: "back-off 1m20s restarting failed container=kube-proxy pod=kube-proxy-2vg6r_kube-system(d4463fe1-d07b-4377-ad89-a9993bff619f)" Aug 09 16:41:44 cube-node1 kubelet[703]: W0809 16:41:44.783903 703 cni.go:239] Unable to update cni config: no networks found in /etc/cni/net.d Aug 09 16:41:45 cube-node1 kubelet[703]: W0809 16:41:45.503553 703 container.go:549] Failed to update stats for container "/docker/80f77a517e3d60b21ae99ede09a47cd98bed2bb9a9ca4e2d0a0126c6d8722d6c": unable to determine device info for dir: /var/lib/docker/overlay2/4b58bde0611b7ec04e859c8a2a37acf40bd441b63d9ec93af83f8a9c3b5ea468/diff: could not find device with major: 0, minor: 41 in cached partitions map, continuing to push stats Aug 09 16:41:46 cube-node1 kubelet[703]: E0809 16:41:46.463148 703 kubelet.go:2100] Container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:docker: network plugin is not ready: cni config uninitialized Aug 09 16:41:49 cube-node1 kubelet[703]: W0809 16:41:49.784231 703 cni.go:239] Unable to update cni config: no networks found in /etc/cni/net.d Aug 09 16:41:51 cube-node1 kubelet[703]: I0809 16:41:51.061582 703 topology_manager.go:219] [topologymanager] RemoveContainer - Container ID: ea55242559317f04426e22cf70bf84c37a896f68287defd0b0495e9a9f99bca9 Aug 09 16:41:51 cube-node1 kubelet[703]: I0809 16:41:51.061774 703 topology_manager.go:219] [topologymanager] RemoveContainer - Container ID: 76baed69e682c202e31f190446abdfdec1b79345732bd5c9709130d26616e66b Aug 09 16:41:51 cube-node1 kubelet[703]: E0809 16:41:51.062534 703 pod_workers.go:191] Error syncing pod 9801ab59-a02b-461a-b8eb-293863bdf7da ("weave-net-5986m_kube-system(9801ab59-a02b-461a-b8eb-293863bdf7da)"), skipping: [failed to "StartContainer" for "weave" with CrashLoopBackOff: "back-off 1m20s restarting failed container=weave pod=weave-net-5986m_kube-system(9801ab59-a02b-461a-b8eb-293863bdf7da)", failed to "StartContainer" for "weave-npc" with CrashLoopBackOff: "back-off 1m20s restarting failed container=weave-npc pod=weave-net-5986m_kube-system(9801ab59-a02b-461a-b8eb-293863bdf7da)"] Aug 09 16:41:51 cube-node1 kubelet[703]: E0809 16:41:51.472341 703 kubelet.go:2100] Container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:docker: network plugin is not ready: cni config uninitialized