Upgrade from Stable 2605.8.0 to 2605.9.0 breaks Kubernetes installed with RKE

[Crazyigor1987]

Our Kubernetes isn't available since the autoupdate on 07.12.2020 from stable channel. I am not quite sure where exactly the problem is, but it looks like the internal network communication between the pods aren't working anymore. We've installed Kubernetes 1.16 with rke on bare metal. The dmesg on every nodes is full with those messages.

[ 4020.042233] SELinux: mount invalid. Same superblock, different security settings for (dev mqueue, type mqueue) [ 4040.766546] SELinux: mount invalid. Same superblock, different security settings for (dev mqueue, type mqueue) [ 4061.421123] SELinux: mount invalid. Same superblock, different security settings for (dev mqueue, type mqueue) [ 4082.028689] SELinux: mount invalid. Same superblock, different security settings for (dev mqueue, type mqueue) [ 4102.974200] SELinux: mount invalid. Same superblock, different security settings for (dev mqueue, type mqueue) [ 4123.629207] SELinux: mount invalid. Same superblock, different security settings for (dev mqueue, type mqueue) [ 4144.201674] SELinux: mount invalid. Same superblock, different security settings for (dev mqueue, type mqueue) [ 4165.058893] SELinux: mount invalid. Same superblock, different security settings for (dev mqueue, type mqueue)

The kube-api is not able to start up, because it has no route to etcd, even on the same Node. Kubeproxy does not create any Routing entries, because it cannot communicate with kube-api

E1208 13:45:32.719863 1159 reflector.go:123] k8s.io/client-go/informers/factory.go:134: Failed to list v1.Service: Get https://127.0.0.1:6443/api/v1/services?labelSelector=%21service.kubernetes.io%2Fheadless%2C%21service.kubernetes.io%2Fservice-proxy-name&limit=500&resourceVersion=0: net/http: TLS handshake timeout E1208 13:45:32.719834 1159 reflector.go:123] k8s.io/client-go/informers/factory.go:134: Failed to list v1.Endpoints: Get https://127.0.0.1:6443/api/v1/endpoints?labelSelector=%21service.kubernetes.io%2Fheadless%2C%21service.kubernetes.io%2Fservice-proxy-name&limit=500&resourceVersion=0: net/http: TLS handshake timeout I1208 13:45:53.277972 1159 trace.go:116] Trace[495832311]: "Reflector ListAndWatch" name:k8s.io/client-go/informers/factory.go:134 (started: 2020-12-08 13:45:33.720020238 +0000 UTC m=+5027.597767847) (total time: 19.557865314s): Trace[495832311]: [19.557865314s] [19.557865314s] END E1208 13:45:53.278237 1159 reflector.go:123] k8s.io/client-go/informers/factory.go:134: Failed to list v1.Service: Get https://127.0.0.1:6443/api/v1/services?labelSelector=%21service.kubernetes.io%2Fheadless%2C%21service.kubernetes.io%2Fservice-proxy-name&limit=500&resourceVersion=0: net/http: TLS handshake timeout I1208 13:45:53.278078 1159 trace.go:116] Trace[1028377127]: "Reflector ListAndWatch" name:k8s.io/client-go/informers/factory.go:134 (started: 2020-12-08 13:45:33.721666243 +0000 UTC m=+5027.599413877) (total time: 19.55635865s): Trace[1028377127]: [19.55635865s] [19.55635865s] END E1208 13:45:53.278859 1159 reflector.go:123] k8s.io/client-go/informers/factory.go:134: Failed to list v1.Endpoints: Get https://127.0.0.1:6443/api/v1/endpoints?labelSelector=%21service.kubernetes.io%2Fheadless%2C%21service.kubernetes.io%2Fservice-proxy-name&limit=500&resourceVersion=0: net/http: TLS handshake timeout I1208 13:45:54.443382 1159 proxier.go:678] Not syncing iptables until Services and Endpoints have been received from master I1208 13:46:13.744119 1159 trace.go:116] Trace[1362926035]: "Reflector ListAndWatch" name:k8s.io/client-go/informers/factory.go:134 (started: 2020-12-08 13:45:54.28044788 +0000 UTC m=+5048.158195480) (total time: 19.463615684s): Trace[1362926035]: [19.463615684s] [19.463615684s] END E1208 13:46:13.744167 1159 reflector.go:123] k8s.io/client-go/informers/factory.go:134: Failed to list v1.Endpoints: Get https://127.0.0.1:6443/api/v1/endpoints?labelSelector=%21service.kubernetes.io%2Fheadless%2C%21service.kubernetes.io%2Fservice-proxy-name&limit=500&resourceVersion=0: net/http: TLS handshake timeout I1208 13:46:13.744432 1159 trace.go:116] Trace[262738453]: "Reflector ListAndWatch" name:k8s.io/client-go/informers/factory.go:134 (started: 2020-12-08 13:45:54.278860726 +0000 UTC m=+5048.156608337) (total time: 19.465520053s): Trace[262738453]: [19.465520053s] [19.465520053s] END E1208 13:46:13.744466 1159 reflector.go:123] k8s.io/client-go/informers/factory.go:134: Failed to list v1.Service: Get https://127.0.0.1:6443/api/v1/services?labelSelector=%21service.kubernetes.io%2Fheadless%2C%21service.kubernetes.io%2Fservice-proxy-name&limit=500&resourceVersion=0: net/http: TLS handshake timeout I1208 13:46:24.444098 1159 proxier.go:678] Not syncing iptables until Services and Endpoints have been received from master I1208 13:46:34.173472 1159 trace.go:116] Trace[1906531757]: "Reflector ListAndWatch" name:k8s.io/client-go/informers/factory.go:134 (started: 2020-12-08 13:46:14.745949535 +0000 UTC m=+5068.623697178) (total time: 19.427484779s): Trace[1906531757]: [19.427484779s] [19.427484779s] END I1208 13:46:34.173735 1159 trace.go:116] Trace[652495813]: "Reflector ListAndWatch" name:k8s.io/client-go/informers/factory.go:134 (started: 2020-12-08 13:46:14.74462498 +0000 UTC m=+5068.622372651) (total time: 19.429076752s): Trace[652495813]: [19.429076752s] [19.429076752s] END E1208 13:46:34.174084 1159 reflector.go:123] k8s.io/client-go/informers/factory.go:134: Failed to list v1.Endpoints: Get https://127.0.0.1:6443/api/v1/endpoints?labelSelector=%21service.kubernetes.io%2Fheadless%2C%21service.kubernetes.io%2Fservice-proxy-name&limit=500&resourceVersion=0: net/http: TLS handshake timeout E1208 13:46:34.174447 1159 reflector.go:123] k8s.io/client-go/informers/factory.go:134: Failed to list v1.Service: Get https://127.0.0.1:6443/api/v1/services?labelSelector=%21service.kubernetes.io%2Fheadless%2C%21service.kubernetes.io%2Fservice-proxy-name&limit=500&resourceVersion=0: net/http: TLS handshake timeout

Environment and steps to reproduce

Using latest stable flatcar container linux with rke installed Kubernetes Cluster rke version INFO[0000] Running RKE version: v1.2.3

rke-config.yml ` kubernetes_version: "v1.16.13-rancher1-2" nodes:

• address: imbserv4-master1 internal_address: imbserv4-master1 user: core role: [controlplane,etcd]
• address: imbserv5-master2 internal_address: imbserv5-master2 user: core role: [controlplane,etcd]
• address: imbserv6-master3 internal_address: imbserv6-master3 user: core role: [controlplane,etcd]
• address: imbserv1-ingress1 internal_address: imbserv1-ingress1 user: core role: [worker] labels: publichost: public1 edgelb: true
• address: worker-00000000-0000-0000-0000-ac1f6b4e3f4a internal_address: worker-00000000-0000-0000-0000-ac1f6b4e3f4a user: core role: [worker]
• address: worker-00000000-0000-0000-0000-ac1f6b4e40b6 internal_address: worker-00000000-0000-0000-0000-ac1f6b4e40b6 user: core role: [worker]
• address: worker-00000000-0000-0000-0000-ac1f6bbadfbe internal_address: worker-00000000-0000-0000-0000-ac1f6bbadfbe user: core role: [worker]
• address: worker-00000000-0000-0000-0000-ac1f6b4e40a4 internal_address: worker-00000000-0000-0000-0000-ac1f6b4e40a4 user: core role: [worker]
• address: worker-00000000-0000-0000-0000-ac1f6b4e3db6 internal_address: worker-00000000-0000-0000-0000-ac1f6b4e3db6 user: core role: [worker]
• address: node1-storage internal_address: node1-storage user: core role: [worker] labels: storagehost: storage1 storage: true

services: etcd: snapshot: true creation: 6h retention: 24h kube-controller: extra_args: flex-volume-plugin-dir: /opt/kubernetes/kubelet-plugins/volume/exec/

network: plugin: canal options: canal_flex_volume_plugin_dir: /opt/kubernetes/kubelet-plugins/volume/exec/nodeagent~uds flannel_backend_type: vxlan

ingress: provider: none

`

[t-lo]

Hello @Crazyigor1987 , thanks for the report, and sorry for the bad user experience. The latest Stable release contains a number of user space changes to address security issues. I understand this is a big ask, but would you be able to re-provision the nodes (instead of updating) and check if the problem persists?

[pothos]

If it's a networking problem, something similar is reported here https://github.com/kinvolk/Flatcar/issues/278 I would try to do debug the connectivity issue by altering some of the settings mentioned in the linked issue.

[Crazyigor1987]

Hello @Crazyigor1987 , thanks for the report, and sorry for the bad user experience. The latest Stable release contains a number of user space changes to address security issues. I understand this is a big ask, but would you be able to re-provision the nodes (instead of updating) and check if the problem persists?

Hi @t-lo ,

done it, getting worse. Now the docker containers cannot start up. Here is my journalctl output, when i perform rke up.

Dec 09 08:46:38 imbserv4-master1 systemd-logind[789]: [🡕] Removed session 47.
Dec 09 08:46:38 imbserv4-master1 systemd-logind[789]: [🡕] Removed session 48.
Dec 09 08:46:38 imbserv4-master1 systemd-logind[789]: [🡕] Removed session 52.
Dec 09 08:46:38 imbserv4-master1 systemd-logind[789]: [🡕] Removed session 46.
Dec 09 08:46:38 imbserv4-master1 systemd-logind[789]: [🡕] Removed session 51.
Dec 09 08:46:38 imbserv4-master1 systemd-logind[789]: [🡕] Removed session 41.
Dec 09 08:46:38 imbserv4-master1 systemd-logind[789]: [🡕] Removed session 42.
Dec 09 08:46:38 imbserv4-master1 systemd-logind[789]: [🡕] Removed session 50.
Dec 09 08:46:38 imbserv4-master1 systemd-logind[789]: [🡕] Removed session 43.
Dec 09 08:46:38 imbserv4-master1 systemd-logind[789]: [🡕] Removed session 49.
Dec 09 08:48:26 imbserv4-master1 systemd[1]: /usr/lib/systemd/system/sshd@.service:8: Standard output type syslog is obsolete, automatically updating to journal. Please update your unit file, and consider removing the setting altogether.
Dec 09 08:48:26 imbserv4-master1 systemd[1]: Started OpenSSH per-connection server daemon (10.5.1.155:49606).
Dec 09 08:48:26 imbserv4-master1 sshd[4457]: Accepted publickey for core from 10.5.1.155 port 49606 ssh2: RSA SHA256:UAp3UG8jkM9zVmDXcRsGlj0F0yT+++J9VlC1L1EL51c
Dec 09 08:48:26 imbserv4-master1 sshd[4457]: pam_unix(sshd:session): session opened for user core by (uid=0)
Dec 09 08:48:26 imbserv4-master1 systemd-logind[789]: [🡕] New session 53 of user core.
Dec 09 08:48:26 imbserv4-master1 systemd[1]: Started Session 53 of user core.
Dec 09 08:48:29 imbserv4-master1 systemd[1]: var-lib-docker-overlay2-0ed08e76f176399ae7b2c4b60e8722b673ef980f8cf21e40d20a5953d7dea5dd\x2dinit-merged.mount: Succeeded.
Dec 09 08:48:29 imbserv4-master1 systemd[3652]: var-lib-docker-overlay2-0ed08e76f176399ae7b2c4b60e8722b673ef980f8cf21e40d20a5953d7dea5dd\x2dinit-merged.mount: Succeeded.
Dec 09 08:48:29 imbserv4-master1 systemd[3652]: var-lib-docker-overlay2-0ed08e76f176399ae7b2c4b60e8722b673ef980f8cf21e40d20a5953d7dea5dd-merged.mount: Succeeded.
Dec 09 08:48:29 imbserv4-master1 systemd[1]: var-lib-docker-overlay2-0ed08e76f176399ae7b2c4b60e8722b673ef980f8cf21e40d20a5953d7dea5dd-merged.mount: Succeeded.
Dec 09 08:48:29 imbserv4-master1 systemd-udevd[4468]: ethtool: autonegotiation is unset or enabled, the speed and duplex are not writable.
Dec 09 08:48:29 imbserv4-master1 systemd-udevd[4469]: ethtool: autonegotiation is unset or enabled, the speed and duplex are not writable.
Dec 09 08:48:29 imbserv4-master1 systemd-udevd[4468]: Using default interface naming scheme 'v245'.
Dec 09 08:48:29 imbserv4-master1 systemd-udevd[4469]: Using default interface naming scheme 'v245'.
Dec 09 08:48:29 imbserv4-master1 kernel: docker0: port 1(veth11e8c71) entered blocking state
Dec 09 08:48:29 imbserv4-master1 kernel: docker0: port 1(veth11e8c71) entered disabled state
Dec 09 08:48:29 imbserv4-master1 kernel: device veth11e8c71 entered promiscuous mode
Dec 09 08:48:29 imbserv4-master1 systemd-networkd[685]: veth11e8c71: Link UP
Dec 09 08:48:29 imbserv4-master1 env[807]: time="2020-12-09T08:48:29.935655721Z" level=warning msg="runtime v1 is deprecated since containerd v1.4, consider using runtime v2"
Dec 09 08:48:29 imbserv4-master1 env[807]: time="2020-12-09T08:48:29.936809505Z" level=info msg="shim containerd-shim started" address="unix:///run/containerd/s/3c44085eb0441d58e099f84cf67ca92f436d5766971d14e0d823e73115d69229" debug=true pid=4473
Dec 09 08:48:29 imbserv4-master1 env[807]: time="2020-12-09T08:48:29Z" level=debug msg="registering ttrpc server"
Dec 09 08:48:29 imbserv4-master1 env[807]: time="2020-12-09T08:48:29Z" level=debug msg="serving api on unix socket" socket="[inherited from parent]"
Dec 09 08:48:29 imbserv4-master1 kernel: SELinux: mount invalid.  Same superblock, different security settings for (dev mqueue, type mqueue)
Dec 09 08:48:30 imbserv4-master1 kernel: eth0: renamed from veth81b6e4e
Dec 09 08:48:30 imbserv4-master1 kernel: IPv6: ADDRCONF(NETDEV_CHANGE): veth11e8c71: link becomes ready
Dec 09 08:48:30 imbserv4-master1 kernel: docker0: port 1(veth11e8c71) entered blocking state
Dec 09 08:48:30 imbserv4-master1 kernel: docker0: port 1(veth11e8c71) entered forwarding state
Dec 09 08:48:30 imbserv4-master1 systemd-networkd[685]: veth11e8c71: Gained carrier
Dec 09 08:48:30 imbserv4-master1 systemd-networkd[685]: docker0: Gained carrier
Dec 09 08:48:30 imbserv4-master1 env[807]: time="2020-12-09T08:48:30Z" level=debug msg="process with id:4481 wasn't found" namespace=moby path=/run/docker/libcontainerd/containerd/io.containerd.runtime.v1.linux/moby/b15f00b0602c1e6873d11f128676c96def5a910a3b04d1cd5f6b5f7ae3e61aa7 pid=4473
Dec 09 08:48:30 imbserv4-master1 env[807]: time="2020-12-09T08:48:30Z" level=debug msg="process with id:4509 wasn't found" namespace=moby path=/run/docker/libcontainerd/containerd/io.containerd.runtime.v1.linux/moby/b15f00b0602c1e6873d11f128676c96def5a910a3b04d1cd5f6b5f7ae3e61aa7 pid=4473
Dec 09 08:48:30 imbserv4-master1 env[807]: time="2020-12-09T08:48:30Z" level=debug msg="process with id:4847 wasn't found" namespace=moby path=/run/docker/libcontainerd/containerd/io.containerd.runtime.v1.linux/moby/b15f00b0602c1e6873d11f128676c96def5a910a3b04d1cd5f6b5f7ae3e61aa7 pid=4473
Dec 09 08:48:30 imbserv4-master1 env[807]: time="2020-12-09T08:48:30Z" level=debug msg="process with id:4853 wasn't found" namespace=moby path=/run/docker/libcontainerd/containerd/io.containerd.runtime.v1.linux/moby/b15f00b0602c1e6873d11f128676c96def5a910a3b04d1cd5f6b5f7ae3e61aa7 pid=4473
Dec 09 08:48:30 imbserv4-master1 env[807]: time="2020-12-09T08:48:30.210437902Z" level=info msg="shim reaped" id=b15f00b0602c1e6873d11f128676c96def5a910a3b04d1cd5f6b5f7ae3e61aa7
Dec 09 08:48:30 imbserv4-master1 env[853]: time="2020-12-09T08:48:30.220873607Z" level=info msg="ignoring event" module=libcontainerd namespace=moby topic=/tasks/delete type="*events.TaskDelete"
Dec 09 08:48:30 imbserv4-master1 systemd-networkd[685]: veth11e8c71: Lost carrier
Dec 09 08:48:30 imbserv4-master1 kernel: docker0: port 1(veth11e8c71) entered disabled state
Dec 09 08:48:30 imbserv4-master1 kernel: veth81b6e4e: renamed from eth0
Dec 09 08:48:30 imbserv4-master1 systemd-udevd[4468]: ethtool: autonegotiation is unset or enabled, the speed and duplex are not writable.
Dec 09 08:48:30 imbserv4-master1 systemd-networkd[685]: veth11e8c71: Link DOWN
Dec 09 08:48:30 imbserv4-master1 kernel: docker0: port 1(veth11e8c71) entered disabled state
Dec 09 08:48:30 imbserv4-master1 kernel: device veth11e8c71 left promiscuous mode
Dec 09 08:48:30 imbserv4-master1 kernel: docker0: port 1(veth11e8c71) entered disabled state
Dec 09 08:48:30 imbserv4-master1 systemd-networkd[685]: rtnl: received neighbor for link '33' we don't know about, ignoring.
Dec 09 08:48:30 imbserv4-master1 systemd-networkd[685]: rtnl: received neighbor for link '33' we don't know about, ignoring.
Dec 09 08:48:30 imbserv4-master1 systemd-networkd[685]: veth81b6e4e: Failed to wait for the interface to be initialized: No such device
Dec 09 08:48:30 imbserv4-master1 systemd[3652]: run-docker-netns-91a847ef59f9.mount: Succeeded.
Dec 09 08:48:30 imbserv4-master1 systemd[1]: run-docker-netns-91a847ef59f9.mount: Succeeded.
Dec 09 08:48:30 imbserv4-master1 systemd[3652]: var-lib-docker-containers-b15f00b0602c1e6873d11f128676c96def5a910a3b04d1cd5f6b5f7ae3e61aa7-mounts-shm.mount: Succeeded.
Dec 09 08:48:30 imbserv4-master1 systemd[3652]: var-lib-docker-overlay2-0ed08e76f176399ae7b2c4b60e8722b673ef980f8cf21e40d20a5953d7dea5dd-merged.mount: Succeeded.
Dec 09 08:48:30 imbserv4-master1 systemd[1]: var-lib-docker-containers-b15f00b0602c1e6873d11f128676c96def5a910a3b04d1cd5f6b5f7ae3e61aa7-mounts-shm.mount: Succeeded.
Dec 09 08:48:30 imbserv4-master1 systemd[1]: var-lib-docker-overlay2-0ed08e76f176399ae7b2c4b60e8722b673ef980f8cf21e40d20a5953d7dea5dd-merged.mount: Succeeded.
Dec 09 08:48:30 imbserv4-master1 systemd-networkd[685]: docker0: Lost carrier
Dec 09 08:48:35 imbserv4-master1 systemd[1]: var-lib-docker-overlay2-a4c09bb4c2837c1132a022b5288289e2eb57f4ef8d614ce13b9fc535addaba12\x2dinit-merged.mount: Succeeded.
Dec 09 08:48:35 imbserv4-master1 systemd[3652]: var-lib-docker-overlay2-a4c09bb4c2837c1132a022b5288289e2eb57f4ef8d614ce13b9fc535addaba12\x2dinit-merged.mount: Succeeded.
Dec 09 08:48:35 imbserv4-master1 systemd[3652]: var-lib-docker-overlay2-a4c09bb4c2837c1132a022b5288289e2eb57f4ef8d614ce13b9fc535addaba12-merged.mount: Succeeded.
Dec 09 08:48:35 imbserv4-master1 systemd[1]: var-lib-docker-overlay2-a4c09bb4c2837c1132a022b5288289e2eb57f4ef8d614ce13b9fc535addaba12-merged.mount: Succeeded.
Dec 09 08:48:35 imbserv4-master1 kernel: docker0: port 1(veth7036f46) entered blocking state
Dec 09 08:48:35 imbserv4-master1 kernel: docker0: port 1(veth7036f46) entered disabled state
Dec 09 08:48:35 imbserv4-master1 kernel: device veth7036f46 entered promiscuous mode
Dec 09 08:48:35 imbserv4-master1 systemd-udevd[4860]: ethtool: autonegotiation is unset or enabled, the speed and duplex are not writable.
Dec 09 08:48:35 imbserv4-master1 systemd-udevd[4861]: ethtool: autonegotiation is unset or enabled, the speed and duplex are not writable.
Dec 09 08:48:35 imbserv4-master1 systemd-udevd[4861]: Using default interface naming scheme 'v245'.
Dec 09 08:48:35 imbserv4-master1 systemd-udevd[4860]: Using default interface naming scheme 'v245'.
Dec 09 08:48:35 imbserv4-master1 systemd-networkd[685]: veth7036f46: Link UP
Dec 09 08:48:35 imbserv4-master1 env[807]: time="2020-12-09T08:48:35.422468586Z" level=warning msg="runtime v1 is deprecated since containerd v1.4, consider using runtime v2"
Dec 09 08:48:35 imbserv4-master1 env[807]: time="2020-12-09T08:48:35.422895888Z" level=info msg="shim containerd-shim started" address="unix:///run/containerd/s/46e8a0a66cb9ded5610cdd49b833865f4d87e5860625cb71fa5c75fb811d901d" debug=true pid=4865
Dec 09 08:48:35 imbserv4-master1 env[807]: time="2020-12-09T08:48:35Z" level=debug msg="registering ttrpc server"
Dec 09 08:48:35 imbserv4-master1 env[807]: time="2020-12-09T08:48:35Z" level=debug msg="serving api on unix socket" socket="[inherited from parent]"
Dec 09 08:48:35 imbserv4-master1 kernel: SELinux: mount invalid.  Same superblock, different security settings for (dev mqueue, type mqueue)
Dec 09 08:48:35 imbserv4-master1 kernel: eth0: renamed from vethc8cf0c2
Dec 09 08:48:35 imbserv4-master1 kernel: IPv6: ADDRCONF(NETDEV_CHANGE): veth7036f46: link becomes ready
Dec 09 08:48:35 imbserv4-master1 kernel: docker0: port 1(veth7036f46) entered blocking state
Dec 09 08:48:35 imbserv4-master1 kernel: docker0: port 1(veth7036f46) entered forwarding state
Dec 09 08:48:35 imbserv4-master1 systemd-networkd[685]: veth7036f46: Gained carrier
Dec 09 08:48:35 imbserv4-master1 systemd-networkd[685]: docker0: Gained carrier
Dec 09 08:48:35 imbserv4-master1 env[807]: time="2020-12-09T08:48:35Z" level=debug msg="process with id:4872 wasn't found" namespace=moby path=/run/docker/libcontainerd/containerd/io.containerd.runtime.v1.linux/moby/5e80e0142359c674665c15a56d74bfec7c9acc7a161f0a5cf91aa41e841f944b pid=4865
Dec 09 08:48:35 imbserv4-master1 env[807]: time="2020-12-09T08:48:35Z" level=debug msg="process with id:4902 wasn't found" namespace=moby path=/run/docker/libcontainerd/containerd/io.containerd.runtime.v1.linux/moby/5e80e0142359c674665c15a56d74bfec7c9acc7a161f0a5cf91aa41e841f944b pid=4865
Dec 09 08:48:35 imbserv4-master1 env[807]: time="2020-12-09T08:48:35Z" level=debug msg="process with id:4912 wasn't found" namespace=moby path=/run/docker/libcontainerd/containerd/io.containerd.runtime.v1.linux/moby/5e80e0142359c674665c15a56d74bfec7c9acc7a161f0a5cf91aa41e841f944b pid=4865
Dec 09 08:48:35 imbserv4-master1 env[807]: time="2020-12-09T08:48:35Z" level=debug msg="process with id:4918 wasn't found" namespace=moby path=/run/docker/libcontainerd/containerd/io.containerd.runtime.v1.linux/moby/5e80e0142359c674665c15a56d74bfec7c9acc7a161f0a5cf91aa41e841f944b pid=4865
Dec 09 08:48:35 imbserv4-master1 env[807]: time="2020-12-09T08:48:35.618073522Z" level=info msg="shim reaped" id=5e80e0142359c674665c15a56d74bfec7c9acc7a161f0a5cf91aa41e841f944b
Dec 09 08:48:35 imbserv4-master1 env[853]: time="2020-12-09T08:48:35.628299704Z" level=info msg="ignoring event" module=libcontainerd namespace=moby topic=/tasks/delete type="*events.TaskDelete"
Dec 09 08:48:35 imbserv4-master1 kernel: vethc8cf0c2: renamed from eth0
Dec 09 08:48:35 imbserv4-master1 systemd-networkd[685]: veth7036f46: Lost carrier
Dec 09 08:48:35 imbserv4-master1 kernel: docker0: port 1(veth7036f46) entered disabled state
Dec 09 08:48:35 imbserv4-master1 systemd-udevd[4864]: ethtool: autonegotiation is unset or enabled, the speed and duplex are not writable.
Dec 09 08:48:35 imbserv4-master1 systemd-udevd[4864]: Using default interface naming scheme 'v245'.
Dec 09 08:48:35 imbserv4-master1 systemd-networkd[685]: veth7036f46: Link DOWN
Dec 09 08:48:35 imbserv4-master1 kernel: docker0: port 1(veth7036f46) entered disabled state
Dec 09 08:48:35 imbserv4-master1 kernel: device veth7036f46 left promiscuous mode
Dec 09 08:48:35 imbserv4-master1 kernel: docker0: port 1(veth7036f46) entered disabled state
Dec 09 08:48:35 imbserv4-master1 systemd-networkd[685]: rtnl: received neighbor for link '35' we don't know about, ignoring.
Dec 09 08:48:35 imbserv4-master1 systemd-networkd[685]: rtnl: received neighbor for link '35' we don't know about, ignoring.
Dec 09 08:48:35 imbserv4-master1 systemd-networkd[685]: vethc8cf0c2: Failed to wait for the interface to be initialized: No such device
Dec 09 08:48:35 imbserv4-master1 systemd[3652]: run-docker-netns-aca45cedc05a.mount: Succeeded.
Dec 09 08:48:35 imbserv4-master1 systemd[1]: run-docker-netns-aca45cedc05a.mount: Succeeded.
Dec 09 08:48:35 imbserv4-master1 systemd[3652]: var-lib-docker-containers-5e80e0142359c674665c15a56d74bfec7c9acc7a161f0a5cf91aa41e841f944b-mounts-shm.mount: Succeeded.
Dec 09 08:48:35 imbserv4-master1 systemd[3652]: var-lib-docker-overlay2-a4c09bb4c2837c1132a022b5288289e2eb57f4ef8d614ce13b9fc535addaba12-merged.mount: Succeeded.
Dec 09 08:48:35 imbserv4-master1 systemd[1]: var-lib-docker-containers-5e80e0142359c674665c15a56d74bfec7c9acc7a161f0a5cf91aa41e841f944b-mounts-shm.mount: Succeeded.
Dec 09 08:48:35 imbserv4-master1 systemd[1]: var-lib-docker-overlay2-a4c09bb4c2837c1132a022b5288289e2eb57f4ef8d614ce13b9fc535addaba12-merged.mount: Succeeded.
Dec 09 08:48:36 imbserv4-master1 systemd-networkd[685]: docker0: Lost carrier
Dec 09 08:48:39 imbserv4-master1 systemd[1]: var-lib-docker-overlay2-cb90f4fe4c670eb25df9ce46e60b8720f4b1edada5d57f239b7b4b88f9be5af7\x2dinit-merged.mount: Succeeded.
Dec 09 08:48:39 imbserv4-master1 systemd[3652]: var-lib-docker-overlay2-cb90f4fe4c670eb25df9ce46e60b8720f4b1edada5d57f239b7b4b88f9be5af7\x2dinit-merged.mount: Succeeded.
Dec 09 08:48:39 imbserv4-master1 systemd[3652]: var-lib-docker-overlay2-cb90f4fe4c670eb25df9ce46e60b8720f4b1edada5d57f239b7b4b88f9be5af7-merged.mount: Succeeded.
Dec 09 08:48:39 imbserv4-master1 systemd[1]: var-lib-docker-overlay2-cb90f4fe4c670eb25df9ce46e60b8720f4b1edada5d57f239b7b4b88f9be5af7-merged.mount: Succeeded.
Dec 09 08:48:39 imbserv4-master1 systemd-udevd[4933]: ethtool: autonegotiation is unset or enabled, the speed and duplex are not writable.
Dec 09 08:48:39 imbserv4-master1 systemd-udevd[4933]: Using default interface naming scheme 'v245'.
Dec 09 08:48:39 imbserv4-master1 systemd-udevd[4934]: ethtool: autonegotiation is unset or enabled, the speed and duplex are not writable.
Dec 09 08:48:39 imbserv4-master1 systemd-udevd[4934]: Using default interface naming scheme 'v245'.
Dec 09 08:48:39 imbserv4-master1 kernel: docker0: port 1(veth1dc186b) entered blocking state
Dec 09 08:48:39 imbserv4-master1 kernel: docker0: port 1(veth1dc186b) entered disabled state
Dec 09 08:48:39 imbserv4-master1 kernel: device veth1dc186b entered promiscuous mode
Dec 09 08:48:39 imbserv4-master1 systemd-networkd[685]: veth1dc186b: Link UP
Dec 09 08:48:39 imbserv4-master1 env[807]: time="2020-12-09T08:48:39.331540628Z" level=warning msg="runtime v1 is deprecated since containerd v1.4, consider using runtime v2"
Dec 09 08:48:39 imbserv4-master1 env[807]: time="2020-12-09T08:48:39.331980154Z" level=info msg="shim containerd-shim started" address="unix:///run/containerd/s/51b5d4b9618e82d9c62b1abbd37d3940ad09e989831a8a5d16c2cca53829b45e" debug=true pid=4938
Dec 09 08:48:39 imbserv4-master1 env[807]: time="2020-12-09T08:48:39Z" level=debug msg="registering ttrpc server"
Dec 09 08:48:39 imbserv4-master1 env[807]: time="2020-12-09T08:48:39Z" level=debug msg="serving api on unix socket" socket="[inherited from parent]"
Dec 09 08:48:39 imbserv4-master1 kernel: SELinux: mount invalid.  Same superblock, different security settings for (dev mqueue, type mqueue)
Dec 09 08:48:39 imbserv4-master1 kernel: eth0: renamed from veth70efa72
Dec 09 08:48:39 imbserv4-master1 kernel: IPv6: ADDRCONF(NETDEV_CHANGE): veth1dc186b: link becomes ready
Dec 09 08:48:39 imbserv4-master1 kernel: docker0: port 1(veth1dc186b) entered blocking state
Dec 09 08:48:39 imbserv4-master1 kernel: docker0: port 1(veth1dc186b) entered forwarding state
Dec 09 08:48:39 imbserv4-master1 systemd-networkd[685]: veth1dc186b: Gained carrier
Dec 09 08:48:39 imbserv4-master1 systemd-networkd[685]: docker0: Gained carrier
Dec 09 08:48:39 imbserv4-master1 env[807]: time="2020-12-09T08:48:39Z" level=debug msg="process with id:4945 wasn't found" namespace=moby path=/run/docker/libcontainerd/containerd/io.containerd.runtime.v1.linux/moby/aaa5b32a020f3e5db801c5f0e98d1535b807599c9cc28e0b5538c9f0a4026df5 pid=4938
Dec 09 08:48:39 imbserv4-master1 env[807]: time="2020-12-09T08:48:39Z" level=debug msg="process with id:4975 wasn't found" namespace=moby path=/run/docker/libcontainerd/containerd/io.containerd.runtime.v1.linux/moby/aaa5b32a020f3e5db801c5f0e98d1535b807599c9cc28e0b5538c9f0a4026df5 pid=4938
Dec 09 08:48:39 imbserv4-master1 env[807]: time="2020-12-09T08:48:39Z" level=debug msg="process with id:4985 wasn't found" namespace=moby path=/run/docker/libcontainerd/containerd/io.containerd.runtime.v1.linux/moby/aaa5b32a020f3e5db801c5f0e98d1535b807599c9cc28e0b5538c9f0a4026df5 pid=4938
Dec 09 08:48:39 imbserv4-master1 env[807]: time="2020-12-09T08:48:39Z" level=debug msg="process with id:4991 wasn't found" namespace=moby path=/run/docker/libcontainerd/containerd/io.containerd.runtime.v1.linux/moby/aaa5b32a020f3e5db801c5f0e98d1535b807599c9cc28e0b5538c9f0a4026df5 pid=4938
Dec 09 08:48:39 imbserv4-master1 env[807]: time="2020-12-09T08:48:39.539494644Z" level=info msg="shim reaped" id=aaa5b32a020f3e5db801c5f0e98d1535b807599c9cc28e0b5538c9f0a4026df5
Dec 09 08:48:39 imbserv4-master1 env[853]: time="2020-12-09T08:48:39.549929035Z" level=info msg="ignoring event" module=libcontainerd namespace=moby topic=/tasks/delete type="*events.TaskDelete"
Dec 09 08:48:39 imbserv4-master1 kernel: veth70efa72: renamed from eth0
Dec 09 08:48:39 imbserv4-master1 kernel: docker0: port 1(veth1dc186b) entered disabled state
Dec 09 08:48:39 imbserv4-master1 systemd-networkd[685]: veth1dc186b: Lost carrier
Dec 09 08:48:39 imbserv4-master1 systemd-udevd[4933]: ethtool: autonegotiation is unset or enabled, the speed and duplex are not writable.
Dec 09 08:48:39 imbserv4-master1 systemd-networkd[685]: veth1dc186b: Link DOWN
Dec 09 08:48:39 imbserv4-master1 kernel: docker0: port 1(veth1dc186b) entered disabled state
Dec 09 08:48:39 imbserv4-master1 kernel: device veth1dc186b left promiscuous mode
Dec 09 08:48:39 imbserv4-master1 kernel: docker0: port 1(veth1dc186b) entered disabled state
Dec 09 08:48:39 imbserv4-master1 systemd-networkd[685]: rtnl: received neighbor for link '37' we don't know about, ignoring.
Dec 09 08:48:39 imbserv4-master1 systemd-networkd[685]: rtnl: received neighbor for link '37' we don't know about, ignoring.
Dec 09 08:48:39 imbserv4-master1 systemd-networkd[685]: veth70efa72: Failed to wait for the interface to be initialized: No such device
Dec 09 08:48:39 imbserv4-master1 systemd[1]: run-docker-netns-29a97bb762ed.mount: Succeeded.
Dec 09 08:48:39 imbserv4-master1 systemd[3652]: run-docker-netns-29a97bb762ed.mount: Succeeded.
Dec 09 08:48:39 imbserv4-master1 systemd[1]: var-lib-docker-containers-aaa5b32a020f3e5db801c5f0e98d1535b807599c9cc28e0b5538c9f0a4026df5-mounts-shm.mount: Succeeded.
Dec 09 08:48:39 imbserv4-master1 systemd[1]: var-lib-docker-overlay2-cb90f4fe4c670eb25df9ce46e60b8720f4b1edada5d57f239b7b4b88f9be5af7-merged.mount: Succeeded.
Dec 09 08:48:39 imbserv4-master1 systemd[3652]: var-lib-docker-containers-aaa5b32a020f3e5db801c5f0e98d1535b807599c9cc28e0b5538c9f0a4026df5-mounts-shm.mount: Succeeded.
Dec 09 08:48:39 imbserv4-master1 systemd[3652]: var-lib-docker-overlay2-cb90f4fe4c670eb25df9ce46e60b8720f4b1edada5d57f239b7b4b88f9be5af7-merged.mount: Succeeded.
Dec 09 08:48:40 imbserv4-master1 systemd-networkd[685]: docker0: Lost carrier
Dec 09 08:48:43 imbserv4-master1 systemd[1]: /usr/lib/systemd/system/sshd@.service:8: Standard output type syslog is obsolete, automatically updating to journal. Please update your unit file, and consider removing the setting altogether.
Dec 09 08:48:43 imbserv4-master1 systemd[1]: Started OpenSSH per-connection server daemon (10.5.1.155:49622).
Dec 09 08:48:43 imbserv4-master1 sshd[5007]: Accepted publickey for core from 10.5.1.155 port 49622 ssh2: RSA SHA256:UAp3UG8jkM9zVmDXcRsGlj0F0yT+++J9VlC1L1EL51c
Dec 09 08:48:43 imbserv4-master1 sshd[5007]: pam_unix(sshd:session): session opened for user core by (uid=0)
Dec 09 08:48:43 imbserv4-master1 systemd-logind[789]: [🡕] New session 54 of user core.
Dec 09 08:48:43 imbserv4-master1 systemd[1]: Started Session 54 of user core.
Dec 09 08:48:43 imbserv4-master1 sshd[5010]: connect_to localhost port 6443: failed.