search

clearcontainers / runtime

OCI (Open Containers Initiative) compatible runtime using Virtual Machines
Apache License 2.0
589 stars 70 forks source link

cc-runtime does not pass an IP to a container using CRI-O 1.9.8 and above #1069

Closed chavafg closed 6 years ago

chavafg commented 6 years ago

Description of problem

When trying to update the CRI-O version that we use for testing to v1.9.9, openshift and k8s tests failed.

not ok 1 Hello Openshift using CC
# (in test file hello_world.bats, line 51)
#   `curl "${container_ip}:${container_port}" &> "$output_file"' failed with status 7
# The connection to the server 10.88.0.4:8080 was refused - did you specify the right host or port?

Seems like the pod gets an IP address, but it is not passed to the container. 

fuentess@latest-crio:~/go/src/github.com/clearcontainers/tests/integration/kubernetes$ sudo -E kubectl describe pod test01-899db7777-gt8k4 | grep IP 
IP:             10.88.0.5

fuentess@latest-crio:~/go/src/github.com/clearcontainers/tests/integration/kubernetes$ ping 10.88.0.5
PING 10.88.0.5 (10.88.0.5) 56(84) bytes of data.
64 bytes from 10.88.0.5: icmp_seq=1 ttl=64 time=0.097 ms
64 bytes from 10.88.0.5: icmp_seq=2 ttl=64 time=0.059 ms

But when getting inside the container, it does not have an assigned IP. 

fuentess@latest-crio:~/go/src/github.com/clearcontainers/tests/integration/kubernetes$ sudo -E kubectl attach test01-899db7777-gt8k4 -c test01 -i -t 
If you don't see a command prompt, try pressing enter.
/ # ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: sit0@NONE: <NOARP> mtu 1480 qdisc noop qlen 1000
    link/sit 0.0.0.0 brd 0.0.0.0

This failures are reproducible using CRI-O 1.9.8, 1.9.9 and 1.9.10

Expected result

Using CRI-O 1.9.7, you get an IP inside the container. 

fuentess@latest-crio:~/go/src/github.com/clearcontainers/tests/integration/kubernetes$ sudo -E kubectl run test01 -ti --image busybox -- sh
If you don't see a command prompt, try pressing enter.
/ # ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc mq qlen 1000
    link/ether 0a:58:0a:f4:00:05 brd ff:ff:ff:ff:ff:ff
    inet 10.244.0.5/24 scope global eth0
       valid_lft forever preferred_lft forever
    inet6 fe80::858:aff:fef4:5/64 scope link 
       valid_lft forever preferred_lft forever
3: sit0@NONE: <NOARP> mtu 1480 qdisc noop qlen 1000
    link/sit 0.0.0.0 brd 0.0.0.0
/ # Session ended, resume using 'kubectl attach test01-899db7777-xck2s -c test01 -i -t' command when the pod is running
fuentess@latest-crio:~/go/src/github.com/clearcontainers/tests/integration/kubernetes$ crio --version
crio version 1.9.7
commit: "a98f9c902421a2d02b650f20a587b345a2ca17f1"

Running cc-collect-data.sh version 3.0.21 (commit b2b6837e16b793c604923e999b825ac18a9002e4-dirty) at 2018-03-13.15:11:58.328449322+0000.

Runtime is /usr/local/bin/cc-runtime.

cc-env

Output of "/usr/local/bin/cc-runtime cc-env":

[Meta]
  Version = "1.0.9"

[Runtime]
  Debug = false
  [Runtime.Version]
    Semver = "3.0.21"
    Commit = "b2b6837e16b793c604923e999b825ac18a9002e4-dirty"
    OCI = "1.0.1"
  [Runtime.Config]
    Path = "/usr/share/defaults/clear-containers/configuration.toml"

[Hypervisor]
  MachineType = "pc"
  Version = "QEMU emulator version 2.7.0, Copyright (c) 2003-2016 Fabrice Bellard and the QEMU Project developers"
  Path = "/usr/bin/qemu-lite-system-x86_64"
  Debug = false
  BlockDeviceDriver = "virtio-scsi"

[Image]
  Path = "/usr/share/clear-containers/cc-20640-agent-6f6e9e.img"

[Kernel]
  Path = "/usr/share/clear-containers/vmlinuz-4.14.22-86.container"
  Parameters = ""

[Proxy]
  Type = "ccProxy"
  Version = "Version: 3.0.21-2d500edc813034b4a59a57bad72d55a36cc2e0da"
  Path = "/usr/libexec/clear-containers/cc-proxy"
  Debug = true

[Shim]
  Type = "ccShim"
  Version = "shim version: 3.0.21 (commit: c8f732c99484e947f18cf29ab0d3d26e3e6bfbfe)"
  Path = "/usr/libexec/clear-containers/cc-shim"
  Debug = true

[Agent]
  Type = "hyperstart"
  Version = "<<unknown>>"

[Host]
  Kernel = "4.13.0-1011-azure"
  Architecture = "amd64"
  VMContainerCapable = true
  [Host.Distro]
    Name = "Ubuntu"
    Version = "16.04"
  [Host.CPU]
    Vendor = "GenuineIntel"
    Model = "Intel(R) Xeon(R) CPU E5-2673 v3 @ 2.40GHz"

Runtime config files

Runtime default config files

/etc/clear-containers/configuration.toml
/usr/share/defaults/clear-containers/configuration.toml

Runtime config file contents

Config file /etc/clear-containers/configuration.toml not found Output of "cat "/usr/share/defaults/clear-containers/configuration.toml"":

# XXX: WARNING: this file is auto-generated.
# XXX:
# XXX: Source file: "config/configuration.toml.in"
# XXX: Project:
# XXX:   Name: Intel® Clear Containers
# XXX:   Type: cc

[hypervisor.qemu]
path = "/usr/bin/qemu-lite-system-x86_64"
kernel = "/usr/share/clear-containers/vmlinuz.container"
image = "/usr/share/clear-containers/clear-containers.img"
machine_type = "pc"

# Optional space-separated list of options to pass to the guest kernel.
# For example, use `kernel_params = "vsyscall=emulate"` if you are having
# trouble running pre-2.15 glibc.
#
# WARNING: - any parameter specified here will take priority over the default
# parameter value of the same name used to start the virtual machine.
# Do not set values here unless you understand the impact of doing so as you
# may stop the virtual machine from booting.
# To see the list of default parameters, enable hypervisor debug, create a
# container and look for 'default-kernel-parameters' log entries.
kernel_params = ""

# Path to the firmware.
# If you want that qemu uses the default firmware leave this option empty
firmware = ""

# Machine accelerators
# comma-separated list of machine accelerators to pass to the hypervisor.
# For example, `machine_accelerators = "nosmm,nosmbus,nosata,nopit,static-prt,nofw"`
machine_accelerators=""

# Default number of vCPUs per POD/VM:
# unspecified or 0                --> will be set to 1
# < 0                             --> will be set to the actual number of physical cores
# > 0 <= number of physical cores --> will be set to the specified number
# > number of physical cores      --> will be set to the actual number of physical cores
default_vcpus = 1

# Bridges can be used to hot plug devices.
# Limitations:
# * Currently only pci bridges are supported
# * Until 30 devices per bridge can be hot plugged.
# * Until 5 PCI bridges can be cold plugged per VM.
#   This limitation could be a bug in qemu or in the kernel
# Default number of bridges per POD/VM:
# unspecified or 0   --> will be set to 1
# > 1 <= 5           --> will be set to the specified number
# > 5                --> will be set to 5
default_bridges = 1

# Default memory size in MiB for POD/VM.
# If unspecified then it will be set 2048 MiB.
#default_memory = 2048

# Disable block device from being used for a container's rootfs.
# In case of a storage driver like devicemapper where a container's 
# root file system is backed by a block device, the block device is passed
# directly to the hypervisor for performance reasons. 
# This flag prevents the block device from being passed to the hypervisor, 
# 9pfs is used instead to pass the rootfs.
disable_block_device_use = false

# Block storage driver to be used for the hypervisor in case the container
# rootfs is backed by a block device. This is either virtio-scsi or 
# virtio-blk.
block_device_driver = "virtio-scsi"

# Enable pre allocation of VM RAM, default false
# Enabling this will result in lower container density
# as all of the memory will be allocated and locked
# This is useful when you want to reserve all the memory
# upfront or in the cases where you want memory latencies
# to be very predictable
# Default false
#enable_mem_prealloc = true

# Enable huge pages for VM RAM, default false
# Enabling this will result in the VM memory
# being allocated using huge pages.
# This is useful when you want to use vhost-user network
# stacks within the container. This will automatically 
# result in memory pre allocation
#enable_hugepages = true

# Enable swap of vm memory. Default false.
# The behaviour is undefined if mem_prealloc is also set to true
#enable_swap = true

# This option changes the default hypervisor and kernel parameters
# to enable debug output where available. This extra output is added
# to the proxy logs, but only when proxy debug is also enabled.
# 
# Default false
enable_debug = true

# Disable the customizations done in the runtime when it detects
# that it is running on top a VMM. This will result in the runtime
# behaving as it would when running on bare metal.
# 
#disable_nesting_checks = true

[proxy.cc]
path = "/usr/libexec/clear-containers/cc-proxy"

# If enabled, proxy messages will be sent to the system log
# (default: disabled)
enable_debug = true

[shim.cc]
path = "/usr/libexec/clear-containers/cc-shim"

# If enabled, shim messages will be sent to the system log
# (default: disabled)
enable_debug = true

[agent.cc]
# There is no field for this section. The goal is only to be able to
# specify which type of agent the user wants to use.

[runtime]
# If enabled, the runtime will log additional debug messages to the
# system log
# (default: disabled)
enable_debug = true
#
# Internetworking model
# Determines how the VM should be connected to the
# the container network interface
# Options:
#
#   - bridged
#     Uses a linux bridge to interconnect the container interface to
#     the VM. Works for most cases except macvlan and ipvlan.
#
#   - macvtap
#     Used when the Container network interface can be bridged using
#     macvtap.
internetworking_model="macvtap"

Logfiles

Runtime logs

Recent runtime problems found in system journal:

time="2018-03-13T14:26:38.051039591Z" level=debug msg="Could not retrieve anything from storage" arch=amd64 name=cc-runtime pid=57048 source=virtcontainers subsystem=hyper
time="2018-03-13T14:26:38.057331739Z" level=debug arch=amd64 default-kernel-parameters="root=/dev/pmem0p1 rootflags=dax,data=ordered,errors=remount-ro rw rootfstype=ext4 tsc=reliable no_timer_check rcupdate.rcu_
expedited=1 i8042.direct=1 i8042.dumbkbd=1 i8042.nopnp=1 i8042.noaux=1 noreplace-smp reboot=k console=hvc0 console=hvc1 iommu=off cryptomgr.notests net.ifnames=0 pci=lastbus=0 debug systemd.show_status=true syst
emd.log_level=debug" name=cc-runtime pid=57048 source=virtcontainers subsystem=qemu
time="2018-03-13T14:45:20.07012642Z" level=debug msg="Could not retrieve anything from storage" arch=amd64 name=cc-runtime pid=60701 source=virtcontainers subsystem=hyper
time="2018-03-13T14:45:20.072820146Z" level=debug arch=amd64 default-kernel-parameters="root=/dev/pmem0p1 rootflags=dax,data=ordered,errors=remount-ro rw rootfstype=ext4 tsc=reliable no_timer_check rcupdate.rcu_
expedited=1 i8042.direct=1 i8042.dumbkbd=1 i8042.nopnp=1 i8042.noaux=1 noreplace-smp reboot=k console=hvc0 console=hvc1 iommu=off cryptomgr.notests net.ifnames=0 pci=lastbus=0 debug systemd.show_status=true syst
emd.log_level=debug" name=cc-runtime pid=60701 source=virtcontainers subsystem=qemu
time="2018-03-13T14:45:20.203531371Z" level=debug msg="Could not retrieve anything from storage" arch=amd64 name=cc-runtime pid=60752 source=virtcontainers subsystem=hyper
time="2018-03-13T14:45:20.211538946Z" level=debug arch=amd64 default-kernel-parameters="root=/dev/pmem0p1 rootflags=dax,data=ordered,errors=remount-ro rw rootfstype=ext4 tsc=reliable no_timer_check rcupdate.rcu_
expedited=1 i8042.direct=1 i8042.dumbkbd=1 i8042.nopnp=1 i8042.noaux=1 noreplace-smp reboot=k console=hvc0 console=hvc1 iommu=off cryptomgr.notests net.ifnames=0 pci=lastbus=0 debug systemd.show_status=true syst
emd.log_level=debug" name=cc-runtime pid=60752 source=virtcontainers subsystem=qemu
time="2018-03-13T14:47:57.79850143Z" level=debug msg="Could not retrieve anything from storage" arch=amd64 name=cc-runtime pid=61417 source=virtcontainers subsystem=hyper
time="2018-03-13T14:47:57.801782859Z" level=debug arch=amd64 default-kernel-parameters="root=/dev/pmem0p1 rootflags=dax,data=ordered,errors=remount-ro rw rootfstype=ext4 tsc=reliable no_timer_check rcupdate.rcu_
expedited=1 i8042.direct=1 i8042.dumbkbd=1 i8042.nopnp=1 i8042.noaux=1 noreplace-smp reboot=k console=hvc0 console=hvc1 iommu=off cryptomgr.notests net.ifnames=0 pci=lastbus=0 debug systemd.show_status=true syst
emd.log_level=debug" name=cc-runtime pid=61417 source=virtcontainers subsystem=qemu
time="2018-03-13T15:10:11.219123663Z" level=warning msg="Retrying proxy connection" arch=amd64 attempt=12 error="dial unix /run/virtcontainers/pods/62e04086797ce38140a9124f2a4314def8e9589d15adafaf3d1802e3ad955ff
e/proxy.sock: connect: connection refused" name=cc-runtime pid=64251 proxy-address=/run/virtcontainers/pods/62e04086797ce38140a9124f2a4314def8e9589d15adafaf3d1802e3ad955ffe/proxy.sock proxy-network=unix remainin
g-time-secs=4.00 source=virtcontainers subsystem=hyper
time="2018-03-13T15:10:12.221584903Z" level=warning msg="Retrying proxy connection" arch=amd64 attempt=22 error="dial unix /run/virtcontainers/pods/62e04086797ce38140a9124f2a4314def8e9589d15adafaf3d1802e3ad955ff
e/proxy.sock: connect: connection refused" name=cc-runtime pid=64251 proxy-address=/run/virtcontainers/pods/62e04086797ce38140a9124f2a4314def8e9589d15adafaf3d1802e3ad955ffe/proxy.sock proxy-network=unix remainin
g-time-secs=3.00 source=virtcontainers subsystem=hyper
time="2018-03-13T15:10:13.223646328Z" level=warning msg="Retrying proxy connection" arch=amd64 attempt=32 error="dial unix /run/virtcontainers/pods/62e04086797ce38140a9124f2a4314def8e9589d15adafaf3d1802e3ad955ff
e/proxy.sock: connect: connection refused" name=cc-runtime pid=64251 proxy-address=/run/virtcontainers/pods/62e04086797ce38140a9124f2a4314def8e9589d15adafaf3d1802e3ad955ffe/proxy.sock proxy-network=unix remainin
g-time-secs=1.99 source=virtcontainers subsystem=hyper
time="2018-03-13T15:10:14.225786143Z" level=warning msg="Retrying proxy connection" arch=amd64 attempt=42 error="dial unix /run/virtcontainers/pods/62e04086797ce38140a9124f2a4314def8e9589d15adafaf3d1802e3ad955ff
e/proxy.sock: connect: connection refused" name=cc-runtime pid=64251 proxy-address=/run/virtcontainers/pods/62e04086797ce38140a9124f2a4314def8e9589d15adafaf3d1802e3ad955ffe/proxy.sock proxy-network=unix remainin
g-time-secs=0.99 source=virtcontainers subsystem=hyper
time="2018-03-13T15:10:15.228128852Z" level=error msg="failed to connect to proxy after 5s: dial unix /run/virtcontainers/pods/62e04086797ce38140a9124f2a4314def8e9589d15adafaf3d1802e3ad955ffe/proxy.sock: connect
: connection refused" command=delete name=cc-runtime pid=64251 source=runtime
time="2018-03-13T15:10:16.256152711Z" level=warning msg="Retrying proxy connection" arch=amd64 attempt=12 error="dial unix /run/virtcontainers/pods/c5e9e10f01644d249b7b1cf957c90995ef7871249ffcc0e9ec34a6f507ef3c6
3/proxy.sock: connect: connection refused" name=cc-runtime pid=64263 proxy-address=/run/virtcontainers/pods/c5e9e10f01644d249b7b1cf957c90995ef7871249ffcc0e9ec34a6f507ef3c63/proxy.sock proxy-network=unix remainin
g-time-secs=4.00 source=virtcontainers subsystem=hyper
time="2018-03-13T15:10:17.258414197Z" level=warning msg="Retrying proxy connection" arch=amd64 attempt=22 error="dial unix /run/virtcontainers/pods/c5e9e10f01644d249b7b1cf957c90995ef7871249ffcc0e9ec34a6f507ef3c6
3/proxy.sock: connect: connection refused" name=cc-runtime pid=64263 proxy-address=/run/virtcontainers/pods/c5e9e10f01644d249b7b1cf957c90995ef7871249ffcc0e9ec34a6f507ef3c63/proxy.sock proxy-network=unix remainin
g-time-secs=3.00 source=virtcontainers subsystem=hyper
time="2018-03-13T15:10:18.260704074Z" level=warning msg="Retrying proxy connection" arch=amd64 attempt=32 error="dial unix /run/virtcontainers/pods/c5e9e10f01644d249b7b1cf957c90995ef7871249ffcc0e9ec34a6f507ef3c6
3/proxy.sock: connect: connection refused" name=cc-runtime pid=64263 proxy-address=/run/virtcontainers/pods/c5e9e10f01644d249b7b1cf957c90995ef7871249ffcc0e9ec34a6f507ef3c63/proxy.sock proxy-network=unix remainin
g-time-secs=1.99 source=virtcontainers subsystem=hyper
time="2018-03-13T15:10:19.262680837Z" level=warning msg="Retrying proxy connection" arch=amd64 attempt=42 error="dial unix /run/virtcontainers/pods/c5e9e10f01644d249b7b1cf957c90995ef7871249ffcc0e9ec34a6f507ef3c6
3/proxy.sock: connect: connection refused" name=cc-runtime pid=64263 proxy-address=/run/virtcontainers/pods/c5e9e10f01644d249b7b1cf957c90995ef7871249ffcc0e9ec34a6f507ef3c63/proxy.sock proxy-network=unix remainin
g-time-secs=0.99 source=virtcontainers subsystem=hyper
time="2018-03-13T15:10:20.264640189Z" level=error msg="failed to connect to proxy after 5s: dial unix /run/virtcontainers/pods/c5e9e10f01644d249b7b1cf957c90995ef7871249ffcc0e9ec34a6f507ef3c63/proxy.sock: connect
: connection refused" command=delete name=cc-runtime pid=64263 source=runtime
time="2018-03-13T15:11:21.29239634Z" level=warning msg="Retrying proxy connection" arch=amd64 attempt=12 error="dial unix /run/virtcontainers/pods/7d162da51d56b7ae2a9a75936b079b5c4854a25fb1fc1ec06b75b6ae8b64ea55
/proxy.sock: connect: connection refused" name=cc-runtime pid=64649 proxy-address=/run/virtcontainers/pods/7d162da51d56b7ae2a9a75936b079b5c4854a25fb1fc1ec06b75b6ae8b64ea55/proxy.sock proxy-network=unix remaining
-time-secs=4.00 source=virtcontainers subsystem=hyper
time="2018-03-13T15:11:22.294393109Z" level=warning msg="Retrying proxy connection" arch=amd64 attempt=22 error="dial unix /run/virtcontainers/pods/7d162da51d56b7ae2a9a75936b079b5c4854a25fb1fc1ec06b75b6ae8b64ea5
5/proxy.sock: connect: connection refused" name=cc-runtime pid=64649 proxy-address=/run/virtcontainers/pods/7d162da51d56b7ae2a9a75936b079b5c4854a25fb1fc1ec06b75b6ae8b64ea55/proxy.sock proxy-network=unix remainin
g-time-secs=3.00 source=virtcontainers subsystem=hyper
time="2018-03-13T15:11:23.296326668Z" level=warning msg="Retrying proxy connection" arch=amd64 attempt=32 error="dial unix /run/virtcontainers/pods/7d162da51d56b7ae2a9a75936b079b5c4854a25fb1fc1ec06b75b6ae8b64ea5
5/proxy.sock: connect: connection refused" name=cc-runtime pid=64649 proxy-address=/run/virtcontainers/pods/7d162da51d56b7ae2a9a75936b079b5c4854a25fb1fc1ec06b75b6ae8b64ea55/proxy.sock proxy-network=unix remainin
g-time-secs=1.99 source=virtcontainers subsystem=hyper
time="2018-03-13T15:11:24.298274719Z" level=warning msg="Retrying proxy connection" arch=amd64 attempt=42 error="dial unix /run/virtcontainers/pods/7d162da51d56b7ae2a9a75936b079b5c4854a25fb1fc1ec06b75b6ae8b64ea5
5/proxy.sock: connect: connection refused" name=cc-runtime pid=64649 proxy-address=/run/virtcontainers/pods/7d162da51d56b7ae2a9a75936b079b5c4854a25fb1fc1ec06b75b6ae8b64ea55/proxy.sock proxy-network=unix remainin
g-time-secs=0.99 source=virtcontainers subsystem=hyper
time="2018-03-13T15:11:25.300321362Z" level=error msg="failed to connect to proxy after 5s: dial unix /run/virtcontainers/pods/7d162da51d56b7ae2a9a75936b079b5c4854a25fb1fc1ec06b75b6ae8b64ea55/proxy.sock: connect
: connection refused" command=delete name=cc-runtime pid=64649 source=runtime
time="2018-03-13T15:11:26.440953353Z" level=warning msg="Retrying proxy connection" arch=amd64 attempt=12 error="dial unix /run/virtcontainers/pods/62e04086797ce38140a9124f2a4314def8e9589d15adafaf3d1802e3ad955ff
e/proxy.sock: connect: connection refused" name=cc-runtime pid=64713 proxy-address=/run/virtcontainers/pods/62e04086797ce38140a9124f2a4314def8e9589d15adafaf3d1802e3ad955ffe/proxy.sock proxy-network=unix remainin
g-time-secs=4.00 source=virtcontainers subsystem=hyper
time="2018-03-13T15:11:27.442607872Z" level=warning msg="Retrying proxy connection" arch=amd64 attempt=22 error="dial unix /run/virtcontainers/pods/62e04086797ce38140a9124f2a4314def8e9589d15adafaf3d1802e3ad955ff
e/proxy.sock: connect: connection refused" name=cc-runtime pid=64713 proxy-address=/run/virtcontainers/pods/62e04086797ce38140a9124f2a4314def8e9589d15adafaf3d1802e3ad955ffe/proxy.sock proxy-network=unix remainin
g-time-secs=3.00 source=virtcontainers subsystem=hyper
time="2018-03-13T15:11:28.444659987Z" level=warning msg="Retrying proxy connection" arch=amd64 attempt=32 error="dial unix /run/virtcontainers/pods/62e04086797ce38140a9124f2a4314def8e9589d15adafaf3d1802e3ad955ff
e/proxy.sock: connect: connection refused" name=cc-runtime pid=64713 proxy-address=/run/virtcontainers/pods/62e04086797ce38140a9124f2a4314def8e9589d15adafaf3d1802e3ad955ffe/proxy.sock proxy-network=unix remainin
g-time-secs=1.99 source=virtcontainers subsystem=hyper
time="2018-03-13T15:11:29.446879296Z" level=warning msg="Retrying proxy connection" arch=amd64 attempt=42 error="dial unix /run/virtcontainers/pods/62e04086797ce38140a9124f2a4314def8e9589d15adafaf3d1802e3ad955ff
e/proxy.sock: connect: connection refused" name=cc-runtime pid=64713 proxy-address=/run/virtcontainers/pods/62e04086797ce38140a9124f2a4314def8e9589d15adafaf3d1802e3ad955ffe/proxy.sock proxy-network=unix remainin
g-time-secs=0.99 source=virtcontainers subsystem=hyper
time="2018-03-13T15:11:30.448903994Z" level=error msg="failed to connect to proxy after 5s: dial unix /run/virtcontainers/pods/62e04086797ce38140a9124f2a4314def8e9589d15adafaf3d1802e3ad955ffe/proxy.sock: connect
: connection refused" command=delete name=cc-runtime pid=64713 source=runtime

Proxy logs

Recent proxy problems found in system journal:

time="2018-03-13T15:09:54.339860124Z" level=debug msg="[ 1473.353246] sd 0:0:0:1: [sdb] Synchronize Cache(10) failed: Result: hostbyte=0x00 driverbyte=0x08" name=cc-proxy pid=60782 source=qemu vm=62e04086797ce38
140a9124f2a4314def8e9589d15adafaf3d1802e3ad955ffe
time="2018-03-13T15:09:54.347081027Z" level=debug msg="{\\\"channel\\\":\\\"ctl\\\",\\\"command\\\":\\\"removecontainer\\\",\\\"error\\\":\\\"Container 376cadd7299472a4ee718739d490f0ed98d6f43c8ed9051cccba36b803d
954e3 running, impossible to remove\\\",\\\"level\\\":\\\"info\\\",\\\"msg\\\":\\\"command failed\\\",\\\"name\\\":\\\"cc-agent\\\",\\\"pid\\\":108,\\\"time\\\":\\\"2018-03-13T15:09:54.326127873Z\\\"}" name=cc-p
roxy pid=57089 source=qemu vm=0d856a79a7c84b726dce37c78cd4b1f5245724f02b31ab717aafac74f2926169
time="2018-03-13T15:09:54.375830138Z" level=debug msg="{\\\"channel\\\":\\\"ctl\\\",\\\"command\\\":\\\"removecontainer\\\",\\\"error\\\":\\\"Container 376cadd7299472a4ee718739d490f0ed98d6f43c8ed9051cccba36b803d
954e3 running, impossible to remove\\\",\\\"level\\\":\\\"info\\\",\\\"msg\\\":\\\"command failed\\\",\\\"name\\\":\\\"cc-agent\\\",\\\"pid\\\":108,\\\"time\\\":\\\"2018-03-13T15:09:54.354895645Z\\\"}" name=cc-p
roxy pid=57089 source=qemu vm=0d856a79a7c84b726dce37c78cd4b1f5245724f02b31ab717aafac74f2926169
time="2018-03-13T15:09:54.395202814Z" level=debug msg="{\\\"channel\\\":\\\"ctl\\\",\\\"command\\\":\\\"removecontainer\\\",\\\"error\\\":\\\"Container 376cadd7299472a4ee718739d490f0ed98d6f43c8ed9051cccba36b803d
954e3 running, impossible to remove\\\",\\\"level\\\":\\\"info\\\",\\\"msg\\\":\\\"command failed\\\",\\\"name\\\":\\\"cc-agent\\\",\\\"pid\\\":108,\\\"time\\\":\\\"2018-03-13T15:09:54.374354338Z\\\"}" name=cc-p
roxy pid=57089 source=qemu vm=0d856a79a7c84b726dce37c78cd4b1f5245724f02b31ab717aafac74f2926169
time="2018-03-13T15:09:54.497287873Z" level=debug msg="{\\\"channel\\\":\\\"ctl\\\",\\\"command\\\":\\\"removecontainer\\\",\\\"error\\\":\\\"Container 376cadd7299472a4ee718739d490f0ed98d6f43c8ed9051cccba36b803d
954e3 running, impossible to remove\\\",\\\"level\\\":\\\"info\\\",\\\"msg\\\":\\\"command failed\\\",\\\"name\\\":\\\"cc-agent\\\",\\\"pid\\\":108,\\\"time\\\":\\\"2018-03-13T15:09:54.476364434Z\\\"}" name=cc-p
roxy pid=57089 source=qemu vm=0d856a79a7c84b726dce37c78cd4b1f5245724f02b31ab717aafac74f2926169
time="2018-03-13T15:09:54.500462418Z" level=debug msg="{\\\"channel\\\":\\\"ctl\\\",\\\"command\\\":\\\"removecontainer\\\",\\\"error\\\":\\\"Container 376cadd7299472a4ee718739d490f0ed98d6f43c8ed9051cccba36b803d
954e3 running, impossible to remove\\\",\\\"level\\\":\\\"info\\\",\\\"msg\\\":\\\"command failed\\\",\\\"name\\\":\\\"cc-agent\\\",\\\"pid\\\":108,\\\"time\\\":\\\"2018-03-13T15:09:54.47954061Z\\\"}" name=cc-pr
oxy pid=57089 source=qemu vm=0d856a79a7c84b726dce37c78cd4b1f5245724f02b31ab717aafac74f2926169
time="2018-03-13T15:09:54.52159762Z" level=debug msg="{\\\"channel\\\":\\\"ctl\\\",\\\"command\\\":\\\"removecontainer\\\",\\\"error\\\":\\\"Container 376cadd7299472a4ee718739d490f0ed98d6f43c8ed9051cccba36b803d9
54e3 running, impossible to remove\\\",\\\"level\\\":\\\"info\\\",\\\"msg\\\":\\\"command failed\\\",\\\"name\\\":\\\"cc-agent\\\",\\\"pid\\\":108,\\\"time\\\":\\\"2018-03-13T15:09:54.500629069Z\\\"}" name=cc-pr
oxy pid=57089 source=qemu vm=0d856a79a7c84b726dce37c78cd4b1f5245724f02b31ab717aafac74f2926169
time="2018-03-13T15:09:54.609250872Z" level=debug msg="{\\\"channel\\\":\\\"ctl\\\",\\\"command\\\":\\\"removecontainer\\\",\\\"error\\\":\\\"Container 376cadd7299472a4ee718739d490f0ed98d6f43c8ed9051cccba36b803d
954e3 running, impossible to remove\\\",\\\"level\\\":\\\"info\\\",\\\"msg\\\":\\\"command failed\\\",\\\"name\\\":\\\"cc-agent\\\",\\\"pid\\\":108,\\\"time\\\":\\\"2018-03-13T15:09:54.588405162Z\\\"}" name=cc-p
roxy pid=57089 source=qemu vm=0d856a79a7c84b726dce37c78cd4b1f5245724f02b31ab717aafac74f2926169
time="2018-03-13T15:09:54.619074212Z" level=debug msg="{\\\"channel\\\":\\\"ctl\\\",\\\"command\\\":\\\"removecontainer\\\",\\\"error\\\":\\\"Container 376cadd7299472a4ee718739d490f0ed98d6f43c8ed9051cccba36b803d
954e3 running, impossible to remove\\\",\\\"level\\\":\\\"info\\\",\\\"msg\\\":\\\"command failed\\\",\\\"name\\\":\\\"cc-agent\\\",\\\"pid\\\":108,\\\"time\\\":\\\"2018-03-13T15:09:54.59815238Z\\\"}" name=cc-pr
oxy pid=57089 source=qemu vm=0d856a79a7c84b726dce37c78cd4b1f5245724f02b31ab717aafac74f2926169
time="2018-03-13T15:09:54.626542919Z" level=debug msg="{\\\"channel\\\":\\\"ctl\\\",\\\"command\\\":\\\"removecontainer\\\",\\\"error\\\":\\\"Container 376cadd7299472a4ee718739d490f0ed98d6f43c8ed9051cccba36b803d
954e3 running, impossible to remove\\\",\\\"level\\\":\\\"info\\\",\\\"msg\\\":\\\"command failed\\\",\\\"name\\\":\\\"cc-agent\\\",\\\"pid\\\":108,\\\"time\\\":\\\"2018-03-13T15:09:54.605603958Z\\\"}" name=cc-p
roxy pid=57089 source=qemu vm=0d856a79a7c84b726dce37c78cd4b1f5245724f02b31ab717aafac74f2926169
time="2018-03-13T15:09:54.635392945Z" level=debug msg="{\\\"error\\\":\\\"EOF\\\",\\\"level\\\":\\\"info\\\",\\\"msg\\\":\\\"Sequence has been closed\\\",\\\"name\\\":\\\"cc-agent\\\",\\\"pid\\\":113,\\\"sequenc
e\\\":4,\\\"time\\\":\\\"2018-03-13T15:09:54.614981994Z\\\"}" name=cc-proxy pid=60744 source=qemu vm=c5e9e10f01644d249b7b1cf957c90995ef7871249ffcc0e9ec34a6f507ef3c63
time="2018-03-13T15:09:54.635494247Z" level=debug msg="{\\\"error\\\":\\\"EOF\\\",\\\"level\\\":\\\"info\\\",\\\"msg\\\":\\\"Sequence has been closed\\\",\\\"name\\\":\\\"cc-agent\\\",\\\"pid\\\":113,\\\"sequenc
e\\\":3,\\\"time\\\":\\\"2018-03-13T15:09:54.615473272Z\\\"}" name=cc-proxy pid=60744 source=qemu vm=c5e9e10f01644d249b7b1cf957c90995ef7871249ffcc0e9ec34a6f507ef3c63
time="2018-03-13T15:09:54.638896495Z" level=debug msg="{\\\"container\\\":\\\"c5e9e10f01644d249b7b1cf957c90995ef7871249ffcc0e9ec34a6f507ef3c63\\\",\\\"container-pid\\\":\\\"3\\\",\\\"error\\\":\\\"invalid argume
nt\\\",\\\"level\\\":\\\"warning\\\",\\\"msg\\\":\\\"Could not close container stream\\\",\\\"name\\\":\\\"cc-agent\\\",\\\"pid\\\":113,\\\"stream\\\":\\\"stdin\\\",\\\"time\\\":\\\"2018-03-13T15:09:54.617244948
Z\\\"}" name=cc-proxy pid=60744 source=qemu vm=c5e9e10f01644d249b7b1cf957c90995ef7871249ffcc0e9ec34a6f507ef3c63
time="2018-03-13T15:09:54.639031297Z" level=error msg="error serving client: write unix /run/virtcontainers/pods/c5e9e10f01644d249b7b1cf957c90995ef7871249ffcc0e9ec34a6f507ef3c63/proxy.sock->@: use of closed netw
ork connection" client=4 name=cc-proxy pid=60744 source=proxy
time="2018-03-13T15:09:55.274151066Z" level=debug msg="[ 1474.363807] sd 0:0:0:0: [sda] Synchronize Cache(10) failed: Result: hostbyte=0x04 driverbyte=0x00" name=cc-proxy pid=60744 source=qemu vm=c5e9e10f01644d2
49b7b1cf957c90995ef7871249ffcc0e9ec34a6f507ef3c63
time="2018-03-13T15:09:55.288076664Z" level=debug msg="{\\\"channel\\\":\\\"ctl\\\",\\\"command\\\":\\\"removecontainer\\\",\\\"error\\\":\\\"Container 376cadd7299472a4ee718739d490f0ed98d6f43c8ed9051cccba36b803d
954e3 running, impossible to remove\\\",\\\"level\\\":\\\"info\\\",\\\"msg\\\":\\\"command failed\\\",\\\"name\\\":\\\"cc-agent\\\",\\\"pid\\\":108,\\\"time\\\":\\\"2018-03-13T15:09:55.267187523Z\\\"}" name=cc-p
roxy pid=57089 source=qemu vm=0d856a79a7c84b726dce37c78cd4b1f5245724f02b31ab717aafac74f2926169
time="2018-03-13T15:09:55.303534185Z" level=debug msg="{\\\"channel\\\":\\\"ctl\\\",\\\"command\\\":\\\"removecontainer\\\",\\\"error\\\":\\\"Container 376cadd7299472a4ee718739d490f0ed98d6f43c8ed9051cccba36b803d
954e3 running, impossible to remove\\\",\\\"level\\\":\\\"info\\\",\\\"msg\\\":\\\"command failed\\\",\\\"name\\\":\\\"cc-agent\\\",\\\"pid\\\":108,\\\"time\\\":\\\"2018-03-13T15:09:55.282550979Z\\\"}" name=cc-p
roxy pid=57089 source=qemu vm=0d856a79a7c84b726dce37c78cd4b1f5245724f02b31ab717aafac74f2926169
time="2018-03-13T15:09:55.326588514Z" level=debug msg="{\\\"channel\\\":\\\"ctl\\\",\\\"command\\\":\\\"removecontainer\\\",\\\"error\\\":\\\"Container 376cadd7299472a4ee718739d490f0ed98d6f43c8ed9051cccba36b803d
954e3 running, impossible to remove\\\",\\\"level\\\":\\\"info\\\",\\\"msg\\\":\\\"command failed\\\",\\\"name\\\":\\\"cc-agent\\\",\\\"pid\\\":108,\\\"time\\\":\\\"2018-03-13T15:09:55.303570956Z\\\"}" name=cc-p
roxy pid=57089 source=qemu vm=0d856a79a7c84b726dce37c78cd4b1f5245724f02b31ab717aafac74f2926169
time="2018-03-13T15:09:55.418601027Z" level=debug msg="{\\\"channel\\\":\\\"ctl\\\",\\\"command\\\":\\\"removecontainer\\\",\\\"error\\\":\\\"Container 376cadd7299472a4ee718739d490f0ed98d6f43c8ed9051cccba36b803d
954e3 running, impossible to remove\\\",\\\"level\\\":\\\"info\\\",\\\"msg\\\":\\\"command failed\\\",\\\"name\\\":\\\"cc-agent\\\",\\\"pid\\\":108,\\\"time\\\":\\\"2018-03-13T15:09:55.39758103Z\\\"}" name=cc-pr
oxy pid=57089 source=qemu vm=0d856a79a7c84b726dce37c78cd4b1f5245724f02b31ab717aafac74f2926169
time="2018-03-13T15:09:55.426597441Z" level=debug msg="{\\\"error\\\":\\\"EOF\\\",\\\"level\\\":\\\"info\\\",\\\"msg\\\":\\\"Sequence has been closed\\\",\\\"name\\\":\\\"cc-agent\\\",\\\"pid\\\":99,\\\"sequence
\\\":4,\\\"time\\\":\\\"2018-03-13T15:09:55.40477418Z\\\"}" name=cc-proxy pid=60782 source=qemu vm=62e04086797ce38140a9124f2a4314def8e9589d15adafaf3d1802e3ad955ffe
time="2018-03-13T15:09:55.426746044Z" level=debug msg="{\\\"error\\\":\\\"EOF\\\",\\\"level\\\":\\\"info\\\",\\\"msg\\\":\\\"Sequence has been closed\\\",\\\"name\\\":\\\"cc-agent\\\",\\\"pid\\\":99,\\\"sequence
\\\":3,\\\"time\\\":\\\"2018-03-13T15:09:55.405193194Z\\\"}" name=cc-proxy pid=60782 source=qemu vm=62e04086797ce38140a9124f2a4314def8e9589d15adafaf3d1802e3ad955ffe
time="2018-03-13T15:09:55.427023648Z" level=debug msg="{\\\"container\\\":\\\"62e04086797ce38140a9124f2a4314def8e9589d15adafaf3d1802e3ad955ffe\\\",\\\"container-pid\\\":\\\"3\\\",\\\"error\\\":\\\"invalid argume
nt\\\",\\\"level\\\":\\\"warning\\\",\\\"msg\\\":\\\"Could not close container stream\\\",\\\"name\\\":\\\"cc-agent\\\",\\\"pid\\\":99,\\\"stream\\\":\\\"stdin\\\",\\\"time\\\":\\\"2018-03-13T15:09:55.406677203Z
\\\"}" name=cc-proxy pid=60782 source=qemu vm=62e04086797ce38140a9124f2a4314def8e9589d15adafaf3d1802e3ad955ffe
time="2018-03-13T15:09:55.427344952Z" level=error msg="error serving client: write unix /run/virtcontainers/pods/62e04086797ce38140a9124f2a4314def8e9589d15adafaf3d1802e3ad955ffe/proxy.sock->@: use of closed netw
ork connection" client=4 name=cc-proxy pid=60782 source=proxy
time="2018-03-13T15:09:55.642688026Z" level=debug msg="[ 1474.655490] sd 0:0:0:0: [sda] Synchronize Cache(10) failed: Result: hostbyte=0x04 driverbyte=0x00" name=cc-proxy pid=60782 source=qemu vm=62e04086797ce38
140a9124f2a4314def8e9589d15adafaf3d1802e3ad955ffe
time="2018-03-13T15:09:55.655113603Z" level=debug msg="{\\\"channel\\\":\\\"ctl\\\",\\\"command\\\":\\\"removecontainer\\\",\\\"error\\\":\\\"Container 376cadd7299472a4ee718739d490f0ed98d6f43c8ed9051cccba36b803d
954e3 running, impossible to remove\\\",\\\"level\\\":\\\"info\\\",\\\"msg\\\":\\\"command failed\\\",\\\"name\\\":\\\"cc-agent\\\",\\\"pid\\\":108,\\\"time\\\":\\\"2018-03-13T15:09:55.633838602Z\\\"}" name=cc-p
roxy pid=57089 source=qemu vm=0d856a79a7c84b726dce37c78cd4b1f5245724f02b31ab717aafac74f2926169
time="2018-03-13T15:09:55.675535394Z" level=debug msg="{\\\"channel\\\":\\\"ctl\\\",\\\"command\\\":\\\"removecontainer\\\",\\\"error\\\":\\\"Container 376cadd7299472a4ee718739d490f0ed98d6f43c8ed9051cccba36b803d
954e3 running, impossible to remove\\\",\\\"level\\\":\\\"info\\\",\\\"msg\\\":\\\"command failed\\\",\\\"name\\\":\\\"cc-agent\\\",\\\"pid\\\":108,\\\"time\\\":\\\"2018-03-13T15:09:55.654594025Z\\\"}" name=cc-p
roxy pid=57089 source=qemu vm=0d856a79a7c84b726dce37c78cd4b1f5245724f02b31ab717aafac74f2926169
time="2018-03-13T15:09:55.813017657Z" level=debug msg="{\\\"channel\\\":\\\"ctl\\\",\\\"command\\\":\\\"removecontainer\\\",\\\"error\\\":\\\"Container 376cadd7299472a4ee718739d490f0ed98d6f43c8ed9051cccba36b803d
954e3 running, impossible to remove\\\",\\\"level\\\":\\\"info\\\",\\\"msg\\\":\\\"command failed\\\",\\\"name\\\":\\\"cc-agent\\\",\\\"pid\\\":108,\\\"time\\\":\\\"2018-03-13T15:09:55.792273181Z\\\"}" name=cc-p
roxy pid=57089 source=qemu vm=0d856a79a7c84b726dce37c78cd4b1f5245724f02b31ab717aafac74f2926169
time="2018-03-13T15:10:10.214721985Z" level=debug msg="{\\\"channel\\\":\\\"ctl\\\",\\\"command\\\":\\\"removecontainer\\\",\\\"error\\\":\\\"Container 376cadd7299472a4ee718739d490f0ed98d6f43c8ed9051cccba36b803d
954e3 running, impossible to remove\\\",\\\"level\\\":\\\"info\\\",\\\"msg\\\":\\\"command failed\\\",\\\"name\\\":\\\"cc-agent\\\",\\\"pid\\\":108,\\\"time\\\":\\\"2018-03-13T15:10:10.193921117Z\\\"}" name=cc-p
roxy pid=57089 source=qemu vm=0d856a79a7c84b726dce37c78cd4b1f5245724f02b31ab717aafac74f2926169
time="2018-03-13T15:10:15.252071988Z" level=debug msg="{\\\"channel\\\":\\\"ctl\\\",\\\"command\\\":\\\"removecontainer\\\",\\\"error\\\":\\\"Container 376cadd7299472a4ee718739d490f0ed98d6f43c8ed9051cccba36b803d
954e3 running, impossible to remove\\\",\\\"level\\\":\\\"info\\\",\\\"msg\\\":\\\"command failed\\\",\\\"name\\\":\\\"cc-agent\\\",\\\"pid\\\":108,\\\"time\\\":\\\"2018-03-13T15:10:15.231075743Z\\\"}" name=cc-p
roxy pid=57089 source=qemu vm=0d856a79a7c84b726dce37c78cd4b1f5245724f02b31ab717aafac74f2926169
time="2018-03-13T15:10:21.695023932Z" level=debug msg="{\\\"channel\\\":\\\"ctl\\\",\\\"command\\\":\\\"removecontainer\\\",\\\"error\\\":\\\"Container 376cadd7299472a4ee718739d490f0ed98d6f43c8ed9051cccba36b803d
954e3 running, impossible to remove\\\",\\\"level\\\":\\\"info\\\",\\\"msg\\\":\\\"command failed\\\",\\\"name\\\":\\\"cc-agent\\\",\\\"pid\\\":108,\\\"time\\\":\\\"2018-03-13T15:10:21.674231844Z\\\"}" name=cc-p
roxy pid=57089 source=qemu vm=0d856a79a7c84b726dce37c78cd4b1f5245724f02b31ab717aafac74f2926169
time="2018-03-13T15:10:31.724823708Z" level=debug msg="{\\\"error\\\":\\\"read /dev/ptmx: input/output error\\\",\\\"level\\\":\\\"info\\\",\\\"msg\\\":\\\"Sequence has been closed\\\",\\\"name\\\":\\\"cc-agent\
\\",\\\"pid\\\":113,\\\"sequence\\\":9,\\\"time\\\":\\\"2018-03-13T15:10:31.690332684Z\\\"}" name=cc-proxy pid=61442 source=qemu vm=7d162da51d56b7ae2a9a75936b079b5c4854a25fb1fc1ec06b75b6ae8b64ea55
time="2018-03-13T15:10:31.728876364Z" level=error msg="error serving client: write unix /run/virtcontainers/pods/7d162da51d56b7ae2a9a75936b079b5c4854a25fb1fc1ec06b75b6ae8b64ea55/proxy.sock->@: use of closed netw
ork connection" client=15 name=cc-proxy pid=61442 source=proxy
time="2018-03-13T15:10:31.997935207Z" level=debug msg="[ 1353.472702] sd 0:0:0:3: [sdb] Synchronize Cache(10) failed: Result: hostbyte=0x00 driverbyte=0x08" name=cc-proxy pid=61442 source=qemu vm=7d162da51d56b7a
e2a9a75936b079b5c4854a25fb1fc1ec06b75b6ae8b64ea55
time="2018-03-13T15:10:32.012084503Z" level=debug msg="{\\\"channel\\\":\\\"ctl\\\",\\\"command\\\":\\\"removecontainer\\\",\\\"error\\\":\\\"Container 376cadd7299472a4ee718739d490f0ed98d6f43c8ed9051cccba36b803d
954e3 running, impossible to remove\\\",\\\"level\\\":\\\"info\\\",\\\"msg\\\":\\\"command failed\\\",\\\"name\\\":\\\"cc-agent\\\",\\\"pid\\\":108,\\\"time\\\":\\\"2018-03-13T15:10:31.991345621Z\\\"}" name=cc-p
roxy pid=57089 source=qemu vm=0d856a79a7c84b726dce37c78cd4b1f5245724f02b31ab717aafac74f2926169
time="2018-03-13T15:10:32.030462259Z" level=debug msg="{\\\"channel\\\":\\\"ctl\\\",\\\"command\\\":\\\"removecontainer\\\",\\\"error\\\":\\\"Container 376cadd7299472a4ee718739d490f0ed98d6f43c8ed9051cccba36b803d
954e3 running, impossible to remove\\\",\\\"level\\\":\\\"info\\\",\\\"msg\\\":\\\"command failed\\\",\\\"name\\\":\\\"cc-agent\\\",\\\"pid\\\":108,\\\"time\\\":\\\"2018-03-13T15:10:32.009562841Z\\\"}" name=cc-p
roxy pid=57089 source=qemu vm=0d856a79a7c84b726dce37c78cd4b1f5245724f02b31ab717aafac74f2926169
time="2018-03-13T15:10:32.102308657Z" level=debug msg="{\\\"channel\\\":\\\"ctl\\\",\\\"command\\\":\\\"removecontainer\\\",\\\"error\\\":\\\"Container 376cadd7299472a4ee718739d490f0ed98d6f43c8ed9051cccba36b803d
954e3 running, impossible to remove\\\",\\\"level\\\":\\\"info\\\",\\\"msg\\\":\\\"command failed\\\",\\\"name\\\":\\\"cc-agent\\\",\\\"pid\\\":108,\\\"time\\\":\\\"2018-03-13T15:10:32.081527362Z\\\"}" name=cc-p
roxy pid=57089 source=qemu vm=0d856a79a7c84b726dce37c78cd4b1f5245724f02b31ab717aafac74f2926169
time="2018-03-13T15:10:32.141617604Z" level=debug msg="{\\\"channel\\\":\\\"ctl\\\",\\\"command\\\":\\\"removecontainer\\\",\\\"error\\\":\\\"Container 376cadd7299472a4ee718739d490f0ed98d6f43c8ed9051cccba36b803d
954e3 running, impossible to remove\\\",\\\"level\\\":\\\"info\\\",\\\"msg\\\":\\\"command failed\\\",\\\"name\\\":\\\"cc-agent\\\",\\\"pid\\\":108,\\\"time\\\":\\\"2018-03-13T15:10:32.120266779Z\\\"}" name=cc-p
roxy pid=57089 source=qemu vm=0d856a79a7c84b726dce37c78cd4b1f5245724f02b31ab717aafac74f2926169
time="2018-03-13T15:10:32.187589243Z" level=debug msg="{\\\"channel\\\":\\\"ctl\\\",\\\"command\\\":\\\"removecontainer\\\",\\\"error\\\":\\\"Container 376cadd7299472a4ee718739d490f0ed98d6f43c8ed9051cccba36b803d
954e3 running, impossible to remove\\\",\\\"level\\\":\\\"info\\\",\\\"msg\\\":\\\"command failed\\\",\\\"name\\\":\\\"cc-agent\\\",\\\"pid\\\":108,\\\"time\\\":\\\"2018-03-13T15:10:32.1668637Z\\\"}" name=cc-pro
xy pid=57089 source=qemu vm=0d856a79a7c84b726dce37c78cd4b1f5245724f02b31ab717aafac74f2926169
time="2018-03-13T15:10:32.299999805Z" level=debug msg="{\\\"channel\\\":\\\"ctl\\\",\\\"command\\\":\\\"removecontainer\\\",\\\"error\\\":\\\"Container 376cadd7299472a4ee718739d490f0ed98d6f43c8ed9051cccba36b803d
954e3 running, impossible to remove\\\",\\\"level\\\":\\\"info\\\",\\\"msg\\\":\\\"command failed\\\",\\\"name\\\":\\\"cc-agent\\\",\\\"pid\\\":108,\\\"time\\\":\\\"2018-03-13T15:10:32.27923507Z\\\"}" name=cc-pr
oxy pid=57089 source=qemu vm=0d856a79a7c84b726dce37c78cd4b1f5245724f02b31ab717aafac74f2926169
time="2018-03-13T15:10:32.310342949Z" level=debug msg="{\\\"error\\\":\\\"EOF\\\",\\\"level\\\":\\\"info\\\",\\\"msg\\\":\\\"Sequence has been closed\\\",\\\"name\\\":\\\"cc-agent\\\",\\\"pid\\\":113,\\\"sequenc
e\\\":4,\\\"time\\\":\\\"2018-03-13T15:10:32.275752942Z\\\"}" name=cc-proxy pid=61442 source=qemu vm=7d162da51d56b7ae2a9a75936b079b5c4854a25fb1fc1ec06b75b6ae8b64ea55
time="2018-03-13T15:10:32.310950057Z" level=debug msg="{\\\"error\\\":\\\"EOF\\\",\\\"level\\\":\\\"info\\\",\\\"msg\\\":\\\"Sequence has been closed\\\",\\\"name\\\":\\\"cc-agent\\\",\\\"pid\\\":113,\\\"sequenc
e\\\":3,\\\"time\\\":\\\"2018-03-13T15:10:32.276406926Z\\\"}" name=cc-proxy pid=61442 source=qemu vm=7d162da51d56b7ae2a9a75936b079b5c4854a25fb1fc1ec06b75b6ae8b64ea55
time="2018-03-13T15:10:32.314592008Z" level=error msg="error serving client: write unix /run/virtcontainers/pods/7d162da51d56b7ae2a9a75936b079b5c4854a25fb1fc1ec06b75b6ae8b64ea55/proxy.sock->@: use of closed netw
ork connection" client=4 name=cc-proxy pid=61442 source=proxy
time="2018-03-13T15:10:32.31546342Z" level=debug msg="{\\\"container\\\":\\\"7d162da51d56b7ae2a9a75936b079b5c4854a25fb1fc1ec06b75b6ae8b64ea55\\\",\\\"container-pid\\\":\\\"3\\\",\\\"error\\\":\\\"invalid argumen
t\\\",\\\"level\\\":\\\"warning\\\",\\\"msg\\\":\\\"Could not close container stream\\\",\\\"name\\\":\\\"cc-agent\\\",\\\"pid\\\":113,\\\"stream\\\":\\\"stdin\\\",\\\"time\\\":\\\"2018-03-13T15:10:32.278490716Z
\\\"}" name=cc-proxy pid=61442 source=qemu vm=7d162da51d56b7ae2a9a75936b079b5c4854a25fb1fc1ec06b75b6ae8b64ea55
time="2018-03-13T15:10:32.550899292Z" level=debug msg="[ 1354.027086] sd 0:0:0:0: [sda] Synchronize Cache(10) failed: Result: hostbyte=0x04 driverbyte=0x00" name=cc-proxy pid=61442 source=qemu vm=7d162da51d56b7a
e2a9a75936b079b5c4854a25fb1fc1ec06b75b6ae8b64ea55
time="2018-03-13T15:10:32.561062134Z" level=debug msg="{\\\"channel\\\":\\\"ctl\\\",\\\"command\\\":\\\"removecontainer\\\",\\\"error\\\":\\\"Container 376cadd7299472a4ee718739d490f0ed98d6f43c8ed9051cccba36b803d
954e3 running, impossible to remove\\\",\\\"level\\\":\\\"info\\\",\\\"msg\\\":\\\"command failed\\\",\\\"name\\\":\\\"cc-agent\\\",\\\"pid\\\":108,\\\"time\\\":\\\"2018-03-13T15:10:32.540010117Z\\\"}" name=cc-p
roxy pid=57089 source=qemu vm=0d856a79a7c84b726dce37c78cd4b1f5245724f02b31ab717aafac74f2926169
time="2018-03-13T15:10:32.579712493Z" level=debug msg="{\\\"channel\\\":\\\"ctl\\\",\\\"command\\\":\\\"removecontainer\\\",\\\"error\\\":\\\"Container 376cadd7299472a4ee718739d490f0ed98d6f43c8ed9051cccba36b803d
954e3 running, impossible to remove\\\",\\\"level\\\":\\\"info\\\",\\\"msg\\\":\\\"command failed\\\",\\\"name\\\":\\\"cc-agent\\\",\\\"pid\\\":108,\\\"time\\\":\\\"2018-03-13T15:10:32.558848687Z\\\"}" name=cc-p
roxy pid=57089 source=qemu vm=0d856a79a7c84b726dce37c78cd4b1f5245724f02b31ab717aafac74f2926169
time="2018-03-13T15:10:33.020431718Z" level=debug msg="{\\\"channel\\\":\\\"ctl\\\",\\\"command\\\":\\\"removecontainer\\\",\\\"error\\\":\\\"Container 376cadd7299472a4ee718739d490f0ed98d6f43c8ed9051cccba36b803d
954e3 running, impossible to remove\\\",\\\"level\\\":\\\"info\\\",\\\"msg\\\":\\\"command failed\\\",\\\"name\\\":\\\"cc-agent\\\",\\\"pid\\\":108,\\\"time\\\":\\\"2018-03-13T15:10:32.999593523Z\\\"}" name=cc-p
roxy pid=57089 source=qemu vm=0d856a79a7c84b726dce37c78cd4b1f5245724f02b31ab717aafac74f2926169
time="2018-03-13T15:10:33.606566159Z" level=debug msg="{\\\"channel\\\":\\\"ctl\\\",\\\"command\\\":\\\"removecontainer\\\",\\\"error\\\":\\\"Container 376cadd7299472a4ee718739d490f0ed98d6f43c8ed9051cccba36b803d
954e3 running, impossible to remove\\\",\\\"level\\\":\\\"info\\\",\\\"msg\\\":\\\"command failed\\\",\\\"name\\\":\\\"cc-agent\\\",\\\"pid\\\":108,\\\"time\\\":\\\"2018-03-13T15:10:33.585639537Z\\\"}" name=cc-p
roxy pid=57089 source=qemu vm=0d856a79a7c84b726dce37c78cd4b1f5245724f02b31ab717aafac74f2926169
time="2018-03-13T15:11:20.288570938Z" level=debug msg="{\\\"channel\\\":\\\"ctl\\\",\\\"command\\\":\\\"removecontainer\\\",\\\"error\\\":\\\"Container 376cadd7299472a4ee718739d490f0ed98d6f43c8ed9051cccba36b803d
954e3 running, impossible to remove\\\",\\\"level\\\":\\\"info\\\",\\\"msg\\\":\\\"command failed\\\",\\\"name\\\":\\\"cc-agent\\\",\\\"pid\\\":108,\\\"time\\\":\\\"2018-03-13T15:11:20.267787647Z\\\"}" name=cc-p
roxy pid=57089 source=qemu vm=0d856a79a7c84b726dce37c78cd4b1f5245724f02b31ab717aafac74f2926169
time="2018-03-13T15:11:25.436108882Z" level=debug msg="{\\\"channel\\\":\\\"ctl\\\",\\\"command\\\":\\\"removecontainer\\\",\\\"error\\\":\\\"Container 376cadd7299472a4ee718739d490f0ed98d6f43c8ed9051cccba36b803d
954e3 running, impossible to remove\\\",\\\"level\\\":\\\"info\\\",\\\"msg\\\":\\\"command failed\\\",\\\"name\\\":\\\"cc-agent\\\",\\\"pid\\\":108,\\\"time\\\":\\\"2018-03-13T15:11:25.41519953Z\\\"}" name=cc-pr
oxy pid=57089 source=qemu vm=0d856a79a7c84b726dce37c78cd4b1f5245724f02b31ab717aafac74f2926169

Shim logs

Recent shim problems found in system journal:

time="2018-03-13T14:28:25.254337342+0000" level="debug" pid=1 function="send_proxy_message" line=300 source="shim" name="cc-shim" msg="Sending frame of type:0, opcode:5, payload:(empty)"
time="2018-03-13T14:29:26.768538445+0000" level="debug" pid=1 function="send_proxy_message" line=300 source="shim" name="cc-shim" msg="Sending frame of type:0, opcode:5, payload:(empty)"
time="2018-03-13T14:32:27.289891231+0000" level="debug" pid=1 function="send_proxy_message" line=300 source="shim" name="cc-shim" msg="Sending frame of type:0, opcode:5, payload:(empty)"
time="2018-03-13T14:49:43.930275317+0000" level="debug" pid=1 function="send_proxy_message" line=300 source="shim" name="cc-shim" msg="Sending frame of type:0, opcode:5, payload:(empty)"
time="2018-03-13T15:00:20.371915303+0000" level="debug" pid=1 function="send_proxy_message" line=300 source="shim" name="cc-shim" msg="Sending frame of type:0, opcode:5, payload:(empty)"
time="2018-03-13T15:09:53.825502276+0000" level="debug" pid=1 function="send_proxy_message" line=300 source="shim" name="cc-shim" msg="Sending frame of type:0, opcode:5, payload:(empty)"
time="2018-03-13T15:09:53.870072013+0000" level="debug" pid=1 function="send_proxy_message" line=300 source="shim" name="cc-shim" msg="Sending frame of type:0, opcode:5, payload:(empty)"
time="2018-03-13T15:09:54.636731064+0000" level="debug" pid=1 function="send_proxy_message" line=300 source="shim" name="cc-shim" msg="Sending frame of type:0, opcode:5, payload:(empty)"
time="2018-03-13T15:09:55.427222950+0000" level="debug" pid=1 function="send_proxy_message" line=300 source="shim" name="cc-shim" msg="Sending frame of type:0, opcode:5, payload:(empty)"
time="2018-03-13T15:10:31.726371129+0000" level="debug" pid=1 function="send_proxy_message" line=300 source="shim" name="cc-shim" msg="Sending frame of type:0, opcode:5, payload:(empty)"
time="2018-03-13T15:10:32.313201089+0000" level="debug" pid=1 function="send_proxy_message" line=300 source="shim" name="cc-shim" msg="Sending frame of type:0, opcode:5, payload:(empty)"

Container manager details

Have docker

Docker

Output of "docker version":

Client:
 Version:       17.12.1-ce
 API version:   1.35
 Go version:    go1.9.4
 Git commit:    7390fc6
 Built: Tue Feb 27 22:17:40 2018
 OS/Arch:       linux/amd64
Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?

Output of "docker info":

Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?

Output of "systemctl show docker":

Type=notify
Restart=on-failure
NotifyAccess=main
RestartUSec=100ms
TimeoutStartUSec=infinity
TimeoutStopUSec=1min 30s
RuntimeMaxUSec=infinity
WatchdogUSec=0
WatchdogTimestampMonotonic=0
FailureAction=none
PermissionsStartOnly=no
RootDirectoryStartOnly=no
RemainAfterExit=no
GuessMainPID=yes
MainPID=0
ControlPID=0
FileDescriptorStoreMax=0
NFileDescriptorStore=0
StatusErrno=0
Result=success
ExecMainStartTimestamp=Tue 2018-03-13 14:10:59 UTC
ExecMainStartTimestampMonotonic=756850203
ExecMainExitTimestamp=Tue 2018-03-13 15:11:31 UTC
ExecMainExitTimestampMonotonic=4389523525
ExecMainPID=26586
ExecMainCode=1
ExecMainStatus=0
ExecStart={ path=/usr/bin/dockerd ; argv[]=/usr/bin/dockerd -D --add-runtime cc-runtime=/usr/local/bin/cc-runtime --default-runtime=cc-runtime --storage-driver=overlay2 ; ignore_errors=no ; start_time=[n/a] ; st
op_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }
ExecReload={ path=/bin/kill ; argv[]=/bin/kill -s HUP $MAINPID ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }
Slice=system.slice
MemoryCurrent=18446744073709551615
CPUUsageNSec=18446744073709551615
TasksCurrent=18446744073709551615
Delegate=yes
CPUAccounting=no
CPUShares=18446744073709551615
StartupCPUShares=18446744073709551615
CPUQuotaPerSecUSec=infinity
BlockIOAccounting=no
BlockIOWeight=18446744073709551615
StartupBlockIOWeight=18446744073709551615
MemoryAccounting=no
MemoryLimit=18446744073709551615
DevicePolicy=auto
TasksAccounting=no
TasksMax=18446744073709551615
UMask=0022
LimitCPU=18446744073709551615
LimitCPUSoft=18446744073709551615
LimitFSIZE=18446744073709551615
LimitFSIZESoft=18446744073709551615
LimitDATA=18446744073709551615
LimitDATASoft=18446744073709551615
LimitSTACK=18446744073709551615
LimitSTACKSoft=8388608
LimitCORE=18446744073709551615
LimitCORESoft=18446744073709551615
LimitRSS=18446744073709551615
LimitRSSSoft=18446744073709551615
LimitNOFILE=1048576
LimitNOFILESoft=1048576
LimitAS=18446744073709551615
LimitASSoft=18446744073709551615
LimitNPROC=18446744073709551615
LimitNPROCSoft=18446744073709551615
LimitMEMLOCK=65536
LimitMEMLOCKSoft=65536
LimitLOCKS=18446744073709551615
LimitLOCKSSoft=18446744073709551615
LimitSIGPENDING=63993
LimitSIGPENDINGSoft=63993
LimitMSGQUEUE=819200
LimitMSGQUEUESoft=819200
LimitNICE=0
LimitNICESoft=0
LimitRTPRIO=0
LimitRTPRIOSoft=0
LimitRTTIME=18446744073709551615
LimitRTTIMESoft=18446744073709551615
OOMScoreAdjust=0
Nice=0
IOScheduling=0
CPUSchedulingPolicy=0
CPUSchedulingPriority=0
TimerSlackNSec=50000
CPUSchedulingResetOnFork=no
NonBlocking=no
StandardInput=null
StandardOutput=journal
StandardError=inherit
TTYReset=no
TTYVHangup=no
TTYVTDisallocate=no
SyslogPriority=30
SyslogLevelPrefix=yes
SyslogLevel=6
SyslogFacility=3
SecureBits=0
CapabilityBoundingSet=18446744073709551615
AmbientCapabilities=0
MountFlags=0
PrivateTmp=no
PrivateNetwork=no
PrivateDevices=no
ProtectHome=no
ProtectSystem=no
SameProcessGroup=no
UtmpMode=init
IgnoreSIGPIPE=yes
NoNewPrivileges=no
SystemCallErrorNumber=0
RuntimeDirectoryMode=0755
KillMode=process
KillSignal=15
SendSIGKILL=yes
SendSIGHUP=no
Id=docker.service
Names=docker.service
Requires=system.slice sysinit.target docker.socket
Wants=network-online.target
WantedBy=multi-user.target
ConsistsOf=docker.socket
Conflicts=shutdown.target
Before=multi-user.target shutdown.target
After=sysinit.target network-online.target basic.target systemd-journald.socket docker.socket firewalld.service system.slice
TriggeredBy=docker.socket
Documentation=https://docs.docker.com
Description=Docker Application Container Engine
LoadState=loaded
ActiveState=inactive
SubState=dead
FragmentPath=/lib/systemd/system/docker.service
DropInPaths=/etc/systemd/system/docker.service.d/clear-containers.conf
UnitFileState=enabled
UnitFilePreset=enabled
StateChangeTimestamp=Tue 2018-03-13 15:11:31 UTC
StateChangeTimestampMonotonic=4389523620
InactiveExitTimestamp=Tue 2018-03-13 14:10:59 UTC
InactiveExitTimestampMonotonic=756850246
ActiveEnterTimestamp=Tue 2018-03-13 14:11:01 UTC
ActiveEnterTimestampMonotonic=759599115
ActiveExitTimestamp=Tue 2018-03-13 15:11:30 UTC
ActiveExitTimestampMonotonic=4388501743
InactiveEnterTimestamp=Tue 2018-03-13 15:11:31 UTC
InactiveEnterTimestampMonotonic=4389523620
CanStart=yes
CanStop=yes
CanReload=yes
CanIsolate=no
StopWhenUnneeded=no
RefuseManualStart=no
RefuseManualStop=no
AllowIsolate=no
DefaultDependencies=yes
OnFailureJobMode=replace
IgnoreOnIsolate=no
NeedDaemonReload=no
JobTimeoutUSec=infinity
JobTimeoutAction=none
ConditionResult=yes
AssertResult=yes
ConditionTimestamp=Tue 2018-03-13 14:10:59 UTC
ConditionTimestampMonotonic=756848958
AssertTimestamp=Tue 2018-03-13 14:10:59 UTC
AssertTimestampMonotonic=756848958
Transient=no
StartLimitInterval=60000000
StartLimitBurst=3
StartLimitAction=none

Have kubectl

Kubernetes

Output of "kubectl version":

Client Version: version.Info{Major:"1", Minor:"9", GitVersion:"v1.9.3", GitCommit:"d2835416544f298c919e2ead3be3d0864b52323b", GitTreeState:"clean", BuildDate:"2018-02-07T12:22:21Z", GoVersion:"go1.9.2", Compile$
:"gc", Platform:"linux/amd64"}
The connection to the server localhost:8080 was refused - did you specify the right host or port?

Output of "kubectl config view":

apiVersion: v1
clusters: []
contexts: []
current-context: ""
kind: Config
preferences: {}
users: []

Output of "systemctl show kubelet":

Type=simple
Restart=always
NotifyAccess=none
RestartUSec=10s
TimeoutStartUSec=1min 30s
TimeoutStopUSec=1min 30s
RuntimeMaxUSec=infinity
WatchdogUSec=0
WatchdogTimestampMonotonic=0
FailureAction=none
PermissionsStartOnly=no
RootDirectoryStartOnly=no
RemainAfterExit=no
GuessMainPID=yes
MainPID=0
ControlPID=0
FileDescriptorStoreMax=0
NFileDescriptorStore=0
StatusErrno=0
Result=success
ExecMainStartTimestamp=Tue 2018-03-13 14:24:58 UTC
ExecMainStartTimestampMonotonic=1596351969
ExecMainExitTimestamp=Tue 2018-03-13 15:11:22 UTC
ExecMainExitTimestampMonotonic=4380088367
ExecMainPID=55587
ExecMainCode=2
ExecMainStatus=15
ExecStart={ path=/usr/bin/kubelet ; argv[]=/usr/bin/kubelet $KUBELET_KUBECONFIG_ARGS $KUBELET_SYSTEM_PODS_ARGS $KUBELET_NETWORK_ARGS $KUBELET_DNS_ARGS $KUBELET_AUTHZ_ARGS $KUBELET_CADVISOR_ARGS $KUBELET_CERTIFI$
ATE_ARGS $KUBELET_EXTRA_ARGS ; ignore_errors=no ; start_time=[Tue 2018-03-13 14:24:58 UTC] ; stop_time=[Tue 2018-03-13 15:11:22 UTC] ; pid=55587 ; code=killed ; status=15/TERM }
Slice=system.slice
MemoryCurrent=18446744073709551615
CPUUsageNSec=18446744073709551615
TasksCurrent=18446744073709551615
Delegate=no
CPUAccounting=no
CPUShares=18446744073709551615
StartupCPUShares=18446744073709551615
CPUQuotaPerSecUSec=infinity
BlockIOAccounting=no
BlockIOWeight=18446744073709551615
StartupBlockIOWeight=18446744073709551615
MemoryAccounting=no
MemoryLimit=18446744073709551615
DevicePolicy=auto
TasksAccounting=no
TasksMax=18446744073709551615
Environment=KUBELET_KUBECONFIG_ARGS=--bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.conf\x20--kubeconfig=/etc/kubernetes/kubelet.conf KUBELET_SYSTEM_PODS_ARGS=--pod-manifest-path=/etc/kubernetes/manife$
ts\x20--allow-privileged=true KUBELET_NETWORK_ARGS=--network-plugin=cni\x20--cni-conf-dir=/etc/cni/net.d\x20--cni-bin-dir=/opt/cni/bin KUBELET_DNS_ARGS=--cluster-dns=10.96.0.10\x20--cluster-domain=cluster.local 
KUBELET_AUTHZ_ARGS=--authorization-mode=Webhook\x20--client-ca-file=/etc/kubernetes/pki/ca.crt KUBELET_EXTRA_ARGS=--container-runtime=remote\x20--container-runtime-endpoint=unix:///var/run/crio/crio.sock\x20--r$
ntime-request-timeout=30m KUBELET_CADVISOR_ARGS=--cadvisor-port=0 KUBELET_CERTIFICATE_ARGS=--rotate-certificates=true\x20--cert-dir=/var/lib/kubelet/pki
UMask=0022
LimitCPU=18446744073709551615
LimitCPUSoft=18446744073709551615
LimitFSIZE=18446744073709551615
LimitFSIZESoft=18446744073709551615
LimitDATA=18446744073709551615
LimitDATASoft=18446744073709551615
LimitSTACK=18446744073709551615
LimitSTACKSoft=8388608
LimitCORE=18446744073709551615
LimitCORESoft=0
LimitRSS=18446744073709551615
LimitRSSSoft=18446744073709551615
LimitNOFILE=4096
LimitNOFILESoft=1024
LimitAS=18446744073709551615
LimitASSoft=18446744073709551615
LimitNPROC=63993
LimitNPROCSoft=63993
LimitMEMLOCK=65536
LimitMEMLOCKSoft=65536
LimitLOCKS=18446744073709551615
LimitLOCKSSoft=18446744073709551615
LimitSIGPENDING=63993
LimitSIGPENDINGSoft=63993
LimitMSGQUEUE=819200
LimitMSGQUEUESoft=819200
LimitNICE=0
LimitNICESoft=0
LimitRTPRIO=0
LimitRTPRIOSoft=0
LimitRTTIME=18446744073709551615
LimitRTTIMESoft=18446744073709551615
OOMScoreAdjust=0
Nice=0
IOScheduling=0
CPUSchedulingPolicy=0
CPUSchedulingPriority=0
TimerSlackNSec=50000
CPUSchedulingResetOnFork=no
NonBlocking=no
StandardInput=null
StandardOutput=journal
StandardError=inherit
TTYReset=no
TTYVHangup=no
TTYVTDisallocate=no
SyslogPriority=30
SyslogLevelPrefix=yes
SyslogLevel=6
SyslogFacility=3
SecureBits=0
CapabilityBoundingSet=18446744073709551615
AmbientCapabilities=0
MountFlags=0
PrivateTmp=no
PrivateNetwork=no
PrivateDevices=no
ProtectHome=no
ProtectSystem=no
SameProcessGroup=no
UtmpMode=init
IgnoreSIGPIPE=yes
NoNewPrivileges=no
SystemCallErrorNumber=0
RuntimeDirectoryMode=0755
KillMode=control-group
KillSignal=15
SendSIGKILL=yes
SendSIGHUP=no
Id=kubelet.service
Names=kubelet.service
Requires=system.slice sysinit.target
WantedBy=multi-user.target
Conflicts=shutdown.target
Before=multi-user.target shutdown.target
After=sysinit.target systemd-journald.socket basic.target system.slice
Documentation=http://kubernetes.io/docs/
Description=kubelet: The Kubernetes Node Agent
LoadState=loaded
ActiveState=inactive
SubState=dead
FragmentPath=/lib/systemd/system/kubelet.service
DropInPaths=/etc/systemd/system/kubelet.service.d/10-kubeadm.conf
UnitFileState=enabled
UnitFilePreset=enabled
StateChangeTimestamp=Tue 2018-03-13 15:11:22 UTC
StateChangeTimestampMonotonic=4380088532
InactiveExitTimestamp=Tue 2018-03-13 14:24:58 UTC
InactiveExitTimestampMonotonic=1596352086
ActiveEnterTimestamp=Tue 2018-03-13 14:24:58 UTC
ActiveEnterTimestampMonotonic=1596352086
ActiveExitTimestamp=Tue 2018-03-13 15:11:22 UTC
ActiveExitTimestampMonotonic=4380045202
InactiveEnterTimestamp=Tue 2018-03-13 15:11:22 UTC
InactiveEnterTimestampMonotonic=4380088532
CanStart=yes
CanStop=yes
CanReload=no
CanIsolate=no
StopWhenUnneeded=no
RefuseManualStart=no
RefuseManualStop=no
AllowIsolate=no
DefaultDependencies=yes
OnFailureJobMode=replace
IgnoreOnIsolate=no
NeedDaemonReload=no
JobTimeoutUSec=infinity
JobTimeoutAction=none
ConditionResult=yes
AssertResult=yes
ConditionTimestamp=Tue 2018-03-13 14:24:58 UTC
ConditionTimestampMonotonic=1596350663
AssertTimestamp=Tue 2018-03-13 14:24:58 UTC
AssertTimestampMonotonic=1596350663
Transient=no
StartLimitInterval=0
StartLimitBurst=5
StartLimitAction=none

Have crio Output of "crio --version":

crio version 1.9.9
commit: "4d7e7dcdbdb7e071cdddc7c9b4ff2c91f5444070"

Output of "systemctl show crio":

Type=simple
Restart=on-failure
NotifyAccess=none
RestartUSec=5s
TimeoutStartUSec=1min 30s
TimeoutStopUSec=1min 30s
RuntimeMaxUSec=infinity
WatchdogUSec=0
WatchdogTimestamp=Tue 2018-03-13 14:24:47 UTC
WatchdogTimestampMonotonic=1585520958
FailureAction=none
PermissionsStartOnly=no
RootDirectoryStartOnly=no
RemainAfterExit=no
GuessMainPID=yes
MainPID=55433
ControlPID=0
FileDescriptorStoreMax=0
NFileDescriptorStore=0
StatusErrno=0
Result=success
ExecMainStartTimestamp=Tue 2018-03-13 14:24:47 UTC
ExecMainStartTimestampMonotonic=1585520925
ExecMainExitTimestampMonotonic=0
ExecMainPID=55433
ExecMainCode=0
ExecMainStatus=0
ExecStart={ path=/usr/local/bin/crio ; argv[]=/usr/local/bin/crio --log-level debug ; ignore_errors=no ; start_time=[Tue 2018-03-13 14:24:47 UTC] ; stop_time=[n/a] ; pid=55433 ; code=(null) ; status=0/0 }
Slice=system.slice
ControlGroup=/system.slice/crio.service
MemoryCurrent=85020672
CPUUsageNSec=57453930920
TasksCurrent=14
Delegate=no
CPUAccounting=no
CPUShares=18446744073709551615
StartupCPUShares=18446744073709551615
CPUQuotaPerSecUSec=infinity
BlockIOAccounting=no
BlockIOWeight=18446744073709551615
StartupBlockIOWeight=18446744073709551615
MemoryAccounting=no
MemoryLimit=18446744073709551615
DevicePolicy=auto
TasksAccounting=no
TasksMax=18446744073709551615
UMask=0022
LimitCPU=18446744073709551615
LimitCPUSoft=18446744073709551615
LimitFSIZE=18446744073709551615
LimitFSIZESoft=18446744073709551615
LimitDATA=18446744073709551615
LimitDATASoft=18446744073709551615
LimitSTACK=18446744073709551615
LimitSTACKSoft=8388608
LimitCORE=18446744073709551615
LimitCORESoft=0
LimitRSS=18446744073709551615
LimitRSSSoft=18446744073709551615
LimitNOFILE=4096
LimitNOFILESoft=1024
LimitAS=18446744073709551615
LimitASSoft=18446744073709551615
LimitNPROC=63993
LimitNPROCSoft=63993
LimitMEMLOCK=65536
LimitMEMLOCKSoft=65536
LimitLOCKS=18446744073709551615
LimitLOCKSSoft=18446744073709551615
LimitSIGPENDING=63993
LimitSIGPENDINGSoft=63993
LimitMSGQUEUE=819200
LimitMSGQUEUESoft=819200
LimitNICE=0
LimitNICESoft=0
LimitRTPRIO=0
LimitRTPRIOSoft=0
LimitRTTIME=18446744073709551615
LimitRTTIMESoft=18446744073709551615
OOMScoreAdjust=0
Nice=0
IOScheduling=0
CPUSchedulingPolicy=0
CPUSchedulingPriority=0
TimerSlackNSec=50000
CPUSchedulingResetOnFork=no
NonBlocking=no
StandardInput=null
StandardOutput=journal
StandardError=inherit
TTYReset=no
TTYVHangup=no
TTYVTDisallocate=no
SyslogPriority=30
SyslogLevelPrefix=yes
SyslogLevel=6
SyslogFacility=3
SecureBits=0
CapabilityBoundingSet=18446744073709551615
AmbientCapabilities=0
MountFlags=0
PrivateTmp=no
PrivateNetwork=no
PrivateDevices=no
ProtectHome=no
ProtectSystem=no
SameProcessGroup=no
UtmpMode=init
IgnoreSIGPIPE=yes
NoNewPrivileges=no
SystemCallErrorNumber=0
RuntimeDirectoryMode=0755
KillMode=control-group
KillSignal=15
SendSIGKILL=yes
SendSIGHUP=no
Id=crio.service
Names=crio.service
Requires=system.slice sysinit.target
Conflicts=shutdown.target
Before=shutdown.target
After=sysinit.target systemd-journald.socket basic.target system.slice
Documentation=https://github.com/kubernetes-incubator/cri-o
Description=CRI-O daemon
LoadState=loaded
ActiveState=active
SubState=running
FragmentPath=/etc/systemd/system/crio.service
UnitFileState=disabled
UnitFilePreset=enabled
StateChangeTimestamp=Tue 2018-03-13 14:24:47 UTC
StateChangeTimestampMonotonic=1585520959
InactiveExitTimestamp=Tue 2018-03-13 14:24:47 UTC
InactiveExitTimestampMonotonic=1585520959
ActiveEnterTimestamp=Tue 2018-03-13 14:24:47 UTC
ActiveEnterTimestampMonotonic=1585520959
ActiveExitTimestampMonotonic=0
InactiveEnterTimestampMonotonic=0
CanStart=yes
CanStop=yes
CanReload=no
CanIsolate=no
StopWhenUnneeded=no
RefuseManualStart=no
RefuseManualStop=no
AllowIsolate=no
DefaultDependencies=yes
OnFailureJobMode=replace
IgnoreOnIsolate=no
NeedDaemonReload=no
JobTimeoutUSec=infinity
JobTimeoutAction=none
ConditionResult=yes
AssertResult=yes
ConditionTimestamp=Tue 2018-03-13 14:24:47 UTC
ConditionTimestampMonotonic=1585519681
AssertTimestamp=Tue 2018-03-13 14:24:47 UTC
AssertTimestampMonotonic=1585519682
Transient=no
StartLimitInterval=10000000
StartLimitBurst=5
StartLimitAction=none

Packages

Have dpkg Output of "dpkg -l|egrep "(cc-oci-runtime|cc-proxy|cc-runtime|cc-shim|kata-proxy|kata-runtime|kata-shim|clear-containers-image|linux-container|qemu-lite|qemu-system-x86)"":

ii  qemu-lite-bin                       741f430a960b5b67745670e8270db91aeb083c5f-30 amd64        bin components for the qemu-lite package.
ii  qemu-lite-data                      741f430a960b5b67745670e8270db91aeb083c5f-30 amd64        data components for the qemu-lite package.

Have rpm Output of "rpm -qa|egrep "(cc-oci-runtime|cc-proxy|cc-runtime|cc-shim|kata-proxy|kata-runtime|kata-shim|clear-containers-image|linux-container|qemu-lite|qemu-system-x86)"":

chavafg commented 6 years ago

@sboeuf @amshinde @mcastelino @sameo

Seems like something changed in CRI-O 1.9.8, that breaks our networking.

chavafg commented 6 years ago

Now we need to add manage_network_ns_lifecycle = true to the [crio.runtime] section in the /etc/crio/crio.conf file. By adding this config option everything works correctly. Closing this issue.