search

kata-containers / runtime

Kata Containers version 1.x runtime (for version 2.x see https://github.com/kata-containers/kata-containers).
https://katacontainers.io/
Apache License 2.0
2.1k stars 375 forks source link

Mount points are left after performing a docker cp #794

Closed GabyCT closed 5 years ago

GabyCT commented 6 years ago

While doing the following

$ docker run -d --name devtest -v /dev:/dev busybox sh -c "sleep 30"
$ docker cp testfile devtest:/dev
$ docker rm -f devtest

It is being observed that mount points are left

$ mount | grep devtmpfs
udev on /dev type devtmpfs (rw,nosuid,relatime,size=4074044k,nr_inodes=1018511,mode=755)
udev on /run/kata-containers/shared/sandboxes/26e37bd224ce6ded8559fd245382cc09810d516730ceb0d9239efbd85e0b9849/26e37bd224ce6ded8559fd245382cc09810d516730ceb0d9239efbd85e0b9849/rootfs/dev type devtmpfs (rw,nosuid,relatime,size=4074044k,nr_inodes=1018511,mode=755)

And they can increase by the number of docker cp that you performed. To reproduce this, I used the following configuration

Meta details

Running kata-collect-data.sh version 1.3.0-rc1 (commit 0d99a4f49f60d936fa15e5a0842cfd4d03a24e8f) at 2018-09-27.19:59:08.823534274+0000.

Runtime is /usr/local/bin/kata-runtime.

kata-env

Output of "/usr/local/bin/kata-runtime kata-env":

[Meta]
  Version = "1.0.18"

[Runtime]
  Debug = true
  Path = "/usr/local/bin/kata-runtime"
  [Runtime.Version]
    Semver = "1.3.0-rc1"
    Commit = "0d99a4f49f60d936fa15e5a0842cfd4d03a24e8f"
    OCI = "1.0.1"
  [Runtime.Config]
    Path = "/usr/share/defaults/kata-containers/configuration.toml"

[Hypervisor]
  MachineType = "pc"
  Version = "QEMU emulator version 2.11.0\nCopyright (c) 2003-2017 Fabrice Bellard and the QEMU Project developers"
  Path = "/usr/bin/qemu-lite-system-x86_64"
  BlockDeviceDriver = "virtio-scsi"
  EntropySource = "/dev/urandom"
  Msize9p = 8192
  MemorySlots = 10
  Debug = true
  UseVSock = false

[Image]
  Path = "/usr/share/kata-containers/kata-containers-image_clearlinux_1.3.0-rc1_agent_1ee972176ae.img"

[Kernel]
  Path = "/usr/share/kata-containers/vmlinuz-4.14.67-12"
  Parameters = "agent.log=debug"

[Initrd]
  Path = ""

[Proxy]
  Type = "kataProxy"
  Version = "kata-proxy version 1.3.0-6ddb006ad3f709cab018af9dc0bf9e756c3ce2cd"
  Path = "/usr/libexec/kata-containers/kata-proxy"
  Debug = true

[Shim]
  Type = "kataShim"
  Version = "kata-shim version 1.3.0-rc1-9b2891cfb153967fa4a65e44b2928255c889f643"
  Path = "/usr/libexec/kata-containers/kata-shim"
  Debug = true

[Agent]
  Type = "kata"

[Host]
  Kernel = "4.15.0-1023-azure"
  Architecture = "amd64"
  VMContainerCapable = true
  SupportVSocks = false
  [Host.Distro]
    Name = "Ubuntu"
    Version = "16.04"
  [Host.CPU]
    Vendor = "GenuineIntel"
    Model = "Intel(R) Xeon(R) CPU E5-2673 v4 @ 2.30GHz"

[Netmon]
  Version = "kata-netmon version 1.3.0-rc1"
  Path = "/usr/libexec/kata-containers/kata-netmon"
  Debug = true
  Enable = false

Runtime config files

Runtime default config files

/etc/kata-containers/configuration.toml
/usr/share/defaults/kata-containers/configuration.toml

Runtime config file contents

Config file /etc/kata-containers/configuration.toml not found Output of "cat "/usr/share/defaults/kata-containers/configuration.toml"":

# Copyright (c) 2017-2018 Intel Corporation
#
# SPDX-License-Identifier: Apache-2.0
#

# XXX: WARNING: this file is auto-generated.
# XXX:
# XXX: Source file: "cli/config/configuration.toml.in"
# XXX: Project:
# XXX:   Name: Kata Containers
# XXX:   Type: kata

[hypervisor.qemu]
path = "/usr/bin/qemu-lite-system-x86_64"
kernel = "/usr/share/kata-containers/vmlinuz.container"
image = "/usr/share/kata-containers/kata-containers.img"
machine_type = "pc"

# Optional space-separated list of options to pass to the guest kernel.
# For example, use `kernel_params = "vsyscall=emulate"` if you are having
# trouble running pre-2.15 glibc.
#
# WARNING: - any parameter specified here will take priority over the default
# parameter value of the same name used to start the virtual machine.
# Do not set values here unless you understand the impact of doing so as you
# may stop the virtual machine from booting.
# To see the list of default parameters, enable hypervisor debug, create a
# container and look for 'default-kernel-parameters' log entries.
kernel_params = " agent.log=debug"

# Path to the firmware.
# If you want that qemu uses the default firmware leave this option empty
firmware = ""

# Machine accelerators
# comma-separated list of machine accelerators to pass to the hypervisor.
# For example, `machine_accelerators = "nosmm,nosmbus,nosata,nopit,static-prt,nofw"`
machine_accelerators=""

# Default number of vCPUs per SB/VM:
# unspecified or 0                --> will be set to 1
# < 0                             --> will be set to the actual number of physical cores
# > 0 <= number of physical cores --> will be set to the specified number
# > number of physical cores      --> will be set to the actual number of physical cores
default_vcpus = 1

# Default maximum number of vCPUs per SB/VM:
# unspecified or == 0             --> will be set to the actual number of physical cores or to the maximum number
#                                     of vCPUs supported by KVM if that number is exceeded
# > 0 <= number of physical cores --> will be set to the specified number
# > number of physical cores      --> will be set to the actual number of physical cores or to the maximum number
#                                     of vCPUs supported by KVM if that number is exceeded
# WARNING: Depending of the architecture, the maximum number of vCPUs supported by KVM is used when
# the actual number of physical cores is greater than it.
# WARNING: Be aware that this value impacts the virtual machine's memory footprint and CPU
# the hotplug functionality. For example, `default_maxvcpus = 240` specifies that until 240 vCPUs
# can be added to a SB/VM, but the memory footprint will be big. Another example, with
# `default_maxvcpus = 8` the memory footprint will be small, but 8 will be the maximum number of
# vCPUs supported by the SB/VM. In general, we recommend that you do not edit this variable,
# unless you know what are you doing.
default_maxvcpus = 0

# Bridges can be used to hot plug devices.
# Limitations:
# * Currently only pci bridges are supported
# * Until 30 devices per bridge can be hot plugged.
# * Until 5 PCI bridges can be cold plugged per VM.
#   This limitation could be a bug in qemu or in the kernel
# Default number of bridges per SB/VM:
# unspecified or 0   --> will be set to 1
# > 1 <= 5           --> will be set to the specified number
# > 5                --> will be set to 5
default_bridges = 1

# Default memory size in MiB for SB/VM.
# If unspecified then it will be set 2048 MiB.
#default_memory = 2048
#
# Default memory slots per SB/VM.
# If unspecified then it will be set 10.
# This is will determine the times that memory will be hotadded to sandbox/VM.
#memory_slots = 10

# Disable block device from being used for a container's rootfs.
# In case of a storage driver like devicemapper where a container's
# root file system is backed by a block device, the block device is passed
# directly to the hypervisor for performance reasons.
# This flag prevents the block device from being passed to the hypervisor,
# 9pfs is used instead to pass the rootfs.
disable_block_device_use = false

# Block storage driver to be used for the hypervisor in case the container
# rootfs is backed by a block device. This is either virtio-scsi or
# virtio-blk.
block_device_driver = "virtio-scsi"

# Enable iothreads (data-plane) to be used. This causes IO to be
# handled in a separate IO thread. This is currently only implemented
# for SCSI.
#
enable_iothreads = false

# Enable pre allocation of VM RAM, default false
# Enabling this will result in lower container density
# as all of the memory will be allocated and locked
# This is useful when you want to reserve all the memory
# upfront or in the cases where you want memory latencies
# to be very predictable
# Default false
#enable_mem_prealloc = true

# Enable huge pages for VM RAM, default false
# Enabling this will result in the VM memory
# being allocated using huge pages.
# This is useful when you want to use vhost-user network
# stacks within the container. This will automatically
# result in memory pre allocation
#enable_hugepages = true

# Enable swap of vm memory. Default false.
# The behaviour is undefined if mem_prealloc is also set to true
#enable_swap = true

# This option changes the default hypervisor and kernel parameters
# to enable debug output where available. This extra output is added
# to the proxy logs, but only when proxy debug is also enabled.
#
# Default false
enable_debug = true

# Disable the customizations done in the runtime when it detects
# that it is running on top a VMM. This will result in the runtime
# behaving as it would when running on bare metal.
#
#disable_nesting_checks = true

# This is the msize used for 9p shares. It is the number of bytes
# used for 9p packet payload.
#msize_9p = 8192

# If true and vsocks are supported, use vsocks to communicate directly
# with the agent and no proxy is started, otherwise use unix
# sockets and start a proxy to communicate with the agent.
# Default false
#use_vsock = true

# VFIO devices are hotplugged on a bridge by default.
# Enable hotplugging on root bus. This may be required for devices with
# a large PCI bar, as this is a current limitation with hotplugging on
# a bridge. This value is valid for "pc" machine type.
# Default false
#hotplug_vfio_on_root_bus = true

# If host doesn't support vhost_net, set to true. Thus we won't create vhost fds for nics.
# Default false
#disable_vhost_net = true
#
# Default entropy source.
# The path to a host source of entropy (including a real hardware RNG)
# /dev/urandom and /dev/random are two main options.
# Be aware that /dev/random is a blocking source of entropy.  If the host
# runs out of entropy, the VMs boot time will increase leading to get startup
# timeouts.
# The source of entropy /dev/urandom is non-blocking and provides a
# generally acceptable source of entropy. It should work well for pretty much
# all practical purposes.
#entropy_source= "/dev/urandom"

[factory]
# VM templating support. Once enabled, new VMs are created from template
# using vm cloning. They will share the same initial kernel, initramfs and
# agent memory by mapping it readonly. It helps speeding up new container
# creation and saves a lot of memory if there are many kata containers running
# on the same host.
#
# When disabled, new VMs are created from scratch.
#
# Default false
#enable_template = true

[proxy.kata]
path = "/usr/libexec/kata-containers/kata-proxy"

# If enabled, proxy messages will be sent to the system log
# (default: disabled)
enable_debug = true

[shim.kata]
path = "/usr/libexec/kata-containers/kata-shim"

# If enabled, shim messages will be sent to the system log
# (default: disabled)
enable_debug = true

[agent.kata]
# There is no field for this section. The goal is only to be able to
# specify which type of agent the user wants to use.

[netmon]
# If enabled, the network monitoring process gets started when the
# sandbox is created. This allows for the detection of some additional
# network being added to the existing network namespace, after the
# sandbox has been created.
# (default: disabled)
#enable_netmon = true

# Specify the path to the netmon binary.
path = "/usr/libexec/kata-containers/kata-netmon"

# If enabled, netmon messages will be sent to the system log
# (default: disabled)
enable_debug = true

[runtime]
# If enabled, the runtime will log additional debug messages to the
# system log
# (default: disabled)
enable_debug = true
#
# Internetworking model
# Determines how the VM should be connected to the
# the container network interface
# Options:
#
#   - bridged
#     Uses a linux bridge to interconnect the container interface to
#     the VM. Works for most cases except macvlan and ipvlan.
#
#   - macvtap
#     Used when the Container network interface can be bridged using
#     macvtap.
internetworking_model="macvtap"

# If enabled, the runtime will create opentracing.io traces and spans.
# (See https://www.jaegertracing.io/docs/getting-started).
# (default: disabled)
#enable_tracing = true

KSM throttler

version

Output of "--version":

/usr/local/bin/kata-collect-data.sh: line 168: --version: command not found

systemd service

Image details

---
osbuilder:
  url: "https://github.com/kata-containers/osbuilder"
  version: "unknown"
rootfs-creation-time: "2018-09-13T22:26:12.472554196+0000Z"
description: "osbuilder rootfs"
file-format-version: "0.0.2"
architecture: "x86_64"
base-distro:
  name: "Clear"
  version: "25000"
  packages:
    default:
      - "iptables-bin"
      - "libudev0-shim"
      - "systemd"
    extra:

agent:
  url: "https://github.com/kata-containers/agent"
  name: "kata-agent"
  version: "1.3.0-rc1-1ee972176ae437bcace0a37227818c506bb64ba9"
  agent-is-init-daemon: "no"

Initrd details

No initrd

Logfiles

Runtime logs

Recent runtime problems found in system journal:

time="2018-09-27T19:25:32.261844503Z" level=debug msg="Could not retrieve anything from storage" arch=amd64 command=create container=66ae83edd972e758611bcd6ca51fe4aa92c4b146de862f2a371ce55860d04466 name=kata-runtime pid=7170 source=virtcontainers subsystem=kata_agent
time="2018-09-27T19:25:32.261947402Z" level=warning msg="fetch sandbox device failed" arch=amd64 command=create container=66ae83edd972e758611bcd6ca51fe4aa92c4b146de862f2a371ce55860d04466 error="open /run/vc/sbs/66ae83edd972e758611bcd6ca51fe4aa92c4b146de862f2a371ce55860d04466/devices.json: no such file or directory" name=kata-runtime pid=7170 sandbox=66ae83edd972e758611bcd6ca51fe4aa92c4b146de862f2a371ce55860d04466 sandboxid=66ae83edd972e758611bcd6ca51fe4aa92c4b146de862f2a371ce55860d04466 source=virtcontainers subsystem=sandbox
time="2018-09-27T19:25:32.264833099Z" level=debug arch=amd64 command=create container=66ae83edd972e758611bcd6ca51fe4aa92c4b146de862f2a371ce55860d04466 default-kernel-parameters="tsc=reliable no_timer_check rcupdate.rcu_expedited=1 i8042.direct=1 i8042.dumbkbd=1 i8042.nopnp=1 i8042.noaux=1 noreplace-smp reboot=k console=hvc0 console=hvc1 iommu=off cryptomgr.notests net.ifnames=0 pci=lastbus=0 root=/dev/pmem0p1 rootflags=dax,data=ordered,errors=remount-ro rw rootfstype=ext4 debug systemd.show_status=true systemd.log_level=debug" name=kata-runtime pid=7170 source=virtcontainers subsystem=qemu
time="2018-09-27T19:25:36.008519816Z" level=error msg="Container not ready, running or paused, impossible to signal the container" arch=amd64 command=kill container=66ae83edd972e758611bcd6ca51fe4aa92c4b146de862f2a371ce55860d04466 name=kata-runtime pid=7287 sandbox=66ae83edd972e758611bcd6ca51fe4aa92c4b146de862f2a371ce55860d04466 source=runtime
time="2018-09-27T19:25:36.056432664Z" level=error msg="Container 66ae83edd972e758611bcd6ca51fe4aa92c4b146de862f2a371ce55860d04466 not ready, running or paused, cannot send a signal" arch=amd64 command=kill container=66ae83edd972e758611bcd6ca51fe4aa92c4b146de862f2a371ce55860d04466 name=kata-runtime pid=7314 sandbox=66ae83edd972e758611bcd6ca51fe4aa92c4b146de862f2a371ce55860d04466 source=runtime
time="2018-09-27T19:25:37.145623177Z" level=debug msg="Could not retrieve anything from storage" arch=amd64 command=create container=fa138897becc662bf46e9e1ffdf5d3a102ae2bc4e20897c00e78be483fe13200 name=kata-runtime pid=7430 source=virtcontainers subsystem=kata_agent
time="2018-09-27T19:25:37.145718076Z" level=warning msg="fetch sandbox device failed" arch=amd64 command=create container=fa138897becc662bf46e9e1ffdf5d3a102ae2bc4e20897c00e78be483fe13200 error="open /run/vc/sbs/fa138897becc662bf46e9e1ffdf5d3a102ae2bc4e20897c00e78be483fe13200/devices.json: no such file or directory" name=kata-runtime pid=7430 sandbox=fa138897becc662bf46e9e1ffdf5d3a102ae2bc4e20897c00e78be483fe13200 sandboxid=fa138897becc662bf46e9e1ffdf5d3a102ae2bc4e20897c00e78be483fe13200 source=virtcontainers subsystem=sandbox
time="2018-09-27T19:25:37.148573373Z" level=debug arch=amd64 command=create container=fa138897becc662bf46e9e1ffdf5d3a102ae2bc4e20897c00e78be483fe13200 default-kernel-parameters="tsc=reliable no_timer_check rcupdate.rcu_expedited=1 i8042.direct=1 i8042.dumbkbd=1 i8042.nopnp=1 i8042.noaux=1 noreplace-smp reboot=k console=hvc0 console=hvc1 iommu=off cryptomgr.notests net.ifnames=0 pci=lastbus=0 root=/dev/pmem0p1 rootflags=dax,data=ordered,errors=remount-ro rw rootfstype=ext4 debug systemd.show_status=true systemd.log_level=debug" name=kata-runtime pid=7430 source=virtcontainers subsystem=qemu
time="2018-09-27T19:25:40.650977856Z" level=error msg="Container not ready, running or paused, impossible to signal the container" arch=amd64 command=kill container=fa138897becc662bf46e9e1ffdf5d3a102ae2bc4e20897c00e78be483fe13200 name=kata-runtime pid=7519 sandbox=fa138897becc662bf46e9e1ffdf5d3a102ae2bc4e20897c00e78be483fe13200 source=runtime
time="2018-09-27T19:25:40.704461598Z" level=error msg="Container fa138897becc662bf46e9e1ffdf5d3a102ae2bc4e20897c00e78be483fe13200 not ready, running or paused, cannot send a signal" arch=amd64 command=kill container=fa138897becc662bf46e9e1ffdf5d3a102ae2bc4e20897c00e78be483fe13200 name=kata-runtime pid=7547 sandbox=fa138897becc662bf46e9e1ffdf5d3a102ae2bc4e20897c00e78be483fe13200 source=runtime
time="2018-09-27T19:25:41.793780811Z" level=debug msg="Could not retrieve anything from storage" arch=amd64 command=create container=ec6052a1516b0df2250cb6b6d3d18ed750945203d851ed49678e09fc8002b67a name=kata-runtime pid=7661 source=virtcontainers subsystem=kata_agent
time="2018-09-27T19:25:41.793918811Z" level=warning msg="fetch sandbox device failed" arch=amd64 command=create container=ec6052a1516b0df2250cb6b6d3d18ed750945203d851ed49678e09fc8002b67a error="open /run/vc/sbs/ec6052a1516b0df2250cb6b6d3d18ed750945203d851ed49678e09fc8002b67a/devices.json: no such file or directory" name=kata-runtime pid=7661 sandbox=ec6052a1516b0df2250cb6b6d3d18ed750945203d851ed49678e09fc8002b67a sandboxid=ec6052a1516b0df2250cb6b6d3d18ed750945203d851ed49678e09fc8002b67a source=virtcontainers subsystem=sandbox
time="2018-09-27T19:25:41.796675108Z" level=debug arch=amd64 command=create container=ec6052a1516b0df2250cb6b6d3d18ed750945203d851ed49678e09fc8002b67a default-kernel-parameters="tsc=reliable no_timer_check rcupdate.rcu_expedited=1 i8042.direct=1 i8042.dumbkbd=1 i8042.nopnp=1 i8042.noaux=1 noreplace-smp reboot=k console=hvc0 console=hvc1 iommu=off cryptomgr.notests net.ifnames=0 pci=lastbus=0 root=/dev/pmem0p1 rootflags=dax,data=ordered,errors=remount-ro rw rootfstype=ext4 debug systemd.show_status=true systemd.log_level=debug" name=kata-runtime pid=7661 source=virtcontainers subsystem=qemu
time="2018-09-27T19:25:45.671005988Z" level=error msg="Container not ready, running or paused, impossible to signal the container" arch=amd64 command=kill container=ec6052a1516b0df2250cb6b6d3d18ed750945203d851ed49678e09fc8002b67a name=kata-runtime pid=7795 sandbox=ec6052a1516b0df2250cb6b6d3d18ed750945203d851ed49678e09fc8002b67a source=runtime
time="2018-09-27T19:25:45.715806239Z" level=error msg="Container ec6052a1516b0df2250cb6b6d3d18ed750945203d851ed49678e09fc8002b67a not ready, running or paused, cannot send a signal" arch=amd64 command=kill container=ec6052a1516b0df2250cb6b6d3d18ed750945203d851ed49678e09fc8002b67a name=kata-runtime pid=7821 sandbox=ec6052a1516b0df2250cb6b6d3d18ed750945203d851ed49678e09fc8002b67a source=runtime
time="2018-09-27T19:25:46.768833493Z" level=debug msg="Could not retrieve anything from storage" arch=amd64 command=create container=267c9a894632e65c4d86409a80a53102c9332e42a4248364777b81fc80bfdf21 name=kata-runtime pid=7934 source=virtcontainers subsystem=kata_agent
time="2018-09-27T19:25:46.768903693Z" level=warning msg="fetch sandbox device failed" arch=amd64 command=create container=267c9a894632e65c4d86409a80a53102c9332e42a4248364777b81fc80bfdf21 error="open /run/vc/sbs/267c9a894632e65c4d86409a80a53102c9332e42a4248364777b81fc80bfdf21/devices.json: no such file or directory" name=kata-runtime pid=7934 sandbox=267c9a894632e65c4d86409a80a53102c9332e42a4248364777b81fc80bfdf21 sandboxid=267c9a894632e65c4d86409a80a53102c9332e42a4248364777b81fc80bfdf21 source=virtcontainers subsystem=sandbox
time="2018-09-27T19:25:46.77145509Z" level=debug arch=amd64 command=create container=267c9a894632e65c4d86409a80a53102c9332e42a4248364777b81fc80bfdf21 default-kernel-parameters="tsc=reliable no_timer_check rcupdate.rcu_expedited=1 i8042.direct=1 i8042.dumbkbd=1 i8042.nopnp=1 i8042.noaux=1 noreplace-smp reboot=k console=hvc0 console=hvc1 iommu=off cryptomgr.notests net.ifnames=0 pci=lastbus=0 root=/dev/pmem0p1 rootflags=dax,data=ordered,errors=remount-ro rw rootfstype=ext4 debug systemd.show_status=true systemd.log_level=debug" name=kata-runtime pid=7934 source=virtcontainers subsystem=qemu
time="2018-09-27T19:25:50.282866768Z" level=error msg="Container not ready, running or paused, impossible to signal the container" arch=amd64 command=kill container=267c9a894632e65c4d86409a80a53102c9332e42a4248364777b81fc80bfdf21 name=kata-runtime pid=8025 sandbox=267c9a894632e65c4d86409a80a53102c9332e42a4248364777b81fc80bfdf21 source=runtime
time="2018-09-27T19:25:50.330823316Z" level=error msg="Container 267c9a894632e65c4d86409a80a53102c9332e42a4248364777b81fc80bfdf21 not ready, running or paused, cannot send a signal" arch=amd64 command=kill container=267c9a894632e65c4d86409a80a53102c9332e42a4248364777b81fc80bfdf21 name=kata-runtime pid=8052 sandbox=267c9a894632e65c4d86409a80a53102c9332e42a4248364777b81fc80bfdf21 source=runtime
time="2018-09-27T19:25:55.413845487Z" level=debug msg="Could not retrieve anything from storage" arch=amd64 command=create container=0515a4f16877ff7f2146d40e1abe05a2f1e4b853320dd2fa98cd7a4cf37de2c3 name=kata-runtime pid=8579 source=virtcontainers subsystem=kata_agent
time="2018-09-27T19:25:55.413929887Z" level=warning msg="fetch sandbox device failed" arch=amd64 command=create container=0515a4f16877ff7f2146d40e1abe05a2f1e4b853320dd2fa98cd7a4cf37de2c3 error="open /run/vc/sbs/0515a4f16877ff7f2146d40e1abe05a2f1e4b853320dd2fa98cd7a4cf37de2c3/devices.json: no such file or directory" name=kata-runtime pid=8579 sandbox=0515a4f16877ff7f2146d40e1abe05a2f1e4b853320dd2fa98cd7a4cf37de2c3 sandboxid=0515a4f16877ff7f2146d40e1abe05a2f1e4b853320dd2fa98cd7a4cf37de2c3 source=virtcontainers subsystem=sandbox
time="2018-09-27T19:25:55.416930684Z" level=debug arch=amd64 command=create container=0515a4f16877ff7f2146d40e1abe05a2f1e4b853320dd2fa98cd7a4cf37de2c3 default-kernel-parameters="tsc=reliable no_timer_check rcupdate.rcu_expedited=1 i8042.direct=1 i8042.dumbkbd=1 i8042.nopnp=1 i8042.noaux=1 noreplace-smp reboot=k console=hvc0 console=hvc1 iommu=off cryptomgr.notests net.ifnames=0 pci=lastbus=0 root=/dev/pmem0p1 rootflags=dax,data=ordered,errors=remount-ro rw rootfstype=ext4 debug systemd.show_status=true systemd.log_level=debug" name=kata-runtime pid=8579 source=virtcontainers subsystem=qemu
time="2018-09-27T19:25:59.717994008Z" level=error msg="Container 0515a4f16877ff7f2146d40e1abe05a2f1e4b853320dd2fa98cd7a4cf37de2c3 not ready, running or paused, cannot send a signal" arch=amd64 command=kill container=0515a4f16877ff7f2146d40e1abe05a2f1e4b853320dd2fa98cd7a4cf37de2c3 name=kata-runtime pid=8749 sandbox=0515a4f16877ff7f2146d40e1abe05a2f1e4b853320dd2fa98cd7a4cf37de2c3 source=runtime
time="2018-09-27T19:25:59.760568362Z" level=error msg="Container 0515a4f16877ff7f2146d40e1abe05a2f1e4b853320dd2fa98cd7a4cf37de2c3 not ready, running or paused, cannot send a signal" arch=amd64 command=kill container=0515a4f16877ff7f2146d40e1abe05a2f1e4b853320dd2fa98cd7a4cf37de2c3 name=kata-runtime pid=8772 sandbox=0515a4f16877ff7f2146d40e1abe05a2f1e4b853320dd2fa98cd7a4cf37de2c3 source=runtime
time="2018-09-27T19:26:00.709833831Z" level=debug msg="Could not retrieve anything from storage" arch=amd64 command=create container=29dcdb990202c8214387b22288df38dffd7384b7fc5258de86a65ee7bd34ed33 name=kata-runtime pid=8884 source=virtcontainers subsystem=kata_agent
time="2018-09-27T19:26:00.70990563Z" level=warning msg="fetch sandbox device failed" arch=amd64 command=create container=29dcdb990202c8214387b22288df38dffd7384b7fc5258de86a65ee7bd34ed33 error="open /run/vc/sbs/29dcdb990202c8214387b22288df38dffd7384b7fc5258de86a65ee7bd34ed33/devices.json: no such file or directory" name=kata-runtime pid=8884 sandbox=29dcdb990202c8214387b22288df38dffd7384b7fc5258de86a65ee7bd34ed33 sandboxid=29dcdb990202c8214387b22288df38dffd7384b7fc5258de86a65ee7bd34ed33 source=virtcontainers subsystem=sandbox
time="2018-09-27T19:26:00.712702027Z" level=debug arch=amd64 command=create container=29dcdb990202c8214387b22288df38dffd7384b7fc5258de86a65ee7bd34ed33 default-kernel-parameters="tsc=reliable no_timer_check rcupdate.rcu_expedited=1 i8042.direct=1 i8042.dumbkbd=1 i8042.nopnp=1 i8042.noaux=1 noreplace-smp reboot=k console=hvc0 console=hvc1 iommu=off cryptomgr.notests net.ifnames=0 pci=lastbus=0 root=/dev/pmem0p1 rootflags=dax,data=ordered,errors=remount-ro rw rootfstype=ext4 debug systemd.show_status=true systemd.log_level=debug" name=kata-runtime pid=8884 source=virtcontainers subsystem=qemu
time="2018-09-27T19:26:03.961191399Z" level=error msg="Container not ready, running or paused, impossible to signal the container" arch=amd64 command=kill container=29dcdb990202c8214387b22288df38dffd7384b7fc5258de86a65ee7bd34ed33 name=kata-runtime pid=9006 sandbox=29dcdb990202c8214387b22288df38dffd7384b7fc5258de86a65ee7bd34ed33 source=runtime
time="2018-09-27T19:26:04.016315539Z" level=error msg="Container 29dcdb990202c8214387b22288df38dffd7384b7fc5258de86a65ee7bd34ed33 not ready, running or paused, cannot send a signal" arch=amd64 command=kill container=29dcdb990202c8214387b22288df38dffd7384b7fc5258de86a65ee7bd34ed33 name=kata-runtime pid=9033 sandbox=29dcdb990202c8214387b22288df38dffd7384b7fc5258de86a65ee7bd34ed33 source=runtime
time="2018-09-27T19:27:12.833948336Z" level=debug msg="Could not retrieve anything from storage" arch=amd64 command=create container=eb7243603a837426640912f5dd9eb0aada8bd795596b6713a61545d9f3f005b7 name=kata-runtime pid=9595 source=virtcontainers subsystem=kata_agent
time="2018-09-27T19:27:12.834020236Z" level=warning msg="fetch sandbox device failed" arch=amd64 command=create container=eb7243603a837426640912f5dd9eb0aada8bd795596b6713a61545d9f3f005b7 error="open /run/vc/sbs/eb7243603a837426640912f5dd9eb0aada8bd795596b6713a61545d9f3f005b7/devices.json: no such file or directory" name=kata-runtime pid=9595 sandbox=eb7243603a837426640912f5dd9eb0aada8bd795596b6713a61545d9f3f005b7 sandboxid=eb7243603a837426640912f5dd9eb0aada8bd795596b6713a61545d9f3f005b7 source=virtcontainers subsystem=sandbox
time="2018-09-27T19:27:12.836809933Z" level=debug arch=amd64 command=create container=eb7243603a837426640912f5dd9eb0aada8bd795596b6713a61545d9f3f005b7 default-kernel-parameters="tsc=reliable no_timer_check rcupdate.rcu_expedited=1 i8042.direct=1 i8042.dumbkbd=1 i8042.nopnp=1 i8042.noaux=1 noreplace-smp reboot=k console=hvc0 console=hvc1 iommu=off cryptomgr.notests net.ifnames=0 pci=lastbus=0 root=/dev/pmem0p1 rootflags=dax,data=ordered,errors=remount-ro rw rootfstype=ext4 debug systemd.show_status=true systemd.log_level=debug" name=kata-runtime pid=9595 source=virtcontainers subsystem=qemu
time="2018-09-27T19:27:16.279662127Z" level=error msg="Container eb7243603a837426640912f5dd9eb0aada8bd795596b6713a61545d9f3f005b7 not ready, running or paused, cannot send a signal" arch=amd64 command=kill container=eb7243603a837426640912f5dd9eb0aada8bd795596b6713a61545d9f3f005b7 name=kata-runtime pid=9750 sandbox=eb7243603a837426640912f5dd9eb0aada8bd795596b6713a61545d9f3f005b7 source=runtime
time="2018-09-27T19:27:16.326669477Z" level=error msg="Container eb7243603a837426640912f5dd9eb0aada8bd795596b6713a61545d9f3f005b7 not ready, running or paused, cannot send a signal" arch=amd64 command=kill container=eb7243603a837426640912f5dd9eb0aada8bd795596b6713a61545d9f3f005b7 name=kata-runtime pid=9773 sandbox=eb7243603a837426640912f5dd9eb0aada8bd795596b6713a61545d9f3f005b7 source=runtime
time="2018-09-27T19:34:24.700976579Z" level=debug msg="Could not retrieve anything from storage" arch=amd64 command=create container=72b021ab0526582beedda2205f74c92fb86d5e39d2c97d098b7835310c99e405 name=kata-runtime pid=11758 source=virtcontainers subsystem=kata_agent
time="2018-09-27T19:34:24.701060779Z" level=warning msg="fetch sandbox device failed" arch=amd64 command=create container=72b021ab0526582beedda2205f74c92fb86d5e39d2c97d098b7835310c99e405 error="open /run/vc/sbs/72b021ab0526582beedda2205f74c92fb86d5e39d2c97d098b7835310c99e405/devices.json: no such file or directory" name=kata-runtime pid=11758 sandbox=72b021ab0526582beedda2205f74c92fb86d5e39d2c97d098b7835310c99e405 sandboxid=72b021ab0526582beedda2205f74c92fb86d5e39d2c97d098b7835310c99e405 source=virtcontainers subsystem=sandbox
time="2018-09-27T19:34:24.703600376Z" level=debug arch=amd64 command=create container=72b021ab0526582beedda2205f74c92fb86d5e39d2c97d098b7835310c99e405 default-kernel-parameters="tsc=reliable no_timer_check rcupdate.rcu_expedited=1 i8042.direct=1 i8042.dumbkbd=1 i8042.nopnp=1 i8042.noaux=1 noreplace-smp reboot=k console=hvc0 console=hvc1 iommu=off cryptomgr.notests net.ifnames=0 pci=lastbus=0 root=/dev/pmem0p1 rootflags=dax,data=ordered,errors=remount-ro rw rootfstype=ext4 debug systemd.show_status=true systemd.log_level=debug" name=kata-runtime pid=11758 source=virtcontainers subsystem=qemu
time="2018-09-27T19:34:27.970290584Z" level=error msg="Container 72b021ab0526582beedda2205f74c92fb86d5e39d2c97d098b7835310c99e405 not ready, running or paused, cannot send a signal" arch=amd64 command=kill container=72b021ab0526582beedda2205f74c92fb86d5e39d2c97d098b7835310c99e405 name=kata-runtime pid=11914 sandbox=72b021ab0526582beedda2205f74c92fb86d5e39d2c97d098b7835310c99e405 source=runtime
time="2018-09-27T19:34:28.017230735Z" level=error msg="Container 72b021ab0526582beedda2205f74c92fb86d5e39d2c97d098b7835310c99e405 not ready, running or paused, cannot send a signal" arch=amd64 command=kill container=72b021ab0526582beedda2205f74c92fb86d5e39d2c97d098b7835310c99e405 name=kata-runtime pid=11938 sandbox=72b021ab0526582beedda2205f74c92fb86d5e39d2c97d098b7835310c99e405 source=runtime
time="2018-09-27T19:36:39.494335461Z" level=debug msg="Could not retrieve anything from storage" arch=amd64 command=create container=4151a31551f2f5afbafb7338e4e1bc9fa37f8d4d43da83032c49ee42e6b67674 name=kata-runtime pid=12757 source=virtcontainers subsystem=kata_agent
time="2018-09-27T19:36:39.494406761Z" level=warning msg="fetch sandbox device failed" arch=amd64 command=create container=4151a31551f2f5afbafb7338e4e1bc9fa37f8d4d43da83032c49ee42e6b67674 error="open /run/vc/sbs/4151a31551f2f5afbafb7338e4e1bc9fa37f8d4d43da83032c49ee42e6b67674/devices.json: no such file or directory" name=kata-runtime pid=12757 sandbox=4151a31551f2f5afbafb7338e4e1bc9fa37f8d4d43da83032c49ee42e6b67674 sandboxid=4151a31551f2f5afbafb7338e4e1bc9fa37f8d4d43da83032c49ee42e6b67674 source=virtcontainers subsystem=sandbox
time="2018-09-27T19:36:39.497414257Z" level=debug arch=amd64 command=create container=4151a31551f2f5afbafb7338e4e1bc9fa37f8d4d43da83032c49ee42e6b67674 default-kernel-parameters="tsc=reliable no_timer_check rcupdate.rcu_expedited=1 i8042.direct=1 i8042.dumbkbd=1 i8042.nopnp=1 i8042.noaux=1 noreplace-smp reboot=k console=hvc0 console=hvc1 iommu=off cryptomgr.notests net.ifnames=0 pci=lastbus=0 root=/dev/pmem0p1 rootflags=dax,data=ordered,errors=remount-ro rw rootfstype=ext4 debug systemd.show_status=true systemd.log_level=debug" name=kata-runtime pid=12757 source=virtcontainers subsystem=qemu
time="2018-09-27T19:36:42.677372276Z" level=error msg="Container 4151a31551f2f5afbafb7338e4e1bc9fa37f8d4d43da83032c49ee42e6b67674 not ready, running or paused, cannot send a signal" arch=amd64 command=kill container=4151a31551f2f5afbafb7338e4e1bc9fa37f8d4d43da83032c49ee42e6b67674 name=kata-runtime pid=12886 sandbox=4151a31551f2f5afbafb7338e4e1bc9fa37f8d4d43da83032c49ee42e6b67674 source=runtime
time="2018-09-27T19:36:42.724551028Z" level=error msg="Container 4151a31551f2f5afbafb7338e4e1bc9fa37f8d4d43da83032c49ee42e6b67674 not ready, running or paused, cannot send a signal" arch=amd64 command=kill container=4151a31551f2f5afbafb7338e4e1bc9fa37f8d4d43da83032c49ee42e6b67674 name=kata-runtime pid=12912 sandbox=4151a31551f2f5afbafb7338e4e1bc9fa37f8d4d43da83032c49ee42e6b67674 source=runtime
time="2018-09-27T19:52:40.134174288Z" level=debug msg="Could not retrieve anything from storage" arch=amd64 command=create container=af3933ea5cf55053cc8dfacfa78bb5b91bb06dca34e5f7c087636b8ef3001246 name=kata-runtime pid=17056 source=virtcontainers subsystem=kata_agent
time="2018-09-27T19:52:40.134356188Z" level=warning msg="fetch sandbox device failed" arch=amd64 command=create container=af3933ea5cf55053cc8dfacfa78bb5b91bb06dca34e5f7c087636b8ef3001246 error="open /run/vc/sbs/af3933ea5cf55053cc8dfacfa78bb5b91bb06dca34e5f7c087636b8ef3001246/devices.json: no such file or directory" name=kata-runtime pid=17056 sandbox=af3933ea5cf55053cc8dfacfa78bb5b91bb06dca34e5f7c087636b8ef3001246 sandboxid=af3933ea5cf55053cc8dfacfa78bb5b91bb06dca34e5f7c087636b8ef3001246 source=virtcontainers subsystem=sandbox
time="2018-09-27T19:52:40.137140785Z" level=debug arch=amd64 command=create container=af3933ea5cf55053cc8dfacfa78bb5b91bb06dca34e5f7c087636b8ef3001246 default-kernel-parameters="tsc=reliable no_timer_check rcupdate.rcu_expedited=1 i8042.direct=1 i8042.dumbkbd=1 i8042.nopnp=1 i8042.noaux=1 noreplace-smp reboot=k console=hvc0 console=hvc1 iommu=off cryptomgr.notests net.ifnames=0 pci=lastbus=0 root=/dev/pmem0p1 rootflags=dax,data=ordered,errors=remount-ro rw rootfstype=ext4 debug systemd.show_status=true systemd.log_level=debug" name=kata-runtime pid=17056 source=virtcontainers subsystem=qemu
time="2018-09-27T19:52:43.426725652Z" level=error msg="Container af3933ea5cf55053cc8dfacfa78bb5b91bb06dca34e5f7c087636b8ef3001246 not ready, running or paused, cannot send a signal" arch=amd64 command=kill container=af3933ea5cf55053cc8dfacfa78bb5b91bb06dca34e5f7c087636b8ef3001246 name=kata-runtime pid=17213 sandbox=af3933ea5cf55053cc8dfacfa78bb5b91bb06dca34e5f7c087636b8ef3001246 source=runtime
time="2018-09-27T19:52:43.473424405Z" level=error msg="Container af3933ea5cf55053cc8dfacfa78bb5b91bb06dca34e5f7c087636b8ef3001246 not ready, running or paused, cannot send a signal" arch=amd64 command=kill container=af3933ea5cf55053cc8dfacfa78bb5b91bb06dca34e5f7c087636b8ef3001246 name=kata-runtime pid=17237 sandbox=af3933ea5cf55053cc8dfacfa78bb5b91bb06dca34e5f7c087636b8ef3001246 source=runtime

Proxy logs

Recent proxy problems found in system journal:

time="2018-09-27T19:34:27.284195896Z" level=info msg="time=\"2018-09-27T19:34:27.241148315Z\" level=debug msg=\"request end\" duration=62.952616ms name=kata-agent pid=119 request=/grpc.AgentService/StartContainer resp=\"&Empty{}\" sandbox=72b021ab0526582beedda2205f74c92fb86d5e39d2c97d098b7835310c99e405 source=agent\n" name=kata-proxy pid=11795 sandbox=72b021ab0526582beedda2205f74c92fb86d5e39d2c97d098b7835310c99e405 source=agent
time="2018-09-27T19:34:27.725193438Z" level=info msg="time=\"2018-09-27T19:34:27.684182437Z\" level=debug msg=\"request end\" duration=1.879416ms name=kata-agent pid=119 request=/grpc.AgentService/SignalProcess resp=\"&Empty{}\" sandbox=72b021ab0526582beedda2205f74c92fb86d5e39d2c97d098b7835310c99e405 source=agent\n" name=kata-proxy pid=11795 sandbox=72b021ab0526582beedda2205f74c92fb86d5e39d2c97d098b7835310c99e405 source=agent
time="2018-09-27T19:34:27.732585231Z" level=info msg="time=\"2018-09-27T19:34:27.691879639Z\" level=debug msg=\"request end\" duration=420.122753ms error=\"read /dev/ptmx: input/output error\" name=kata-agent pid=119 request=/grpc.AgentService/ReadStdout resp= sandbox=72b021ab0526582beedda2205f74c92fb86d5e39d2c97d098b7835310c99e405 source=agent\n" name=kata-proxy pid=11795 sandbox=72b021ab0526582beedda2205f74c92fb86d5e39d2c97d098b7835310c99e405 source=agent
time="2018-09-27T19:34:27.735620328Z" level=info msg="time=\"2018-09-27T19:34:27.694750839Z\" level=info msg=\"ignoring unexpected signal\" name=kata-agent pid=119 sandbox=72b021ab0526582beedda2205f74c92fb86d5e39d2c97d098b7835310c99e405 signal=\"child exited\" source=agent\n" name=kata-proxy pid=11795 sandbox=72b021ab0526582beedda2205f74c92fb86d5e39d2c97d098b7835310c99e405 source=agent
time="2018-09-27T19:34:27.833677926Z" level=info msg="time=\"2018-09-27T19:34:27.79306689Z\" level=debug msg=\"request end\" duration=1.02652ms name=kata-agent pid=119 request=/grpc.AgentService/SignalProcess resp=\"&Empty{}\" sandbox=72b021ab0526582beedda2205f74c92fb86d5e39d2c97d098b7835310c99e405 source=agent\n" name=kata-proxy pid=11795 sandbox=72b021ab0526582beedda2205f74c92fb86d5e39d2c97d098b7835310c99e405 source=agent
time="2018-09-27T19:34:27.838097921Z" level=info msg="time=\"2018-09-27T19:34:27.797319296Z\" level=debug msg=\"request end\" duration=\"4.715µs\" error=\"rpc error: code = NotFound desc = Process 72b021ab0526582beedda2205f74c92fb86d5e39d2c97d098b7835310c99e405 not found (container 72b021ab0526582beedda2205f74c92fb86d5e39d2c97d098b7835310c99e405)\" name=kata-agent pid=119 request=/grpc.AgentService/WaitProcess resp= sandbox=72b021ab0526582beedda2205f74c92fb86d5e39d2c97d098b7835310c99e405 source=agent\n" name=kata-proxy pid=11795 sandbox=72b021ab0526582beedda2205f74c92fb86d5e39d2c97d098b7835310c99e405 source=agent
time="2018-09-27T19:34:27.842447917Z" level=info msg="time=\"2018-09-27T19:34:27.80192582Z\" level=debug msg=\"request end\" duration=\"578.24µs\" name=kata-agent pid=119 request=/grpc.AgentService/RemoveContainer resp=\"&Empty{}\" sandbox=72b021ab0526582beedda2205f74c92fb86d5e39d2c97d098b7835310c99e405 source=agent\n" name=kata-proxy pid=11795 sandbox=72b021ab0526582beedda2205f74c92fb86d5e39d2c97d098b7835310c99e405 source=agent
time="2018-09-27T19:34:27.913535843Z" level=info msg="time=\"2018-09-27T19:34:27.872923365Z\" level=debug msg=\"request end\" duration=10.704102ms name=kata-agent pid=119 request=/grpc.AgentService/DestroySandbox resp=\"&Empty{}\" sandbox=72b021ab0526582beedda2205f74c92fb86d5e39d2c97d098b7835310c99e405 source=agent\n" name=kata-proxy pid=11795 sandbox=72b021ab0526582beedda2205f74c92fb86d5e39d2c97d098b7835310c99e405 source=agent
time="2018-09-27T19:34:27.914840542Z" level=fatal msg="failed to handle exit signal" error="close unix @->/run/vc/vm/72b021ab0526582beedda2205f74c92fb86d5e39d2c97d098b7835310c99e405/kata.sock: use of closed network connection" name=kata-proxy pid=11795 sandbox=72b021ab0526582beedda2205f74c92fb86d5e39d2c97d098b7835310c99e405 source=proxy
time="2018-09-27T19:36:41.037371568Z" level=info msg="[    0.417353] EXT4-fs (pmem0p1): DAX enabled. Warning: EXPERIMENTAL, use at your own risk\n" name=kata-proxy pid=12795 sandbox=4151a31551f2f5afbafb7338e4e1bc9fa37f8d4d43da83032c49ee42e6b67674 source=agent
time="2018-09-27T19:36:41.039086967Z" level=info msg="[    0.418863] EXT4-fs (pmem0p1): mounted filesystem with ordered data mode. Opts: dax,data=ordered,errors=remount-ro\n" name=kata-proxy pid=12795 sandbox=4151a31551f2f5afbafb7338e4e1bc9fa37f8d4d43da83032c49ee42e6b67674 source=agent
time="2018-09-27T19:36:41.089096415Z" level=info msg="[    0.468850] systemd-gpt-auto-generator[68]: Failed to chase block device '/', ignoring: No such file or directory\n" name=kata-proxy pid=12795 sandbox=4151a31551f2f5afbafb7338e4e1bc9fa37f8d4d43da83032c49ee42e6b67674 source=agent
time="2018-09-27T19:36:41.138651964Z" level=info msg="[    0.518779] systemd[69]: tmp.mount: Failed to execute command: No such file or directory\n" name=kata-proxy pid=12795 sandbox=4151a31551f2f5afbafb7338e4e1bc9fa37f8d4d43da83032c49ee42e6b67674 source=agent
time="2018-09-27T19:36:41.163581838Z" level=info msg="[    0.543370] systemd-journald[77]: Failed to open configuration file '/etc/systemd/journald.conf': No such file or directory\n" name=kata-proxy pid=12795 sandbox=4151a31551f2f5afbafb7338e4e1bc9fa37f8d4d43da83032c49ee42e6b67674 source=agent
time="2018-09-27T19:36:41.187177514Z" level=info msg="[\x1b[0;1;31mFAILED\x1b[0m] Failed to mount Temporary Directory (/tmp).\n" name=kata-proxy pid=12795 sandbox=4151a31551f2f5afbafb7338e4e1bc9fa37f8d4d43da83032c49ee42e6b67674 source=agent
time="2018-09-27T19:36:41.192168809Z" level=info msg="[\x1b[0;1;33mDEPEND\x1b[0m] Dependency failed for Network Time Synchronization.\n" name=kata-proxy pid=12795 sandbox=4151a31551f2f5afbafb7338e4e1bc9fa37f8d4d43da83032c49ee42e6b67674 source=agent
time="2018-09-27T19:36:41.563601025Z" level=info msg="time=\"2018-09-27T19:36:41.544100387Z\" level=debug msg=\"request end\" duration=24.738298ms name=kata-agent pid=107 request=/grpc.AgentService/CreateSandbox resp=\"&Empty{}\" sandbox=4151a31551f2f5afbafb7338e4e1bc9fa37f8d4d43da83032c49ee42e6b67674 source=agent\n" name=kata-proxy pid=12795 sandbox=4151a31551f2f5afbafb7338e4e1bc9fa37f8d4d43da83032c49ee42e6b67674 source=agent
time="2018-09-27T19:36:41.841540739Z" level=info msg="time=\"2018-09-27T19:36:41.821986755Z\" level=debug msg=\"request end\" duration=201.268709ms name=kata-agent pid=107 request=/grpc.AgentService/CreateContainer resp=\"&Empty{}\" sandbox=4151a31551f2f5afbafb7338e4e1bc9fa37f8d4d43da83032c49ee42e6b67674 source=agent\n" name=kata-proxy pid=12795 sandbox=4151a31551f2f5afbafb7338e4e1bc9fa37f8d4d43da83032c49ee42e6b67674 source=agent
time="2018-09-27T19:36:41.843897036Z" level=info msg="time=\"2018-09-27T19:36:41.823503552Z\" level=info msg=\"ignoring unexpected signal\" name=kata-agent pid=107 sandbox=4151a31551f2f5afbafb7338e4e1bc9fa37f8d4d43da83032c49ee42e6b67674 signal=\"child exited\" source=agent\n" name=kata-proxy pid=12795 sandbox=4151a31551f2f5afbafb7338e4e1bc9fa37f8d4d43da83032c49ee42e6b67674 source=agent
time="2018-09-27T19:36:41.849261431Z" level=debug msg="Copy stream error" error="read unix /run/vc/sbs/4151a31551f2f5afbafb7338e4e1bc9fa37f8d4d43da83032c49ee42e6b67674/proxy.sock->@: read: connection reset by peer" name=kata-proxy pid=12795 sandbox=4151a31551f2f5afbafb7338e4e1bc9fa37f8d4d43da83032c49ee42e6b67674 source=proxy
time="2018-09-27T19:36:41.962672414Z" level=info msg="time=\"2018-09-27T19:36:41.943103282Z\" level=debug msg=\"request end\" duration=57.429587ms name=kata-agent pid=107 request=/grpc.AgentService/StartContainer resp=\"&Empty{}\" sandbox=4151a31551f2f5afbafb7338e4e1bc9fa37f8d4d43da83032c49ee42e6b67674 source=agent\n" name=kata-proxy pid=12795 sandbox=4151a31551f2f5afbafb7338e4e1bc9fa37f8d4d43da83032c49ee42e6b67674 source=agent
time="2018-09-27T19:36:42.427385834Z" level=info msg="time=\"2018-09-27T19:36:42.40514046Z\" level=debug msg=\"request end\" duration=\"533.144µs\" name=kata-agent pid=107 request=/grpc.AgentService/SignalProcess resp=\"&Empty{}\" sandbox=4151a31551f2f5afbafb7338e4e1bc9fa37f8d4d43da83032c49ee42e6b67674 source=agent\n" name=kata-proxy pid=12795 sandbox=4151a31551f2f5afbafb7338e4e1bc9fa37f8d4d43da83032c49ee42e6b67674 source=agent
time="2018-09-27T19:36:42.432426629Z" level=info msg="time=\"2018-09-27T19:36:42.412327816Z\" level=debug msg=\"request end\" duration=423.151377ms error=\"read /dev/ptmx: input/output error\" name=kata-agent pid=107 request=/grpc.AgentService/ReadStdout resp= sandbox=4151a31551f2f5afbafb7338e4e1bc9fa37f8d4d43da83032c49ee42e6b67674 source=agent\n" name=kata-proxy pid=12795 sandbox=4151a31551f2f5afbafb7338e4e1bc9fa37f8d4d43da83032c49ee42e6b67674 source=agent
time="2018-09-27T19:36:42.439401422Z" level=info msg="time=\"2018-09-27T19:36:42.419668538Z\" level=info msg=\"ignoring unexpected signal\" name=kata-agent pid=107 sandbox=4151a31551f2f5afbafb7338e4e1bc9fa37f8d4d43da83032c49ee42e6b67674 signal=\"child exited\" source=agent\n" name=kata-proxy pid=12795 sandbox=4151a31551f2f5afbafb7338e4e1bc9fa37f8d4d43da83032c49ee42e6b67674 source=agent
time="2018-09-27T19:36:42.544418714Z" level=info msg="time=\"2018-09-27T19:36:42.524752912Z\" level=debug msg=\"request end\" duration=\"702.664µs\" name=kata-agent pid=107 request=/grpc.AgentService/SignalProcess resp=\"&Empty{}\" sandbox=4151a31551f2f5afbafb7338e4e1bc9fa37f8d4d43da83032c49ee42e6b67674 source=agent\n" name=kata-proxy pid=12795 sandbox=4151a31551f2f5afbafb7338e4e1bc9fa37f8d4d43da83032c49ee42e6b67674 source=agent
time="2018-09-27T19:36:42.545897812Z" level=debug msg="Copy stream error" error="write unix /run/vc/sbs/4151a31551f2f5afbafb7338e4e1bc9fa37f8d4d43da83032c49ee42e6b67674/proxy.sock->@: use of closed network connection" name=kata-proxy pid=12795 sandbox=4151a31551f2f5afbafb7338e4e1bc9fa37f8d4d43da83032c49ee42e6b67674 source=proxy
time="2018-09-27T19:36:42.550116308Z" level=info msg="time=\"2018-09-27T19:36:42.530000992Z\" level=debug msg=\"request end\" duration=\"6.926µs\" error=\"rpc error: code = NotFound desc = Process 4151a31551f2f5afbafb7338e4e1bc9fa37f8d4d43da83032c49ee42e6b67674 not found (container 4151a31551f2f5afbafb7338e4e1bc9fa37f8d4d43da83032c49ee42e6b67674)\" name=kata-agent pid=107 request=/grpc.AgentService/WaitProcess resp= sandbox=4151a31551f2f5afbafb7338e4e1bc9fa37f8d4d43da83032c49ee42e6b67674 source=agent\n" name=kata-proxy pid=12795 sandbox=4151a31551f2f5afbafb7338e4e1bc9fa37f8d4d43da83032c49ee42e6b67674 source=agent
time="2018-09-27T19:36:42.556516101Z" level=info msg="time=\"2018-09-27T19:36:42.537086171Z\" level=debug msg=\"request end\" duration=\"479.85µs\" name=kata-agent pid=107 request=/grpc.AgentService/RemoveContainer resp=\"&Empty{}\" sandbox=4151a31551f2f5afbafb7338e4e1bc9fa37f8d4d43da83032c49ee42e6b67674 source=agent\n" name=kata-proxy pid=12795 sandbox=4151a31551f2f5afbafb7338e4e1bc9fa37f8d4d43da83032c49ee42e6b67674 source=agent
time="2018-09-27T19:36:42.626727929Z" level=info msg="time=\"2018-09-27T19:36:42.607094036Z\" level=debug msg=\"request end\" duration=10.693327ms name=kata-agent pid=107 request=/grpc.AgentService/DestroySandbox resp=\"&Empty{}\" sandbox=4151a31551f2f5afbafb7338e4e1bc9fa37f8d4d43da83032c49ee42e6b67674 source=agent\n" name=kata-proxy pid=12795 sandbox=4151a31551f2f5afbafb7338e4e1bc9fa37f8d4d43da83032c49ee42e6b67674 source=agent
time="2018-09-27T19:36:42.627910127Z" level=fatal msg="failed to handle exit signal" error="close unix @->/run/vc/vm/4151a31551f2f5afbafb7338e4e1bc9fa37f8d4d43da83032c49ee42e6b67674/kata.sock: use of closed network connection" name=kata-proxy pid=12795 sandbox=4151a31551f2f5afbafb7338e4e1bc9fa37f8d4d43da83032c49ee42e6b67674 source=proxy
time="2018-09-27T19:52:41.666946435Z" level=info msg="[    0.414858] EXT4-fs (pmem0p1): DAX enabled. Warning: EXPERIMENTAL, use at your own risk\n" name=kata-proxy pid=17093 sandbox=af3933ea5cf55053cc8dfacfa78bb5b91bb06dca34e5f7c087636b8ef3001246 source=agent
time="2018-09-27T19:52:41.668734834Z" level=info msg="[    0.416538] EXT4-fs (pmem0p1): mounted filesystem with ordered data mode. Opts: dax,data=ordered,errors=remount-ro\n" name=kata-proxy pid=17093 sandbox=af3933ea5cf55053cc8dfacfa78bb5b91bb06dca34e5f7c087636b8ef3001246 source=agent
time="2018-09-27T19:52:41.727487074Z" level=info msg="[    0.475293] systemd-gpt-auto-generator[68]: Failed to chase block device '/', ignoring: No such file or directory\n" name=kata-proxy pid=17093 sandbox=af3933ea5cf55053cc8dfacfa78bb5b91bb06dca34e5f7c087636b8ef3001246 source=agent
time="2018-09-27T19:52:41.781470819Z" level=info msg="[    0.529295] systemd-journald[74]: Failed to open configuration file '/etc/systemd/journald.conf': No such file or directory\n" name=kata-proxy pid=17093 sandbox=af3933ea5cf55053cc8dfacfa78bb5b91bb06dca34e5f7c087636b8ef3001246 source=agent
time="2018-09-27T19:52:41.845397055Z" level=info msg="[    0.593326] systemd[83]: tmp.mount: Failed to execute command: No such file or directory\n" name=kata-proxy pid=17093 sandbox=af3933ea5cf55053cc8dfacfa78bb5b91bb06dca34e5f7c087636b8ef3001246 source=agent
time="2018-09-27T19:52:41.876894123Z" level=info msg="[\x1b[0;1;31mFAILED\x1b[0m] Failed to mount Temporary Directory (/tmp).\n" name=kata-proxy pid=17093 sandbox=af3933ea5cf55053cc8dfacfa78bb5b91bb06dca34e5f7c087636b8ef3001246 source=agent
time="2018-09-27T19:52:41.881546618Z" level=info msg="[\x1b[0;1;33mDEPEND\x1b[0m] Dependency failed for Network Time Synchronization.\n" name=kata-proxy pid=17093 sandbox=af3933ea5cf55053cc8dfacfa78bb5b91bb06dca34e5f7c087636b8ef3001246 source=agent
time="2018-09-27T19:52:42.236447758Z" level=info msg="time=\"2018-09-27T19:52:42.218838977Z\" level=debug msg=\"request end\" duration=23.161298ms name=kata-agent pid=112 request=/grpc.AgentService/CreateSandbox resp=\"&Empty{}\" sandbox=af3933ea5cf55053cc8dfacfa78bb5b91bb06dca34e5f7c087636b8ef3001246 source=agent\n" name=kata-proxy pid=17093 sandbox=af3933ea5cf55053cc8dfacfa78bb5b91bb06dca34e5f7c087636b8ef3001246 source=agent
time="2018-09-27T19:52:42.526291665Z" level=info msg="time=\"2018-09-27T19:52:42.508565542Z\" level=debug msg=\"request end\" duration=213.425761ms name=kata-agent pid=112 request=/grpc.AgentService/CreateContainer resp=\"&Empty{}\" sandbox=af3933ea5cf55053cc8dfacfa78bb5b91bb06dca34e5f7c087636b8ef3001246 source=agent\n" name=kata-proxy pid=17093 sandbox=af3933ea5cf55053cc8dfacfa78bb5b91bb06dca34e5f7c087636b8ef3001246 source=agent
time="2018-09-27T19:52:42.528673762Z" level=info msg="time=\"2018-09-27T19:52:42.510227892Z\" level=info msg=\"ignoring unexpected signal\" name=kata-agent pid=112 sandbox=af3933ea5cf55053cc8dfacfa78bb5b91bb06dca34e5f7c087636b8ef3001246 signal=\"child exited\" source=agent\n" name=kata-proxy pid=17093 sandbox=af3933ea5cf55053cc8dfacfa78bb5b91bb06dca34e5f7c087636b8ef3001246 source=agent
time="2018-09-27T19:52:42.532211159Z" level=debug msg="Copy stream error" error="read unix /run/vc/sbs/af3933ea5cf55053cc8dfacfa78bb5b91bb06dca34e5f7c087636b8ef3001246/proxy.sock->@: read: connection reset by peer" name=kata-proxy pid=17093 sandbox=af3933ea5cf55053cc8dfacfa78bb5b91bb06dca34e5f7c087636b8ef3001246 source=proxy
time="2018-09-27T19:52:42.684372304Z" level=info msg="time=\"2018-09-27T19:52:42.666585285Z\" level=debug msg=\"request end\" duration=57.92026ms name=kata-agent pid=112 request=/grpc.AgentService/StartContainer resp=\"&Empty{}\" sandbox=af3933ea5cf55053cc8dfacfa78bb5b91bb06dca34e5f7c087636b8ef3001246 source=agent\n" name=kata-proxy pid=17093 sandbox=af3933ea5cf55053cc8dfacfa78bb5b91bb06dca34e5f7c087636b8ef3001246 source=agent
time="2018-09-27T19:52:43.183183899Z" level=info msg="time=\"2018-09-27T19:52:43.164010097Z\" level=debug msg=\"request end\" duration=1.157722ms name=kata-agent pid=112 request=/grpc.AgentService/SignalProcess resp=\"&Empty{}\" sandbox=af3933ea5cf55053cc8dfacfa78bb5b91bb06dca34e5f7c087636b8ef3001246 source=agent\n" name=kata-proxy pid=17093 sandbox=af3933ea5cf55053cc8dfacfa78bb5b91bb06dca34e5f7c087636b8ef3001246 source=agent
time="2018-09-27T19:52:43.188519294Z" level=info msg="time=\"2018-09-27T19:52:43.169654065Z\" level=debug msg=\"request end\" duration=417.459148ms error=\"read /dev/ptmx: input/output error\" name=kata-agent pid=112 request=/grpc.AgentService/ReadStdout resp= sandbox=af3933ea5cf55053cc8dfacfa78bb5b91bb06dca34e5f7c087636b8ef3001246 source=agent\n" name=kata-proxy pid=17093 sandbox=af3933ea5cf55053cc8dfacfa78bb5b91bb06dca34e5f7c087636b8ef3001246 source=agent
time="2018-09-27T19:52:43.190352292Z" level=info msg="time=\"2018-09-27T19:52:43.17253144Z\" level=info msg=\"ignoring unexpected signal\" name=kata-agent pid=112 sandbox=af3933ea5cf55053cc8dfacfa78bb5b91bb06dca34e5f7c087636b8ef3001246 signal=\"child exited\" source=agent\n" name=kata-proxy pid=17093 sandbox=af3933ea5cf55053cc8dfacfa78bb5b91bb06dca34e5f7c087636b8ef3001246 source=agent
time="2018-09-27T19:52:43.295562685Z" level=info msg="time=\"2018-09-27T19:52:43.277923421Z\" level=debug msg=\"request end\" duration=\"855.534µs\" name=kata-agent pid=112 request=/grpc.AgentService/SignalProcess resp=\"&Empty{}\" sandbox=af3933ea5cf55053cc8dfacfa78bb5b91bb06dca34e5f7c087636b8ef3001246 source=agent\n" name=kata-proxy pid=17093 sandbox=af3933ea5cf55053cc8dfacfa78bb5b91bb06dca34e5f7c087636b8ef3001246 source=agent
time="2018-09-27T19:52:43.301323779Z" level=info msg="time=\"2018-09-27T19:52:43.283308416Z\" level=debug msg=\"request end\" duration=\"5.392µs\" error=\"rpc error: code = NotFound desc = Process af3933ea5cf55053cc8dfacfa78bb5b91bb06dca34e5f7c087636b8ef3001246 not found (container af3933ea5cf55053cc8dfacfa78bb5b91bb06dca34e5f7c087636b8ef3001246)\" name=kata-agent pid=112 request=/grpc.AgentService/WaitProcess resp= sandbox=af3933ea5cf55053cc8dfacfa78bb5b91bb06dca34e5f7c087636b8ef3001246 source=agent\n" name=kata-proxy pid=17093 sandbox=af3933ea5cf55053cc8dfacfa78bb5b91bb06dca34e5f7c087636b8ef3001246 source=agent
time="2018-09-27T19:52:43.308400572Z" level=info msg="time=\"2018-09-27T19:52:43.290683457Z\" level=debug msg=\"request end\" duration=\"971.064µs\" name=kata-agent pid=112 request=/grpc.AgentService/RemoveContainer resp=\"&Empty{}\" sandbox=af3933ea5cf55053cc8dfacfa78bb5b91bb06dca34e5f7c087636b8ef3001246 source=agent\n" name=kata-proxy pid=17093 sandbox=af3933ea5cf55053cc8dfacfa78bb5b91bb06dca34e5f7c087636b8ef3001246 source=agent
time="2018-09-27T19:52:43.3794305Z" level=info msg="time=\"2018-09-27T19:52:43.361866113Z\" level=debug msg=\"request end\" duration=9.035406ms name=kata-agent pid=112 request=/grpc.AgentService/DestroySandbox resp=\"&Empty{}\" sandbox=af3933ea5cf55053cc8dfacfa78bb5b91bb06dca34e5f7c087636b8ef3001246 source=agent\n" name=kata-proxy pid=17093 sandbox=af3933ea5cf55053cc8dfacfa78bb5b91bb06dca34e5f7c087636b8ef3001246 source=agent
time="2018-09-27T19:52:43.380433999Z" level=fatal msg="failed to handle exit signal" error="close unix @->/run/vc/vm/af3933ea5cf55053cc8dfacfa78bb5b91bb06dca34e5f7c087636b8ef3001246/kata.sock: use of closed network connection" name=kata-proxy pid=17093 sandbox=af3933ea5cf55053cc8dfacfa78bb5b91bb06dca34e5f7c087636b8ef3001246 source=proxy

Shim logs

Recent shim problems found in system journal:

time="2018-09-27T19:24:07.908699123Z" level=info msg="copy stdout failed" container=3bb7b68b162242d4f5ec192774908acb00feefe4a61bbf18cbb05419784bd470 error="rpc error: code = Unknown desc = EOF" exec-id=3bb7b68b162242d4f5ec192774908acb00feefe4a61bbf18cbb05419784bd470 name=kata-shim pid=1 source=shim
time="2018-09-27T19:24:08.374908707Z" level=info msg="copy stderr failed" container=6672dda770c68c18fec5ea2167bf766b69de66819680a17d962a4b7a2f4b9dd4 error="rpc error: code = Unknown desc = EOF" exec-id=6672dda770c68c18fec5ea2167bf766b69de66819680a17d962a4b7a2f4b9dd4 name=kata-shim pid=1 source=shim
time="2018-09-27T19:24:08.393468787Z" level=info msg="copy stdout failed" container=6672dda770c68c18fec5ea2167bf766b69de66819680a17d962a4b7a2f4b9dd4 error="rpc error: code = Unknown desc = EOF" exec-id=6672dda770c68c18fec5ea2167bf766b69de66819680a17d962a4b7a2f4b9dd4 name=kata-shim pid=1 source=shim
time="2018-09-27T19:24:12.677840553Z" level=info msg="copy stdout failed" container=c85054b6b9c27dd1c7005efdab828a5f65bf2e51f98c2176021bb8372001b008 error="rpc error: code = Unknown desc = read /dev/ptmx: input/output error" exec-id=c85054b6b9c27dd1c7005efdab828a5f65bf2e51f98c2176021bb8372001b008 name=kata-shim pid=1 source=shim
time="2018-09-27T19:24:16.815666784Z" level=info msg="copy stdout failed" container=f88eef2eda5907ea96dc8bacd39915b49ebe4ed255426a6c0876da9959a17592 error="rpc error: code = Unknown desc = EOF" exec-id=f88eef2eda5907ea96dc8bacd39915b49ebe4ed255426a6c0876da9959a17592 name=kata-shim pid=1 source=shim
time="2018-09-27T19:24:16.817520682Z" level=info msg="copy stderr failed" container=f88eef2eda5907ea96dc8bacd39915b49ebe4ed255426a6c0876da9959a17592 error="rpc error: code = Unknown desc = EOF" exec-id=f88eef2eda5907ea96dc8bacd39915b49ebe4ed255426a6c0876da9959a17592 name=kata-shim pid=1 source=shim
time="2018-09-27T19:24:20.874035106Z" level=info msg="copy stdout failed" container=b48911bafc19288aa5ea79f132a68c2197b3d2d8d6763b69f24a2b81f187d8cc error="rpc error: code = Unknown desc = read /dev/ptmx: input/output error" exec-id=b48911bafc19288aa5ea79f132a68c2197b3d2d8d6763b69f24a2b81f187d8cc name=kata-shim pid=1 source=shim
time="2018-09-27T19:24:36.232838885Z" level=info msg="copy stdout failed" container=edd49129f438b1d48e86a556b4f7f0c79e04df3950e8a450d8ea66132a1f4cb3 error="rpc error: code = Unknown desc = EOF" exec-id=edd49129f438b1d48e86a556b4f7f0c79e04df3950e8a450d8ea66132a1f4cb3 name=kata-shim pid=1 source=shim
time="2018-09-27T19:24:36.234216283Z" level=info msg="copy stderr failed" container=edd49129f438b1d48e86a556b4f7f0c79e04df3950e8a450d8ea66132a1f4cb3 error="rpc error: code = Unknown desc = EOF" exec-id=edd49129f438b1d48e86a556b4f7f0c79e04df3950e8a450d8ea66132a1f4cb3 name=kata-shim pid=1 source=shim
time="2018-09-27T19:24:41.31996629Z" level=info msg="copy stdout failed" container=d27dbd348315ebae3e1f0cb35f7f7e1f6783bb3083ea44c6f980760a5d08eb2f error="rpc error: code = Unknown desc = EOF" exec-id=d27dbd348315ebae3e1f0cb35f7f7e1f6783bb3083ea44c6f980760a5d08eb2f name=kata-shim pid=1 source=shim
time="2018-09-27T19:24:41.322099087Z" level=info msg="copy stderr failed" container=d27dbd348315ebae3e1f0cb35f7f7e1f6783bb3083ea44c6f980760a5d08eb2f error="rpc error: code = Unknown desc = EOF" exec-id=d27dbd348315ebae3e1f0cb35f7f7e1f6783bb3083ea44c6f980760a5d08eb2f name=kata-shim pid=1 source=shim
time="2018-09-27T19:24:50.844935426Z" level=info msg="copy stderr failed" container=d146a3e395a3e8cda165cfdd5768ad465a9b30792caf05772d39751d3d9c7428 error="rpc error: code = Unknown desc = EOF" exec-id=d146a3e395a3e8cda165cfdd5768ad465a9b30792caf05772d39751d3d9c7428 name=kata-shim pid=1 source=shim
time="2018-09-27T19:24:50.845037825Z" level=info msg="copy stdout failed" container=d146a3e395a3e8cda165cfdd5768ad465a9b30792caf05772d39751d3d9c7428 error="rpc error: code = Unknown desc = EOF" exec-id=d146a3e395a3e8cda165cfdd5768ad465a9b30792caf05772d39751d3d9c7428 name=kata-shim pid=1 source=shim
time="2018-09-27T19:24:56.276018366Z" level=info msg="copy stdout failed" container=b8b4fbc8b817514b9f0c91c5509fb32f42659f25d4acea40c31634c7b8d4fc5a error="rpc error: code = Unknown desc = EOF" exec-id=b8b4fbc8b817514b9f0c91c5509fb32f42659f25d4acea40c31634c7b8d4fc5a name=kata-shim pid=1 source=shim
time="2018-09-27T19:24:56.278252964Z" level=info msg="copy stderr failed" container=b8b4fbc8b817514b9f0c91c5509fb32f42659f25d4acea40c31634c7b8d4fc5a error="rpc error: code = Unknown desc = EOF" exec-id=b8b4fbc8b817514b9f0c91c5509fb32f42659f25d4acea40c31634c7b8d4fc5a name=kata-shim pid=1 source=shim
time="2018-09-27T19:24:57.054944112Z" level=info msg="copy stderr failed" container=c34364ee397a95c70b72efda4b96a3eb1d872093dfcf77f0abe151a7bab7b410 error="rpc error: code = Unknown desc = EOF" exec-id=c34364ee397a95c70b72efda4b96a3eb1d872093dfcf77f0abe151a7bab7b410 name=kata-shim pid=1 source=shim
time="2018-09-27T19:24:57.076148988Z" level=info msg="copy stdout failed" container=c34364ee397a95c70b72efda4b96a3eb1d872093dfcf77f0abe151a7bab7b410 error="rpc error: code = Unknown desc = EOF" exec-id=c34364ee397a95c70b72efda4b96a3eb1d872093dfcf77f0abe151a7bab7b410 name=kata-shim pid=1 source=shim
time="2018-09-27T19:25:01.853519951Z" level=info msg="copy stderr failed" container=0ba0edc6045eeb33c712f56336fcf5cd69838d0d5149bceb5f6e839dd3126f71 error="rpc error: code = Unknown desc = EOF" exec-id=0ba0edc6045eeb33c712f56336fcf5cd69838d0d5149bceb5f6e839dd3126f71 name=kata-shim pid=1 source=shim
time="2018-09-27T19:25:01.874252728Z" level=info msg="copy stdout failed" container=0ba0edc6045eeb33c712f56336fcf5cd69838d0d5149bceb5f6e839dd3126f71 error="rpc error: code = Unknown desc = EOF" exec-id=0ba0edc6045eeb33c712f56336fcf5cd69838d0d5149bceb5f6e839dd3126f71 name=kata-shim pid=1 source=shim
time="2018-09-27T19:25:06.837578491Z" level=info msg="copy stdout failed" container=5065e0a68ff1a3ad2fc67003c976dbf25d01dc6e43971d53d6d87788a578c439 error="rpc error: code = Unknown desc = EOF" exec-id=5065e0a68ff1a3ad2fc67003c976dbf25d01dc6e43971d53d6d87788a578c439 name=kata-shim pid=1 source=shim
time="2018-09-27T19:25:06.837582691Z" level=info msg="copy stderr failed" container=5065e0a68ff1a3ad2fc67003c976dbf25d01dc6e43971d53d6d87788a578c439 error="rpc error: code = Unknown desc = EOF" exec-id=5065e0a68ff1a3ad2fc67003c976dbf25d01dc6e43971d53d6d87788a578c439 name=kata-shim pid=1 source=shim
time="2018-09-27T19:25:11.47893501Z" level=info msg="copy stdout failed" container=6ceb47bdc0cb96babe08b202813798650614de90ad5ebe1ab8e6f5b294c79e21 error="rpc error: code = Unknown desc = EOF" exec-id=6ceb47bdc0cb96babe08b202813798650614de90ad5ebe1ab8e6f5b294c79e21 name=kata-shim pid=1 source=shim
time="2018-09-27T19:25:11.480601908Z" level=info msg="copy stderr failed" container=6ceb47bdc0cb96babe08b202813798650614de90ad5ebe1ab8e6f5b294c79e21 error="rpc error: code = Unknown desc = EOF" exec-id=6ceb47bdc0cb96babe08b202813798650614de90ad5ebe1ab8e6f5b294c79e21 name=kata-shim pid=1 source=shim
time="2018-09-27T19:25:11.808952749Z" level=info msg="copy stdout failed" container=78a0d5a21d9dce3c4fb6d83799c16cadf78b6ee01664f2e556bfec793861d117 error="rpc error: code = Unknown desc = EOF" exec-id=78a0d5a21d9dce3c4fb6d83799c16cadf78b6ee01664f2e556bfec793861d117 name=kata-shim pid=1 source=shim
time="2018-09-27T19:25:11.810536647Z" level=info msg="copy stderr failed" container=78a0d5a21d9dce3c4fb6d83799c16cadf78b6ee01664f2e556bfec793861d117 error="rpc error: code = Unknown desc = EOF" exec-id=78a0d5a21d9dce3c4fb6d83799c16cadf78b6ee01664f2e556bfec793861d117 name=kata-shim pid=1 source=shim
time="2018-09-27T19:25:16.398502229Z" level=info msg="copy stdout failed" container=747f8a7032d10419288bec27b5c6dbc09d42ca6432b2685d579fca77cf9e4f54 error="rpc error: code = Unknown desc = EOF" exec-id=747f8a7032d10419288bec27b5c6dbc09d42ca6432b2685d579fca77cf9e4f54 name=kata-shim pid=1 source=shim
time="2018-09-27T19:25:16.400577826Z" level=info msg="copy stderr failed" container=747f8a7032d10419288bec27b5c6dbc09d42ca6432b2685d579fca77cf9e4f54 error="rpc error: code = Unknown desc = EOF" exec-id=747f8a7032d10419288bec27b5c6dbc09d42ca6432b2685d579fca77cf9e4f54 name=kata-shim pid=1 source=shim
time="2018-09-27T19:25:16.848842536Z" level=info msg="copy stderr failed" container=57de31eb303c813b487907f0532a556712aa2958d24648b2771507c49ed2cfb9 error="rpc error: code = Unknown desc = EOF" exec-id=57de31eb303c813b487907f0532a556712aa2958d24648b2771507c49ed2cfb9 name=kata-shim pid=1 source=shim
time="2018-09-27T19:25:16.858275826Z" level=info msg="copy stdout failed" container=57de31eb303c813b487907f0532a556712aa2958d24648b2771507c49ed2cfb9 error="rpc error: code = Unknown desc = EOF" exec-id=57de31eb303c813b487907f0532a556712aa2958d24648b2771507c49ed2cfb9 name=kata-shim pid=1 source=shim
time="2018-09-27T19:25:21.611411731Z" level=info msg="copy stderr failed" container=4142077ed729480c91e4c49df986990f86e51dd75d8570da375806abb2bd30af error="rpc error: code = Unknown desc = EOF" exec-id=4142077ed729480c91e4c49df986990f86e51dd75d8570da375806abb2bd30af name=kata-shim pid=1 source=shim
time="2018-09-27T19:25:21.626361314Z" level=info msg="copy stdout failed" container=4142077ed729480c91e4c49df986990f86e51dd75d8570da375806abb2bd30af error="rpc error: code = Unknown desc = EOF" exec-id=4142077ed729480c91e4c49df986990f86e51dd75d8570da375806abb2bd30af name=kata-shim pid=1 source=shim
time="2018-09-27T19:25:26.392702008Z" level=info msg="copy stderr failed" container=1ea60254a3e5a3a9ea6ffd4dc3ec0994b932895dfa43d0c18905aa7cf1771841 error="rpc error: code = Unknown desc = EOF" exec-id=1ea60254a3e5a3a9ea6ffd4dc3ec0994b932895dfa43d0c18905aa7cf1771841 name=kata-shim pid=1 source=shim
time="2018-09-27T19:25:26.408614591Z" level=info msg="copy stdout failed" container=1ea60254a3e5a3a9ea6ffd4dc3ec0994b932895dfa43d0c18905aa7cf1771841 error="rpc error: code = Unknown desc = EOF" exec-id=1ea60254a3e5a3a9ea6ffd4dc3ec0994b932895dfa43d0c18905aa7cf1771841 name=kata-shim pid=1 source=shim
time="2018-09-27T19:25:31.01799506Z" level=info msg="copy stderr failed" container=4aa0bb5b6072038d5b8a1f351731c2654a87a14519df97a84f5a997fcd94fd63 error="rpc error: code = Unknown desc = EOF" exec-id=4aa0bb5b6072038d5b8a1f351731c2654a87a14519df97a84f5a997fcd94fd63 name=kata-shim pid=1 source=shim
time="2018-09-27T19:25:31.023991153Z" level=info msg="copy stdout failed" container=4aa0bb5b6072038d5b8a1f351731c2654a87a14519df97a84f5a997fcd94fd63 error="rpc error: code = Unknown desc = EOF" exec-id=4aa0bb5b6072038d5b8a1f351731c2654a87a14519df97a84f5a997fcd94fd63 name=kata-shim pid=1 source=shim
time="2018-09-27T19:25:35.855999383Z" level=info msg="copy stdout failed" container=66ae83edd972e758611bcd6ca51fe4aa92c4b146de862f2a371ce55860d04466 error="rpc error: code = Unknown desc = EOF" exec-id=66ae83edd972e758611bcd6ca51fe4aa92c4b146de862f2a371ce55860d04466 name=kata-shim pid=1 source=shim
time="2018-09-27T19:25:35.863097475Z" level=info msg="copy stderr failed" container=66ae83edd972e758611bcd6ca51fe4aa92c4b146de862f2a371ce55860d04466 error="rpc error: code = Unknown desc = EOF" exec-id=66ae83edd972e758611bcd6ca51fe4aa92c4b146de862f2a371ce55860d04466 name=kata-shim pid=1 source=shim
time="2018-09-27T19:25:40.509062111Z" level=info msg="copy stderr failed" container=fa138897becc662bf46e9e1ffdf5d3a102ae2bc4e20897c00e78be483fe13200 error="rpc error: code = Unknown desc = EOF" exec-id=fa138897becc662bf46e9e1ffdf5d3a102ae2bc4e20897c00e78be483fe13200 name=kata-shim pid=1 source=shim
time="2018-09-27T19:25:40.50995231Z" level=info msg="copy stdout failed" container=fa138897becc662bf46e9e1ffdf5d3a102ae2bc4e20897c00e78be483fe13200 error="rpc error: code = Unknown desc = EOF" exec-id=fa138897becc662bf46e9e1ffdf5d3a102ae2bc4e20897c00e78be483fe13200 name=kata-shim pid=1 source=shim
time="2018-09-27T19:25:45.527536444Z" level=info msg="copy stderr failed" container=ec6052a1516b0df2250cb6b6d3d18ed750945203d851ed49678e09fc8002b67a error="rpc error: code = Unknown desc = EOF" exec-id=ec6052a1516b0df2250cb6b6d3d18ed750945203d851ed49678e09fc8002b67a name=kata-shim pid=1 source=shim
time="2018-09-27T19:25:45.532542639Z" level=info msg="copy stdout failed" container=ec6052a1516b0df2250cb6b6d3d18ed750945203d851ed49678e09fc8002b67a error="rpc error: code = Unknown desc = EOF" exec-id=ec6052a1516b0df2250cb6b6d3d18ed750945203d851ed49678e09fc8002b67a name=kata-shim pid=1 source=shim
time="2018-09-27T19:25:50.144288819Z" level=info msg="copy stderr failed" container=267c9a894632e65c4d86409a80a53102c9332e42a4248364777b81fc80bfdf21 error="rpc error: code = Unknown desc = EOF" exec-id=267c9a894632e65c4d86409a80a53102c9332e42a4248364777b81fc80bfdf21 name=kata-shim pid=1 source=shim
time="2018-09-27T19:25:50.164557097Z" level=info msg="copy stdout failed" container=267c9a894632e65c4d86409a80a53102c9332e42a4248364777b81fc80bfdf21 error="rpc error: code = Unknown desc = EOF" exec-id=267c9a894632e65c4d86409a80a53102c9332e42a4248364777b81fc80bfdf21 name=kata-shim pid=1 source=shim
time="2018-09-27T19:25:59.434461916Z" level=info msg="copy stdout failed" container=0515a4f16877ff7f2146d40e1abe05a2f1e4b853320dd2fa98cd7a4cf37de2c3 error="rpc error: code = Unknown desc = read /dev/ptmx: input/output error" exec-id=0515a4f16877ff7f2146d40e1abe05a2f1e4b853320dd2fa98cd7a4cf37de2c3 name=kata-shim pid=1 source=shim
time="2018-09-27T19:26:03.84963352Z" level=info msg="copy stdout failed" container=29dcdb990202c8214387b22288df38dffd7384b7fc5258de86a65ee7bd34ed33 error="rpc error: code = Unknown desc = EOF" exec-id=29dcdb990202c8214387b22288df38dffd7384b7fc5258de86a65ee7bd34ed33 name=kata-shim pid=1 source=shim
time="2018-09-27T19:26:03.850112219Z" level=info msg="copy stderr failed" container=29dcdb990202c8214387b22288df38dffd7384b7fc5258de86a65ee7bd34ed33 error="rpc error: code = Unknown desc = EOF" exec-id=29dcdb990202c8214387b22288df38dffd7384b7fc5258de86a65ee7bd34ed33 name=kata-shim pid=1 source=shim
time="2018-09-27T19:27:16.037489188Z" level=info msg="copy stdout failed" container=eb7243603a837426640912f5dd9eb0aada8bd795596b6713a61545d9f3f005b7 error="rpc error: code = Unknown desc = read /dev/ptmx: input/output error" exec-id=eb7243603a837426640912f5dd9eb0aada8bd795596b6713a61545d9f3f005b7 name=kata-shim pid=1 source=shim
time="2018-09-27T19:34:27.73317523Z" level=info msg="copy stdout failed" container=72b021ab0526582beedda2205f74c92fb86d5e39d2c97d098b7835310c99e405 error="rpc error: code = Unknown desc = read /dev/ptmx: input/output error" exec-id=72b021ab0526582beedda2205f74c92fb86d5e39d2c97d098b7835310c99e405 name=kata-shim pid=1 source=shim
time="2018-09-27T19:36:42.434712027Z" level=info msg="copy stdout failed" container=4151a31551f2f5afbafb7338e4e1bc9fa37f8d4d43da83032c49ee42e6b67674 error="rpc error: code = Unknown desc = read /dev/ptmx: input/output error" exec-id=4151a31551f2f5afbafb7338e4e1bc9fa37f8d4d43da83032c49ee42e6b67674 name=kata-shim pid=1 source=shim
time="2018-09-27T19:52:43.193393089Z" level=info msg="copy stdout failed" container=af3933ea5cf55053cc8dfacfa78bb5b91bb06dca34e5f7c087636b8ef3001246 error="rpc error: code = Unknown desc = read /dev/ptmx: input/output error" exec-id=af3933ea5cf55053cc8dfacfa78bb5b91bb06dca34e5f7c087636b8ef3001246 name=kata-shim pid=1 source=shim

Throttler logs

No recent throttler problems found in system journal.

Container manager details

Have docker

Docker

Output of "docker version":

Client:
 Version:           18.06.1-ce
 API version:       1.38
 Go version:        go1.10.3
 Git commit:        e68fc7a
 Built:             Tue Aug 21 17:24:56 2018
 OS/Arch:           linux/amd64
 Experimental:      false

Server:
 Engine:
  Version:          18.06.1-ce
  API version:      1.38 (minimum version 1.12)
  Go version:       go1.10.3
  Git commit:       e68fc7a
  Built:            Tue Aug 21 17:23:21 2018
  OS/Arch:          linux/amd64
  Experimental:     false

Output of "docker info":

Containers: 2
 Running: 0
 Paused: 0
 Stopped: 2
Images: 7
Server Version: 18.06.1-ce
Storage Driver: overlay2
 Backing Filesystem: extfs
 Supports d_type: true
 Native Overlay Diff: false
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
 Volume: local
 Network: bridge host macvlan null overlay
 Log: awslogs fluentd gcplogs gelf journald json-file logentries splunk syslog
Swarm: inactive
Runtimes: kata-runtime runc
Default Runtime: kata-runtime
Init Binary: docker-init
containerd version: 468a545b9edcd5932818eb9de8e72413e616e86e
runc version: 0d99a4f49f60d936fa15e5a0842cfd4d03a24e8f (expected: 69663f0bd4b60df09991c08812a60108003fa340)
init version: fec3683
Security Options:
 apparmor
 seccomp
  Profile: default
Kernel Version: 4.15.0-1023-azure
Operating System: Ubuntu 16.04.5 LTS
OSType: linux
Architecture: x86_64
CPUs: 2
Total Memory: 7.791GiB
Name: gabypnp
ID: 5TIH:5YPR:375O:JVJ3:KXX7:TTJO:TDTA:AEI2:ZHZ5:DBIV:GTLY:Y7C4
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): true
 File Descriptors: 35
 Goroutines: 44
 System Time: 2018-09-27T19:59:11.022336658Z
 EventsListeners: 0
Registry: https://index.docker.io/v1/
Labels:
Experimental: false
Insecure Registries:
 127.0.0.0/8
Live Restore Enabled: false

WARNING: No swap limit support

Output of "systemctl show docker":

Type=notify
Restart=on-failure
NotifyAccess=main
RestartUSec=100ms
TimeoutStartUSec=infinity
TimeoutStopUSec=1min 30s
RuntimeMaxUSec=infinity
WatchdogUSec=0
WatchdogTimestamp=Wed 2018-09-26 16:21:30 UTC
WatchdogTimestampMonotonic=3061530170
FailureAction=none
PermissionsStartOnly=no
RootDirectoryStartOnly=no
RemainAfterExit=no
GuessMainPID=yes
MainPID=1198
ControlPID=0
FileDescriptorStoreMax=0
NFileDescriptorStore=0
StatusErrno=0
Result=success
ExecMainStartTimestamp=Wed 2018-09-26 16:21:14 UTC
ExecMainStartTimestampMonotonic=3045688776
ExecMainExitTimestampMonotonic=0
ExecMainPID=1198
ExecMainCode=0
ExecMainStatus=0
ExecStart={ path=/usr/bin/dockerd ; argv[]=/usr/bin/dockerd -D --add-runtime kata-runtime=/usr/local/bin/kata-runtime --default-runtime=kata-runtime --storage-driver=overlay2 ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }
ExecReload={ path=/bin/kill ; argv[]=/bin/kill -s HUP $MAINPID ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }
Slice=system.slice
ControlGroup=/system.slice/docker.service
MemoryCurrent=2166378496
CPUUsageNSec=1213552716942
TasksCurrent=194
Delegate=yes
CPUAccounting=no
CPUShares=18446744073709551615
StartupCPUShares=18446744073709551615
CPUQuotaPerSecUSec=infinity
BlockIOAccounting=no
BlockIOWeight=18446744073709551615
StartupBlockIOWeight=18446744073709551615
MemoryAccounting=no
MemoryLimit=18446744073709551615
DevicePolicy=auto
TasksAccounting=no
TasksMax=18446744073709551615
UMask=0022
LimitCPU=18446744073709551615
LimitCPUSoft=18446744073709551615
LimitFSIZE=18446744073709551615
LimitFSIZESoft=18446744073709551615
LimitDATA=18446744073709551615
LimitDATASoft=18446744073709551615
LimitSTACK=18446744073709551615
LimitSTACKSoft=8388608
LimitCORE=18446744073709551615
LimitCORESoft=18446744073709551615
LimitRSS=18446744073709551615
LimitRSSSoft=18446744073709551615
LimitNOFILE=1048576
LimitNOFILESoft=1048576
LimitAS=18446744073709551615
LimitASSoft=18446744073709551615
LimitNPROC=18446744073709551615
LimitNPROCSoft=18446744073709551615
LimitMEMLOCK=65536
LimitMEMLOCKSoft=65536
LimitLOCKS=18446744073709551615
LimitLOCKSSoft=18446744073709551615
LimitSIGPENDING=31828
LimitSIGPENDINGSoft=31828
LimitMSGQUEUE=819200
LimitMSGQUEUESoft=819200
LimitNICE=0
LimitNICESoft=0
LimitRTPRIO=0
LimitRTPRIOSoft=0
LimitRTTIME=18446744073709551615
LimitRTTIMESoft=18446744073709551615
OOMScoreAdjust=0
Nice=0
IOScheduling=0
CPUSchedulingPolicy=0
CPUSchedulingPriority=0
TimerSlackNSec=50000
CPUSchedulingResetOnFork=no
NonBlocking=no
StandardInput=null
StandardOutput=journal
StandardError=inherit
TTYReset=no
TTYVHangup=no
TTYVTDisallocate=no
SyslogPriority=30
SyslogLevelPrefix=yes
SyslogLevel=6
SyslogFacility=3
SecureBits=0
CapabilityBoundingSet=18446744073709551615
AmbientCapabilities=0
MountFlags=0
PrivateTmp=no
PrivateNetwork=no
PrivateDevices=no
ProtectHome=no
ProtectSystem=no
SameProcessGroup=no
UtmpMode=init
IgnoreSIGPIPE=yes
NoNewPrivileges=no
SystemCallErrorNumber=0
RuntimeDirectoryMode=0755
KillMode=process
KillSignal=15
SendSIGKILL=yes
SendSIGHUP=no
Id=docker.service
Names=docker.service
Requires=docker.socket system.slice sysinit.target
Wants=network-online.target
WantedBy=multi-user.target
ConsistsOf=docker.socket
Conflicts=shutdown.target
Before=multi-user.target shutdown.target
After=sysinit.target network-online.target system.slice docker.socket firewalld.service basic.target systemd-journald.socket
TriggeredBy=docker.socket
Documentation=https://docs.docker.com
Description=Docker Application Container Engine
LoadState=loaded
ActiveState=active
SubState=running
FragmentPath=/lib/systemd/system/docker.service
DropInPaths=/etc/systemd/system/docker.service.d/kata-containers.conf
UnitFileState=enabled
UnitFilePreset=enabled
StateChangeTimestamp=Wed 2018-09-26 16:21:30 UTC
StateChangeTimestampMonotonic=3061530172
InactiveExitTimestamp=Wed 2018-09-26 16:21:14 UTC
InactiveExitTimestampMonotonic=3045688830
ActiveEnterTimestamp=Wed 2018-09-26 16:21:30 UTC
ActiveEnterTimestampMonotonic=3061530172
ActiveExitTimestamp=Wed 2018-09-26 16:21:13 UTC
ActiveExitTimestampMonotonic=3044547194
InactiveEnterTimestamp=Wed 2018-09-26 16:21:14 UTC
InactiveEnterTimestampMonotonic=3045552660
CanStart=yes
CanStop=yes
CanReload=yes
CanIsolate=no
StopWhenUnneeded=no
RefuseManualStart=no
RefuseManualStop=no
AllowIsolate=no
DefaultDependencies=yes
OnFailureJobMode=replace
IgnoreOnIsolate=no
NeedDaemonReload=no
JobTimeoutUSec=infinity
JobTimeoutAction=none
ConditionResult=yes
AssertResult=yes
ConditionTimestamp=Wed 2018-09-26 16:21:14 UTC
ConditionTimestampMonotonic=3045687959
AssertTimestamp=Wed 2018-09-26 16:21:14 UTC
AssertTimestampMonotonic=3045687959
Transient=no
StartLimitInterval=60000000
StartLimitBurst=3
StartLimitAction=none

Have kubectl

Kubernetes

Output of "kubectl version":

Client Version: version.Info{Major:"1", Minor:"10", GitVersion:"v1.10.5", GitCommit:"32ac1c9073b132b8ba18aa830f46b77dcceb0723", GitTreeState:"clean", BuildDate:"2018-06-21T11:46:00Z", GoVersion:"go1.9.3", Compiler:"gc", Platform:"linux/amd64"}
The connection to the server localhost:8080 was refused - did you specify the right host or port?

Output of "kubectl config view":

apiVersion: v1
clusters: []
contexts: []
current-context: ""
kind: Config
preferences: {}
users: []

Output of "systemctl show kubelet":

Type=simple
Restart=always
NotifyAccess=none
RestartUSec=10s
TimeoutStartUSec=1min 30s
TimeoutStopUSec=1min 30s
RuntimeMaxUSec=infinity
WatchdogUSec=0
WatchdogTimestamp=Thu 2018-09-27 19:59:11 UTC
WatchdogTimestampMonotonic=102522631019
FailureAction=none
PermissionsStartOnly=no
RootDirectoryStartOnly=no
RemainAfterExit=no
GuessMainPID=yes
MainPID=18941
ControlPID=0
FileDescriptorStoreMax=0
NFileDescriptorStore=0
StatusErrno=0
Result=success
ExecMainStartTimestamp=Thu 2018-09-27 19:59:11 UTC
ExecMainStartTimestampMonotonic=102522630987
ExecMainExitTimestampMonotonic=0
ExecMainPID=18941
ExecMainCode=0
ExecMainStatus=0
ExecStart={ path=/usr/bin/kubelet ; argv[]=/usr/bin/kubelet $KUBELET_KUBECONFIG_ARGS $KUBELET_SYSTEM_PODS_ARGS $KUBELET_NETWORK_ARGS $KUBELET_DNS_ARGS $KUBELET_AUTHZ_ARGS $KUBELET_CADVISOR_ARGS $KUBELET_CERTIFICATE_ARGS $KUBELET_EXTRA_ARGS ; ignore_errors=no ; start_time=[Thu 2018-09-27 19:59:11 UTC] ; stop_time=[n/a] ; pid=18941 ; code=(null) ; status=0/0 }
Slice=system.slice
ControlGroup=/system.slice/kubelet.service
MemoryCurrent=0
CPUUsageNSec=0
TasksCurrent=0
Delegate=no
CPUAccounting=no
CPUShares=18446744073709551615
StartupCPUShares=18446744073709551615
CPUQuotaPerSecUSec=infinity
BlockIOAccounting=no
BlockIOWeight=18446744073709551615
StartupBlockIOWeight=18446744073709551615
MemoryAccounting=no
MemoryLimit=18446744073709551615
DevicePolicy=auto
TasksAccounting=no
TasksMax=18446744073709551615
Environment=KUBELET_KUBECONFIG_ARGS=--bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.conf\x20--kubeconfig=/etc/kubernetes/kubelet.conf KUBELET_SYSTEM_PODS_ARGS=--pod-manifest-path=/etc/kubernetes/manifests\x20--allow-privileged=true KUBELET_NETWORK_ARGS=--network-plugin=cni\x20--cni-conf-dir=/etc/cni/net.d\x20--cni-bin-dir=/opt/cni/bin KUBELET_DNS_ARGS=--cluster-dns=10.96.0.10\x20--cluster-domain=cluster.local KUBELET_AUTHZ_ARGS=--authorization-mode=Webhook\x20--client-ca-file=/etc/kubernetes/pki/ca.crt KUBELET_EXTRA_ARGS=--container-runtime=remote\x20--container-runtime-endpoint=unix:///var/run/crio/crio.sock\x20--runtime-request-timeout=30m KUBELET_CADVISOR_ARGS=--cadvisor-port=0 KUBELET_CERTIFICATE_ARGS=--rotate-certificates=true\x20--cert-dir=/var/lib/kubelet/pki
UMask=0022
LimitCPU=18446744073709551615
LimitCPUSoft=18446744073709551615
LimitFSIZE=18446744073709551615
LimitFSIZESoft=18446744073709551615
LimitDATA=18446744073709551615
LimitDATASoft=18446744073709551615
LimitSTACK=18446744073709551615
LimitSTACKSoft=8388608
LimitCORE=18446744073709551615
LimitCORESoft=0
LimitRSS=18446744073709551615
LimitRSSSoft=18446744073709551615
LimitNOFILE=4096
LimitNOFILESoft=1024
LimitAS=18446744073709551615
LimitASSoft=18446744073709551615
LimitNPROC=31828
LimitNPROCSoft=31828
LimitMEMLOCK=65536
LimitMEMLOCKSoft=65536
LimitLOCKS=18446744073709551615
LimitLOCKSSoft=18446744073709551615
LimitSIGPENDING=31828
LimitSIGPENDINGSoft=31828
LimitMSGQUEUE=819200
LimitMSGQUEUESoft=819200
LimitNICE=0
LimitNICESoft=0
LimitRTPRIO=0
LimitRTPRIOSoft=0
LimitRTTIME=18446744073709551615
LimitRTTIMESoft=18446744073709551615
OOMScoreAdjust=0
Nice=0
IOScheduling=0
CPUSchedulingPolicy=0
CPUSchedulingPriority=0
TimerSlackNSec=50000
CPUSchedulingResetOnFork=no
NonBlocking=no
StandardInput=null
StandardOutput=journal
StandardError=inherit
TTYReset=no
TTYVHangup=no
TTYVTDisallocate=no
SyslogPriority=30
SyslogLevelPrefix=yes
SyslogLevel=6
SyslogFacility=3
SecureBits=0
CapabilityBoundingSet=18446744073709551615
AmbientCapabilities=0
MountFlags=0
PrivateTmp=no
PrivateNetwork=no
PrivateDevices=no
ProtectHome=no
ProtectSystem=no
SameProcessGroup=no
UtmpMode=init
IgnoreSIGPIPE=yes
NoNewPrivileges=no
SystemCallErrorNumber=0
RuntimeDirectoryMode=0755
KillMode=control-group
KillSignal=15
SendSIGKILL=yes
SendSIGHUP=no
Id=kubelet.service
Names=kubelet.service
Requires=system.slice sysinit.target
WantedBy=multi-user.target
Conflicts=shutdown.target
Before=multi-user.target shutdown.target
After=sysinit.target systemd-journald.socket basic.target system.slice
Documentation=http://kubernetes.io/docs/
Description=kubelet: The Kubernetes Node Agent
LoadState=loaded
ActiveState=active
SubState=running
FragmentPath=/lib/systemd/system/kubelet.service
DropInPaths=/etc/systemd/system/kubelet.service.d/10-kubeadm.conf
UnitFileState=enabled
UnitFilePreset=enabled
StateChangeTimestamp=Thu 2018-09-27 19:59:11 UTC
StateChangeTimestampMonotonic=102522631020
InactiveExitTimestamp=Thu 2018-09-27 19:59:11 UTC
InactiveExitTimestampMonotonic=102522631020
ActiveEnterTimestamp=Thu 2018-09-27 19:59:11 UTC
ActiveEnterTimestampMonotonic=102522631020
ActiveExitTimestamp=Thu 2018-09-27 19:59:01 UTC
ActiveExitTimestampMonotonic=102512608110
InactiveEnterTimestamp=Thu 2018-09-27 19:59:11 UTC
InactiveEnterTimestampMonotonic=102522630052
CanStart=yes
CanStop=yes
CanReload=no
CanIsolate=no
StopWhenUnneeded=no
RefuseManualStart=no
RefuseManualStop=no
AllowIsolate=no
DefaultDependencies=yes
OnFailureJobMode=replace
IgnoreOnIsolate=no
NeedDaemonReload=no
JobTimeoutUSec=infinity
JobTimeoutAction=none
ConditionResult=yes
AssertResult=yes
ConditionTimestamp=Thu 2018-09-27 19:59:11 UTC
ConditionTimestampMonotonic=102522630131
AssertTimestamp=Thu 2018-09-27 19:59:11 UTC
AssertTimestampMonotonic=102522630131
Transient=no
StartLimitInterval=0
StartLimitBurst=5
StartLimitAction=none

Have crio Output of "crio --version":

crio version 1.10.7-dev
commit: "6273bea4c9ed788aeb3d051ebf2d030060c05b6c"

Output of "systemctl show crio":

Type=simple
Restart=on-failure
NotifyAccess=none
RestartUSec=5s
TimeoutStartUSec=1min 30s
TimeoutStopUSec=1min 30s
RuntimeMaxUSec=infinity
WatchdogUSec=0
WatchdogTimestampMonotonic=0
FailureAction=none
PermissionsStartOnly=no
RootDirectoryStartOnly=no
RemainAfterExit=no
GuessMainPID=yes
MainPID=0
ControlPID=0
FileDescriptorStoreMax=0
NFileDescriptorStore=0
StatusErrno=0
Result=success
ExecMainStartTimestampMonotonic=0
ExecMainExitTimestampMonotonic=0
ExecMainPID=0
ExecMainCode=0
ExecMainStatus=0
ExecStart={ path=/usr/local/bin/crio ; argv[]=/usr/local/bin/crio --log-level debug ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }
Slice=system.slice
MemoryCurrent=18446744073709551615
CPUUsageNSec=18446744073709551615
TasksCurrent=18446744073709551615
Delegate=no
CPUAccounting=no
CPUShares=18446744073709551615
StartupCPUShares=18446744073709551615
CPUQuotaPerSecUSec=infinity
BlockIOAccounting=no
BlockIOWeight=18446744073709551615
StartupBlockIOWeight=18446744073709551615
MemoryAccounting=no
MemoryLimit=18446744073709551615
DevicePolicy=auto
TasksAccounting=no
TasksMax=18446744073709551615
UMask=0022
LimitCPU=18446744073709551615
LimitCPUSoft=18446744073709551615
LimitFSIZE=18446744073709551615
LimitFSIZESoft=18446744073709551615
LimitDATA=18446744073709551615
LimitDATASoft=18446744073709551615
LimitSTACK=18446744073709551615
LimitSTACKSoft=8388608
LimitCORE=18446744073709551615
LimitCORESoft=0
LimitRSS=18446744073709551615
LimitRSSSoft=18446744073709551615
LimitNOFILE=4096
LimitNOFILESoft=1024
LimitAS=18446744073709551615
LimitASSoft=18446744073709551615
LimitNPROC=31828
LimitNPROCSoft=31828
LimitMEMLOCK=65536
LimitMEMLOCKSoft=65536
LimitLOCKS=18446744073709551615
LimitLOCKSSoft=18446744073709551615
LimitSIGPENDING=31828
LimitSIGPENDINGSoft=31828
LimitMSGQUEUE=819200
LimitMSGQUEUESoft=819200
LimitNICE=0
LimitNICESoft=0
LimitRTPRIO=0
LimitRTPRIOSoft=0
LimitRTTIME=18446744073709551615
LimitRTTIMESoft=18446744073709551615
OOMScoreAdjust=0
Nice=0
IOScheduling=0
CPUSchedulingPolicy=0
CPUSchedulingPriority=0
TimerSlackNSec=50000
CPUSchedulingResetOnFork=no
NonBlocking=no
StandardInput=null
StandardOutput=journal
StandardError=inherit
TTYReset=no
TTYVHangup=no
TTYVTDisallocate=no
SyslogPriority=30
SyslogLevelPrefix=yes
SyslogLevel=6
SyslogFacility=3
SecureBits=0
CapabilityBoundingSet=18446744073709551615
AmbientCapabilities=0
MountFlags=0
PrivateTmp=no
PrivateNetwork=no
PrivateDevices=no
ProtectHome=no
ProtectSystem=no
SameProcessGroup=no
UtmpMode=init
IgnoreSIGPIPE=yes
NoNewPrivileges=no
SystemCallErrorNumber=0
RuntimeDirectoryMode=0755
KillMode=control-group
KillSignal=15
SendSIGKILL=yes
SendSIGHUP=no
Id=crio.service
Names=crio.service
Requires=system.slice sysinit.target
Conflicts=shutdown.target
Before=shutdown.target
After=sysinit.target systemd-journald.socket basic.target system.slice
Documentation=https://github.com/kubernetes-incubator/cri-o
Description=CRI-O daemon
LoadState=loaded
ActiveState=inactive
SubState=dead
FragmentPath=/etc/systemd/system/crio.service
UnitFileState=disabled
UnitFilePreset=enabled
StateChangeTimestampMonotonic=0
InactiveExitTimestampMonotonic=0
ActiveEnterTimestampMonotonic=0
ActiveExitTimestampMonotonic=0
InactiveEnterTimestampMonotonic=0
CanStart=yes
CanStop=yes
CanReload=no
CanIsolate=no
StopWhenUnneeded=no
RefuseManualStart=no
RefuseManualStop=no
AllowIsolate=no
DefaultDependencies=yes
OnFailureJobMode=replace
IgnoreOnIsolate=no
NeedDaemonReload=no
JobTimeoutUSec=infinity
JobTimeoutAction=none
ConditionResult=no
AssertResult=no
ConditionTimestampMonotonic=0
AssertTimestampMonotonic=0
Transient=no
StartLimitInterval=10000000
StartLimitBurst=5
StartLimitAction=none

Packages

Have dpkg Output of "dpkg -l|egrep "(cc-oci-runtimecc-runtimerunv|kata-proxy|kata-runtime|kata-shim|kata-ksm-throttler|kata-containers-image|linux-container|qemu-)"":

ii  kata-containers-image               1.3.0~rc1-34                               amd64        Kata containers image
ii  qemu-lite                           2.11.0+git.f886228056-50                   amd64        linux kernel optimised for container-like workloads.

Have rpm Output of "rpm -qa|egrep "(cc-oci-runtimecc-runtimerunv|kata-proxy|kata-runtime|kata-shim|kata-ksm-throttler|kata-containers-image|linux-container|qemu-)"":

caoruidong commented 6 years ago

Is it because https://github.com/kata-containers/runtime/pull/778?

devimc commented 6 years ago

@caoruidong nop, this issue is because of lack of mnt namespace support

caoruidong commented 6 years ago

@devimc Oh, I see. Thanks