search

kata-containers / runtime

Kata Containers version 1.x runtime (for version 2.x see https://github.com/kata-containers/kata-containers).
https://katacontainers.io/
Apache License 2.0
2.1k stars 375 forks source link

Failed to luanch android image based on kata #1552

Closed zhiminghufighting closed 3 years ago

zhiminghufighting commented 5 years ago

Description of problem

  1. Enable host OS binder driver and ashmem then launch android image by runc successfully;

  2. Rebuild kata guest kernel and add two android depending on modelers: binder driver and ashmem and then failed to launch the same android image by kata;

Expected result

launch the same android image by kata successfully;

Actual result

Meta details

Running kata-collect-data.sh version 1.6.0-rc1 (commit 9f8d4e1) at 2019-04-17.17:43:19.365712194+0800.

Runtime is /usr/bin/kata-runtime.

kata-env

Output of "/usr/bin/kata-runtime kata-env":

[Meta]
  Version = "1.0.20"

[Runtime]
  Debug = true
  Trace = false
  DisableGuestSeccomp = true
  DisableNewNetNs = false
  Path = "/usr/bin/kata-runtime"
  [Runtime.Version]
    Semver = "1.6.0-rc1"
    Commit = "9f8d4e1"
    OCI = "1.0.1-dev"
  [Runtime.Config]
    Path = "/usr/share/defaults/kata-containers/configuration.toml"

[Hypervisor]
  MachineType = "pc"
  Version = "QEMU emulator version 2.11.0\nCopyright (c) 2003-2017 Fabrice Bellard and the QEMU Project developers"
  Path = "/usr/bin/qemu-lite-system-x86_64"
  BlockDeviceDriver = "virtio-scsi"
  EntropySource = "/dev/urandom"
  Msize9p = 8192
  MemorySlots = 10
  Debug = false
  UseVSock = false

[Image]
  Path = "/usr/share/kata-containers/kata-containers-image_clearlinux_1.6.0-rc1_agent_a2037c08531.img"

[Kernel]
  Path = "/usr/share/kata-containers/kata-vmlinuz-4.14-andriod.container"
  Parameters = "init=/usr/lib/systemd/systemd systemd.unit=kata-containers.target systemd.mask=systemd-networkd.service systemd.mask=systemd-networkd.socket"

[Initrd]
  Path = ""

[Proxy] 
  Type = "kataProxy"
  Version = "kata-proxy version 1.6.0-rc1-2085829"
  Path = "/usr/libexec/kata-containers/kata-proxy"
  Debug = true

[Shim] 
  Type = "kataShim"
  Version = "kata-shim version 1.6.0-rc1-0f41347"
  Path = "/usr/libexec/kata-containers/kata-shim"
  Debug = true

[Agent] 
  Type = "kata"

[Host] 
  Kernel = "3.10.0-862.el7.x86_64"
  Architecture = "amd64"
  VMContainerCapable = true
  SupportVSocks = false
  [Host.Distro]
    Name = "CentOS Linux"
    Version = "7"
  [Host.CPU]
    Vendor = "GenuineIntel"
    Model = "Intel(R) Xeon(R) CPU E5-2680 v4 @ 2.40GHz"

[Netmon]
  Version = "kata-netmon version 1.6.0-rc1"
  Path = "/usr/libexec/kata-containers/kata-netmon"
  Debug = false
  Enable = false

Runtime config files

Runtime default config files

/etc/kata-containers/configuration.toml
/usr/share/defaults/kata-containers/configuration.toml

Runtime config file contents

Config file /etc/kata-containers/configuration.toml not found Output of "cat "/usr/share/defaults/kata-containers/configuration.toml"":

# Copyright (c) 2017-2019 Intel Corporation
# 
# SPDX-License-Identifier: Apache-2.0
# 

# XXX: WARNING: this file is auto-generated.
# XXX: 
# XXX: Source file: "cli/config/configuration-qemu.toml.in"
# XXX: Project:
# XXX:   Name: Kata Containers
# XXX:   Type: kata

[hypervisor.qemu]
path = "/usr/bin/qemu-lite-system-x86_64"
#kernel = "/usr/share/kata-containers/vmlinuz.container"
kernel = "/usr/share/kata-containers/vmlinuz-andriod.container"
image = "/usr/share/kata-containers/kata-containers.img"
machine_type = "pc"

# Optional space-separated list of options to pass to the guest kernel.
# For example, use `kernel_params = "vsyscall=emulate"` if you are having
# trouble running pre-2.15 glibc.
# 
# WARNING: - any parameter specified here will take priority over the default
# parameter value of the same name used to start the virtual machine.
# Do not set values here unless you understand the impact of doing so as you
# may stop the virtual machine from booting.
# To see the list of default parameters, enable hypervisor debug, create a
# container and look for 'default-kernel-parameters' log entries.
kernel_params = ""

# Path to the firmware.
# If you want that qemu uses the default firmware leave this option empty
firmware = ""

# Machine accelerators
# comma-separated list of machine accelerators to pass to the hypervisor.
# For example, `machine_accelerators = "nosmm,nosmbus,nosata,nopit,static-prt,nofw"`
machine_accelerators=""

# Default number of vCPUs per SB/VM:
# unspecified or 0                --> will be set to 1
# < 0                             --> will be set to the actual number of physical cores
# > 0 <= number of physical cores --> will be set to the specified number
# > number of physical cores      --> will be set to the actual number of physical cores
default_vcpus = 1

# Default maximum number of vCPUs per SB/VM:
# unspecified or == 0             --> will be set to the actual number of physical cores or to the maximum number
#                                     of vCPUs supported by KVM if that number is exceeded
# > 0 <= number of physical cores --> will be set to the specified number
# > number of physical cores      --> will be set to the actual number of physical cores or to the maximum number
#                                     of vCPUs supported by KVM if that number is exceeded
# WARNING: Depending of the architecture, the maximum number of vCPUs supported by KVM is used when
# the actual number of physical cores is greater than it.
# WARNING: Be aware that this value impacts the virtual machine's memory footprint and CPU
# the hotplug functionality. For example, `default_maxvcpus = 240` specifies that until 240 vCPUs
# can be added to a SB/VM, but the memory footprint will be big. Another example, with
# `default_maxvcpus = 8` the memory footprint will be small, but 8 will be the maximum number of
# vCPUs supported by the SB/VM. In general, we recommend that you do not edit this variable,
# unless you know what are you doing.
default_maxvcpus = 0

# Bridges can be used to hot plug devices.
# Limitations:
# * Currently only pci bridges are supported
# * Until 30 devices per bridge can be hot plugged.
# * Until 5 PCI bridges can be cold plugged per VM.
#   This limitation could be a bug in qemu or in the kernel
# Default number of bridges per SB/VM:
# unspecified or 0   --> will be set to 1
# > 1 <= 5           --> will be set to the specified number
# > 5                --> will be set to 5
default_bridges = 1

# Default memory size in MiB for SB/VM.
# If unspecified then it will be set 2048 MiB.
default_memory = 2048
# 
# Default memory slots per SB/VM.
# If unspecified then it will be set 10.
# This is will determine the times that memory will be hotadded to sandbox/VM.
#memory_slots = 10

# The size in MiB will be plused to max memory of hypervisor.
# It is the memory address space for the NVDIMM devie.
# If set block storage driver (block_device_driver) to "nvdimm",
# should set memory_offset to the size of block device.
# Default 0
#memory_offset = 0

# Disable block device from being used for a container's rootfs.
# In case of a storage driver like devicemapper where a container's 
# root file system is backed by a block device, the block device is passed
# directly to the hypervisor for performance reasons. 
# This flag prevents the block device from being passed to the hypervisor, 
# 9pfs is used instead to pass the rootfs.
disable_block_device_use = false

# Block storage driver to be used for the hypervisor in case the container
# rootfs is backed by a block device. This is virtio-scsi, virtio-blk
# or nvdimm.
block_device_driver = "virtio-scsi"

# Specifies cache-related options will be set to block devices or not.
# Default false
#block_device_cache_set = true

# Specifies cache-related options for block devices.
# Denotes whether use of O_DIRECT (bypass the host page cache) is enabled.
# Default false
#block_device_cache_direct = true

# Specifies cache-related options for block devices.
# Denotes whether flush requests for the device are ignored.
# Default false
#block_device_cache_noflush = true

# Enable iothreads (data-plane) to be used. This causes IO to be
# handled in a separate IO thread. This is currently only implemented
# for SCSI.
# 
enable_iothreads = false

# Enable pre allocation of VM RAM, default false
# Enabling this will result in lower container density
# as all of the memory will be allocated and locked
# This is useful when you want to reserve all the memory
# upfront or in the cases where you want memory latencies
# to be very predictable
# Default false
#enable_mem_prealloc = true

# Enable huge pages for VM RAM, default false
# Enabling this will result in the VM memory
# being allocated using huge pages.
# This is useful when you want to use vhost-user network
# stacks within the container. This will automatically 
# result in memory pre allocation
#enable_hugepages = true

# Enable swap of vm memory. Default false.
# The behaviour is undefined if mem_prealloc is also set to true
#enable_swap = true

# This option changes the default hypervisor and kernel parameters
# to enable debug output where available. This extra output is added
# to the proxy logs, but only when proxy debug is also enabled.
#  
# Default false
#enable_debug = true

# Disable the customizations done in the runtime when it detects
# that it is running on top a VMM. This will result in the runtime
# behaving as it would when running on bare metal.
#  
#disable_nesting_checks = true

# This is the msize used for 9p shares. It is the number of bytes 
# used for 9p packet payload.
#msize_9p = 8192

# If true and vsocks are supported, use vsocks to communicate directly
# with the agent and no proxy is started, otherwise use unix
# sockets and start a proxy to communicate with the agent.
# Default false
#use_vsock = true

# VFIO devices are hotplugged on a bridge by default. 
# Enable hotplugging on root bus. This may be required for devices with
# a large PCI bar, as this is a current limitation with hotplugging on 
# a bridge. This value is valid for "pc" machine type.
# Default false
#hotplug_vfio_on_root_bus = true

# If host doesn't support vhost_net, set to true. Thus we won't create vhost fds for nics.
# Default false
#disable_vhost_net = true
# 
# Default entropy source.
# The path to a host source of entropy (including a real hardware RNG)
# /dev/urandom and /dev/random are two main options.
# Be aware that /dev/random is a blocking source of entropy.  If the host
# runs out of entropy, the VMs boot time will increase leading to get startup
# timeouts.
# The source of entropy /dev/urandom is non-blocking and provides a
# generally acceptable source of entropy. It should work well for pretty much
# all practical purposes.
#entropy_source= "/dev/urandom"

# Path to OCI hook binaries in the *guest rootfs*.
# This does not affect host-side hooks which must instead be added to
# the OCI spec passed to the runtime.
# 
# You can create a rootfs with hooks by customizing the osbuilder scripts:
# https://github.com/kata-containers/osbuilder
# 
# Hooks must be stored in a subdirectory of guest_hook_path according to their
# hook type, i.e. "guest_hook_path/{prestart,postart,poststop}".
# The agent will scan these directories for executable files and add them, in
# lexicographical order, to the lifecycle of the guest container.
# Hooks are executed in the runtime namespace of the guest. See the official documentation:
# https://github.com/opencontainers/runtime-spec/blob/v1.0.1/config.md#posix-platform-hooks
# Warnings will be logged if any error is encountered will scanning for hooks,
# but it will not abort container execution.
#guest_hook_path = "/usr/share/oci/hooks"

[factory]
# VM templating support. Once enabled, new VMs are created from template
# using vm cloning. They will share the same initial kernel, initramfs and
# agent memory by mapping it readonly. It helps speeding up new container
# creation and saves a lot of memory if there are many kata containers running
# on the same host.
# 
# When disabled, new VMs are created from scratch.
# 
# Note: Requires "initrd=" to be set ("image=" is not supported).
# 
# Default false
#enable_template = true

[proxy.kata]
path = "/usr/libexec/kata-containers/kata-proxy"

# If enabled, proxy messages will be sent to the system log
# (default: disabled)
enable_debug = true

[shim.kata]
path = "/usr/libexec/kata-containers/kata-shim"

# If enabled, shim messages will be sent to the system log
# (default: disabled)
enable_debug = true

# If enabled, the shim will create opentracing.io traces and spans.
# (See https://www.jaegertracing.io/docs/getting-started).
# 
# Note: By default, the shim runs in a separate network namespace. Therefore,
# to allow it to send trace details to the Jaeger agent running on the host,
# it is necessary to set 'disable_new_netns=true' so that it runs in the host
# network namespace.
# 
# (default: disabled)
#enable_tracing = true

[agent.kata]
# There is no field for this section. The goal is only to be able to
# specify which type of agent the user wants to use.

[netmon]
# If enabled, the network monitoring process gets started when the
# sandbox is created. This allows for the detection of some additional
# network being added to the existing network namespace, after the
# sandbox has been created.
# (default: disabled)
#enable_netmon = true

# Specify the path to the netmon binary.
path = "/usr/libexec/kata-containers/kata-netmon"

# If enabled, netmon messages will be sent to the system log
# (default: disabled)
#enable_debug = true

[runtime]
# If enabled, the runtime will log additional debug messages to the
# system log
# (default: disabled)
enable_debug = true
# 
# Internetworking model
# Determines how the VM should be connected to the
# the container network interface
# Options:
# 
#   - bridged
#     Uses a linux bridge to interconnect the container interface to
#     the VM. Works for most cases except macvlan and ipvlan.
# 
#   - macvtap
#     Used when the Container network interface can be bridged using
#     macvtap.
# 
#   - none
#     Used when customize network. Only creates a tap device. No veth pair.
# 
#   - tcfilter
#     Uses tc filter rules to redirect traffic from the network interface
#     provided by plugin to a tap interface connected to the VM.
# 
internetworking_model="macvtap"

# disable guest seccomp
# Determines whether container seccomp profiles are passed to the virtual
# machine and applied by the kata agent. If set to true, seccomp is not applied
# within the guest
# (default: true)
disable_guest_seccomp=true

# If enabled, the runtime will create opentracing.io traces and spans.
# (See https://www.jaegertracing.io/docs/getting-started).
# (default: disabled)
#enable_tracing = true

# If enabled, the runtime will not create a network namespace for shim and hypervisor processes.
# This option may have some potential impacts to your host. It should only be used when you know what you're doing.
# `disable_new_netns` conflicts with `enable_netmon`
# `disable_new_netns` conflicts with `internetworking_model=bridged` and `internetworking_model=macvtap`. It works only
# with `internetworking_model=none`. The tap device will be in the host network namespace and can connect to a bridge
# (like OVS) directly.
# If you are using docker, `disable_new_netns` only works with `docker run --net=none`
# (default: false)
#disable_new_netns = true

KSM throttler

version

Output of "/usr/libexec/kata-ksm-throttler/kata-ksm-throttler --version":

kata-ksm-throttler version 1.6.0-rc1-ce5b765

Output of "/usr/lib/systemd/system/kata-ksm-throttler.service --version":

/usr/bin/kata-collect-data.sh: line 168: /usr/lib/systemd/system/kata-ksm-throttler.service: Permission denied

systemd service

Image details

--- 
osbuilder:
  url: "https://github.com/kata-containers/osbuilder"
  version: "unknown"
rootfs-creation-time: "2019-03-04T21:52:40.031090715+0000Z"
description: "osbuilder rootfs"
file-format-version: "0.0.2"
architecture: "x86_64"
base-distro:
  name: "Clear"
  version: "28100"
  packages:
    default:
      - "iptables-bin"
      - "libudev0-shim"
      - "systemd"
    extra:

agent: 
  url: "https://github.com/kata-containers/agent"
  name: "kata-agent"
  version: "1.6.0-rc1-a2037c08531e4982003d8fc9b4837d27c35c0966"
  agent-is-init-daemon: "no"

Initrd details

No initrd

320284d29c85c2 name=kata-runtime pid=272038 source=virtcontainers subsystem=kata_agent time="2019-04-15T16:43:56.959834186+08:00" level=warning msg="load sandbox devices failed" arch=amd64 command=create container=3122d618a896589702f918616847f014ae48030eba0e7f2df2320284d29c8 5c2 error="open /run/vc/sbs/3122d618a896589702f918616847f014ae48030eba0e7f2df2320284d29c85c2/devices.json: no such file or directory" name=kata-runtime pid=272038 sandbox=3122d618a89658970 2f918616847f014ae48030eba0e7f2df2320284d29c85c2 sandboxid=3122d618a896589702f918616847f014ae48030eba0e7f2df2320284d29c85c2 source=virtcontainers subsystem=sandbox time="2019-04-16T17:07:44.438555073+08:00" level=debug msg="Could not retrieve anything from storage" arch=amd64 command=create container=6d4b102a48e19d90eeb1ecc753c2ed246ca8d3f7f4d0840ea8 88207b1addb6b7 name=kata-runtime pid=82304 source=virtcontainers subsystem=kata_agent time="2019-04-16T17:07:44.438599578+08:00" level=warning msg="load sandbox devices failed" arch=amd64 command=create container=6d4b102a48e19d90eeb1ecc753c2ed246ca8d3f7f4d0840ea888207b1addb 6b7 error="open /run/vc/sbs/6d4b102a48e19d90eeb1ecc753c2ed246ca8d3f7f4d0840ea888207b1addb6b7/devices.json: no such file or directory" name=kata-runtime pid=82304 sandbox=6d4b102a48e19d90ee b1ecc753c2ed246ca8d3f7f4d0840ea888207b1addb6b7 sandboxid=6d4b102a48e19d90eeb1ecc753c2ed246ca8d3f7f4d0840ea888207b1addb6b7 source=virtcontainers subsystem=sandbox time="2019-04-16T17:07:45.91855327+08:00" level=info msg="{\"error\": {\"class\": \"GenericError\", \"desc\": \"Could not open '/dev/fd0': No such device or address\"}}" arch=amd64 command =create container=6d4b102a48e19d90eeb1ecc753c2ed246ca8d3f7f4d0840ea888207b1addb6b7 name=kata-runtime pid=82304 source=virtcontainers subsystem=qmp time="2019-04-16T17:07:46.00200023+08:00" level=error msg="QMP command failed: Could not open '/dev/fd0': No such device or address" arch=amd64 command=create container=6d4b102a48e19d90eeb 1ecc753c2ed246ca8d3f7f4d0840ea888207b1addb6b7 name=kata-runtime pid=82304 source=runtime time="2019-04-16T17:07:58.932782354+08:00" level=debug msg="Could not retrieve anything from storage" arch=amd64 command=create container=2b20edf0dc79b902b096386202f8b37caf27f1badfa8085d33 84dd251ad75d28 name=kata-runtime pid=82403 source=virtcontainers subsystem=kata_agent time="2019-04-16T17:07:58.932821958+08:00" level=warning msg="load sandbox devices failed" arch=amd64 command=create container=2b20edf0dc79b902b096386202f8b37caf27f1badfa8085d3384dd251ad75 d28 error="open /run/vc/sbs/2b20edf0dc79b902b096386202f8b37caf27f1badfa8085d3384dd251ad75d28/devices.json: no such file or directory" name=kata-runtime pid=82403 sandbox=2b20edf0dc79b902b0 96386202f8b37caf27f1badfa8085d3384dd251ad75d28 sandboxid=2b20edf0dc79b902b096386202f8b37caf27f1badfa8085d3384dd251ad75d28 source=virtcontainers subsystem=sandbox time="2019-04-16T17:08:00.261576928+08:00" level=info msg="{\"error\": {\"class\": \"GenericError\", \"desc\": \"Could not open '/dev/fd0': No such device or address\"}}" arch=amd64 comman d=create container=2b20edf0dc79b902b096386202f8b37caf27f1badfa8085d3384dd251ad75d28 name=kata-runtime pid=82403 source=virtcontainers subsystem=qmp time="2019-04-16T17:08:00.28784754+08:00" level=error msg="QMP command failed: Could not open '/dev/fd0': No such device or address" arch=amd64 command=create container=2b20edf0dc79b902b09 6386202f8b37caf27f1badfa8085d3384dd251ad75d28 name=kata-runtime pid=82403 source=runtime time="2019-04-16T17:17:32.642085739+08:00" level=debug msg="Could not retrieve anything from storage" arch=amd64 command=create container=37cb8ad988f540887bd9aa44d178c207f4d93f5dd44220c09f 0a4356465a9e0d name=kata-runtime pid=83072 source=virtcontainers subsystem=kata_agent time="2019-04-16T17:17:32.642133575+08:00" level=warning msg="load sandbox devices failed" arch=amd64 command=create container=37cb8ad988f540887bd9aa44d178c207f4d93f5dd44220c09f0a4356465a9 e0d error="open /run/vc/sbs/37cb8ad988f540887bd9aa44d178c207f4d93f5dd44220c09f0a4356465a9e0d/devices.json: no such file or directory" name=kata-runtime pid=83072 sandbox=37cb8ad988f540887b d9aa44d178c207f4d93f5dd44220c09f0a4356465a9e0d sandboxid=37cb8ad988f540887bd9aa44d178c207f4d93f5dd44220c09f0a4356465a9e0d source=virtcontainers subsystem=sandbox time="2019-04-16T17:17:34.013555599+08:00" level=info msg="{\"error\": {\"class\": \"GenericError\", \"desc\": \"Could not open '/dev/fd0': No such device or address\"}}" arch=amd64 comman d=create container=37cb8ad988f540887bd9aa44d178c207f4d93f5dd44220c09f0a4356465a9e0d name=kata-runtime pid=83072 source=virtcontainers subsystem=qmp time="2019-04-16T17:17:34.040044399+08:00" level=error msg="QMP command failed: Could not open '/dev/fd0': No such device or address" arch=amd64 command=create container=37cb8ad988f540887b d9aa44d178c207f4d93f5dd44220c09f0a4356465a9e0d name=kata-runtime pid=83072 source=runtime time="2019-04-16T17:18:20.403285719+08:00" level=debug msg="Could not retrieve anything from storage" arch=amd64 command=create container=c727a814256d307b45a49fe7cb4822bffe418037577b968d2f 37bef37fb21267 name=kata-runtime pid=83190 source=virtcontainers subsystem=kata_agent time="2019-04-16T17:18:20.403328737+08:00" level=warning msg="load sandbox devices failed" arch=amd64 command=create container=c727a814256d307b45a49fe7cb4822bffe418037577b968d2f37bef37fb21 267 error="open /run/vc/sbs/c727a814256d307b45a49fe7cb4822bffe418037577b968d2f37bef37fb21267/devices.json: no such file or directory" name=kata-runtime pid=83190 sandbox=c727a814256d307b45 a49fe7cb4822bffe418037577b968d2f37bef37fb21267 sandboxid=c727a814256d307b45a49fe7cb4822bffe418037577b968d2f37bef37fb21267 source=virtcontainers subsystem=sandbox time="2019-04-16T17:18:21.705482712+08:00" level=info msg="{\"error\": {\"class\": \"GenericError\", \"desc\": \"Could not open '/dev/fd0': No such device or address\"}}" arch=amd64 comman d=create container=c727a814256d307b45a49fe7cb4822bffe418037577b968d2f37bef37fb21267 name=kata-runtime pid=83190 source=virtcontainers subsystem=qmp time="2019-04-16T17:18:21.760923996+08:00" level=error msg="QMP command failed: Could not open '/dev/fd0': No such device or address" arch=amd64 command=create container=c727a814256d307b45 a49fe7cb4822bffe418037577b968d2f37bef37fb21267 name=kata-runtime pid=83190 source=runtime time="2019-04-16T17:23:14.254669589+08:00" level=debug msg="Could not retrieve anything from storage" arch=amd64 command=create container=d4c091664726ad991659f8e28e8e238b931e9f95a303cb6c39 01eac4e389fcd6 name=kata-runtime pid=83809 source=virtcontainers subsystem=kata_agent time="2019-04-16T17:23:14.254709281+08:00" level=warning msg="load sandbox devices failed" arch=amd64 command=create container=d4c091664726ad991659f8e28e8e238b931e9f95a303cb6c3901eac4e389f cd6 error="open /run/vc/sbs/d4c091664726ad991659f8e28e8e238b931e9f95a303cb6c3901eac4e389fcd6/devices.json: no such file or directory" name=kata-runtime pid=83809 sandbox=d4c091664726ad9916 59f8e28e8e238b931e9f95a303cb6c3901eac4e389fcd6 sandboxid=d4c091664726ad991659f8e28e8e238b931e9f95a303cb6c3901eac4e389fcd6 source=virtcontainers subsystem=sandbox time="2019-04-16T17:23:15.621579547+08:00" level=info msg="{\"error\": {\"class\": \"GenericError\", \"desc\": \"Could not open '/dev/fd0': No such device or address\"}}" arch=amd64 comman d=create container=d4c091664726ad991659f8e28e8e238b931e9f95a303cb6c3901eac4e389fcd6 name=kata-runtime pid=83809 source=virtcontainers subsystem=qmp time="2019-04-16T17:23:15.647877804+08:00" level=error msg="QMP command failed: Could not open '/dev/fd0': No such device or address" arch=amd64 command=create container=d4c091664726ad9916 59f8e28e8e238b931e9f95a303cb6c3901eac4e389fcd6 name=kata-runtime pid=83809 source=runtime time="2019-04-17T15:52:11.762224737+08:00" level=debug msg="Could not retrieve anything from storage" arch=amd64 command=create container=d2e84d608f77a85c98f1d37115254a6e0b769b020ee111455a 1412d92b45904b name=kata-runtime pid=166837 source=virtcontainers subsystem=kata_agent time="2019-04-17T15:52:11.762283796+08:00" level=warning msg="load sandbox devices failed" arch=amd64 command=create container=d2e84d608f77a85c98f1d37115254a6e0b769b020ee111455a1412d92b459 04b error="open /run/vc/sbs/d2e84d608f77a85c98f1d37115254a6e0b769b020ee111455a1412d92b45904b/devices.json: no such file or directory" name=kata-runtime pid=166837 sandbox=d2e84d608f77a85c9 8f1d37115254a6e0b769b020ee111455a1412d92b45904b sandboxid=d2e84d608f77a85c98f1d37115254a6e0b769b020ee111455a1412d92b45904b source=virtcontainers subsystem=sandbox time="2019-04-17T15:52:13.130480717+08:00" level=info msg="{\"error\": {\"class\": \"GenericError\", \"desc\": \"Could not open '/dev/fd0': No such device or address\"}}" arch=amd64 comman d=create container=d2e84d608f77a85c98f1d37115254a6e0b769b020ee111455a1412d92b45904b name=kata-runtime pid=166837 source=virtcontainers subsystem=qmp time="2019-04-17T15:52:13.15586313+08:00" level=error msg="QMP command failed: Could not open '/dev/fd0': No such device or address" arch=amd64 command=create container=d2e84d608f77a85c98f 1d37115254a6e0b769b020ee111455a1412d92b45904b name=kata-runtime pid=166837 source=runtime time="2019-04-17T16:34:43.829706497+08:00" level=debug msg="Could not retrieve anything from storage" arch=amd64 command=create container=a98c2d8a7bb80dca033eca387ba837754f55316876411dae7e 4c0fad1600021b name=kata-runtime pid=169567 source=virtcontainers subsystem=kata_agent time="2019-04-17T16:34:43.829758305+08:00" level=warning msg="load sandbox devices failed" arch=amd64 command=create container=a98c2d8a7bb80dca033eca387ba837754f55316876411dae7e4c0fad16000 21b error="open /run/vc/sbs/a98c2d8a7bb80dca033eca387ba837754f55316876411dae7e4c0fad1600021b/devices.json: no such file or directory" name=kata-runtime pid=169567 sandbox=a98c2d8a7bb80dca0 33eca387ba837754f55316876411dae7e4c0fad1600021b sandboxid=a98c2d8a7bb80dca033eca387ba837754f55316876411dae7e4c0fad1600021b source=virtcontainers subsystem=sandbox time="2019-04-17T16:34:45.179639719+08:00" level=info msg="{\"error\": {\"class\": \"GenericError\", \"desc\": \"Could not open '/dev/fd0': No such device or address\"}}" arch=amd64 comman d=create container=a98c2d8a7bb80dca033eca387ba837754f55316876411dae7e4c0fad1600021b name=kata-runtime pid=169567 source=virtcontainers subsystem=qmp time="2019-04-17T16:34:45.200887173+08:00" level=error msg="QMP command failed: Could not open '/dev/fd0': No such device or address" arch=amd64 command=create container=a98c2d8a7bb80dca03 3eca387ba837754f55316876411dae7e4c0fad1600021b name=kata-runtime pid=169567 source=runtime time="2019-04-17T16:35:07.581504419+08:00" level=debug msg="Could not retrieve anything from storage" arch=amd64 command=create container=5b3633d8595090c15d16429796858e4856e0b4b542408a48a2 5a7fdb454f3062 name=kata-runtime pid=169665 source=virtcontainers subsystem=kata_agent time="2019-04-17T16:35:07.581544822+08:00" level=warning msg="load sandbox devices failed" arch=amd64 command=create container=5b3633d8595090c15d16429796858e4856e0b4b542408a48a25a7fdb454f3 062 error="open /run/vc/sbs/5b3633d8595090c15d16429796858e4856e0b4b542408a48a25a7fdb454f3062/devices.json: no such file or directory" name=kata-runtime pid=169665 sandbox=5b3633d8595090c15 d16429796858e4856e0b4b542408a48a25a7fdb454f3062 sandboxid=5b3633d8595090c15d16429796858e4856e0b4b542408a48a25a7fdb454f3062 source=virtcontainers subsystem=sandbox time="2019-04-17T16:35:08.922462833+08:00" level=info msg="{\"error\": {\"class\": \"GenericError\", \"desc\": \"Could not open '/dev/fd0': No such device or address\"}}" arch=amd64 comman d=create container=5b3633d8595090c15d16429796858e4856e0b4b542408a48a25a7fdb454f3062 name=kata-runtime pid=169665 source=virtcontainers subsystem=qmp time="2019-04-17T16:35:08.94287472+08:00" level=error msg="QMP command failed: Could not open '/dev/fd0': No such device or address" arch=amd64 command=create container=5b3633d8595090c15d1 6429796858e4856e0b4b542408a48a25a7fdb454f3062 name=kata-runtime pid=169665 source=runtime time="2019-04-17T17:41:17.155291155+08:00" level=debug msg="Could not retrieve anything from storage" arch=amd64 command=create container=b0618b3f02321e121f65f005a8099408fdd6b73af0a6e73141 44ecf5dc75a02e name=kata-runtime pid=173988 source=virtcontainers subsystem=kata_agent time="2019-04-17T17:41:17.155336873+08:00" level=warning msg="load sandbox devices failed" arch=amd64 command=create container=b0618b3f02321e121f65f005a8099408fdd6b73af0a6e7314144ecf5dc75a 02e error="open /run/vc/sbs/b0618b3f02321e121f65f005a8099408fdd6b73af0a6e7314144ecf5dc75a02e/devices.json: no such file or directory" name=kata-runtime pid=173988 sandbox=b0618b3f02321e121 f65f005a8099408fdd6b73af0a6e7314144ecf5dc75a02e sandboxid=b0618b3f02321e121f65f005a8099408fdd6b73af0a6e7314144ecf5dc75a02e source=virtcontainers subsystem=sandbox


## Proxy logs

Recent proxy problems found in system journal:

time="2019-04-10T15:31:02.747544202+08:00" level=info msg="time=\"2019-04-10T07:31:02.739084201Z\" level=info msg=\"ignoring unexpected signal\" name=kata-agent pid=97 sandbox=6456e69ec541 b10a7d04e00d70364490165f9205bdaf27e6829216320047fb47 signal=\"child exited\" source=agent\n" name=kata-proxy pid=103319 sandbox=6456e69ec541b10a7d04e00d70364490165f9205bdaf27e6829216320047 fb47 source=agent time="2019-04-10T15:31:02.876833586+08:00" level=fatal msg="channel error" error="accept unix /run/vc/sbs/6456e69ec541b10a7d04e00d70364490165f9205bdaf27e6829216320047fb47/proxy.sock: use o f closed network connection" name=kata-proxy pid=103319 sandbox=6456e69ec541b10a7d04e00d70364490165f9205bdaf27e6829216320047fb47 source=proxy time="2019-04-10T16:55:28.524929817+08:00" level=info msg="[ 0.338048] systemd[73]: tmp.mount: Failed to execute command: No such file or directory\n" name=kata-proxy pid=108738 sandbox =40037cb4147f224187eed7ace0bfbb382e6ccd33acc5fb0f5b2ab447b6173c77 source=agent time="2019-04-10T16:55:28.969383688+08:00" level=info msg="time=\"2019-04-10T08:55:28.96116418Z\" level=info msg=\"ignoring unexpected signal\" name=kata-agent pid=100 sandbox=40037cb4147f 224187eed7ace0bfbb382e6ccd33acc5fb0f5b2ab447b6173c77 signal=\"child exited\" source=agent\n" name=kata-proxy pid=108738 sandbox=40037cb4147f224187eed7ace0bfbb382e6ccd33acc5fb0f5b2ab447b617 3c77 source=agent time="2019-04-10T16:55:28.969525647+08:00" level=info msg="time=\"2019-04-10T08:55:28.96137286Z\" level=info msg=\"ignoring unexpected signal\" name=kata-agent pid=100 sandbox=40037cb4147f 224187eed7ace0bfbb382e6ccd33acc5fb0f5b2ab447b6173c77 signal=\"child exited\" source=agent\n" name=kata-proxy pid=108738 sandbox=40037cb4147f224187eed7ace0bfbb382e6ccd33acc5fb0f5b2ab447b617 3c77 source=agent time="2019-04-10T16:55:29.899920789+08:00" level=info msg="time=\"2019-04-10T08:55:29.891775121Z\" level=info msg=\"ignoring unexpected signal\" name=kata-agent pid=100 sandbox=40037cb4147 f224187eed7ace0bfbb382e6ccd33acc5fb0f5b2ab447b6173c77 signal=\"child exited\" source=agent\n" name=kata-proxy pid=108738 sandbox=40037cb4147f224187eed7ace0bfbb382e6ccd33acc5fb0f5b2ab447b61 73c77 source=agent time="2019-04-10T16:55:29.987071929+08:00" level=fatal msg="failed to handle exit signal" error="close unix @->/run/vc/vm/40037cb4147f224187eed7ace0bfbb382e6ccd33acc5fb0f5b2ab447b6173c77/k ata.sock: use of closed network connection" name=kata-proxy pid=108738 sandbox=40037cb4147f224187eed7ace0bfbb382e6ccd33acc5fb0f5b2ab447b6173c77 source=proxy time="2019-04-10T17:09:48.228477356+08:00" level=info msg="[ 0.331080] systemd[69]: tmp.mount: Failed to execute command: No such file or directory\n" name=kata-proxy pid=109884 sandbox =7a0027b0aab1f9896da8181f151a4f15d4b148f652763fa5b82579dfafea88f0 source=agent time="2019-04-10T17:09:48.698428965+08:00" level=info msg="time=\"2019-04-10T09:09:48.690667899Z\" level=info msg=\"ignoring unexpected signal\" name=kata-agent pid=95 sandbox=7a0027b0aab1 f9896da8181f151a4f15d4b148f652763fa5b82579dfafea88f0 signal=\"child exited\" source=agent\n" name=kata-proxy pid=109884 sandbox=7a0027b0aab1f9896da8181f151a4f15d4b148f652763fa5b82579dfafea 88f0 source=agent time="2019-04-10T17:09:48.698499521+08:00" level=info msg="time=\"2019-04-10T09:09:48.690901383Z\" level=info msg=\"ignoring unexpected signal\" name=kata-agent pid=95 sandbox=7a0027b0aab1 f9896da8181f151a4f15d4b148f652763fa5b82579dfafea88f0 signal=\"child exited\" source=agent\n" name=kata-proxy pid=109884 sandbox=7a0027b0aab1f9896da8181f151a4f15d4b148f652763fa5b82579dfafea 88f0 source=agent time="2019-04-10T17:09:48.699522008+08:00" level=debug msg="Copy stream error" error="read unix /run/vc/sbs/7a0027b0aab1f9896da8181f151a4f15d4b148f652763fa5b82579dfafea88f0/proxy.sock->@: read: connection reset by peer" name=kata-proxy pid=109884 sandbox=7a0027b0aab1f9896da8181f151a4f15d4b148f652763fa5b82579dfafea88f0 source=proxy time="2019-04-10T22:27:21.598428554+08:00" level=info msg="time=\"2019-04-10T14:27:21.589703056Z\" level=info msg=\"ignoring unexpected signal\" name=kata-agent pid=95 sandbox=7a0027b0aab1 f9896da8181f151a4f15d4b148f652763fa5b82579dfafea88f0 signal=\"child exited\" source=agent\n" name=kata-proxy pid=109884 sandbox=7a0027b0aab1f9896da8181f151a4f15d4b148f652763fa5b82579dfafea 88f0 source=agent time="2019-04-10T22:27:21.600380976+08:00" level=debug msg="Copy stream error" error="write unix /run/vc/sbs/7a0027b0aab1f9896da8181f151a4f15d4b148f652763fa5b82579dfafea88f0/proxy.sock->@: write: broken pipe" name=kata-proxy pid=109884 sandbox=7a0027b0aab1f9896da8181f151a4f15d4b148f652763fa5b82579dfafea88f0 source=proxy time="2019-04-10T22:27:21.732280423+08:00" level=fatal msg="channel error" error="accept unix /run/vc/sbs/7a0027b0aab1f9896da8181f151a4f15d4b148f652763fa5b82579dfafea88f0/proxy.sock: use o f closed network connection" name=kata-proxy pid=109884 sandbox=7a0027b0aab1f9896da8181f151a4f15d4b148f652763fa5b82579dfafea88f0 source=proxy time="2019-04-10T22:28:01.919382173+08:00" level=info msg="[ 0.378555] systemd[78]: tmp.mount: Failed to execute command: No such file or directory\n" name=kata-proxy pid=129799 sandbox =5af7576b444d7663300dbc50c74b2665ad84ddde0b3e74f8cdafe351c5759ef4 source=agent time="2019-04-10T22:28:02.404004604+08:00" level=info msg="time=\"2019-04-10T14:28:02.39414142Z\" level=info msg=\"ignoring unexpected signal\" name=kata-agent pid=96 sandbox=5af7576b444d7 663300dbc50c74b2665ad84ddde0b3e74f8cdafe351c5759ef4 signal=\"child exited\" source=agent\n" name=kata-proxy pid=129799 sandbox=5af7576b444d7663300dbc50c74b2665ad84ddde0b3e74f8cdafe351c5759 ef4 source=agent time="2019-04-10T22:28:02.404200896+08:00" level=info msg="time=\"2019-04-10T14:28:02.394478279Z\" level=info msg=\"ignoring unexpected signal\" name=kata-agent pid=96 sandbox=5af7576b444d 7663300dbc50c74b2665ad84ddde0b3e74f8cdafe351c5759ef4 signal=\"child exited\" source=agent\n" name=kata-proxy pid=129799 sandbox=5af7576b444d7663300dbc50c74b2665ad84ddde0b3e74f8cdafe351c575 9ef4 source=agent time="2019-04-10T22:28:02.529178492+08:00" level=debug msg="Copy stream error" error="read unix /run/vc/sbs/5af7576b444d7663300dbc50c74b2665ad84ddde0b3e74f8cdafe351c5759ef4/proxy.sock->@: read: connection reset by peer" name=kata-proxy pid=129799 sandbox=5af7576b444d7663300dbc50c74b2665ad84ddde0b3e74f8cdafe351c5759ef4 source=proxy time="2019-04-10T22:28:03.279321071+08:00" level=info msg="time=\"2019-04-10T14:28:03.269653035Z\" level=info msg=\"ignoring unexpected signal\" name=kata-agent pid=96 sandbox=5af7576b444d 7663300dbc50c74b2665ad84ddde0b3e74f8cdafe351c5759ef4 signal=\"child exited\" source=agent\n" name=kata-proxy pid=129799 sandbox=5af7576b444d7663300dbc50c74b2665ad84ddde0b3e74f8cdafe351c575 9ef4 source=agent time="2019-04-10T22:28:03.280975359+08:00" level=debug msg="Copy stream error" error="read unix /run/vc/sbs/5af7576b444d7663300dbc50c74b2665ad84ddde0b3e74f8cdafe351c5759ef4/proxy.sock->@: read: connection reset by peer" name=kata-proxy pid=129799 sandbox=5af7576b444d7663300dbc50c74b2665ad84ddde0b3e74f8cdafe351c5759ef4 source=proxy time="2019-04-10T22:28:03.388393597+08:00" level=fatal msg="channel error" error="accept unix /run/vc/sbs/5af7576b444d7663300dbc50c74b2665ad84ddde0b3e74f8cdafe351c5759ef4/proxy.sock: use o f closed network connection" name=kata-proxy pid=129799 sandbox=5af7576b444d7663300dbc50c74b2665ad84ddde0b3e74f8cdafe351c5759ef4 source=proxy time="2019-04-15T16:43:58.00227813+08:00" level=info msg="[ 0.353248] systemd[72]: tmp.mount: Failed to execute command: No such file or directory\n" name=kata-proxy pid=272076 sandbox= 3122d618a896589702f918616847f014ae48030eba0e7f2df2320284d29c85c2 source=agent time="2019-04-15T16:43:58.495244558+08:00" level=info msg="time=\"2019-04-15T08:43:58.485880146Z\" level=info msg=\"ignoring unexpected signal\" name=kata-agent pid=97 sandbox=3122d618a896 589702f918616847f014ae48030eba0e7f2df2320284d29c85c2 signal=\"child exited\" source=agent\n" name=kata-proxy pid=272076 sandbox=3122d618a896589702f918616847f014ae48030eba0e7f2df2320284d29c 85c2 source=agent time="2019-04-15T16:43:58.499142281+08:00" level=info msg="time=\"2019-04-15T08:43:58.486823321Z\" level=info msg=\"ignoring unexpected signal\" name=kata-agent pid=97 sandbox=3122d618a896 589702f918616847f014ae48030eba0e7f2df2320284d29c85c2 signal=\"child exited\" source=agent\n" name=kata-proxy pid=272076 sandbox=3122d618a896589702f918616847f014ae48030eba0e7f2df2320284d29c 85c2 source=agent time="2019-04-15T16:43:58.644905396+08:00" level=debug msg="Copy stream error" error="read unix /run/vc/sbs/3122d618a896589702f918616847f014ae48030eba0e7f2df2320284d29c85c2/proxy.sock->@: read: connection reset by peer" name=kata-proxy pid=272076 sandbox=3122d618a896589702f918616847f014ae48030eba0e7f2df2320284d29c85c2 source=proxy time="2019-04-16T17:07:45.521864015+08:00" level=info msg="[ 0.367060] systemd[72]: tmp.mount: Failed to execute command: No such file or directory\n" name=kata-proxy pid=82343 sandbox= 6d4b102a48e19d90eeb1ecc753c2ed246ca8d3f7f4d0840ea888207b1addb6b7 source=agent time="2019-04-16T17:07:45.927047559+08:00" level=fatal msg="failed to handle exit signal" error="close unix @->/run/vc/vm/6d4b102a48e19d90eeb1ecc753c2ed246ca8d3f7f4d0840ea888207b1addb6b7/k ata.sock: use of closed network connection" name=kata-proxy pid=82343 sandbox=6d4b102a48e19d90eeb1ecc753c2ed246ca8d3f7f4d0840ea888207b1addb6b7 source=proxy time="2019-04-16T17:07:59.878318099+08:00" level=info msg="[ 0.338545] systemd[73]: tmp.mount: Failed to execute command: No such file or directory\n" name=kata-proxy pid=82441 sandbox= 2b20edf0dc79b902b096386202f8b37caf27f1badfa8085d3384dd251ad75d28 source=agent time="2019-04-16T17:08:00.269508126+08:00" level=fatal msg="failed to handle exit signal" error="close unix @->/run/vc/vm/2b20edf0dc79b902b096386202f8b37caf27f1badfa8085d3384dd251ad75d28/k ata.sock: use of closed network connection" name=kata-proxy pid=82441 sandbox=2b20edf0dc79b902b096386202f8b37caf27f1badfa8085d3384dd251ad75d28 source=proxy time="2019-04-16T17:17:33.624211252+08:00" level=info msg="[ 0.356064] systemd[71]: tmp.mount: Failed to execute command: No such file or directory\n" name=kata-proxy pid=83110 sandbox= 37cb8ad988f540887bd9aa44d178c207f4d93f5dd44220c09f0a4356465a9e0d source=agent time="2019-04-16T17:17:34.021742069+08:00" level=fatal msg="failed to handle exit signal" error="close unix @->/run/vc/vm/37cb8ad988f540887bd9aa44d178c207f4d93f5dd44220c09f0a4356465a9e0d/k ata.sock: use of closed network connection" name=kata-proxy pid=83110 sandbox=37cb8ad988f540887bd9aa44d178c207f4d93f5dd44220c09f0a4356465a9e0d source=proxy time="2019-04-16T17:18:21.348444408+08:00" level=info msg="[ 0.317085] systemd[71]: tmp.mount: Failed to execute command: No such file or directory\n" name=kata-proxy pid=83227 sandbox= c727a814256d307b45a49fe7cb4822bffe418037577b968d2f37bef37fb21267 source=agent time="2019-04-16T17:18:21.711495692+08:00" level=fatal msg="channel error" error="accept unix /run/vc/sbs/c727a814256d307b45a49fe7cb4822bffe418037577b968d2f37bef37fb21267/proxy.sock: use o f closed network connection" name=kata-proxy pid=83227 sandbox=c727a814256d307b45a49fe7cb4822bffe418037577b968d2f37bef37fb21267 source=proxy time="2019-04-16T17:22:38.141326232+08:00" level=info msg="time=\"2019-04-16T09:22:38.127723743Z\" level=info msg=\"ignoring unexpected signal\" name=kata-agent pid=97 sandbox=3122d618a896 589702f918616847f014ae48030eba0e7f2df2320284d29c85c2 signal=\"child exited\" source=agent\n" name=kata-proxy pid=272076 sandbox=3122d618a896589702f918616847f014ae48030eba0e7f2df2320284d29c 85c2 source=agent time="2019-04-16T17:22:38.147641515+08:00" level=debug msg="Copy stream error" error="read unix /run/vc/sbs/3122d618a896589702f918616847f014ae48030eba0e7f2df2320284d29c85c2/proxy.sock->@: read: connection reset by peer" name=kata-proxy pid=272076 sandbox=3122d618a896589702f918616847f014ae48030eba0e7f2df2320284d29c85c2 source=proxy time="2019-04-16T17:22:38.280308978+08:00" level=fatal msg="failed to handle exit signal" error="close unix @->/run/vc/vm/3122d618a896589702f918616847f014ae48030eba0e7f2df2320284d29c85c2/k ata.sock: use of closed network connection" name=kata-proxy pid=272076 sandbox=3122d618a896589702f918616847f014ae48030eba0e7f2df2320284d29c85c2 source=proxy time="2019-04-16T17:23:15.113981787+08:00" level=info msg="[ 0.543932] systemd[75]: tmp.mount: Failed to execute command: No such file or directory\n" name=kata-proxy pid=83847 sandbox= d4c091664726ad991659f8e28e8e238b931e9f95a303cb6c3901eac4e389fcd6 source=agent time="2019-04-16T17:23:15.627971406+08:00" level=fatal msg="channel error" error="accept unix /run/vc/sbs/d4c091664726ad991659f8e28e8e238b931e9f95a303cb6c3901eac4e389fcd6/proxy.sock: use o f closed network connection" name=kata-proxy pid=83847 sandbox=d4c091664726ad991659f8e28e8e238b931e9f95a303cb6c3901eac4e389fcd6 source=proxy time="2019-04-17T15:52:12.664205393+08:00" level=info msg="[ 0.585070] systemd[82]: tmp.mount: Failed to execute command: No such file or directory\n" name=kata-proxy pid=166876 sandbox =d2e84d608f77a85c98f1d37115254a6e0b769b020ee111455a1412d92b45904b source=agent time="2019-04-17T15:52:13.13977264+08:00" level=fatal msg="channel error" error="accept unix /run/vc/sbs/d2e84d608f77a85c98f1d37115254a6e0b769b020ee111455a1412d92b45904b/proxy.sock: use of closed network connection" name=kata-proxy pid=166876 sandbox=d2e84d608f77a85c98f1d37115254a6e0b769b020ee111455a1412d92b45904b source=proxy time="2019-04-17T16:34:44.696314399+08:00" level=info msg="[ 0.545928] systemd[76]: tmp.mount: Failed to execute command: No such file or directory\n" name=kata-proxy pid=169606 sandbox =a98c2d8a7bb80dca033eca387ba837754f55316876411dae7e4c0fad1600021b source=agent time="2019-04-17T16:34:45.186585631+08:00" level=debug msg="Copy stream error" error="read unix /run/vc/sbs/a98c2d8a7bb80dca033eca387ba837754f55316876411dae7e4c0fad1600021b/proxy.sock->@: use of closed network connection" name=kata-proxy pid=169606 sandbox=a98c2d8a7bb80dca033eca387ba837754f55316876411dae7e4c0fad1600021b source=proxy time="2019-04-17T16:34:45.186593065+08:00" level=fatal msg="failed to handle exit signal" error="close unix @->/run/vc/vm/a98c2d8a7bb80dca033eca387ba837754f55316876411dae7e4c0fad1600021b/k ata.sock: use of closed network connection" name=kata-proxy pid=169606 sandbox=a98c2d8a7bb80dca033eca387ba837754f55316876411dae7e4c0fad1600021b source=proxy time="2019-04-17T16:35:08.412348611+08:00" level=info msg="[ 0.527783] systemd[75]: tmp.mount: Failed to execute command: No such file or directory\n" name=kata-proxy pid=169702 sandbox =5b3633d8595090c15d16429796858e4856e0b4b542408a48a25a7fdb454f3062 source=agent time="2019-04-17T16:35:08.927530209+08:00" level=fatal msg="failed to handle exit signal" error="close unix @->/run/vc/vm/5b3633d8595090c15d16429796858e4856e0b4b542408a48a25a7fdb454f3062/k ata.sock: use of closed network connection" name=kata-proxy pid=169702 sandbox=5b3633d8595090c15d16429796858e4856e0b4b542408a48a25a7fdb454f3062 source=proxy time="2019-04-17T17:41:18.014469476+08:00" level=info msg="[ 0.349249] systemd[73]: tmp.mount: Failed to execute command: No such file or directory\n" name=kata-proxy pid=174026 sandbox =b0618b3f02321e121f65f005a8099408fdd6b73af0a6e7314144ecf5dc75a02e source=agent time="2019-04-17T17:41:18.469755612+08:00" level=info msg="time=\"2019-04-17T09:41:18.462272003Z\" level=info msg=\"ignoring unexpected signal\" name=kata-agent pid=108 sandbox=b0618b3f023 21e121f65f005a8099408fdd6b73af0a6e7314144ecf5dc75a02e signal=\"child exited\" source=agent\n" name=kata-proxy pid=174026 sandbox=b0618b3f02321e121f65f005a8099408fdd6b73af0a6e7314144ecf5dc7 5a02e source=agent time="2019-04-17T17:41:18.469915366+08:00" level=info msg="time=\"2019-04-17T09:41:18.462491794Z\" level=info msg=\"ignoring unexpected signal\" name=kata-agent pid=108 sandbox=b0618b3f023 21e121f65f005a8099408fdd6b73af0a6e7314144ecf5dc75a02e signal=\"child exited\" source=agent\n" name=kata-proxy pid=174026 sandbox=b0618b3f02321e121f65f005a8099408fdd6b73af0a6e7314144ecf5dc7 5a02e source=agent time="2019-04-17T17:41:18.60879889+08:00" level=info msg="time=\"2019-04-17T09:41:18.601252407Z\" level=info msg=\"ignoring unexpected signal\" name=kata-agent pid=108 sandbox=b0618b3f0232 1e121f65f005a8099408fdd6b73af0a6e7314144ecf5dc75a02e signal=\"child exited\" source=agent\n" name=kata-proxy pid=174026 sandbox=b0618b3f02321e121f65f005a8099408fdd6b73af0a6e7314144ecf5dc75 a02e source=agent time="2019-04-17T17:41:18.720054909+08:00" level=fatal msg="failed to handle exit signal" error="close unix @->/run/vc/vm/b0618b3f02321e121f65f005a8099408fdd6b73af0a6e7314144ecf5dc75a02e/k ata.sock: use of closed network connection" name=kata-proxy pid=174026 sandbox=b0618b3f02321e121f65f005a8099408fdd6b73af0a6e7314144ecf5dc75a02e source=proxy


## Shim logs

Recent shim problems found in system journal:

time="2019-04-09T10:58:57.469193961+08:00" level=info msg="copy stdout failed" container=c215edafcbba900ab77412b5a074529afd7cfd92a88a75c4c0a3586d3f9c266f error="rpc error: code = Unknown d esc = read /dev/ptmx: input/output error" exec-id=c215edafcbba900ab77412b5a074529afd7cfd92a88a75c4c0a3586d3f9c266f name=kata-shim pid=1 source=shim time="2019-04-10T15:31:02.74639583+08:00" level=info msg="copy stdout failed" container=6456e69ec541b10a7d04e00d70364490165f9205bdaf27e6829216320047fb47 error="rpc error: code = Unknown de sc = read /dev/ptmx: input/output error" exec-id=6456e69ec541b10a7d04e00d70364490165f9205bdaf27e6829216320047fb47 name=kata-shim pid=1 source=shim time="2019-04-10T16:55:29.898108347+08:00" level=info msg="copy stdout failed" container=40037cb4147f224187eed7ace0bfbb382e6ccd33acc5fb0f5b2ab447b6173c77 error="rpc error: code = Unknown d esc = read /dev/ptmx: input/output error" exec-id=40037cb4147f224187eed7ace0bfbb382e6ccd33acc5fb0f5b2ab447b6173c77 name=kata-shim pid=1 source=shim time="2019-04-10T22:27:21.596785617+08:00" level=info msg="copy stdout failed" container=7a0027b0aab1f9896da8181f151a4f15d4b148f652763fa5b82579dfafea88f0 error="rpc error: code = Unknown d esc = read /dev/ptmx: input/output error" exec-id=7a0027b0aab1f9896da8181f151a4f15d4b148f652763fa5b82579dfafea88f0 name=kata-shim pid=1 source=shim time="2019-04-10T22:28:03.277721921+08:00" level=info msg="copy stdout failed" container=5af7576b444d7663300dbc50c74b2665ad84ddde0b3e74f8cdafe351c5759ef4 error="rpc error: code = Unknown d esc = read /dev/ptmx: input/output error" exec-id=5af7576b444d7663300dbc50c74b2665ad84ddde0b3e74f8cdafe351c5759ef4 name=kata-shim pid=1 source=shim time="2019-04-16T17:22:38.137872318+08:00" level=info msg="copy stdout failed" container=3122d618a896589702f918616847f014ae48030eba0e7f2df2320284d29c85c2 error="rpc error: code = Unknown d esc = read /dev/ptmx: input/output error" exec-id=3122d618a896589702f918616847f014ae48030eba0e7f2df2320284d29c85c2 name=kata-shim pid=1 source=shim time="2019-04-17T17:41:18.611470323+08:00" level=info msg="copy stdout failed" container=b0618b3f02321e121f65f005a8099408fdd6b73af0a6e7314144ecf5dc75a02e error="rpc error: code = Unknown d esc = read /dev/ptmx: input/output error" exec-id=b0618b3f02321e121f65f005a8099408fdd6b73af0a6e7314144ecf5dc75a02e name=kata-shim pid=1 source=shim


## Throttler logs

No recent throttler problems found in system journal.

--- 

# Container manager details

Have `docker`

## Docker

Output of "`docker version`":

Client: Version: 18.06.2-ce API version: 1.38 Go version: go1.10.3 Git commit: 6d37f41 Built: Sun Feb 10 03:46:03 2019 OS/Arch: linux/amd64 Experimental: false

Server: Engine: Version: 18.06.2-ce API version: 1.38 (minimum version 1.12) Go version: go1.10.3 Git commit: 6d37f41 Built: Sun Feb 10 03:48:29 2019 OS/Arch: linux/amd64 Experimental: false

Output of "`docker info`":

Containers: 14 Running: 0 Paused: 0 Stopped: 14 Images: 9 Server Version: 18.06.2-ce Storage Driver: overlay2 Backing Filesystem: extfs Supports d_type: true Native Overlay Diff: true Logging Driver: json-file Cgroup Driver: cgroupfs Plugins: Volume: local Network: bridge host macvlan null overlay Log: awslogs fluentd gcplogs gelf journald json-file logentries splunk syslog Swarm: inactive Runtimes: kata-runtime runc Default Runtime: kata-runtime Init Binary: docker-init containerd version: 468a545b9edcd5932818eb9de8e72413e616e86e runc version: 9f8d4e1 (expected: 69663f0bd4b60df09991c08812a60108003fa340) init version: fec3683 Security Options: seccomp Profile: default Kernel Version: 3.10.0-862.el7.x86_64 Operating System: CentOS Linux 7 (Core) OSType: linux Architecture: x86_64 CPUs: 4 Total Memory: 7.639GiB Name: centos-k8s014.sh.intel.com ID: GL32:HRHJ:IVSY:WKDP:UZ2S:FMVO:HP6B:A5KV:X7T6:DBBU:VT26:X6RP Docker Root Dir: /var/lib/docker Debug Mode (client): false Debug Mode (server): true File Descriptors: 22 Goroutines: 44 System Time: 2019-04-17T17:43:20.205014289+08:00 EventsListeners: 0 HTTP Proxy: http://child-prc.intel.com:913/ Registry: https://index.docker.io/v1/ Labels: Experimental: false Insecure Registries: 127.0.0.0/8 Live Restore Enabled: false

Output of "`systemctl show docker`":

Type=notify Restart=on-failure NotifyAccess=main RestartUSec=100ms TimeoutStartUSec=0 TimeoutStopUSec=1min 30s WatchdogUSec=0 WatchdogTimestamp=Wed 2019-04-17 17:40:25 CST WatchdogTimestampMonotonic=2534654669193 StartLimitInterval=60000000 StartLimitBurst=3 StartLimitAction=none FailureAction=none PermissionsStartOnly=no RootDirectoryStartOnly=no RemainAfterExit=no GuessMainPID=yes MainPID=173758 ControlPID=0 FileDescriptorStoreMax=0 StatusErrno=0 Result=success ExecMainStartTimestamp=Wed 2019-04-17 17:40:25 CST ExecMainStartTimestampMonotonic=2534654295380 ExecMainExitTimestampMonotonic=0 ExecMainPID=173758 ExecMainCode=0 ExecMainStatus=0 ExecStart={ path=/usr/bin/dockerd ; argv[]=/usr/bin/dockerd -D --add-runtime kata-runtime=/usr/bin/kata-runtime --default-runtime=kata-runtime ; ignore_errors=no ; start_time=[Wed 2019-04- 17 17:40:25 CST] ; stop_time=[n/a] ; pid=173758 ; code=(null) ; status=0/0 } ExecReload={ path=/bin/kill ; argv[]=/bin/kill -s HUP $MAINPID ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 } Slice=system.slice ControlGroup=/system.slice/docker.service MemoryCurrent=181837824 TasksCurrent=53 Delegate=yes CPUAccounting=no CPUShares=18446744073709551615 StartupCPUShares=18446744073709551615 CPUQuotaPerSecUSec=infinity BlockIOAccounting=no BlockIOWeight=18446744073709551615 StartupBlockIOWeight=18446744073709551615 MemoryAccounting=no MemoryLimit=18446744073709551615 DevicePolicy=auto TasksAccounting=no TasksMax=18446744073709551615 Environment=HTTP_PROXY=http://child-prc.intel.com:913/ UMask=0022 LimitCPU=18446744073709551615 LimitFSIZE=18446744073709551615 LimitDATA=18446744073709551615 LimitSTACK=18446744073709551615 LimitCORE=18446744073709551615 LimitRSS=18446744073709551615 LimitNOFILE=18446744073709551615 LimitAS=18446744073709551615 LimitNPROC=18446744073709551615 LimitMEMLOCK=65536 LimitLOCKS=18446744073709551615 LimitSIGPENDING=31165 LimitMSGQUEUE=819200 LimitNICE=0 LimitRTPRIO=0 LimitRTTIME=18446744073709551615 OOMScoreAdjust=0 Nice=0 IOScheduling=0 CPUSchedulingPolicy=0 CPUSchedulingPriority=0 TimerSlackNSec=50000 CPUSchedulingResetOnFork=no NonBlocking=no StandardInput=null StandardOutput=journal StandardError=inherit TTYReset=no TTYVHangup=no TTYVTDisallocate=no SyslogPriority=30 SyslogLevelPrefix=yes SecureBits=0 CapabilityBoundingSet=18446744073709551615 AmbientCapabilities=0 MountFlags=0 PrivateTmp=no PrivateNetwork=no PrivateDevices=no ProtectHome=no ProtectSystem=no SameProcessGroup=no IgnoreSIGPIPE=yes NoNewPrivileges=no SystemCallErrorNumber=0 RuntimeDirectoryMode=0755 KillMode=process KillSignal=15 SendSIGKILL=yes SendSIGHUP=no Id=docker.service Names=docker.service Requires=basic.target Wants=system.slice network-online.target Conflicts=shutdown.target Before=shutdown.target After=network-online.target firewalld.service systemd-journald.socket system.slice basic.target Documentation=https://docs.docker.com Description=Docker Application Container Engine LoadState=loaded ActiveState=active SubState=running FragmentPath=/usr/lib/systemd/system/docker.service DropInPaths=/etc/systemd/system/docker.service.d/kata-container.conf UnitFileState=disabled UnitFilePreset=disabled InactiveExitTimestamp=Wed 2019-04-17 17:40:25 CST InactiveExitTimestampMonotonic=2534654295426 ActiveEnterTimestamp=Wed 2019-04-17 17:40:25 CST ActiveEnterTimestampMonotonic=2534654669240 ActiveExitTimestamp=Wed 2019-04-17 17:40:24 CST ActiveExitTimestampMonotonic=2534653287397 InactiveEnterTimestamp=Wed 2019-04-17 17:40:25 CST InactiveEnterTimestampMonotonic=2534654294093 CanStart=yes CanStop=yes CanReload=yes CanIsolate=no StopWhenUnneeded=no RefuseManualStart=no RefuseManualStop=no AllowIsolate=no DefaultDependencies=yes OnFailureJobMode=replace IgnoreOnIsolate=no IgnoreOnSnapshot=no NeedDaemonReload=no JobTimeoutUSec=0 JobTimeoutAction=none ConditionResult=yes AssertResult=yes ConditionTimestamp=Wed 2019-04-17 17:40:25 CST ConditionTimestampMonotonic=2534654294509 AssertTimestamp=Wed 2019-04-17 17:40:25 CST AssertTimestampMonotonic=2534654294509 Transient=no

No `kubectl`

--- 

# Packages

No `dpkg`
Have `rpm`
Output of "`rpm -qa|egrep "(cc-oci-runtimecc-runtimerunv|kata-proxy|kata-runtime|kata-shim|kata-ksm-throttler|kata-containers-image|linux-container|qemu-)"`":

qemu-lite-data-2.11.0+git.87517afd72-21.1.x86_64 kata-proxy-bin-1.6.0~rc1+git.2085829-18.1.x86_64 kata-ksm-throttler-1.6.0~rc1.git+ce5b765-20.1.x86_64 qemu-lite-bin-2.11.0+git.87517afd72-21.1.x86_64 kata-containers-image-1.6.0~rc1-17.1.x86_64 qemu-vanilla-2.11.2+git.0982a56a55-21.1.x86_64 qemu-kvm-1.5.3-156.el7.x86_64 kata-runtime-1.6.0~rc1+git.9f8d4e1-26.1.x86_64 libvirt-daemon-driver-qemu-3.9.0-14.el7.x86_64 qemu-vanilla-bin-2.11.2+git.0982a56a55-21.1.x86_64 kata-linux-container-4.19.24.25-20.1.x86_64 qemu-system-x86-2.0.0-1.el7.6.x86_64 qemu-img-1.5.3-156.el7.x86_64 qemu-kvm-common-1.5.3-156.el7.x86_64 kata-shim-1.6.0~rc1+git.0f41347-16.1.x86_64 qemu-guest-agent-2.8.0-2.el7.x86_64 kata-shim-bin-1.6.0~rc1+git.0f41347-16.1.x86_64 ipxe-roms-qemu-20170123-1.git4e85b27.el7_4.1.noarch kata-proxy-1.6.0~rc1+git.2085829-18.1.x86_64 qemu-vanilla-data-2.11.2+git.0982a56a55-21.1.x86_64 qemu-lite-2.11.0+git.87517afd72-21.1.x86_64 qemu-common-2.0.0-1.el7.6.x86_64



---
amshinde commented 5 years ago

@zhiminghufighting this error here looks similar to the one in #1551. Can you give this a try with the fix I mentioned there.

zhiminghufighting commented 5 years ago

@amshinde Yes. the fix for '--privileged' works for this issue. Thanks!