search

kata-containers / runtime

Kata Containers version 1.x runtime (for version 2.x see https://github.com/kata-containers/kata-containers).
https://katacontainers.io/
Apache License 2.0
2.1k stars 375 forks source link

Binding cpu failed #1278

Closed free2k closed 5 years ago

free2k commented 5 years ago

Description of problem

docker run --runtime kata-runtime --name kata-test --cpus 8 --cpuset-cpus="1-8" --memory 16G -itd daocloud.io/library/centos /bin/sh This command can be successfully executed, and you can see that the cpu is bound to the 1-8 core.

docker run --runtime kata-runtime --name kata-test --cpus 8 --cpuset-cpus="9-16" --memory 16G -itd daocloud.io/library/centos /bin/sh This command can fail to execute and cannot be bound to the 9-16 core. The physical machine is 40 core

Expected result

Can be successfully bound to the 9-16 core

Actual result

docker run --runtime kata-runtime --name kata-test --cpus 8 --cpuset-cpus="9-16" --memory 16G -itd daocloud.io/library/centos /bin/sh

39dcbc28f5ec08034207f57050834d5909e0004f3392ce4570adbf4371471b13 /usr/bin/docker-current: Error response from daemon: oci runtime error: rpc error: code = Internal desc = Could not run process: container_linux.go:348: starting container process caused "process_linux.go:279: applying cgroup configuration for process caused \"failed to write 9-16 to cpuset.cpus: write /sys/fs/cgroup/cpuset/system.slice:docker:39dcbc28f5ec08034207f57050834d5909e0004f3392ce4570adbf4371471b13/cpuset.cpus: invalid argument\"".

Meta details

Running kata-collect-data.sh version 1.5.0 (commit 5f7fcd7) at 2019-02-25.20:55:50.235871978+0800.

Runtime is /bin/kata-runtime.

kata-env

Output of "/bin/kata-runtime kata-env":

[Meta]
  Version = "1.0.20"

[Runtime]
  Debug = false
  Trace = false
  DisableGuestSeccomp = true
  DisableNewNetNs = false
  Path = "/usr/bin/kata-runtime"
  [Runtime.Version]
    Semver = "1.5.0"
    Commit = "5f7fcd7"
    OCI = "1.0.1-dev"
  [Runtime.Config]
    Path = "/usr/share/defaults/kata-containers/configuration.toml"

[Hypervisor]
  MachineType = "pc"
  Version = "QEMU emulator version 2.11.0\nCopyright (c) 2003-2017 Fabrice Bellard and the QEMU Project developers"
  Path = "/usr/bin/qemu-lite-system-x86_64"
  BlockDeviceDriver = "virtio-scsi"
  EntropySource = "/dev/urandom"
  Msize9p = 8192
  MemorySlots = 10
  Debug = false
  UseVSock = false

[Image]
  Path = "/usr/share/kata-containers/kata-containers-image_clearlinux_1.5.0_agent_a581aebf473.img"

[Kernel]
  Path = "/usr/share/kata-containers/vmlinuz-4.14.67.22-18.1.container"
  Parameters = "init=/usr/lib/systemd/systemd systemd.unit=kata-containers.target systemd.mask=systemd-networkd.service systemd.mask=systemd-networkd.socket"

[Initrd]
  Path = ""

[Proxy]
  Type = "kataProxy"
  Version = "kata-proxy version 1.5.0-9e77a0b"
  Path = "/usr/libexec/kata-containers/kata-proxy"
  Debug = false

[Shim]
  Type = "kataShim"
  Version = "kata-shim version 1.5.0-efbf3bb"
  Path = "/usr/libexec/kata-containers/kata-shim"
  Debug = false

[Agent]
  Type = "kata"

[Host]
  Kernel = "3.10.0-862.mt20181225.115.el7.x86_64"
  Architecture = "amd64"
  VMContainerCapable = true
  SupportVSocks = false
  [Host.Distro]
    Name = "CentOS Linux"
    Version = "7"
  [Host.CPU]
    Vendor = "GenuineIntel"
    Model = "Intel(R) Xeon(R) CPU E5-2630 v4 @ 2.20GHz"

[Netmon]
  Version = "kata-netmon version 1.5.0"
  Path = "/usr/libexec/kata-containers/kata-netmon"
  Debug = false
  Enable = false

Runtime config files

Runtime default config files

/etc/kata-containers/configuration.toml
/usr/share/defaults/kata-containers/configuration.toml

Runtime config file contents

Config file /etc/kata-containers/configuration.toml not found Output of "cat "/usr/share/defaults/kata-containers/configuration.toml"":

# Copyright (c) 2017-2019 Intel Corporation
#
# SPDX-License-Identifier: Apache-2.0
#

# XXX: WARNING: this file is auto-generated.
# XXX:
# XXX: Source file: "cli/config/configuration-qemu.toml.in"
# XXX: Project:
# XXX:   Name: Kata Containers
# XXX:   Type: kata

[hypervisor.qemu]
path = "/usr/bin/qemu-lite-system-x86_64"
kernel = "/usr/share/kata-containers/vmlinuz.container"
image = "/usr/share/kata-containers/kata-containers.img"
machine_type = "pc"

# Optional space-separated list of options to pass to the guest kernel.
# For example, use `kernel_params = "vsyscall=emulate"` if you are having
# trouble running pre-2.15 glibc.
#
# WARNING: - any parameter specified here will take priority over the default
# parameter value of the same name used to start the virtual machine.
# Do not set values here unless you understand the impact of doing so as you
# may stop the virtual machine from booting.
# To see the list of default parameters, enable hypervisor debug, create a
# container and look for 'default-kernel-parameters' log entries.
kernel_params = ""

# Path to the firmware.
# If you want that qemu uses the default firmware leave this option empty
firmware = ""

# Machine accelerators
# comma-separated list of machine accelerators to pass to the hypervisor.
# For example, `machine_accelerators = "nosmm,nosmbus,nosata,nopit,static-prt,nofw"`
machine_accelerators=""

# Default number of vCPUs per SB/VM:
# unspecified or 0                --> will be set to 1
# < 0                             --> will be set to the actual number of physical cores
# > 0 <= number of physical cores --> will be set to the specified number
# > number of physical cores      --> will be set to the actual number of physical cores
default_vcpus = 1

# Default maximum number of vCPUs per SB/VM:
# unspecified or == 0             --> will be set to the actual number of physical cores or to the maximum number
#                                     of vCPUs supported by KVM if that number is exceeded
# > 0 <= number of physical cores --> will be set to the specified number
# > number of physical cores      --> will be set to the actual number of physical cores or to the maximum number
#                                     of vCPUs supported by KVM if that number is exceeded
# WARNING: Depending of the architecture, the maximum number of vCPUs supported by KVM is used when
# the actual number of physical cores is greater than it.
# WARNING: Be aware that this value impacts the virtual machine's memory footprint and CPU
# the hotplug functionality. For example, `default_maxvcpus = 240` specifies that until 240 vCPUs
# can be added to a SB/VM, but the memory footprint will be big. Another example, with
# `default_maxvcpus = 8` the memory footprint will be small, but 8 will be the maximum number of
# vCPUs supported by the SB/VM. In general, we recommend that you do not edit this variable,
# unless you know what are you doing.
default_maxvcpus = 0

# Bridges can be used to hot plug devices.
# Limitations:
# * Currently only pci bridges are supported
# * Until 30 devices per bridge can be hot plugged.
# * Until 5 PCI bridges can be cold plugged per VM.
#   This limitation could be a bug in qemu or in the kernel
# Default number of bridges per SB/VM:
# unspecified or 0   --> will be set to 1
# > 1 <= 5           --> will be set to the specified number
# > 5                --> will be set to 5
default_bridges = 1

# Default memory size in MiB for SB/VM.
# If unspecified then it will be set 2048 MiB.
default_memory = 2048
#
# Default memory slots per SB/VM.
# If unspecified then it will be set 10.
# This is will determine the times that memory will be hotadded to sandbox/VM.
#memory_slots = 10

# The size in MiB will be plused to max memory of hypervisor.
# It is the memory address space for the NVDIMM devie.
# If set block storage driver (block_device_driver) to "nvdimm",
# should set memory_offset to the size of block device.
# Default 0
#memory_offset = 0

# Disable block device from being used for a container's rootfs.
# In case of a storage driver like devicemapper where a container's
# root file system is backed by a block device, the block device is passed
# directly to the hypervisor for performance reasons.
# This flag prevents the block device from being passed to the hypervisor,
# 9pfs is used instead to pass the rootfs.
disable_block_device_use = false

# Block storage driver to be used for the hypervisor in case the container
# rootfs is backed by a block device. This is virtio-scsi, virtio-blk
# or nvdimm.
block_device_driver = "virtio-scsi"

# Specifies cache-related options will be set to block devices or not.
# Default false
#block_device_cache_set = true

# Specifies cache-related options for block devices.
# Denotes whether use of O_DIRECT (bypass the host page cache) is enabled.
# Default false
#block_device_cache_direct = true

# Specifies cache-related options for block devices.
# Denotes whether flush requests for the device are ignored.
# Default false
#block_device_cache_noflush = true

# Enable iothreads (data-plane) to be used. This causes IO to be
# handled in a separate IO thread. This is currently only implemented
# for SCSI.
#
enable_iothreads = false

# Enable pre allocation of VM RAM, default false
# Enabling this will result in lower container density
# as all of the memory will be allocated and locked
# This is useful when you want to reserve all the memory
# upfront or in the cases where you want memory latencies
# to be very predictable
# Default false
#enable_mem_prealloc = true

# Enable huge pages for VM RAM, default false
# Enabling this will result in the VM memory
# being allocated using huge pages.
# This is useful when you want to use vhost-user network
# stacks within the container. This will automatically
# result in memory pre allocation
#enable_hugepages = true

# Enable swap of vm memory. Default false.
# The behaviour is undefined if mem_prealloc is also set to true
#enable_swap = true

# This option changes the default hypervisor and kernel parameters
# to enable debug output where available. This extra output is added
# to the proxy logs, but only when proxy debug is also enabled.
#
# Default false
#enable_debug = true

# Disable the customizations done in the runtime when it detects
# that it is running on top a VMM. This will result in the runtime
# behaving as it would when running on bare metal.
#
#disable_nesting_checks = true

# This is the msize used for 9p shares. It is the number of bytes
# used for 9p packet payload.
#msize_9p = 8192

# If true and vsocks are supported, use vsocks to communicate directly
# with the agent and no proxy is started, otherwise use unix
# sockets and start a proxy to communicate with the agent.
# Default false
#use_vsock = true

# VFIO devices are hotplugged on a bridge by default.
# Enable hotplugging on root bus. This may be required for devices with
# a large PCI bar, as this is a current limitation with hotplugging on
# a bridge. This value is valid for "pc" machine type.
# Default false
#hotplug_vfio_on_root_bus = true

# If host doesn't support vhost_net, set to true. Thus we won't create vhost fds for nics.
# Default false
#disable_vhost_net = true
#
# Default entropy source.
# The path to a host source of entropy (including a real hardware RNG)
# /dev/urandom and /dev/random are two main options.
# Be aware that /dev/random is a blocking source of entropy.  If the host
# runs out of entropy, the VMs boot time will increase leading to get startup
# timeouts.
# The source of entropy /dev/urandom is non-blocking and provides a
# generally acceptable source of entropy. It should work well for pretty much
# all practical purposes.
#entropy_source= "/dev/urandom"

# Path to OCI hook binaries in the *guest rootfs*.
# This does not affect host-side hooks which must instead be added to
# the OCI spec passed to the runtime.
#
# You can create a rootfs with hooks by customizing the osbuilder scripts:
# https://github.com/kata-containers/osbuilder
#
# Hooks must be stored in a subdirectory of guest_hook_path according to their
# hook type, i.e. "guest_hook_path/{prestart,postart,poststop}".
# The agent will scan these directories for executable files and add them, in
# lexicographical order, to the lifecycle of the guest container.
# Hooks are executed in the runtime namespace of the guest. See the official documentation:
# https://github.com/opencontainers/runtime-spec/blob/v1.0.1/config.md#posix-platform-hooks
# Warnings will be logged if any error is encountered will scanning for hooks,
# but it will not abort container execution.
#guest_hook_path = "/usr/share/oci/hooks"

[factory]
# VM templating support. Once enabled, new VMs are created from template
# using vm cloning. They will share the same initial kernel, initramfs and
# agent memory by mapping it readonly. It helps speeding up new container
# creation and saves a lot of memory if there are many kata containers running
# on the same host.
#
# When disabled, new VMs are created from scratch.
#
# Note: Requires "initrd=" to be set ("image=" is not supported).
#
# Default false
#enable_template = true

[proxy.kata]
path = "/usr/libexec/kata-containers/kata-proxy"

# If enabled, proxy messages will be sent to the system log
# (default: disabled)
#enable_debug = true

[shim.kata]
path = "/usr/libexec/kata-containers/kata-shim"

# If enabled, shim messages will be sent to the system log
# (default: disabled)
#enable_debug = true

# If enabled, the shim will create opentracing.io traces and spans.
# (See https://www.jaegertracing.io/docs/getting-started).
#
# Note: By default, the shim runs in a separate network namespace. Therefore,
# to allow it to send trace details to the Jaeger agent running on the host,
# it is necessary to set 'disable_new_netns=true' so that it runs in the host
# network namespace.
#
# (default: disabled)
#enable_tracing = true

[agent.kata]
# There is no field for this section. The goal is only to be able to
# specify which type of agent the user wants to use.

[netmon]
# If enabled, the network monitoring process gets started when the
# sandbox is created. This allows for the detection of some additional
# network being added to the existing network namespace, after the
# sandbox has been created.
# (default: disabled)
#enable_netmon = true

# Specify the path to the netmon binary.
path = "/usr/libexec/kata-containers/kata-netmon"

# If enabled, netmon messages will be sent to the system log
# (default: disabled)
#enable_debug = true

[runtime]
# If enabled, the runtime will log additional debug messages to the
# system log
# (default: disabled)
#enable_debug = true
#
# Internetworking model
# Determines how the VM should be connected to the
# the container network interface
# Options:
#
#   - bridged
#     Uses a linux bridge to interconnect the container interface to
#     the VM. Works for most cases except macvlan and ipvlan.
#
#   - macvtap
#     Used when the Container network interface can be bridged using
#     macvtap.
#
#   - none
#     Used when customize network. Only creates a tap device. No veth pair.
#
#   - tcfilter
#     Uses tc filter rules to redirect traffic from the network interface
#     provided by plugin to a tap interface connected to the VM.
#
internetworking_model="macvtap"

# disable guest seccomp
# Determines whether container seccomp profiles are passed to the virtual
# machine and applied by the kata agent. If set to true, seccomp is not applied
# within the guest
# (default: true)
disable_guest_seccomp=true

# If enabled, the runtime will create opentracing.io traces and spans.
# (See https://www.jaegertracing.io/docs/getting-started).
# (default: disabled)
#enable_tracing = true

# If enabled, the runtime will not create a network namespace for shim and hypervisor processes.
# This option may have some potential impacts to your host. It should only be used when you know what you're doing.
# `disable_new_netns` conflicts with `enable_netmon`
# `disable_new_netns` conflicts with `internetworking_model=bridged` and `internetworking_model=macvtap`. It works only
# with `internetworking_model=none`. The tap device will be in the host network namespace and can connect to a bridge
# (like OVS) directly.
# If you are using docker, `disable_new_netns` only works with `docker run --net=none`
# (default: false)
#disable_new_netns = true

KSM throttler

version

Output of "/usr/libexec/kata-ksm-throttler/kata-ksm-throttler --version":

kata-ksm-throttler version 1.5.0-3dd4c9f

Output of "/usr/lib/systemd/system/kata-ksm-throttler.service --version":

/bin/kata-collect-data.sh: line 168: /usr/lib/systemd/system/kata-ksm-throttler.service: Permission denied

systemd service

Image details

---
osbuilder:
  url: "https://github.com/kata-containers/osbuilder"
  version: "unknown"
rootfs-creation-time: "2019-01-22T21:10:11.082469062+0000Z"
description: "osbuilder rootfs"
file-format-version: "0.0.2"
architecture: "x86_64"
base-distro:
  name: "Clear"
  version: "27390"
  packages:
    default:
      - "iptables-bin"
      - "libudev0-shim"
      - "systemd"
    extra:

agent:
  url: "https://github.com/kata-containers/agent"
  name: "kata-agent"
  version: "1.5.0-a581aebf47386df570f796cb835bd23820448b9b"
  agent-is-init-daemon: "no"

Initrd details

No initrd

Logfiles

Runtime logs

Recent runtime problems found in system journal:

time="2019-02-24T14:44:20.214746117+08:00" level=warning msg="unsupported address" address="fe80::42:acff:fe11:2/64" arch=amd64 command=create container=8213cb0d7710154d5298732314501c1101bd0bed35b9037ef05efc41824b3a35 name=kata-runtime pid=301489 source=virtcontainers subsystem=network unsupported-address-type=ipv6
time="2019-02-24T14:44:20.214905094+08:00" level=warning msg="unsupported route" arch=amd64 command=create container=8213cb0d7710154d5298732314501c1101bd0bed35b9037ef05efc41824b3a35 destination="fe80::/64" name=kata-runtime pid=301489 source=virtcontainers subsystem=network unsupported-route-type=ipv6
time="2019-02-24T14:44:33.09212893+08:00" level=error msg="Container ID (8213cb0d7710154d5298732314501c1101bd0bed35b9037ef05efc41824b3a35) does not exist" arch=amd64 command=delete container=8213cb0d7710154d5298732314501c1101bd0bed35b9037ef05efc41824b3a35 name=kata-runtime pid=302100 source=runtime
time="2019-02-24T14:44:51.153606564+08:00" level=warning msg="fetch sandbox device failed" arch=amd64 command=create container=bb06265a82e51c11d26fc3000dd11f02f05be58d3420b34c27b2b55b36d75659 error="open /run/vc/sbs/bb06265a82e51c11d26fc3000dd11f02f05be58d3420b34c27b2b55b36d75659/devices.json: no such file or directory" name=kata-runtime pid=302655 sandbox=bb06265a82e51c11d26fc3000dd11f02f05be58d3420b34c27b2b55b36d75659 sandboxid=bb06265a82e51c11d26fc3000dd11f02f05be58d3420b34c27b2b55b36d75659 source=virtcontainers subsystem=sandbox
time="2019-02-24T14:44:52.054495516+08:00" level=warning msg="unsupported address" address="fe80::42:acff:fe11:2/64" arch=amd64 command=create container=bb06265a82e51c11d26fc3000dd11f02f05be58d3420b34c27b2b55b36d75659 name=kata-runtime pid=302655 source=virtcontainers subsystem=network unsupported-address-type=ipv6
time="2019-02-24T14:44:52.054667274+08:00" level=warning msg="unsupported route" arch=amd64 command=create container=bb06265a82e51c11d26fc3000dd11f02f05be58d3420b34c27b2b55b36d75659 destination="fe80::/64" name=kata-runtime pid=302655 source=virtcontainers subsystem=network unsupported-route-type=ipv6
time="2019-02-24T14:44:53.158124581+08:00" level=error msg="rpc error: code = Internal desc = Could not run process: container_linux.go:348: starting container process caused \"process_linux.go:279: applying cgroup configuration for process caused \\\"failed to write 10 to cpuset.cpus: write /sys/fs/cgroup/cpuset/system.slice:docker:bb06265a82e51c11d26fc3000dd11f02f05be58d3420b34c27b2b55b36d75659/cpuset.cpus: invalid argument\\\"\"" arch=amd64 command=create container=bb06265a82e51c11d26fc3000dd11f02f05be58d3420b34c27b2b55b36d75659 name=kata-runtime pid=302655 source=runtime
time="2019-02-24T14:44:53.209520424+08:00" level=error msg="Container ID (bb06265a82e51c11d26fc3000dd11f02f05be58d3420b34c27b2b55b36d75659) does not exist" arch=amd64 command=delete container=bb06265a82e51c11d26fc3000dd11f02f05be58d3420b34c27b2b55b36d75659 name=kata-runtime pid=302792 source=runtime
time="2019-02-24T14:44:53.276153904+08:00" level=error msg="Container ID (bb06265a82e51c11d26fc3000dd11f02f05be58d3420b34c27b2b55b36d75659) does not exist" arch=amd64 command=delete container=bb06265a82e51c11d26fc3000dd11f02f05be58d3420b34c27b2b55b36d75659 name=kata-runtime pid=302813 source=runtime
time="2019-02-24T14:49:36.3972274+08:00" level=warning msg="fetch sandbox device failed" arch=amd64 command=create container=0748d542839b6c6c364fcffdd17cf51455e9faad71b4573c4a5ec35a2e749e47 error="open /run/vc/sbs/0748d542839b6c6c364fcffdd17cf51455e9faad71b4573c4a5ec35a2e749e47/devices.json: no such file or directory" name=kata-runtime pid=311057 sandbox=0748d542839b6c6c364fcffdd17cf51455e9faad71b4573c4a5ec35a2e749e47 sandboxid=0748d542839b6c6c364fcffdd17cf51455e9faad71b4573c4a5ec35a2e749e47 source=virtcontainers subsystem=sandbox
time="2019-02-24T14:49:37.362370882+08:00" level=warning msg="unsupported address" address="fe80::42:acff:fe11:2/64" arch=amd64 command=create container=0748d542839b6c6c364fcffdd17cf51455e9faad71b4573c4a5ec35a2e749e47 name=kata-runtime pid=311057 source=virtcontainers subsystem=network unsupported-address-type=ipv6
time="2019-02-24T14:49:37.36257257+08:00" level=warning msg="unsupported route" arch=amd64 command=create container=0748d542839b6c6c364fcffdd17cf51455e9faad71b4573c4a5ec35a2e749e47 destination="fe80::/64" name=kata-runtime pid=311057 source=virtcontainers subsystem=network unsupported-route-type=ipv6
time="2019-02-24T14:49:38.400230282+08:00" level=error msg="rpc error: code = Internal desc = Could not run process: container_linux.go:348: starting container process caused \"process_linux.go:279: applying cgroup configuration for process caused \\\"failed to write 35 to cpuset.cpus: write /sys/fs/cgroup/cpuset/system.slice:docker:0748d542839b6c6c364fcffdd17cf51455e9faad71b4573c4a5ec35a2e749e47/cpuset.cpus: invalid argument\\\"\"" arch=amd64 command=create container=0748d542839b6c6c364fcffdd17cf51455e9faad71b4573c4a5ec35a2e749e47 name=kata-runtime pid=311057 source=runtime
time="2019-02-24T14:49:38.454654315+08:00" level=error msg="Container ID (0748d542839b6c6c364fcffdd17cf51455e9faad71b4573c4a5ec35a2e749e47) does not exist" arch=amd64 command=delete container=0748d542839b6c6c364fcffdd17cf51455e9faad71b4573c4a5ec35a2e749e47 name=kata-runtime pid=311197 source=runtime
time="2019-02-24T14:49:38.515404114+08:00" level=error msg="Container ID (0748d542839b6c6c364fcffdd17cf51455e9faad71b4573c4a5ec35a2e749e47) does not exist" arch=amd64 command=delete container=0748d542839b6c6c364fcffdd17cf51455e9faad71b4573c4a5ec35a2e749e47 name=kata-runtime pid=311215 source=runtime
time="2019-02-24T14:51:30.272090477+08:00" level=warning msg="fetch sandbox device failed" arch=amd64 command=create container=fa33f6b0f2c8b71b830684002744e0770df2a2da9a41b921f578b4d015cc030a error="open /run/vc/sbs/fa33f6b0f2c8b71b830684002744e0770df2a2da9a41b921f578b4d015cc030a/devices.json: no such file or directory" name=kata-runtime pid=314507 sandbox=fa33f6b0f2c8b71b830684002744e0770df2a2da9a41b921f578b4d015cc030a sandboxid=fa33f6b0f2c8b71b830684002744e0770df2a2da9a41b921f578b4d015cc030a source=virtcontainers subsystem=sandbox
time="2019-02-24T14:51:31.240010978+08:00" level=warning msg="unsupported address" address="fe80::42:acff:fe11:2/64" arch=amd64 command=create container=fa33f6b0f2c8b71b830684002744e0770df2a2da9a41b921f578b4d015cc030a name=kata-runtime pid=314507 source=virtcontainers subsystem=network unsupported-address-type=ipv6
time="2019-02-24T14:51:31.240146586+08:00" level=warning msg="unsupported route" arch=amd64 command=create container=fa33f6b0f2c8b71b830684002744e0770df2a2da9a41b921f578b4d015cc030a destination="fe80::/64" name=kata-runtime pid=314507 source=virtcontainers subsystem=network unsupported-route-type=ipv6
time="2019-02-24T14:51:32.267147095+08:00" level=error msg="rpc error: code = Internal desc = Could not run process: container_linux.go:348: starting container process caused \"process_linux.go:279: applying cgroup configuration for process caused \\\"failed to write 10 to cpuset.cpus: write /sys/fs/cgroup/cpuset/system.slice:docker:fa33f6b0f2c8b71b830684002744e0770df2a2da9a41b921f578b4d015cc030a/cpuset.cpus: invalid argument\\\"\"" arch=amd64 command=create container=fa33f6b0f2c8b71b830684002744e0770df2a2da9a41b921f578b4d015cc030a name=kata-runtime pid=314507 source=runtime
time="2019-02-24T14:51:32.322790648+08:00" level=error msg="Container ID (fa33f6b0f2c8b71b830684002744e0770df2a2da9a41b921f578b4d015cc030a) does not exist" arch=amd64 command=delete container=fa33f6b0f2c8b71b830684002744e0770df2a2da9a41b921f578b4d015cc030a name=kata-runtime pid=314691 source=runtime
time="2019-02-24T14:51:32.374351959+08:00" level=error msg="Container ID (fa33f6b0f2c8b71b830684002744e0770df2a2da9a41b921f578b4d015cc030a) does not exist" arch=amd64 command=delete container=fa33f6b0f2c8b71b830684002744e0770df2a2da9a41b921f578b4d015cc030a name=kata-runtime pid=314709 source=runtime
time="2019-02-24T14:51:50.395718463+08:00" level=warning msg="fetch sandbox device failed" arch=amd64 command=create container=879d2f317bc7ba1e737f611f9f78647f2e9c539cf8347f6a8664a96587995236 error="open /run/vc/sbs/879d2f317bc7ba1e737f611f9f78647f2e9c539cf8347f6a8664a96587995236/devices.json: no such file or directory" name=kata-runtime pid=315246 sandbox=879d2f317bc7ba1e737f611f9f78647f2e9c539cf8347f6a8664a96587995236 sandboxid=879d2f317bc7ba1e737f611f9f78647f2e9c539cf8347f6a8664a96587995236 source=virtcontainers subsystem=sandbox
time="2019-02-24T14:51:51.31166005+08:00" level=warning msg="unsupported address" address="fe80::42:acff:fe11:2/64" arch=amd64 command=create container=879d2f317bc7ba1e737f611f9f78647f2e9c539cf8347f6a8664a96587995236 name=kata-runtime pid=315246 source=virtcontainers subsystem=network unsupported-address-type=ipv6
time="2019-02-24T14:51:51.311782006+08:00" level=warning msg="unsupported route" arch=amd64 command=create container=879d2f317bc7ba1e737f611f9f78647f2e9c539cf8347f6a8664a96587995236 destination="fe80::/64" name=kata-runtime pid=315246 source=virtcontainers subsystem=network unsupported-route-type=ipv6
time="2019-02-24T14:51:52.37129584+08:00" level=error msg="rpc error: code = Internal desc = Could not run process: container_linux.go:348: starting container process caused \"process_linux.go:279: applying cgroup configuration for process caused \\\"failed to write 10 to cpuset.cpus: write /sys/fs/cgroup/cpuset/system.slice:docker:879d2f317bc7ba1e737f611f9f78647f2e9c539cf8347f6a8664a96587995236/cpuset.cpus: invalid argument\\\"\"" arch=amd64 command=create container=879d2f317bc7ba1e737f611f9f78647f2e9c539cf8347f6a8664a96587995236 name=kata-runtime pid=315246 source=runtime
time="2019-02-24T14:51:52.425554899+08:00" level=error msg="Container ID (879d2f317bc7ba1e737f611f9f78647f2e9c539cf8347f6a8664a96587995236) does not exist" arch=amd64 command=delete container=879d2f317bc7ba1e737f611f9f78647f2e9c539cf8347f6a8664a96587995236 name=kata-runtime pid=315394 source=runtime
time="2019-02-24T14:51:52.48949183+08:00" level=error msg="Container ID (879d2f317bc7ba1e737f611f9f78647f2e9c539cf8347f6a8664a96587995236) does not exist" arch=amd64 command=delete container=879d2f317bc7ba1e737f611f9f78647f2e9c539cf8347f6a8664a96587995236 name=kata-runtime pid=315412 source=runtime
time="2019-02-25T16:40:57.668611876+08:00" level=warning msg="fetch sandbox device failed" arch=amd64 command=create container=7e3b21b05db1826728d27ddd7c349940fc19f7214ae544564fd1ee81f62f2b44 error="open /run/vc/sbs/7e3b21b05db1826728d27ddd7c349940fc19f7214ae544564fd1ee81f62f2b44/devices.json: no such file or directory" name=kata-runtime pid=59449 sandbox=7e3b21b05db1826728d27ddd7c349940fc19f7214ae544564fd1ee81f62f2b44 sandboxid=7e3b21b05db1826728d27ddd7c349940fc19f7214ae544564fd1ee81f62f2b44 source=virtcontainers subsystem=sandbox
time="2019-02-25T16:40:59.159336755+08:00" level=warning msg="unsupported address" address="fe80::42:acff:fe11:3/64" arch=amd64 command=create container=7e3b21b05db1826728d27ddd7c349940fc19f7214ae544564fd1ee81f62f2b44 name=kata-runtime pid=59449 source=virtcontainers subsystem=network unsupported-address-type=ipv6
time="2019-02-25T16:40:59.15947607+08:00" level=warning msg="unsupported route" arch=amd64 command=create container=7e3b21b05db1826728d27ddd7c349940fc19f7214ae544564fd1ee81f62f2b44 destination="fe80::/64" name=kata-runtime pid=59449 source=virtcontainers subsystem=network unsupported-route-type=ipv6
time="2019-02-25T17:07:40.662725107+08:00" level=error msg="Container ID (7e3b21b05db1826728d27ddd7c349940fc19f7214ae544564fd1ee81f62f2b44) does not exist" arch=amd64 command=delete container=7e3b21b05db1826728d27ddd7c349940fc19f7214ae544564fd1ee81f62f2b44 name=kata-runtime pid=106665 source=runtime
time="2019-02-25T17:08:16.807794037+08:00" level=warning msg="fetch sandbox device failed" arch=amd64 command=create container=9b1349890e6b633ff7ff0fb3eec6315f9dc9d7381ccac37e85f5369614d9810a error="open /run/vc/sbs/9b1349890e6b633ff7ff0fb3eec6315f9dc9d7381ccac37e85f5369614d9810a/devices.json: no such file or directory" name=kata-runtime pid=107784 sandbox=9b1349890e6b633ff7ff0fb3eec6315f9dc9d7381ccac37e85f5369614d9810a sandboxid=9b1349890e6b633ff7ff0fb3eec6315f9dc9d7381ccac37e85f5369614d9810a source=virtcontainers subsystem=sandbox
time="2019-02-25T17:08:17.715655836+08:00" level=warning msg="unsupported address" address="fe80::42:acff:fe11:2/64" arch=amd64 command=create container=9b1349890e6b633ff7ff0fb3eec6315f9dc9d7381ccac37e85f5369614d9810a name=kata-runtime pid=107784 source=virtcontainers subsystem=network unsupported-address-type=ipv6
time="2019-02-25T17:08:17.715815077+08:00" level=warning msg="unsupported route" arch=amd64 command=create container=9b1349890e6b633ff7ff0fb3eec6315f9dc9d7381ccac37e85f5369614d9810a destination="fe80::/64" name=kata-runtime pid=107784 source=virtcontainers subsystem=network unsupported-route-type=ipv6
time="2019-02-25T17:08:36.527803783+08:00" level=error msg="Container ID (9b1349890e6b633ff7ff0fb3eec6315f9dc9d7381ccac37e85f5369614d9810a) does not exist" arch=amd64 command=delete container=9b1349890e6b633ff7ff0fb3eec6315f9dc9d7381ccac37e85f5369614d9810a name=kata-runtime pid=108569 source=runtime
time="2019-02-25T17:08:48.032978547+08:00" level=warning msg="fetch sandbox device failed" arch=amd64 command=create container=28a5d8f3887d57375cee9ec42c53ba5f98f1b4424de94b1b0622d35ddb55bc4e error="open /run/vc/sbs/28a5d8f3887d57375cee9ec42c53ba5f98f1b4424de94b1b0622d35ddb55bc4e/devices.json: no such file or directory" name=kata-runtime pid=108927 sandbox=28a5d8f3887d57375cee9ec42c53ba5f98f1b4424de94b1b0622d35ddb55bc4e sandboxid=28a5d8f3887d57375cee9ec42c53ba5f98f1b4424de94b1b0622d35ddb55bc4e source=virtcontainers subsystem=sandbox
time="2019-02-25T17:08:48.949245612+08:00" level=warning msg="unsupported address" address="fe80::42:acff:fe11:2/64" arch=amd64 command=create container=28a5d8f3887d57375cee9ec42c53ba5f98f1b4424de94b1b0622d35ddb55bc4e name=kata-runtime pid=108927 source=virtcontainers subsystem=network unsupported-address-type=ipv6
time="2019-02-25T17:08:48.949400659+08:00" level=warning msg="unsupported route" arch=amd64 command=create container=28a5d8f3887d57375cee9ec42c53ba5f98f1b4424de94b1b0622d35ddb55bc4e destination="fe80::/64" name=kata-runtime pid=108927 source=virtcontainers subsystem=network unsupported-route-type=ipv6
time="2019-02-25T17:08:49.990045752+08:00" level=error msg="rpc error: code = Internal desc = Could not run process: container_linux.go:348: starting container process caused \"process_linux.go:279: applying cgroup configuration for process caused \\\"failed to write 9 to cpuset.cpus: write /sys/fs/cgroup/cpuset/system.slice:docker:28a5d8f3887d57375cee9ec42c53ba5f98f1b4424de94b1b0622d35ddb55bc4e/cpuset.cpus: invalid argument\\\"\"" arch=amd64 command=create container=28a5d8f3887d57375cee9ec42c53ba5f98f1b4424de94b1b0622d35ddb55bc4e name=kata-runtime pid=108927 source=runtime
time="2019-02-25T17:08:50.043497521+08:00" level=error msg="Container ID (28a5d8f3887d57375cee9ec42c53ba5f98f1b4424de94b1b0622d35ddb55bc4e) does not exist" arch=amd64 command=delete container=28a5d8f3887d57375cee9ec42c53ba5f98f1b4424de94b1b0622d35ddb55bc4e name=kata-runtime pid=109081 source=runtime
time="2019-02-25T17:08:50.108294146+08:00" level=error msg="Container ID (28a5d8f3887d57375cee9ec42c53ba5f98f1b4424de94b1b0622d35ddb55bc4e) does not exist" arch=amd64 command=delete container=28a5d8f3887d57375cee9ec42c53ba5f98f1b4424de94b1b0622d35ddb55bc4e name=kata-runtime pid=109098 source=runtime
time="2019-02-25T17:10:58.170182476+08:00" level=warning msg="fetch sandbox device failed" arch=amd64 command=create container=39dcbc28f5ec08034207f57050834d5909e0004f3392ce4570adbf4371471b13 error="open /run/vc/sbs/39dcbc28f5ec08034207f57050834d5909e0004f3392ce4570adbf4371471b13/devices.json: no such file or directory" name=kata-runtime pid=112805 sandbox=39dcbc28f5ec08034207f57050834d5909e0004f3392ce4570adbf4371471b13 sandboxid=39dcbc28f5ec08034207f57050834d5909e0004f3392ce4570adbf4371471b13 source=virtcontainers subsystem=sandbox
time="2019-02-25T17:10:59.053831318+08:00" level=warning msg="unsupported address" address="fe80::42:acff:fe11:2/64" arch=amd64 command=create container=39dcbc28f5ec08034207f57050834d5909e0004f3392ce4570adbf4371471b13 name=kata-runtime pid=112805 source=virtcontainers subsystem=network unsupported-address-type=ipv6
time="2019-02-25T17:10:59.053957419+08:00" level=warning msg="unsupported route" arch=amd64 command=create container=39dcbc28f5ec08034207f57050834d5909e0004f3392ce4570adbf4371471b13 destination="fe80::/64" name=kata-runtime pid=112805 source=virtcontainers subsystem=network unsupported-route-type=ipv6
time="2019-02-25T17:11:00.087316736+08:00" level=error msg="rpc error: code = Internal desc = Could not run process: container_linux.go:348: starting container process caused \"process_linux.go:279: applying cgroup configuration for process caused \\\"failed to write 9-16 to cpuset.cpus: write /sys/fs/cgroup/cpuset/system.slice:docker:39dcbc28f5ec08034207f57050834d5909e0004f3392ce4570adbf4371471b13/cpuset.cpus: invalid argument\\\"\"" arch=amd64 command=create container=39dcbc28f5ec08034207f57050834d5909e0004f3392ce4570adbf4371471b13 name=kata-runtime pid=112805 source=runtime
time="2019-02-25T17:11:00.145717785+08:00" level=error msg="Container ID (39dcbc28f5ec08034207f57050834d5909e0004f3392ce4570adbf4371471b13) does not exist" arch=amd64 command=delete container=39dcbc28f5ec08034207f57050834d5909e0004f3392ce4570adbf4371471b13 name=kata-runtime pid=112963 source=runtime
time="2019-02-25T17:11:00.201735715+08:00" level=error msg="Container ID (39dcbc28f5ec08034207f57050834d5909e0004f3392ce4570adbf4371471b13) does not exist" arch=amd64 command=delete container=39dcbc28f5ec08034207f57050834d5909e0004f3392ce4570adbf4371471b13 name=kata-runtime pid=112984 source=runtime
time="2019-02-25T17:11:13.871140381+08:00" level=warning msg="fetch sandbox device failed" arch=amd64 command=create container=92d5a7a9679d837bb031421f0a1c956e3abf95926877b32b359af17eca558434 error="open /run/vc/sbs/92d5a7a9679d837bb031421f0a1c956e3abf95926877b32b359af17eca558434/devices.json: no such file or directory" name=kata-runtime pid=113557 sandbox=92d5a7a9679d837bb031421f0a1c956e3abf95926877b32b359af17eca558434 sandboxid=92d5a7a9679d837bb031421f0a1c956e3abf95926877b32b359af17eca558434 source=virtcontainers subsystem=sandbox
time="2019-02-25T17:11:14.838357752+08:00" level=warning msg="unsupported address" address="fe80::42:acff:fe11:2/64" arch=amd64 command=create container=92d5a7a9679d837bb031421f0a1c956e3abf95926877b32b359af17eca558434 name=kata-runtime pid=113557 source=virtcontainers subsystem=network unsupported-address-type=ipv6
time="2019-02-25T17:11:14.83848142+08:00" level=warning msg="unsupported route" arch=amd64 command=create container=92d5a7a9679d837bb031421f0a1c956e3abf95926877b32b359af17eca558434 destination="fe80::/64" name=kata-runtime pid=113557 source=virtcontainers subsystem=network unsupported-route-type=ipv6

Proxy logs

Recent proxy problems found in system journal:

time="2019-02-21T15:36:43.952136793+08:00" level=fatal msg="channel error" error="accept unix /run/vc/sbs/51c23b618fc3dc40fc68f4eddaeee62646fe6a77a8dcf20340693613da571432/proxy.sock: use of closed network connection" name=kata-proxy pid=33848 sandbox=51c23b618fc3dc40fc68f4eddaeee62646fe6a77a8dcf20340693613da571432 source=proxy
time="2019-02-21T15:37:04.606206995+08:00" level=fatal msg="failed to handle exit signal" error="close unix @->/run/vc/vm/682371cdf166c1651190671824d787267589d388d8f6d9031d8ce1c73a125038/kata.sock: use of closed network connection" name=kata-proxy pid=34697 sandbox=682371cdf166c1651190671824d787267589d388d8f6d9031d8ce1c73a125038 source=proxy
time="2019-02-21T15:38:07.616620734+08:00" level=fatal msg="failed to handle exit signal" error="close unix @->/run/vc/vm/1fed6ca96cda6355a11b78a5201b4d5c97540468e7b7a822e27779805f717923/kata.sock: use of closed network connection" name=kata-proxy pid=40942 sandbox=1fed6ca96cda6355a11b78a5201b4d5c97540468e7b7a822e27779805f717923 source=proxy
time="2019-02-21T15:40:14.031074118+08:00" level=fatal msg="failed to handle exit signal" error="close unix @->/run/vc/vm/c8c0257d08bd213f4eff9f022d86bb3e48836f0b82044e7c497fe0f81ec86f4c/kata.sock: use of closed network connection" name=kata-proxy pid=46800 sandbox=c8c0257d08bd213f4eff9f022d86bb3e48836f0b82044e7c497fe0f81ec86f4c source=proxy
time="2019-02-21T15:42:33.743351362+08:00" level=fatal msg="failed to handle exit signal" error="close unix @->/run/vc/vm/6fec355e82eb1037e0728e7374b2a870d7180eea688462717456e36722619e5f/kata.sock: use of closed network connection" name=kata-proxy pid=59977 sandbox=6fec355e82eb1037e0728e7374b2a870d7180eea688462717456e36722619e5f source=proxy
time="2019-02-21T15:43:26.523144268+08:00" level=fatal msg="failed to handle exit signal" error="close unix @->/run/vc/vm/c42096b13fb4ba2dc5eb3ceda407b62880c8d86bbd87c76ce1e0c7cfa918e26e/kata.sock: use of closed network connection" name=kata-proxy pid=61917 sandbox=c42096b13fb4ba2dc5eb3ceda407b62880c8d86bbd87c76ce1e0c7cfa918e26e source=proxy
time="2019-02-21T15:43:54.126313144+08:00" level=fatal msg="failed to handle exit signal" error="close unix @->/run/vc/vm/583da0976bf5176aa004af2200735821ccc1de5ef19f3a51cf5a4c1fccc71830/kata.sock: use of closed network connection" name=kata-proxy pid=62977 sandbox=583da0976bf5176aa004af2200735821ccc1de5ef19f3a51cf5a4c1fccc71830 source=proxy
time="2019-02-21T15:44:18.327687345+08:00" level=fatal msg="channel error" error="accept unix /run/vc/sbs/12e29d0d5b19cf5ec6b0270f519a612dc768a6a64eb5ec598ba5115869f5ba16/proxy.sock: use of closed network connection" name=kata-proxy pid=64030 sandbox=12e29d0d5b19cf5ec6b0270f519a612dc768a6a64eb5ec598ba5115869f5ba16 source=proxy
time="2019-02-21T15:44:30.309332587+08:00" level=fatal msg="channel error" error="accept unix /run/vc/sbs/7b4441a3bea6fa5e990ba081f0822b0c7dc231ba57b78a223ec82c68310b4076/proxy.sock: use of closed network connection" name=kata-proxy pid=64935 sandbox=7b4441a3bea6fa5e990ba081f0822b0c7dc231ba57b78a223ec82c68310b4076 source=proxy
time="2019-02-21T15:47:08.218647098+08:00" level=fatal msg="failed to handle exit signal" error="close unix @->/run/vc/vm/78d18f7e658c45108c3a5cf214ae564b47fee3aa41082a1457f394086a613c3c/kata.sock: use of closed network connection" name=kata-proxy pid=65748 sandbox=78d18f7e658c45108c3a5cf214ae564b47fee3aa41082a1457f394086a613c3c source=proxy
time="2019-02-21T15:54:30.660040268+08:00" level=fatal msg="channel error" error="accept unix /run/vc/sbs/66ac416f69586d59cb279cdeb5ecaf042cc562f049091b7851d6cad33ef76568/proxy.sock: use of closed network connection" name=kata-proxy pid=72123 sandbox=66ac416f69586d59cb279cdeb5ecaf042cc562f049091b7851d6cad33ef76568 source=proxy
time="2019-02-21T16:05:54.429196086+08:00" level=fatal msg="failed to handle exit signal" error="close unix @->/run/vc/vm/539358e23eac42610e386bc06e680f53a3de30d10a69d97c16a2f0ae35cc3038/kata.sock: use of closed network connection" name=kata-proxy pid=83480 sandbox=539358e23eac42610e386bc06e680f53a3de30d10a69d97c16a2f0ae35cc3038 source=proxy
time="2019-02-21T16:08:57.764464768+08:00" level=fatal msg="failed to handle exit signal" error="close unix @->/run/vc/vm/7145edeb087ef13150cf1ef0ef77301a59164ffead9fc525c8e7c2ced156fe2f/kata.sock: use of closed network connection" name=kata-proxy pid=122916 sandbox=7145edeb087ef13150cf1ef0ef77301a59164ffead9fc525c8e7c2ced156fe2f source=proxy
time="2019-02-21T16:09:22.099961029+08:00" level=fatal msg="failed to handle exit signal" error="close unix @->/run/vc/vm/d811e8b92738a11cdcecc6f7d3974c559542c66658d23b30430427c0c355f023/kata.sock: use of closed network connection" name=kata-proxy pid=126007 sandbox=d811e8b92738a11cdcecc6f7d3974c559542c66658d23b30430427c0c355f023 source=proxy
time="2019-02-21T16:17:46.491382524+08:00" level=fatal msg="failed to handle exit signal" error="close unix @->/run/vc/vm/eeb3eee0904505331cb19885565182303a1cc27156d05767c561fafa779f1517/kata.sock: use of closed network connection" name=kata-proxy pid=140747 sandbox=eeb3eee0904505331cb19885565182303a1cc27156d05767c561fafa779f1517 source=proxy
time="2019-02-21T16:23:47.478932832+08:00" level=fatal msg="channel error" error="accept unix /run/vc/sbs/3b1fb52f8c21763d4afd95bae65e8d5b8147763d236d2f5c56af2de37c2d2573/proxy.sock: use of closed network connection" name=kata-proxy pid=141319 sandbox=3b1fb52f8c21763d4afd95bae65e8d5b8147763d236d2f5c56af2de37c2d2573 source=proxy
time="2019-02-24T13:57:46.067833353+08:00" level=fatal msg="channel error" error="accept unix /run/vc/sbs/74297df6dba3f21b10d3dac95eddff1a05e02d120bdca01df70b1db8655ea319/proxy.sock: use of closed network connection" name=kata-proxy pid=194276 sandbox=74297df6dba3f21b10d3dac95eddff1a05e02d120bdca01df70b1db8655ea319 source=proxy
time="2019-02-24T13:57:51.522066892+08:00" level=fatal msg="failed to handle exit signal" error="close unix @->/run/vc/vm/f74f70cde202296497677b85fee04531f33005a1191c3059481f054305c3fe2e/kata.sock: use of closed network connection" name=kata-proxy pid=214685 sandbox=f74f70cde202296497677b85fee04531f33005a1191c3059481f054305c3fe2e source=proxy
time="2019-02-24T13:58:49.757502913+08:00" level=fatal msg="channel error" error="accept unix /run/vc/sbs/160dd405b9d4795abe2e8537dbf26e16e881f39f5cacbc52ae6298042691bb13/proxy.sock: use of closed network connection" name=kata-proxy pid=216538 sandbox=160dd405b9d4795abe2e8537dbf26e16e881f39f5cacbc52ae6298042691bb13 source=proxy
time="2019-02-24T13:59:21.645394727+08:00" level=fatal msg="failed to handle exit signal" error="close unix @->/run/vc/vm/7f5352d91b7fc10bfc6cef5967cf0b3d1c163aeaaf6bd52b392ea8bd8024fe70/kata.sock: use of closed network connection" name=kata-proxy pid=217676 sandbox=7f5352d91b7fc10bfc6cef5967cf0b3d1c163aeaaf6bd52b392ea8bd8024fe70 source=proxy
time="2019-02-24T14:07:38.194876525+08:00" level=fatal msg="channel error" error="accept unix /run/vc/sbs/27027f7124eebb5115aac7cbca96644b4530275bc39b8e7fee9037a8ebd9a537/proxy.sock: use of closed network connection" name=kata-proxy pid=232660 sandbox=27027f7124eebb5115aac7cbca96644b4530275bc39b8e7fee9037a8ebd9a537 source=proxy
time="2019-02-24T14:14:27.470402013+08:00" level=fatal msg="failed to handle exit signal" error="close unix @->/run/vc/vm/4a89e36bb7d547372a0d890bbec9db43f1082f5eda192c0798ad98b3bff72f47/kata.sock: use of closed network connection" name=kata-proxy pid=243510 sandbox=4a89e36bb7d547372a0d890bbec9db43f1082f5eda192c0798ad98b3bff72f47 source=proxy
time="2019-02-24T14:17:32.744929497+08:00" level=fatal msg="channel error" error="accept unix /run/vc/sbs/2ada9cc0f64ce42b9e94163d313f0a74f89635d6ce2b413e2c86dcf73e04752b/proxy.sock: use of closed network connection" name=kata-proxy pid=246674 sandbox=2ada9cc0f64ce42b9e94163d313f0a74f89635d6ce2b413e2c86dcf73e04752b source=proxy
time="2019-02-24T14:19:26.332168548+08:00" level=fatal msg="channel error" error="accept unix /run/vc/sbs/798ecdce0dcb43116f7c5d849d190861130d57236f097452dfe2000ad0b8a752/proxy.sock: use of closed network connection" name=kata-proxy pid=250262 sandbox=798ecdce0dcb43116f7c5d849d190861130d57236f097452dfe2000ad0b8a752 source=proxy
time="2019-02-24T14:19:33.979570849+08:00" level=fatal msg="channel error" error="accept unix /run/vc/sbs/f2dcd6b11e067d32348c5a58104901728819273448b8252ec5c18d76cfec8d29/proxy.sock: use of closed network connection" name=kata-proxy pid=253786 sandbox=f2dcd6b11e067d32348c5a58104901728819273448b8252ec5c18d76cfec8d29 source=proxy
time="2019-02-24T14:20:18.900201758+08:00" level=fatal msg="channel error" error="accept unix /run/vc/sbs/819af73bfd5e6b6d940c61e3c948b34e69abea65235408146e756cbaaa51cd27/proxy.sock: use of closed network connection" name=kata-proxy pid=255408 sandbox=819af73bfd5e6b6d940c61e3c948b34e69abea65235408146e756cbaaa51cd27 source=proxy
time="2019-02-24T14:20:38.212678869+08:00" level=fatal msg="failed to handle exit signal" error="close unix @->/run/vc/vm/8976a53a8efc4e1df2e95a39b0781e23b04df00e767bab494d4fec80a7e79a1a/kata.sock: use of closed network connection" name=kata-proxy pid=256140 sandbox=8976a53a8efc4e1df2e95a39b0781e23b04df00e767bab494d4fec80a7e79a1a source=proxy
time="2019-02-24T14:20:54.942259967+08:00" level=fatal msg="failed to handle exit signal" error="close unix @->/run/vc/vm/4468f4f6d251b20563623aeca440cc0425d0f9381e36850c65302b7fec03e761/kata.sock: use of closed network connection" name=kata-proxy pid=256791 sandbox=4468f4f6d251b20563623aeca440cc0425d0f9381e36850c65302b7fec03e761 source=proxy
time="2019-02-24T14:27:41.484155521+08:00" level=fatal msg="failed to handle exit signal" error="close unix @->/run/vc/vm/0f74ce0f9a85f4368d25eb7d1dc022070f1bd40b196bd402daa67b5ba9bfcb43/kata.sock: use of closed network connection" name=kata-proxy pid=268852 sandbox=0f74ce0f9a85f4368d25eb7d1dc022070f1bd40b196bd402daa67b5ba9bfcb43 source=proxy
time="2019-02-24T14:28:08.246746923+08:00" level=fatal msg="failed to handle exit signal" error="close unix @->/run/vc/vm/e6ae35e7c9630c2379d416f819f6024a66400394af35c64db7842a1ae9458151/kata.sock: use of closed network connection" name=kata-proxy pid=269909 sandbox=e6ae35e7c9630c2379d416f819f6024a66400394af35c64db7842a1ae9458151 source=proxy
time="2019-02-24T14:28:42.88483776+08:00" level=fatal msg="failed to handle exit signal" error="close unix @->/run/vc/vm/d3f00c07cefcfd1f3c1b495887eba518edb62e6253e16e703a9a371ad2bd3d31/kata.sock: use of closed network connection" name=kata-proxy pid=271539 sandbox=d3f00c07cefcfd1f3c1b495887eba518edb62e6253e16e703a9a371ad2bd3d31 source=proxy
time="2019-02-24T14:28:58.958817307+08:00" level=fatal msg="channel error" error="accept unix /run/vc/sbs/3b7f852992773e0acfba6c3a5e214045be55a7cb842cdc4e7bd00ed470b0c005/proxy.sock: use of closed network connection" name=kata-proxy pid=272145 sandbox=3b7f852992773e0acfba6c3a5e214045be55a7cb842cdc4e7bd00ed470b0c005 source=proxy
time="2019-02-24T14:29:25.829351624+08:00" level=fatal msg="failed to handle exit signal" error="close unix @->/run/vc/vm/c94f6b101af99d2342938c240da06b522c1019fb08dd8f25681588309a8eb20d/kata.sock: use of closed network connection" name=kata-proxy pid=272811 sandbox=c94f6b101af99d2342938c240da06b522c1019fb08dd8f25681588309a8eb20d source=proxy
time="2019-02-24T14:30:02.906272023+08:00" level=fatal msg="failed to handle exit signal" error="close unix @->/run/vc/vm/47ec18998757c806948bcc9d5cbad33328afcb1b22d2975295692c8180dcb904/kata.sock: use of closed network connection" name=kata-proxy pid=274415 sandbox=47ec18998757c806948bcc9d5cbad33328afcb1b22d2975295692c8180dcb904 source=proxy
time="2019-02-24T14:33:40.126140727+08:00" level=fatal msg="failed to handle exit signal" error="close unix @->/run/vc/vm/289ecaf13321706c47070d5d9da5ac3ca2c53b2dfed4332a07595b3f67da006c/kata.sock: use of closed network connection" name=kata-proxy pid=280993 sandbox=289ecaf13321706c47070d5d9da5ac3ca2c53b2dfed4332a07595b3f67da006c source=proxy
time="2019-02-24T14:34:12.203428486+08:00" level=fatal msg="failed to handle exit signal" error="close unix @->/run/vc/vm/cc5cab1e38d1814f1ad0f43a69ae314361832e8637e1c53666d18b4720e5221b/kata.sock: use of closed network connection" name=kata-proxy pid=282614 sandbox=cc5cab1e38d1814f1ad0f43a69ae314361832e8637e1c53666d18b4720e5221b source=proxy
time="2019-02-24T14:36:03.635107439+08:00" level=fatal msg="failed to handle exit signal" error="close unix @->/run/vc/vm/6084eeb7a639cc505f6c1a8731d5d97ca98843b5d1c18303a9c76978b5180169/kata.sock: use of closed network connection" name=kata-proxy pid=283215 sandbox=6084eeb7a639cc505f6c1a8731d5d97ca98843b5d1c18303a9c76978b5180169 source=proxy
time="2019-02-24T14:36:08.801114018+08:00" level=fatal msg="failed to handle exit signal" error="close unix @->/run/vc/vm/d5b86488c6dcdb6d8c5085a9f2bf5d51712f25833b181873eeff64380c9f8649/kata.sock: use of closed network connection" name=kata-proxy pid=286560 sandbox=d5b86488c6dcdb6d8c5085a9f2bf5d51712f25833b181873eeff64380c9f8649 source=proxy
time="2019-02-24T14:37:04.788177619+08:00" level=fatal msg="failed to handle exit signal" error="close unix @->/run/vc/vm/5bb245d275541f638e5b6ea6c18f1a1b99b8e22175a058537df739457bfef24b/kata.sock: use of closed network connection" name=kata-proxy pid=287845 sandbox=5bb245d275541f638e5b6ea6c18f1a1b99b8e22175a058537df739457bfef24b source=proxy
time="2019-02-24T14:41:44.249899824+08:00" level=fatal msg="failed to handle exit signal" error="close unix @->/run/vc/vm/a77721a81167d2042dd7a97cedaf060d2d93c20dff75fc19b88cb6436d689858/kata.sock: use of closed network connection" name=kata-proxy pid=296236 sandbox=a77721a81167d2042dd7a97cedaf060d2d93c20dff75fc19b88cb6436d689858 source=proxy
time="2019-02-24T14:41:56.851218502+08:00" level=fatal msg="channel error" error="accept unix /run/vc/sbs/f13148c56c7a6d399cf8e7286f40033a46d05edc6b3924080996dcaaf06dbe5d/proxy.sock: use of closed network connection" name=kata-proxy pid=297174 sandbox=f13148c56c7a6d399cf8e7286f40033a46d05edc6b3924080996dcaaf06dbe5d source=proxy
time="2019-02-24T14:44:32.92030984+08:00" level=fatal msg="failed to handle exit signal" error="close unix @->/run/vc/vm/8213cb0d7710154d5298732314501c1101bd0bed35b9037ef05efc41824b3a35/kata.sock: use of closed network connection" name=kata-proxy pid=301567 sandbox=8213cb0d7710154d5298732314501c1101bd0bed35b9037ef05efc41824b3a35 source=proxy
time="2019-02-24T14:44:53.079654012+08:00" level=fatal msg="failed to handle exit signal" error="close unix @->/run/vc/vm/bb06265a82e51c11d26fc3000dd11f02f05be58d3420b34c27b2b55b36d75659/kata.sock: use of closed network connection" name=kata-proxy pid=302704 sandbox=bb06265a82e51c11d26fc3000dd11f02f05be58d3420b34c27b2b55b36d75659 source=proxy
time="2019-02-24T14:49:38.313222812+08:00" level=fatal msg="channel error" error="accept unix /run/vc/sbs/0748d542839b6c6c364fcffdd17cf51455e9faad71b4573c4a5ec35a2e749e47/proxy.sock: use of closed network connection" name=kata-proxy pid=311109 sandbox=0748d542839b6c6c364fcffdd17cf51455e9faad71b4573c4a5ec35a2e749e47 source=proxy
time="2019-02-24T14:51:32.192144491+08:00" level=fatal msg="failed to handle exit signal" error="close unix @->/run/vc/vm/fa33f6b0f2c8b71b830684002744e0770df2a2da9a41b921f578b4d015cc030a/kata.sock: use of closed network connection" name=kata-proxy pid=314560 sandbox=fa33f6b0f2c8b71b830684002744e0770df2a2da9a41b921f578b4d015cc030a source=proxy
time="2019-02-24T14:51:52.291264712+08:00" level=fatal msg="channel error" error="accept unix /run/vc/sbs/879d2f317bc7ba1e737f611f9f78647f2e9c539cf8347f6a8664a96587995236/proxy.sock: use of closed network connection" name=kata-proxy pid=315305 sandbox=879d2f317bc7ba1e737f611f9f78647f2e9c539cf8347f6a8664a96587995236 source=proxy
time="2019-02-25T17:07:40.484997029+08:00" level=fatal msg="failed to handle exit signal" error="close unix @->/run/vc/vm/7e3b21b05db1826728d27ddd7c349940fc19f7214ae544564fd1ee81f62f2b44/kata.sock: use of closed network connection" name=kata-proxy pid=59521 sandbox=7e3b21b05db1826728d27ddd7c349940fc19f7214ae544564fd1ee81f62f2b44 source=proxy
time="2019-02-25T17:08:36.357814135+08:00" level=fatal msg="failed to handle exit signal" error="close unix @->/run/vc/vm/9b1349890e6b633ff7ff0fb3eec6315f9dc9d7381ccac37e85f5369614d9810a/kata.sock: use of closed network connection" name=kata-proxy pid=107851 sandbox=9b1349890e6b633ff7ff0fb3eec6315f9dc9d7381ccac37e85f5369614d9810a source=proxy
time="2019-02-25T17:08:49.910940014+08:00" level=fatal msg="failed to handle exit signal" error="close unix @->/run/vc/vm/28a5d8f3887d57375cee9ec42c53ba5f98f1b4424de94b1b0622d35ddb55bc4e/kata.sock: use of closed network connection" name=kata-proxy pid=108992 sandbox=28a5d8f3887d57375cee9ec42c53ba5f98f1b4424de94b1b0622d35ddb55bc4e source=proxy
time="2019-02-25T17:11:00.000460752+08:00" level=fatal msg="channel error" error="accept unix /run/vc/sbs/39dcbc28f5ec08034207f57050834d5909e0004f3392ce4570adbf4371471b13/proxy.sock: use of closed network connection" name=kata-proxy pid=112883 sandbox=39dcbc28f5ec08034207f57050834d5909e0004f3392ce4570adbf4371471b13 source=proxy

Shim logs

No recent shim problems found in system journal.

Throttler logs

No recent throttler problems found in system journal.

Container manager details

Have docker

Docker

Output of "docker version":

Client:
 Version:         1.13.1
 API version:     1.26
 Package version: docker-1.13.1-63.mt20190220.gitd365784.el7.centos.x86_64
 Go version:      go1.7.5
 Git commit:      d365784/1.13.1
 Built:           Wed Feb 20 03:10:55 2019
 OS/Arch:         linux/amd64

Server:
 Version:         1.13.1
 API version:     1.26 (minimum version 1.12)
 Package version: docker-1.13.1-63.mt20190220.gitd365784.el7.centos.x86_64
 Go version:      go1.7.5
 Git commit:      d365784/1.13.1
 Built:           Wed Feb 20 03:10:55 2019
 OS/Arch:         linux/amd64
 Experimental:    false

Output of "docker info":

Containers: 1
 Running: 1
 Paused: 0
 Stopped: 0
Images: 180
Server Version: 1.13.1
Storage Driver: devicemapper
 Pool Name: dockervg-thinpool
 Pool Blocksize: 524.3 kB
 Base Device Size: 10.74 GB
 Backing Filesystem: ext4
 Data file:
 Metadata file:
 Data Space Used: 12.38 GB
 Data Space Total: 973.7 GB
 Data Space Available: 961.4 GB
 Metadata Space Used: 9.806 MB
 Metadata Space Total: 10.25 GB
 Metadata Space Available: 10.24 GB
 Thin Pool Minimum Free Space: 97.37 GB
 Udev Sync Supported: true
 Deferred Removal Enabled: true
 Deferred Deletion Enabled: true
 Deferred Deleted Device Count: 0
 Library Version: 1.02.140-RHEL7 (2017-05-03)
Logging Driver: json-file
Cgroup Driver: systemd
Plugins:
 Volume: local
 Network: bridge host macvlan null overlay
Swarm: inactive
Runtimes: kata-runtime runc
Default Runtime: runc
Init Binary: /usr/libexec/docker/docker-init-current
containerd version: d365784586ef8dfc171f524aa329e928dc355c76 (expected: aa8187dbd3b7ad67d8e5e3a15115d3eef43a7ed1)
runc version: e9ab997cfc4151ebcca8e54cf047311121fd4849 (expected: 9df8b306d01f59d3a8029be411de015b7304dd8f)
init version: 5b117de7f824f3d3825737cf09581645abbe35d4 (expected: 949e6facb77383876aeff8a6944dde66b3089574)
Security Options:
 seccomp
  WARNING: You're not using the default seccomp profile
  Profile: /etc/docker/seccomp.json
Kernel Version: 3.10.0-862.mt20181225.115.el7.x86_64
Operating System: CentOS Linux 7 (Core)
OSType: linux
Architecture: x86_64
Number of Docker Hooks: 3
CPUs: 40
Total Memory: 377.3 GiB
Name: gh-hulk-k8s-ep14.gh.sankuai.com
ID: 4GNB:L2NJ:RTSE:2HCQ:IHIN:QXND:HANJ:77JA:BS7K:NFHY:OSKA:TRIF
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
Experimental: false
Insecure Registries:
 127.0.0.0/8
Registry Mirrors:
 http://127.0.0.1:65001
 registry-hulk.sankuai.com
Live Restore Enabled: true
Registries: docker.io (secure)

Output of "systemctl show docker":

Type=notify
Restart=on-abnormal
NotifyAccess=all
RestartUSec=100ms
TimeoutStartUSec=0
TimeoutStopUSec=1min 30s
WatchdogUSec=0
WatchdogTimestamp=Sun 2019-02-24 14:13:34 CST
WatchdogTimestampMonotonic=273445442614
StartLimitInterval=10000000
StartLimitBurst=5
StartLimitAction=none
FailureAction=none
PermissionsStartOnly=no
RootDirectoryStartOnly=no
RemainAfterExit=no
GuessMainPID=yes
MainPID=242136
ControlPID=0
FileDescriptorStoreMax=0
StatusErrno=0
Result=success
ExecMainStartTimestamp=Sun 2019-02-24 14:13:33 CST
ExecMainStartTimestampMonotonic=273444041707
ExecMainExitTimestampMonotonic=0
ExecMainPID=242136
ExecMainCode=0
ExecMainStatus=0
ExecStart={ path=/usr/bin/dockerd-current ; argv[]=/usr/bin/dockerd-current --exec-opt native.cgroupdriver=systemd --userland-proxy-path=/usr/libexec/docker/docker-proxy-current --init-path=/usr/libexec/docker/docker-init-current --seccomp-profile=/etc/docker/seccomp.json $OPTIONS $DOCKER_STORAGE_OPTIONS $DOCKER_NETWORK_OPTIONS $ADD_REGISTRY $BLOCK_REGISTRY $INSECURE_REGISTRY $REGISTRIES ; ignore_errors=no ; start_time=[Sun 2019-02-24 14:13:33 CST] ; stop_time=[n/a] ; pid=242136 ; code=(null) ; status=0/0 }
ExecReload={ path=/bin/kill ; argv[]=/bin/kill -s HUP $MAINPID ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }
Slice=system.slice
ControlGroup=/system.slice/docker.service
MemoryCurrent=949891072
TasksCurrent=153
Delegate=no
CPUAccounting=no
CPUShares=18446744073709551615
StartupCPUShares=18446744073709551615
CPUQuotaPerSecUSec=infinity
BlockIOAccounting=no
BlockIOWeight=18446744073709551615
StartupBlockIOWeight=18446744073709551615
MemoryAccounting=no
MemoryLimit=18446744073709551615
DevicePolicy=auto
TasksAccounting=no
TasksMax=18446744073709551615
Environment=GOTRACEBACK=crash DOCKER_HTTP_HOST_COMPAT=1 PATH=/usr/libexec/docker:/usr/bin:/usr/sbin
EnvironmentFile=/run/containers/registries.conf (ignore_errors=yes)
EnvironmentFile=/etc/sysconfig/docker (ignore_errors=yes)
EnvironmentFile=/etc/sysconfig/docker-storage (ignore_errors=yes)
EnvironmentFile=/etc/sysconfig/docker-network (ignore_errors=yes)
UMask=0022
LimitCPU=18446744073709551615
LimitFSIZE=18446744073709551615
LimitDATA=18446744073709551615
LimitSTACK=18446744073709551615
LimitCORE=18446744073709551615
LimitRSS=18446744073709551615
LimitNOFILE=1048576
LimitAS=18446744073709551615
LimitNPROC=1048576
LimitMEMLOCK=65536
LimitLOCKS=18446744073709551615
LimitSIGPENDING=1545530
LimitMSGQUEUE=819200
LimitNICE=0
LimitRTPRIO=0
LimitRTTIME=18446744073709551615
OOMScoreAdjust=0
Nice=0
IOScheduling=0
CPUSchedulingPolicy=0
CPUSchedulingPriority=0
TimerSlackNSec=50000
CPUSchedulingResetOnFork=no
NonBlocking=no
StandardInput=null
StandardOutput=journal
StandardError=inherit
TTYReset=no
TTYVHangup=no
TTYVTDisallocate=no
SyslogPriority=30
SyslogLevelPrefix=yes
SecureBits=0
CapabilityBoundingSet=18446744073709551615
AmbientCapabilities=0
MountFlags=0
PrivateTmp=no
PrivateNetwork=no
PrivateDevices=no
ProtectHome=no
ProtectSystem=no
SameProcessGroup=no
IgnoreSIGPIPE=yes
NoNewPrivileges=no
SystemCallErrorNumber=0
RuntimeDirectoryMode=0755
KillMode=process
KillSignal=15
SendSIGKILL=yes
SendSIGHUP=no
Id=docker.service
Names=docker.service
Requires=basic.target docker-cleanup.timer
Wants=system.slice docker-storage-setup.service
RequiredBy=docker-novolume-plugin.service docker-cleanup.service
Conflicts=shutdown.target
Before=shutdown.target
After=basic.target network.target system.slice hulk-host-init.service docker-novolume-plugin.service registries.service systemd-journald.socket docker-storage-setup.service rhel-push-plugin.socket
Documentation=http://docs.docker.com
Description=Docker Application Container Engine
LoadState=loaded
ActiveState=active
SubState=running
FragmentPath=/usr/lib/systemd/system/docker.service
UnitFileState=disabled
UnitFilePreset=disabled
InactiveExitTimestamp=Sun 2019-02-24 14:13:33 CST
InactiveExitTimestampMonotonic=273444041740
ActiveEnterTimestamp=Sun 2019-02-24 14:13:34 CST
ActiveEnterTimestampMonotonic=273445442662
ActiveExitTimestamp=Sun 2019-02-24 14:13:24 CST
ActiveExitTimestampMonotonic=273434675279
InactiveEnterTimestamp=Sun 2019-02-24 14:13:25 CST
InactiveEnterTimestampMonotonic=273435727751
CanStart=yes
CanStop=yes
CanReload=yes
CanIsolate=no
StopWhenUnneeded=no
RefuseManualStart=no
RefuseManualStop=no
AllowIsolate=no
DefaultDependencies=yes
OnFailureJobMode=replace
IgnoreOnIsolate=no
IgnoreOnSnapshot=no
NeedDaemonReload=no
JobTimeoutUSec=0
JobTimeoutAction=none
ConditionResult=yes
AssertResult=yes
ConditionTimestamp=Sun 2019-02-24 14:13:33 CST
ConditionTimestampMonotonic=273444040829
AssertTimestamp=Sun 2019-02-24 14:13:33 CST
AssertTimestampMonotonic=273444040829
Transient=no

No kubectl

Packages

No dpkg Have rpm Output of "rpm -qa|egrep "(cc-oci-runtimecc-runtimerunv|kata-proxy|kata-runtime|kata-shim|kata-ksm-throttler|kata-containers-image|linux-container|qemu-)"":

kata-ksm-throttler-1.5.0.git+3dd4c9f-18.1.x86_64
kata-linux-container-debug-4.14.67.22-18.1.x86_64
qemu-vanilla-2.11.2+git.0982a56a55-19.1.x86_64
kata-linux-container-4.14.67.22-18.1.x86_64
qemu-lite-bin-2.11.0+git.87517afd72-19.1.x86_64
kata-containers-image-1.5.0-15.1.x86_64
qemu-vanilla-data-2.11.2+git.0982a56a55-19.1.x86_64
qemu-lite-2.11.0+git.87517afd72-19.1.x86_64
kata-proxy-bin-1.5.0+git.9e77a0b-16.1.x86_64
kata-runtime-1.5.0+git.5f7fcd7-24.1.x86_64
qemu-vanilla-bin-2.11.2+git.0982a56a55-19.1.x86_64
kata-proxy-1.5.0+git.9e77a0b-16.1.x86_64
kata-shim-bin-1.5.0+git.efbf3bb-14.1.x86_64
qemu-lite-data-2.11.0+git.87517afd72-19.1.x86_64
kata-shim-1.5.0+git.efbf3bb-14.1.x86_64
grahamwhaley commented 5 years ago

thanks for reporting @free2k - /cc @devimc @jcvenegas for input.

devimc commented 5 years ago

@free2k you need to update the kata-agent or wait for the next release (1.6.0)

jcvenegas commented 5 years ago

@free2k yes as @devimc metion , next release will fix, this in two sides you are going to get host cpuset isolation and you wont have this issue from guest side.

free2k commented 5 years ago

Thank you for your answer, then I can only wait for the next version. What is the next release? Is there a certain date?

caoruidong commented 5 years ago

We already have 1.6.0 rc1. The formal release will come soon

devimc commented 5 years ago

@free2k please try with 1.5.1

devimc commented 5 years ago

@free2k can we close this?