search

kata-containers / runtime

Kata Containers version 1.x runtime (for version 2.x see https://github.com/kata-containers/kata-containers).
https://katacontainers.io/
Apache License 2.0
2.1k stars 375 forks source link

--cpuset-cpus invalid argument #1093

Closed free2k closed 3 years ago

free2k commented 5 years ago

Description of problem

  1. docker run --runtime kata-runtime --cpuset-cpus="0" -it busybox /bin/sh If bound to 0 core, it can run normally without any error.

2.docker run --runtime kata-runtime --cpuset-cpus="1" -it busybox /bin/sh If you bind to a non-zero core, then he will report an error.

/usr/bin/docker-current: Error response from daemon: oci runtime error: rpc error: code = Internal desc = Could not run process: container_linux.go:348: starting container process caused "process_linux.go:279: applying cgroup configuration for process caused \"failed to write 1 to cpuset.cpus: write /sys/fs/cgroup/cpuset/system.slice:docker:f3ee1e8406433c46bae3b450c63e5117e2a72ccecc57a1776781031a24eaf916/cpuset.cpus: invalid argument\"".

Expected result

No matter which core is bound to it, it should run correctly.

Actual result

Binding error

`# Meta details

Running kata-collect-data.sh version 1.3.0 (commit a786643) at 2019-01-07.14:44:21.206714732+0800.

Runtime is /bin/kata-runtime.

kata-env

Output of "/bin/kata-runtime kata-env":

[Meta]
  Version = "1.0.18"

[Runtime]
  Debug = false
  Path = "/usr/bin/kata-runtime"
  [Runtime.Version]
    Semver = "1.3.0"
    Commit = "a786643"
    OCI = "1.0.1"
  [Runtime.Config]
    Path = "/usr/share/defaults/kata-containers/configuration.toml"

[Hypervisor]
  MachineType = "pc"
  Version = "QEMU emulator version 2.11.0\nCopyright (c) 2003-2017 Fabrice Bellard and the QEMU Project developers"
  Path = "/usr/bin/qemu-lite-system-x86_64"
  BlockDeviceDriver = "virtio-scsi"
  EntropySource = "/dev/urandom"
  Msize9p = 8192
  MemorySlots = 10
  Debug = false
  UseVSock = false

[Image]
  Path = "/usr/share/kata-containers/kata-containers-image_clearlinux_1.3.0_agent_042c3ebd71c.img"

[Kernel]
  Path = "/usr/share/kata-containers/vmlinuz-4.14.67.12-10.1.container"
  Parameters = ""

[Initrd]
  Path = ""

[Proxy]
  Type = "kataProxy"
  Version = "kata-proxy version 1.3.0-6ddb006"
  Path = "/usr/libexec/kata-containers/kata-proxy"
  Debug = false

[Shim]
  Type = "kataShim"
  Version = "kata-shim version 1.3.0-5fbf1f0"
  Path = "/usr/libexec/kata-containers/kata-shim"
  Debug = false

[Agent]
  Type = "kata"

[Host]
  Kernel = "3.10.0-693.mt20180601.62.el7.x86_64"
  Architecture = "amd64"
  VMContainerCapable = true
  SupportVSocks = false
  [Host.Distro]
    Name = "CentOS Linux"
    Version = "7"
  [Host.CPU]
    Vendor = "GenuineIntel"
    Model = "Intel(R) Xeon(R) CPU E5-2630 v4 @ 2.20GHz"

[Netmon]
  Version = "kata-netmon version 1.3.0"
  Path = "/usr/libexec/kata-containers/kata-netmon"
  Debug = false
  Enable = false

Runtime config files

Runtime default config files

/etc/kata-containers/configuration.toml
/usr/share/defaults/kata-containers/configuration.toml

Runtime config file contents

Config file /etc/kata-containers/configuration.toml not found Output of "cat "/usr/share/defaults/kata-containers/configuration.toml"":

# Copyright (c) 2017-2018 Intel Corporation
#
# SPDX-License-Identifier: Apache-2.0
#

# XXX: WARNING: this file is auto-generated.
# XXX:
# XXX: Source file: "cli/config/configuration.toml.in"
# XXX: Project:
# XXX:   Name: Kata Containers
# XXX:   Type: kata

[hypervisor.qemu]
path = "/usr/bin/qemu-lite-system-x86_64"
kernel = "/usr/share/kata-containers/vmlinuz.container"
image = "/usr/share/kata-containers/kata-containers.img"
machine_type = "pc"

# Optional space-separated list of options to pass to the guest kernel.
# For example, use `kernel_params = "vsyscall=emulate"` if you are having
# trouble running pre-2.15 glibc.
#
# WARNING: - any parameter specified here will take priority over the default
# parameter value of the same name used to start the virtual machine.
# Do not set values here unless you understand the impact of doing so as you
# may stop the virtual machine from booting.
# To see the list of default parameters, enable hypervisor debug, create a
# container and look for 'default-kernel-parameters' log entries.
kernel_params = ""

# Path to the firmware.
# If you want that qemu uses the default firmware leave this option empty
firmware = ""

# Machine accelerators
# comma-separated list of machine accelerators to pass to the hypervisor.
# For example, `machine_accelerators = "nosmm,nosmbus,nosata,nopit,static-prt,nofw"`
machine_accelerators=""

# Default number of vCPUs per SB/VM:
# unspecified or 0                --> will be set to 1
# < 0                             --> will be set to the actual number of physical cores
# > 0 <= number of physical cores --> will be set to the specified number
# > number of physical cores      --> will be set to the actual number of physical cores
default_vcpus = 1

# Default maximum number of vCPUs per SB/VM:
# unspecified or == 0             --> will be set to the actual number of physical cores or to the maximum number
#                                     of vCPUs supported by KVM if that number is exceeded
# > 0 <= number of physical cores --> will be set to the specified number
# > number of physical cores      --> will be set to the actual number of physical cores or to the maximum number
#                                     of vCPUs supported by KVM if that number is exceeded
# WARNING: Depending of the architecture, the maximum number of vCPUs supported by KVM is used when
# the actual number of physical cores is greater than it.
# WARNING: Be aware that this value impacts the virtual machine's memory footprint and CPU
# the hotplug functionality. For example, `default_maxvcpus = 240` specifies that until 240 vCPUs
# can be added to a SB/VM, but the memory footprint will be big. Another example, with
# `default_maxvcpus = 8` the memory footprint will be small, but 8 will be the maximum number of
# vCPUs supported by the SB/VM. In general, we recommend that you do not edit this variable,
# unless you know what are you doing.
default_maxvcpus = 0

# Bridges can be used to hot plug devices.
# Limitations:
# * Currently only pci bridges are supported
# * Until 30 devices per bridge can be hot plugged.
# * Until 5 PCI bridges can be cold plugged per VM.
#   This limitation could be a bug in qemu or in the kernel
# Default number of bridges per SB/VM:
# unspecified or 0   --> will be set to 1
# > 1 <= 5           --> will be set to the specified number
# > 5                --> will be set to 5
default_bridges = 1

# Default memory size in MiB for SB/VM.
# If unspecified then it will be set 2048 MiB.
#default_memory = 2048
#
# Default memory slots per SB/VM.
# If unspecified then it will be set 10.
# This is will determine the times that memory will be hotadded to sandbox/VM.
#memory_slots = 10

# Disable block device from being used for a container's rootfs.
# In case of a storage driver like devicemapper where a container's
# root file system is backed by a block device, the block device is passed
# directly to the hypervisor for performance reasons.
# This flag prevents the block device from being passed to the hypervisor,
# 9pfs is used instead to pass the rootfs.
disable_block_device_use = false

# Block storage driver to be used for the hypervisor in case the container
# rootfs is backed by a block device. This is either virtio-scsi or
# virtio-blk.
block_device_driver = "virtio-scsi"

# Enable iothreads (data-plane) to be used. This causes IO to be
# handled in a separate IO thread. This is currently only implemented
# for SCSI.
#
enable_iothreads = false

# Enable pre allocation of VM RAM, default false
# Enabling this will result in lower container density
# as all of the memory will be allocated and locked
# This is useful when you want to reserve all the memory
# upfront or in the cases where you want memory latencies
# to be very predictable
# Default false
#enable_mem_prealloc = true

# Enable huge pages for VM RAM, default false
# Enabling this will result in the VM memory
# being allocated using huge pages.
# This is useful when you want to use vhost-user network
# stacks within the container. This will automatically
# result in memory pre allocation
#enable_hugepages = true

# Enable swap of vm memory. Default false.
# The behaviour is undefined if mem_prealloc is also set to true
#enable_swap = true

# This option changes the default hypervisor and kernel parameters
# to enable debug output where available. This extra output is added
# to the proxy logs, but only when proxy debug is also enabled.
#
# Default false
#enable_debug = true

# Disable the customizations done in the runtime when it detects
# that it is running on top a VMM. This will result in the runtime
# behaving as it would when running on bare metal.
#
#disable_nesting_checks = true

# This is the msize used for 9p shares. It is the number of bytes
# used for 9p packet payload.
#msize_9p = 8192

# If true and vsocks are supported, use vsocks to communicate directly
# with the agent and no proxy is started, otherwise use unix
# sockets and start a proxy to communicate with the agent.
# Default false
#use_vsock = true

# VFIO devices are hotplugged on a bridge by default.
# Enable hotplugging on root bus. This may be required for devices with
# a large PCI bar, as this is a current limitation with hotplugging on
# a bridge. This value is valid for "pc" machine type.
# Default false
#hotplug_vfio_on_root_bus = true

# If host doesn't support vhost_net, set to true. Thus we won't create vhost fds for nics.
# Default false
#disable_vhost_net = true
#
# Default entropy source.
# The path to a host source of entropy (including a real hardware RNG)
# /dev/urandom and /dev/random are two main options.
# Be aware that /dev/random is a blocking source of entropy.  If the host
# runs out of entropy, the VMs boot time will increase leading to get startup
# timeouts.
# The source of entropy /dev/urandom is non-blocking and provides a
# generally acceptable source of entropy. It should work well for pretty much
# all practical purposes.
#entropy_source= "/dev/urandom"

[factory]
# VM templating support. Once enabled, new VMs are created from template
# using vm cloning. They will share the same initial kernel, initramfs and
# agent memory by mapping it readonly. It helps speeding up new container
# creation and saves a lot of memory if there are many kata containers running
# on the same host.
#
# When disabled, new VMs are created from scratch.
#
# Default false
#enable_template = true

[proxy.kata]
path = "/usr/libexec/kata-containers/kata-proxy"

# If enabled, proxy messages will be sent to the system log
# (default: disabled)
#enable_debug = true

[shim.kata]
path = "/usr/libexec/kata-containers/kata-shim"

# If enabled, shim messages will be sent to the system log
# (default: disabled)
#enable_debug = true

[agent.kata]
# There is no field for this section. The goal is only to be able to
# specify which type of agent the user wants to use.

[netmon]
# If enabled, the network monitoring process gets started when the
# sandbox is created. This allows for the detection of some additional
# network being added to the existing network namespace, after the
# sandbox has been created.
# (default: disabled)
#enable_netmon = true

# Specify the path to the netmon binary.
path = "/usr/libexec/kata-containers/kata-netmon"

# If enabled, netmon messages will be sent to the system log
# (default: disabled)
#enable_debug = true

[runtime]
# If enabled, the runtime will log additional debug messages to the
# system log
# (default: disabled)
#enable_debug = true
#
# Internetworking model
# Determines how the VM should be connected to the
# the container network interface
# Options:
#
#   - bridged
#     Uses a linux bridge to interconnect the container interface to
#     the VM. Works for most cases except macvlan and ipvlan.
#
#   - macvtap
#     Used when the Container network interface can be bridged using
#     macvtap.
internetworking_model="macvtap"

# If enabled, the runtime will create opentracing.io traces and spans.
# (See https://www.jaegertracing.io/docs/getting-started).
# (default: disabled)
#enable_tracing = true

KSM throttler

version

Output of "--version":

/usr/bin/kata-collect-data.sh: line 168: --version: command not found

systemd service

Image details

---
osbuilder:
  url: "https://github.com/kata-containers/osbuilder"
  version: "unknown"
rootfs-creation-time: "2018-09-27T09:46:58.640358658+0000Z"
description: "osbuilder rootfs"
file-format-version: "0.0.2"
architecture: "x86_64"
base-distro:
  name: "Clear"
  version: "25180"
  packages:
    default:
      - "iptables-bin"
      - "libudev0-shim"
      - "systemd"
    extra:

agent:
  url: "https://github.com/kata-containers/agent"
  name: "kata-agent"
  version: "1.3.0-042c3ebd71c2ca425aa0c70fadcbf6370116659b"
  agent-is-init-daemon: "no"

Initrd details

No initrd

Logfiles

Runtime logs

Recent runtime problems found in system journal:

time="2019-01-07T11:35:06.425607102+08:00" level=warning msg="fetch sandbox device failed" arch=amd64 command=create container=da5664fbb0e8085db3e2b1b280164a60d475bc8f468ab54179a84266024bff77 error="open /run/vc/sbs/da5664fbb0e8085db3e2b1b280164a60d475bc8f468ab54179a84266024bff77/devices.json: no such file or directory" name=kata-runtime pid=22639 sandbox=da5664fbb0e8085db3e2b1b280164a60d475bc8f468ab54179a84266024bff77 sandboxid=da5664fbb0e8085db3e2b1b280164a60d475bc8f468ab54179a84266024bff77 source=virtcontainers subsystem=sandbox
time="2019-01-07T11:35:07.690450807+08:00" level=warning msg="unsupported address" address="fe80::42:acff:fe11:3/64" arch=amd64 command=create container=da5664fbb0e8085db3e2b1b280164a60d475bc8f468ab54179a84266024bff77 name=kata-runtime pid=22639 source=virtcontainers subsystem=network unsupported-address-type=ipv6
time="2019-01-07T11:35:07.690573348+08:00" level=warning msg="unsupported route" arch=amd64 command=create container=da5664fbb0e8085db3e2b1b280164a60d475bc8f468ab54179a84266024bff77 destination="fe80::/64" name=kata-runtime pid=22639 source=virtcontainers subsystem=network unsupported-route-type=ipv6
time="2019-01-07T14:20:42.410219121+08:00" level=error msg="Container not ready, running or paused, impossible to signal the container" arch=amd64 command=kill container=da5664fbb0e8085db3e2b1b280164a60d475bc8f468ab54179a84266024bff77 name=kata-runtime pid=322968 sandbox=da5664fbb0e8085db3e2b1b280164a60d475bc8f468ab54179a84266024bff77 source=runtime
time="2019-01-07T14:20:42.506162176+08:00" level=error msg="Container ID (da5664fbb0e8085db3e2b1b280164a60d475bc8f468ab54179a84266024bff77) does not exist" arch=amd64 command=delete container=da5664fbb0e8085db3e2b1b280164a60d475bc8f468ab54179a84266024bff77 name=kata-runtime pid=323007 source=runtime
time="2019-01-07T14:21:11.52508922+08:00" level=warning msg="fetch sandbox device failed" arch=amd64 command=create container=ccf16de541600f6fd40663d940149f2e522683ad7714fb15c110ed2032c36447 error="open /run/vc/sbs/ccf16de541600f6fd40663d940149f2e522683ad7714fb15c110ed2032c36447/devices.json: no such file or directory" name=kata-runtime pid=324059 sandbox=ccf16de541600f6fd40663d940149f2e522683ad7714fb15c110ed2032c36447 sandboxid=ccf16de541600f6fd40663d940149f2e522683ad7714fb15c110ed2032c36447 source=virtcontainers subsystem=sandbox
time="2019-01-07T14:21:12.327462975+08:00" level=warning msg="unsupported address" address="fe80::42:acff:fe11:3/64" arch=amd64 command=create container=ccf16de541600f6fd40663d940149f2e522683ad7714fb15c110ed2032c36447 name=kata-runtime pid=324059 source=virtcontainers subsystem=network unsupported-address-type=ipv6
time="2019-01-07T14:21:12.32758555+08:00" level=warning msg="unsupported route" arch=amd64 command=create container=ccf16de541600f6fd40663d940149f2e522683ad7714fb15c110ed2032c36447 destination="fe80::/64" name=kata-runtime pid=324059 source=virtcontainers subsystem=network unsupported-route-type=ipv6
time="2019-01-07T14:21:12.465424869+08:00" level=error msg="rpc error: code = Internal desc = Could not run process: container_linux.go:348: starting container process caused \"process_linux.go:279: applying cgroup configuration for process caused \\\"failed to write 0-3 to cpuset.cpus: write /sys/fs/cgroup/cpuset/system.slice:docker:ccf16de541600f6fd40663d940149f2e522683ad7714fb15c110ed2032c36447/cpuset.cpus: invalid argument\\\"\"" arch=amd64 command=create container=ccf16de541600f6fd40663d940149f2e522683ad7714fb15c110ed2032c36447 name=kata-runtime pid=324059 source=runtime
time="2019-01-07T14:21:12.48620523+08:00" level=error msg="Container ID (ccf16de541600f6fd40663d940149f2e522683ad7714fb15c110ed2032c36447) does not exist" arch=amd64 command=delete container=ccf16de541600f6fd40663d940149f2e522683ad7714fb15c110ed2032c36447 name=kata-runtime pid=324150 source=runtime
time="2019-01-07T14:21:12.511675167+08:00" level=error msg="Container ID (ccf16de541600f6fd40663d940149f2e522683ad7714fb15c110ed2032c36447) does not exist" arch=amd64 command=delete container=ccf16de541600f6fd40663d940149f2e522683ad7714fb15c110ed2032c36447 name=kata-runtime pid=324161 source=runtime
time="2019-01-07T14:21:27.018914138+08:00" level=warning msg="fetch sandbox device failed" arch=amd64 command=create container=2ba7ac8a5668ff44362fdff1c89d904e5aa1a5cd5cddcb7b35f9fd976094bb47 error="open /run/vc/sbs/2ba7ac8a5668ff44362fdff1c89d904e5aa1a5cd5cddcb7b35f9fd976094bb47/devices.json: no such file or directory" name=kata-runtime pid=324789 sandbox=2ba7ac8a5668ff44362fdff1c89d904e5aa1a5cd5cddcb7b35f9fd976094bb47 sandboxid=2ba7ac8a5668ff44362fdff1c89d904e5aa1a5cd5cddcb7b35f9fd976094bb47 source=virtcontainers subsystem=sandbox
time="2019-01-07T14:21:27.771973028+08:00" level=warning msg="unsupported address" address="fe80::42:acff:fe11:3/64" arch=amd64 command=create container=2ba7ac8a5668ff44362fdff1c89d904e5aa1a5cd5cddcb7b35f9fd976094bb47 name=kata-runtime pid=324789 source=virtcontainers subsystem=network unsupported-address-type=ipv6
time="2019-01-07T14:21:27.772141387+08:00" level=warning msg="unsupported route" arch=amd64 command=create container=2ba7ac8a5668ff44362fdff1c89d904e5aa1a5cd5cddcb7b35f9fd976094bb47 destination="fe80::/64" name=kata-runtime pid=324789 source=virtcontainers subsystem=network unsupported-route-type=ipv6
time="2019-01-07T14:21:27.904427639+08:00" level=error msg="rpc error: code = Internal desc = Could not run process: container_linux.go:348: starting container process caused \"process_linux.go:279: applying cgroup configuration for process caused \\\"failed to write 0-3 to cpuset.cpus: write /sys/fs/cgroup/cpuset/system.slice:docker:2ba7ac8a5668ff44362fdff1c89d904e5aa1a5cd5cddcb7b35f9fd976094bb47/cpuset.cpus: invalid argument\\\"\"" arch=amd64 command=create container=2ba7ac8a5668ff44362fdff1c89d904e5aa1a5cd5cddcb7b35f9fd976094bb47 name=kata-runtime pid=324789 source=runtime
time="2019-01-07T14:21:27.932128954+08:00" level=error msg="Container ID (2ba7ac8a5668ff44362fdff1c89d904e5aa1a5cd5cddcb7b35f9fd976094bb47) does not exist" arch=amd64 command=delete container=2ba7ac8a5668ff44362fdff1c89d904e5aa1a5cd5cddcb7b35f9fd976094bb47 name=kata-runtime pid=324881 source=runtime
time="2019-01-07T14:21:27.968829587+08:00" level=error msg="Container ID (2ba7ac8a5668ff44362fdff1c89d904e5aa1a5cd5cddcb7b35f9fd976094bb47) does not exist" arch=amd64 command=delete container=2ba7ac8a5668ff44362fdff1c89d904e5aa1a5cd5cddcb7b35f9fd976094bb47 name=kata-runtime pid=324892 source=runtime
time="2019-01-07T14:27:15.882547816+08:00" level=warning msg="fetch sandbox device failed" arch=amd64 command=create container=498aeb9001dcdb47c0dc6b9bf1722a24621dc9b4eb043b73b143eaaa43f63a5d error="open /run/vc/sbs/498aeb9001dcdb47c0dc6b9bf1722a24621dc9b4eb043b73b143eaaa43f63a5d/devices.json: no such file or directory" name=kata-runtime pid=8529 sandbox=498aeb9001dcdb47c0dc6b9bf1722a24621dc9b4eb043b73b143eaaa43f63a5d sandboxid=498aeb9001dcdb47c0dc6b9bf1722a24621dc9b4eb043b73b143eaaa43f63a5d source=virtcontainers subsystem=sandbox
time="2019-01-07T14:27:16.65689883+08:00" level=warning msg="unsupported address" address="fe80::42:acff:fe11:3/64" arch=amd64 command=create container=498aeb9001dcdb47c0dc6b9bf1722a24621dc9b4eb043b73b143eaaa43f63a5d name=kata-runtime pid=8529 source=virtcontainers subsystem=network unsupported-address-type=ipv6
time="2019-01-07T14:27:16.657060281+08:00" level=warning msg="unsupported route" arch=amd64 command=create container=498aeb9001dcdb47c0dc6b9bf1722a24621dc9b4eb043b73b143eaaa43f63a5d destination="fe80::/64" name=kata-runtime pid=8529 source=virtcontainers subsystem=network unsupported-route-type=ipv6
time="2019-01-07T14:27:17.877573452+08:00" level=error msg="Container not ready, running or paused, impossible to signal the container" arch=amd64 command=kill container=498aeb9001dcdb47c0dc6b9bf1722a24621dc9b4eb043b73b143eaaa43f63a5d name=kata-runtime pid=8679 sandbox=498aeb9001dcdb47c0dc6b9bf1722a24621dc9b4eb043b73b143eaaa43f63a5d source=runtime
time="2019-01-07T14:27:17.976190014+08:00" level=error msg="Container ID (498aeb9001dcdb47c0dc6b9bf1722a24621dc9b4eb043b73b143eaaa43f63a5d) does not exist" arch=amd64 command=delete container=498aeb9001dcdb47c0dc6b9bf1722a24621dc9b4eb043b73b143eaaa43f63a5d name=kata-runtime pid=8725 source=runtime
time="2019-01-07T14:28:16.594016968+08:00" level=warning msg="fetch sandbox device failed" arch=amd64 command=create container=76b674e549d454142ab04da0a9febf118373bdfb32bfbf5db4df81fa91d0b015 error="open /run/vc/sbs/76b674e549d454142ab04da0a9febf118373bdfb32bfbf5db4df81fa91d0b015/devices.json: no such file or directory" name=kata-runtime pid=10581 sandbox=76b674e549d454142ab04da0a9febf118373bdfb32bfbf5db4df81fa91d0b015 sandboxid=76b674e549d454142ab04da0a9febf118373bdfb32bfbf5db4df81fa91d0b015 source=virtcontainers subsystem=sandbox
time="2019-01-07T14:28:17.400626822+08:00" level=warning msg="unsupported address" address="fe80::42:acff:fe11:3/64" arch=amd64 command=create container=76b674e549d454142ab04da0a9febf118373bdfb32bfbf5db4df81fa91d0b015 name=kata-runtime pid=10581 source=virtcontainers subsystem=network unsupported-address-type=ipv6
time="2019-01-07T14:28:17.400753068+08:00" level=warning msg="unsupported route" arch=amd64 command=create container=76b674e549d454142ab04da0a9febf118373bdfb32bfbf5db4df81fa91d0b015 destination="fe80::/64" name=kata-runtime pid=10581 source=virtcontainers subsystem=network unsupported-route-type=ipv6
time="2019-01-07T14:28:17.536390844+08:00" level=error msg="rpc error: code = Internal desc = Could not run process: container_linux.go:348: starting container process caused \"process_linux.go:279: applying cgroup configuration for process caused \\\"failed to write 1 to cpuset.cpus: write /sys/fs/cgroup/cpuset/system.slice:docker:76b674e549d454142ab04da0a9febf118373bdfb32bfbf5db4df81fa91d0b015/cpuset.cpus: invalid argument\\\"\"" arch=amd64 command=create container=76b674e549d454142ab04da0a9febf118373bdfb32bfbf5db4df81fa91d0b015 name=kata-runtime pid=10581 source=runtime
time="2019-01-07T14:28:17.560211701+08:00" level=error msg="Container ID (76b674e549d454142ab04da0a9febf118373bdfb32bfbf5db4df81fa91d0b015) does not exist" arch=amd64 command=delete container=76b674e549d454142ab04da0a9febf118373bdfb32bfbf5db4df81fa91d0b015 name=kata-runtime pid=10673 source=runtime
time="2019-01-07T14:28:17.587363662+08:00" level=error msg="Container ID (76b674e549d454142ab04da0a9febf118373bdfb32bfbf5db4df81fa91d0b015) does not exist" arch=amd64 command=delete container=76b674e549d454142ab04da0a9febf118373bdfb32bfbf5db4df81fa91d0b015 name=kata-runtime pid=10694 source=runtime
time="2019-01-07T14:41:15.267819222+08:00" level=warning msg="fetch sandbox device failed" arch=amd64 command=create container=f3ee1e8406433c46bae3b450c63e5117e2a72ccecc57a1776781031a24eaf916 error="open /run/vc/sbs/f3ee1e8406433c46bae3b450c63e5117e2a72ccecc57a1776781031a24eaf916/devices.json: no such file or directory" name=kata-runtime pid=34711 sandbox=f3ee1e8406433c46bae3b450c63e5117e2a72ccecc57a1776781031a24eaf916 sandboxid=f3ee1e8406433c46bae3b450c63e5117e2a72ccecc57a1776781031a24eaf916 source=virtcontainers subsystem=sandbox
time="2019-01-07T14:41:16.03019109+08:00" level=warning msg="unsupported address" address="fe80::42:acff:fe11:3/64" arch=amd64 command=create container=f3ee1e8406433c46bae3b450c63e5117e2a72ccecc57a1776781031a24eaf916 name=kata-runtime pid=34711 source=virtcontainers subsystem=network unsupported-address-type=ipv6
time="2019-01-07T14:41:16.030353661+08:00" level=warning msg="unsupported route" arch=amd64 command=create container=f3ee1e8406433c46bae3b450c63e5117e2a72ccecc57a1776781031a24eaf916 destination="fe80::/64" name=kata-runtime pid=34711 source=virtcontainers subsystem=network unsupported-route-type=ipv6
time="2019-01-07T14:41:16.173436305+08:00" level=error msg="rpc error: code = Internal desc = Could not run process: container_linux.go:348: starting container process caused \"process_linux.go:279: applying cgroup configuration for process caused \\\"failed to write 1 to cpuset.cpus: write /sys/fs/cgroup/cpuset/system.slice:docker:f3ee1e8406433c46bae3b450c63e5117e2a72ccecc57a1776781031a24eaf916/cpuset.cpus: invalid argument\\\"\"" arch=amd64 command=create container=f3ee1e8406433c46bae3b450c63e5117e2a72ccecc57a1776781031a24eaf916 name=kata-runtime pid=34711 source=runtime
time="2019-01-07T14:41:16.199197464+08:00" level=error msg="Container ID (f3ee1e8406433c46bae3b450c63e5117e2a72ccecc57a1776781031a24eaf916) does not exist" arch=amd64 command=delete container=f3ee1e8406433c46bae3b450c63e5117e2a72ccecc57a1776781031a24eaf916 name=kata-runtime pid=34802 source=runtime
time="2019-01-07T14:41:16.224737261+08:00" level=error msg="Container ID (f3ee1e8406433c46bae3b450c63e5117e2a72ccecc57a1776781031a24eaf916) does not exist" arch=amd64 command=delete container=f3ee1e8406433c46bae3b450c63e5117e2a72ccecc57a1776781031a24eaf916 name=kata-runtime pid=34811 source=runtime

Proxy logs

Recent proxy problems found in system journal:

time="2019-01-07T14:20:42.428425703+08:00" level=fatal msg="channel error" error="accept unix /run/vc/sbs/da5664fbb0e8085db3e2b1b280164a60d475bc8f468ab54179a84266024bff77/proxy.sock: use of closed network connection" name=kata-proxy pid=22712 sandbox=da5664fbb0e8085db3e2b1b280164a60d475bc8f468ab54179a84266024bff77 source=proxy
time="2019-01-07T14:21:12.411522611+08:00" level=fatal msg="failed to handle exit signal" error="close unix @->/run/vc/vm/ccf16de541600f6fd40663d940149f2e522683ad7714fb15c110ed2032c36447/kata.sock: use of closed network connection" name=kata-proxy pid=324106 sandbox=ccf16de541600f6fd40663d940149f2e522683ad7714fb15c110ed2032c36447 source=proxy
time="2019-01-07T14:21:27.851173048+08:00" level=fatal msg="channel error" error="accept unix /run/vc/sbs/2ba7ac8a5668ff44362fdff1c89d904e5aa1a5cd5cddcb7b35f9fd976094bb47/proxy.sock: use of closed network connection" name=kata-proxy pid=324863 sandbox=2ba7ac8a5668ff44362fdff1c89d904e5aa1a5cd5cddcb7b35f9fd976094bb47 source=proxy
time="2019-01-07T14:27:17.903597261+08:00" level=fatal msg="failed to handle exit signal" error="close unix @->/run/vc/vm/498aeb9001dcdb47c0dc6b9bf1722a24621dc9b4eb043b73b143eaaa43f63a5d/kata.sock: use of closed network connection" name=kata-proxy pid=8575 sandbox=498aeb9001dcdb47c0dc6b9bf1722a24621dc9b4eb043b73b143eaaa43f63a5d source=proxy
time="2019-01-07T14:28:17.480712234+08:00" level=fatal msg="failed to handle exit signal" error="close unix @->/run/vc/vm/76b674e549d454142ab04da0a9febf118373bdfb32bfbf5db4df81fa91d0b015/kata.sock: use of closed network connection" name=kata-proxy pid=10640 sandbox=76b674e549d454142ab04da0a9febf118373bdfb32bfbf5db4df81fa91d0b015 source=proxy
time="2019-01-07T14:41:16.118420908+08:00" level=fatal msg="failed to handle exit signal" error="close unix @->/run/vc/vm/f3ee1e8406433c46bae3b450c63e5117e2a72ccecc57a1776781031a24eaf916/kata.sock: use of closed network connection" name=kata-proxy pid=34760 sandbox=f3ee1e8406433c46bae3b450c63e5117e2a72ccecc57a1776781031a24eaf916 source=proxy

Shim logs

No recent shim problems found in system journal.

Throttler logs

No recent throttler problems found in system journal.

Container manager details

Have docker

Docker

Output of "docker version":

Client:
 Version:         1.13.1
 API version:     1.26
 Package version: <unknown>
 Go version:      go1.7.5
 Git commit:      89b8308/1.13.1
 Built:           Tue Oct 23 08:39:46 2018
 OS/Arch:         linux/amd64

Server:
 Version:         1.13.1
 API version:     1.26 (minimum version 1.12)
 Package version: <unknown>
 Go version:      go1.7.5
 Git commit:      c3848e1-unsupported
 Built:           Wed Jan  2 08:08:41 2019
 OS/Arch:         linux/amd64
 Experimental:    false

Output of "docker info":

Containers: 142
 Running: 1
 Paused: 0
 Stopped: 141
Images: 329
Server Version: 1.13.1
Storage Driver: devicemapper
 Pool Name: dockervg-thinpool
 Pool Blocksize: 524.3 kB
 Base Device Size: 10.74 GB
 Backing Filesystem: ext4
 Data file:
 Metadata file:
 Data Space Used: 79.88 GB
 Data Space Total: 2.55 TB
 Data Space Available: 2.47 TB
 Metadata Space Used: 26.28 MB
 Metadata Space Total: 16.98 GB
 Metadata Space Available: 16.95 GB
 Thin Pool Minimum Free Space: 255 GB
 Udev Sync Supported: true
 Deferred Removal Enabled: true
 Deferred Deletion Enabled: true
 Deferred Deleted Device Count: 0
 Library Version: 1.02.146-RHEL7 (2018-01-22)
Logging Driver: journald
Cgroup Driver: systemd
Plugins:
 Volume: local
 Network: bridge host macvlan null overlay
Swarm: inactive
Runtimes: kata-runtime runc
Default Runtime: runc
Init Binary: /usr/libexec/docker/docker-init-current
containerd version: 89b8308e0c1fc2f72f5d5247ff09894cd17f9e6b (expected: aa8187dbd3b7ad67d8e5e3a15115d3eef43a7ed1)
runc version: e36af54b9034454193dfa0eab35fb068590ce1ca (expected: 9df8b306d01f59d3a8029be411de015b7304dd8f)
init version: 5b117de7f824f3d3825737cf09581645abbe35d4 (expected: 949e6facb77383876aeff8a6944dde66b3089574)
Security Options:
 seccomp
  WARNING: You're not using the default seccomp profile
  Profile: /etc/docker/seccomp.json
Kernel Version: 3.10.0-693.mt20180601.62.el7.x86_64
Operating System: CentOS Linux 7 (Core)
OSType: linux
Architecture: x86_64
Number of Docker Hooks: 3
CPUs: 40
Total Memory: 377.3 GiB
Name: gh-hulk-k8s-ep14.gh.sankuai.com
ID: YBTM:HG7O:FDGB:ZB4C:FTX3:QQX3:LMWY:26AG:WKT3:WHDL:NA4Z:ZGAU
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Username: huxos
Registry: https://index.docker.io/v1/
Experimental: false
Insecure Registries:
 registry.inf.test.sankuai.com
 127.0.0.0/8
Registry Mirrors:
 http://127.0.0.1:65001
 registry-hulk.sankuai.com
Live Restore Enabled: true
Registries: docker.io (secure)

Output of "systemctl show docker":

Type=notify
Restart=on-abnormal
NotifyAccess=all
RestartUSec=100ms
TimeoutStartUSec=0
TimeoutStopUSec=1min 30s
WatchdogUSec=0
WatchdogTimestamp=Mon 2019-01-07 11:24:35 CST
WatchdogTimestampMonotonic=841534743079
StartLimitInterval=10000000
StartLimitBurst=5
StartLimitAction=none
FailureAction=none
PermissionsStartOnly=no
RootDirectoryStartOnly=no
RemainAfterExit=no
GuessMainPID=yes
MainPID=1974
ControlPID=0
FileDescriptorStoreMax=0
StatusErrno=0
Result=success
ExecMainStartTimestamp=Mon 2019-01-07 11:24:33 CST
ExecMainStartTimestampMonotonic=841532722740
ExecMainExitTimestampMonotonic=0
ExecMainPID=1974
ExecMainCode=0
ExecMainStatus=0
ExecStart={ path=/usr/bin/dockerd-current ; argv[]=/usr/bin/dockerd-current --exec-opt native.cgroupdriver=systemd --userland-proxy-path=/usr/libexec/docker/docker-proxy-current --init-path=/usr/libexec/docker/docker-init-current --seccomp-profile=/etc/docker/seccomp.json $OPTIONS $DOCKER_STORAGE_OPTIONS $DOCKER_NETWORK_OPTIONS $ADD_REGISTRY $BLOCK_REGISTRY $INSECURE_REGISTRY $REGISTRIES ; ignore_errors=no ; start_time=[Mon 2019-01-07 11:24:33 CST] ; stop_time=[n/a] ; pid=1974 ; code=(null) ; status=0/0 }
ExecReload={ path=/bin/kill ; argv[]=/bin/kill -s HUP $MAINPID ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }
Slice=system.slice
ControlGroup=/system.slice/docker.service
MemoryCurrent=266698752
TasksCurrent=140
Delegate=no
CPUAccounting=no
CPUShares=18446744073709551615
StartupCPUShares=18446744073709551615
CPUQuotaPerSecUSec=infinity
BlockIOAccounting=no
BlockIOWeight=18446744073709551615
StartupBlockIOWeight=18446744073709551615
MemoryAccounting=no
MemoryLimit=18446744073709551615
DevicePolicy=auto
TasksAccounting=no
TasksMax=18446744073709551615
Environment=GOTRACEBACK=crash DOCKER_HTTP_HOST_COMPAT=1 PATH=/usr/libexec/docker:/usr/bin:/usr/sbin
EnvironmentFile=/run/containers/registries.conf (ignore_errors=yes)
EnvironmentFile=/etc/sysconfig/docker (ignore_errors=yes)
EnvironmentFile=/etc/sysconfig/docker-storage (ignore_errors=yes)
EnvironmentFile=/etc/sysconfig/docker-network (ignore_errors=yes)
UMask=0022
LimitCPU=18446744073709551615
LimitFSIZE=18446744073709551615
LimitDATA=18446744073709551615
LimitSTACK=18446744073709551615
LimitCORE=18446744073709551615
LimitRSS=18446744073709551615
LimitNOFILE=1048576
LimitAS=18446744073709551615
LimitNPROC=1048576
LimitMEMLOCK=65536
LimitLOCKS=18446744073709551615
LimitSIGPENDING=1545541
LimitMSGQUEUE=819200
LimitNICE=0
LimitRTPRIO=0
LimitRTTIME=18446744073709551615
OOMScoreAdjust=0
Nice=0
IOScheduling=0
CPUSchedulingPolicy=0
CPUSchedulingPriority=0
TimerSlackNSec=50000
CPUSchedulingResetOnFork=no
NonBlocking=no
StandardInput=null
StandardOutput=journal
StandardError=inherit
TTYReset=no
TTYVHangup=no
TTYVTDisallocate=no
SyslogPriority=30
SyslogLevelPrefix=yes
SecureBits=0
CapabilityBoundingSet=18446744073709551615
AmbientCapabilities=0
MountFlags=0
PrivateTmp=no
PrivateNetwork=no
PrivateDevices=no
ProtectHome=no
ProtectSystem=no
SameProcessGroup=no
IgnoreSIGPIPE=yes
NoNewPrivileges=no
SystemCallErrorNumber=0
RuntimeDirectoryMode=0755
KillMode=process
KillSignal=15
SendSIGKILL=yes
SendSIGHUP=no
Id=docker.service
Names=docker.service
Requires=basic.target docker-cleanup.timer
Wants=docker-storage-setup.service system.slice
RequiredBy=docker-cleanup.service
WantedBy=multi-user.target
Conflicts=shutdown.target
Before=shutdown.target multi-user.target
After=registries.service rhel-push-plugin.socket system.slice network.target systemd-journald.socket basic.target rc-local.service hulk-host-init.service docker-storage-setup.service
Documentation=http://docs.docker.com
Description=Docker Application Container Engine
LoadState=loaded
ActiveState=active
SubState=running
FragmentPath=/usr/lib/systemd/system/docker.service
DropInPaths=/etc/systemd/system/docker.service.d/override.conf
UnitFileState=enabled
UnitFilePreset=disabled
InactiveExitTimestamp=Mon 2019-01-07 11:24:33 CST
InactiveExitTimestampMonotonic=841532722762
ActiveEnterTimestamp=Mon 2019-01-07 11:24:35 CST
ActiveEnterTimestampMonotonic=841534743136
ActiveExitTimestamp=Mon 2019-01-07 11:24:32 CST
ActiveExitTimestampMonotonic=841531666904
InactiveEnterTimestamp=Mon 2019-01-07 11:24:33 CST
InactiveEnterTimestampMonotonic=841532679258
CanStart=yes
CanStop=yes
CanReload=yes
CanIsolate=no
StopWhenUnneeded=no
RefuseManualStart=no
RefuseManualStop=no
AllowIsolate=no
DefaultDependencies=yes
OnFailureJobMode=replace
IgnoreOnIsolate=no
IgnoreOnSnapshot=no
NeedDaemonReload=no
JobTimeoutUSec=0
JobTimeoutAction=none
ConditionResult=yes
AssertResult=yes
ConditionTimestamp=Mon 2019-01-07 11:24:33 CST
ConditionTimestampMonotonic=841532722234
AssertTimestamp=Mon 2019-01-07 11:24:33 CST
AssertTimestampMonotonic=841532722234
Transient=no

No kubectl

Packages

No dpkg Have rpm Output of "rpm -qa|egrep "(cc-oci-runtimecc-runtimerunv|kata-proxy|kata-runtime|kata-shim|kata-ksm-throttler|kata-containers-image|linux-container|qemu-)"":

kata-proxy-1.3.0+git.6ddb006-9.1.x86_64
qemu-lite-bin-2.11.0+git.f886228056-12.1.x86_64
qemu-img-1.5.3-156.el7_5.5.x86_64
qemu-kvm-1.5.3-156.el7_5.5.x86_64
qemu-system-x86-2.0.0-1.el7.6.x86_64
kata-shim-1.3.0+git.5fbf1f0-8.1.x86_64
qemu-vanilla-data-2.11.2+git.0982a56a55-12.1.x86_64
ipxe-roms-qemu-20170123-1.git4e85b27.el7_4.1.noarch
qemu-lite-2.11.0+git.f886228056-12.1.x86_64
kata-containers-image-1.3.0-9.1.x86_64
libvirt-daemon-driver-qemu-3.9.0-14.el7_5.8.x86_64
kata-linux-container-4.14.67.12-10.1.x86_64
kata-proxy-bin-1.3.0+git.6ddb006-9.1.x86_64
kata-shim-bin-1.3.0+git.5fbf1f0-8.1.x86_64
qemu-kvm-common-1.5.3-156.el7_5.5.x86_64
qemu-common-2.0.0-1.el7.6.x86_64
kata-runtime-1.3.0+git.a786643-14.1.x86_64

---`

grahamwhaley commented 5 years ago

/cc @jcvenegas @devimc

devimc commented 5 years ago

@free2k thanks for raising this, your test case fails because by default kata containers run with 1 vcpu (cpu0) and you're trying to bind it to cpu1, if you want to bind your workload to a specific cpu make sure that the POD (Virtual Machine) has at least n+1 vcpus, for example

docker run --runtime kata-runtime --cpus 2 --cpuset-cpus="1" -it busybox /bin/sh
free2k commented 5 years ago

@devimc Thank you for your answer，There is another problem, I didn't understand it. docker run --runtime kata-runtime --cpus 8 --memory 16G -it daocloud.io/library/centos /bin/sh

sh-4.2# lscpu CPU(s): 9 On-line CPU(s) list: 0-8

I assigned 8 cpus, actually I saw 9 cpus in the virtual machine. But docker's runc is showing 8 cpu

caoruidong commented 5 years ago

See https://github.com/kata-containers/documentation/blob/master/constraints/cpu.md#do-not-waste-resources

jcvenegas commented 5 years ago

@free2k can we close the issue or you see any other problem ? you see one additional because one is let for the VM itself so you see one additional all the time. If you dont request any you only will see one cpu.

jcvenegas commented 5 years ago

@free2k actually I found an issue where this is a problematic error. The cpusets fails here because they dont exist on the VM but, the we should expect that those are applied on the host and may be dont fail or do a map on the guest.

So this is a limitation today in kata that we need to document.

devimc commented 5 years ago

@jcvenegas I think you fixed this error, can we close it?

middlestone commented 4 years ago

Is --cpuset-cpus supportted in kata? now I really confused. when I specify --cpus and --cpuset-cpus at the same time just as below, the result is that container has 2 cpus, not 1 or 4, in this case, it should be 4, correct? BTW, I don't mind the online CPUs is not corresponded with cpuset-cpus at this moment.

root@k8s:~# docker run --rm --runtime kata-runtime --cpus 1 --cpuset-cpus 0-3 -ti be2868bebaba bash root@1b9f7cbc6c85:/# lscpu Architecture: x86_64 CPU op-mode(s): 32-bit, 64-bit Byte Order: Little Endian CPU(s): 2 On-line CPU(s) list: 0,1 Thread(s) per core: 1 Core(s) per socket: 1 Socket(s): 2 Vendor ID: GenuineIntel CPU family: 6 Model: 61 Model name: Intel Core Processor (Broadwell, IBRS) Stepping: 2 CPU MHz: 2599.988 BogoMIPS: 5199.97 Hypervisor vendor: KVM Virtualization type: full L1d cache: 32K L1i cache: 32K L2 cache: 4096K L3 cache: 16384K Flags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ss syscall nx pdpe1gb rdtscp lm constant_tsc rep_good nopl xtopology cpuid tsc_known_freq pni pclmulqdq ssse3 fma cx16 pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand hypervisor lahf_lm abm 3dnowprefetch cpuid_fault invpcid_single pti ssbd ibrs ibpb fsgsbase tsc_adjust bmi1 hle avx2 smep bmi2 erms invpcid rtm mpx avx512f rdseed adx smap avx512cd xsaveopt arat

root@k8s:~# kata-runtime version kata-runtime : 1.10.0-alpha1 commit : <> OCI specs: 1.0.1-dev

devimc commented 4 years ago

@middlestone take a look to this document https://github.com/kata-containers/documentation/blob/master/design/vcpu-handling.md

regarding --cpuset-cpus 0-3 argument, it can't be honoured in the guest (VM) because there are not enough vcpus (4)

middlestone commented 4 years ago

@devimc thanks for your quick reply, I undertand what happen now.