ariel-adam
commented
3 years ago @snir911 please comment :-)
Closed Sh4d1 closed 2 years ago
ariel-adam
commented
3 years ago @snir911 please comment :-)
snir911
commented
3 years ago @Sh4d1, thanks for reporting this, It seems to me it might be related to #1544
Are you getting this error consistently? i.e every single pod you'll try to start is getting this? if not how it could be reproduced?
Sh4d1
commented
3 years ago Yep it looks like I'm getting this every time, with a basic pod:
apiVersion: v1
kind: Pod
metadata:
name: untrust
spec:
runtimeClassName: kata
containers:
- name: task-pv-container
image: nginx
jodh-intel
commented
3 years ago Thanks @Sh4d1. Please could you try enabling full debug, re-running and pasting the output of kata-collect-data.sh again?
Sh4d1
commented
3 years ago Hey!
How did you install Kata?
Basically juste curl -fsSL https://github.com/kata-containers/kata-containers/releases/download/2.0.1/kata-static-2.0.1-x86_64.tar.xz | tar -xJvf - -C / +
[plugins.cri.containerd.runtimes.kata]
runtime_type = "io.containerd.kata.v2"added to containerd's config.
Can you run a Kata Container with containerd running with a tiny busybox image for example?
# ctr --debug run --runtime "io.containerd.kata.v2" --rm -t "docker.io/library/busybox:latest" test-kata uname -r
ctr: Failed to check if grpc server is working: rpc error: code = DeadlineExceeded desc = timed out connecting to vsock 2033866087:1024: unknown
# Meta details
Running `kata-collect-data.sh` version `2.0.1 (commit 3df65f4f3a439c8c7b97dd581f8ff4fc1c877a70)` at `2021-04-06.16:08:47.840892243+0000`.
---
Runtime is `/opt/kata/bin/kata-runtime`.
# `kata-env`
```toml
[Meta]
Version = "1.0.25"
[Runtime]
Debug = true
Trace = false
DisableGuestSeccomp = true
DisableNewNetNs = false
SandboxCgroupOnly = false
Path = "/opt/kata/bin/kata-runtime"
[Runtime.Version]
OCI = "1.0.1-dev"
[Runtime.Version.Version]
Semver = "2.0.1"
Major = 2
Minor = 0
Patch = 1
Commit = "3df65f4f3a439c8c7b97dd581f8ff4fc1c877a70"
[Runtime.Config]
Path = "/opt/kata/share/defaults/kata-containers/configuration-qemu.toml"
[Hypervisor]
MachineType = "pc"
Version = "QEMU emulator version 5.0.0 (kata-static)\nCopyright (c) 2003-2020 Fabrice Bellard and the QEMU Project developers"
Path = "/opt/kata/bin/qemu-system-x86_64"
BlockDeviceDriver = "virtio-scsi"
EntropySource = "/dev/urandom"
SharedFS = "virtio-fs"
VirtioFSDaemon = "/opt/kata/libexec/kata-qemu/virtiofsd"
Msize9p = 8192
MemorySlots = 10
PCIeRootPort = 0
HotplugVFIOOnRootBus = false
Debug = true
[Image]
Path = "/opt/kata/share/kata-containers/kata-containers-image_clearlinux_2.0.1_agent_3df65f4f3a.img"
[Kernel]
Path = "/opt/kata/share/kata-containers/vmlinux-5.4.71-84"
Parameters = "systemd.unit=kata-containers.target systemd.mask=systemd-networkd.service systemd.mask=systemd-networkd.socket scsi_mod.scan=none agent.log=debug agent.log=debug initcall_debug"
[Initrd]
Path = ""
[Agent]
Debug = true
Trace = false
TraceMode = ""
TraceType = ""
[Host]
Kernel = "5.8.0-45-generic"
Architecture = "amd64"
VMContainerCapable = true
SupportVSocks = true
[Host.Distro]
Name = "Ubuntu"
Version = "20.04"
[Host.CPU]
Vendor = "AuthenticAMD"
Model = "AMD EPYC 7281 16-Core Processor"
CPUs = 4
[Host.Memory]
Total = 8148548
Free = 3744672
Available = 7323580
[Netmon]
Path = "/opt/kata/libexec/kata-containers/kata-netmon"
Debug = true
Enable = false
[Netmon.Version]
Semver = "2.0.1"
Major = 2
Minor = 0
Patch = 1
Commit = "<
# Runtime config files
## Runtime default config files
```
/etc/kata-containers/configuration.toml
/opt/kata/share/defaults/kata-containers/configuration.toml
```
## Runtime config file contents
Config file `/etc/kata-containers/configuration.toml` not found
```toml
# Copyright (c) 2017-2019 Intel Corporation
#
# SPDX-License-Identifier: Apache-2.0
#
# XXX: WARNING: this file is auto-generated.
# XXX:
# XXX: Source file: "cli/config/configuration-qemu.toml.in"
# XXX: Project:
# XXX: Name: Kata Containers
# XXX: Type: kata
[hypervisor.qemu]
path = "/opt/kata/bin/qemu-system-x86_64"
kernel = "/opt/kata/share/kata-containers/vmlinux.container"
image = "/opt/kata/share/kata-containers/kata-containers.img"
machine_type = "pc"
# List of valid annotation names for the hypervisor
# Each member of the list is a regular expression, which is the base name
# of the annotation, e.g. "path" for io.katacontainers.config.hypervisor.path"
enable_annotations = []
# List of valid annotations values for the hypervisor
# Each member of the list is a path pattern as described by glob(3).
# The default if not set is empty (all annotations rejected.)
# Your distribution recommends: ["/opt/kata/bin/qemu-system-x86_64"]
valid_hypervisor_paths = ["/opt/kata/bin/qemu-system-x86_64"]
# Optional space-separated list of options to pass to the guest kernel.
# For example, use `kernel_params = "vsyscall=emulate"` if you are having
# trouble running pre-2.15 glibc.
#
# WARNING: - any parameter specified here will take priority over the default
# parameter value of the same name used to start the virtual machine.
# Do not set values here unless you understand the impact of doing so as you
# may stop the virtual machine from booting.
# To see the list of default parameters, enable hypervisor debug, create a
# container and look for 'default-kernel-parameters' log entries.
kernel_params = "agent.log=debug initcall_debug"
# Path to the firmware.
# If you want that qemu uses the default firmware leave this option empty
firmware = ""
# Machine accelerators
# comma-separated list of machine accelerators to pass to the hypervisor.
# For example, `machine_accelerators = "nosmm,nosmbus,nosata,nopit,static-prt,nofw"`
machine_accelerators=""
# CPU features
# comma-separated list of cpu features to pass to the cpu
# For example, `cpu_features = "pmu=off,vmx=off"
cpu_features="pmu=off"
# Default number of vCPUs per SB/VM:
# unspecified or 0 --> will be set to 1
# < 0 --> will be set to the actual number of physical cores
# > 0 <= number of physical cores --> will be set to the specified number
# > number of physical cores --> will be set to the actual number of physical cores
default_vcpus = 1
# Default maximum number of vCPUs per SB/VM:
# unspecified or == 0 --> will be set to the actual number of physical cores or to the maximum number
# of vCPUs supported by KVM if that number is exceeded
# > 0 <= number of physical cores --> will be set to the specified number
# > number of physical cores --> will be set to the actual number of physical cores or to the maximum number
# of vCPUs supported by KVM if that number is exceeded
# WARNING: Depending of the architecture, the maximum number of vCPUs supported by KVM is used when
# the actual number of physical cores is greater than it.
# WARNING: Be aware that this value impacts the virtual machine's memory footprint and CPU
# the hotplug functionality. For example, `default_maxvcpus = 240` specifies that until 240 vCPUs
# can be added to a SB/VM, but the memory footprint will be big. Another example, with
# `default_maxvcpus = 8` the memory footprint will be small, but 8 will be the maximum number of
# vCPUs supported by the SB/VM. In general, we recommend that you do not edit this variable,
# unless you know what are you doing.
# NOTICE: on arm platform with gicv2 interrupt controller, set it to 8.
default_maxvcpus = 0
# Bridges can be used to hot plug devices.
# Limitations:
# * Currently only pci bridges are supported
# * Until 30 devices per bridge can be hot plugged.
# * Until 5 PCI bridges can be cold plugged per VM.
# This limitation could be a bug in qemu or in the kernel
# Default number of bridges per SB/VM:
# unspecified or 0 --> will be set to 1
# > 1 <= 5 --> will be set to the specified number
# > 5 --> will be set to 5
default_bridges = 1
# Default memory size in MiB for SB/VM.
# If unspecified then it will be set 2048 MiB.
default_memory = 2048
#
# Default memory slots per SB/VM.
# If unspecified then it will be set 10.
# This is will determine the times that memory will be hotadded to sandbox/VM.
#memory_slots = 10
# The size in MiB will be plused to max memory of hypervisor.
# It is the memory address space for the NVDIMM devie.
# If set block storage driver (block_device_driver) to "nvdimm",
# should set memory_offset to the size of block device.
# Default 0
#memory_offset = 0
# Specifies virtio-mem will be enabled or not.
# Please note that this option should be used with the command
# "echo 1 > /proc/sys/vm/overcommit_memory".
# Default false
#enable_virtio_mem = true
# Disable block device from being used for a container's rootfs.
# In case of a storage driver like devicemapper where a container's
# root file system is backed by a block device, the block device is passed
# directly to the hypervisor for performance reasons.
# This flag prevents the block device from being passed to the hypervisor,
# 9pfs is used instead to pass the rootfs.
disable_block_device_use = false
# Shared file system type:
# - virtio-9p (default)
# - virtio-fs
shared_fs = "virtio-fs"
# Path to vhost-user-fs daemon.
virtio_fs_daemon = "/opt/kata/libexec/kata-qemu/virtiofsd"
# List of valid annotations values for the virtiofs daemon
# The default if not set is empty (all annotations rejected.)
# Your distribution recommends: ["/opt/kata/libexec/kata-qemu/virtiofsd"]
valid_virtio_fs_daemon_paths = ["/opt/kata/libexec/kata-qemu/virtiofsd"]
# Default size of DAX cache in MiB
virtio_fs_cache_size = 0
# Extra args for virtiofsd daemon
#
# Format example:
# ["-o", "arg1=xxx,arg2", "-o", "hello world", "--arg3=yyy"]
#
# see `virtiofsd -h` for possible options.
virtio_fs_extra_args = ["--thread-pool-size=1"]
# Cache mode:
#
# - none
# Metadata, data, and pathname lookup are not cached in guest. They are
# always fetched from host and any changes are immediately pushed to host.
#
# - auto
# Metadata and pathname lookup cache expires after a configured amount of
# time (default is 1 second). Data is cached while the file is open (close
# to open consistency).
#
# - always
# Metadata, data, and pathname lookup are cached in guest and never expire.
virtio_fs_cache = "auto"
# Block storage driver to be used for the hypervisor in case the container
# rootfs is backed by a block device. This is virtio-scsi, virtio-blk
# or nvdimm.
block_device_driver = "virtio-scsi"
# Specifies cache-related options will be set to block devices or not.
# Default false
#block_device_cache_set = true
# Specifies cache-related options for block devices.
# Denotes whether use of O_DIRECT (bypass the host page cache) is enabled.
# Default false
#block_device_cache_direct = true
# Specifies cache-related options for block devices.
# Denotes whether flush requests for the device are ignored.
# Default false
#block_device_cache_noflush = true
# Enable iothreads (data-plane) to be used. This causes IO to be
# handled in a separate IO thread. This is currently only implemented
# for SCSI.
#
enable_iothreads = false
# Enable pre allocation of VM RAM, default false
# Enabling this will result in lower container density
# as all of the memory will be allocated and locked
# This is useful when you want to reserve all the memory
# upfront or in the cases where you want memory latencies
# to be very predictable
# Default false
#enable_mem_prealloc = true
# Enable huge pages for VM RAM, default false
# Enabling this will result in the VM memory
# being allocated using huge pages.
# This is useful when you want to use vhost-user network
# stacks within the container. This will automatically
# result in memory pre allocation
#enable_hugepages = true
# Enable vhost-user storage device, default false
# Enabling this will result in some Linux reserved block type
# major range 240-254 being chosen to represent vhost-user devices.
enable_vhost_user_store = false
# The base directory specifically used for vhost-user devices.
# Its sub-path "block" is used for block devices; "block/sockets" is
# where we expect vhost-user sockets to live; "block/devices" is where
# simulated block device nodes for vhost-user devices to live.
vhost_user_store_path = "/var/run/kata-containers/vhost-user"
# Enable vIOMMU, default false
# Enabling this will result in the VM having a vIOMMU device
# This will also add the following options to the kernel's
# command line: intel_iommu=on,iommu=pt
#enable_iommu = true
# Enable IOMMU_PLATFORM, default false
# Enabling this will result in the VM device having iommu_platform=on set
#enable_iommu_platform = true
# List of valid annotations values for the vhost user store path
# The default if not set is empty (all annotations rejected.)
# Your distribution recommends: ["/var/run/kata-containers/vhost-user"]
valid_vhost_user_store_paths = ["/var/run/kata-containers/vhost-user"]
# Enable file based guest memory support. The default is an empty string which
# will disable this feature. In the case of virtio-fs, this is enabled
# automatically and '/dev/shm' is used as the backing folder.
# This option will be ignored if VM templating is enabled.
#file_mem_backend = ""
# List of valid annotations values for the file_mem_backend annotation
# The default if not set is empty (all annotations rejected.)
# Your distribution recommends: [""]
valid_file_mem_backends = [""]
# Enable swap of vm memory. Default false.
# The behaviour is undefined if mem_prealloc is also set to true
#enable_swap = true
# This option changes the default hypervisor and kernel parameters
# to enable debug output where available.
#
# Default false
enable_debug = true
# Disable the customizations done in the runtime when it detects
# that it is running on top a VMM. This will result in the runtime
# behaving as it would when running on bare metal.
#
#disable_nesting_checks = true
# This is the msize used for 9p shares. It is the number of bytes
# used for 9p packet payload.
#msize_9p = 8192
# If false and nvdimm is supported, use nvdimm device to plug guest image.
# Otherwise virtio-block device is used.
# Default is false
#disable_image_nvdimm = true
# VFIO devices are hotplugged on a bridge by default.
# Enable hotplugging on root bus. This may be required for devices with
# a large PCI bar, as this is a current limitation with hotplugging on
# a bridge. This value is valid for "pc" machine type.
# Default false
#hotplug_vfio_on_root_bus = true
# Before hot plugging a PCIe device, you need to add a pcie_root_port device.
# Use this parameter when using some large PCI bar devices, such as Nvidia GPU
# The value means the number of pcie_root_port
# This value is valid when hotplug_vfio_on_root_bus is true and machine_type is "q35"
# Default 0
#pcie_root_port = 2
# If vhost-net backend for virtio-net is not desired, set to true. Default is false, which trades off
# security (vhost-net runs ring0) for network I/O performance.
#disable_vhost_net = true
#
# Default entropy source.
# The path to a host source of entropy (including a real hardware RNG)
# /dev/urandom and /dev/random are two main options.
# Be aware that /dev/random is a blocking source of entropy. If the host
# runs out of entropy, the VMs boot time will increase leading to get startup
# timeouts.
# The source of entropy /dev/urandom is non-blocking and provides a
# generally acceptable source of entropy. It should work well for pretty much
# all practical purposes.
#entropy_source= "/dev/urandom"
# Path to OCI hook binaries in the *guest rootfs*.
# This does not affect host-side hooks which must instead be added to
# the OCI spec passed to the runtime.
#
# You can create a rootfs with hooks by customizing the osbuilder scripts:
# https://github.com/kata-containers/osbuilder
#
# Hooks must be stored in a subdirectory of guest_hook_path according to their
# hook type, i.e. "guest_hook_path/{prestart,postart,poststop}".
# The agent will scan these directories for executable files and add them, in
# lexicographical order, to the lifecycle of the guest container.
# Hooks are executed in the runtime namespace of the guest. See the official documentation:
# https://github.com/opencontainers/runtime-spec/blob/v1.0.1/config.md#posix-platform-hooks
# Warnings will be logged if any error is encountered will scanning for hooks,
# but it will not abort container execution.
#guest_hook_path = "/usr/share/oci/hooks"
#
# Use rx Rate Limiter to control network I/O inbound bandwidth(size in bits/sec for SB/VM).
# In Qemu, we use classful qdiscs HTB(Hierarchy Token Bucket) to discipline traffic.
# Default 0-sized value means unlimited rate.
#rx_rate_limiter_max_rate = 0
# Use tx Rate Limiter to control network I/O outbound bandwidth(size in bits/sec for SB/VM).
# In Qemu, we use classful qdiscs HTB(Hierarchy Token Bucket) and ifb(Intermediate Functional Block)
# to discipline traffic.
# Default 0-sized value means unlimited rate.
#tx_rate_limiter_max_rate = 0
[factory]
# VM templating support. Once enabled, new VMs are created from template
# using vm cloning. They will share the same initial kernel, initramfs and
# agent memory by mapping it readonly. It helps speeding up new container
# creation and saves a lot of memory if there are many kata containers running
# on the same host.
#
# When disabled, new VMs are created from scratch.
#
# Note: Requires "initrd=" to be set ("image=" is not supported).
#
# Default false
#enable_template = true
# Specifies the path of template.
#
# Default "/run/vc/vm/template"
#template_path = "/run/vc/vm/template"
# The number of caches of VMCache:
# unspecified or == 0 --> VMCache is disabled
# > 0 --> will be set to the specified number
#
# VMCache is a function that creates VMs as caches before using it.
# It helps speed up new container creation.
# The function consists of a server and some clients communicating
# through Unix socket. The protocol is gRPC in protocols/cache/cache.proto.
# The VMCache server will create some VMs and cache them by factory cache.
# It will convert the VM to gRPC format and transport it when gets
# requestion from clients.
# Factory grpccache is the VMCache client. It will request gRPC format
# VM and convert it back to a VM. If VMCache function is enabled,
# kata-runtime will request VM from factory grpccache when it creates
# a new sandbox.
#
# Default 0
#vm_cache_number = 0
# Specify the address of the Unix socket that is used by VMCache.
#
# Default /var/run/kata-containers/cache.sock
#vm_cache_endpoint = "/var/run/kata-containers/cache.sock"
[agent.kata]
# If enabled, make the agent display debug-level messages.
# (default: disabled)
enable_debug = true
# Enable agent tracing.
#
# If enabled, the default trace mode is "dynamic" and the
# default trace type is "isolated". The trace mode and type are set
# explicity with the `trace_type=` and `trace_mode=` options.
#
# Notes:
#
# - Tracing is ONLY enabled when `enable_tracing` is set: explicitly
# setting `trace_mode=` and/or `trace_type=` without setting `enable_tracing`
# will NOT activate agent tracing.
#
# - See https://github.com/kata-containers/agent/blob/master/TRACING.md for
# full details.
#
# (default: disabled)
#enable_tracing = true
#
#trace_mode = "dynamic"
#trace_type = "isolated"
# Comma separated list of kernel modules and their parameters.
# These modules will be loaded in the guest kernel using modprobe(8).
# The following example can be used to load two kernel modules with parameters
# - kernel_modules=["e1000e InterruptThrottleRate=3000,3000,3000 EEE=1", "i915 enable_ppgtt=0"]
# The first word is considered as the module name and the rest as its parameters.
# Container will not be started when:
# * A kernel module is specified and the modprobe command is not installed in the guest
# or it fails loading the module.
# * The module is not available in the guest or it doesn't met the guest kernel
# requirements, like architecture and version.
#
kernel_modules=[]
# Enable debug console.
# If enabled, user can connect guest OS running inside hypervisor
# through "kata-runtime exec
Containerd shim v2 is `/usr/local/bin/containerd-shim-kata-v2`.
```
Kata Containers containerd shim: id: "io.containerd.kata.v2", version: 2.0.1, commit: 3df65f4f3a439c8c7b97dd581f8ff4fc1c877a70
```
# KSM throttler
## version
## systemd service
# Image details
```yaml
---
osbuilder:
url: "https://github.com/kata-containers/kata-containers/tools/osbuilder"
version: "2.0.1-3df65f4f3a439c8c7b97dd581f8ff4fc1c877a70"
rootfs-creation-time: "2021-01-19T20:23:47.239302721+0000Z"
description: "osbuilder rootfs"
file-format-version: "0.0.2"
architecture: "x86_64"
base-distro:
name: "Clear"
version: "34170"
packages:
default:
- "chrony"
- "iptables-bin"
- "kmod-bin"
- "libudev0-shim"
- "systemd"
- "util-linux-bin"
extra:
agent:
url: "https://github.com/kata-containers/kata-containers"
name: "kata-agent"
version: "2.0.1"
agent-is-init-daemon: "no"
```
---
# Initrd details
No initrd
---
# Logfiles
## Runtime logs
No recent runtime problems found in system journal.
No recent throttler problems found in system journal.
Recent problems found in system journal:
```
time="2021-04-06T15:52:34.48509948Z" level=warning msg="sandbox cgroups path is empty" pid=1052756 sandbox=ae67964f484dd455886f00c165a70f611640933553299a76da6e659e4341b2a4 source=virtcontainers subsystem=sandbox
time="2021-04-06T15:53:03.509014725Z" level=warning msg="sandbox cgroups path is empty" pid=1052896 sandbox=fcaa63a7a9437ec76e0e2a857fd742877462d0257a35e9dae32f396c0ca10476 source=virtcontainers subsystem=sandbox
time="2021-04-06T15:53:31.541474737Z" level=warning msg="sandbox cgroups path is empty" pid=1053111 sandbox=ae279b8eccec199cdf9682dc3d3f5c7bacb756683f57b32acbd0d6d5b3de4aee source=virtcontainers subsystem=sandbox
time="2021-04-06T15:54:01.302719836Z" level=warning msg="sandbox cgroups path is empty" pid=1053270 sandbox=850106762e61dcad4fb7b773ed678c548857382b636fea14efd87e0d34cebd41 source=virtcontainers subsystem=sandbox
time="2021-04-06T15:54:29.49288432Z" level=warning msg="sandbox cgroups path is empty" pid=1053419 sandbox=3522c1c892d01f4f8079b5df0fa83cea116f727aee60a829634f1a83e3ee217e source=virtcontainers subsystem=sandbox
time="2021-04-06T15:54:57.493231887Z" level=warning msg="sandbox cgroups path is empty" pid=1053582 sandbox=1ce28cd2e1c121794d2a3990b91355e13b9f4aeb12182504cd4604cc7b0d16fe source=virtcontainers subsystem=sandbox
time="2021-04-06T15:55:28.437895854Z" level=warning msg="sandbox cgroups path is empty" pid=1053736 sandbox=7f5d69be38a881b2a8e361169cfb8906046b93f09a262b4eb15d18e82b2b4a08 source=virtcontainers subsystem=sandbox
time="2021-04-06T15:55:56.213187139Z" level=warning msg="sandbox cgroups path is empty" pid=1054014 sandbox=f1b18b14bfe749aed4c26576848d796c843d8525369b319d298d4300afe516b1 source=virtcontainers subsystem=sandbox
time="2021-04-06T15:56:25.208368763Z" level=warning msg="sandbox cgroups path is empty" pid=1054195 sandbox=840384ba1ded16cff55400d75011603152102ef7eb86db08443f30efd37e66f0 source=virtcontainers subsystem=sandbox
time="2021-04-06T15:56:55.477637399Z" level=warning msg="sandbox cgroups path is empty" pid=1054359 sandbox=3c471cc5361c3b6f2e5d1bda69dff003fcfe6c6b7d739faaa3a971f918a21949 source=virtcontainers subsystem=sandbox
time="2021-04-06T15:57:27.480574674Z" level=warning msg="sandbox cgroups path is empty" pid=1054524 sandbox=1a8970cd40e5b44c862f53cc51e856a0e18f0c932d6e98e19f0e5b6c94861f5a source=virtcontainers subsystem=sandbox
time="2021-04-06T15:57:57.525365669Z" level=warning msg="sandbox cgroups path is empty" pid=1054678 sandbox=1a7d8b8560a167f59566aae854f4d0159616c8849322307c819c36d1b913fb50 source=virtcontainers subsystem=sandbox
time="2021-04-06T15:58:29.460952992Z" level=warning msg="sandbox cgroups path is empty" pid=1054818 sandbox=d60d1bb813126ad061a75b2a24d38ee425bab1d5bc9baabee738a12eeea2dcbd source=virtcontainers subsystem=sandbox
time="2021-04-06T15:58:59.156881591Z" level=warning msg="sandbox cgroups path is empty" pid=1055033 sandbox=e62904898c0886a58a63100b2b9f8dc5300600f92b472e7ed095bfb40fd91223 source=virtcontainers subsystem=sandbox
time="2021-04-06T15:59:27.161483398Z" level=warning msg="sandbox cgroups path is empty" pid=1055175 sandbox=79a13f63c8938ba2e8b54731dde071320b4e4f3e68afb9bd7486f4080bb23c18 source=virtcontainers subsystem=sandbox
time="2021-04-06T15:59:57.493934462Z" level=warning msg="sandbox cgroups path is empty" pid=1055321 sandbox=29a22f61aae0dcdaa33461a174d53257313ee68322bf1b6bb8b1f15dce9d4c3b source=virtcontainers subsystem=sandbox
time="2021-04-06T16:00:27.542153226Z" level=warning msg="sandbox cgroups path is empty" pid=1055467 sandbox=83cf069dd0ac184bb690ad1e3ab5a9b29b41d9a363088307dc239a4747518f7e source=virtcontainers subsystem=sandbox
time="2021-04-06T16:02:41.205936353Z" level=info msg="scanner return error: read unix @->/run/vc/vm/test-kata/qmp.sock: use of closed network connection" pid=1055969 sandbox=test-kata source=virtcontainers subsystem=qmp
time="2021-04-06T16:02:57.33976192Z" level=warning msg="sandbox cgroups path is empty" pid=1055969 sandbox=test-kata source=virtcontainers subsystem=sandbox
time="2021-04-06T16:02:57.340778879Z" level=warning msg="failed to cleanup netns" error="failed to get netns /var/run/netns/cnitest-a0d107d3-0b51-c133-6970-316218969443: failed to Statfs \"/var/run/netns/cnitest-a0d107d3-0b51-c133-6970-316218969443\": no such file or directory" path=/var/run/netns/cnitest-a0d107d3-0b51-c133-6970-316218969443 pid=1055969 sandbox=test-kata source=katautils
time="2021-04-06T16:02:57.348112014Z" level=info msg="scanner return error:
# Container manager details
## Kubernetes
```
Client Version: version.Info{Major:"1", Minor:"20", GitVersion:"v1.20.4", GitCommit:"e87da0bd6e03ec3fea7933c4b5263d151aafd07c", GitTreeState:"clean", BuildDate:"2021-02-18T16:12:00Z", GoVersion:"go1.15.8", Compiler:"gc", Platform:"linux/amd64"}
The connection to the server localhost:8080 was refused - did you specify the right host or port?
```
```
apiVersion: v1
clusters: null
contexts: null
current-context: ""
kind: Config
preferences: {}
users: null
```
```
Type=simple
Restart=on-failure
NotifyAccess=none
RestartUSec=5s
TimeoutStartUSec=1min 30s
TimeoutStopUSec=1min 30s
TimeoutAbortUSec=1min 30s
RuntimeMaxUSec=infinity
WatchdogUSec=0
WatchdogTimestampMonotonic=0
RootDirectoryStartOnly=no
RemainAfterExit=no
GuessMainPID=yes
MainPID=1055565
ControlPID=0
FileDescriptorStoreMax=0
NFileDescriptorStore=0
StatusErrno=0
Result=success
ReloadResult=success
CleanResult=success
UID=[not set]
GID=[not set]
NRestarts=0
OOMPolicy=stop
ExecMainStartTimestamp=Tue 2021-04-06 16:00:32 UTC
ExecMainStartTimestampMonotonic=535408764542
ExecMainExitTimestampMonotonic=0
ExecMainPID=1055565
ExecMainCode=0
ExecMainStatus=0
ExecStart={ path=/usr/local/bin/kubelet ; argv[]=/usr/local/bin/kubelet --container-runtime-endpoint=unix:///var/run/containerd/containerd.sock --container-runtime=remote --image-pull-progress-deadline=2m --kubeconfig=/var/lib/kubelet/kubeconfig --network-plugin=cni --cert-dir=/var/lib/kubelet/pki --experimental-allocatable-ignore-eviction --node-labels=${NODELABELS} --pod-infra-container-image=gcr.io/google-containers/pause:3.2 --cloud-provider=external --hostname-override=${NODE_NAME} --provider-id=${PROVIDER_ID} --config=/var/lib/kubelet/kubelet.conf --image-credential-provider-config=/var/lib/kubelet/icp/config.yml --image-credential-provider-bin-dir=/var/lib/kubelet/icp/bin --v=2 ; ignore_errors=no ; start_time=[Tue 2021-04-06 16:00:32 UTC] ; stop_time=[n/a] ; pid=1055565 ; code=(null) ; status=0/0 }
ExecStartEx={ path=/usr/local/bin/kubelet ; argv[]=/usr/local/bin/kubelet --container-runtime-endpoint=unix:///var/run/containerd/containerd.sock --container-runtime=remote --image-pull-progress-deadline=2m --kubeconfig=/var/lib/kubelet/kubeconfig --network-plugin=cni --cert-dir=/var/lib/kubelet/pki --experimental-allocatable-ignore-eviction --node-labels=${NODELABELS} --pod-infra-container-image=gcr.io/google-containers/pause:3.2 --cloud-provider=external --hostname-override=${NODE_NAME} --provider-id=${PROVIDER_ID} --config=/var/lib/kubelet/kubelet.conf --image-credential-provider-config=/var/lib/kubelet/icp/config.yml --image-credential-provider-bin-dir=/var/lib/kubelet/icp/bin --v=2 ; flags= ; start_time=[Tue 2021-04-06 16:00:32 UTC] ; stop_time=[n/a] ; pid=1055565 ; code=(null) ; status=0/0 }
Slice=system.slice
ControlGroup=/system.slice/kubelet.service
MemoryCurrent=45281280
CPUUsageNSec=[not set]
EffectiveCPUs=
EffectiveMemoryNodes=
TasksCurrent=17
IPIngressBytes=[no data]
IPIngressPackets=[no data]
IPEgressBytes=[no data]
IPEgressPackets=[no data]
IOReadBytes=18446744073709551615
IOReadOperations=18446744073709551615
IOWriteBytes=18446744073709551615
IOWriteOperations=18446744073709551615
Delegate=no
CPUAccounting=no
CPUWeight=[not set]
StartupCPUWeight=[not set]
CPUShares=[not set]
StartupCPUShares=[not set]
CPUQuotaPerSecUSec=infinity
CPUQuotaPeriodUSec=infinity
AllowedCPUs=
AllowedMemoryNodes=
IOAccounting=no
IOWeight=[not set]
StartupIOWeight=[not set]
BlockIOAccounting=no
BlockIOWeight=[not set]
StartupBlockIOWeight=[not set]
MemoryAccounting=yes
DefaultMemoryLow=0
DefaultMemoryMin=0
MemoryMin=0
MemoryLow=0
MemoryHigh=infinity
MemoryMax=infinity
MemorySwapMax=infinity
MemoryLimit=infinity
DevicePolicy=auto
TasksAccounting=yes
TasksMax=9498
IPAccounting=no
EnvironmentFiles=/var/lib/kubelet/kubelet.env (ignore_errors=no)
UMask=0022
LimitCPU=infinity
LimitCPUSoft=infinity
LimitFSIZE=infinity
LimitFSIZESoft=infinity
LimitDATA=infinity
LimitDATASoft=infinity
LimitSTACK=infinity
LimitSTACKSoft=8388608
LimitCORE=infinity
LimitCORESoft=0
LimitRSS=infinity
LimitRSSSoft=infinity
LimitNOFILE=524288
LimitNOFILESoft=1024
LimitAS=infinity
LimitASSoft=infinity
LimitNPROC=31661
LimitNPROCSoft=31661
LimitMEMLOCK=65536
LimitMEMLOCKSoft=65536
LimitLOCKS=infinity
LimitLOCKSSoft=infinity
LimitSIGPENDING=31661
LimitSIGPENDINGSoft=31661
LimitMSGQUEUE=819200
LimitMSGQUEUESoft=819200
LimitNICE=0
LimitNICESoft=0
LimitRTPRIO=0
LimitRTPRIOSoft=0
LimitRTTIME=infinity
LimitRTTIMESoft=infinity
OOMScoreAdjust=0
Nice=0
IOSchedulingClass=0
IOSchedulingPriority=0
CPUSchedulingPolicy=0
CPUSchedulingPrFailed to parse bus message: Invalid argument
iority=0
CPUAffinity=
CPUAffinityFromNUMA=no
NUMAPolicy=n/a
NUMAMask=
TimerSlackNSec=50000
CPUSchedulingResetOnFork=no
NonBlocking=no
StandardInput=null
StandardInputData=
StandardOutput=journal
StandardError=inherit
TTYReset=no
TTYVHangup=no
TTYVTDisallocate=no
SyslogPriority=30
SyslogLevelPrefix=yes
SyslogLevel=6
SyslogFacility=3
LogLevelMax=-1
LogRateLimitIntervalUSec=0
LogRateLimitBurst=0
SecureBits=0
```
## containerd
```
containerd github.com/containerd/containerd v1.4.4 05f951a3781f4f2c1911b05e61c160e9c30eaa8e
```
```
Failed to parse bus message: Invalid argument
Type=notify
Restart=always
NotifyAccess=main
RestartUSec=5s
TimeoutStartUSec=1min 30s
TimeoutStopUSec=1min 30s
TimeoutAbortUSec=1min 30s
RuntimeMaxUSec=infinity
WatchdogUSec=0
WatchdogTimestampMonotonic=0
RootDirectoryStartOnly=no
RemainAfterExit=no
GuessMainPID=yes
MainPID=1055554
ControlPID=0
FileDescriptorStoreMax=0
NFileDescriptorStore=0
StatusErrno=0
Result=success
ReloadResult=success
CleanResult=success
UID=[not set]
GID=[not set]
NRestarts=0
OOMPolicy=continue
ExecMainStartTimestamp=Tue 2021-04-06 16:00:31 UTC
ExecMainStartTimestampMonotonic=535408605187
ExecMainExitTimestampMonotonic=0
ExecMainPID=1055554
ExecMainCode=0
ExecMainStatus=0
ExecStartPre={ path=/sbin/modprobe ; argv[]=/sbin/modprobe overlay ; ignore_errors=yes ; start_time=[Tue 2021-04-06 16:00:31 UTC] ; stop_time=[Tue 2021-04-06 16:00:31 UTC] ; pid=1055553 ; code=exited ; status=0 }
ExecStartPreEx={ path=/sbin/modprobe ; argv[]=/sbin/modprobe overlay ; flags=ignore-failure ; start_time=[Tue 2021-04-06 16:00:31 UTC] ; stop_time=[Tue 2021-04-06 16:00:31 UTC] ; pid=1055553 ; code=exited ; status=0 }
ExecStart={ path=/usr/local/bin/containerd ; argv[]=/usr/local/bin/containerd ; ignore_errors=no ; start_time=[Tue 2021-04-06 16:00:31 UTC] ; stop_time=[n/a] ; pid=1055554 ; code=(null) ; status=0/0 }
ExecStartEx={ path=/usr/local/bin/containerd ; argv[]=/usr/local/bin/containerd ; flags= ; start_time=[Tue 2021-04-06 16:00:31 UTC] ; stop_time=[n/a] ; pid=1055554 ; code=(null) ; status=0/0 }
Slice=system.slice
ControlGroup=/system.slice/containerd.service
MemoryCurrent=1520263168
CPUUsageNSec=[not set]
EffectiveCPUs=
EffectiveMemoryNodes=
TasksCurrent=140
IPIngressBytes=[no data]
IPIngressPackets=[no data]
IPEgressBytes=[no data]
IPEgressPackets=[no data]
IOReadBytes=18446744073709551615
IOReadOperations=18446744073709551615
IOWriteBytes=18446744073709551615
IOWriteOperations=18446744073709551615
Delegate=yes
DelegateControllers=cpu cpuacct cpuset io blkio memory devices pids bpf-firewall bpf-devices
CPUAccounting=no
CPUWeight=[not set]
StartupCPUWeight=[not set]
CPUShares=[not set]
StartupCPUShares=[not set]
CPUQuotaPerSecUSec=infinity
CPUQuotaPeriodUSec=infinity
AllowedCPUs=
AllowedMemoryNodes=
IOAccounting=no
IOWeight=[not set]
StartupIOWeight=[not set]
BlockIOAccounting=no
BlockIOWeight=[not set]
StartupBlockIOWeight=[not set]
MemoryAccounting=yes
DefaultMemoryLow=0
DefaultMemoryMin=0
MemoryMin=0
MemoryLow=0
MemoryHigh=infinity
MemoryMax=infinity
MemorySwapMax=infinity
MemoryLimit=infinity
DevicePolicy=auto
TasksAccounting=yes
TasksMax=infinity
IPAccounting=no
UMask=0022
LimitCPU=infinity
LimitCPUSoft=infinity
LimitFSIZE=infinity
LimitFSIZESoft=infinity
LimitDATA=infinity
LimitDATASoft=infinity
LimitSTACK=infinity
LimitSTACKSoft=8388608
LimitCORE=infinity
LimitCORESoft=infinity
LimitRSS=infinity
LimitRSSSoft=infinity
LimitNOFILE=1048576
LimitNOFILESoft=1048576
LimitAS=infinity
LimitASSoft=infinity
LimitNPROC=infinity
LimitNPROCSoft=infinity
LimitMEMLOCK=65536
LimitMEMLOCKSoft=65536
LimitLOCKS=infinity
LimitLOCKSSoft=infinity
LimitSIGPENDING=31661
LimitSIGPENDINGSoft=31661
LimitMSGQUEUE=819200
LimitMSGQUEUESoft=819200
LimitNICE=0
LimitNICESoft=0
LimitRTPRIO=0
LimitRTPRIOSoft=0
LimitRTTIME=infinity
LimitRTTIMESoft=infinity
OOMScoreAdjust=-999
Nice=0
IOSchedulingClass=0
IOSchedulingPriority=0
CPUSchedulingPolicy=0
CPUSchedulingPriority=0
CPUAffinity=
CPUAffinityFromNUMA=no
NUMAPolicy=n/a
NUMAMask=
TimerSlackNSec=50000
CPUSchedulingResetOnFork=no
NonBlocking=no
StandardInput=null
StandardInputData=
StandardOutput=journal
StandardError=inherit
TTYReset=no
TTYVHangup=no
TTYVTDisallocate=no
SyslogPriority=30
SyslogLevelPrefix=yes
SyslogLevel=6
SyslogFacility=3
LogLevelMax=-1
LogRateLimitIntervalUSec=0
LogRateLimitBurst=0
SecureBits=0
```
```toml
[debug]
level = "debug"
[plugins.cri.containerd]
no_pivot = false
[plugins.cri.containerd.runtimes]
[plugins.cri.containerd.runtimes.runc]
runtime_type = "io.containerd.runc.v2"
[plugins.cri.containerd.runtimes.runc.options]
NoPivotRoot = false
NoNewKeyring = false
ShimCgroup = ""
IoUid = 0
IoGid = 0
BinaryName = "runc"
Root = ""
CriuPath = ""
SystemdCgroup = false
[plugins.cri.containerd.untrusted]
runtime_type = "io.containerd.runsc.v1"
[plugins.cri.containerd.runtimes.runsc]
runtime_type = "io.containerd.runsc.v1"
[plugins.cri.containerd.runtimes.runsc.options]
NoPivotRoot = false
NoNewKeyring = false
ShimCgroup = ""
IoUid = 0
IoGid = 0
BinaryName = "/usr/local/bin/runsc"
Root = ""
CriuPath = ""
SystemdCgroup = false
[plugins.cri.containerd.runtimes.kata]
runtime_type = "io.containerd.kata.v2"
```
# Packages
Have `dpkg`
```
```
Runtime
---
/opt/kata/bin/kata-runtime kata-env
Runtime config files
Config file `/usr/share/defaults/kata-containers/configuration.toml` not found
---
cat "/opt/kata/share/defaults/kata-containers/configuration.toml"
Containerd shim v2
---
containerd-shim-kata-v2 --version
KSM throttler
Image details
Initrd details
Logfiles
## Throttler logs
Runtime logs
## Kata Containerd Shim v2 logs
Throttler logs
---
Kata Containerd Shim v2
Container manager details
Kubernetes
kubectl version
kubectl config view
systemctl show kubelet
---
containerd
containerd --version
systemctl show containerd
cat /etc/containerd/config.toml
Packages
No `rpm`
---
dpkg -l|egrep "(cc-oci-runtime|cc-runtime|runv|kata-runtime|kata-ksm-throttler|kata-containers-image|linux-container|qemu-)"
Sh4d1
commented
3 years ago Tried to get all containerd logs between 2 timeouts -> https://gist.github.com/Sh4d1/ac6a641cab146a56deffb1828a39c5ab
snir911
commented
3 years ago It seems like agent doesn't start for some reason, maybe try the initrd img, use: initrd = "/opt/kata/share/kata-containers/kata-containers-initrd.img" instead of image = "/opt/kata/share/kata-containers/kata-containers.img" in configuration.toml
Sh4d1
commented
3 years ago Just tried, same issue :/
Sh4d1
commented
3 years ago Almost the same issue, pod events :
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 11m default-scheduler Successfully assigned default/untrust to scw-test-default-5b325c396f274f1d863d38b2a7216
Warning FailedCreatePodSandBox 11m kubelet Failed to create pod sandbox: rpc error: code = Unknown desc = failed to create containerd task: Failed to check if grpc server is working: rpc error: code = DeadlineExceeded desc = timed out connecting to vsock 1641159211:1024: unknown
Warning FailedCreatePodSandBox 8m45s kubelet Failed to create pod sandbox: rpc error: code = DeadlineExceeded desc = failed to create containerd task: context deadline exceeded
Warning FailedCreatePodSandBox 8m15s kubelet Failed to create pod sandbox: rpc error: code = Unknown desc = failed to create containerd task: Failed to check if grpc server is working: rpc error: code = DeadlineExceeded desc = timed out connecting to vsock 3053871500:1024: unknown
Warning FailedCreatePodSandBox 7m44s kubelet Failed to create pod sandbox: rpc error: code = Unknown desc = failed to create containerd task: Failed to check if grpc server is working: rpc error: code = DeadlineExceeded desc = timed out connecting to vsock 3119628993:1024: unknown
Warning FailedCreatePodSandBox 7m13s kubelet Failed to create pod sandbox: rpc error: code = Unknown desc = failed to create containerd task: Failed to check if grpc server is working: rpc error: code = DeadlineExceeded desc = timed out connecting to vsock 4033628167:1024: unknown
Warning FailedCreatePodSandBox 6m41s kubelet Failed to create pod sandbox: rpc error: code = Unknown desc = failed to create containerd task: Failed to check if grpc server is working: rpc error: code = DeadlineExceeded desc = timed out connecting to vsock 863226773:1024: unknown
Warning FailedCreatePodSandBox 6m10s kubelet Failed to create pod sandbox: rpc error: code = Unknown desc = failed to create containerd task: Failed to check if grpc server is working: rpc error: code = DeadlineExceeded desc = timed out connecting to vsock 3452980632:1024: unknown
Warning FailedCreatePodSandBox 5m39s kubelet Failed to create pod sandbox: rpc error: code = Unknown desc = failed to create containerd task: Failed to check if grpc server is working: rpc error: code = DeadlineExceeded desc = timed out connecting to vsock 3685485082:1024: unknown
Warning FailedCreatePodSandBox 5m11s kubelet Failed to create pod sandbox: rpc error: code = Unknown desc = failed to create containerd task: Failed to check if grpc server is working: rpc error: code = DeadlineExceeded desc = timed out connecting to vsock 2272206044:1024: unknown
Normal Pulling 113s kubelet Pulling image "nginx"
Normal Pulled 104s kubelet Successfully pulled image "nginx" in 8.260531041s
Normal Created 104s kubelet Created container task-pv-container
Warning Failed 82s kubelet Error: failed to create containerd task: ttrpc: closed: unknown
Normal SandboxChanged 27s (x4 over 81s) kubelet Pod sandbox changed, it will be killed and re-created.
Warning FailedCreatePodSandBox 10s (x10 over 4m40s) kubelet (combined from similar events): Failed to create pod sandbox: rpc error: code = Unknown desc = failed to create containerd task: Failed to check if grpc server is working: rpc error: code = DeadlineExceeded desc = timed out connecting to vsock 3491360140:1024: unknown
fidencio
commented
3 years ago @Sh4d1, out of curiosity, are you trying to use kata-containers on a VM or on your host? If you're running on a VM, are you using Virtualbox, KVM, something else? And what's the host OS?
I'm just asking because I remember some similar reports coming from a user running Kata Containers, on a guest created using Virtualbox, on Windows.
Sh4d1
commented
3 years ago It's on a Scaleway VM. Thing is it was working with Kernel 5.4, but not on 5.8 anymore 🤔
Sh4d1
commented
3 years ago And underlying HV is a Linux host, and it's KVM based
fidencio
commented
3 years ago Ack! So, it's a regression, and may be on the kernel side. @stefano-garzarella, does this ring a bell?
stefano-garzarella
commented
3 years ago Sh4d1
commented
3 years ago If only it was available in the Ubuntu repos 😓 I can try yep! But that won't be a long term solution for me since we are based on the upstream Ubuntu packages.
stefano-garzarella
commented
3 years ago If only it was available in the Ubuntu repos sweat I can try yep! But that won't be a long term solution for me since we are based on the upstream Ubuntu packages.
I checked and it seems that Ubuntu internally maintains 5.8 backporting patches from other branches (like upstream 5.10). I' ll check further to see if all patches have been backported.
Sh4d1
commented
3 years ago Ah true indeed! Thanks very much appreciated 😄
Sh4d1
commented
3 years ago Tested with latest official ubuntu kernel : 5.8.0-48-generic and with 5.10.28-051028-generic (taken from https://kernel.ubuntu.com/~kernel-ppa/mainline/v5.10.28/) and still the same issue 🤔
stefano-garzarella
commented
3 years ago Tested with latest official ubuntu kernel :
5.8.0-48-genericand with5.10.28-051028-generic(taken from https://kernel.ubuntu.com/~kernel-ppa/mainline/v5.10.28/) and still the same issue thinking
Thanks for testing, 5.10.28 should contains all the fixes, so this one can be a new issue.
Can you attach the output of lsmod | grep vsock ?
I'll try to replicate the environment to see what's going on.
Sh4d1
commented
3 years ago Yep!
root@scw-kata-please-default-eec578f9cff34e0a833bdd:~# lsmod | grep vsock
vhost_vsock 24576 1
vmw_vsock_virtio_transport_common 36864 1 vhost_vsock
vhost 53248 2 vhost_vsock,vhost_net
vsock 45056 2 vmw_vsock_virtio_transport_common,vhost_vsock
root@scw-kata-please-default-eec578f9cff34e0a833bdd:~# uname -a
Linux scw-kata-please-default-eec578f9cff34e0a833bdd 5.10.28-051028-generic #202104071032 SMP Wed Apr 7 14:35:55 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux@Sh4d1 thanks! In Linux 5.5 we introduced support for multiple transports in vsock to handle nested VMs, but in this case I only see vhost_vsock, so that shouldn't be a problem.
I'll try to replicate and let you know.
Sh4d1
commented
3 years ago Thanks! Don't hesitate if you need more info 😄
Sh4d1
commented
3 years ago Just to be sure I was not miles away, I tried again with 5.4.0-70-generic and it works :p
snir911
commented
3 years ago Hi, I tried on ubuntu 20.04.2, 5.8.0-48-generic, i was not able to reproduce, it just worked. According to you logs last guest console output is "Run /sbin/init as init process" which if i'm not mistaken it's usually should be followed by kata-agent logs (which you enabled), hence, i suspect agent is not starting for some reason :P Could someone confirm this theory?
@Sh4d1 in 2.1.0-alpha2 timeout is 30s instead of 15s, I'd give it a try to see if it helps
Sh4d1
commented
3 years ago I'd say the agent is not running too :p but weird thing is that if I downgrade the kernel, it just works, without changing the kata configuration, which seems weird 🤔
@Sh4d1 in 2.1.0-alpha2 timeout is 30s instead of 15s, I'd give it a try to see if it helps
I'll try to test it today!
Sh4d1
commented
3 years ago Hum, same with 2.1.0-alpha2 🤔
vsxen
commented
3 years ago same question with kata 2.0.3 centos 7 kernel 4.20
snir911
commented
3 years ago :( another wild suggestion would be to check logs after adding to kernel_params in configuration.toml " init=/usr/bin/kata-agent "
Sh4d1
commented
3 years ago The logs : https://gist.github.com/Sh4d1/66b3b94ef8648b3e6120128f8548c21b
I can't seem to see why there is a:
Apr 13 06:56:54 scw-kata-21-default-92494fc088b4493b92bc447331 kata[152811]: time="2021-04-13T06:56:54.686417001Z" level=info msg="Stopping Sandbox" name=containerd-shim-v2 pid=152811 sandbox=9218960f55bd40991150833f2c4ef3277285c8a5482f5f6ad57766b928a
9d704 source=virtcontainers subsystem=qemu🤔
snir911
commented
3 years ago @Sh4d1 ,sorry, i meant in addition to what you had before, hence, "agent.log=debug initcall_debug init=/usr/bin/kata-agent " (as I'd expect to see guest console output with the booting logs as before)
I think the reason sandbox is being stopped is just the result of reaching the timeout without agent response, unless i'm missing something..
Sh4d1
commented
3 years ago Yep that's what I did, or rather init=/usr/bin/kata-agent agent.log=debug initcall_debug but I'd say it's the same :p
Sh4d1
commented
3 years ago Weirdest thing is I just downgrade the kernel + reboot and everything works 🤔
fidencio
commented
2 years ago @Sh4d1, I'm closing this one for now. Please, feel free to re-open it, to update it, or even to open a new issue in case you happen to hit this again.
lixd
commented
2 years ago same question with kata 2.4.0 centos 7 kernel 3.10 when update to kernel 5.17 it works
norbjd
commented
2 years ago Hello :wave:
We are facing the exact same errors, in the same conditions. We are using Ubuntu 20.04.
Everything works with kernel v5.7.19 (last v5.7 kernel), and we have timed out connecting to vsock error when using a v5.8 kernel (tested with v5.8.0 and even newer ones like v5.9, v5.15).
There is no need to use Kubernetes to get the error; just create a new Ubuntu 20.04 VM, install a kernel newer than 5.8.0. We use:
wget https://raw.githubusercontent.com/pimlie/ubuntu-mainline-kernel.sh/master/ubuntu-mainline-kernel.sh
chmod +x ubuntu-mainline-kernel.sh
./ubuntu-mainline-kernel.sh -i v5.8.0
rebootAfter rebooting the VM, we just run nerdctl run --rm --runtime io.containerd.run.kata.v2 ubuntu uname -r and we faced the timeout error.
To install kata, we followed this gist: https://gist.github.com/norbjd/1773b2930227d89c347f223f1d9b4ff0#install-prerequisites.
norbjd
commented
2 years ago Following my previous message.
So the issue is between 5.7 and 5.8 kernels.
After git bisecting the Linux kernel and around 14 custom kernels built and tested, I have succeeded to isolate the problematic commit: https://github.com/torvalds/linux/commit/c9d40913ac5a21eb2b976bb221a4677540e84eba have changed the arch/x86/kvm/x86.c file.
So I have tried to build a custom 5.8 kernel (with the changes in arch/x86/kvm/x86.c reverted), and got no issue anymore. I have also tried to build a custom 5.15 kernel without that commit and it worked too.
I'm not an expert, so I can't tell why this commit is problematic; however, it seems related to virtualization (kvm).
I don't know if the issue is really related to Kata, or if this issue is related to the Cloud provider I'm using (Scaleway). If someone could try to follow my Gist to install Kata (https://gist.github.com/norbjd/1773b2930227d89c347f223f1d9b4ff0#install-prerequisites) on a Ubuntu 20.04 with Kernel 5.8+ on another cloud provider, it would be much appreciated!
EDIT
From discussions on Slack, I've noticed that Kata is tested mostly on bare metal instances. Also, the guide to install Kata on AWS states that it requires bare metal instances:
Kata Containers on Amazon Web Services (AWS) makes use of i3.metal instances.
So I've decided to run Kata on Scaleway's Elastic Metal offer (bare metal instances). The issue with 5.8 kernel disappeared on these bare metal instances, and I've also noticed better performance (less CPU usage when starting multiple containers at the same time).
My conclusion is that Kata does not play very well with already virtualized environments. I guess it depends on the hypervisor used under the instances (KVM for Scaleway), and the issues can be different from one cloud provider to another.
soulseen
commented
1 year ago same question with kata 2.5.1 Debian 4.19.249
norbjd
commented
1 year ago @soulseen are you using a bare metal instance? Which cloud provider are you using?
soulseen
commented
1 year ago @norbjd No, I am using a VM.
Show kata-collect-data.sh details
# Meta details Running `kata-collect-data.sh` version `2.0.1 (commit 3df65f4f3a439c8c7b97dd581f8ff4fc1c877a70)` at `2021-04-06.13:22:03.156075323+0000`. ---
Runtime
Runtime is `/opt/kata/bin/kata-runtime`. # `kata-env`
---
/opt/kata/bin/kata-runtime kata-env
```toml [Meta] Version = "1.0.25" [Runtime] Debug = false Trace = false DisableGuestSeccomp = true DisableNewNetNs = false SandboxCgroupOnly = false Path = "/opt/kata/bin/kata-runtime" [Runtime.Version] OCI = "1.0.1-dev" [Runtime.Version.Version] Semver = "2.0.1" Major = 2 Minor = 0 Patch = 1 Commit = "3df65f4f3a439c8c7b97dd581f8ff4fc1c877a70" [Runtime.Config] Path = "/opt/kata/share/defaults/kata-containers/configuration-qemu.toml" [Hypervisor] MachineType = "pc" Version = "QEMU emulator version 5.0.0 (kata-static)\nCopyright (c) 2003-2020 Fabrice Bellard and the QEMU Project developers" Path = "/opt/kata/bin/qemu-system-x86_64" BlockDeviceDriver = "virtio-scsi" EntropySource = "/dev/urandom" SharedFS = "virtio-fs" VirtioFSDaemon = "/opt/kata/libexec/kata-qemu/virtiofsd" Msize9p = 8192 MemorySlots = 10 PCIeRootPort = 0 HotplugVFIOOnRootBus = false Debug = false [Image] Path = "/opt/kata/share/kata-containers/kata-containers-image_clearlinux_2.0.1_agent_3df65f4f3a.img" [Kernel] Path = "/opt/kata/share/kata-containers/vmlinux-5.4.71-84" Parameters = "systemd.unit=kata-containers.target systemd.mask=systemd-networkd.service systemd.mask=systemd-networkd.socket scsi_mod.scan=none" [Initrd] Path = "" [Agent] Debug = false Trace = false TraceMode = "" TraceType = "" [Host] Kernel = "5.8.0-45-generic" Architecture = "amd64" VMContainerCapable = true SupportVSocks = true [Host.Distro] Name = "Ubuntu" Version = "20.04" [Host.CPU] Vendor = "AuthenticAMD" Model = "AMD EPYC 7281 16-Core Processor" CPUs = 4 [Host.Memory] Total = 8148548 Free = 3632816 Available = 7207824 [Netmon] Path = "/opt/kata/libexec/kata-containers/kata-netmon" Debug = false Enable = false [Netmon.Version] Semver = "2.0.1" Major = 2 Minor = 0 Patch = 1 Commit = "<>"
```
Runtime config files
# Runtime config files ## Runtime default config files ``` /etc/kata-containers/configuration.toml /opt/kata/share/defaults/kata-containers/configuration.toml ``` ## Runtime config file contents Config file `/etc/kata-containers/configuration.toml` not found
Config file `/usr/share/defaults/kata-containers/configuration.toml` not found
---
cat "/opt/kata/share/defaults/kata-containers/configuration.toml"
```toml # Copyright (c) 2017-2019 Intel Corporation # # SPDX-License-Identifier: Apache-2.0 # # XXX: WARNING: this file is auto-generated. # XXX: # XXX: Source file: "cli/config/configuration-qemu.toml.in" # XXX: Project: # XXX: Name: Kata Containers # XXX: Type: kata [hypervisor.qemu] path = "/opt/kata/bin/qemu-system-x86_64" kernel = "/opt/kata/share/kata-containers/vmlinux.container" image = "/opt/kata/share/kata-containers/kata-containers.img" machine_type = "pc" # List of valid annotation names for the hypervisor # Each member of the list is a regular expression, which is the base name # of the annotation, e.g. "path" for io.katacontainers.config.hypervisor.path" enable_annotations = [] # List of valid annotations values for the hypervisor # Each member of the list is a path pattern as described by glob(3). # The default if not set is empty (all annotations rejected.) # Your distribution recommends: ["/opt/kata/bin/qemu-system-x86_64"] valid_hypervisor_paths = ["/opt/kata/bin/qemu-system-x86_64"] # Optional space-separated list of options to pass to the guest kernel. # For example, use `kernel_params = "vsyscall=emulate"` if you are having # trouble running pre-2.15 glibc. # # WARNING: - any parameter specified here will take priority over the default # parameter value of the same name used to start the virtual machine. # Do not set values here unless you understand the impact of doing so as you # may stop the virtual machine from booting. # To see the list of default parameters, enable hypervisor debug, create a # container and look for 'default-kernel-parameters' log entries. kernel_params = "" # Path to the firmware. # If you want that qemu uses the default firmware leave this option empty firmware = "" # Machine accelerators # comma-separated list of machine accelerators to pass to the hypervisor. # For example, `machine_accelerators = "nosmm,nosmbus,nosata,nopit,static-prt,nofw"` machine_accelerators="" # CPU features # comma-separated list of cpu features to pass to the cpu # For example, `cpu_features = "pmu=off,vmx=off" cpu_features="pmu=off" # Default number of vCPUs per SB/VM: # unspecified or 0 --> will be set to 1 # < 0 --> will be set to the actual number of physical cores # > 0 <= number of physical cores --> will be set to the specified number # > number of physical cores --> will be set to the actual number of physical cores default_vcpus = 1 # Default maximum number of vCPUs per SB/VM: # unspecified or == 0 --> will be set to the actual number of physical cores or to the maximum number # of vCPUs supported by KVM if that number is exceeded # > 0 <= number of physical cores --> will be set to the specified number # > number of physical cores --> will be set to the actual number of physical cores or to the maximum number # of vCPUs supported by KVM if that number is exceeded # WARNING: Depending of the architecture, the maximum number of vCPUs supported by KVM is used when # the actual number of physical cores is greater than it. # WARNING: Be aware that this value impacts the virtual machine's memory footprint and CPU # the hotplug functionality. For example, `default_maxvcpus = 240` specifies that until 240 vCPUs # can be added to a SB/VM, but the memory footprint will be big. Another example, with # `default_maxvcpus = 8` the memory footprint will be small, but 8 will be the maximum number of # vCPUs supported by the SB/VM. In general, we recommend that you do not edit this variable, # unless you know what are you doing. # NOTICE: on arm platform with gicv2 interrupt controller, set it to 8. default_maxvcpus = 0 # Bridges can be used to hot plug devices. # Limitations: # * Currently only pci bridges are supported # * Until 30 devices per bridge can be hot plugged. # * Until 5 PCI bridges can be cold plugged per VM. # This limitation could be a bug in qemu or in the kernel # Default number of bridges per SB/VM: # unspecified or 0 --> will be set to 1 # > 1 <= 5 --> will be set to the specified number # > 5 --> will be set to 5 default_bridges = 1 # Default memory size in MiB for SB/VM. # If unspecified then it will be set 2048 MiB. default_memory = 2048 # # Default memory slots per SB/VM. # If unspecified then it will be set 10. # This is will determine the times that memory will be hotadded to sandbox/VM. #memory_slots = 10 # The size in MiB will be plused to max memory of hypervisor. # It is the memory address space for the NVDIMM devie. # If set block storage driver (block_device_driver) to "nvdimm", # should set memory_offset to the size of block device. # Default 0 #memory_offset = 0 # Specifies virtio-mem will be enabled or not. # Please note that this option should be used with the command # "echo 1 > /proc/sys/vm/overcommit_memory". # Default false #enable_virtio_mem = true # Disable block device from being used for a container's rootfs. # In case of a storage driver like devicemapper where a container's # root file system is backed by a block device, the block device is passed # directly to the hypervisor for performance reasons. # This flag prevents the block device from being passed to the hypervisor, # 9pfs is used instead to pass the rootfs. disable_block_device_use = false # Shared file system type: # - virtio-9p (default) # - virtio-fs shared_fs = "virtio-fs" # Path to vhost-user-fs daemon. virtio_fs_daemon = "/opt/kata/libexec/kata-qemu/virtiofsd" # List of valid annotations values for the virtiofs daemon # The default if not set is empty (all annotations rejected.) # Your distribution recommends: ["/opt/kata/libexec/kata-qemu/virtiofsd"] valid_virtio_fs_daemon_paths = ["/opt/kata/libexec/kata-qemu/virtiofsd"] # Default size of DAX cache in MiB virtio_fs_cache_size = 0 # Extra args for virtiofsd daemon # # Format example: # ["-o", "arg1=xxx,arg2", "-o", "hello world", "--arg3=yyy"] # # see `virtiofsd -h` for possible options. virtio_fs_extra_args = ["--thread-pool-size=1"] # Cache mode: # # - none # Metadata, data, and pathname lookup are not cached in guest. They are # always fetched from host and any changes are immediately pushed to host. # # - auto # Metadata and pathname lookup cache expires after a configured amount of # time (default is 1 second). Data is cached while the file is open (close # to open consistency). # # - always # Metadata, data, and pathname lookup are cached in guest and never expire. virtio_fs_cache = "auto" # Block storage driver to be used for the hypervisor in case the container # rootfs is backed by a block device. This is virtio-scsi, virtio-blk # or nvdimm. block_device_driver = "virtio-scsi" # Specifies cache-related options will be set to block devices or not. # Default false #block_device_cache_set = true # Specifies cache-related options for block devices. # Denotes whether use of O_DIRECT (bypass the host page cache) is enabled. # Default false #block_device_cache_direct = true # Specifies cache-related options for block devices. # Denotes whether flush requests for the device are ignored. # Default false #block_device_cache_noflush = true # Enable iothreads (data-plane) to be used. This causes IO to be # handled in a separate IO thread. This is currently only implemented # for SCSI. # enable_iothreads = false # Enable pre allocation of VM RAM, default false # Enabling this will result in lower container density # as all of the memory will be allocated and locked # This is useful when you want to reserve all the memory # upfront or in the cases where you want memory latencies # to be very predictable # Default false #enable_mem_prealloc = true # Enable huge pages for VM RAM, default false # Enabling this will result in the VM memory # being allocated using huge pages. # This is useful when you want to use vhost-user network # stacks within the container. This will automatically # result in memory pre allocation #enable_hugepages = true # Enable vhost-user storage device, default false # Enabling this will result in some Linux reserved block type # major range 240-254 being chosen to represent vhost-user devices. enable_vhost_user_store = false # The base directory specifically used for vhost-user devices. # Its sub-path "block" is used for block devices; "block/sockets" is # where we expect vhost-user sockets to live; "block/devices" is where # simulated block device nodes for vhost-user devices to live. vhost_user_store_path = "/var/run/kata-containers/vhost-user" # Enable vIOMMU, default false # Enabling this will result in the VM having a vIOMMU device # This will also add the following options to the kernel's # command line: intel_iommu=on,iommu=pt #enable_iommu = true # Enable IOMMU_PLATFORM, default false # Enabling this will result in the VM device having iommu_platform=on set #enable_iommu_platform = true # List of valid annotations values for the vhost user store path # The default if not set is empty (all annotations rejected.) # Your distribution recommends: ["/var/run/kata-containers/vhost-user"] valid_vhost_user_store_paths = ["/var/run/kata-containers/vhost-user"] # Enable file based guest memory support. The default is an empty string which # will disable this feature. In the case of virtio-fs, this is enabled # automatically and '/dev/shm' is used as the backing folder. # This option will be ignored if VM templating is enabled. #file_mem_backend = "" # List of valid annotations values for the file_mem_backend annotation # The default if not set is empty (all annotations rejected.) # Your distribution recommends: [""] valid_file_mem_backends = [""] # Enable swap of vm memory. Default false. # The behaviour is undefined if mem_prealloc is also set to true #enable_swap = true # This option changes the default hypervisor and kernel parameters # to enable debug output where available. # # Default false #enable_debug = true # Disable the customizations done in the runtime when it detects # that it is running on top a VMM. This will result in the runtime # behaving as it would when running on bare metal. # #disable_nesting_checks = true # This is the msize used for 9p shares. It is the number of bytes # used for 9p packet payload. #msize_9p = 8192 # If false and nvdimm is supported, use nvdimm device to plug guest image. # Otherwise virtio-block device is used. # Default is false #disable_image_nvdimm = true # VFIO devices are hotplugged on a bridge by default. # Enable hotplugging on root bus. This may be required for devices with # a large PCI bar, as this is a current limitation with hotplugging on # a bridge. This value is valid for "pc" machine type. # Default false #hotplug_vfio_on_root_bus = true # Before hot plugging a PCIe device, you need to add a pcie_root_port device. # Use this parameter when using some large PCI bar devices, such as Nvidia GPU # The value means the number of pcie_root_port # This value is valid when hotplug_vfio_on_root_bus is true and machine_type is "q35" # Default 0 #pcie_root_port = 2 # If vhost-net backend for virtio-net is not desired, set to true. Default is false, which trades off # security (vhost-net runs ring0) for network I/O performance. #disable_vhost_net = true # # Default entropy source. # The path to a host source of entropy (including a real hardware RNG) # /dev/urandom and /dev/random are two main options. # Be aware that /dev/random is a blocking source of entropy. If the host # runs out of entropy, the VMs boot time will increase leading to get startup # timeouts. # The source of entropy /dev/urandom is non-blocking and provides a # generally acceptable source of entropy. It should work well for pretty much # all practical purposes. #entropy_source= "/dev/urandom" # Path to OCI hook binaries in the *guest rootfs*. # This does not affect host-side hooks which must instead be added to # the OCI spec passed to the runtime. # # You can create a rootfs with hooks by customizing the osbuilder scripts: # https://github.com/kata-containers/osbuilder # # Hooks must be stored in a subdirectory of guest_hook_path according to their # hook type, i.e. "guest_hook_path/{prestart,postart,poststop}". # The agent will scan these directories for executable files and add them, in # lexicographical order, to the lifecycle of the guest container. # Hooks are executed in the runtime namespace of the guest. See the official documentation: # https://github.com/opencontainers/runtime-spec/blob/v1.0.1/config.md#posix-platform-hooks # Warnings will be logged if any error is encountered will scanning for hooks, # but it will not abort container execution. #guest_hook_path = "/usr/share/oci/hooks" # # Use rx Rate Limiter to control network I/O inbound bandwidth(size in bits/sec for SB/VM). # In Qemu, we use classful qdiscs HTB(Hierarchy Token Bucket) to discipline traffic. # Default 0-sized value means unlimited rate. #rx_rate_limiter_max_rate = 0 # Use tx Rate Limiter to control network I/O outbound bandwidth(size in bits/sec for SB/VM). # In Qemu, we use classful qdiscs HTB(Hierarchy Token Bucket) and ifb(Intermediate Functional Block) # to discipline traffic. # Default 0-sized value means unlimited rate. #tx_rate_limiter_max_rate = 0 [factory] # VM templating support. Once enabled, new VMs are created from template # using vm cloning. They will share the same initial kernel, initramfs and # agent memory by mapping it readonly. It helps speeding up new container # creation and saves a lot of memory if there are many kata containers running # on the same host. # # When disabled, new VMs are created from scratch. # # Note: Requires "initrd=" to be set ("image=" is not supported). # # Default false #enable_template = true # Specifies the path of template. # # Default "/run/vc/vm/template" #template_path = "/run/vc/vm/template" # The number of caches of VMCache: # unspecified or == 0 --> VMCache is disabled # > 0 --> will be set to the specified number # # VMCache is a function that creates VMs as caches before using it. # It helps speed up new container creation. # The function consists of a server and some clients communicating # through Unix socket. The protocol is gRPC in protocols/cache/cache.proto. # The VMCache server will create some VMs and cache them by factory cache. # It will convert the VM to gRPC format and transport it when gets # requestion from clients. # Factory grpccache is the VMCache client. It will request gRPC format # VM and convert it back to a VM. If VMCache function is enabled, # kata-runtime will request VM from factory grpccache when it creates # a new sandbox. # # Default 0 #vm_cache_number = 0 # Specify the address of the Unix socket that is used by VMCache. # # Default /var/run/kata-containers/cache.sock #vm_cache_endpoint = "/var/run/kata-containers/cache.sock" [agent.kata] # If enabled, make the agent display debug-level messages. # (default: disabled) #enable_debug = true # Enable agent tracing. # # If enabled, the default trace mode is "dynamic" and the # default trace type is "isolated". The trace mode and type are set # explicity with the `trace_type=` and `trace_mode=` options. # # Notes: # # - Tracing is ONLY enabled when `enable_tracing` is set: explicitly # setting `trace_mode=` and/or `trace_type=` without setting `enable_tracing` # will NOT activate agent tracing. # # - See https://github.com/kata-containers/agent/blob/master/TRACING.md for # full details. # # (default: disabled) #enable_tracing = true # #trace_mode = "dynamic" #trace_type = "isolated" # Comma separated list of kernel modules and their parameters. # These modules will be loaded in the guest kernel using modprobe(8). # The following example can be used to load two kernel modules with parameters # - kernel_modules=["e1000e InterruptThrottleRate=3000,3000,3000 EEE=1", "i915 enable_ppgtt=0"] # The first word is considered as the module name and the rest as its parameters. # Container will not be started when: # * A kernel module is specified and the modprobe command is not installed in the guest # or it fails loading the module. # * The module is not available in the guest or it doesn't met the guest kernel # requirements, like architecture and version. # kernel_modules=[] # Enable debug console. # If enabled, user can connect guest OS running inside hypervisor # through "kata-runtime exec" command
#debug_console_enabled = true
[netmon]
# If enabled, the network monitoring process gets started when the
# sandbox is created. This allows for the detection of some additional
# network being added to the existing network namespace, after the
# sandbox has been created.
# (default: disabled)
#enable_netmon = true
# Specify the path to the netmon binary.
path = "/opt/kata/libexec/kata-containers/kata-netmon"
# If enabled, netmon messages will be sent to the system log
# (default: disabled)
#enable_debug = true
[runtime]
# If enabled, the runtime will log additional debug messages to the
# system log
# (default: disabled)
#enable_debug = true
#
# Internetworking model
# Determines how the VM should be connected to the
# the container network interface
# Options:
#
# - macvtap
# Used when the Container network interface can be bridged using
# macvtap.
#
# - none
# Used when customize network. Only creates a tap device. No veth pair.
#
# - tcfilter
# Uses tc filter rules to redirect traffic from the network interface
# provided by plugin to a tap interface connected to the VM.
#
internetworking_model="tcfilter"
# disable guest seccomp
# Determines whether container seccomp profiles are passed to the virtual
# machine and applied by the kata agent. If set to true, seccomp is not applied
# within the guest
# (default: true)
disable_guest_seccomp=true
# If enabled, the runtime will create opentracing.io traces and spans.
# (See https://www.jaegertracing.io/docs/getting-started).
# (default: disabled)
#enable_tracing = true
# If enabled, the runtime will not create a network namespace for shim and hypervisor processes.
# This option may have some potential impacts to your host. It should only be used when you know what you're doing.
# `disable_new_netns` conflicts with `enable_netmon`
# `disable_new_netns` conflicts with `internetworking_model=tcfilter` and `internetworking_model=macvtap`. It works only
# with `internetworking_model=none`. The tap device will be in the host network namespace and can connect to a bridge
# (like OVS) directly.
# If you are using docker, `disable_new_netns` only works with `docker run --net=none`
# (default: false)
#disable_new_netns = true
# if enabled, the runtime will add all the kata processes inside one dedicated cgroup.
# The container cgroups in the host are not created, just one single cgroup per sandbox.
# The runtime caller is free to restrict or collect cgroup stats of the overall Kata sandbox.
# The sandbox cgroup path is the parent cgroup of a container with the PodSandbox annotation.
# The sandbox cgroup is constrained if there is no container type annotation.
# See: https://godoc.org/github.com/kata-containers/runtime/virtcontainers#ContainerType
sandbox_cgroup_only=false
# Enabled experimental feature list, format: ["a", "b"].
# Experimental features are features not stable enough for production,
# they may break compatibility, and are prepared for a big version bump.
# Supported experimental features:
# (default: [])
experimental=[]
# If enabled, user can run pprof tools with shim v2 process through kata-monitor.
# (default: false)
# EnablePprof = true
```
Containerd shim v2
Containerd shim v2 is `/usr/local/bin/containerd-shim-kata-v2`.
---
containerd-shim-kata-v2 --version
``` Kata Containers containerd shim: id: "io.containerd.kata.v2", version: 2.0.1, commit: 3df65f4f3a439c8c7b97dd581f8ff4fc1c877a70 ```
KSM throttler
# KSM throttler ## version ## systemd service
Image details
# Image details ```yaml --- osbuilder: url: "https://github.com/kata-containers/kata-containers/tools/osbuilder" version: "2.0.1-3df65f4f3a439c8c7b97dd581f8ff4fc1c877a70" rootfs-creation-time: "2021-01-19T20:23:47.239302721+0000Z" description: "osbuilder rootfs" file-format-version: "0.0.2" architecture: "x86_64" base-distro: name: "Clear" version: "34170" packages: default: - "chrony" - "iptables-bin" - "kmod-bin" - "libudev0-shim" - "systemd" - "util-linux-bin" extra: agent: url: "https://github.com/kata-containers/kata-containers" name: "kata-agent" version: "2.0.1" agent-is-init-daemon: "no" ``` ---
Initrd details
# Initrd details No initrd ---
Logfiles
# Logfiles ## Runtime logs
## Throttler logs
## Kata Containerd Shim v2 logs
---
Runtime logs
No recent runtime problems found in system journal.
Throttler logs
No recent throttler problems found in system journal.
Kata Containerd Shim v2
Recent problems found in system journal: ``` time="2021-04-06T13:10:29.268863552Z" level=warning msg="sandbox cgroups path is empty" pid=1005355 sandbox=905febf403288c82eb56c7b29fb6916ef2295f401ec2568c21c1939121a9e40c source=virtcontainers subsystem=sandbox time="2021-04-06T13:11:01.208109576Z" level=warning msg="sandbox cgroups path is empty" pid=1005507 sandbox=765899992d7719c26ee7c239da54073ef164e58bc60c9897a25679800b3d5ca2 source=virtcontainers subsystem=sandbox time="2021-04-06T13:11:30.293091526Z" level=warning msg="sandbox cgroups path is empty" pid=1005648 sandbox=2981347558726fbd02e99a4f832924b71819aa85a9781fbcb33eab05e40d2861 source=virtcontainers subsystem=sandbox time="2021-04-06T13:12:01.973111272Z" level=warning msg="sandbox cgroups path is empty" pid=1005789 sandbox=56b31edb26aa6a760b5418ef1505ed4b9807bb5fb8dc6e695ac381310d1c0856 source=virtcontainers subsystem=sandbox time="2021-04-06T13:12:31.030259964Z" level=warning msg="sandbox cgroups path is empty" pid=1005926 sandbox=390bfccf7d05068618cc9ae09bd8bdd1232d9f0997540f387002da86e46fb75a source=virtcontainers subsystem=sandbox time="2021-04-06T13:13:01.141275323Z" level=warning msg="sandbox cgroups path is empty" pid=1006072 sandbox=4df516d5eadf7d479944ef81aa3d1dd9546be24a59ad9958975b00491fdf0cc8 source=virtcontainers subsystem=sandbox time="2021-04-06T13:13:32.119304039Z" level=warning msg="sandbox cgroups path is empty" pid=1006215 sandbox=3933bde6c9a3176aef951377d3c11fa0774b83081e242775853b3991f7186afc source=virtcontainers subsystem=sandbox time="2021-04-06T13:14:04.341415258Z" level=warning msg="sandbox cgroups path is empty" pid=1006360 sandbox=5bd5b62405491103e698b376ff530317e81d6ce82712e62c6fd20acd2b900e43 source=virtcontainers subsystem=sandbox time="2021-04-06T13:14:35.446225899Z" level=warning msg="sandbox cgroups path is empty" pid=1006504 sandbox=040784f96bc45bdeb3f53568eeda9dde533a69ddca11d52250eae7e2503c89d3 source=virtcontainers subsystem=sandbox time="2021-04-06T13:15:05.397096033Z" level=warning msg="sandbox cgroups path is empty" pid=1006647 sandbox=248888e0424ad6d7d57d4b4ab41003540e6381a9b480e188e9e9ec2ab447492a source=virtcontainers subsystem=sandbox time="2021-04-06T13:15:35.061363615Z" level=warning msg="sandbox cgroups path is empty" pid=1006790 sandbox=cff659796fe71956c6dcb0e1ead675ddb3fcdb80265de814bf1bb5a013b089b8 source=virtcontainers subsystem=sandbox time="2021-04-06T13:16:03.225126448Z" level=warning msg="sandbox cgroups path is empty" pid=1006931 sandbox=c73b452274e14e0fe86606868fc02aab4847f46dc2b93c2856196610a1fc47e1 source=virtcontainers subsystem=sandbox time="2021-04-06T13:16:33.461061175Z" level=warning msg="sandbox cgroups path is empty" pid=1007075 sandbox=383ffe8e440cae8625eadba79685ae1b657ee0a645418e2d86b0df73a64a49af source=virtcontainers subsystem=sandbox time="2021-04-06T13:17:04.276489294Z" level=warning msg="sandbox cgroups path is empty" pid=1007222 sandbox=ebc4e9e0250b553e2f3a5e75609e065d1979f93832b4b06ceaea3b6d3372be22 source=virtcontainers subsystem=sandbox time="2021-04-06T13:17:32.438335534Z" level=warning msg="sandbox cgroups path is empty" pid=1007362 sandbox=ffbcfeb5fedc88a0fd694f787c558284c48b9a4630f081264a7efef09eb5850a source=virtcontainers subsystem=sandbox time="2021-04-06T13:18:02.516549584Z" level=warning msg="sandbox cgroups path is empty" pid=1007601 sandbox=65f9ac466c617ef80d154fa9e73015fae4e54a0c40057506b628018555da27cb source=virtcontainers subsystem=sandbox time="2021-04-06T13:18:34.165732463Z" level=warning msg="sandbox cgroups path is empty" pid=1007817 sandbox=adacaf50bd0dbe34b8f7c4fc36126fb135a272a5784a4253ac5d60a7b630e561 source=virtcontainers subsystem=sandbox time="2021-04-06T13:19:06.19758607Z" level=warning msg="sandbox cgroups path is empty" pid=1007960 sandbox=6dfc52284792813d78d67a028c090d278c07a882b30e6cd8b01c626a01dc3398 source=virtcontainers subsystem=sandbox time="2021-04-06T13:19:35.449039983Z" level=warning msg="sandbox cgroups path is empty" pid=1008109 sandbox=42128cfadd17406812dd061b50caeaaa32986d2d1b20d8d8bcd78375351ea4c3 source=virtcontainers subsystem=sandbox time="2021-04-06T13:20:07.414213577Z" level=warning msg="sandbox cgroups path is empty" pid=1008255 sandbox=97b7deed49ee143d5f06aacfd9bcf46ff95b8fab9e53589349debf1099a2b493 source=virtcontainers subsystem=sandbox time="2021-04-06T13:20:38.16466577Z" level=warning msg="sandbox cgroups path is empty" pid=1008400 sandbox=1a690785ceb21f8c68ed6ec4baa5984ad32439d55db6eaaafdc66fce29d102f8 source=virtcontainers subsystem=sandbox time="2021-04-06T13:21:06.423355757Z" level=warning msg="sandbox cgroups path is empty" pid=1008560 sandbox=a18d4b6ec5e2c46333d97de18fe66ef3e4103d0fc02762546698412bd19c82f9 source=virtcontainers subsystem=sandbox time="2021-04-06T13:21:35.158288146Z" level=warning msg="sandbox cgroups path is empty" pid=1008708 sandbox=47ea332d73a797a5cbd019efd4347f85e64da5fef37165b3bb41f5549a1e2f83 source=virtcontainers subsystem=sandbox ```
Container manager details
# Container manager details
---
Kubernetes
## Kubernetes
kubectl version
``` Client Version: version.Info{Major:"1", Minor:"20", GitVersion:"v1.20.4", GitCommit:"e87da0bd6e03ec3fea7933c4b5263d151aafd07c", GitTreeState:"clean", BuildDate:"2021-02-18T16:12:00Z", GoVersion:"go1.15.8", Compiler:"gc", Platform:"linux/amd64"} The connection to the server localhost:8080 was refused - did you specify the right host or port? ```
kubectl config view
``` apiVersion: v1 clusters: null contexts: null current-context: "" kind: Config preferences: {} users: null ```
systemctl show kubelet
``` Type=simple Restart=on-failure NotifyAccess=none RestartUSec=5s TimeoutStartUSec=1min 30s TimeoutStopUSec=1min 30s TimeoutAbortUSec=1min 30s RuntimeMaxUSec=infinity WatchdogUSec=0 WatchdogTimestampMonotonic=0 RootDirectoryStartOnly=no RemainAfterExit=no GuessMainPID=yes MainPID=1152 ControlPID=0 FileDescriptorStoreMax=0 NFileDescriptorStore=0 StatusErrno=0 Result=success ReloadResult=success CleanResult=success UID=[not set] GID=[not set] NRestarts=0 OOMPolicy=stop ExecMainStartTimestamp=Wed 2021-03-31 11:18:19 UTC ExecMainStartTimestampMonotonic=76584939 ExecMainExitTimestampMonotonic=0 ExecMainPID=1152 ExecMainCode=0 ExecMainStatus=0 ExecStart={ path=/usr/local/bin/kubelet ; argv[]=/usr/local/bin/kubelet --container-runtime-endpoint=unix:///var/run/containerd/containerd.sock --container-runtime=remote --image-pull-progress-deadline=2m --kubeconfig=/var/lib/kubelet/kubeconfig --network-plugin=cni --cert-dir=/var/lib/kubelet/pki --experimental-allocatable-ignore-eviction --node-labels=${NODELABELS} --pod-infra-container-image=gcr.io/google-containers/pause:3.2 --cloud-provider=external --hostname-override=${NODE_NAME} --provider-id=${PROVIDER_ID} --config=/var/lib/kubelet/kubelet.conf --image-credential-provider-config=/var/lib/kubelet/icp/config.yml --image-credential-provider-bin-dir=/var/lib/kubelet/icp/bin --v=2 ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 } ExecStartEx={ path=/usr/local/bin/kubelet ; argv[]=/usr/local/bin/kubelet --container-runtime-endpoint=unix:///var/run/containerd/containerd.sock --container-runtime=remote --image-pull-progress-deadline=2m --kubeconfig=/var/lib/kubelet/kubeconfig --network-plugin=cni --cert-dir=/var/lib/kubelet/pki --experimental-allocatable-ignore-eviction --node-labels=${NODELABELS} --pod-infra-container-image=gcr.io/google-containers/pause:3.2 --cloud-provider=external --hostname-override=${NODE_NAME} --provider-id=${PROVIDER_ID} --config=/var/lib/kubelet/kubelet.conf --image-credential-provider-config=/var/lib/kubelet/icp/config.yml --image-credential-provider-bin-dir=/var/lib/kubelet/icp/bin --v=2 ; flags= ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 } Slice=system.slice ControlGroup=/system.slice/kubelet.service MemoryCurrent=65355776 CPUUsageNSec=[not set] EffectiveCPUs= EffectiveMemoryNodes= TasksCurrent=18 IPIngressBytes=[no data] IPIngressPackets=[no data] IPEgressBytes=[no data] IPEgressPackets=[no data] IOReadBytes=18446744073709551615 IOReadOperations=18446744073709551615 IOWriteBytes=18446744073709551615 IOWriteOperations=18446744073709551615 Delegate=no CPUAccounting=no CPUWeight=[not set] StartupCPUWeight=[not set] CPUShares=[not set] StartupCPUShares=[not set] CPUQuotaPerSecUSec=infinity CPUQuotaPeriodUSec=infinity AllowedCPUs= AllowedMemoryNodes= IOAccounting=no IOWeight=[not set] StartupIOWeight=[not set] BlockIOAccounting=no BlockIOWeight=[not set] StartupBlockIOWeight=[not set] MemoryAccounting=yes DefaultMemoryLow=0 DefaultMemoryMin=0 MemoryMin=0 MemoryLow=0 MemoryHigh=infinity MemoryMax=infinity MemorySwapMax=infinity MemoryLimit=infinity DevicePolicy=auto TasksAccounting=yes TasksMax=9498 IPAccounting=no EnvironmentFiles=/var/lib/kubelet/kubelet.env (ignore_errors=no) UMask=0022 LimitCPU=infinity LimitCPUSoft=infinity LimitFSIZE=infinity LimitFSIZESoft=infinity LimitDATA=infinity LimitDATASoft=infinity LimitSTACK=infinity LimitSTACKSoft=8388608 LimitCORE=infinity LimitCORESoft=0 LimitRSS=infinity LimitRSSSoft=infinity LimitNOFILE=524288 LimitNOFILESoft=1024 LimitAS=infinity LimitASSoft=infinity LimitNPROC=31661 LimitNPROCSoft=31661 LimitMEMLOCK=65536 LimitMEMLOCKSoft=65536 LimitLOCKS=infinity LimitLOCKSSoft=infinity LimitSIGPENDING=31661 LimitSIGPENDINGSoft=31661 LimitMSGQUEUE=819200 LimitMSGQUEUESoft=819200 LimitNICE=0 LimitNICESoft=0 LimitRTPRIO=0 LimitRTPRIOSoft=0 LimitRTTIME=infinity LimitRTTIMESoft=infinity OOMScoreAdjust=0 Nice=0 IOSchedulingClass=0 IOSchedulingPriority=0 CPUSchedulingPolicy=0 CPUSchedulingPriority=0 CPUAffinity= CPUAffinityFromNUMA=no NUMAPolicy=n/a NUMAMask= Failed to parse bus message: Invalid argument TimerSlackNSec=50000 CPUSchedulingResetOnFork=no NonBlocking=no StandardInput=null StandardInputData= StandardOutput=journal StandardError=inherit TTYReset=no TTYVHangup=no TTYVTDisallocate=no SyslogPriority=30 SyslogLevelPrefix=yes SyslogLevel=6 SyslogFacility=3 LogLevelMax=-1 LogRateLimitIntervalUSec=0 LogRateLimitBurst=0 SecureBits=0 ```
containerd
## containerd
containerd --version
``` containerd github.com/containerd/containerd v1.4.4 05f951a3781f4f2c1911b05e61c160e9c30eaa8e ```
systemctl show containerd
``` Failed to parse bus message: Invalid argument Type=notify Restart=always NotifyAccess=main RestartUSec=5s TimeoutStartUSec=1min 30s TimeoutStopUSec=1min 30s TimeoutAbortUSec=1min 30s RuntimeMaxUSec=infinity WatchdogUSec=0 WatchdogTimestampMonotonic=0 RootDirectoryStartOnly=no RemainAfterExit=no GuessMainPID=yes MainPID=720 ControlPID=0 FileDescriptorStoreMax=0 NFileDescriptorStore=0 StatusErrno=0 Result=success ReloadResult=success CleanResult=success UID=[not set] GID=[not set] NRestarts=0 OOMPolicy=continue ExecMainStartTimestamp=Wed 2021-03-31 11:18:16 UTC ExecMainStartTimestampMonotonic=74177404 ExecMainExitTimestampMonotonic=0 ExecMainPID=720 ExecMainCode=0 ExecMainStatus=0 ExecStartPre={ path=/sbin/modprobe ; argv[]=/sbin/modprobe overlay ; ignore_errors=yes ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 } ExecStartPreEx={ path=/sbin/modprobe ; argv[]=/sbin/modprobe overlay ; flags=ignore-failure ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 } ExecStart={ path=/usr/local/bin/containerd ; argv[]=/usr/local/bin/containerd ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 } ExecStartEx={ path=/usr/local/bin/containerd ; argv[]=/usr/local/bin/containerd ; flags= ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 } Slice=system.slice ControlGroup=/system.slice/containerd.service MemoryCurrent=1653202944 CPUUsageNSec=[not set] EffectiveCPUs= EffectiveMemoryNodes= TasksCurrent=161 IPIngressBytes=[no data] IPIngressPackets=[no data] IPEgressBytes=[no data] IPEgressPackets=[no data] IOReadBytes=18446744073709551615 IOReadOperations=18446744073709551615 IOWriteBytes=18446744073709551615 IOWriteOperations=18446744073709551615 Delegate=yes DelegateControllers=cpu cpuacct cpuset io blkio memory devices pids bpf-firewall bpf-devices CPUAccounting=no CPUWeight=[not set] StartupCPUWeight=[not set] CPUShares=[not set] StartupCPUShares=[not set] CPUQuotaPerSecUSec=infinity CPUQuotaPeriodUSec=infinity AllowedCPUs= AllowedMemoryNodes= IOAccounting=no IOWeight=[not set] StartupIOWeight=[not set] BlockIOAccounting=no BlockIOWeight=[not set] StartupBlockIOWeight=[not set] MemoryAccounting=yes DefaultMemoryLow=0 DefaultMemoryMin=0 MemoryMin=0 MemoryLow=0 MemoryHigh=infinity MemoryMax=infinity MemorySwapMax=infinity MemoryLimit=infinity DevicePolicy=auto TasksAccounting=yes TasksMax=infinity IPAccounting=no UMask=0022 LimitCPU=infinity LimitCPUSoft=infinity LimitFSIZE=infinity LimitFSIZESoft=infinity LimitDATA=infinity LimitDATASoft=infinity LimitSTACK=infinity LimitSTACKSoft=8388608 LimitCORE=infinity LimitCORESoft=infinity LimitRSS=infinity LimitRSSSoft=infinity LimitNOFILE=1048576 LimitNOFILESoft=1048576 LimitAS=infinity LimitASSoft=infinity LimitNPROC=infinity LimitNPROCSoft=infinity LimitMEMLOCK=65536 LimitMEMLOCKSoft=65536 LimitLOCKS=infinity LimitLOCKSSoft=infinity LimitSIGPENDING=31661 LimitSIGPENDINGSoft=31661 LimitMSGQUEUE=819200 LimitMSGQUEUESoft=819200 LimitNICE=0 LimitNICESoft=0 LimitRTPRIO=0 LimitRTPRIOSoft=0 LimitRTTIME=infinity LimitRTTIMESoft=infinity OOMScoreAdjust=-999 Nice=0 IOSchedulingClass=0 IOSchedulingPriority=0 CPUSchedulingPolicy=0 CPUSchedulingPriority=0 CPUAffinity= CPUAffinityFromNUMA=no NUMAPolicy=n/a NUMAMask= TimerSlackNSec=50000 CPUSchedulingResetOnFork=no NonBlocking=no StandardInput=null StandardInputData= StandardOutput=journal StandardError=inherit TTYReset=no TTYVHangup=no TTYVTDisallocate=no SyslogPriority=30 SyslogLevelPrefix=yes SyslogLevel=6 SyslogFacility=3 LogLevelMax=-1 LogRateLimitIntervalUSec=0 LogRateLimitBurst=0 SecureBits=0 ```
cat /etc/containerd/config.toml
```toml [plugins.cri.containerd] no_pivot = false [plugins.cri.containerd.runtimes] [plugins.cri.containerd.runtimes.runc] runtime_type = "io.containerd.runc.v2" [plugins.cri.containerd.runtimes.runc.options] NoPivotRoot = false NoNewKeyring = false ShimCgroup = "" IoUid = 0 IoGid = 0 BinaryName = "runc" Root = "" CriuPath = "" SystemdCgroup = false [plugins.cri.containerd.untrusted] runtime_type = "io.containerd.runsc.v1" [plugins.cri.containerd.runtimes.runsc] runtime_type = "io.containerd.runsc.v1" [plugins.cri.containerd.runtimes.runsc.options] NoPivotRoot = false NoNewKeyring = false ShimCgroup = "" IoUid = 0 IoGid = 0 BinaryName = "/usr/local/bin/runsc" Root = "" CriuPath = "" SystemdCgroup = false [plugins.cri.containerd.runtimes.kata] runtime_type = "io.containerd.kata.v2" ```
Packages
# Packages Have `dpkg`
No `rpm`
---
dpkg -l|egrep "(cc-oci-runtime|cc-runtime|runv|kata-runtime|kata-ksm-throttler|kata-containers-image|linux-container|qemu-)"
``` ```
Description of problem
Installed kata via the static binaries:
Configured with containerd, and try running a pod, without success:
Expected result
The pod to be running
Actual result
Further information
It worked on kernel 5.4 on Ubuntu 20.04 but not on 5.8 (available in Ubuntu repos) (
5.8.0-45-generic)