Does the kernel in kata containers support Page Cache and Dirty Page Writeback?

Show kata-collect-data.sh details

# Meta details Running `kata-collect-data.sh` version `1.12.0-rc0 (commit )` at `2020-12-07.19:02:13.348148264+0800`. --- Runtime is `/usr/bin/kata-runtime`. # `kata-env` Output of "`/usr/bin/kata-runtime kata-env`": ```toml [Meta] Version = "1.0.24" [Runtime] Debug = false Trace = false DisableGuestSeccomp = true DisableNewNetNs = false SandboxCgroupOnly = false Path = "/usr/bin/kata-runtime" [Runtime.Version] OCI = "1.0.1-dev" [Runtime.Version.Version] Semver = "1.12.0-rc0" Major = 1 Minor = 12 Patch = 0 Commit = "" [Runtime.Config] Path = "/usr/share/defaults/kata-containers/configuration.toml" [Hypervisor] MachineType = "pc" Version = "QEMU emulator version 5.0.0\nCopyright (c) 2003-2020 Fabrice Bellard and the QEMU Project developers" Path = "/usr/bin/qemu-vanilla-system-x86_64" BlockDeviceDriver = "virtio-scsi" EntropySource = "/dev/urandom" SharedFS = "virtio-9p" VirtioFSDaemon = "/usr/bin/virtiofsd" Msize9p = 8192 MemorySlots = 10 PCIeRootPort = 0 HotplugVFIOOnRootBus = false Debug = false UseVSock = false [Image] Path = "/usr/share/kata-containers/kata-containers-image_clearlinux_1.12.0-rc0_agent_5cfb8ec960.img" [Kernel] Path = "/usr/share/kata-containers/vmlinuz-5.4.60.89-51.container" Parameters = "systemd.unit=kata-containers.target systemd.mask=systemd-networkd.service systemd.mask=systemd-networkd.socket scsi_mod.scan=none" [Initrd] Path = "" [Proxy] Type = "kataProxy" Path = "/usr/libexec/kata-containers/kata-proxy" Debug = false [Proxy.Version] Semver = "1.12.0-rc0-adde733" Major = 1 Minor = 12 Patch = 0 Commit = "<>" [Shim] Type = "kataShim" Path = "/usr/libexec/kata-containers/kata-shim" Debug = false [Shim.Version] Semver = "<>" Major = 0 Minor = 0 Patch = 0 Commit = "<>" [Agent] Type = "kata" Debug = false Trace = false TraceMode = "" TraceType = "" [Host] Kernel = "5.4.0-56-generic" Architecture = "amd64" VMContainerCapable = true SupportVSocks = true [Host.Distro] Name = "Ubuntu" Version = "18.04" [Host.CPU] Vendor = "GenuineIntel" Model = "Intel(R) Xeon(R) CPU E5-2630 v4 @ 2.20GHz" [Netmon] Path = "/usr/libexec/kata-containers/kata-netmon" Debug = false Enable = false [Netmon.Version] Semver = "1.12.0-rc0" Major = 1 Minor = 12 Patch = 0 Commit = "<>" ``` --- # Runtime config files ## Runtime default config files ``` /etc/kata-containers/configuration.toml /usr/share/defaults/kata-containers/configuration.toml ``` ## Runtime config file contents Config file `/etc/kata-containers/configuration.toml` not found Output of "`cat "/usr/share/defaults/kata-containers/configuration.toml"`": ```toml # Copyright (c) 2017-2019 Intel Corporation # # SPDX-License-Identifier: Apache-2.0 # # XXX: WARNING: this file is auto-generated. # XXX: # XXX: Source file: "cli/config/configuration-qemu.toml.in" # XXX: Project: # XXX: Name: Kata Containers # XXX: Type: kata [hypervisor.qemu] path = "/usr/bin/qemu-vanilla-system-x86_64" kernel = "/usr/share/kata-containers/vmlinuz.container" image = "/usr/share/kata-containers/kata-containers.img" machine_type = "pc" # Optional space-separated list of options to pass to the guest kernel. # For example, use `kernel_params = "vsyscall=emulate"` if you are having # trouble running pre-2.15 glibc. # # WARNING: - any parameter specified here will take priority over the default # parameter value of the same name used to start the virtual machine. # Do not set values here unless you understand the impact of doing so as you # may stop the virtual machine from booting. # To see the list of default parameters, enable hypervisor debug, create a # container and look for 'default-kernel-parameters' log entries. kernel_params = "" # Path to the firmware. # If you want that qemu uses the default firmware leave this option empty firmware = "" # Machine accelerators # comma-separated list of machine accelerators to pass to the hypervisor. # For example, `machine_accelerators = "nosmm,nosmbus,nosata,nopit,static-prt,nofw"` machine_accelerators="" # CPU features # comma-separated list of cpu features to pass to the cpu # For example, `cpu_features = "pmu=off,vmx=off" cpu_features="pmu=off" # Default number of vCPUs per SB/VM: # unspecified or 0 --> will be set to 1 # < 0 --> will be set to the actual number of physical cores # > 0 <= number of physical cores --> will be set to the specified number # > number of physical cores --> will be set to the actual number of physical cores default_vcpus = 1 # Default maximum number of vCPUs per SB/VM: # unspecified or == 0 --> will be set to the actual number of physical cores or to the maximum number # of vCPUs supported by KVM if that number is exceeded # > 0 <= number of physical cores --> will be set to the specified number # > number of physical cores --> will be set to the actual number of physical cores or to the maximum number # of vCPUs supported by KVM if that number is exceeded # WARNING: Depending of the architecture, the maximum number of vCPUs supported by KVM is used when # the actual number of physical cores is greater than it. # WARNING: Be aware that this value impacts the virtual machine's memory footprint and CPU # the hotplug functionality. For example, `default_maxvcpus = 240` specifies that until 240 vCPUs # can be added to a SB/VM, but the memory footprint will be big. Another example, with # `default_maxvcpus = 8` the memory footprint will be small, but 8 will be the maximum number of # vCPUs supported by the SB/VM. In general, we recommend that you do not edit this variable, # unless you know what are you doing. # NOTICE: on arm platform with gicv2 interrupt controller, set it to 8. default_maxvcpus = 0 # Bridges can be used to hot plug devices. # Limitations: # * Currently only pci bridges are supported # * Until 30 devices per bridge can be hot plugged. # * Until 5 PCI bridges can be cold plugged per VM. # This limitation could be a bug in qemu or in the kernel # Default number of bridges per SB/VM: # unspecified or 0 --> will be set to 1 # > 1 <= 5 --> will be set to the specified number # > 5 --> will be set to 5 default_bridges = 1 # Default memory size in MiB for SB/VM. # If unspecified then it will be set 2048 MiB. default_memory = 2048 # # Default memory slots per SB/VM. # If unspecified then it will be set 10. # This is will determine the times that memory will be hotadded to sandbox/VM. #memory_slots = 10 # The size in MiB will be plused to max memory of hypervisor. # It is the memory address space for the NVDIMM devie. # If set block storage driver (block_device_driver) to "nvdimm", # should set memory_offset to the size of block device. # Default 0 #memory_offset = 0 # Specifies virtio-mem will be enabled or not. # Please note that this option should be used with the command # "echo 1 > /proc/sys/vm/overcommit_memory". # Default false #enable_virtio_mem = true # Disable block device from being used for a container's rootfs. # In case of a storage driver like devicemapper where a container's # root file system is backed by a block device, the block device is passed # directly to the hypervisor for performance reasons. # This flag prevents the block device from being passed to the hypervisor, # 9pfs is used instead to pass the rootfs. disable_block_device_use = false # Shared file system type: # - virtio-9p (default) # - virtio-fs shared_fs = "virtio-9p" # Path to vhost-user-fs daemon. virtio_fs_daemon = "/usr/bin/virtiofsd" # Default size of DAX cache in MiB virtio_fs_cache_size = 0 # Extra args for virtiofsd daemon # # Format example: # ["-o", "arg1=xxx,arg2", "-o", "hello world", "--arg3=yyy"] # # see `virtiofsd -h` for possible options. virtio_fs_extra_args = [] # Cache mode: # # - none # Metadata, data, and pathname lookup are not cached in guest. They are # always fetched from host and any changes are immediately pushed to host. # # - auto # Metadata and pathname lookup cache expires after a configured amount of # time (default is 1 second). Data is cached while the file is open (close # to open consistency). # # - always # Metadata, data, and pathname lookup are cached in guest and never expire. virtio_fs_cache = "auto" # Block storage driver to be used for the hypervisor in case the container # rootfs is backed by a block device. This is virtio-scsi, virtio-blk # or nvdimm. block_device_driver = "virtio-scsi" # Specifies cache-related options will be set to block devices or not. # Default false #block_device_cache_set = true # Specifies cache-related options for block devices. # Denotes whether use of O_DIRECT (bypass the host page cache) is enabled. # Default false #block_device_cache_direct = true # Specifies cache-related options for block devices. # Denotes whether flush requests for the device are ignored. # Default false #block_device_cache_noflush = true # Enable iothreads (data-plane) to be used. This causes IO to be # handled in a separate IO thread. This is currently only implemented # for SCSI. # enable_iothreads = false # Enable pre allocation of VM RAM, default false # Enabling this will result in lower container density # as all of the memory will be allocated and locked # This is useful when you want to reserve all the memory # upfront or in the cases where you want memory latencies # to be very predictable # Default false #enable_mem_prealloc = true # Enable huge pages for VM RAM, default false # Enabling this will result in the VM memory # being allocated using huge pages. # This is useful when you want to use vhost-user network # stacks within the container. This will automatically # result in memory pre allocation #enable_hugepages = true # Enable vhost-user storage device, default false # Enabling this will result in some Linux reserved block type # major range 240-254 being chosen to represent vhost-user devices. enable_vhost_user_store = false # The base directory specifically used for vhost-user devices. # Its sub-path "block" is used for block devices; "block/sockets" is # where we expect vhost-user sockets to live; "block/devices" is where # simulated block device nodes for vhost-user devices to live. vhost_user_store_path = "/var/run/kata-containers/vhost-user" # Enable vIOMMU, default false # Enabling this will result in the VM having a vIOMMU device # This will also add the following options to the kernel's # command line: intel_iommu=on,iommu=pt #enable_iommu = true # Enable IOMMU_PLATFORM, default false # Enabling this will result in the VM device having iommu_platform=on set #enable_iommu_platform = true # Enable file based guest memory support. The default is an empty string which # will disable this feature. In the case of virtio-fs, this is enabled # automatically and '/dev/shm' is used as the backing folder. # This option will be ignored if VM templating is enabled. #file_mem_backend = "" # Enable swap of vm memory. Default false. # The behaviour is undefined if mem_prealloc is also set to true #enable_swap = true # This option changes the default hypervisor and kernel parameters # to enable debug output where available. This extra output is added # to the proxy logs, but only when proxy debug is also enabled. # # Default false #enable_debug = true # Disable the customizations done in the runtime when it detects # that it is running on top a VMM. This will result in the runtime # behaving as it would when running on bare metal. # #disable_nesting_checks = true # This is the msize used for 9p shares. It is the number of bytes # used for 9p packet payload. #msize_9p = 8192 # If true and vsocks are supported, use vsocks to communicate directly # with the agent and no proxy is started, otherwise use unix # sockets and start a proxy to communicate with the agent. # Default false #use_vsock = true # If false and nvdimm is supported, use nvdimm device to plug guest image. # Otherwise virtio-block device is used. # Default is false #disable_image_nvdimm = true # VFIO devices are hotplugged on a bridge by default. # Enable hotplugging on root bus. This may be required for devices with # a large PCI bar, as this is a current limitation with hotplugging on # a bridge. This value is valid for "pc" machine type. # Default false #hotplug_vfio_on_root_bus = true # Before hot plugging a PCIe device, you need to add a pcie_root_port device. # Use this parameter when using some large PCI bar devices, such as Nvidia GPU # The value means the number of pcie_root_port # This value is valid when hotplug_vfio_on_root_bus is true and machine_type is "q35" # Default 0 #pcie_root_port = 2 # If vhost-net backend for virtio-net is not desired, set to true. Default is false, which trades off # security (vhost-net runs ring0) for network I/O performance. #disable_vhost_net = true # # Default entropy source. # The path to a host source of entropy (including a real hardware RNG) # /dev/urandom and /dev/random are two main options. # Be aware that /dev/random is a blocking source of entropy. If the host # runs out of entropy, the VMs boot time will increase leading to get startup # timeouts. # The source of entropy /dev/urandom is non-blocking and provides a # generally acceptable source of entropy. It should work well for pretty much # all practical purposes. #entropy_source= "/dev/urandom" # Path to OCI hook binaries in the *guest rootfs*. # This does not affect host-side hooks which must instead be added to # the OCI spec passed to the runtime. # # You can create a rootfs with hooks by customizing the osbuilder scripts: # https://github.com/kata-containers/osbuilder # # Hooks must be stored in a subdirectory of guest_hook_path according to their # hook type, i.e. "guest_hook_path/{prestart,postart,poststop}". # The agent will scan these directories for executable files and add them, in # lexicographical order, to the lifecycle of the guest container. # Hooks are executed in the runtime namespace of the guest. See the official documentation: # https://github.com/opencontainers/runtime-spec/blob/v1.0.1/config.md#posix-platform-hooks # Warnings will be logged if any error is encountered will scanning for hooks, # but it will not abort container execution. #guest_hook_path = "/usr/share/oci/hooks" [factory] # VM templating support. Once enabled, new VMs are created from template # using vm cloning. They will share the same initial kernel, initramfs and # agent memory by mapping it readonly. It helps speeding up new container # creation and saves a lot of memory if there are many kata containers running # on the same host. # # When disabled, new VMs are created from scratch. # # Note: Requires "initrd=" to be set ("image=" is not supported). # # Default false #enable_template = true # Specifies the path of template. # # Default "/run/vc/vm/template" #template_path = "/run/vc/vm/template" # The number of caches of VMCache: # unspecified or == 0 --> VMCache is disabled # > 0 --> will be set to the specified number # # VMCache is a function that creates VMs as caches before using it. # It helps speed up new container creation. # The function consists of a server and some clients communicating # through Unix socket. The protocol is gRPC in protocols/cache/cache.proto. # The VMCache server will create some VMs and cache them by factory cache. # It will convert the VM to gRPC format and transport it when gets # requestion from clients. # Factory grpccache is the VMCache client. It will request gRPC format # VM and convert it back to a VM. If VMCache function is enabled, # kata-runtime will request VM from factory grpccache when it creates # a new sandbox. # # Default 0 #vm_cache_number = 0 # Specify the address of the Unix socket that is used by VMCache. # # Default /var/run/kata-containers/cache.sock #vm_cache_endpoint = "/var/run/kata-containers/cache.sock" [proxy.kata] path = "/usr/libexec/kata-containers/kata-proxy" # If enabled, proxy messages will be sent to the system log # (default: disabled) #enable_debug = true [shim.kata] path = "/usr/libexec/kata-containers/kata-shim" # If enabled, shim messages will be sent to the system log # (default: disabled) #enable_debug = true # If enabled, the shim will create opentracing.io traces and spans. # (See https://www.jaegertracing.io/docs/getting-started). # # Note: By default, the shim runs in a separate network namespace. Therefore, # to allow it to send trace details to the Jaeger agent running on the host, # it is necessary to set 'disable_new_netns=true' so that it runs in the host # network namespace. # # (default: disabled) #enable_tracing = true [agent.kata] # If enabled, make the agent display debug-level messages. # (default: disabled) #enable_debug = true # Enable agent tracing. # # If enabled, the default trace mode is "dynamic" and the # default trace type is "isolated". The trace mode and type are set # explicity with the `trace_type=` and `trace_mode=` options. # # Notes: # # - Tracing is ONLY enabled when `enable_tracing` is set: explicitly # setting `trace_mode=` and/or `trace_type=` without setting `enable_tracing` # will NOT activate agent tracing. # # - See https://github.com/kata-containers/agent/blob/master/TRACING.md for # full details. # # (default: disabled) #enable_tracing = true # #trace_mode = "dynamic" #trace_type = "isolated" # Comma separated list of kernel modules and their parameters. # These modules will be loaded in the guest kernel using modprobe(8). # The following example can be used to load two kernel modules with parameters # - kernel_modules=["e1000e InterruptThrottleRate=3000,3000,3000 EEE=1", "i915 enable_ppgtt=0"] # The first word is considered as the module name and the rest as its parameters. # Container will not be started when: # * A kernel module is specified and the modprobe command is not installed in the guest # or it fails loading the module. # * The module is not available in the guest or it doesn't met the guest kernel # requirements, like architecture and version. # kernel_modules=[] [netmon] # If enabled, the network monitoring process gets started when the # sandbox is created. This allows for the detection of some additional # network being added to the existing network namespace, after the # sandbox has been created. # (default: disabled) #enable_netmon = true # Specify the path to the netmon binary. path = "/usr/libexec/kata-containers/kata-netmon" # If enabled, netmon messages will be sent to the system log # (default: disabled) #enable_debug = true [runtime] # If enabled, the runtime will log additional debug messages to the # system log # (default: disabled) #enable_debug = true # # Internetworking model # Determines how the VM should be connected to the # the container network interface # Options: # # - macvtap # Used when the Container network interface can be bridged using # macvtap. # # - none # Used when customize network. Only creates a tap device. No veth pair. # # - tcfilter # Uses tc filter rules to redirect traffic from the network interface # provided by plugin to a tap interface connected to the VM. # internetworking_model="tcfilter" # disable guest seccomp # Determines whether container seccomp profiles are passed to the virtual # machine and applied by the kata agent. If set to true, seccomp is not applied # within the guest # (default: true) disable_guest_seccomp=true # If enabled, the runtime will create opentracing.io traces and spans. # (See https://www.jaegertracing.io/docs/getting-started). # (default: disabled) #enable_tracing = true # If enabled, the runtime will not create a network namespace for shim and hypervisor processes. # This option may have some potential impacts to your host. It should only be used when you know what you're doing. # `disable_new_netns` conflicts with `enable_netmon` # `disable_new_netns` conflicts with `internetworking_model=tcfilter` and `internetworking_model=macvtap`. It works only # with `internetworking_model=none`. The tap device will be in the host network namespace and can connect to a bridge # (like OVS) directly. # If you are using docker, `disable_new_netns` only works with `docker run --net=none` # (default: false) #disable_new_netns = true # if enabled, the runtime will add all the kata processes inside one dedicated cgroup. # The container cgroups in the host are not created, just one single cgroup per sandbox. # The runtime caller is free to restrict or collect cgroup stats of the overall Kata sandbox. # The sandbox cgroup path is the parent cgroup of a container with the PodSandbox annotation. # The sandbox cgroup is constrained if there is no container type annotation. # See: https://godoc.org/github.com/kata-containers/runtime/virtcontainers#ContainerType sandbox_cgroup_only=false # Enabled experimental feature list, format: ["a", "b"]. # Experimental features are features not stable enough for production, # they may break compatibility, and are prepared for a big version bump. # Supported experimental features: # (default: []) experimental=[] # If enabled, containers are allowed to join the pid namespace of the agent # when the env variable KATA_AGENT_PIDNS is set for a container. # Use this with caution and only when required, as this option allows the container # to access the agent process. It is recommended to enable this option # only in debug scenarios and with containers with lowered priveleges. #enable_agent_pidns = true ``` --- # KSM throttler ## version Output of "`/usr/libexec/kata-ksm-throttler/kata-ksm-throttler --version`": ``` kata-ksm-throttler version 1.12.0-rc0-23f5da9 ``` ## systemd service # Image details ```yaml --- osbuilder: url: "https://github.com/kata-containers/osbuilder" version: "unknown" rootfs-creation-time: "2020-10-22T16:04:30.991754775+0000Z" description: "osbuilder rootfs" file-format-version: "0.0.2" architecture: "x86_64" base-distro: name: "Clear" version: "33850" packages: default: - "chrony" - "iptables-bin" - "kmod-bin" - "libudev0-shim" - "systemd" - "util-linux-bin" extra: agent: url: "https://github.com/kata-containers/agent" name: "kata-agent" version: "1.12.0-rc0-5cfb8ec9601a2e397bcb8f7e81c09f56cf6e7a17" agent-is-init-daemon: "no" ``` --- # Initrd details No initrd --- # Logfiles ## Runtime logs Recent runtime problems found in system journal: ``` time="2020-12-07T17:53:36.823232772+08:00" level=warning msg="failed to cleanup netns" arch=amd64 command=create container=53c7c475e7048a32f87ef43e5c52f8e0846c49e3b4deed97cb282685684a2691 error="failed to get netns /var/run/netns/cnitest-24486b2f-3a92-634b-3893-95b9530d8b33: failed to Statfs \"/var/run/netns/cnitest-24486b2f-3a92-634b-3893-95b9530d8b33\": no such file or directory" name=kata-runtime path=/var/run/netns/cnitest-24486b2f-3a92-634b-3893-95b9530d8b33 pid=39067 source=katautils time="2020-12-07T17:53:36.823335812+08:00" level=error msg="Unable to hotplug 262144 MiB memory, the SB has 2048 MiB and the maximum amount is 257723 MiB" arch=amd64 command=create container=53c7c475e7048a32f87ef43e5c52f8e0846c49e3b4deed97cb282685684a2691 name=kata-runtime pid=39067 source=runtime time="2020-12-07T17:54:44.940485827+08:00" level=info msg="sanner return error: read unix @->/run/vc/vm/11183ca8789b19f55c5a130b7b5382a60601dcf8b1771c9ad5f84520eade90f9/qmp.sock: use of closed network connection" arch=amd64 command=create container=11183ca8789b19f55c5a130b7b5382a60601dcf8b1771c9ad5f84520eade90f9 name=kata-runtime pid=39678 source=virtcontainers subsystem=qmp time="2020-12-07T17:54:46.078511115+08:00" level=info msg="sanner return error: " arch=amd64 command=create container=11183ca8789b19f55c5a130b7b5382a60601dcf8b1771c9ad5f84520eade90f9 name=kata-runtime pid=39678 source=virtcontainers subsystem=qmp time="2020-12-07T17:54:46.180794522+08:00" level=error msg="Could not read qemu pid file" arch=amd64 command=create container=11183ca8789b19f55c5a130b7b5382a60601dcf8b1771c9ad5f84520eade90f9 error="open /run/vc/vm/11183ca8789b19f55c5a130b7b5382a60601dcf8b1771c9ad5f84520eade90f9/pid: no such file or directory" name=kata-runtime pid=39678 source=virtcontainers subsystem=qemu time="2020-12-07T17:54:46.181671496+08:00" level=warning msg="sandbox cgroups path is empty" arch=amd64 command=create container=11183ca8789b19f55c5a130b7b5382a60601dcf8b1771c9ad5f84520eade90f9 name=kata-runtime pid=39678 sandbox=11183ca8789b19f55c5a130b7b5382a60601dcf8b1771c9ad5f84520eade90f9 source=virtcontainers subsystem=sandbox time="2020-12-07T17:54:46.463356256+08:00" level=error msg="failed to cleanup vm path /run/kata-containers/shared/sandboxes/11183ca8789b19f55c5a130b7b5382a60601dcf8b1771c9ad5f84520eade90f9" arch=amd64 command=create container=11183ca8789b19f55c5a130b7b5382a60601dcf8b1771c9ad5f84520eade90f9 error="unlinkat /run/kata-containers/shared/sandboxes/11183ca8789b19f55c5a130b7b5382a60601dcf8b1771c9ad5f84520eade90f9/mounts/11183ca8789b19f55c5a130b7b5382a60601dcf8b1771c9ad5f84520eade90f9-394bcd80b34bf642-hosts: device or resource busy" name=kata-runtime pid=39678 source=virtcontainers subsystem=kata_agent time="2020-12-07T17:54:46.46357822+08:00" level=warning msg="failed to cleanup netns" arch=amd64 command=create container=11183ca8789b19f55c5a130b7b5382a60601dcf8b1771c9ad5f84520eade90f9 error="failed to get netns /var/run/netns/cnitest-7dfdc905-17da-98a1-f0a5-88e4ad206b94: failed to Statfs \"/var/run/netns/cnitest-7dfdc905-17da-98a1-f0a5-88e4ad206b94\": no such file or directory" name=kata-runtime path=/var/run/netns/cnitest-7dfdc905-17da-98a1-f0a5-88e4ad206b94 pid=39678 source=katautils time="2020-12-07T17:54:46.463641438+08:00" level=error msg="Unable to hotplug 257792 MiB memory, the SB has 2048 MiB and the maximum amount is 257723 MiB" arch=amd64 command=create container=11183ca8789b19f55c5a130b7b5382a60601dcf8b1771c9ad5f84520eade90f9 name=kata-runtime pid=39678 source=runtime time="2020-12-07T17:55:25.126157422+08:00" level=info msg="sanner return error: read unix @->/run/vc/vm/87faf522948de0ab94d655b77b026002ab00941e9d4bbd5c7b0f8b6f660e3cbb/qmp.sock: use of closed network connection" arch=amd64 command=create container=87faf522948de0ab94d655b77b026002ab00941e9d4bbd5c7b0f8b6f660e3cbb name=kata-runtime pid=40223 source=virtcontainers subsystem=qmp time="2020-12-07T17:55:26.310443581+08:00" level=info msg="sanner return error: " arch=amd64 command=create container=87faf522948de0ab94d655b77b026002ab00941e9d4bbd5c7b0f8b6f660e3cbb name=kata-runtime pid=40223 source=virtcontainers subsystem=qmp time="2020-12-07T17:55:26.433668471+08:00" level=error msg="Could not read qemu pid file" arch=amd64 command=create container=87faf522948de0ab94d655b77b026002ab00941e9d4bbd5c7b0f8b6f660e3cbb error="open /run/vc/vm/87faf522948de0ab94d655b77b026002ab00941e9d4bbd5c7b0f8b6f660e3cbb/pid: no such file or directory" name=kata-runtime pid=40223 source=virtcontainers subsystem=qemu time="2020-12-07T17:55:26.435108519+08:00" level=warning msg="sandbox cgroups path is empty" arch=amd64 command=create container=87faf522948de0ab94d655b77b026002ab00941e9d4bbd5c7b0f8b6f660e3cbb name=kata-runtime pid=40223 sandbox=87faf522948de0ab94d655b77b026002ab00941e9d4bbd5c7b0f8b6f660e3cbb source=virtcontainers subsystem=sandbox time="2020-12-07T17:55:26.671990784+08:00" level=error msg="failed to cleanup vm path /run/kata-containers/shared/sandboxes/87faf522948de0ab94d655b77b026002ab00941e9d4bbd5c7b0f8b6f660e3cbb" arch=amd64 command=create container=87faf522948de0ab94d655b77b026002ab00941e9d4bbd5c7b0f8b6f660e3cbb error="unlinkat /run/kata-containers/shared/sandboxes/87faf522948de0ab94d655b77b026002ab00941e9d4bbd5c7b0f8b6f660e3cbb/mounts/87faf522948de0ab94d655b77b026002ab00941e9d4bbd5c7b0f8b6f660e3cbb-7c87aec746fdfaa1-hosts: device or resource busy" name=kata-runtime pid=40223 source=virtcontainers subsystem=kata_agent time="2020-12-07T17:55:26.672202646+08:00" level=warning msg="failed to cleanup netns" arch=amd64 command=create container=87faf522948de0ab94d655b77b026002ab00941e9d4bbd5c7b0f8b6f660e3cbb error="failed to get netns /var/run/netns/cnitest-4f5cbd75-5dfd-fae5-5bb6-76a791a3bb13: failed to Statfs \"/var/run/netns/cnitest-4f5cbd75-5dfd-fae5-5bb6-76a791a3bb13\": no such file or directory" name=kata-runtime path=/var/run/netns/cnitest-4f5cbd75-5dfd-fae5-5bb6-76a791a3bb13 pid=40223 source=katautils time="2020-12-07T17:55:26.672266212+08:00" level=error msg="Unable to hotplug 256000 MiB memory, the SB has 2048 MiB and the maximum amount is 257723 MiB" arch=amd64 command=create container=87faf522948de0ab94d655b77b026002ab00941e9d4bbd5c7b0f8b6f660e3cbb name=kata-runtime pid=40223 source=runtime time="2020-12-07T17:56:21.959127543+08:00" level=info msg="sanner return error: read unix @->/run/vc/vm/65ebb989192736084cd540aa3f11674fdf582fbc530b9992a6ed5dafaaada6f3/qmp.sock: use of closed network connection" arch=amd64 command=create container=65ebb989192736084cd540aa3f11674fdf582fbc530b9992a6ed5dafaaada6f3 name=kata-runtime pid=40788 source=virtcontainers subsystem=qmp time="2020-12-07T17:56:24.257081599+08:00" level=warning msg="sandbox's cgroup won't be updated: cgroup path is empty" arch=amd64 command=create container=65ebb989192736084cd540aa3f11674fdf582fbc530b9992a6ed5dafaaada6f3 name=kata-runtime pid=40788 sandbox=65ebb989192736084cd540aa3f11674fdf582fbc530b9992a6ed5dafaaada6f3 source=virtcontainers subsystem=sandbox time="2020-12-07T17:56:24.259835109+08:00" level=info msg="sanner return error: read unix @->/run/vc/vm/65ebb989192736084cd540aa3f11674fdf582fbc530b9992a6ed5dafaaada6f3/qmp.sock: use of closed network connection" arch=amd64 command=create container=65ebb989192736084cd540aa3f11674fdf582fbc530b9992a6ed5dafaaada6f3 name=kata-runtime pid=40788 source=virtcontainers subsystem=qmp time="2020-12-07T17:56:24.539423666+08:00" level=error msg="Failed to start container" arch=amd64 command=start container=65ebb989192736084cd540aa3f11674fdf582fbc530b9992a6ed5dafaaada6f3 error="rpc error: code = Unavailable desc = transport is closing" name=kata-runtime pid=370 sandbox=65ebb989192736084cd540aa3f11674fdf582fbc530b9992a6ed5dafaaada6f3 source=virtcontainers subsystem=container time="2020-12-07T17:56:24.641582033+08:00" level=warning msg="Unable to connect to unix socket (/run/vc/vm/65ebb989192736084cd540aa3f11674fdf582fbc530b9992a6ed5dafaaada6f3/qmp.sock): dial unix /run/vc/vm/65ebb989192736084cd540aa3f11674fdf582fbc530b9992a6ed5dafaaada6f3/qmp.sock: connect: no such file or directory" arch=amd64 command=start container=65ebb989192736084cd540aa3f11674fdf582fbc530b9992a6ed5dafaaada6f3 name=kata-runtime pid=370 sandbox=65ebb989192736084cd540aa3f11674fdf582fbc530b9992a6ed5dafaaada6f3 source=virtcontainers subsystem=qmp time="2020-12-07T17:56:24.641738004+08:00" level=error msg="Failed to connect to QEMU instance" arch=amd64 command=start container=65ebb989192736084cd540aa3f11674fdf582fbc530b9992a6ed5dafaaada6f3 error="dial unix /run/vc/vm/65ebb989192736084cd540aa3f11674fdf582fbc530b9992a6ed5dafaaada6f3/qmp.sock: connect: no such file or directory" name=kata-runtime pid=370 sandbox=65ebb989192736084cd540aa3f11674fdf582fbc530b9992a6ed5dafaaada6f3 source=virtcontainers subsystem=qemu time="2020-12-07T17:56:24.641846394+08:00" level=error msg="Failed to unplug block device" arch=amd64 command=start container=65ebb989192736084cd540aa3f11674fdf582fbc530b9992a6ed5dafaaada6f3 error="dial unix /run/vc/vm/65ebb989192736084cd540aa3f11674fdf582fbc530b9992a6ed5dafaaada6f3/qmp.sock: connect: no such file or directory" name=kata-runtime pid=370 sandbox=65ebb989192736084cd540aa3f11674fdf582fbc530b9992a6ed5dafaaada6f3 source=virtcontainers subsystem=device time="2020-12-07T17:56:24.642017747+08:00" level=warning msg="Could not remove container share dir" arch=amd64 command=start container=65ebb989192736084cd540aa3f11674fdf582fbc530b9992a6ed5dafaaada6f3 error="no such file or directory" name=kata-runtime pid=370 sandbox=65ebb989192736084cd540aa3f11674fdf582fbc530b9992a6ed5dafaaada6f3 share-dir=/run/kata-containers/shared/sandboxes/65ebb989192736084cd540aa3f11674fdf582fbc530b9992a6ed5dafaaada6f3/65ebb989192736084cd540aa3f11674fdf582fbc530b9992a6ed5dafaaada6f3 source=virtcontainers subsystem=container time="2020-12-07T17:56:24.642156861+08:00" level=error msg="Could not read qemu pid file" arch=amd64 command=start container=65ebb989192736084cd540aa3f11674fdf582fbc530b9992a6ed5dafaaada6f3 error="open /run/vc/vm/65ebb989192736084cd540aa3f11674fdf582fbc530b9992a6ed5dafaaada6f3/pid: no such file or directory" name=kata-runtime pid=370 sandbox=65ebb989192736084cd540aa3f11674fdf582fbc530b9992a6ed5dafaaada6f3 source=virtcontainers subsystem=qemu time="2020-12-07T17:56:24.643450203+08:00" level=error msg="Could not read qemu pid file" arch=amd64 command=start container=65ebb989192736084cd540aa3f11674fdf582fbc530b9992a6ed5dafaaada6f3 error="open /run/vc/vm/65ebb989192736084cd540aa3f11674fdf582fbc530b9992a6ed5dafaaada6f3/pid: no such file or directory" name=kata-runtime pid=370 sandbox=65ebb989192736084cd540aa3f11674fdf582fbc530b9992a6ed5dafaaada6f3 source=virtcontainers subsystem=qemu time="2020-12-07T17:56:24.644598701+08:00" level=error msg="rpc error: code = Unavailable desc = transport is closing" arch=amd64 command=start container=65ebb989192736084cd540aa3f11674fdf582fbc530b9992a6ed5dafaaada6f3 name=kata-runtime pid=370 sandbox=65ebb989192736084cd540aa3f11674fdf582fbc530b9992a6ed5dafaaada6f3 source=runtime time="2020-12-07T17:56:24.728994279+08:00" level=warning msg="Agent did not stop sandbox" arch=amd64 command=delete container=65ebb989192736084cd540aa3f11674fdf582fbc530b9992a6ed5dafaaada6f3 error="Proxy is not running" name=kata-runtime pid=416 sandbox=65ebb989192736084cd540aa3f11674fdf582fbc530b9992a6ed5dafaaada6f3 sandboxid=65ebb989192736084cd540aa3f11674fdf582fbc530b9992a6ed5dafaaada6f3 source=virtcontainers subsystem=sandbox time="2020-12-07T17:56:24.729144938+08:00" level=warning msg="Unable to connect to unix socket (/run/vc/vm/65ebb989192736084cd540aa3f11674fdf582fbc530b9992a6ed5dafaaada6f3/qmp.sock): dial unix /run/vc/vm/65ebb989192736084cd540aa3f11674fdf582fbc530b9992a6ed5dafaaada6f3/qmp.sock: connect: no such file or directory" arch=amd64 command=delete container=65ebb989192736084cd540aa3f11674fdf582fbc530b9992a6ed5dafaaada6f3 name=kata-runtime pid=416 sandbox=65ebb989192736084cd540aa3f11674fdf582fbc530b9992a6ed5dafaaada6f3 source=virtcontainers subsystem=qmp time="2020-12-07T17:56:24.729194531+08:00" level=error msg="Failed to connect to QEMU instance" arch=amd64 command=delete container=65ebb989192736084cd540aa3f11674fdf582fbc530b9992a6ed5dafaaada6f3 error="dial unix /run/vc/vm/65ebb989192736084cd540aa3f11674fdf582fbc530b9992a6ed5dafaaada6f3/qmp.sock: connect: no such file or directory" name=kata-runtime pid=416 sandbox=65ebb989192736084cd540aa3f11674fdf582fbc530b9992a6ed5dafaaada6f3 source=virtcontainers subsystem=qemu time="2020-12-07T17:56:24.729387015+08:00" level=error msg="dial unix /run/vc/vm/65ebb989192736084cd540aa3f11674fdf582fbc530b9992a6ed5dafaaada6f3/qmp.sock: connect: no such file or directory" arch=amd64 command=delete container=65ebb989192736084cd540aa3f11674fdf582fbc530b9992a6ed5dafaaada6f3 name=kata-runtime pid=416 sandbox=65ebb989192736084cd540aa3f11674fdf582fbc530b9992a6ed5dafaaada6f3 source=runtime time="2020-12-07T17:57:25.222254339+08:00" level=info msg="sanner return error: read unix @->/run/vc/vm/1f1838d24a7c388b28f8fbd1dac39bdd3468f64279f7b850701019dfbc872888/qmp.sock: use of closed network connection" arch=amd64 command=create container=1f1838d24a7c388b28f8fbd1dac39bdd3468f64279f7b850701019dfbc872888 name=kata-runtime pid=831 source=virtcontainers subsystem=qmp time="2020-12-07T17:57:27.075158183+08:00" level=warning msg="sandbox's cgroup won't be updated: cgroup path is empty" arch=amd64 command=create container=1f1838d24a7c388b28f8fbd1dac39bdd3468f64279f7b850701019dfbc872888 name=kata-runtime pid=831 sandbox=1f1838d24a7c388b28f8fbd1dac39bdd3468f64279f7b850701019dfbc872888 source=virtcontainers subsystem=sandbox time="2020-12-07T17:57:27.07773734+08:00" level=info msg="sanner return error: read unix @->/run/vc/vm/1f1838d24a7c388b28f8fbd1dac39bdd3468f64279f7b850701019dfbc872888/qmp.sock: use of closed network connection" arch=amd64 command=create container=1f1838d24a7c388b28f8fbd1dac39bdd3468f64279f7b850701019dfbc872888 name=kata-runtime pid=831 source=virtcontainers subsystem=qmp time="2020-12-07T17:57:27.331465824+08:00" level=error msg="Failed to start container" arch=amd64 command=start container=1f1838d24a7c388b28f8fbd1dac39bdd3468f64279f7b850701019dfbc872888 error="rpc error: code = Unavailable desc = transport is closing" name=kata-runtime pid=1068 sandbox=1f1838d24a7c388b28f8fbd1dac39bdd3468f64279f7b850701019dfbc872888 source=virtcontainers subsystem=container time="2020-12-07T17:57:47.373599657+08:00" level=warning msg="Unable to connect to unix socket (/run/vc/vm/1f1838d24a7c388b28f8fbd1dac39bdd3468f64279f7b850701019dfbc872888/qmp.sock): dial unix /run/vc/vm/1f1838d24a7c388b28f8fbd1dac39bdd3468f64279f7b850701019dfbc872888/qmp.sock: connect: no such file or directory" arch=amd64 command=start container=1f1838d24a7c388b28f8fbd1dac39bdd3468f64279f7b850701019dfbc872888 name=kata-runtime pid=1068 sandbox=1f1838d24a7c388b28f8fbd1dac39bdd3468f64279f7b850701019dfbc872888 source=virtcontainers subsystem=qmp time="2020-12-07T17:57:47.37370348+08:00" level=error msg="Failed to connect to QEMU instance" arch=amd64 command=start container=1f1838d24a7c388b28f8fbd1dac39bdd3468f64279f7b850701019dfbc872888 error="dial unix /run/vc/vm/1f1838d24a7c388b28f8fbd1dac39bdd3468f64279f7b850701019dfbc872888/qmp.sock: connect: no such file or directory" name=kata-runtime pid=1068 sandbox=1f1838d24a7c388b28f8fbd1dac39bdd3468f64279f7b850701019dfbc872888 source=virtcontainers subsystem=qemu time="2020-12-07T17:57:47.373789317+08:00" level=error msg="Failed to unplug block device" arch=amd64 command=start container=1f1838d24a7c388b28f8fbd1dac39bdd3468f64279f7b850701019dfbc872888 error="dial unix /run/vc/vm/1f1838d24a7c388b28f8fbd1dac39bdd3468f64279f7b850701019dfbc872888/qmp.sock: connect: no such file or directory" name=kata-runtime pid=1068 sandbox=1f1838d24a7c388b28f8fbd1dac39bdd3468f64279f7b850701019dfbc872888 source=virtcontainers subsystem=device time="2020-12-07T17:57:47.37388901+08:00" level=warning msg="Could not remove container share dir" arch=amd64 command=start container=1f1838d24a7c388b28f8fbd1dac39bdd3468f64279f7b850701019dfbc872888 error="no such file or directory" name=kata-runtime pid=1068 sandbox=1f1838d24a7c388b28f8fbd1dac39bdd3468f64279f7b850701019dfbc872888 share-dir=/run/kata-containers/shared/sandboxes/1f1838d24a7c388b28f8fbd1dac39bdd3468f64279f7b850701019dfbc872888/1f1838d24a7c388b28f8fbd1dac39bdd3468f64279f7b850701019dfbc872888 source=virtcontainers subsystem=container time="2020-12-07T17:57:47.374011287+08:00" level=error msg="Could not read qemu pid file" arch=amd64 command=start container=1f1838d24a7c388b28f8fbd1dac39bdd3468f64279f7b850701019dfbc872888 error="open /run/vc/vm/1f1838d24a7c388b28f8fbd1dac39bdd3468f64279f7b850701019dfbc872888/pid: no such file or directory" name=kata-runtime pid=1068 sandbox=1f1838d24a7c388b28f8fbd1dac39bdd3468f64279f7b850701019dfbc872888 source=virtcontainers subsystem=qemu time="2020-12-07T17:57:47.375344935+08:00" level=error msg="Could not read qemu pid file" arch=amd64 command=start container=1f1838d24a7c388b28f8fbd1dac39bdd3468f64279f7b850701019dfbc872888 error="open /run/vc/vm/1f1838d24a7c388b28f8fbd1dac39bdd3468f64279f7b850701019dfbc872888/pid: no such file or directory" name=kata-runtime pid=1068 sandbox=1f1838d24a7c388b28f8fbd1dac39bdd3468f64279f7b850701019dfbc872888 source=virtcontainers subsystem=qemu time="2020-12-07T17:57:47.376340077+08:00" level=error msg="Failed to stop container so stopping dangling shim" arch=amd64 command=start container=1f1838d24a7c388b28f8fbd1dac39bdd3468f64279f7b850701019dfbc872888 name=kata-runtime pid=1068 sandbox=1f1838d24a7c388b28f8fbd1dac39bdd3468f64279f7b850701019dfbc872888 source=virtcontainers subsystem=container time="2020-12-07T17:57:47.376708564+08:00" level=error msg="rpc error: code = Unavailable desc = transport is closing" arch=amd64 command=start container=1f1838d24a7c388b28f8fbd1dac39bdd3468f64279f7b850701019dfbc872888 name=kata-runtime pid=1068 sandbox=1f1838d24a7c388b28f8fbd1dac39bdd3468f64279f7b850701019dfbc872888 source=runtime time="2020-12-07T17:57:47.483378036+08:00" level=warning msg="Agent did not stop sandbox" arch=amd64 command=delete container=1f1838d24a7c388b28f8fbd1dac39bdd3468f64279f7b850701019dfbc872888 error="Proxy is not running" name=kata-runtime pid=1178 sandbox=1f1838d24a7c388b28f8fbd1dac39bdd3468f64279f7b850701019dfbc872888 sandboxid=1f1838d24a7c388b28f8fbd1dac39bdd3468f64279f7b850701019dfbc872888 source=virtcontainers subsystem=sandbox time="2020-12-07T17:57:47.483529558+08:00" level=warning msg="Unable to connect to unix socket (/run/vc/vm/1f1838d24a7c388b28f8fbd1dac39bdd3468f64279f7b850701019dfbc872888/qmp.sock): dial unix /run/vc/vm/1f1838d24a7c388b28f8fbd1dac39bdd3468f64279f7b850701019dfbc872888/qmp.sock: connect: no such file or directory" arch=amd64 command=delete container=1f1838d24a7c388b28f8fbd1dac39bdd3468f64279f7b850701019dfbc872888 name=kata-runtime pid=1178 sandbox=1f1838d24a7c388b28f8fbd1dac39bdd3468f64279f7b850701019dfbc872888 source=virtcontainers subsystem=qmp time="2020-12-07T17:57:47.483575271+08:00" level=error msg="Failed to connect to QEMU instance" arch=amd64 command=delete container=1f1838d24a7c388b28f8fbd1dac39bdd3468f64279f7b850701019dfbc872888 error="dial unix /run/vc/vm/1f1838d24a7c388b28f8fbd1dac39bdd3468f64279f7b850701019dfbc872888/qmp.sock: connect: no such file or directory" name=kata-runtime pid=1178 sandbox=1f1838d24a7c388b28f8fbd1dac39bdd3468f64279f7b850701019dfbc872888 source=virtcontainers subsystem=qemu time="2020-12-07T17:57:47.483771939+08:00" level=error msg="dial unix /run/vc/vm/1f1838d24a7c388b28f8fbd1dac39bdd3468f64279f7b850701019dfbc872888/qmp.sock: connect: no such file or directory" arch=amd64 command=delete container=1f1838d24a7c388b28f8fbd1dac39bdd3468f64279f7b850701019dfbc872888 name=kata-runtime pid=1178 sandbox=1f1838d24a7c388b28f8fbd1dac39bdd3468f64279f7b850701019dfbc872888 source=runtime time="2020-12-07T17:58:51.750710026+08:00" level=info msg="sanner return error: read unix @->/run/vc/vm/230bdfc371da4e547d214f972132ffe2788c659bdb088234a7826cf27b5ee55e/qmp.sock: use of closed network connection" arch=amd64 command=create container=230bdfc371da4e547d214f972132ffe2788c659bdb088234a7826cf27b5ee55e name=kata-runtime pid=1529 source=virtcontainers subsystem=qmp time="2020-12-07T17:58:52.951300619+08:00" level=warning msg="sandbox's cgroup won't be updated: cgroup path is empty" arch=amd64 command=create container=230bdfc371da4e547d214f972132ffe2788c659bdb088234a7826cf27b5ee55e name=kata-runtime pid=1529 sandbox=230bdfc371da4e547d214f972132ffe2788c659bdb088234a7826cf27b5ee55e source=virtcontainers subsystem=sandbox time="2020-12-07T17:58:52.95404748+08:00" level=info msg="sanner return error: read unix @->/run/vc/vm/230bdfc371da4e547d214f972132ffe2788c659bdb088234a7826cf27b5ee55e/qmp.sock: use of closed network connection" arch=amd64 command=create container=230bdfc371da4e547d214f972132ffe2788c659bdb088234a7826cf27b5ee55e name=kata-runtime pid=1529 source=virtcontainers subsystem=qmp ``` ## Proxy logs Recent proxy problems found in system journal: ``` time="2020-12-01T13:16:31.29559804+08:00" level=fatal msg="failed to handle exit signal" error="close unix @->/run/vc/vm/a8f9d2a964289f19e29ab4bc1ccbac6efba68227b0f8549d61efc93d1c2c9851/kata.sock: use of closed network connection" name=kata-proxy pid=6918 sandbox=a8f9d2a964289f19e29ab4bc1ccbac6efba68227b0f8549d61efc93d1c2c9851 source=proxy time="2020-12-01T13:32:43.789343585+08:00" level=fatal msg="failed to handle exit signal" error="close unix @->/run/vc/vm/3bd462e07d144834452ae71b576e2d3b40a593e5e47210823108342259715f35/kata.sock: use of closed network connection" name=kata-proxy pid=8309 sandbox=3bd462e07d144834452ae71b576e2d3b40a593e5e47210823108342259715f35 source=proxy time="2020-12-01T13:58:24.982819749+08:00" level=fatal msg="channel error" error="accept unix /run/vc/sbs/ff189f93118f11d52cca1f409d495958696b352aa5dbf7a4a1fd983e04ecd2f1/proxy.sock: use of closed network connection" name=kata-proxy pid=8825 sandbox=ff189f93118f11d52cca1f409d495958696b352aa5dbf7a4a1fd983e04ecd2f1 source=proxy time="2020-12-01T14:08:52.646314407+08:00" level=fatal msg="failed to handle exit signal" error="close unix @->/run/vc/vm/53b59b67cdd48f52bf53a2faf18a261ab8edcc100b5111fc52cd2f7161165680/kata.sock: use of closed network connection" name=kata-proxy pid=7502 sandbox=53b59b67cdd48f52bf53a2faf18a261ab8edcc100b5111fc52cd2f7161165680 source=proxy time="2020-12-01T14:08:56.244349144+08:00" level=fatal msg="failed to handle exit signal" error="close unix @->/run/vc/vm/6ae641e6cb47e4669a8b3fc31617c1d1766077c2fd81fb172d88c6c6f1c6e659/kata.sock: use of closed network connection" name=kata-proxy pid=7128 sandbox=6ae641e6cb47e4669a8b3fc31617c1d1766077c2fd81fb172d88c6c6f1c6e659 source=proxy time="2020-12-01T15:20:05.339956403+08:00" level=fatal msg="failed to handle exit signal" error="close unix @->/run/vc/vm/6c8a33420d5c63cb42dc3d2c0bff43f71a7170fe919636369ec2b27fd1184640/kata.sock: use of closed network connection" name=kata-proxy pid=10528 sandbox=6c8a33420d5c63cb42dc3d2c0bff43f71a7170fe919636369ec2b27fd1184640 source=proxy time="2020-12-01T21:17:33.144786096+08:00" level=fatal msg="failed to handle exit signal" error="close unix @->/run/vc/vm/2089215418d9a497e536a0c4d0b9e4348dde029b9a14066b65cfa774283c360e/kata.sock: use of closed network connection" name=kata-proxy pid=12212 sandbox=2089215418d9a497e536a0c4d0b9e4348dde029b9a14066b65cfa774283c360e source=proxy time="2020-12-01T21:19:20.556979558+08:00" level=fatal msg="failed to handle exit signal" error="close unix @->/run/vc/vm/2089215418d9a497e536a0c4d0b9e4348dde029b9a14066b65cfa774283c360e/kata.sock: use of closed network connection" name=kata-proxy pid=30769 sandbox=2089215418d9a497e536a0c4d0b9e4348dde029b9a14066b65cfa774283c360e source=proxy time="2020-12-01T21:20:54.884505034+08:00" level=fatal msg="failed to handle exit signal" error="close unix @->/run/vc/vm/8f3fbacb223c4d4157cec94d41138c43cd572ad504d074532cab4f209d2f0926/kata.sock: use of closed network connection" name=kata-proxy pid=31847 sandbox=8f3fbacb223c4d4157cec94d41138c43cd572ad504d074532cab4f209d2f0926 source=proxy time="2020-12-01T21:21:39.687566301+08:00" level=fatal msg="channel error" error="accept unix /run/vc/sbs/c25b6654138cf2bdc1a20287e6fc01f8be0239220b9267c401ba5c2e99b49edd/proxy.sock: use of closed network connection" name=kata-proxy pid=32042 sandbox=c25b6654138cf2bdc1a20287e6fc01f8be0239220b9267c401ba5c2e99b49edd source=proxy time="2020-12-02T00:27:20.175740464+08:00" level=fatal msg="failed to handle exit signal" error="close unix @->/run/vc/vm/06401bada8e64848f298fc959a3fd863305fddc6b2e47601bd42f995c8ba1365/kata.sock: use of closed network connection" name=kata-proxy pid=32220 sandbox=06401bada8e64848f298fc959a3fd863305fddc6b2e47601bd42f995c8ba1365 source=proxy time="2020-12-03T16:22:01.854361651+08:00" level=fatal msg="channel error" error="accept unix /run/vc/sbs/2089215418d9a497e536a0c4d0b9e4348dde029b9a14066b65cfa774283c360e/proxy.sock: use of closed network connection" name=kata-proxy pid=31180 sandbox=2089215418d9a497e536a0c4d0b9e4348dde029b9a14066b65cfa774283c360e source=proxy time="2020-12-03T16:37:25.352530852+08:00" level=fatal msg="failed to handle exit signal" error="close unix @->/run/vc/vm/2089215418d9a497e536a0c4d0b9e4348dde029b9a14066b65cfa774283c360e/kata.sock: use of closed network connection" name=kata-proxy pid=3760 sandbox=2089215418d9a497e536a0c4d0b9e4348dde029b9a14066b65cfa774283c360e source=proxy time="2020-12-03T17:22:12.894752711+08:00" level=fatal msg="failed to handle exit signal" error="close unix @->/run/vc/vm/2089215418d9a497e536a0c4d0b9e4348dde029b9a14066b65cfa774283c360e/kata.sock: use of closed network connection" name=kata-proxy pid=4580 sandbox=2089215418d9a497e536a0c4d0b9e4348dde029b9a14066b65cfa774283c360e source=proxy time="2020-12-03T19:17:03.955155936+08:00" level=fatal msg="channel error" error="accept unix /run/vc/sbs/2089215418d9a497e536a0c4d0b9e4348dde029b9a14066b65cfa774283c360e/proxy.sock: use of closed network connection" name=kata-proxy pid=3890 sandbox=2089215418d9a497e536a0c4d0b9e4348dde029b9a14066b65cfa774283c360e source=proxy time="2020-12-04T00:01:40.715294526+08:00" level=fatal msg="channel error" error="accept unix /run/vc/sbs/4862c399eb5e24ad7dd052f4c4710285f9273750bcb2b101f27aef085c9e50a8/proxy.sock: use of closed network connection" name=kata-proxy pid=10378 sandbox=4862c399eb5e24ad7dd052f4c4710285f9273750bcb2b101f27aef085c9e50a8 source=proxy time="2020-12-04T12:38:44.259677407+08:00" level=fatal msg="failed to handle exit signal" error="close unix @->/run/vc/vm/bb5e6a7e14e9340b883f1fc4824eeef34615d50d4074f900aa7f94ab9471d8bf/kata.sock: use of closed network connection" name=kata-proxy pid=3792 sandbox=bb5e6a7e14e9340b883f1fc4824eeef34615d50d4074f900aa7f94ab9471d8bf source=proxy time="2020-12-04T12:52:52.006342563+08:00" level=fatal msg="failed to handle exit signal" error="close unix @->/run/vc/vm/bb5e6a7e14e9340b883f1fc4824eeef34615d50d4074f900aa7f94ab9471d8bf/kata.sock: use of closed network connection" name=kata-proxy pid=4381 sandbox=bb5e6a7e14e9340b883f1fc4824eeef34615d50d4074f900aa7f94ab9471d8bf source=proxy time="2020-12-04T12:54:43.350148753+08:00" level=fatal msg="channel error" error="accept unix /run/vc/sbs/a2c2541a612392d4e299fda0993a615ff0827a0622e0bf5af57353676bb6e589/proxy.sock: use of closed network connection" name=kata-proxy pid=5206 sandbox=a2c2541a612392d4e299fda0993a615ff0827a0622e0bf5af57353676bb6e589 source=proxy time="2020-12-04T12:58:11.596762464+08:00" level=fatal msg="channel error" error="accept unix /run/vc/sbs/44fcfd00ee5f1d5a38c9f84b06d2368a1363fc5d351d4166b67a22fe7d8b5279/proxy.sock: use of closed network connection" name=kata-proxy pid=5778 sandbox=44fcfd00ee5f1d5a38c9f84b06d2368a1363fc5d351d4166b67a22fe7d8b5279 source=proxy time="2020-12-04T12:58:41.691041403+08:00" level=fatal msg="failed to handle exit signal" error="close unix @->/run/vc/vm/44fcfd00ee5f1d5a38c9f84b06d2368a1363fc5d351d4166b67a22fe7d8b5279/kata.sock: use of closed network connection" name=kata-proxy pid=6231 sandbox=44fcfd00ee5f1d5a38c9f84b06d2368a1363fc5d351d4166b67a22fe7d8b5279 source=proxy time="2020-12-04T13:00:10.210230859+08:00" level=fatal msg="failed to handle exit signal" error="close unix @->/run/vc/vm/a591ae4c778dcfd454f84efe5d21633c8c34ded0663f25eaf4cb0fc6c933d550/kata.sock: use of closed network connection" name=kata-proxy pid=6704 sandbox=a591ae4c778dcfd454f84efe5d21633c8c34ded0663f25eaf4cb0fc6c933d550 source=proxy time="2020-12-04T13:01:14.646797334+08:00" level=fatal msg="failed to handle exit signal" error="close unix @->/run/vc/vm/2ce3db5908ebb66dc2e30b6761ecd482398957213aef9f1a3c19e2f19500cdd9/kata.sock: use of closed network connection" name=kata-proxy pid=7128 sandbox=2ce3db5908ebb66dc2e30b6761ecd482398957213aef9f1a3c19e2f19500cdd9 source=proxy time="2020-12-04T13:01:55.736461994+08:00" level=fatal msg="failed to handle exit signal" error="close unix @->/run/vc/vm/0254f1ed99a98665f15af93c7dfc1d279c67894133da759123e30b21024d8ab2/kata.sock: use of closed network connection" name=kata-proxy pid=7433 sandbox=0254f1ed99a98665f15af93c7dfc1d279c67894133da759123e30b21024d8ab2 source=proxy time="2020-12-04T13:02:53.098722291+08:00" level=fatal msg="failed to handle exit signal" error="close unix @->/run/vc/vm/1dfce96a39428e64205b301f7a8bf56604c13207eaf63efa8433adee7148d31f/kata.sock: use of closed network connection" name=kata-proxy pid=7860 sandbox=1dfce96a39428e64205b301f7a8bf56604c13207eaf63efa8433adee7148d31f source=proxy time="2020-12-04T13:03:30.760580745+08:00" level=fatal msg="failed to handle exit signal" error="close unix @->/run/vc/vm/c75b7f8481091f9bf2f6814922bba01da8962b3ab598c45ff467a96faf5087c8/kata.sock: use of closed network connection" name=kata-proxy pid=8255 sandbox=c75b7f8481091f9bf2f6814922bba01da8962b3ab598c45ff467a96faf5087c8 source=proxy time="2020-12-04T13:06:52.635702806+08:00" level=fatal msg="channel error" error="accept unix /run/vc/sbs/836f977f9c3df797a84da66d31ebb5583283cb969d5d26fd02f6865d5a4838f4/proxy.sock: use of closed network connection" name=kata-proxy pid=8789 sandbox=836f977f9c3df797a84da66d31ebb5583283cb969d5d26fd02f6865d5a4838f4 source=proxy time="2020-12-04T13:07:14.771163688+08:00" level=fatal msg="failed to handle exit signal" error="close unix @->/run/vc/vm/836f977f9c3df797a84da66d31ebb5583283cb969d5d26fd02f6865d5a4838f4/kata.sock: use of closed network connection" name=kata-proxy pid=9163 sandbox=836f977f9c3df797a84da66d31ebb5583283cb969d5d26fd02f6865d5a4838f4 source=proxy time="2020-12-04T13:08:13.204489502+08:00" level=fatal msg="failed to handle exit signal" error="close unix @->/run/vc/vm/bf1a56010736411fc522271f2f0987f87d392c4b2e6d0b2dfd7f963b0e27c0c8/kata.sock: use of closed network connection" name=kata-proxy pid=9543 sandbox=bf1a56010736411fc522271f2f0987f87d392c4b2e6d0b2dfd7f963b0e27c0c8 source=proxy time="2020-12-04T13:08:45.426650292+08:00" level=fatal msg="failed to handle exit signal" error="close unix @->/run/vc/vm/2610dd0b4dd06533df787ed0a4cbacffe352dd2c10ee721e19ee06f49acfd0e9/kata.sock: use of closed network connection" name=kata-proxy pid=9802 sandbox=2610dd0b4dd06533df787ed0a4cbacffe352dd2c10ee721e19ee06f49acfd0e9 source=proxy time="2020-12-04T13:10:36.53424473+08:00" level=fatal msg="failed to handle exit signal" error="close unix @->/run/vc/vm/9e259ce02fb04b969cdb2eb79c4b6a56333beef27c291d1b916499f0d7ff451f/kata.sock: use of closed network connection" name=kata-proxy pid=10188 sandbox=9e259ce02fb04b969cdb2eb79c4b6a56333beef27c291d1b916499f0d7ff451f source=proxy time="2020-12-04T18:30:54.263316477+08:00" level=fatal msg="failed to handle exit signal" error="close unix @->/run/vc/vm/9486b1381158d36bd26b09ef031e52937e87a3161b8045238e5170173ba9d625/kata.sock: use of closed network connection" name=kata-proxy pid=14434 sandbox=9486b1381158d36bd26b09ef031e52937e87a3161b8045238e5170173ba9d625 source=proxy time="2020-12-04T18:30:54.265997004+08:00" level=fatal msg="channel error" error="accept unix /run/vc/sbs/375e9900dd1ffde793e9ecab614288f0b5f5fbc1e7807b436b7751af187ac2a4/proxy.sock: use of closed network connection" name=kata-proxy pid=10700 sandbox=375e9900dd1ffde793e9ecab614288f0b5f5fbc1e7807b436b7751af187ac2a4 source=proxy time="2020-12-05T12:54:57.142065174+08:00" level=fatal msg="failed to handle exit signal" error="close unix @->/run/vc/vm/13539266fa207f6b27b72854940dc2aa5f9ed130bf633efcbb04bdd94fa89f6e/kata.sock: use of closed network connection" name=kata-proxy pid=14160 sandbox=13539266fa207f6b27b72854940dc2aa5f9ed130bf633efcbb04bdd94fa89f6e source=proxy time="2020-12-05T13:18:37.089227335+08:00" level=fatal msg="failed to handle exit signal" error="close unix @->/run/vc/vm/c20694bb5bc05aab42488bf5dc2c7a84ec454aa9c5a5e07976eff6ce53e73b37/kata.sock: use of closed network connection" name=kata-proxy pid=14627 sandbox=c20694bb5bc05aab42488bf5dc2c7a84ec454aa9c5a5e07976eff6ce53e73b37 source=proxy time="2020-12-05T13:42:05.0516949+08:00" level=fatal msg="failed to handle exit signal" error="close unix @->/run/vc/vm/c20694bb5bc05aab42488bf5dc2c7a84ec454aa9c5a5e07976eff6ce53e73b37/kata.sock: use of closed network connection" name=kata-proxy pid=15459 sandbox=c20694bb5bc05aab42488bf5dc2c7a84ec454aa9c5a5e07976eff6ce53e73b37 source=proxy time="2020-12-05T16:37:57.201284781+08:00" level=fatal msg="failed to handle exit signal" error="close unix @->/run/vc/vm/b81bc4dd59bea7183d6271b06bf702951d6e110d4ee9364f9e6c849619111718/kata.sock: use of closed network connection" name=kata-proxy pid=16228 sandbox=b81bc4dd59bea7183d6271b06bf702951d6e110d4ee9364f9e6c849619111718 source=proxy time="2020-12-05T19:41:38.438895332+08:00" level=fatal msg="failed to handle exit signal" error="close unix @->/run/vc/vm/b81bc4dd59bea7183d6271b06bf702951d6e110d4ee9364f9e6c849619111718/kata.sock: use of closed network connection" name=kata-proxy pid=20300 sandbox=b81bc4dd59bea7183d6271b06bf702951d6e110d4ee9364f9e6c849619111718 source=proxy time="2020-12-05T20:41:12.47801523+08:00" level=fatal msg="failed to handle exit signal" error="close unix @->/run/vc/vm/663727e7bccdc828701545324f3ddf7bd7ababd9fa2ca4f70f19ee7139babb46/kata.sock: use of closed network connection" name=kata-proxy pid=4019 sandbox=663727e7bccdc828701545324f3ddf7bd7ababd9fa2ca4f70f19ee7139babb46 source=proxy time="2020-12-07T13:59:41.450265023+08:00" level=fatal msg="failed to handle exit signal" error="close unix @->/run/vc/vm/13539266fa207f6b27b72854940dc2aa5f9ed130bf633efcbb04bdd94fa89f6e/kata.sock: use of closed network connection" name=kata-proxy pid=29802 sandbox=13539266fa207f6b27b72854940dc2aa5f9ed130bf633efcbb04bdd94fa89f6e source=proxy time="2020-12-07T14:28:53.035290307+08:00" level=fatal msg="failed to handle exit signal" error="close unix @->/run/vc/vm/769582b7b9e8b0b92058a84f6ee6f983ffc0ebc30958935f281ecd8c8317a876/kata.sock: use of closed network connection" name=kata-proxy pid=30252 sandbox=769582b7b9e8b0b92058a84f6ee6f983ffc0ebc30958935f281ecd8c8317a876 source=proxy time="2020-12-07T14:57:11.948714542+08:00" level=fatal msg="failed to handle exit signal" error="close unix @->/run/vc/vm/5a2e37787c4d9b01ef8fa8e0bcab648bbb0df0a5bf16feba41884a20a417d226/kata.sock: use of closed network connection" name=kata-proxy pid=4214 sandbox=5a2e37787c4d9b01ef8fa8e0bcab648bbb0df0a5bf16feba41884a20a417d226 source=proxy time="2020-12-07T17:43:43.07133785+08:00" level=fatal msg="channel error" error="accept unix /run/vc/sbs/89a24578298fadbb203da9dd610e5dfd157a4a4a5f7adaca9cdc68f8d6027df7/proxy.sock: use of closed network connection" name=kata-proxy pid=4196 sandbox=89a24578298fadbb203da9dd610e5dfd157a4a4a5f7adaca9cdc68f8d6027df7 source=proxy time="2020-12-07T17:48:53.146507203+08:00" level=fatal msg="failed to handle exit signal" error="close unix @->/run/vc/vm/eeb3b1e50b1e202786e485c8534c8593ec2c1ca8b1c4db1a72ac4da3b5948acd/kata.sock: use of closed network connection" name=kata-proxy pid=36293 sandbox=eeb3b1e50b1e202786e485c8534c8593ec2c1ca8b1c4db1a72ac4da3b5948acd source=proxy time="2020-12-07T17:51:16.199098937+08:00" level=fatal msg="failed to handle exit signal" error="close unix @->/run/vc/vm/f1f3e9ecb2de3ef256227fe03049c56594a9c86122fb320aff0cc1a53132cd15/kata.sock: use of closed network connection" name=kata-proxy pid=38201 sandbox=f1f3e9ecb2de3ef256227fe03049c56594a9c86122fb320aff0cc1a53132cd15 source=proxy time="2020-12-07T17:53:36.404187996+08:00" level=fatal msg="failed to handle exit signal" error="close unix @->/run/vc/vm/53c7c475e7048a32f87ef43e5c52f8e0846c49e3b4deed97cb282685684a2691/kata.sock: use of closed network connection" name=kata-proxy pid=39135 sandbox=53c7c475e7048a32f87ef43e5c52f8e0846c49e3b4deed97cb282685684a2691 source=proxy time="2020-12-07T17:54:46.026793721+08:00" level=fatal msg="failed to handle exit signal" error="close unix @->/run/vc/vm/11183ca8789b19f55c5a130b7b5382a60601dcf8b1771c9ad5f84520eade90f9/kata.sock: use of closed network connection" name=kata-proxy pid=39742 sandbox=11183ca8789b19f55c5a130b7b5382a60601dcf8b1771c9ad5f84520eade90f9 source=proxy time="2020-12-07T17:55:26.231270046+08:00" level=fatal msg="failed to handle exit signal" error="close unix @->/run/vc/vm/87faf522948de0ab94d655b77b026002ab00941e9d4bbd5c7b0f8b6f660e3cbb/kata.sock: use of closed network connection" name=kata-proxy pid=40285 sandbox=87faf522948de0ab94d655b77b026002ab00941e9d4bbd5c7b0f8b6f660e3cbb source=proxy time="2020-12-07T17:56:24.539620107+08:00" level=fatal msg="channel error" error="session shutdown" name=kata-proxy pid=40846 sandbox=65ebb989192736084cd540aa3f11674fdf582fbc530b9992a6ed5dafaaada6f3 source=proxy time="2020-12-07T17:57:27.331588424+08:00" level=fatal msg="channel error" error="session shutdown" name=kata-proxy pid=901 sandbox=1f1838d24a7c388b28f8fbd1dac39bdd3468f64279f7b850701019dfbc872888 source=proxy ``` ## Shim logs Recent shim problems found in system journal: ``` time="2020-12-01T12:35:16.681969915+08:00" level=error msg="failed waiting for process" container=523a9fc1e8ba056382b7d8d30b5c524e5f929458da0288dd322b163623c13f9b error="rpc error: code = Unavailable desc = transport is closing" exec-id=523a9fc1e8ba056382b7d8d30b5c524e5f929458da0288dd322b163623c13f9b name=kata-shim pid=1 source=shim time="2020-12-01T12:40:44.061821013+08:00" level=error msg="failed waiting for process" container=e13b25f77f5e9a39006b6b400145a7c111ba7743c6d2e933e8dd7ca600e0c686 error="rpc error: code = Unavailable desc = transport is closing" exec-id=e13b25f77f5e9a39006b6b400145a7c111ba7743c6d2e933e8dd7ca600e0c686 name=kata-shim pid=1 source=shim time="2020-12-05T16:37:52.365354468+08:00" level=error msg="forward signal failed" container=b81bc4dd59bea7183d6271b06bf702951d6e110d4ee9364f9e6c849619111718 error="rpc error: code = Unknown desc = no such process" exec-id=7237cc0b-748c-47ab-8071-3284b3549ecb name=kata-shim pid=25 signal="urgent I/O condition" source=shim time="2020-12-07T14:57:11.948914089+08:00" level=error msg="forward signal failed" container=5a2e37787c4d9b01ef8fa8e0bcab648bbb0df0a5bf16feba41884a20a417d226 error="rpc error: code = Unavailable desc = transport is closing" exec-id=5a2e37787c4d9b01ef8fa8e0bcab648bbb0df0a5bf16feba41884a20a417d226 name=kata-shim pid=1 signal=hangup source=shim time="2020-12-07T17:51:16.19855803+08:00" level=error msg="failed waiting for process" container=f1f3e9ecb2de3ef256227fe03049c56594a9c86122fb320aff0cc1a53132cd15 error="rpc error: code = NotFound desc = Container f1f3e9ecb2de3ef256227fe03049c56594a9c86122fb320aff0cc1a53132cd15 not found" exec-id=f1f3e9ecb2de3ef256227fe03049c56594a9c86122fb320aff0cc1a53132cd15 name=kata-shim pid=1 source=shim time="2020-12-07T17:53:36.402030889+08:00" level=error msg="failed waiting for process" container=53c7c475e7048a32f87ef43e5c52f8e0846c49e3b4deed97cb282685684a2691 error="rpc error: code = NotFound desc = Container 53c7c475e7048a32f87ef43e5c52f8e0846c49e3b4deed97cb282685684a2691 not found" exec-id=53c7c475e7048a32f87ef43e5c52f8e0846c49e3b4deed97cb282685684a2691 name=kata-shim pid=1 source=shim time="2020-12-07T17:54:46.026320778+08:00" level=error msg="failed waiting for process" container=11183ca8789b19f55c5a130b7b5382a60601dcf8b1771c9ad5f84520eade90f9 error="rpc error: code = NotFound desc = Container 11183ca8789b19f55c5a130b7b5382a60601dcf8b1771c9ad5f84520eade90f9 not found" exec-id=11183ca8789b19f55c5a130b7b5382a60601dcf8b1771c9ad5f84520eade90f9 name=kata-shim pid=1 source=shim time="2020-12-07T17:55:26.231013276+08:00" level=error msg="failed waiting for process" container=87faf522948de0ab94d655b77b026002ab00941e9d4bbd5c7b0f8b6f660e3cbb error="rpc error: code = NotFound desc = Container 87faf522948de0ab94d655b77b026002ab00941e9d4bbd5c7b0f8b6f660e3cbb not found" exec-id=87faf522948de0ab94d655b77b026002ab00941e9d4bbd5c7b0f8b6f660e3cbb name=kata-shim pid=1 source=shim time="2020-12-07T17:56:24.54007432+08:00" level=error msg="failed waiting for process" container=65ebb989192736084cd540aa3f11674fdf582fbc530b9992a6ed5dafaaada6f3 error="rpc error: code = Unavailable desc = all SubConns are in TransientFailure, latest connection error: connection error: desc = \"transport: failed to write client preface: write unix @->/run/vc/sbs/65ebb989192736084cd540aa3f11674fdf582fbc530b9992a6ed5dafaaada6f3/proxy.sock: write: broken pipe\"" exec-id=65ebb989192736084cd540aa3f11674fdf582fbc530b9992a6ed5dafaaada6f3 name=kata-shim pid=1 source=shim ``` ## Throttler logs No recent throttler problems found in system journal. --- # Container manager details Have `docker` ## Docker Output of "`docker version`": ``` Client: Docker Engine - Community Version: 19.03.13 API version: 1.40 Go version: go1.13.15 Git commit: 4484c46d9d Built: Wed Sep 16 17:02:36 2020 OS/Arch: linux/amd64 Experimental: false Server: Docker Engine - Community Engine: Version: 19.03.13 API version: 1.40 (minimum version 1.12) Go version: go1.13.15 Git commit: 4484c46d9d Built: Wed Sep 16 17:01:06 2020 OS/Arch: linux/amd64 Experimental: false containerd: Version: 1.3.7 GitCommit: 8fba4e9a7d01810a393d5d25a3621dc101981175 docker-init: Version: 0.18.0 GitCommit: fec3683 ``` Output of "`docker info`": ``` Client: Debug Mode: false Server: Containers: 25 Running: 1 Paused: 0 Stopped: 24 Images: 11 Server Version: 19.03.13 Storage Driver: overlay2 Backing Filesystem: extfs Supports d_type: true Native Overlay Diff: true Logging Driver: json-file Cgroup Driver: cgroupfs Plugins: Volume: local Network: bridge host ipvlan macvlan null overlay Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog Swarm: inactive Runtimes: kata-fc kata-qemu kata-qemu-virtiofs kata-runtime runc runsc kata-acrn kata-clh Default Runtime: kata-runtime Init Binary: docker-init containerd version: 8fba4e9a7d01810a393d5d25a3621dc101981175 runc version: N/A init version: fec3683 Security Options: apparmor seccomp Profile: default Kernel Version: 5.4.0-56-generic Operating System: Ubuntu 18.04.5 LTS OSType: linux Architecture: x86_64 CPUs: 40 Total Memory: 251.7GiB Name: dell-07 ID: WF4K:6AZT:V45K:GEBX:QUNB:6L3W:YPSI:XBXU:LYQW:4YYC:7D7L:DEB6 Docker Root Dir: /var/lib/docker Debug Mode: true File Descriptors: 38 Goroutines: 60 System Time: 2020-12-07T19:02:19.001946714+08:00 EventsListeners: 0 Registry: https://index.docker.io/v1/ Labels: Experimental: false Insecure Registries: 127.0.0.0/8 Live Restore Enabled: false ``` Output of "`systemctl show docker`": ``` Type=notify Restart=always NotifyAccess=main RestartUSec=2s TimeoutStartUSec=infinity TimeoutStopUSec=infinity RuntimeMaxUSec=infinity WatchdogUSec=0 WatchdogTimestamp=Mon 2020-12-07 15:01:17 CST WatchdogTimestampMonotonic=54894221 PermissionsStartOnly=no RootDirectoryStartOnly=no RemainAfterExit=no GuessMainPID=yes MainPID=3025 ControlPID=0 FileDescriptorStoreMax=0 NFileDescriptorStore=0 StatusErrno=0 Result=success UID=[not set] GID=[not set] NRestarts=0 ExecMainStartTimestamp=Mon 2020-12-07 15:01:10 CST ExecMainStartTimestampMonotonic=47115541 ExecMainExitTimestampMonotonic=0 ExecMainPID=3025 ExecMainCode=0 ExecMainStatus=0 ExecStart={ path=/usr/bin/dockerd ; argv[]=/usr/bin/dockerd -D --add-runtime kata-runtime=/usr/bin/kata-runtime --default-runtime=kata-runtime ; ignore_errors=no ; start_time=[Mon 2020-12-07 15:01:10 CST] ; stop_time=[n/a] ; pid=3025 ; code=(null) ; status=0/0 } ExecReload={ path=/bin/kill ; argv[]=/bin/kill -s HUP $MAINPID ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 } Slice=system.slice ControlGroup=/system.slice/docker.service MemoryCurrent=[not set] CPUUsageNSec=[not set] TasksCurrent=59 IPIngressBytes=18446744073709551615 IPIngressPackets=18446744073709551615 IPEgressBytes=18446744073709551615 IPEgressPackets=18446744073709551615 Delegate=yes DelegateControllers=cpu cpuacct io blkio memory devices pids CPUAccounting=no CPUWeight=[not set] StartupCPUWeight=[not set] CPUShares=[not set] StartupCPUShares=[not set] CPUQuotaPerSecUSec=infinity IOAccounting=no IOWeight=[not set] StartupIOWeight=[not set] BlockIOAccounting=no BlockIOWeight=[not set] StartupBlockIOWeight=[not set] MemoryAccounting=no MemoryLow=0 MemoryHigh=infinity MemoryMax=infinity MemorySwapMax=infinity MemoryLimit=infinity DevicePolicy=auto TasksAccounting=yes TasksMax=infinity IPAccounting=no UMask=0022 LimitCPU=infinity LimitCPUSoft=infinity LimitFSIZE=infinity LimitFSIZESoft=infinity LimitDATA=infinity LimitDATASoft=infinity LimitSTACK=infinity LimitSTACKSoft=8388608 LimitCORE=infinity LimitCORESoft=infinity LimitRSS=infinity LimitRSSSoft=infinity LimitNOFILE=infinity LimitNOFILESoft=infinity LimitAS=infinity LimitASSoft=infinity LimitNPROC=infinity LimitNPROCSoft=infinity LimitMEMLOCK=16777216 LimitMEMLOCKSoft=16777216 LimitLOCKS=infinity LimitLOCKSSoft=infinity LimitSIGPENDING=1030698 LimitSIGPENDINGSoft=1030698 LimitMSGQUEUE=819200 LimitMSGQUEUESoft=819200 LimitNICE=0 LimitNICESoft=0 LimitRTPRIO=0 LimitRTPRIOSoft=0 LimitRTTIME=infinity LimitRTTIMESoft=infinity OOMScoreAdjust=0 Nice=0 IOSchedulingClass=0 IOSchedulingPriority=0 CPUSchedulingPolicy=0 CPUSchedulingPriority=0 TimerSlackNSec=50000 CPUSchedulingResetOnFork=no NonBlocking=no StandardInput=null StandardInputData= StandardOutput=journal StandardError=inherit TTYReset=no TTYVHangup=no TTYVTDisallocate=no SyslogPriority=30 SyslogLevelPrefix=yes SyslogLevel=6 SyslogFacility=3 LogLevelMax=-1 SecureBits=0 CapabilityBoundingSet=cap_chown cap_dac_override cap_dac_read_search cap_fowner cap_fsetid cap_kill cap_setgid cap_setuid cap_setpcap cap_linux_immutable cap_net_bind_service cap_net_broadcast cap_net_admin cap_net_raw cap_ipc_lock cap_ipc_owner cap_sys_module cap_sys_rawio cap_sys_chroot cap_sys_ptrace cap_sys_pacct cap_sys_admin cap_sys_boot cap_sys_nice cap_sys_resource cap_sys_time cap_sys_tty_config cap_mknod cap_lease cap_audit_write cap_audit_control cap_setfcap cap_mac_override cap_mac_admin cap_syslog cap_wake_alarm cap_block_suspend AmbientCapabilities= DynamicUser=no RemoveIPC=no MountFlags= PrivateTmp=no PrivateDevices=no ProtectKernelTunables=no ProtectKernelModules=no ProtectControlGroups=no PrivateNetwork=no PrivateUsers=no ProtectHome=no ProtectSystem=no SameProcessGroup=no UtmpMode=init IgnoreSIGPIPE=yes NoNewPrivileges=no SystemCallErrorNumber=0 LockPersonality=no RuntimeDirectoryPreserve=no RuntimeDirectoryMode=0755 StateDirectoryMode=0755 CacheDirectoryMode=0755 LogsDirectoryMode=0755 ConfigurationDirectoryMode=0755 MemoryDenyWriteExecute=no RestrictRealtime=no RestrictSUIDSGID=no RestrictNamespaces=no MountAPIVFS=no KeyringMode=private KillMode=process KillSignal=15 SendSIGKILL=yes SendSIGHUP=no Id=docker.service Names=docker.service Requires=sysinit.target system.slice docker.socket Wants=network-online.target BindsTo=containerd.service WantedBy=multi-user.target ConsistsOf=docker.socket Conflicts=shutdown.target Before=shutdown.target multi-user.target After=docker.socket sysinit.target network-online.target firewalld.service systemd-journald.socket containerd.service basic.target system.slice TriggeredBy=docker.socket Documentation=https://docs.docker.com Description=Docker Application Container Engine LoadState=loaded ActiveState=active SubState=running FragmentPath=/lib/systemd/system/docker.service DropInPaths=/etc/systemd/system/docker.service.d/kata-containers.conf UnitFileState=enabled UnitFilePreset=enabled StateChangeTimestamp=Mon 2020-12-07 15:01:17 CST StateChangeTimestampMonotonic=54894225 InactiveExitTimestamp=Mon 2020-12-07 15:01:10 CST InactiveExitTimestampMonotonic=47115594 ActiveEnterTimestamp=Mon 2020-12-07 15:01:17 CST ActiveEnterTimestampMonotonic=54894225 ActiveExitTimestampMonotonic=0 InactiveEnterTimestampMonotonic=0 CanStart=yes CanStop=yes CanReload=yes CanIsolate=no StopWhenUnneeded=no RefuseManualStart=no RefuseManualStop=no AllowIsolate=no DefaultDependencies=yes OnFailureJobMode=replace IgnoreOnIsolate=no NeedDaemonReload=no JobTimeoutUSec=infinity JobRunningTimeoutUSec=infinity JobTimeoutAction=none ConditionResult=yes AssertResult=yes ConditionTimestamp=Mon 2020-12-07 15:01:10 CST ConditionTimestampMonotonic=47113872 AssertTimestamp=Mon 2020-12-07 15:01:10 CST AssertTimestampMonotonic=47113872 Transient=no Perpetual=no StartLimitIntervalUSec=1min StartLimitBurst=3 StartLimitAction=none FailureAction=none SuccessAction=none InvocationID=41dbcd37c922461f82656270018a0eb1 CollectMode=inactive ``` No `kubectl` No `crio` Have `containerd` ## containerd Output of "`containerd --version`": ``` containerd containerd.io 1.3.7 8fba4e9a7d01810a393d5d25a3621dc101981175 ``` Output of "`systemctl show containerd`": ``` Type=notify Restart=always NotifyAccess=main RestartUSec=100ms TimeoutStartUSec=1min 30s TimeoutStopUSec=1min 30s RuntimeMaxUSec=infinity WatchdogUSec=0 WatchdogTimestamp=Mon 2020-12-07 15:00:59 CST WatchdogTimestampMonotonic=36248993 PermissionsStartOnly=no RootDirectoryStartOnly=no RemainAfterExit=no GuessMainPID=yes MainPID=1878 ControlPID=0 FileDescriptorStoreMax=0 NFileDescriptorStore=0 StatusErrno=0 Result=success UID=[not set] GID=[not set] NRestarts=0 ExecMainStartTimestamp=Mon 2020-12-07 15:00:55 CST ExecMainStartTimestampMonotonic=32436090 ExecMainExitTimestampMonotonic=0 ExecMainPID=1878 ExecMainCode=0 ExecMainStatus=0 ExecStartPre={ path=/sbin/modprobe ; argv[]=/sbin/modprobe overlay ; ignore_errors=yes ; start_time=[Mon 2020-12-07 15:00:55 CST] ; stop_time=[Mon 2020-12-07 15:00:55 CST] ; pid=1855 ; code=exited ; status=0 } ExecStart={ path=/usr/bin/containerd ; argv[]=/usr/bin/containerd ; ignore_errors=no ; start_time=[Mon 2020-12-07 15:00:55 CST] ; stop_time=[n/a] ; pid=1878 ; code=(null) ; status=0/0 } Slice=system.slice ControlGroup=/system.slice/containerd.service MemoryCurrent=[not set] CPUUsageNSec=[not set] TasksCurrent=237 IPIngressBytes=18446744073709551615 IPIngressPackets=18446744073709551615 IPEgressBytes=18446744073709551615 IPEgressPackets=18446744073709551615 Delegate=yes DelegateControllers=cpu cpuacct io blkio memory devices pids CPUAccounting=no CPUWeight=[not set] StartupCPUWeight=[not set] CPUShares=[not set] StartupCPUShares=[not set] CPUQuotaPerSecUSec=infinity IOAccounting=no IOWeight=[not set] StartupIOWeight=[not set] BlockIOAccounting=no BlockIOWeight=[not set] StartupBlockIOWeight=[not set] MemoryAccounting=no MemoryLow=0 MemoryHigh=infinity MemoryMax=infinity MemorySwapMax=infinity MemoryLimit=infinity DevicePolicy=auto TasksAccounting=yes TasksMax=infinity IPAccounting=no UMask=0022 LimitCPU=infinity LimitCPUSoft=infinity LimitFSIZE=infinity LimitFSIZESoft=infinity LimitDATA=infinity LimitDATASoft=infinity LimitSTACK=infinity LimitSTACKSoft=8388608 LimitCORE=infinity LimitCORESoft=infinity LimitRSS=infinity LimitRSSSoft=infinity LimitNOFILE=1048576 LimitNOFILESoft=1048576 LimitAS=infinity LimitASSoft=infinity LimitNPROC=infinity LimitNPROCSoft=infinity LimitMEMLOCK=16777216 LimitMEMLOCKSoft=16777216 LimitLOCKS=infinity LimitLOCKSSoft=infinity LimitSIGPENDING=1030698 LimitSIGPENDINGSoft=1030698 LimitMSGQUEUE=819200 LimitMSGQUEUESoft=819200 LimitNICE=0 LimitNICESoft=0 LimitRTPRIO=0 LimitRTPRIOSoft=0 LimitRTTIME=infinity LimitRTTIMESoft=infinity OOMScoreAdjust=0 Nice=0 IOSchedulingClass=0 IOSchedulingPriority=0 CPUSchedulingPolicy=0 CPUSchedulingPriority=0 TimerSlackNSec=50000 CPUSchedulingResetOnFork=no NonBlocking=no StandardInput=null StandardInputData= StandardOutput=journal StandardError=inherit TTYReset=no TTYVHangup=no TTYVTDisallocate=no SyslogPriority=30 SyslogLevelPrefix=yes SyslogLevel=6 SyslogFacility=3 LogLevelMax=-1 SecureBits=0 CapabilityBoundingSet=cap_chown cap_dac_override cap_dac_read_search cap_fowner cap_fsetid cap_kill cap_setgid cap_setuid cap_setpcap cap_linux_immutable cap_net_bind_service cap_net_broadcast cap_net_admin cap_net_raw cap_ipc_lock cap_ipc_owner cap_sys_module cap_sys_rawio cap_sys_chroot cap_sys_ptrace cap_sys_pacct cap_sys_admin cap_sys_boot cap_sys_nice cap_sys_resource cap_sys_time cap_sys_tty_config cap_mknod cap_lease cap_audit_write cap_audit_control cap_setfcap cap_mac_override cap_mac_admin cap_syslog cap_wake_alarm cap_block_suspend AmbientCapabilities= DynamicUser=no RemoveIPC=no MountFlags= PrivateTmp=no PrivateDevices=no ProtectKernelTunables=no ProtectKernelModules=no ProtectControlGroups=no PrivateNetwork=no PrivateUsers=no ProtectHome=no ProtectSystem=no SameProcessGroup=no UtmpMode=init IgnoreSIGPIPE=yes NoNewPrivileges=no SystemCallErrorNumber=0 LockPersonality=no RuntimeDirectoryPreserve=no RuntimeDirectoryMode=0755 StateDirectoryMode=0755 CacheDirectoryMode=0755 LogsDirectoryMode=0755 ConfigurationDirectoryMode=0755 MemoryDenyWriteExecute=no RestrictRealtime=no RestrictSUIDSGID=no RestrictNamespaces=no MountAPIVFS=no KeyringMode=private KillMode=process KillSignal=15 SendSIGKILL=yes SendSIGHUP=no Id=containerd.service Names=containerd.service Requires=sysinit.target system.slice WantedBy=multi-user.target BoundBy=docker.service Conflicts=shutdown.target Before=multi-user.target shutdown.target docker.service After=network.target local-fs.target systemd-journald.socket sysinit.target basic.target system.slice Documentation=https://containerd.io Description=containerd container runtime LoadState=loaded ActiveState=active SubState=running FragmentPath=/lib/systemd/system/containerd.service UnitFileState=enabled UnitFilePreset=enabled StateChangeTimestamp=Mon 2020-12-07 15:00:59 CST StateChangeTimestampMonotonic=36248995 InactiveExitTimestamp=Mon 2020-12-07 15:00:55 CST InactiveExitTimestampMonotonic=32425761 ActiveEnterTimestamp=Mon 2020-12-07 15:00:59 CST ActiveEnterTimestampMonotonic=36248995 ActiveExitTimestampMonotonic=0 InactiveEnterTimestampMonotonic=0 CanStart=yes CanStop=yes CanReload=no CanIsolate=no StopWhenUnneeded=no RefuseManualStart=no RefuseManualStop=no AllowIsolate=no DefaultDependencies=yes OnFailureJobMode=replace IgnoreOnIsolate=no NeedDaemonReload=no JobTimeoutUSec=infinity JobRunningTimeoutUSec=infinity JobTimeoutAction=none ConditionResult=yes AssertResult=yes ConditionTimestamp=Mon 2020-12-07 15:00:55 CST ConditionTimestampMonotonic=32424706 AssertTimestamp=Mon 2020-12-07 15:00:55 CST AssertTimestampMonotonic=32424706 Transient=no Perpetual=no StartLimitIntervalUSec=10s StartLimitBurst=5 StartLimitAction=none FailureAction=none SuccessAction=none InvocationID=a9ce546253de40b9a1f4c7be76acb7a8 CollectMode=inactive ``` Output of "`cat /etc/containerd/config.toml`": ``` # Copyright 2018-2020 Docker Inc. # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # http://www.apache.org/licenses/LICENSE-2.0 # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. disabled_plugins = ["cri"] #root = "/var/lib/containerd" #state = "/run/containerd" #subreaper = true #oom_score = 0 #[grpc] # address = "/run/containerd/containerd.sock" # uid = 0 # gid = 0 #[debug] # address = "/run/containerd/debug.sock" # uid = 0 # gid = 0 # level = "info" ``` --- # Packages Have `dpkg` Output of "`dpkg -l|egrep "(cc-oci-runtimecc-runtimerunv|kata-proxy|kata-runtime|kata-shim|kata-ksm-throttler|kata-containers-image|linux-container|qemu-)"`": ``` ii ipxe-qemu-256k-compat-efi-roms 1.0.0+git-20150424.a25a16d-0ubuntu2 all PXE boot firmware - Compat EFI ROM images for qemu ii kata-containers-image 1.12.0~rc0-48 amd64 Kata containers image ii kata-ksm-throttler 1.12.0~rc0-51 amd64 ii kata-linux-container 5.4.60.89-51 amd64 linux kernel optimised for container-like workloads. ii kata-proxy 1.12.0~rc0-49 amd64 ii kata-runtime 1.12.0~rc0-56 amd64 ii kata-shim 1.12.0~rc0-47 amd64 ii qemu-block-extra:amd64 1:2.11+dfsg-1ubuntu7.34 amd64 extra block backend modules for qemu-system and qemu-utils ii qemu-kvm 1:2.11+dfsg-1ubuntu7.34 amd64 QEMU Full virtualization on x86 hardware ii qemu-slof 20170724+dfsg-1ubuntu1 all Slimline Open Firmware -- QEMU PowerPC version ii qemu-system 1:2.11+dfsg-1ubuntu7.34 amd64 QEMU full system emulation binaries ii qemu-system-arm 1:2.11+dfsg-1ubuntu7.34 amd64 QEMU full system emulation binaries (arm) ii qemu-system-common 1:2.11+dfsg-1ubuntu7.34 amd64 QEMU full system emulation binaries (common files) ii qemu-system-mips 1:2.11+dfsg-1ubuntu7.34 amd64 QEMU full system emulation binaries (mips) ii qemu-system-misc 1:2.11+dfsg-1ubuntu7.34 amd64 QEMU full system emulation binaries (miscellaneous) ii qemu-system-ppc 1:2.11+dfsg-1ubuntu7.34 amd64 QEMU full system emulation binaries (ppc) ii qemu-system-s390x 1:2.11+dfsg-1ubuntu7.34 amd64 QEMU full system emulation binaries (s390x) ii qemu-system-sparc 1:2.11+dfsg-1ubuntu7.34 amd64 QEMU full system emulation binaries (sparc) ii qemu-system-x86 1:2.11+dfsg-1ubuntu7.34 amd64 QEMU full system emulation binaries (x86) ii qemu-user 1:2.11+dfsg-1ubuntu7.34 amd64 QEMU user mode emulation binaries ii qemu-user-binfmt 1:2.11+dfsg-1ubuntu7.34 amd64 QEMU user mode binfmt registration for qemu-user ii qemu-utils 1:2.11+dfsg-1ubuntu7.34 amd64 QEMU utilities ii qemu-vanilla 5.0.0+git.fdd76fecdd-52 amd64 linux kernel optimised for container-like workloads. ``` No `rpm` ---

kata-containers / documentation

Does the kernel in kata containers support Page Cache and Dirty Page Writeback? #770

Description of problem

Expected result

Actual result

Further information