Errors with podman 1.8.1

Here are the logs information:

Show kata-collect-data.sh details

# Meta details Running `kata-collect-data.sh` version `1.11.0-alpha0 (commit aaa4e5c66183a709db1d8960d45aef43dba22baf)` at `2020-03-17.18:32:27.731939205+0000`. --- Runtime is `/usr/local/bin/kata-runtime`. # `kata-env` Output of "`/usr/local/bin/kata-runtime kata-env`": ```toml [Meta] Version = "1.0.24" [Runtime] Debug = true Trace = false DisableGuestSeccomp = true DisableNewNetNs = false SandboxCgroupOnly = false Path = "/usr/local/bin/kata-runtime" [Runtime.Version] OCI = "1.0.1-dev" [Runtime.Version.Version] Semver = "1.11.0-alpha0" Major = 1 Minor = 11 Patch = 0 Commit = "aaa4e5c66183a709db1d8960d45aef43dba22baf" [Runtime.Config] Path = "/etc/kata-containers/configuration.toml" [Hypervisor] MachineType = "pc" Version = "QEMU emulator version 4.1.1 (v4.1.1-dirty)\nCopyright (c) 2003-2019 Fabrice Bellard and the QEMU Project developers" Path = "/usr/bin/qemu-system-x86_64" BlockDeviceDriver = "virtio-scsi" EntropySource = "/dev/urandom" SharedFS = "virtio-9p" VirtioFSDaemon = "/usr/bin/virtiofsd" Msize9p = 8192 MemorySlots = 10 PCIeRootPort = 0 HotplugVFIOOnRootBus = false Debug = true UseVSock = false [Image] Path = "" [Kernel] Path = "/usr/share/kata-containers/vmlinuz-5.4.15-69" Parameters = "agent.log=debug agent.log=debug" [Initrd] Path = "/usr/share/kata-containers/kata-containers-alpine-3.7-osbuilder-73eb5c1-agent-2725dda.initrd" [Proxy] Type = "kataProxy" Path = "/usr/libexec/kata-containers/kata-proxy" Debug = true [Proxy.Version] Semver = "1.11.0-alpha0-4ec94c8e7a933af8778e10c8035f4c3a4ab7c9c1" Major = 1 Minor = 11 Patch = 0 Commit = "4ec94c8e7a933af8778e10c8035f4c3a4ab7c9c1" [Shim] Type = "kataShim" Path = "/usr/libexec/kata-containers/kata-shim" Debug = true [Shim.Version] Semver = "1.11.0-alpha1-4728ac44c159565aeb9213a9f4607e24be415cc0" Major = 1 Minor = 11 Patch = 0 Commit = "4728ac44c159565aeb9213a9f4607e24be415cc0" [Agent] Type = "kata" Debug = true Trace = false TraceMode = "" TraceType = "" [Host] Kernel = "5.3.7-301.fc31.x86_64" Architecture = "amd64" VMContainerCapable = true SupportVSocks = true [Host.Distro] Name = "Fedora" Version = "31" [Host.CPU] Vendor = "GenuineIntel" Model = "Intel(R) Xeon(R) CPU E5-2673 v4 @ 2.30GHz" [Netmon] Path = "/usr/libexec/kata-containers/kata-netmon" Debug = true Enable = false [Netmon.Version] Semver = "1.11.0-alpha0" Major = 1 Minor = 11 Patch = 0 Commit = "<>" ``` --- # Runtime config files ## Runtime default config files ``` /etc/kata-containers/configuration.toml /usr/share/defaults/kata-containers/configuration.toml ``` ## Runtime config file contents Output of "`cat "/etc/kata-containers/configuration.toml"`": ```toml # Copyright (c) 2017-2019 Intel Corporation # # SPDX-License-Identifier: Apache-2.0 # # XXX: WARNING: this file is auto-generated. # XXX: # XXX: Source file: "cli/config/configuration-qemu.toml.in" # XXX: Project: # XXX: Name: Kata Containers # XXX: Type: kata [hypervisor.qemu] path = "/usr/bin/qemu-system-x86_64" kernel = "/usr/share/kata-containers/vmlinuz.container" initrd = "/usr/share/kata-containers/kata-containers-initrd.img" machine_type = "pc" # Optional space-separated list of options to pass to the guest kernel. # For example, use `kernel_params = "vsyscall=emulate"` if you are having # trouble running pre-2.15 glibc. # # WARNING: - any parameter specified here will take priority over the default # parameter value of the same name used to start the virtual machine. # Do not set values here unless you understand the impact of doing so as you # may stop the virtual machine from booting. # To see the list of default parameters, enable hypervisor debug, create a # container and look for 'default-kernel-parameters' log entries. kernel_params = " agent.log=debug" # Path to the firmware. # If you want that qemu uses the default firmware leave this option empty firmware = "" # Machine accelerators # comma-separated list of machine accelerators to pass to the hypervisor. # For example, `machine_accelerators = "nosmm,nosmbus,nosata,nopit,static-prt,nofw"` machine_accelerators="" # Default number of vCPUs per SB/VM: # unspecified or 0 --> will be set to 1 # < 0 --> will be set to the actual number of physical cores # > 0 <= number of physical cores --> will be set to the specified number # > number of physical cores --> will be set to the actual number of physical cores default_vcpus = 1 # Default maximum number of vCPUs per SB/VM: # unspecified or == 0 --> will be set to the actual number of physical cores or to the maximum number # of vCPUs supported by KVM if that number is exceeded # > 0 <= number of physical cores --> will be set to the specified number # > number of physical cores --> will be set to the actual number of physical cores or to the maximum number # of vCPUs supported by KVM if that number is exceeded # WARNING: Depending of the architecture, the maximum number of vCPUs supported by KVM is used when # the actual number of physical cores is greater than it. # WARNING: Be aware that this value impacts the virtual machine's memory footprint and CPU # the hotplug functionality. For example, `default_maxvcpus = 240` specifies that until 240 vCPUs # can be added to a SB/VM, but the memory footprint will be big. Another example, with # `default_maxvcpus = 8` the memory footprint will be small, but 8 will be the maximum number of # vCPUs supported by the SB/VM. In general, we recommend that you do not edit this variable, # unless you know what are you doing. default_maxvcpus = 0 # Bridges can be used to hot plug devices. # Limitations: # * Currently only pci bridges are supported # * Until 30 devices per bridge can be hot plugged. # * Until 5 PCI bridges can be cold plugged per VM. # This limitation could be a bug in qemu or in the kernel # Default number of bridges per SB/VM: # unspecified or 0 --> will be set to 1 # > 1 <= 5 --> will be set to the specified number # > 5 --> will be set to 5 default_bridges = 1 # Default memory size in MiB for SB/VM. # If unspecified then it will be set 2048 MiB. default_memory = 2048 # # Default memory slots per SB/VM. # If unspecified then it will be set 10. # This is will determine the times that memory will be hotadded to sandbox/VM. #memory_slots = 10 # The size in MiB will be plused to max memory of hypervisor. # It is the memory address space for the NVDIMM devie. # If set block storage driver (block_device_driver) to "nvdimm", # should set memory_offset to the size of block device. # Default 0 #memory_offset = 0 # Specifies virtio-mem will be enabled or not. # Please note that this option should be used with the command # "echo 1 > /proc/sys/vm/overcommit_memory". # Default false #enable_virtio_mem = true # Disable block device from being used for a container's rootfs. # In case of a storage driver like devicemapper where a container's # root file system is backed by a block device, the block device is passed # directly to the hypervisor for performance reasons. # This flag prevents the block device from being passed to the hypervisor, # 9pfs is used instead to pass the rootfs. disable_block_device_use = false # Shared file system type: # - virtio-9p (default) # - virtio-fs shared_fs = "virtio-9p" # Path to vhost-user-fs daemon. virtio_fs_daemon = "/usr/bin/virtiofsd" # Default size of DAX cache in MiB virtio_fs_cache_size = 1024 # Extra args for virtiofsd daemon # # Format example: # ["-o", "arg1=xxx,arg2", "-o", "hello world", "--arg3=yyy"] # # see `virtiofsd -h` for possible options. virtio_fs_extra_args = [] # Cache mode: # # - none # Metadata, data, and pathname lookup are not cached in guest. They are # always fetched from host and any changes are immediately pushed to host. # # - auto # Metadata and pathname lookup cache expires after a configured amount of # time (default is 1 second). Data is cached while the file is open (close # to open consistency). # # - always # Metadata, data, and pathname lookup are cached in guest and never expire. virtio_fs_cache = "always" # Block storage driver to be used for the hypervisor in case the container # rootfs is backed by a block device. This is virtio-scsi, virtio-blk # or nvdimm. block_device_driver = "virtio-scsi" # Specifies cache-related options will be set to block devices or not. # Default false #block_device_cache_set = true # Specifies cache-related options for block devices. # Denotes whether use of O_DIRECT (bypass the host page cache) is enabled. # Default false #block_device_cache_direct = true # Specifies cache-related options for block devices. # Denotes whether flush requests for the device are ignored. # Default false #block_device_cache_noflush = true # Enable iothreads (data-plane) to be used. This causes IO to be # handled in a separate IO thread. This is currently only implemented # for SCSI. # enable_iothreads = false # Enable pre allocation of VM RAM, default false # Enabling this will result in lower container density # as all of the memory will be allocated and locked # This is useful when you want to reserve all the memory # upfront or in the cases where you want memory latencies # to be very predictable # Default false #enable_mem_prealloc = true # Enable huge pages for VM RAM, default false # Enabling this will result in the VM memory # being allocated using huge pages. # This is useful when you want to use vhost-user network # stacks within the container. This will automatically # result in memory pre allocation #enable_hugepages = true # Enable file based guest memory support. The default is an empty string which # will disable this feature. In the case of virtio-fs, this is enabled # automatically and '/dev/shm' is used as the backing folder. # This option will be ignored if VM templating is enabled. #file_mem_backend = "" # Enable swap of vm memory. Default false. # The behaviour is undefined if mem_prealloc is also set to true #enable_swap = true # This option changes the default hypervisor and kernel parameters # to enable debug output where available. This extra output is added # to the proxy logs, but only when proxy debug is also enabled. # # Default false enable_debug = true # Disable the customizations done in the runtime when it detects # that it is running on top a VMM. This will result in the runtime # behaving as it would when running on bare metal. # #disable_nesting_checks = true # This is the msize used for 9p shares. It is the number of bytes # used for 9p packet payload. #msize_9p = 8192 # If true and vsocks are supported, use vsocks to communicate directly # with the agent and no proxy is started, otherwise use unix # sockets and start a proxy to communicate with the agent. # Default false #use_vsock = true # If false and nvdimm is supported, use nvdimm device to plug guest image. # Otherwise virtio-block device is used. # Default is false #disable_image_nvdimm = true # VFIO devices are hotplugged on a bridge by default. # Enable hotplugging on root bus. This may be required for devices with # a large PCI bar, as this is a current limitation with hotplugging on # a bridge. This value is valid for "pc" machine type. # Default false #hotplug_vfio_on_root_bus = true # Before hot plugging a PCIe device, you need to add a pcie_root_port device. # Use this parameter when using some large PCI bar devices, such as Nvidia GPU # The value means the number of pcie_root_port # This value is valid when hotplug_vfio_on_root_bus is true and machine_type is "q35" # Default 0 #pcie_root_port = 2 # If vhost-net backend for virtio-net is not desired, set to true. Default is false, which trades off # security (vhost-net runs ring0) for network I/O performance. disable_vhost_net = true # # Default entropy source. # The path to a host source of entropy (including a real hardware RNG) # /dev/urandom and /dev/random are two main options. # Be aware that /dev/random is a blocking source of entropy. If the host # runs out of entropy, the VMs boot time will increase leading to get startup # timeouts. # The source of entropy /dev/urandom is non-blocking and provides a # generally acceptable source of entropy. It should work well for pretty much # all practical purposes. #entropy_source= "/dev/urandom" # Path to OCI hook binaries in the *guest rootfs*. # This does not affect host-side hooks which must instead be added to # the OCI spec passed to the runtime. # # You can create a rootfs with hooks by customizing the osbuilder scripts: # https://github.com/kata-containers/osbuilder # # Hooks must be stored in a subdirectory of guest_hook_path according to their # hook type, i.e. "guest_hook_path/{prestart,postart,poststop}". # The agent will scan these directories for executable files and add them, in # lexicographical order, to the lifecycle of the guest container. # Hooks are executed in the runtime namespace of the guest. See the official documentation: # https://github.com/opencontainers/runtime-spec/blob/v1.0.1/config.md#posix-platform-hooks # Warnings will be logged if any error is encountered will scanning for hooks, # but it will not abort container execution. #guest_hook_path = "/usr/share/oci/hooks" [factory] # VM templating support. Once enabled, new VMs are created from template # using vm cloning. They will share the same initial kernel, initramfs and # agent memory by mapping it readonly. It helps speeding up new container # creation and saves a lot of memory if there are many kata containers running # on the same host. # # When disabled, new VMs are created from scratch. # # Note: Requires "initrd=" to be set ("image=" is not supported). # # Default false #enable_template = true # Specifies the path of template. # # Default "/run/vc/vm/template" #template_path = "/run/vc/vm/template" # The number of caches of VMCache: # unspecified or == 0 --> VMCache is disabled # > 0 --> will be set to the specified number # # VMCache is a function that creates VMs as caches before using it. # It helps speed up new container creation. # The function consists of a server and some clients communicating # through Unix socket. The protocol is gRPC in protocols/cache/cache.proto. # The VMCache server will create some VMs and cache them by factory cache. # It will convert the VM to gRPC format and transport it when gets # requestion from clients. # Factory grpccache is the VMCache client. It will request gRPC format # VM and convert it back to a VM. If VMCache function is enabled, # kata-runtime will request VM from factory grpccache when it creates # a new sandbox. # # Default 0 #vm_cache_number = 0 # Specify the address of the Unix socket that is used by VMCache. # # Default /var/run/kata-containers/cache.sock #vm_cache_endpoint = "/var/run/kata-containers/cache.sock" [proxy.kata] path = "/usr/libexec/kata-containers/kata-proxy" # If enabled, proxy messages will be sent to the system log # (default: disabled) enable_debug = true [shim.kata] path = "/usr/libexec/kata-containers/kata-shim" # If enabled, shim messages will be sent to the system log # (default: disabled) enable_debug = true # If enabled, the shim will create opentracing.io traces and spans. # (See https://www.jaegertracing.io/docs/getting-started). # # Note: By default, the shim runs in a separate network namespace. Therefore, # to allow it to send trace details to the Jaeger agent running on the host, # it is necessary to set 'disable_new_netns=true' so that it runs in the host # network namespace. # # (default: disabled) #enable_tracing = true [agent.kata] # If enabled, make the agent display debug-level messages. # (default: disabled) enable_debug = true # Enable agent tracing. # # If enabled, the default trace mode is "dynamic" and the # default trace type is "isolated". The trace mode and type are set # explicity with the `trace_type=` and `trace_mode=` options. # # Notes: # # - Tracing is ONLY enabled when `enable_tracing` is set: explicitly # setting `trace_mode=` and/or `trace_type=` without setting `enable_tracing` # will NOT activate agent tracing. # # - See https://github.com/kata-containers/agent/blob/master/TRACING.md for # full details. # # (default: disabled) #enable_tracing = true # #trace_mode = "dynamic" #trace_type = "isolated" # Comma separated list of kernel modules and their parameters. # These modules will be loaded in the guest kernel using modprobe(8). # The following example can be used to load two kernel modules with parameters # - kernel_modules=["e1000e InterruptThrottleRate=3000,3000,3000 EEE=1", "i915 enable_ppgtt=0"] # The first word is considered as the module name and the rest as its parameters. # Container will not be started when: # * A kernel module is specified and the modprobe command is not installed in the guest # or it fails loading the module. # * The module is not available in the guest or it doesn't met the guest kernel # requirements, like architecture and version. # kernel_modules=[] [netmon] # If enabled, the network monitoring process gets started when the # sandbox is created. This allows for the detection of some additional # network being added to the existing network namespace, after the # sandbox has been created. # (default: disabled) #enable_netmon = true # Specify the path to the netmon binary. path = "/usr/libexec/kata-containers/kata-netmon" # If enabled, netmon messages will be sent to the system log # (default: disabled) enable_debug = true [runtime] # If enabled, the runtime will log additional debug messages to the # system log # (default: disabled) enable_debug = true # # Internetworking model # Determines how the VM should be connected to the # the container network interface # Options: # # - macvtap # Used when the Container network interface can be bridged using # macvtap. # # - none # Used when customize network. Only creates a tap device. No veth pair. # # - tcfilter # Uses tc filter rules to redirect traffic from the network interface # provided by plugin to a tap interface connected to the VM. # internetworking_model="tcfilter" # disable guest seccomp # Determines whether container seccomp profiles are passed to the virtual # machine and applied by the kata agent. If set to true, seccomp is not applied # within the guest # (default: true) disable_guest_seccomp=true # If enabled, the runtime will create opentracing.io traces and spans. # (See https://www.jaegertracing.io/docs/getting-started). # (default: disabled) #enable_tracing = true # If enabled, the runtime will not create a network namespace for shim and hypervisor processes. # This option may have some potential impacts to your host. It should only be used when you know what you're doing. # `disable_new_netns` conflicts with `enable_netmon` # `disable_new_netns` conflicts with `internetworking_model=tcfilter` and `internetworking_model=macvtap`. It works only # with `internetworking_model=none`. The tap device will be in the host network namespace and can connect to a bridge # (like OVS) directly. # If you are using docker, `disable_new_netns` only works with `docker run --net=none` # (default: false) #disable_new_netns = true # if enabled, the runtime will add all the kata processes inside one dedicated cgroup. # The container cgroups in the host are not created, just one single cgroup per sandbox. # The runtime caller is free to restrict or collect cgroup stats of the overall Kata sandbox. # The sandbox cgroup path is the parent cgroup of a container with the PodSandbox annotation. # The sandbox cgroup is constrained if there is no container type annotation. # See: https://godoc.org/github.com/kata-containers/runtime/virtcontainers#ContainerType sandbox_cgroup_only=false # Enabled experimental feature list, format: ["a", "b"]. # Experimental features are features not stable enough for production, # they may break compatibility, and are prepared for a big version bump. # Supported experimental features: # (default: []) experimental=[] ``` Output of "`cat "/usr/share/defaults/kata-containers/configuration.toml"`": ```toml # Copyright (c) 2017-2019 Intel Corporation # # SPDX-License-Identifier: Apache-2.0 # # XXX: WARNING: this file is auto-generated. # XXX: # XXX: Source file: "cli/config/configuration-qemu.toml.in" # XXX: Project: # XXX: Name: Kata Containers # XXX: Type: kata [hypervisor.qemu] path = "/usr/bin/qemu-system-x86_64" kernel = "/usr/share/kata-containers/vmlinuz.container" initrd = "/usr/share/kata-containers/kata-containers-initrd.img" machine_type = "pc" # Optional space-separated list of options to pass to the guest kernel. # For example, use `kernel_params = "vsyscall=emulate"` if you are having # trouble running pre-2.15 glibc. # # WARNING: - any parameter specified here will take priority over the default # parameter value of the same name used to start the virtual machine. # Do not set values here unless you understand the impact of doing so as you # may stop the virtual machine from booting. # To see the list of default parameters, enable hypervisor debug, create a # container and look for 'default-kernel-parameters' log entries. kernel_params = " agent.log=debug" # Path to the firmware. # If you want that qemu uses the default firmware leave this option empty firmware = "" # Machine accelerators # comma-separated list of machine accelerators to pass to the hypervisor. # For example, `machine_accelerators = "nosmm,nosmbus,nosata,nopit,static-prt,nofw"` machine_accelerators="" # Default number of vCPUs per SB/VM: # unspecified or 0 --> will be set to 1 # < 0 --> will be set to the actual number of physical cores # > 0 <= number of physical cores --> will be set to the specified number # > number of physical cores --> will be set to the actual number of physical cores default_vcpus = 1 # Default maximum number of vCPUs per SB/VM: # unspecified or == 0 --> will be set to the actual number of physical cores or to the maximum number # of vCPUs supported by KVM if that number is exceeded # > 0 <= number of physical cores --> will be set to the specified number # > number of physical cores --> will be set to the actual number of physical cores or to the maximum number # of vCPUs supported by KVM if that number is exceeded # WARNING: Depending of the architecture, the maximum number of vCPUs supported by KVM is used when # the actual number of physical cores is greater than it. # WARNING: Be aware that this value impacts the virtual machine's memory footprint and CPU # the hotplug functionality. For example, `default_maxvcpus = 240` specifies that until 240 vCPUs # can be added to a SB/VM, but the memory footprint will be big. Another example, with # `default_maxvcpus = 8` the memory footprint will be small, but 8 will be the maximum number of # vCPUs supported by the SB/VM. In general, we recommend that you do not edit this variable, # unless you know what are you doing. default_maxvcpus = 0 # Bridges can be used to hot plug devices. # Limitations: # * Currently only pci bridges are supported # * Until 30 devices per bridge can be hot plugged. # * Until 5 PCI bridges can be cold plugged per VM. # This limitation could be a bug in qemu or in the kernel # Default number of bridges per SB/VM: # unspecified or 0 --> will be set to 1 # > 1 <= 5 --> will be set to the specified number # > 5 --> will be set to 5 default_bridges = 1 # Default memory size in MiB for SB/VM. # If unspecified then it will be set 2048 MiB. default_memory = 2048 # # Default memory slots per SB/VM. # If unspecified then it will be set 10. # This is will determine the times that memory will be hotadded to sandbox/VM. #memory_slots = 10 # The size in MiB will be plused to max memory of hypervisor. # It is the memory address space for the NVDIMM devie. # If set block storage driver (block_device_driver) to "nvdimm", # should set memory_offset to the size of block device. # Default 0 #memory_offset = 0 # Specifies virtio-mem will be enabled or not. # Please note that this option should be used with the command # "echo 1 > /proc/sys/vm/overcommit_memory". # Default false #enable_virtio_mem = true # Disable block device from being used for a container's rootfs. # In case of a storage driver like devicemapper where a container's # root file system is backed by a block device, the block device is passed # directly to the hypervisor for performance reasons. # This flag prevents the block device from being passed to the hypervisor, # 9pfs is used instead to pass the rootfs. disable_block_device_use = false # Shared file system type: # - virtio-9p (default) # - virtio-fs shared_fs = "virtio-9p" # Path to vhost-user-fs daemon. virtio_fs_daemon = "/usr/bin/virtiofsd" # Default size of DAX cache in MiB virtio_fs_cache_size = 1024 # Extra args for virtiofsd daemon # # Format example: # ["-o", "arg1=xxx,arg2", "-o", "hello world", "--arg3=yyy"] # # see `virtiofsd -h` for possible options. virtio_fs_extra_args = [] # Cache mode: # # - none # Metadata, data, and pathname lookup are not cached in guest. They are # always fetched from host and any changes are immediately pushed to host. # # - auto # Metadata and pathname lookup cache expires after a configured amount of # time (default is 1 second). Data is cached while the file is open (close # to open consistency). # # - always # Metadata, data, and pathname lookup are cached in guest and never expire. virtio_fs_cache = "always" # Block storage driver to be used for the hypervisor in case the container # rootfs is backed by a block device. This is virtio-scsi, virtio-blk # or nvdimm. block_device_driver = "virtio-scsi" # Specifies cache-related options will be set to block devices or not. # Default false #block_device_cache_set = true # Specifies cache-related options for block devices. # Denotes whether use of O_DIRECT (bypass the host page cache) is enabled. # Default false #block_device_cache_direct = true # Specifies cache-related options for block devices. # Denotes whether flush requests for the device are ignored. # Default false #block_device_cache_noflush = true # Enable iothreads (data-plane) to be used. This causes IO to be # handled in a separate IO thread. This is currently only implemented # for SCSI. # enable_iothreads = false # Enable pre allocation of VM RAM, default false # Enabling this will result in lower container density # as all of the memory will be allocated and locked # This is useful when you want to reserve all the memory # upfront or in the cases where you want memory latencies # to be very predictable # Default false #enable_mem_prealloc = true # Enable huge pages for VM RAM, default false # Enabling this will result in the VM memory # being allocated using huge pages. # This is useful when you want to use vhost-user network # stacks within the container. This will automatically # result in memory pre allocation #enable_hugepages = true # Enable file based guest memory support. The default is an empty string which # will disable this feature. In the case of virtio-fs, this is enabled # automatically and '/dev/shm' is used as the backing folder. # This option will be ignored if VM templating is enabled. #file_mem_backend = "" # Enable swap of vm memory. Default false. # The behaviour is undefined if mem_prealloc is also set to true #enable_swap = true # This option changes the default hypervisor and kernel parameters # to enable debug output where available. This extra output is added # to the proxy logs, but only when proxy debug is also enabled. # # Default false enable_debug = true # Disable the customizations done in the runtime when it detects # that it is running on top a VMM. This will result in the runtime # behaving as it would when running on bare metal. # #disable_nesting_checks = true # This is the msize used for 9p shares. It is the number of bytes # used for 9p packet payload. #msize_9p = 8192 # If true and vsocks are supported, use vsocks to communicate directly # with the agent and no proxy is started, otherwise use unix # sockets and start a proxy to communicate with the agent. # Default false #use_vsock = true # If false and nvdimm is supported, use nvdimm device to plug guest image. # Otherwise virtio-block device is used. # Default is false #disable_image_nvdimm = true # VFIO devices are hotplugged on a bridge by default. # Enable hotplugging on root bus. This may be required for devices with # a large PCI bar, as this is a current limitation with hotplugging on # a bridge. This value is valid for "pc" machine type. # Default false #hotplug_vfio_on_root_bus = true # Before hot plugging a PCIe device, you need to add a pcie_root_port device. # Use this parameter when using some large PCI bar devices, such as Nvidia GPU # The value means the number of pcie_root_port # This value is valid when hotplug_vfio_on_root_bus is true and machine_type is "q35" # Default 0 #pcie_root_port = 2 # If vhost-net backend for virtio-net is not desired, set to true. Default is false, which trades off # security (vhost-net runs ring0) for network I/O performance. #disable_vhost_net = true # # Default entropy source. # The path to a host source of entropy (including a real hardware RNG) # /dev/urandom and /dev/random are two main options. # Be aware that /dev/random is a blocking source of entropy. If the host # runs out of entropy, the VMs boot time will increase leading to get startup # timeouts. # The source of entropy /dev/urandom is non-blocking and provides a # generally acceptable source of entropy. It should work well for pretty much # all practical purposes. #entropy_source= "/dev/urandom" # Path to OCI hook binaries in the *guest rootfs*. # This does not affect host-side hooks which must instead be added to # the OCI spec passed to the runtime. # # You can create a rootfs with hooks by customizing the osbuilder scripts: # https://github.com/kata-containers/osbuilder # # Hooks must be stored in a subdirectory of guest_hook_path according to their # hook type, i.e. "guest_hook_path/{prestart,postart,poststop}". # The agent will scan these directories for executable files and add them, in # lexicographical order, to the lifecycle of the guest container. # Hooks are executed in the runtime namespace of the guest. See the official documentation: # https://github.com/opencontainers/runtime-spec/blob/v1.0.1/config.md#posix-platform-hooks # Warnings will be logged if any error is encountered will scanning for hooks, # but it will not abort container execution. #guest_hook_path = "/usr/share/oci/hooks" [factory] # VM templating support. Once enabled, new VMs are created from template # using vm cloning. They will share the same initial kernel, initramfs and # agent memory by mapping it readonly. It helps speeding up new container # creation and saves a lot of memory if there are many kata containers running # on the same host. # # When disabled, new VMs are created from scratch. # # Note: Requires "initrd=" to be set ("image=" is not supported). # # Default false #enable_template = true # Specifies the path of template. # # Default "/run/vc/vm/template" #template_path = "/run/vc/vm/template" # The number of caches of VMCache: # unspecified or == 0 --> VMCache is disabled # > 0 --> will be set to the specified number # # VMCache is a function that creates VMs as caches before using it. # It helps speed up new container creation. # The function consists of a server and some clients communicating # through Unix socket. The protocol is gRPC in protocols/cache/cache.proto. # The VMCache server will create some VMs and cache them by factory cache. # It will convert the VM to gRPC format and transport it when gets # requestion from clients. # Factory grpccache is the VMCache client. It will request gRPC format # VM and convert it back to a VM. If VMCache function is enabled, # kata-runtime will request VM from factory grpccache when it creates # a new sandbox. # # Default 0 #vm_cache_number = 0 # Specify the address of the Unix socket that is used by VMCache. # # Default /var/run/kata-containers/cache.sock #vm_cache_endpoint = "/var/run/kata-containers/cache.sock" [proxy.kata] path = "/usr/libexec/kata-containers/kata-proxy" # If enabled, proxy messages will be sent to the system log # (default: disabled) enable_debug = true [shim.kata] path = "/usr/libexec/kata-containers/kata-shim" # If enabled, shim messages will be sent to the system log # (default: disabled) enable_debug = true # If enabled, the shim will create opentracing.io traces and spans. # (See https://www.jaegertracing.io/docs/getting-started). # # Note: By default, the shim runs in a separate network namespace. Therefore, # to allow it to send trace details to the Jaeger agent running on the host, # it is necessary to set 'disable_new_netns=true' so that it runs in the host # network namespace. # # (default: disabled) #enable_tracing = true [agent.kata] # If enabled, make the agent display debug-level messages. # (default: disabled) enable_debug = true # Enable agent tracing. # # If enabled, the default trace mode is "dynamic" and the # default trace type is "isolated". The trace mode and type are set # explicity with the `trace_type=` and `trace_mode=` options. # # Notes: # # - Tracing is ONLY enabled when `enable_tracing` is set: explicitly # setting `trace_mode=` and/or `trace_type=` without setting `enable_tracing` # will NOT activate agent tracing. # # - See https://github.com/kata-containers/agent/blob/master/TRACING.md for # full details. # # (default: disabled) #enable_tracing = true # #trace_mode = "dynamic" #trace_type = "isolated" # Comma separated list of kernel modules and their parameters. # These modules will be loaded in the guest kernel using modprobe(8). # The following example can be used to load two kernel modules with parameters # - kernel_modules=["e1000e InterruptThrottleRate=3000,3000,3000 EEE=1", "i915 enable_ppgtt=0"] # The first word is considered as the module name and the rest as its parameters. # Container will not be started when: # * A kernel module is specified and the modprobe command is not installed in the guest # or it fails loading the module. # * The module is not available in the guest or it doesn't met the guest kernel # requirements, like architecture and version. # kernel_modules=[] [netmon] # If enabled, the network monitoring process gets started when the # sandbox is created. This allows for the detection of some additional # network being added to the existing network namespace, after the # sandbox has been created. # (default: disabled) #enable_netmon = true # Specify the path to the netmon binary. path = "/usr/libexec/kata-containers/kata-netmon" # If enabled, netmon messages will be sent to the system log # (default: disabled) enable_debug = true [runtime] # If enabled, the runtime will log additional debug messages to the # system log # (default: disabled) enable_debug = true # # Internetworking model # Determines how the VM should be connected to the # the container network interface # Options: # # - macvtap # Used when the Container network interface can be bridged using # macvtap. # # - none # Used when customize network. Only creates a tap device. No veth pair. # # - tcfilter # Uses tc filter rules to redirect traffic from the network interface # provided by plugin to a tap interface connected to the VM. # internetworking_model="tcfilter" # disable guest seccomp # Determines whether container seccomp profiles are passed to the virtual # machine and applied by the kata agent. If set to true, seccomp is not applied # within the guest # (default: true) disable_guest_seccomp=true # If enabled, the runtime will create opentracing.io traces and spans. # (See https://www.jaegertracing.io/docs/getting-started). # (default: disabled) #enable_tracing = true # If enabled, the runtime will not create a network namespace for shim and hypervisor processes. # This option may have some potential impacts to your host. It should only be used when you know what you're doing. # `disable_new_netns` conflicts with `enable_netmon` # `disable_new_netns` conflicts with `internetworking_model=tcfilter` and `internetworking_model=macvtap`. It works only # with `internetworking_model=none`. The tap device will be in the host network namespace and can connect to a bridge # (like OVS) directly. # If you are using docker, `disable_new_netns` only works with `docker run --net=none` # (default: false) #disable_new_netns = true # if enabled, the runtime will add all the kata processes inside one dedicated cgroup. # The container cgroups in the host are not created, just one single cgroup per sandbox. # The runtime caller is free to restrict or collect cgroup stats of the overall Kata sandbox. # The sandbox cgroup path is the parent cgroup of a container with the PodSandbox annotation. # The sandbox cgroup is constrained if there is no container type annotation. # See: https://godoc.org/github.com/kata-containers/runtime/virtcontainers#ContainerType sandbox_cgroup_only=false # Enabled experimental feature list, format: ["a", "b"]. # Experimental features are features not stable enough for production, # they may break compatibility, and are prepared for a big version bump. # Supported experimental features: # (default: []) experimental=[] ``` --- # KSM throttler ## version find: ‘/usr/lib*’: No such file or directory Output of "` --version`": ``` kata-collect-data.sh: line 178: --version: command not found ``` ## systemd service # Image details No image --- # Initrd details ```yaml --- osbuilder: url: "https://github.com/kata-containers/osbuilder" version: "unknown" rootfs-creation-time: "2020-03-17T07:32:25.639302038+0000Z" description: "osbuilder rootfs" file-format-version: "0.0.2" architecture: "x86_64" base-distro: name: "Alpine" version: "latest-stable" packages: default: - "iptables" extra: agent: url: "https://github.com/kata-containers/agent" name: "kata-agent" version: "1.11.0-alpha0-2725ddac66124d669838437705376f21efcc6ed0" agent-is-init-daemon: "yes" ``` --- # Logfiles ## Runtime logs Recent runtime problems found in system journal: ``` time="2020-03-17T17:50:55.853372543Z" level=warning msg="sandbox's cgroup won't be updated: cgroup path is empty" arch=amd64 command=create container=50a87dd6139cd3158d413c1c8e5460610a283381ef9e1150378b48f0a5f9045d name=kata-runtime pid=213117 sandbox=50a87dd6139cd3158d413c1c8e5460610a283381ef9e1150378b48f0a5f9045d source=virtcontainers subsystem=sandbox time="2020-03-17T17:50:55.854459048Z" level=info msg="sanner return error: read unix @->/run/user/1000/run/vc/vm/50a87dd6139cd3158d413c1c8e5460610a283381ef9e1150378b48f0a5f9045d/qmp.sock: use of closed network connection" arch=amd64 command=create container=50a87dd6139cd3158d413c1c8e5460610a283381ef9e1150378b48f0a5f9045d name=kata-runtime pid=213117 source=virtcontainers subsystem=qmp time="2020-03-17T17:50:55.904562056Z" level=warning msg="failed to get sandbox config from old store: open /run/user/1000/var/lib/vc/sbs/50a87dd6139cd3158d413c1c8e5460610a283381ef9e1150378b48f0a5f9045d/config.json: no such file or directory" arch=amd64 command=start container=50a87dd6139cd3158d413c1c8e5460610a283381ef9e1150378b48f0a5f9045d name=kata-runtime pid=213163 source=virtcontainers time="2020-03-17T17:50:55.905860862Z" level=error msg="unknown endpoint type" arch=amd64 command=start container=50a87dd6139cd3158d413c1c8e5460610a283381ef9e1150378b48f0a5f9045d endpoint-type=tuntap name=kata-runtime pid=213163 sandbox=50a87dd6139cd3158d413c1c8e5460610a283381ef9e1150378b48f0a5f9045d source=virtcontainers subsystem=sandbox time="2020-03-17T17:50:55.90770667Z" level=warning msg="failed to get sandbox config from old store: open /run/user/1000/var/lib/vc/sbs/50a87dd6139cd3158d413c1c8e5460610a283381ef9e1150378b48f0a5f9045d/config.json: no such file or directory" arch=amd64 command=start container=50a87dd6139cd3158d413c1c8e5460610a283381ef9e1150378b48f0a5f9045d name=kata-runtime pid=213163 sandbox=50a87dd6139cd3158d413c1c8e5460610a283381ef9e1150378b48f0a5f9045d source=virtcontainers time="2020-03-17T17:50:55.908286472Z" level=error msg="unknown endpoint type" arch=amd64 command=start container=50a87dd6139cd3158d413c1c8e5460610a283381ef9e1150378b48f0a5f9045d endpoint-type=tuntap name=kata-runtime pid=213163 sandbox=50a87dd6139cd3158d413c1c8e5460610a283381ef9e1150378b48f0a5f9045d source=virtcontainers subsystem=sandbox time="2020-03-17T17:50:56.170292563Z" level=warning msg="failed to get sandbox config from old store: open /run/user/1000/var/lib/vc/sbs/50a87dd6139cd3158d413c1c8e5460610a283381ef9e1150378b48f0a5f9045d/config.json: no such file or directory" arch=amd64 command=delete container=50a87dd6139cd3158d413c1c8e5460610a283381ef9e1150378b48f0a5f9045d name=kata-runtime pid=213206 source=virtcontainers time="2020-03-17T17:50:56.184182821Z" level=warning msg="failed to get sandbox config from old store: open /run/user/1000/var/lib/vc/sbs/50a87dd6139cd3158d413c1c8e5460610a283381ef9e1150378b48f0a5f9045d/config.json: no such file or directory" arch=amd64 command=delete container=50a87dd6139cd3158d413c1c8e5460610a283381ef9e1150378b48f0a5f9045d name=kata-runtime pid=213206 sandbox=50a87dd6139cd3158d413c1c8e5460610a283381ef9e1150378b48f0a5f9045d source=virtcontainers time="2020-03-17T17:50:56.18644493Z" level=warning msg="failed to get sandbox config from old store: open /run/user/1000/var/lib/vc/sbs/50a87dd6139cd3158d413c1c8e5460610a283381ef9e1150378b48f0a5f9045d/config.json: no such file or directory" arch=amd64 command=delete container=50a87dd6139cd3158d413c1c8e5460610a283381ef9e1150378b48f0a5f9045d name=kata-runtime pid=213206 sandbox=50a87dd6139cd3158d413c1c8e5460610a283381ef9e1150378b48f0a5f9045d source=virtcontainers time="2020-03-17T17:50:56.205985812Z" level=error msg="Could not read qemu pid file" arch=amd64 command=delete container=50a87dd6139cd3158d413c1c8e5460610a283381ef9e1150378b48f0a5f9045d error="open /run/user/1000/run/vc/vm/50a87dd6139cd3158d413c1c8e5460610a283381ef9e1150378b48f0a5f9045d/pid: no such file or directory" name=kata-runtime pid=213206 sandbox=50a87dd6139cd3158d413c1c8e5460610a283381ef9e1150378b48f0a5f9045d source=virtcontainers subsystem=qemu time="2020-03-17T17:50:56.206272313Z" level=error msg="Could not read qemu pid file" arch=amd64 command=delete container=50a87dd6139cd3158d413c1c8e5460610a283381ef9e1150378b48f0a5f9045d error="open /run/user/1000/run/vc/vm/50a87dd6139cd3158d413c1c8e5460610a283381ef9e1150378b48f0a5f9045d/pid: no such file or directory" name=kata-runtime pid=213206 sandbox=50a87dd6139cd3158d413c1c8e5460610a283381ef9e1150378b48f0a5f9045d source=virtcontainers subsystem=qemu time="2020-03-17T17:50:56.206561414Z" level=info msg="sanner return error: read unix @->/run/user/1000/run/vc/vm/50a87dd6139cd3158d413c1c8e5460610a283381ef9e1150378b48f0a5f9045d/qmp.sock: use of closed network connection" arch=amd64 command=delete container=50a87dd6139cd3158d413c1c8e5460610a283381ef9e1150378b48f0a5f9045d name=kata-runtime pid=213206 sandbox=50a87dd6139cd3158d413c1c8e5460610a283381ef9e1150378b48f0a5f9045d source=virtcontainers subsystem=qmp time="2020-03-17T17:50:56.206770215Z" level=warning msg="failed to get sandbox config from old store: open /run/user/1000/var/lib/vc/sbs/50a87dd6139cd3158d413c1c8e5460610a283381ef9e1150378b48f0a5f9045d/config.json: no such file or directory" arch=amd64 command=delete container=50a87dd6139cd3158d413c1c8e5460610a283381ef9e1150378b48f0a5f9045d name=kata-runtime pid=213206 sandbox=50a87dd6139cd3158d413c1c8e5460610a283381ef9e1150378b48f0a5f9045d source=virtcontainers time="2020-03-17T17:50:56.212307238Z" level=error msg="Could not read qemu pid file" arch=amd64 command=delete container=50a87dd6139cd3158d413c1c8e5460610a283381ef9e1150378b48f0a5f9045d error="open /run/user/1000/run/vc/vm/50a87dd6139cd3158d413c1c8e5460610a283381ef9e1150378b48f0a5f9045d/pid: no such file or directory" name=kata-runtime pid=213206 sandbox=50a87dd6139cd3158d413c1c8e5460610a283381ef9e1150378b48f0a5f9045d source=virtcontainers subsystem=qemu time="2020-03-17T17:50:58.336685585Z" level=warning msg="Failed to get container, force will not fail: Container ID (50a87dd6139cd3158d413c1c8e5460610a283381ef9e1150378b48f0a5f9045d) does not exist" arch=amd64 command=delete container=50a87dd6139cd3158d413c1c8e5460610a283381ef9e1150378b48f0a5f9045d name=kata-runtime pid=213291 source=runtime time="2020-03-17T17:50:58.810524058Z" level=debug msg="restore sandbox failed" arch=amd64 command=create container=15288558b81ea3a58b706746f9825aa4137cd10bb39263bc353e622ba3153eb3 error="open /run/user/1000/run/vc/sbs/15288558b81ea3a58b706746f9825aa4137cd10bb39263bc353e622ba3153eb3/persist.json: no such file or directory" name=kata-runtime pid=213318 sandbox=15288558b81ea3a58b706746f9825aa4137cd10bb39263bc353e622ba3153eb3 source=virtcontainers subsystem=sandbox time="2020-03-17T17:50:58.905110352Z" level=info msg="sanner return error: read unix @->/run/user/1000/run/vc/vm/15288558b81ea3a58b706746f9825aa4137cd10bb39263bc353e622ba3153eb3/qmp.sock: use of closed network connection" arch=amd64 command=create container=15288558b81ea3a58b706746f9825aa4137cd10bb39263bc353e622ba3153eb3 name=kata-runtime pid=213318 source=virtcontainers subsystem=qmp time="2020-03-17T17:50:59.728827283Z" level=warning msg="sandbox's cgroup won't be updated: cgroup path is empty" arch=amd64 command=create container=15288558b81ea3a58b706746f9825aa4137cd10bb39263bc353e622ba3153eb3 name=kata-runtime pid=213318 sandbox=15288558b81ea3a58b706746f9825aa4137cd10bb39263bc353e622ba3153eb3 source=virtcontainers subsystem=sandbox time="2020-03-17T17:50:59.729662286Z" level=info msg="sanner return error: read unix @->/run/user/1000/run/vc/vm/15288558b81ea3a58b706746f9825aa4137cd10bb39263bc353e622ba3153eb3/qmp.sock: use of closed network connection" arch=amd64 command=create container=15288558b81ea3a58b706746f9825aa4137cd10bb39263bc353e622ba3153eb3 name=kata-runtime pid=213318 source=virtcontainers subsystem=qmp time="2020-03-17T17:50:59.773110567Z" level=warning msg="failed to get sandbox config from old store: open /run/user/1000/var/lib/vc/sbs/15288558b81ea3a58b706746f9825aa4137cd10bb39263bc353e622ba3153eb3/config.json: no such file or directory" arch=amd64 command=start container=15288558b81ea3a58b706746f9825aa4137cd10bb39263bc353e622ba3153eb3 name=kata-runtime pid=213367 source=virtcontainers time="2020-03-17T17:50:59.774537273Z" level=error msg="unknown endpoint type" arch=amd64 command=start container=15288558b81ea3a58b706746f9825aa4137cd10bb39263bc353e622ba3153eb3 endpoint-type=tuntap name=kata-runtime pid=213367 sandbox=15288558b81ea3a58b706746f9825aa4137cd10bb39263bc353e622ba3153eb3 source=virtcontainers subsystem=sandbox time="2020-03-17T17:50:59.776843183Z" level=warning msg="failed to get sandbox config from old store: open /run/user/1000/var/lib/vc/sbs/15288558b81ea3a58b706746f9825aa4137cd10bb39263bc353e622ba3153eb3/config.json: no such file or directory" arch=amd64 command=start container=15288558b81ea3a58b706746f9825aa4137cd10bb39263bc353e622ba3153eb3 name=kata-runtime pid=213367 sandbox=15288558b81ea3a58b706746f9825aa4137cd10bb39263bc353e622ba3153eb3 source=virtcontainers time="2020-03-17T17:50:59.777364085Z" level=error msg="unknown endpoint type" arch=amd64 command=start container=15288558b81ea3a58b706746f9825aa4137cd10bb39263bc353e622ba3153eb3 endpoint-type=tuntap name=kata-runtime pid=213367 sandbox=15288558b81ea3a58b706746f9825aa4137cd10bb39263bc353e622ba3153eb3 source=virtcontainers subsystem=sandbox time="2020-03-17T17:50:59.963814461Z" level=warning msg="failed to get sandbox config from old store: open /run/user/1000/var/lib/vc/sbs/15288558b81ea3a58b706746f9825aa4137cd10bb39263bc353e622ba3153eb3/config.json: no such file or directory" arch=amd64 command=delete container=15288558b81ea3a58b706746f9825aa4137cd10bb39263bc353e622ba3153eb3 name=kata-runtime pid=213398 source=virtcontainers time="2020-03-17T17:50:59.980101829Z" level=warning msg="failed to get sandbox config from old store: open /run/user/1000/var/lib/vc/sbs/15288558b81ea3a58b706746f9825aa4137cd10bb39263bc353e622ba3153eb3/config.json: no such file or directory" arch=amd64 command=delete container=15288558b81ea3a58b706746f9825aa4137cd10bb39263bc353e622ba3153eb3 name=kata-runtime pid=213398 sandbox=15288558b81ea3a58b706746f9825aa4137cd10bb39263bc353e622ba3153eb3 source=virtcontainers time="2020-03-17T17:50:59.98262924Z" level=warning msg="failed to get sandbox config from old store: open /run/user/1000/var/lib/vc/sbs/15288558b81ea3a58b706746f9825aa4137cd10bb39263bc353e622ba3153eb3/config.json: no such file or directory" arch=amd64 command=delete container=15288558b81ea3a58b706746f9825aa4137cd10bb39263bc353e622ba3153eb3 name=kata-runtime pid=213398 sandbox=15288558b81ea3a58b706746f9825aa4137cd10bb39263bc353e622ba3153eb3 source=virtcontainers time="2020-03-17T17:51:00.002226421Z" level=error msg="Could not read qemu pid file" arch=amd64 command=delete container=15288558b81ea3a58b706746f9825aa4137cd10bb39263bc353e622ba3153eb3 error="open /run/user/1000/run/vc/vm/15288558b81ea3a58b706746f9825aa4137cd10bb39263bc353e622ba3153eb3/pid: no such file or directory" name=kata-runtime pid=213398 sandbox=15288558b81ea3a58b706746f9825aa4137cd10bb39263bc353e622ba3153eb3 source=virtcontainers subsystem=qemu time="2020-03-17T17:51:00.002848424Z" level=error msg="Could not read qemu pid file" arch=amd64 command=delete container=15288558b81ea3a58b706746f9825aa4137cd10bb39263bc353e622ba3153eb3 error="open /run/user/1000/run/vc/vm/15288558b81ea3a58b706746f9825aa4137cd10bb39263bc353e622ba3153eb3/pid: no such file or directory" name=kata-runtime pid=213398 sandbox=15288558b81ea3a58b706746f9825aa4137cd10bb39263bc353e622ba3153eb3 source=virtcontainers subsystem=qemu time="2020-03-17T17:51:00.003259626Z" level=info msg="sanner return error: read unix @->/run/user/1000/run/vc/vm/15288558b81ea3a58b706746f9825aa4137cd10bb39263bc353e622ba3153eb3/qmp.sock: use of closed network connection" arch=amd64 command=delete container=15288558b81ea3a58b706746f9825aa4137cd10bb39263bc353e622ba3153eb3 name=kata-runtime pid=213398 sandbox=15288558b81ea3a58b706746f9825aa4137cd10bb39263bc353e622ba3153eb3 source=virtcontainers subsystem=qmp time="2020-03-17T17:51:00.003502627Z" level=warning msg="failed to get sandbox config from old store: open /run/user/1000/var/lib/vc/sbs/15288558b81ea3a58b706746f9825aa4137cd10bb39263bc353e622ba3153eb3/config.json: no such file or directory" arch=amd64 command=delete container=15288558b81ea3a58b706746f9825aa4137cd10bb39263bc353e622ba3153eb3 name=kata-runtime pid=213398 sandbox=15288558b81ea3a58b706746f9825aa4137cd10bb39263bc353e622ba3153eb3 source=virtcontainers time="2020-03-17T17:51:00.006162238Z" level=error msg="Could not read qemu pid file" arch=amd64 command=delete container=15288558b81ea3a58b706746f9825aa4137cd10bb39263bc353e622ba3153eb3 error="open /run/user/1000/run/vc/vm/15288558b81ea3a58b706746f9825aa4137cd10bb39263bc353e622ba3153eb3/pid: no such file or directory" name=kata-runtime pid=213398 sandbox=15288558b81ea3a58b706746f9825aa4137cd10bb39263bc353e622ba3153eb3 source=virtcontainers subsystem=qemu time="2020-03-17T17:51:02.12952128Z" level=warning msg="Failed to get container, force will not fail: Container ID (15288558b81ea3a58b706746f9825aa4137cd10bb39263bc353e622ba3153eb3) does not exist" arch=amd64 command=delete container=15288558b81ea3a58b706746f9825aa4137cd10bb39263bc353e622ba3153eb3 name=kata-runtime pid=213484 source=runtime time="2020-03-17T17:51:04.426536446Z" level=debug msg="restore sandbox failed" arch=amd64 command=create container=992b94d81a146c5d122715cd970ec29221bad138201dd6fae14fd3f74061477c error="open /run/user/1000/run/vc/sbs/992b94d81a146c5d122715cd970ec29221bad138201dd6fae14fd3f74061477c/persist.json: no such file or directory" name=kata-runtime pid=213733 sandbox=992b94d81a146c5d122715cd970ec29221bad138201dd6fae14fd3f74061477c source=virtcontainers subsystem=sandbox time="2020-03-17T17:51:04.515760518Z" level=info msg="sanner return error: read unix @->/run/user/1000/run/vc/vm/992b94d81a146c5d122715cd970ec29221bad138201dd6fae14fd3f74061477c/qmp.sock: use of closed network connection" arch=amd64 command=create container=992b94d81a146c5d122715cd970ec29221bad138201dd6fae14fd3f74061477c name=kata-runtime pid=213733 source=virtcontainers subsystem=qmp time="2020-03-17T17:51:05.439886966Z" level=warning msg="sandbox's cgroup won't be updated: cgroup path is empty" arch=amd64 command=create container=992b94d81a146c5d122715cd970ec29221bad138201dd6fae14fd3f74061477c name=kata-runtime pid=213733 sandbox=992b94d81a146c5d122715cd970ec29221bad138201dd6fae14fd3f74061477c source=virtcontainers subsystem=sandbox time="2020-03-17T17:51:05.44066667Z" level=info msg="sanner return error: read unix @->/run/user/1000/run/vc/vm/992b94d81a146c5d122715cd970ec29221bad138201dd6fae14fd3f74061477c/qmp.sock: use of closed network connection" arch=amd64 command=create container=992b94d81a146c5d122715cd970ec29221bad138201dd6fae14fd3f74061477c name=kata-runtime pid=213733 source=virtcontainers subsystem=qmp time="2020-03-17T17:51:05.664929304Z" level=warning msg="failed to get sandbox config from old store: open /run/user/1000/var/lib/vc/sbs/992b94d81a146c5d122715cd970ec29221bad138201dd6fae14fd3f74061477c/config.json: no such file or directory" arch=amd64 command=kill container=992b94d81a146c5d122715cd970ec29221bad138201dd6fae14fd3f74061477c name=kata-runtime pid=213802 source=virtcontainers time="2020-03-17T17:51:05.666326609Z" level=error msg="unknown endpoint type" arch=amd64 command=kill container=992b94d81a146c5d122715cd970ec29221bad138201dd6fae14fd3f74061477c endpoint-type=tuntap name=kata-runtime pid=213802 sandbox=992b94d81a146c5d122715cd970ec29221bad138201dd6fae14fd3f74061477c source=virtcontainers subsystem=sandbox time="2020-03-17T17:51:05.668326218Z" level=warning msg="failed to get sandbox config from old store: open /run/user/1000/var/lib/vc/sbs/992b94d81a146c5d122715cd970ec29221bad138201dd6fae14fd3f74061477c/config.json: no such file or directory" arch=amd64 command=kill container=992b94d81a146c5d122715cd970ec29221bad138201dd6fae14fd3f74061477c name=kata-runtime pid=213802 sandbox=992b94d81a146c5d122715cd970ec29221bad138201dd6fae14fd3f74061477c source=virtcontainers time="2020-03-17T17:51:05.671729332Z" level=error msg="unknown endpoint type" arch=amd64 command=kill container=992b94d81a146c5d122715cd970ec29221bad138201dd6fae14fd3f74061477c endpoint-type=tuntap name=kata-runtime pid=213802 sandbox=992b94d81a146c5d122715cd970ec29221bad138201dd6fae14fd3f74061477c source=virtcontainers subsystem=sandbox time="2020-03-17T17:51:05.677064954Z" level=warning msg="failed to get sandbox config from old store: open /run/user/1000/var/lib/vc/sbs/992b94d81a146c5d122715cd970ec29221bad138201dd6fae14fd3f74061477c/config.json: no such file or directory" arch=amd64 command=kill container=992b94d81a146c5d122715cd970ec29221bad138201dd6fae14fd3f74061477c name=kata-runtime pid=213802 sandbox=992b94d81a146c5d122715cd970ec29221bad138201dd6fae14fd3f74061477c source=virtcontainers time="2020-03-17T17:51:05.678083458Z" level=error msg="unknown endpoint type" arch=amd64 command=kill container=992b94d81a146c5d122715cd970ec29221bad138201dd6fae14fd3f74061477c endpoint-type=tuntap name=kata-runtime pid=213802 sandbox=992b94d81a146c5d122715cd970ec29221bad138201dd6fae14fd3f74061477c source=virtcontainers subsystem=sandbox time="2020-03-17T17:51:05.814283726Z" level=error msg="Could not read qemu pid file" arch=amd64 command=kill container=992b94d81a146c5d122715cd970ec29221bad138201dd6fae14fd3f74061477c error="open /run/user/1000/run/vc/vm/992b94d81a146c5d122715cd970ec29221bad138201dd6fae14fd3f74061477c/pid: no such file or directory" name=kata-runtime pid=213802 sandbox=992b94d81a146c5d122715cd970ec29221bad138201dd6fae14fd3f74061477c source=virtcontainers subsystem=qemu time="2020-03-17T17:51:05.814530427Z" level=error msg="Could not read qemu pid file" arch=amd64 command=kill container=992b94d81a146c5d122715cd970ec29221bad138201dd6fae14fd3f74061477c error="open /run/user/1000/run/vc/vm/992b94d81a146c5d122715cd970ec29221bad138201dd6fae14fd3f74061477c/pid: no such file or directory" name=kata-runtime pid=213802 sandbox=992b94d81a146c5d122715cd970ec29221bad138201dd6fae14fd3f74061477c source=virtcontainers subsystem=qemu time="2020-03-17T17:51:05.814863428Z" level=info msg="sanner return error: read unix @->/run/user/1000/run/vc/vm/992b94d81a146c5d122715cd970ec29221bad138201dd6fae14fd3f74061477c/qmp.sock: use of closed network connection" arch=amd64 command=kill container=992b94d81a146c5d122715cd970ec29221bad138201dd6fae14fd3f74061477c name=kata-runtime pid=213802 sandbox=992b94d81a146c5d122715cd970ec29221bad138201dd6fae14fd3f74061477c source=virtcontainers subsystem=qmp time="2020-03-17T17:51:07.921443301Z" level=warning msg="failed to get sandbox config from old store: open /run/user/1000/var/lib/vc/sbs/992b94d81a146c5d122715cd970ec29221bad138201dd6fae14fd3f74061477c/config.json: no such file or directory" arch=amd64 command=delete container=992b94d81a146c5d122715cd970ec29221bad138201dd6fae14fd3f74061477c name=kata-runtime pid=213837 source=virtcontainers time="2020-03-17T17:51:07.925090316Z" level=warning msg="failed to get sandbox config from old store: open /run/user/1000/var/lib/vc/sbs/992b94d81a146c5d122715cd970ec29221bad138201dd6fae14fd3f74061477c/config.json: no such file or directory" arch=amd64 command=delete container=992b94d81a146c5d122715cd970ec29221bad138201dd6fae14fd3f74061477c name=kata-runtime pid=213837 sandbox=992b94d81a146c5d122715cd970ec29221bad138201dd6fae14fd3f74061477c source=virtcontainers time="2020-03-17T17:51:07.927375226Z" level=warning msg="failed to get sandbox config from old store: open /run/user/1000/var/lib/vc/sbs/992b94d81a146c5d122715cd970ec29221bad138201dd6fae14fd3f74061477c/config.json: no such file or directory" arch=amd64 command=delete container=992b94d81a146c5d122715cd970ec29221bad138201dd6fae14fd3f74061477c name=kata-runtime pid=213837 sandbox=992b94d81a146c5d122715cd970ec29221bad138201dd6fae14fd3f74061477c source=virtcontainers time="2020-03-17T17:51:07.930173937Z" level=error msg="Could not read qemu pid file" arch=amd64 command=delete container=992b94d81a146c5d122715cd970ec29221bad138201dd6fae14fd3f74061477c error="open /run/user/1000/run/vc/vm/992b94d81a146c5d122715cd970ec29221bad138201dd6fae14fd3f74061477c/pid: no such file or directory" name=kata-runtime pid=213837 sandbox=992b94d81a146c5d122715cd970ec29221bad138201dd6fae14fd3f74061477c source=virtcontainers subsystem=qemu time="2020-03-17T17:51:08.164346612Z" level=warning msg="Failed to get container, force will not fail: Container ID (992b94d81a146c5d122715cd970ec29221bad138201dd6fae14fd3f74061477c) does not exist" arch=amd64 command=delete container=992b94d81a146c5d122715cd970ec29221bad138201dd6fae14fd3f74061477c name=kata-runtime pid=213874 source=runtime ``` ## Proxy logs Recent proxy problems found in system journal: ``` time="2020-03-17T17:50:39.021226746Z" level=info msg="time=\"2020-03-17T17:50:39.009371568Z\" level=info msg=\"ignoring unexpected signal\" debug_console=false name=kata-agent pid=1 sandbox=4d4a0cfcb196cae46cd97a7a0495945bd201a66267d237c432f1ea42daf07a85 signal=\"child exited\" source=agent\n" name=kata-proxy pid=212039 sandbox=4d4a0cfcb196cae46cd97a7a0495945bd201a66267d237c432f1ea42daf07a85 source=agent time="2020-03-17T17:50:49.656791138Z" level=info msg="[ 0.463510] intel_pstate: CPU model not supported\n" name=kata-proxy pid=212943 sandbox=1284226a23bf29f23c0228907d4080b3a314cccece0b1304adb22f878c3ecf28 source=agent time="2020-03-17T17:50:49.696757404Z" level=info msg="time=\"2020-03-17T17:50:49.685743648Z\" level=debug msg=\"request end\" debug_console=false duration=2.02482ms name=kata-agent pid=1 request=/grpc.AgentService/CreateSandbox resp=\"&Empty{XXX_unrecognized:[],}\" sandbox=1284226a23bf29f23c0228907d4080b3a314cccece0b1304adb22f878c3ecf28 source=agent\n" name=kata-proxy pid=212943 sandbox=1284226a23bf29f23c0228907d4080b3a314cccece0b1304adb22f878c3ecf28 source=agent time="2020-03-17T17:50:49.790333094Z" level=info msg="time=\"2020-03-17T17:50:49.779358402Z\" level=debug msg=\"request end\" debug_console=false duration=84.697562ms name=kata-agent pid=1 request=/grpc.AgentService/CreateContainer resp=\"&Empty{XXX_unrecognized:[],}\" sandbox=1284226a23bf29f23c0228907d4080b3a314cccece0b1304adb22f878c3ecf28 source=agent\n" name=kata-proxy pid=212943 sandbox=1284226a23bf29f23c0228907d4080b3a314cccece0b1304adb22f878c3ecf28 source=agent time="2020-03-17T17:50:49.790691195Z" level=info msg="time=\"2020-03-17T17:50:49.77967681Z\" level=info msg=\"ignoring unexpected signal\" debug_console=false name=kata-agent pid=1 sandbox=1284226a23bf29f23c0228907d4080b3a314cccece0b1304adb22f878c3ecf28 signal=\"child exited\" source=agent\n" name=kata-proxy pid=212943 sandbox=1284226a23bf29f23c0228907d4080b3a314cccece0b1304adb22f878c3ecf28 source=agent time="2020-03-17T17:50:49.790887496Z" level=info msg="time=\"2020-03-17T17:50:49.779902783Z\" level=info msg=\"ignoring unexpected signal\" debug_console=false name=kata-agent pid=1 sandbox=1284226a23bf29f23c0228907d4080b3a314cccece0b1304adb22f878c3ecf28 signal=\"child exited\" source=agent\n" name=kata-proxy pid=212943 sandbox=1284226a23bf29f23c0228907d4080b3a314cccece0b1304adb22f878c3ecf28 source=agent time="2020-03-17T17:50:49.791417798Z" level=debug msg="Copy stream error" error="write unix /run/user/1000/run/vc/sbs/1284226a23bf29f23c0228907d4080b3a314cccece0b1304adb22f878c3ecf28/proxy.sock->@: write: broken pipe" name=kata-proxy pid=212943 sandbox=1284226a23bf29f23c0228907d4080b3a314cccece0b1304adb22f878c3ecf28 source=proxy time="2020-03-17T17:50:49.797752625Z" level=info msg="time=\"2020-03-17T17:50:49.78671683Z\" level=debug msg=\"request end\" debug_console=false duration=\"2.762µs\" name=kata-agent pid=1 request=/grpc.AgentService/OnlineCPUMem resp=\"&Empty{XXX_unrecognized:[],}\" sandbox=1284226a23bf29f23c0228907d4080b3a314cccece0b1304adb22f878c3ecf28 source=agent\n" name=kata-proxy pid=212943 sandbox=1284226a23bf29f23c0228907d4080b3a314cccece0b1304adb22f878c3ecf28 source=agent time="2020-03-17T17:50:49.806439761Z" level=info msg="time=\"2020-03-17T17:50:49.795374499Z\" level=debug msg=\"request end\" debug_console=false duration=\"107.158µs\" name=kata-agent pid=1 request=/grpc.AgentService/CloseStdin resp=\"&Empty{XXX_unrecognized:[],}\" sandbox=1284226a23bf29f23c0228907d4080b3a314cccece0b1304adb22f878c3ecf28 source=agent\n" name=kata-proxy pid=212943 sandbox=1284226a23bf29f23c0228907d4080b3a314cccece0b1304adb22f878c3ecf28 source=agent time="2020-03-17T17:50:51.817669437Z" level=info msg="time=\"2020-03-17T17:50:51.806716307Z\" level=debug msg=\"request end\" debug_console=false duration=5.495806ms name=kata-agent pid=1 request=/grpc.AgentService/StartContainer resp=\"&Empty{XXX_unrecognized:[],}\" sandbox=1284226a23bf29f23c0228907d4080b3a314cccece0b1304adb22f878c3ecf28 source=agent\n" name=kata-proxy pid=212943 sandbox=1284226a23bf29f23c0228907d4080b3a314cccece0b1304adb22f878c3ecf28 source=agent time="2020-03-17T17:50:51.85442749Z" level=info msg="time=\"2020-03-17T17:50:51.843380942Z\" level=info msg=\"ignoring unexpected signal\" debug_console=false name=kata-agent pid=1 sandbox=1284226a23bf29f23c0228907d4080b3a314cccece0b1304adb22f878c3ecf28 signal=\"child exited\" source=agent\n" name=kata-proxy pid=212943 sandbox=1284226a23bf29f23c0228907d4080b3a314cccece0b1304adb22f878c3ecf28 source=agent time="2020-03-17T17:50:52.068523881Z" level=info msg="time=\"2020-03-17T17:50:52.057614575Z\" level=debug msg=\"request end\" debug_console=false duration=\"236.614µs\" name=kata-agent pid=1 request=/grpc.AgentService/SignalProcess resp=\"&Empty{XXX_unrecognized:[],}\" sandbox=1284226a23bf29f23c0228907d4080b3a314cccece0b1304adb22f878c3ecf28 source=agent\n" name=kata-proxy pid=212943 sandbox=1284226a23bf29f23c0228907d4080b3a314cccece0b1304adb22f878c3ecf28 source=agent time="2020-03-17T17:50:52.073917004Z" level=info msg="time=\"2020-03-17T17:50:52.062940996Z\" level=debug msg=\"request end\" debug_console=false duration=\"467.973µs\" name=kata-agent pid=1 request=/grpc.AgentService/RemoveContainer resp=\"&Empty{XXX_unrecognized:[],}\" sandbox=1284226a23bf29f23c0228907d4080b3a314cccece0b1304adb22f878c3ecf28 source=agent\n" name=kata-proxy pid=212943 sandbox=1284226a23bf29f23c0228907d4080b3a314cccece0b1304adb22f878c3ecf28 source=agent time="2020-03-17T17:50:52.095485594Z" level=info msg="time=\"2020-03-17T17:50:52.084588978Z\" level=debug msg=\"request end\" debug_console=false duration=10.854429ms name=kata-agent pid=1 request=/grpc.AgentService/DestroySandbox resp=\"&Empty{XXX_unrecognized:[],}\" sandbox=1284226a23bf29f23c0228907d4080b3a314cccece0b1304adb22f878c3ecf28 source=agent\n" name=kata-proxy pid=212943 sandbox=1284226a23bf29f23c0228907d4080b3a314cccece0b1304adb22f878c3ecf28 source=agent time="2020-03-17T17:50:52.095954196Z" level=fatal msg="channel error" error="accept unix /run/user/1000/run/vc/sbs/1284226a23bf29f23c0228907d4080b3a314cccece0b1304adb22f878c3ecf28/proxy.sock: use of closed network connection" name=kata-proxy pid=212943 sandbox=1284226a23bf29f23c0228907d4080b3a314cccece0b1304adb22f878c3ecf28 source=proxy time="2020-03-17T17:50:55.713126659Z" level=info msg="[ 0.445253] intel_pstate: CPU model not supported\n" name=kata-proxy pid=213136 sandbox=50a87dd6139cd3158d413c1c8e5460610a283381ef9e1150378b48f0a5f9045d source=agent time="2020-03-17T17:50:55.756168238Z" level=info msg="time=\"2020-03-17T17:50:55.747050926Z\" level=debug msg=\"request end\" debug_console=false duration=2.415761ms name=kata-agent pid=1 request=/grpc.AgentService/CreateSandbox resp=\"&Empty{XXX_unrecognized:[],}\" sandbox=50a87dd6139cd3158d413c1c8e5460610a283381ef9e1150378b48f0a5f9045d source=agent\n" name=kata-proxy pid=213136 sandbox=50a87dd6139cd3158d413c1c8e5460610a283381ef9e1150378b48f0a5f9045d source=agent time="2020-03-17T17:50:55.843628303Z" level=info msg="time=\"2020-03-17T17:50:55.834462779Z\" level=debug msg=\"request end\" debug_console=false duration=79.350124ms name=kata-agent pid=1 request=/grpc.AgentService/CreateContainer resp=\"&Empty{XXX_unrecognized:[],}\" sandbox=50a87dd6139cd3158d413c1c8e5460610a283381ef9e1150378b48f0a5f9045d source=agent\n" name=kata-proxy pid=213136 sandbox=50a87dd6139cd3158d413c1c8e5460610a283381ef9e1150378b48f0a5f9045d source=agent time="2020-03-17T17:50:55.844441406Z" level=info msg="time=\"2020-03-17T17:50:55.834849668Z\" level=info msg=\"ignoring unexpected signal\" debug_console=false name=kata-agent pid=1 sandbox=50a87dd6139cd3158d413c1c8e5460610a283381ef9e1150378b48f0a5f9045d signal=\"child exited\" source=agent\n" name=kata-proxy pid=213136 sandbox=50a87dd6139cd3158d413c1c8e5460610a283381ef9e1150378b48f0a5f9045d source=agent time="2020-03-17T17:50:55.850491831Z" level=info msg="time=\"2020-03-17T17:50:55.841253117Z\" level=debug msg=\"request end\" debug_console=false duration=\"3.093µs\" name=kata-agent pid=1 request=/grpc.AgentService/OnlineCPUMem resp=\"&Empty{XXX_unrecognized:[],}\" sandbox=50a87dd6139cd3158d413c1c8e5460610a283381ef9e1150378b48f0a5f9045d source=agent\n" name=kata-proxy pid=213136 sandbox=50a87dd6139cd3158d413c1c8e5460610a283381ef9e1150378b48f0a5f9045d source=agent time="2020-03-17T17:50:55.859040567Z" level=info msg="time=\"2020-03-17T17:50:55.849866297Z\" level=debug msg=\"request end\" debug_console=false duration=\"7.606µs\" name=kata-agent pid=1 request=/grpc.AgentService/CloseStdin resp=\"&Empty{XXX_unrecognized:[],}\" sandbox=50a87dd6139cd3158d413c1c8e5460610a283381ef9e1150378b48f0a5f9045d source=agent\n" name=kata-proxy pid=213136 sandbox=50a87dd6139cd3158d413c1c8e5460610a283381ef9e1150378b48f0a5f9045d source=agent time="2020-03-17T17:50:55.916521106Z" level=info msg="time=\"2020-03-17T17:50:55.907404032Z\" level=debug msg=\"request end\" debug_console=false duration=3.529231ms name=kata-agent pid=1 request=/grpc.AgentService/StartContainer resp=\"&Empty{XXX_unrecognized:[],}\" sandbox=50a87dd6139cd3158d413c1c8e5460610a283381ef9e1150378b48f0a5f9045d source=agent\n" name=kata-proxy pid=213136 sandbox=50a87dd6139cd3158d413c1c8e5460610a283381ef9e1150378b48f0a5f9045d source=agent time="2020-03-17T17:50:55.938656698Z" level=info msg="time=\"2020-03-17T17:50:55.929402478Z\" level=info msg=\"ignoring unexpected signal\" debug_console=false name=kata-agent pid=1 sandbox=50a87dd6139cd3158d413c1c8e5460610a283381ef9e1150378b48f0a5f9045d signal=\"child exited\" source=agent\n" name=kata-proxy pid=213136 sandbox=50a87dd6139cd3158d413c1c8e5460610a283381ef9e1150378b48f0a5f9045d source=agent time="2020-03-17T17:50:56.176270988Z" level=info msg="time=\"2020-03-17T17:50:56.167101814Z\" level=debug msg=\"request end\" debug_console=false duration=\"245.179µs\" name=kata-agent pid=1 request=/grpc.AgentService/SignalProcess resp=\"&Empty{XXX_unrecognized:[],}\" sandbox=50a87dd6139cd3158d413c1c8e5460610a283381ef9e1150378b48f0a5f9045d source=agent\n" name=kata-proxy pid=213136 sandbox=50a87dd6139cd3158d413c1c8e5460610a283381ef9e1150378b48f0a5f9045d source=agent time="2020-03-17T17:50:56.18161401Z" level=info msg="time=\"2020-03-17T17:50:56.17254343Z\" level=debug msg=\"request end\" debug_console=false duration=\"404.225µs\" name=kata-agent pid=1 request=/grpc.AgentService/RemoveContainer resp=\"&Empty{XXX_unrecognized:[],}\" sandbox=50a87dd6139cd3158d413c1c8e5460610a283381ef9e1150378b48f0a5f9045d source=agent\n" name=kata-proxy pid=213136 sandbox=50a87dd6139cd3158d413c1c8e5460610a283381ef9e1150378b48f0a5f9045d source=agent time="2020-03-17T17:50:56.202399497Z" level=info msg="time=\"2020-03-17T17:50:56.193271689Z\" level=debug msg=\"request end\" debug_console=false duration=11.440601ms name=kata-agent pid=1 request=/grpc.AgentService/DestroySandbox resp=\"&Empty{XXX_unrecognized:[],}\" sandbox=50a87dd6139cd3158d413c1c8e5460610a283381ef9e1150378b48f0a5f9045d source=agent\n" name=kata-proxy pid=213136 sandbox=50a87dd6139cd3158d413c1c8e5460610a283381ef9e1150378b48f0a5f9045d source=agent time="2020-03-17T17:50:56.2031277Z" level=fatal msg="failed to handle exit signal" error="close unix @->/run/user/1000/run/vc/vm/50a87dd6139cd3158d413c1c8e5460610a283381ef9e1150378b48f0a5f9045d/kata.sock: use of closed network connection" name=kata-proxy pid=213136 sandbox=50a87dd6139cd3158d413c1c8e5460610a283381ef9e1150378b48f0a5f9045d source=proxy time="2020-03-17T17:50:59.58164907Z" level=info msg="[ 0.445583] intel_pstate: CPU model not supported\n" name=kata-proxy pid=213338 sandbox=15288558b81ea3a58b706746f9825aa4137cd10bb39263bc353e622ba3153eb3 source=agent time="2020-03-17T17:50:59.625321952Z" level=info msg="time=\"2020-03-17T17:50:59.614042984Z\" level=debug msg=\"request end\" debug_console=false duration=2.615888ms name=kata-agent pid=1 request=/grpc.AgentService/CreateSandbox resp=\"&Empty{XXX_unrecognized:[],}\" sandbox=15288558b81ea3a58b706746f9825aa4137cd10bb39263bc353e622ba3153eb3 source=agent\n" name=kata-proxy pid=213338 sandbox=15288558b81ea3a58b706746f9825aa4137cd10bb39263bc353e622ba3153eb3 source=agent time="2020-03-17T17:50:59.717179234Z" level=info msg="time=\"2020-03-17T17:50:59.705958135Z\" level=debug msg=\"request end\" debug_console=false duration=82.975416ms name=kata-agent pid=1 request=/grpc.AgentService/CreateContainer resp=\"&Empty{XXX_unrecognized:[],}\" sandbox=15288558b81ea3a58b706746f9825aa4137cd10bb39263bc353e622ba3153eb3 source=agent\n" name=kata-proxy pid=213338 sandbox=15288558b81ea3a58b706746f9825aa4137cd10bb39263bc353e622ba3153eb3 source=agent time="2020-03-17T17:50:59.717579036Z" level=info msg="time=\"2020-03-17T17:50:59.706363036Z\" level=info msg=\"ignoring unexpected signal\" debug_console=false name=kata-agent pid=1 sandbox=15288558b81ea3a58b706746f9825aa4137cd10bb39263bc353e622ba3153eb3 signal=\"child exited\" source=agent\n" name=kata-proxy pid=213338 sandbox=15288558b81ea3a58b706746f9825aa4137cd10bb39263bc353e622ba3153eb3 source=agent time="2020-03-17T17:50:59.724530165Z" level=info msg="time=\"2020-03-17T17:50:59.713351914Z\" level=debug msg=\"request end\" debug_console=false duration=\"2.998µs\" name=kata-agent pid=1 request=/grpc.AgentService/OnlineCPUMem resp=\"&Empty{XXX_unrecognized:[],}\" sandbox=15288558b81ea3a58b706746f9825aa4137cd10bb39263bc353e622ba3153eb3 source=agent\n" name=kata-proxy pid=213338 sandbox=15288558b81ea3a58b706746f9825aa4137cd10bb39263bc353e622ba3153eb3 source=agent time="2020-03-17T17:50:59.785549919Z" level=info msg="time=\"2020-03-17T17:50:59.774349755Z\" level=debug msg=\"request end\" debug_console=false duration=4.882568ms name=kata-agent pid=1 request=/grpc.AgentService/StartContainer resp=\"&Empty{XXX_unrecognized:[],}\" sandbox=15288558b81ea3a58b706746f9825aa4137cd10bb39263bc353e622ba3153eb3 source=agent\n" name=kata-proxy pid=213338 sandbox=15288558b81ea3a58b706746f9825aa4137cd10bb39263bc353e622ba3153eb3 source=agent time="2020-03-17T17:50:59.8050123Z" level=info msg="time=\"2020-03-17T17:50:59.793885543Z\" level=info msg=\"ignoring unexpected signal\" debug_console=false name=kata-agent pid=1 sandbox=15288558b81ea3a58b706746f9825aa4137cd10bb39263bc353e622ba3153eb3 signal=\"child exited\" source=agent\n" name=kata-proxy pid=213338 sandbox=15288558b81ea3a58b706746f9825aa4137cd10bb39263bc353e622ba3153eb3 source=agent time="2020-03-17T17:50:59.970902291Z" level=info msg="time=\"2020-03-17T17:50:59.959764992Z\" level=debug msg=\"request end\" debug_console=false duration=\"583.6µs\" name=kata-agent pid=1 request=/grpc.AgentService/SignalProcess resp=\"&Empty{XXX_unrecognized:[],}\" sandbox=15288558b81ea3a58b706746f9825aa4137cd10bb39263bc353e622ba3153eb3 source=agent\n" name=kata-proxy pid=213338 sandbox=15288558b81ea3a58b706746f9825aa4137cd10bb39263bc353e622ba3153eb3 source=agent time="2020-03-17T17:50:59.978730223Z" level=info msg="time=\"2020-03-17T17:50:59.967570597Z\" level=debug msg=\"request end\" debug_console=false duration=\"348.587µs\" name=kata-agent pid=1 request=/grpc.AgentService/RemoveContainer resp=\"&Empty{XXX_unrecognized:[],}\" sandbox=15288558b81ea3a58b706746f9825aa4137cd10bb39263bc353e622ba3153eb3 source=agent\n" name=kata-proxy pid=213338 sandbox=15288558b81ea3a58b706746f9825aa4137cd10bb39263bc353e622ba3153eb3 source=agent time="2020-03-17T17:50:59.994307588Z" level=info msg="time=\"2020-03-17T17:50:59.983142751Z\" level=debug msg=\"request end\" debug_console=false duration=7.783972ms name=kata-agent pid=1 request=/grpc.AgentService/DestroySandbox resp=\"&Empty{XXX_unrecognized:[],}\" sandbox=15288558b81ea3a58b706746f9825aa4137cd10bb39263bc353e622ba3153eb3 source=agent\n" name=kata-proxy pid=213338 sandbox=15288558b81ea3a58b706746f9825aa4137cd10bb39263bc353e622ba3153eb3 source=agent time="2020-03-17T17:50:59.995140492Z" level=fatal msg="failed to handle exit signal" error="close unix @->/run/user/1000/run/vc/vm/15288558b81ea3a58b706746f9825aa4137cd10bb39263bc353e622ba3153eb3/kata.sock: use of closed network connection" name=kata-proxy pid=213338 sandbox=15288558b81ea3a58b706746f9825aa4137cd10bb39263bc353e622ba3153eb3 source=proxy time="2020-03-17T17:51:05.269370856Z" level=info msg="[ 0.491428] intel_pstate: CPU model not supported\n" name=kata-proxy pid=213753 sandbox=992b94d81a146c5d122715cd970ec29221bad138201dd6fae14fd3f74061477c source=agent time="2020-03-17T17:51:05.31831916Z" level=info msg="time=\"2020-03-17T17:51:05.305283695Z\" level=debug msg=\"request end\" debug_console=false duration=2.019184ms name=kata-agent pid=1 request=/grpc.AgentService/CreateSandbox resp=\"&Empty{XXX_unrecognized:[],}\" sandbox=992b94d81a146c5d122715cd970ec29221bad138201dd6fae14fd3f74061477c source=agent\n" name=kata-proxy pid=213753 sandbox=992b94d81a146c5d122715cd970ec29221bad138201dd6fae14fd3f74061477c source=agent time="2020-03-17T17:51:05.42863162Z" level=info msg="time=\"2020-03-17T17:51:05.415517695Z\" level=debug msg=\"request end\" debug_console=false duration=100.770652ms name=kata-agent pid=1 request=/grpc.AgentService/CreateContainer resp=\"&Empty{XXX_unrecognized:[],}\" sandbox=992b94d81a146c5d122715cd970ec29221bad138201dd6fae14fd3f74061477c source=agent\n" name=kata-proxy pid=213753 sandbox=992b94d81a146c5d122715cd970ec29221bad138201dd6fae14fd3f74061477c source=agent time="2020-03-17T17:51:05.429365023Z" level=info msg="time=\"2020-03-17T17:51:05.415899896Z\" level=info msg=\"ignoring unexpected signal\" debug_console=false name=kata-agent pid=1 sandbox=992b94d81a146c5d122715cd970ec29221bad138201dd6fae14fd3f74061477c signal=\"child exited\" source=agent\n" name=kata-proxy pid=213753 sandbox=992b94d81a146c5d122715cd970ec29221bad138201dd6fae14fd3f74061477c source=agent time="2020-03-17T17:51:05.435619749Z" level=info msg="time=\"2020-03-17T17:51:05.422482977Z\" level=debug msg=\"request end\" debug_console=false duration=\"3.603µs\" name=kata-agent pid=1 request=/grpc.AgentService/OnlineCPUMem resp=\"&Empty{XXX_unrecognized:[],}\" sandbox=992b94d81a146c5d122715cd970ec29221bad138201dd6fae14fd3f74061477c source=agent\n" name=kata-proxy pid=213753 sandbox=992b94d81a146c5d122715cd970ec29221bad138201dd6fae14fd3f74061477c source=agent time="2020-03-17T17:51:05.447167397Z" level=info msg="time=\"2020-03-17T17:51:05.434061378Z\" level=debug msg=\"request end\" debug_console=false duration=\"10.563µs\" name=kata-agent pid=1 request=/grpc.AgentService/CloseStdin resp=\"&Empty{XXX_unrecognized:[],}\" sandbox=992b94d81a146c5d122715cd970ec29221bad138201dd6fae14fd3f74061477c source=agent\n" name=kata-proxy pid=213753 sandbox=992b94d81a146c5d122715cd970ec29221bad138201dd6fae14fd3f74061477c source=agent time="2020-03-17T17:51:05.676342551Z" level=info msg="time=\"2020-03-17T17:51:05.663210241Z\" level=debug msg=\"request end\" debug_console=false duration=\"243.682µs\" name=kata-agent pid=1 request=/grpc.AgentService/SignalProcess resp=\"&Empty{XXX_unrecognized:[],}\" sandbox=992b94d81a146c5d122715cd970ec29221bad138201dd6fae14fd3f74061477c source=agent\n" name=kata-proxy pid=213753 sandbox=992b94d81a146c5d122715cd970ec29221bad138201dd6fae14fd3f74061477c source=agent time="2020-03-17T17:51:05.682924379Z" level=info msg="time=\"2020-03-17T17:51:05.66990221Z\" level=info msg=\"ignoring unexpected signal\" debug_console=false name=kata-agent pid=1 sandbox=992b94d81a146c5d122715cd970ec29221bad138201dd6fae14fd3f74061477c signal=\"child exited\" source=agent\n" name=kata-proxy pid=213753 sandbox=992b94d81a146c5d122715cd970ec29221bad138201dd6fae14fd3f74061477c source=agent time="2020-03-17T17:51:05.789574523Z" level=info msg="time=\"2020-03-17T17:51:05.776438983Z\" level=debug msg=\"request end\" debug_console=false duration=\"404.01µs\" name=kata-agent pid=1 request=/grpc.AgentService/SignalProcess resp=\"&Empty{XXX_unrecognized:[],}\" sandbox=992b94d81a146c5d122715cd970ec29221bad138201dd6fae14fd3f74061477c source=agent\n" name=kata-proxy pid=213753 sandbox=992b94d81a146c5d122715cd970ec29221bad138201dd6fae14fd3f74061477c source=agent time="2020-03-17T17:51:05.79619665Z" level=info msg="time=\"2020-03-17T17:51:05.783038762Z\" level=debug msg=\"request end\" debug_console=false duration=\"569.236µs\" name=kata-agent pid=1 request=/grpc.AgentService/RemoveContainer resp=\"&Empty{XXX_unrecognized:[],}\" sandbox=992b94d81a146c5d122715cd970ec29221bad138201dd6fae14fd3f74061477c source=agent\n" name=kata-proxy pid=213753 sandbox=992b94d81a146c5d122715cd970ec29221bad138201dd6fae14fd3f74061477c source=agent time="2020-03-17T17:51:05.809982808Z" level=info msg="time=\"2020-03-17T17:51:05.796746591Z\" level=debug msg=\"request end\" debug_console=false duration=10.295769ms name=kata-agent pid=1 request=/grpc.AgentService/DestroySandbox resp=\"&Empty{XXX_unrecognized:[],}\" sandbox=992b94d81a146c5d122715cd970ec29221bad138201dd6fae14fd3f74061477c source=agent\n" name=kata-proxy pid=213753 sandbox=992b94d81a146c5d122715cd970ec29221bad138201dd6fae14fd3f74061477c source=agent time="2020-03-17T17:51:05.81062831Z" level=fatal msg="channel error" error="accept unix /run/user/1000/run/vc/sbs/992b94d81a146c5d122715cd970ec29221bad138201dd6fae14fd3f74061477c/proxy.sock: use of closed network connection" name=kata-proxy pid=213753 sandbox=992b94d81a146c5d122715cd970ec29221bad138201dd6fae14fd3f74061477c source=proxy ``` ## Shim logs Recent shim problems found in system journal: ``` time="2020-03-17T17:46:07.976798581Z" level=info msg="copy stdout failed" container=f7dd98af3dfd6a853b5479100071fddbffc4432bcfef9f718cba6836a2f386bb error="rpc error: code = Unknown desc = EOF" exec-id=f7dd98af3dfd6a853b5479100071fddbffc4432bcfef9f718cba6836a2f386bb name=kata-shim pid=1 source=shim time="2020-03-17T17:46:11.30714405Z" level=info msg="copy stderr failed" container=19c4224fd7ef0e42842a9304f467d9d4c72a85ac7ce0fc0f9ddde5f4e70d6ee9 error="rpc error: code = Unknown desc = EOF" exec-id=d3a5a9db-6c59-4038-857c-d8ab300f54d3 name=kata-shim pid=11 source=shim time="2020-03-17T17:46:11.308122654Z" level=info msg="copy stdout failed" container=19c4224fd7ef0e42842a9304f467d9d4c72a85ac7ce0fc0f9ddde5f4e70d6ee9 error="rpc error: code = Unknown desc = EOF" exec-id=d3a5a9db-6c59-4038-857c-d8ab300f54d3 name=kata-shim pid=11 source=shim time="2020-03-17T17:46:11.558880698Z" level=info msg="copy stderr failed" container=19c4224fd7ef0e42842a9304f467d9d4c72a85ac7ce0fc0f9ddde5f4e70d6ee9 error="rpc error: code = Unknown desc = EOF" exec-id=a88e5687-115a-43ee-a790-141c43accce4 name=kata-shim pid=21 source=shim time="2020-03-17T17:46:11.559648701Z" level=info msg="copy stdout failed" container=19c4224fd7ef0e42842a9304f467d9d4c72a85ac7ce0fc0f9ddde5f4e70d6ee9 error="rpc error: code = Unknown desc = EOF" exec-id=a88e5687-115a-43ee-a790-141c43accce4 name=kata-shim pid=21 source=shim time="2020-03-17T17:46:11.765991261Z" level=info msg="copy stderr failed" container=19c4224fd7ef0e42842a9304f467d9d4c72a85ac7ce0fc0f9ddde5f4e70d6ee9 error="rpc error: code = Unknown desc = EOF" exec-id=19c4224fd7ef0e42842a9304f467d9d4c72a85ac7ce0fc0f9ddde5f4e70d6ee9 name=kata-shim pid=1 source=shim time="2020-03-17T17:46:11.766314762Z" level=info msg="copy stdout failed" container=19c4224fd7ef0e42842a9304f467d9d4c72a85ac7ce0fc0f9ddde5f4e70d6ee9 error="rpc error: code = Unknown desc = EOF" exec-id=19c4224fd7ef0e42842a9304f467d9d4c72a85ac7ce0fc0f9ddde5f4e70d6ee9 name=kata-shim pid=1 source=shim time="2020-03-17T17:46:26.006582666Z" level=info msg="copy stderr failed" container=c92468435143e6894025b9cd642961cf662ae3873ec265e54e1fa5ba0033bf79 error="rpc error: code = Unknown desc = EOF" exec-id=c92468435143e6894025b9cd642961cf662ae3873ec265e54e1fa5ba0033bf79 name=kata-shim pid=1 source=shim time="2020-03-17T17:46:26.006994967Z" level=info msg="copy stdout failed" container=c92468435143e6894025b9cd642961cf662ae3873ec265e54e1fa5ba0033bf79 error="rpc error: code = Unknown desc = EOF" exec-id=c92468435143e6894025b9cd642961cf662ae3873ec265e54e1fa5ba0033bf79 name=kata-shim pid=1 source=shim time="2020-03-17T17:46:39.579946192Z" level=info msg="copy stderr failed" container=db789faa81faee037904085b0f674e7f03a09340ed76fef1103f24d213026193 error="rpc error: code = Unknown desc = EOF" exec-id=db789faa81faee037904085b0f674e7f03a09340ed76fef1103f24d213026193 name=kata-shim pid=1 source=shim time="2020-03-17T17:46:39.580227093Z" level=info msg="copy stdout failed" container=db789faa81faee037904085b0f674e7f03a09340ed76fef1103f24d213026193 error="rpc error: code = Unknown desc = EOF" exec-id=db789faa81faee037904085b0f674e7f03a09340ed76fef1103f24d213026193 name=kata-shim pid=1 source=shim time="2020-03-17T17:49:10.031775049Z" level=info msg="copy stderr failed" container=f70d478b2c7dd8b9a54c0a41928a16203fe6e090ebc8ace18144e3ee0df406e6 error="rpc error: code = Unknown desc = EOF" exec-id=f70d478b2c7dd8b9a54c0a41928a16203fe6e090ebc8ace18144e3ee0df406e6 name=kata-shim pid=1 source=shim time="2020-03-17T17:49:10.032192351Z" level=info msg="copy stdout failed" container=f70d478b2c7dd8b9a54c0a41928a16203fe6e090ebc8ace18144e3ee0df406e6 error="rpc error: code = Unknown desc = EOF" exec-id=f70d478b2c7dd8b9a54c0a41928a16203fe6e090ebc8ace18144e3ee0df406e6 name=kata-shim pid=1 source=shim time="2020-03-17T17:49:10.034763161Z" level=info msg="copy stdout failed" container=95ea1a3842f6dd42ac614a44d73d7dc845fe90947a075f8933a25c6b5a31fc7f error="rpc error: code = Unknown desc = EOF" exec-id=95ea1a3842f6dd42ac614a44d73d7dc845fe90947a075f8933a25c6b5a31fc7f name=kata-shim pid=1 source=shim time="2020-03-17T17:49:10.034853362Z" level=info msg="copy stderr failed" container=95ea1a3842f6dd42ac614a44d73d7dc845fe90947a075f8933a25c6b5a31fc7f error="rpc error: code = Unknown desc = EOF" exec-id=95ea1a3842f6dd42ac614a44d73d7dc845fe90947a075f8933a25c6b5a31fc7f name=kata-shim pid=1 source=shim time="2020-03-17T17:49:21.169769633Z" level=info msg="copy stderr failed" container=99bfd83dad2493b3863504538441031de9fa317419e6695d254a141826b6a40b error="rpc error: code = Unknown desc = EOF" exec-id=99bfd83dad2493b3863504538441031de9fa317419e6695d254a141826b6a40b name=kata-shim pid=1 source=shim time="2020-03-17T17:49:21.170351936Z" level=info msg="copy stdout failed" container=99bfd83dad2493b3863504538441031de9fa317419e6695d254a141826b6a40b error="rpc error: code = Unknown desc = EOF" exec-id=99bfd83dad2493b3863504538441031de9fa317419e6695d254a141826b6a40b name=kata-shim pid=1 source=shim time="2020-03-17T17:49:24.257726793Z" level=info msg="copy stderr failed" container=2f0ac85b90bd256a6159d195e949cb47b684337bcaa2270c10cb4b5d1734ceab error="rpc error: code = Unknown desc = EOF" exec-id=39760008-b226-4f50-8060-799b97684bea name=kata-shim pid=13 source=shim time="2020-03-17T17:49:24.260335204Z" level=info msg="copy stdout failed" container=2f0ac85b90bd256a6159d195e949cb47b684337bcaa2270c10cb4b5d1734ceab error="rpc error: code = Unknown desc = EOF" exec-id=39760008-b226-4f50-8060-799b97684bea name=kata-shim pid=13 source=shim time="2020-03-17T17:49:24.54023907Z" level=info msg="copy stderr failed" container=2f0ac85b90bd256a6159d195e949cb47b684337bcaa2270c10cb4b5d1734ceab error="rpc error: code = Unknown desc = EOF" exec-id=34cd473b-1679-4414-aed4-f4f633f34864 name=kata-shim pid=23 source=shim time="2020-03-17T17:49:24.541479175Z" level=info msg="copy stdout failed" container=2f0ac85b90bd256a6159d195e949cb47b684337bcaa2270c10cb4b5d1734ceab error="rpc error: code = Unknown desc = EOF" exec-id=34cd473b-1679-4414-aed4-f4f633f34864 name=kata-shim pid=23 source=shim time="2020-03-17T17:49:24.708230269Z" level=info msg="copy stderr failed" container=2f0ac85b90bd256a6159d195e949cb47b684337bcaa2270c10cb4b5d1734ceab error="rpc error: code = Unknown desc = EOF" exec-id=2f0ac85b90bd256a6159d195e949cb47b684337bcaa2270c10cb4b5d1734ceab name=kata-shim pid=1 source=shim time="2020-03-17T17:49:24.70837217Z" level=info msg="copy stdout failed" container=2f0ac85b90bd256a6159d195e949cb47b684337bcaa2270c10cb4b5d1734ceab error="rpc error: code = Unknown desc = EOF" exec-id=2f0ac85b90bd256a6159d195e949cb47b684337bcaa2270c10cb4b5d1734ceab name=kata-shim pid=1 source=shim time="2020-03-17T17:49:30.95168917Z" level=info msg="copy stderr failed" container=4aeba198e80db12565b910e386ada5bf4910529a698159966dc2f55151fddbcb error="rpc error: code = Unknown desc = EOF" exec-id=4aeba198e80db12565b910e386ada5bf4910529a698159966dc2f55151fddbcb name=kata-shim pid=1 source=shim time="2020-03-17T17:49:30.951916671Z" level=info msg="copy stdout failed" container=4aeba198e80db12565b910e386ada5bf4910529a698159966dc2f55151fddbcb error="rpc error: code = Unknown desc = EOF" exec-id=4aeba198e80db12565b910e386ada5bf4910529a698159966dc2f55151fddbcb name=kata-shim pid=1 source=shim time="2020-03-17T17:49:32.157371891Z" level=info msg="copy stderr failed" container=d9506d730d1de62bdc80d8773af31703fa939ad3db0ef274106b4a23e1571637 error="rpc error: code = Unknown desc = EOF" exec-id=d9506d730d1de62bdc80d8773af31703fa939ad3db0ef274106b4a23e1571637 name=kata-shim pid=1 source=shim time="2020-03-17T17:49:32.158011894Z" level=info msg="copy stdout failed" container=d9506d730d1de62bdc80d8773af31703fa939ad3db0ef274106b4a23e1571637 error="rpc error: code = Unknown desc = EOF" exec-id=d9506d730d1de62bdc80d8773af31703fa939ad3db0ef274106b4a23e1571637 name=kata-shim pid=1 source=shim time="2020-03-17T17:49:34.386216273Z" level=info msg="copy stderr failed" container=f34bc14af54840968b330ec08abaee0d40c0bd8b9c44f96be01c0365405e2287 error="rpc error: code = Unknown desc = EOF" exec-id=f34bc14af54840968b330ec08abaee0d40c0bd8b9c44f96be01c0365405e2287 name=kata-shim pid=1 source=shim time="2020-03-17T17:49:34.386984876Z" level=info msg="copy stdout failed" container=f34bc14af54840968b330ec08abaee0d40c0bd8b9c44f96be01c0365405e2287 error="rpc error: code = Unknown desc = EOF" exec-id=f34bc14af54840968b330ec08abaee0d40c0bd8b9c44f96be01c0365405e2287 name=kata-shim pid=1 source=shim time="2020-03-17T17:49:37.18284012Z" level=info msg="copy stderr failed" container=30344b3d9d0433dfe01a8ccf2492baa792647bfb178ca74c106aa44eb996e017 error="rpc error: code = Unknown desc = EOF" exec-id=30344b3d9d0433dfe01a8ccf2492baa792647bfb178ca74c106aa44eb996e017 name=kata-shim pid=1 source=shim time="2020-03-17T17:49:37.183194621Z" level=info msg="copy stdout failed" container=30344b3d9d0433dfe01a8ccf2492baa792647bfb178ca74c106aa44eb996e017 error="rpc error: code = Unknown desc = EOF" exec-id=30344b3d9d0433dfe01a8ccf2492baa792647bfb178ca74c106aa44eb996e017 name=kata-shim pid=1 source=shim time="2020-03-17T17:49:41.312063716Z" level=info msg="copy stdout failed" container=65744c7aced36ffc2607a3447d629705e837d0ea9942efb40266df84c204f14e error="rpc error: code = Unknown desc = EOF" exec-id=65744c7aced36ffc2607a3447d629705e837d0ea9942efb40266df84c204f14e name=kata-shim pid=1 source=shim time="2020-03-17T17:49:41.312088716Z" level=info msg="copy stderr failed" container=65744c7aced36ffc2607a3447d629705e837d0ea9942efb40266df84c204f14e error="rpc error: code = Unknown desc = EOF" exec-id=65744c7aced36ffc2607a3447d629705e837d0ea9942efb40266df84c204f14e name=kata-shim pid=1 source=shim time="2020-03-17T17:49:50.023451194Z" level=info msg="copy stderr failed" container=3220b62368442695c531c49e5e312059827fb7b0d70f7cb6b424f26a67f5eed4 error="rpc error: code = Unknown desc = EOF" exec-id=3220b62368442695c531c49e5e312059827fb7b0d70f7cb6b424f26a67f5eed4 name=kata-shim pid=1 source=shim time="2020-03-17T17:49:50.023777396Z" level=info msg="copy stdout failed" container=3220b62368442695c531c49e5e312059827fb7b0d70f7cb6b424f26a67f5eed4 error="rpc error: code = Unknown desc = EOF" exec-id=3220b62368442695c531c49e5e312059827fb7b0d70f7cb6b424f26a67f5eed4 name=kata-shim pid=1 source=shim time="2020-03-17T17:49:54.988696472Z" level=info msg="copy stderr failed" container=dab46fb3024b3b9725e19e3f659d755b873f3b01212f58b8f9634fb01d76a29a error="rpc error: code = Unknown desc = EOF" exec-id=dab46fb3024b3b9725e19e3f659d755b873f3b01212f58b8f9634fb01d76a29a name=kata-shim pid=1 source=shim time="2020-03-17T17:49:54.988943873Z" level=info msg="copy stdout failed" container=dab46fb3024b3b9725e19e3f659d755b873f3b01212f58b8f9634fb01d76a29a error="rpc error: code = Unknown desc = EOF" exec-id=dab46fb3024b3b9725e19e3f659d755b873f3b01212f58b8f9634fb01d76a29a name=kata-shim pid=1 source=shim time="2020-03-17T17:50:02.044177655Z" level=info msg="copy stderr failed" container=0e3f7aff77e58314cbeed58738bebd4a55398d4c4644f91a2f7d363bee619622 error="rpc error: code = Unknown desc = EOF" exec-id=0e3f7aff77e58314cbeed58738bebd4a55398d4c4644f91a2f7d363bee619622 name=kata-shim pid=1 source=shim time="2020-03-17T17:50:02.044606857Z" level=info msg="copy stdout failed" container=0e3f7aff77e58314cbeed58738bebd4a55398d4c4644f91a2f7d363bee619622 error="rpc error: code = Unknown desc = EOF" exec-id=0e3f7aff77e58314cbeed58738bebd4a55398d4c4644f91a2f7d363bee619622 name=kata-shim pid=1 source=shim time="2020-03-17T17:50:25.893721176Z" level=info msg="copy stderr failed" container=6f61cc5d4b8f50da0e3484b2e98d0ec09b27efc782736dfe54b57ea97c9eef0a error="rpc error: code = Unknown desc = EOF" exec-id=6f61cc5d4b8f50da0e3484b2e98d0ec09b27efc782736dfe54b57ea97c9eef0a name=kata-shim pid=1 source=shim time="2020-03-17T17:50:25.894099878Z" level=info msg="copy stdout failed" container=6f61cc5d4b8f50da0e3484b2e98d0ec09b27efc782736dfe54b57ea97c9eef0a error="rpc error: code = Unknown desc = EOF" exec-id=6f61cc5d4b8f50da0e3484b2e98d0ec09b27efc782736dfe54b57ea97c9eef0a name=kata-shim pid=1 source=shim time="2020-03-17T17:50:39.019080037Z" level=info msg="copy stderr failed" container=4d4a0cfcb196cae46cd97a7a0495945bd201a66267d237c432f1ea42daf07a85 error="rpc error: code = Unknown desc = EOF" exec-id=4d4a0cfcb196cae46cd97a7a0495945bd201a66267d237c432f1ea42daf07a85 name=kata-shim pid=1 source=shim time="2020-03-17T17:50:39.019566139Z" level=info msg="copy stdout failed" container=4d4a0cfcb196cae46cd97a7a0495945bd201a66267d237c432f1ea42daf07a85 error="rpc error: code = Unknown desc = EOF" exec-id=4d4a0cfcb196cae46cd97a7a0495945bd201a66267d237c432f1ea42daf07a85 name=kata-shim pid=1 source=shim time="2020-03-17T17:50:51.851371177Z" level=info msg="copy stderr failed" container=1284226a23bf29f23c0228907d4080b3a314cccece0b1304adb22f878c3ecf28 error="rpc error: code = Unknown desc = EOF" exec-id=1284226a23bf29f23c0228907d4080b3a314cccece0b1304adb22f878c3ecf28 name=kata-shim pid=1 source=shim time="2020-03-17T17:50:51.852506682Z" level=info msg="copy stdout failed" container=1284226a23bf29f23c0228907d4080b3a314cccece0b1304adb22f878c3ecf28 error="rpc error: code = Unknown desc = EOF" exec-id=1284226a23bf29f23c0228907d4080b3a314cccece0b1304adb22f878c3ecf28 name=kata-shim pid=1 source=shim time="2020-03-17T17:50:55.935901087Z" level=info msg="copy stderr failed" container=50a87dd6139cd3158d413c1c8e5460610a283381ef9e1150378b48f0a5f9045d error="rpc error: code = Unknown desc = EOF" exec-id=50a87dd6139cd3158d413c1c8e5460610a283381ef9e1150378b48f0a5f9045d name=kata-shim pid=1 source=shim time="2020-03-17T17:50:55.936437789Z" level=info msg="copy stdout failed" container=50a87dd6139cd3158d413c1c8e5460610a283381ef9e1150378b48f0a5f9045d error="rpc error: code = Unknown desc = EOF" exec-id=50a87dd6139cd3158d413c1c8e5460610a283381ef9e1150378b48f0a5f9045d name=kata-shim pid=1 source=shim time="2020-03-17T17:50:59.803540494Z" level=info msg="copy stdout failed" container=15288558b81ea3a58b706746f9825aa4137cd10bb39263bc353e622ba3153eb3 error="rpc error: code = Unknown desc = read /dev/ptmx: input/output error" exec-id=15288558b81ea3a58b706746f9825aa4137cd10bb39263bc353e622ba3153eb3 name=kata-shim pid=1 source=shim time="2020-03-17T17:51:05.680032666Z" level=info msg="copy stderr failed" container=992b94d81a146c5d122715cd970ec29221bad138201dd6fae14fd3f74061477c error="rpc error: code = Unknown desc = EOF" exec-id=992b94d81a146c5d122715cd970ec29221bad138201dd6fae14fd3f74061477c name=kata-shim pid=1 source=shim time="2020-03-17T17:51:05.680248067Z" level=info msg="copy stdout failed" container=992b94d81a146c5d122715cd970ec29221bad138201dd6fae14fd3f74061477c error="rpc error: code = Unknown desc = EOF" exec-id=992b94d81a146c5d122715cd970ec29221bad138201dd6fae14fd3f74061477c name=kata-shim pid=1 source=shim ``` ## Throttler logs No recent throttler problems found in system journal. --- # Container manager details No `docker` No `kubectl` Have `crio` ## crio Output of "`crio --version`": ``` crio version 1.10.7-dev commit: "6273bea4c9ed788aeb3d051ebf2d030060c05b6c-dirty" ``` Output of "`systemctl show crio`": ``` Type=simple Restart=on-failure NotifyAccess=none RestartUSec=5s TimeoutStartUSec=1min 30s TimeoutStopUSec=1min 30s TimeoutAbortUSec=1min 30s RuntimeMaxUSec=infinity WatchdogUSec=0 WatchdogTimestampMonotonic=0 RootDirectoryStartOnly=no RemainAfterExit=no GuessMainPID=yes MainPID=0 ControlPID=0 FileDescriptorStoreMax=0 NFileDescriptorStore=0 StatusErrno=0 Result=success ReloadResult=success CleanResult=success UID=[not set] GID=[not set] NRestarts=0 OOMPolicy=stop ExecMainStartTimestampMonotonic=0 ExecMainExitTimestampMonotonic=0 ExecMainPID=0 ExecMainCode=0 ExecMainStatus=0 ExecStart={ path=/usr/local/bin/crio ; argv[]=/usr/local/bin/crio --log-level debug ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 } ExecStartEx={ path=/usr/local/bin/crio ; argv[]=/usr/local/bin/crio --log-level debug ; flags= ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 } Slice=system.slice MemoryCurrent=[not set] CPUUsageNSec=[not set] TasksCurrent=[not set] IPIngressBytes=[no data] IPIngressPackets=[no data] IPEgressBytes=[no data] IPEgressPackets=[no data] IOReadBytes=18446744073709551615 IOReadOperations=18446744073709551615 IOWriteBytes=18446744073709551615 IOWriteOperations=18446744073709551615 Delegate=no CPUAccounting=yes CPUWeight=[not set] StartupCPUWeight=[not set] CPUShares=[not set] StartupCPUShares=[not set] CPUQuotaPerSecUSec=infinity CPUQuotaPeriodUSec=infinity IOAccounting=no IOWeight=[not set] StartupIOWeight=[not set] BlockIOAccounting=no BlockIOWeight=[not set] StartupBlockIOWeight=[not set] MemoryAccounting=yes DefaultMemoryLow=0 DefaultMemoryMin=0 MemoryMin=0 MemoryLow=0 MemoryHigh=infinity MemoryMax=infinity MemorySwapMax=infinity MemoryLimit=infinity DevicePolicy=auto TasksAccounting=yes TasksMax=19145 IPAccounting=no UMask=0022 LimitCPU=infinity LimitCPUSoft=infinity LimitFSIZE=infinity LimitFSIZESoft=infinity LimitDATA=infinity LimitDATASoft=infinity LimitSTACK=infinity LimitSTACKSoft=8388608 LimitCORE=infinity LimitCORESoft=infinity LimitRSS=infinity LimitRSSSoft=infinity LimitNOFILE=524288 LimitNOFILESoft=1024 LimitAS=infinity LimitASSoft=infinity LimitNPROC=63817 LimitNPROCSoft=63817 LimitMEMLOCK=65536 LimitMEMLOCKSoft=65536 LimitLOCKS=infinity LimitLOCKSSoft=infinity LimitSIGPENDING=63817 LimitSIGPENDINGSoft=63817 LimitMSGQUEUE=819200 LimitMSGQUEUESoft=819200 LimitNICE=0 LimitNICESoft=0 LimitRTPRIO=0 LimitRTPRIOSoft=0 LimitRTTIME=infinity LimitRTTIMESoft=infinity OOMScoreAdjust=0 Nice=0 IOSchedulingClass=0 IOSchedulingPriority=0 CPUSchedulingPolicy=0 CPUSchedulingPriority=0 CPUAffinity= NUMAPolicy=n/a NUMAMask= TimerSlackNSec=50000 CPUSchedulingResetOnFork=no NonBlocking=no StandardInput=null StandardInputData= StandardOutput=journal StandardError=inherit TTYReset=no TTYVHangup=no TTYVTDisallocate=no SyslogPriority=30 SyslogLevelPrefix=yes SyslogLevel=6 SyslogFacility=3 LogLevelMax=-1 LogRateLimitIntervalUSec=0 LogRateLimitBurst=0 SecureBits=0 CapabilityBoundingSet=cap_chown cap_dac_override cap_dac_read_search cap_fowner cap_fsetid cap_kill cap_setgid cap_setuid cap_setpcap cap_linux_immutable cap_net_bind_service cap_net_broadcast cap_net_admin cap_net_raw cap_ipc_lock cap_ipc_owner cap_sys_module cap_sys_rawio cap_sys_chroot cap_sys_ptrace cap_sys_pacct cap_sys_admin cap_sys_boot cap_sys_nice cap_sys_resource cap_sys_time cap_sys_tty_config cap_mknod cap_lease cap_audit_write cap_audit_control cap_setfcap cap_mac_override cap_mac_admin cap_syslog cap_wake_alarm cap_block_suspend cap_audit_read AmbientCapabilities= DynamicUser=no RemoveIPC=no MountFlags= PrivateTmp=no PrivateDevices=no ProtectKernelTunables=no ProtectKernelModules=no ProtectControlGroups=no PrivateNetwork=no PrivateUsers=no PrivateMounts=no ProtectHome=no ProtectSystem=no SameProcessGroup=no UtmpMode=init IgnoreSIGPIPE=yes NoNewPrivileges=no SystemCallErrorNumber=0 LockPersonality=no RuntimeDirectoryPreserve=no RuntimeDirectoryMode=0755 StateDirectoryMode=0755 CacheDirectoryMode=0755 LogsDirectoryMode=0755 ConfigurationDirectoryMode=0755 TimeoutCleanUSec=infinity MemoryDenyWriteExecute=no RestrictRealtime=no RestrictSUIDSGID=no RestrictNamespaces=no MountAPIVFS=no KeyringMode=private ProtectHostname=no KillMode=control-group KillSignal=15 FinalKillSignal=9 SendSIGKILL=yes SendSIGHUP=no WatchdogSignal=6 Id=crio.service Names=crio.service Requires=sysinit.target system.slice Conflicts=shutdown.target Before=shutdown.target After=basic.target system.slice sysinit.target systemd-journald.socket Documentation=https://github.com/cri-o/cri-o Description=CRI-O daemon LoadState=loaded ActiveState=inactive SubState=dead FragmentPath=/etc/systemd/system/crio.service UnitFileState=disabled UnitFilePreset=disabled StateChangeTimestampMonotonic=0 InactiveExitTimestampMonotonic=0 ActiveEnterTimestampMonotonic=0 ActiveExitTimestampMonotonic=0 InactiveEnterTimestampMonotonic=0 CanStart=yes CanStop=yes CanReload=no CanIsolate=no StopWhenUnneeded=no RefuseManualStart=no RefuseManualStop=no AllowIsolate=no DefaultDependencies=yes OnFailureJobMode=replace IgnoreOnIsolate=no NeedDaemonReload=no JobTimeoutUSec=infinity JobRunningTimeoutUSec=infinity JobTimeoutAction=none ConditionResult=no AssertResult=no ConditionTimestampMonotonic=0 AssertTimestampMonotonic=0 Transient=no Perpetual=no StartLimitIntervalUSec=10s StartLimitBurst=5 StartLimitAction=none FailureAction=none SuccessAction=none CollectMode=inactive ``` Output of "`cat /etc/crio/crio.conf`": ``` # The "crio" table contains all of the server options. [crio] # root is a path to the "root directory". CRIO stores all of its data, # including container images, in this directory. root = "/var/lib/containers/storage" # run is a path to the "run directory". CRIO stores all of its state # in this directory. runroot = "/var/run/containers/storage" # storage_driver select which storage driver is used to manage storage # of images and containers. storage_driver = "" # storage_option is used to pass an option to the storage driver. storage_option = [ ] # The "crio.api" table contains settings for the kubelet/gRPC interface. [crio.api] # listen is the path to the AF_LOCAL socket on which crio will listen. listen = "/var/run/crio/crio.sock" # stream_address is the IP address on which the stream server will listen stream_address = "" # stream_port is the port on which the stream server will listen stream_port = "10010" # file_locking is whether file-based locking will be used instead of # in-memory locking file_locking = true # The "crio.runtime" table contains settings pertaining to the OCI # runtime used and options for how to set up and manage the OCI runtime. [crio.runtime] manage_network_ns_lifecycle = true manage_network_ns_lifecycle = true # runtime is the OCI compatible runtime used for trusted container workloads. # This is a mandatory setting as this runtime will be the default one # and will also be used for untrusted container workloads if # runtime_untrusted_workload is not set. runtime = "/usr/local/bin/crio-runc" # runtime_untrusted_workload is the OCI compatible runtime used for untrusted # container workloads. This is an optional setting, except if # default_container_trust is set to "untrusted". runtime_untrusted_workload = "/usr/local/bin/kata-runtime" # default_workload_trust is the default level of trust crio puts in container # workloads. It can either be "trusted" or "untrusted", and the default # is "trusted". # Containers can be run through different container runtimes, depending on # the trust hints we receive from kubelet: # - If kubelet tags a container workload as untrusted, crio will try first to # run it through the untrusted container workload runtime. If it is not set, # crio will use the trusted runtime. # - If kubelet does not provide any information about the container workload trust # level, the selected runtime will depend on the default_container_trust setting. # If it is set to "untrusted", then all containers except for the host privileged # ones, will be run by the runtime_untrusted_workload runtime. Host privileged # containers are by definition trusted and will always use the trusted container # runtime. If default_container_trust is set to "trusted", crio will use the trusted # container runtime for all containers. default_workload_trust = "trusted" # no_pivot instructs the runtime to not use pivot_root, but instead use MS_MOVE no_pivot = false # conmon is the path to conmon binary, used for managing the runtime. conmon = "/usr/local/libexec/crio/conmon" # conmon_env is the environment variable list for conmon process, # used for passing necessary environment variable to conmon or runtime. conmon_env = [ "PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin", ] # selinux indicates whether or not SELinux will be used for pod # separation on the host. If you enable this flag, SELinux must be running # on the host. selinux = false # seccomp_profile is the seccomp json profile path which is used as the # default for the runtime. seccomp_profile = "/etc/crio/seccomp.json" # apparmor_profile is the apparmor profile name which is used as the # default for the runtime. apparmor_profile = "crio-default" # cgroup_manager is the cgroup management implementation to be used # for the runtime. cgroup_manager = "cgroupfs" # hooks_dir_path is the oci hooks directory for automatically executed hooks hooks_dir_path = "/usr/share/containers/oci/hooks.d" # default_mounts is the mounts list to be mounted for the container when created default_mounts = [ ] # pids_limit is the number of processes allowed in a container pids_limit = 1024 # log_size_max is the max limit for the container log size in bytes. # Negative values indicate that no limit is imposed. log_size_max = -1 # The "crio.image" table contains settings pertaining to the # management of OCI images. [crio.image] # default_transport is the prefix we try prepending to an image name if the # image name as we receive it can't be parsed as a valid source reference default_transport = "docker://" # pause_image is the image which we use to instantiate infra containers. pause_image = "k8s.gcr.io/pause" # pause_command is the command to run in a pause_image to have a container just # sit there. If the image contains the necessary information, this value need # not be specified. pause_command = "/pause" # signature_policy is the name of the file which decides what sort of policy we # use when deciding whether or not to trust an image that we've pulled. # Outside of testing situations, it is strongly advised that this be left # unspecified so that the default system-wide policy will be used. signature_policy = "" # image_volumes controls how image volumes are handled. # The valid values are mkdir and ignore. image_volumes = "mkdir" # insecure_registries is used to skip TLS verification when pulling images. insecure_registries = [ ] # registries is used to specify a comma separated list of registries to be used # when pulling an unqualified image (e.g. fedora:rawhide). registries = [ "docker.io" "docker.io" ] # The "crio.network" table contains settings pertaining to the # management of CNI plugins. [crio.network] # network_dir is is where CNI network configuration # files are stored. network_dir = "/etc/cni/net.d/" # plugin_dir is is where CNI plugin binaries are stored. plugin_dir = "/opt/cni/bin/" ``` No `containerd` --- # Packages No `dpkg` Have `rpm` Output of "`rpm -qa|egrep "(cc-oci-runtimecc-runtimerunv|kata-proxy|kata-runtime|kata-shim|kata-ksm-throttler|kata-containers-image|linux-container|qemu-)"`": ``` ``` ---

kata-containers / runtime

Errors with podman 1.8.1 #2535