pmem test not running on virtiofs, initrd and fedora

[GabyCT]

Currently the pmem test is not working on virtiofs, initrd, and fedora.

• Virtiofs problem

  docker: Error response from daemon: OCI runtime create failed: QMP command failed: Lack of libpmem support while setting the 'pmem=on' of memory-backend-file. We can't ensure data persistence.: unknown.

• Initrd problem

  16:28:33 docker: Error response from daemon: OCI runtime create failed: QMP command failed: nvdimm is not enabled: missing 'nvdimm' in '-M': unknown.
  16:28:33 Failed at 46: docker run -d --name "${container_name}" --runtime kata-runtime -v "${test_directory}:/${test_directory_name}" "${image}" sh -c "${payload}"

• Fedora problem

  16:06:51 docker: Error response from daemon: OCI runtime create failed: rpc error: code = Internal desc = Could not mount /dev/pmem1 to /run/kata-containers/shared/containers/22782a95-e072-40aa-903b-8f2d4c39b3da-test_pmem1: invalid argument: unknown.

[GabyCT]

This is the log from fedora

Show kata-collect-data.sh details

# Meta details Running `kata-collect-data.sh` version `1.11.0-alpha1 (commit 705713b4f9bc4d1e754871d5ef1ab5e99ea71aff)` at `2020-03-31.22:07:01.933006614+0000`. --- Runtime is `/usr/local/bin/kata-runtime`. # `kata-env` Output of "`/usr/local/bin/kata-runtime kata-env`": ```toml [Meta] Version = "1.0.24" [Runtime] Debug = false Trace = false DisableGuestSeccomp = true DisableNewNetNs = false SandboxCgroupOnly = false Path = "/usr/local/bin/kata-runtime" [Runtime.Version] OCI = "1.0.1-dev" [Runtime.Version.Version] Semver = "1.11.0-alpha1" Major = 1 Minor = 11 Patch = 0 Commit = "705713b4f9bc4d1e754871d5ef1ab5e99ea71aff" [Runtime.Config] Path = "/usr/share/defaults/kata-containers/configuration.toml" [[Runtime.Experimental]] Name = "newstore" Description = "This is a new storage driver which reorganized disk data structures, it has to be an experimental feature since it breaks backward compatibility." ExpRelease = "2.0" [Hypervisor] MachineType = "pc" Version = "QEMU emulator version 4.1.1 (kata-static)\nCopyright (c) 2003-2019 Fabrice Bellard and the QEMU Project developers" Path = "/usr/bin/qemu-system-x86_64" BlockDeviceDriver = "virtio-scsi" EntropySource = "/dev/urandom" SharedFS = "virtio-9p" VirtioFSDaemon = "/usr/bin/virtiofsd" Msize9p = 8192 MemorySlots = 10 PCIeRootPort = 0 HotplugVFIOOnRootBus = false Debug = false UseVSock = false [Image] Path = "/usr/share/kata-containers/kata-containers-clearlinux-32700-osbuilder-891b61c-agent-73afd1a.img" [Kernel] Path = "/usr/share/kata-containers/vmlinuz-5.4.15-71" Parameters = "systemd.unit=kata-containers.target systemd.mask=systemd-networkd.service systemd.mask=systemd-networkd.socket agent.log=debug" [Initrd] Path = "" [Proxy] Type = "kataProxy" Path = "/usr/libexec/kata-containers/kata-proxy" Debug = false [Proxy.Version] Semver = "1.11.0-alpha1-e7d2214f303fe9dfc433f9045659218e75f4d779" Major = 1 Minor = 11 Patch = 0 Commit = "e7d2214f303fe9dfc433f9045659218e75f4d779" [Shim] Type = "kataShim" Path = "/usr/libexec/kata-containers/kata-shim" Debug = false [Shim.Version] Semver = "1.11.0-alpha1-6a828a430c3d35e6ee22ee50a4fd2ed61280ad42" Major = 1 Minor = 11 Patch = 0 Commit = "6a828a430c3d35e6ee22ee50a4fd2ed61280ad42" [Agent] Type = "kata" Debug = false Trace = false TraceMode = "" TraceType = "" [Host] Kernel = "5.3.7-301.fc31.x86_64" Architecture = "amd64" VMContainerCapable = true SupportVSocks = true [Host.Distro] Name = "Fedora" Version = "31" [Host.CPU] Vendor = "GenuineIntel" Model = "Intel(R) Xeon(R) CPU E5-2673 v4 @ 2.30GHz" [Netmon] Path = "/usr/libexec/kata-containers/kata-netmon" Debug = false Enable = false [Netmon.Version] Semver = "1.11.0-alpha1" Major = 1 Minor = 11 Patch = 0 Commit = "<>" ``` --- # Runtime config files ## Runtime default config files ``` /etc/kata-containers/configuration.toml /usr/share/defaults/kata-containers/configuration.toml ``` ## Runtime config file contents Config file `/etc/kata-containers/configuration.toml` not found Output of "`cat "/usr/share/defaults/kata-containers/configuration.toml"`": ```toml # Copyright (c) 2017-2019 Intel Corporation # # SPDX-License-Identifier: Apache-2.0 # # XXX: WARNING: this file is auto-generated. # XXX: # XXX: Source file: "cli/config/configuration-qemu.toml.in" # XXX: Project: # XXX: Name: Kata Containers # XXX: Type: kata [hypervisor.qemu] path = "/usr/bin/qemu-system-x86_64" kernel = "/usr/share/kata-containers/vmlinuz.container" image = "/usr/share/kata-containers/kata-containers.img" machine_type = "pc" # Optional space-separated list of options to pass to the guest kernel. # For example, use `kernel_params = "vsyscall=emulate"` if you are having # trouble running pre-2.15 glibc. # # WARNING: - any parameter specified here will take priority over the default # parameter value of the same name used to start the virtual machine. # Do not set values here unless you understand the impact of doing so as you # may stop the virtual machine from booting. # To see the list of default parameters, enable hypervisor debug, create a # container and look for 'default-kernel-parameters' log entries. kernel_params = " agent.log=debug" # Path to the firmware. # If you want that qemu uses the default firmware leave this option empty firmware = "" # Machine accelerators # comma-separated list of machine accelerators to pass to the hypervisor. # For example, `machine_accelerators = "nosmm,nosmbus,nosata,nopit,static-prt,nofw"` machine_accelerators="" # Default number of vCPUs per SB/VM: # unspecified or 0 --> will be set to 1 # < 0 --> will be set to the actual number of physical cores # > 0 <= number of physical cores --> will be set to the specified number # > number of physical cores --> will be set to the actual number of physical cores default_vcpus = 1 # Default maximum number of vCPUs per SB/VM: # unspecified or == 0 --> will be set to the actual number of physical cores or to the maximum number # of vCPUs supported by KVM if that number is exceeded # > 0 <= number of physical cores --> will be set to the specified number # > number of physical cores --> will be set to the actual number of physical cores or to the maximum number # of vCPUs supported by KVM if that number is exceeded # WARNING: Depending of the architecture, the maximum number of vCPUs supported by KVM is used when # the actual number of physical cores is greater than it. # WARNING: Be aware that this value impacts the virtual machine's memory footprint and CPU # the hotplug functionality. For example, `default_maxvcpus = 240` specifies that until 240 vCPUs # can be added to a SB/VM, but the memory footprint will be big. Another example, with # `default_maxvcpus = 8` the memory footprint will be small, but 8 will be the maximum number of # vCPUs supported by the SB/VM. In general, we recommend that you do not edit this variable, # unless you know what are you doing. default_maxvcpus = 0 # Bridges can be used to hot plug devices. # Limitations: # * Currently only pci bridges are supported # * Until 30 devices per bridge can be hot plugged. # * Until 5 PCI bridges can be cold plugged per VM. # This limitation could be a bug in qemu or in the kernel # Default number of bridges per SB/VM: # unspecified or 0 --> will be set to 1 # > 1 <= 5 --> will be set to the specified number # > 5 --> will be set to 5 default_bridges = 1 # Default memory size in MiB for SB/VM. # If unspecified then it will be set 2048 MiB. default_memory = 2048 # # Default memory slots per SB/VM. # If unspecified then it will be set 10. # This is will determine the times that memory will be hotadded to sandbox/VM. #memory_slots = 10 # The size in MiB will be plused to max memory of hypervisor. # It is the memory address space for the NVDIMM devie. # If set block storage driver (block_device_driver) to "nvdimm", # should set memory_offset to the size of block device. # Default 0 #memory_offset = 0 # Specifies virtio-mem will be enabled or not. # Please note that this option should be used with the command # "echo 1 > /proc/sys/vm/overcommit_memory". # Default false #enable_virtio_mem = true # Disable block device from being used for a container's rootfs. # In case of a storage driver like devicemapper where a container's # root file system is backed by a block device, the block device is passed # directly to the hypervisor for performance reasons. # This flag prevents the block device from being passed to the hypervisor, # 9pfs is used instead to pass the rootfs. disable_block_device_use = false # Shared file system type: # - virtio-9p (default) # - virtio-fs shared_fs = "virtio-9p" # Path to vhost-user-fs daemon. virtio_fs_daemon = "/usr/bin/virtiofsd" # Default size of DAX cache in MiB virtio_fs_cache_size = 1024 # Extra args for virtiofsd daemon # # Format example: # ["-o", "arg1=xxx,arg2", "-o", "hello world", "--arg3=yyy"] # # see `virtiofsd -h` for possible options. virtio_fs_extra_args = [] # Cache mode: # # - none # Metadata, data, and pathname lookup are not cached in guest. They are # always fetched from host and any changes are immediately pushed to host. # # - auto # Metadata and pathname lookup cache expires after a configured amount of # time (default is 1 second). Data is cached while the file is open (close # to open consistency). # # - always # Metadata, data, and pathname lookup are cached in guest and never expire. virtio_fs_cache = "always" # Block storage driver to be used for the hypervisor in case the container # rootfs is backed by a block device. This is virtio-scsi, virtio-blk # or nvdimm. block_device_driver = "virtio-scsi" # Specifies cache-related options will be set to block devices or not. # Default false #block_device_cache_set = true # Specifies cache-related options for block devices. # Denotes whether use of O_DIRECT (bypass the host page cache) is enabled. # Default false #block_device_cache_direct = true # Specifies cache-related options for block devices. # Denotes whether flush requests for the device are ignored. # Default false #block_device_cache_noflush = true # Enable iothreads (data-plane) to be used. This causes IO to be # handled in a separate IO thread. This is currently only implemented # for SCSI. # enable_iothreads = false # Enable pre allocation of VM RAM, default false # Enabling this will result in lower container density # as all of the memory will be allocated and locked # This is useful when you want to reserve all the memory # upfront or in the cases where you want memory latencies # to be very predictable # Default false #enable_mem_prealloc = true # Enable huge pages for VM RAM, default false # Enabling this will result in the VM memory # being allocated using huge pages. # This is useful when you want to use vhost-user network # stacks within the container. This will automatically # result in memory pre allocation #enable_hugepages = true # Enable vhost-user storage device, default false # Enabling this will result in some Linux reserved block type # major range 240-254 being chosen to represent vhost-user devices. enable_vhost_user_store = false # The base directory specifically used for vhost-user devices. # Its sub-path "block" is used for block devices; "block/sockets" is # where we expect vhost-user sockets to live; "block/devices" is where # simulated block device nodes for vhost-user devices to live. vhost_user_store_path = "/var/run/kata-containers/vhost-user" # Enable file based guest memory support. The default is an empty string which # will disable this feature. In the case of virtio-fs, this is enabled # automatically and '/dev/shm' is used as the backing folder. # This option will be ignored if VM templating is enabled. #file_mem_backend = "" # Enable swap of vm memory. Default false. # The behaviour is undefined if mem_prealloc is also set to true #enable_swap = true # This option changes the default hypervisor and kernel parameters # to enable debug output where available. This extra output is added # to the proxy logs, but only when proxy debug is also enabled. # # Default false #enable_debug = true # Disable the customizations done in the runtime when it detects # that it is running on top a VMM. This will result in the runtime # behaving as it would when running on bare metal. # #disable_nesting_checks = true # This is the msize used for 9p shares. It is the number of bytes # used for 9p packet payload. #msize_9p = 8192 # If true and vsocks are supported, use vsocks to communicate directly # with the agent and no proxy is started, otherwise use unix # sockets and start a proxy to communicate with the agent. # Default false #use_vsock = true # If false and nvdimm is supported, use nvdimm device to plug guest image. # Otherwise virtio-block device is used. # Default is false #disable_image_nvdimm = true # VFIO devices are hotplugged on a bridge by default. # Enable hotplugging on root bus. This may be required for devices with # a large PCI bar, as this is a current limitation with hotplugging on # a bridge. This value is valid for "pc" machine type. # Default false #hotplug_vfio_on_root_bus = true # Before hot plugging a PCIe device, you need to add a pcie_root_port device. # Use this parameter when using some large PCI bar devices, such as Nvidia GPU # The value means the number of pcie_root_port # This value is valid when hotplug_vfio_on_root_bus is true and machine_type is "q35" # Default 0 #pcie_root_port = 2 # If vhost-net backend for virtio-net is not desired, set to true. Default is false, which trades off # security (vhost-net runs ring0) for network I/O performance. #disable_vhost_net = true # # Default entropy source. # The path to a host source of entropy (including a real hardware RNG) # /dev/urandom and /dev/random are two main options. # Be aware that /dev/random is a blocking source of entropy. If the host # runs out of entropy, the VMs boot time will increase leading to get startup # timeouts. # The source of entropy /dev/urandom is non-blocking and provides a # generally acceptable source of entropy. It should work well for pretty much # all practical purposes. #entropy_source= "/dev/urandom" # Path to OCI hook binaries in the *guest rootfs*. # This does not affect host-side hooks which must instead be added to # the OCI spec passed to the runtime. # # You can create a rootfs with hooks by customizing the osbuilder scripts: # https://github.com/kata-containers/osbuilder # # Hooks must be stored in a subdirectory of guest_hook_path according to their # hook type, i.e. "guest_hook_path/{prestart,postart,poststop}". # The agent will scan these directories for executable files and add them, in # lexicographical order, to the lifecycle of the guest container. # Hooks are executed in the runtime namespace of the guest. See the official documentation: # https://github.com/opencontainers/runtime-spec/blob/v1.0.1/config.md#posix-platform-hooks # Warnings will be logged if any error is encountered will scanning for hooks, # but it will not abort container execution. #guest_hook_path = "/usr/share/oci/hooks" [factory] # VM templating support. Once enabled, new VMs are created from template # using vm cloning. They will share the same initial kernel, initramfs and # agent memory by mapping it readonly. It helps speeding up new container # creation and saves a lot of memory if there are many kata containers running # on the same host. # # When disabled, new VMs are created from scratch. # # Note: Requires "initrd=" to be set ("image=" is not supported). # # Default false #enable_template = true # Specifies the path of template. # # Default "/run/vc/vm/template" #template_path = "/run/vc/vm/template" # The number of caches of VMCache: # unspecified or == 0 --> VMCache is disabled # > 0 --> will be set to the specified number # # VMCache is a function that creates VMs as caches before using it. # It helps speed up new container creation. # The function consists of a server and some clients communicating # through Unix socket. The protocol is gRPC in protocols/cache/cache.proto. # The VMCache server will create some VMs and cache them by factory cache. # It will convert the VM to gRPC format and transport it when gets # requestion from clients. # Factory grpccache is the VMCache client. It will request gRPC format # VM and convert it back to a VM. If VMCache function is enabled, # kata-runtime will request VM from factory grpccache when it creates # a new sandbox. # # Default 0 #vm_cache_number = 0 # Specify the address of the Unix socket that is used by VMCache. # # Default /var/run/kata-containers/cache.sock #vm_cache_endpoint = "/var/run/kata-containers/cache.sock" [proxy.kata] path = "/usr/libexec/kata-containers/kata-proxy" # If enabled, proxy messages will be sent to the system log # (default: disabled) #enable_debug = true [shim.kata] path = "/usr/libexec/kata-containers/kata-shim" enable_tracing = false # If enabled, shim messages will be sent to the system log # (default: disabled) #enable_debug = true # If enabled, the shim will create opentracing.io traces and spans. # (See https://www.jaegertracing.io/docs/getting-started). # # Note: By default, the shim runs in a separate network namespace. Therefore, # to allow it to send trace details to the Jaeger agent running on the host, # it is necessary to set 'disable_new_netns=true' so that it runs in the host # network namespace. # # (default: disabled) #enable_tracing = true [agent.kata] # If enabled, make the agent display debug-level messages. # (default: disabled) #enable_debug = true # Enable agent tracing. # # If enabled, the default trace mode is "dynamic" and the # default trace type is "isolated". The trace mode and type are set # explicity with the `trace_type=` and `trace_mode=` options. # # Notes: # # - Tracing is ONLY enabled when `enable_tracing` is set: explicitly # setting `trace_mode=` and/or `trace_type=` without setting `enable_tracing` # will NOT activate agent tracing. # # - See https://github.com/kata-containers/agent/blob/master/TRACING.md for # full details. # # (default: disabled) #enable_tracing = true # #trace_mode = "dynamic" #trace_type = "isolated" # Comma separated list of kernel modules and their parameters. # These modules will be loaded in the guest kernel using modprobe(8). # The following example can be used to load two kernel modules with parameters # - kernel_modules=["e1000e InterruptThrottleRate=3000,3000,3000 EEE=1", "i915 enable_ppgtt=0"] # The first word is considered as the module name and the rest as its parameters. # Container will not be started when: # * A kernel module is specified and the modprobe command is not installed in the guest # or it fails loading the module. # * The module is not available in the guest or it doesn't met the guest kernel # requirements, like architecture and version. # kernel_modules=[] [netmon] # If enabled, the network monitoring process gets started when the # sandbox is created. This allows for the detection of some additional # network being added to the existing network namespace, after the # sandbox has been created. # (default: disabled) #enable_netmon = true # Specify the path to the netmon binary. path = "/usr/libexec/kata-containers/kata-netmon" enable_netmon = false # If enabled, netmon messages will be sent to the system log # (default: disabled) #enable_debug = true [runtime] # If enabled, the runtime will log additional debug messages to the # system log # (default: disabled) #enable_debug = true # # Internetworking model # Determines how the VM should be connected to the # the container network interface # Options: # # - macvtap # Used when the Container network interface can be bridged using # macvtap. # # - none # Used when customize network. Only creates a tap device. No veth pair. # # - tcfilter # Uses tc filter rules to redirect traffic from the network interface # provided by plugin to a tap interface connected to the VM. # internetworking_model="macvtap" # disable guest seccomp # Determines whether container seccomp profiles are passed to the virtual # machine and applied by the kata agent. If set to true, seccomp is not applied # within the guest # (default: true) disable_guest_seccomp=true # If enabled, the runtime will create opentracing.io traces and spans. # (See https://www.jaegertracing.io/docs/getting-started). # (default: disabled) #enable_tracing = true # If enabled, the runtime will not create a network namespace for shim and hypervisor processes. # This option may have some potential impacts to your host. It should only be used when you know what you're doing. # `disable_new_netns` conflicts with `enable_netmon` # `disable_new_netns` conflicts with `internetworking_model=tcfilter` and `internetworking_model=macvtap`. It works only # with `internetworking_model=none`. The tap device will be in the host network namespace and can connect to a bridge # (like OVS) directly. # If you are using docker, `disable_new_netns` only works with `docker run --net=none` # (default: false) #disable_new_netns = true # if enabled, the runtime will add all the kata processes inside one dedicated cgroup. # The container cgroups in the host are not created, just one single cgroup per sandbox. # The runtime caller is free to restrict or collect cgroup stats of the overall Kata sandbox. # The sandbox cgroup path is the parent cgroup of a container with the PodSandbox annotation. # The sandbox cgroup is constrained if there is no container type annotation. # See: https://godoc.org/github.com/kata-containers/runtime/virtcontainers#ContainerType sandbox_cgroup_only=false # Enabled experimental feature list, format: ["a", "b"]. # Experimental features are features not stable enough for production, # they may break compatibility, and are prepared for a big version bump. # Supported experimental features: # (default: []) experimental=[ "newstore" ] enable_tracing = false disable_new_netns = false ``` --- # KSM throttler ## version Output of "` --version`": ``` /usr/local/bin/kata-collect-data.sh: line 178: --version: command not found ``` ## systemd service # Image details ```yaml --- osbuilder: url: "https://github.com/kata-containers/osbuilder" version: "unknown" rootfs-creation-time: "2020-03-30T19:10:00.713681621+0000Z" description: "osbuilder rootfs" file-format-version: "0.0.2" architecture: "x86_64" base-distro: name: "Clear" version: "32700" packages: default: - "chrony" - "iptables-bin" - "kmod-bin" - "libudev0-shim" - "systemd" - "util-linux-bin" extra: agent: url: "https://github.com/kata-containers/agent" name: "kata-agent" version: "1.11.0-alpha1-73afd1a31736490e5fda4c4b779d84f945acb187" agent-is-init-daemon: "no" ``` --- # Initrd details No initrd --- # Logfiles ## Runtime logs Recent runtime problems found in system journal: ``` time="2020-03-31T22:06:29.599122097Z" level=warning msg="cgroups have not been created and cgroup path is empty" source=virtcontainers/pkg/cgroups time="2020-03-31T22:06:29.600798906Z" level=warning msg="failed to get sandbox config from old store: open /var/lib/vc/sbs/17b84496453e3c540d08bde02707d90adabbe43a34d156cc96ebed67d3e11bef/config.json: no such file or directory" arch=amd64 command=delete container=17b84496453e3c540d08bde02707d90adabbe43a34d156cc96ebed67d3e11bef name=kata-runtime pid=136440 sandbox=17b84496453e3c540d08bde02707d90adabbe43a34d156cc96ebed67d3e11bef source=virtcontainers time="2020-03-31T22:06:29.60152521Z" level=warning msg="cgroups have not been created and cgroup path is empty" source=virtcontainers/pkg/cgroups time="2020-03-31T22:06:29.602747016Z" level=warning msg="failed to get sandbox config from old store: open /var/lib/vc/sbs/17b84496453e3c540d08bde02707d90adabbe43a34d156cc96ebed67d3e11bef/config.json: no such file or directory" arch=amd64 command=delete container=17b84496453e3c540d08bde02707d90adabbe43a34d156cc96ebed67d3e11bef name=kata-runtime pid=136440 sandbox=17b84496453e3c540d08bde02707d90adabbe43a34d156cc96ebed67d3e11bef source=virtcontainers time="2020-03-31T22:06:29.603368019Z" level=warning msg="cgroups have not been created and cgroup path is empty" source=virtcontainers/pkg/cgroups time="2020-03-31T22:06:29.607953243Z" level=error msg="Could not read qemu pid file" arch=amd64 command=delete container=17b84496453e3c540d08bde02707d90adabbe43a34d156cc96ebed67d3e11bef error="open /run/vc/vm/17b84496453e3c540d08bde02707d90adabbe43a34d156cc96ebed67d3e11bef/pid: no such file or directory" name=kata-runtime pid=136440 sandbox=17b84496453e3c540d08bde02707d90adabbe43a34d156cc96ebed67d3e11bef source=virtcontainers subsystem=qemu time="2020-03-31T22:06:29.608176744Z" level=warning msg="sandbox cgroups path is empty" arch=amd64 command=delete container=17b84496453e3c540d08bde02707d90adabbe43a34d156cc96ebed67d3e11bef name=kata-runtime pid=136440 sandbox=17b84496453e3c540d08bde02707d90adabbe43a34d156cc96ebed67d3e11bef source=virtcontainers subsystem=sandbox time="2020-03-31T22:06:32.730395959Z" level=info msg="sanner return error: read unix @->/run/vc/vm/319e83e44fa95fec79ebca4c78807096eb2e411dcd5ebafa5cb8f6e016854e23/qmp.sock: use of closed network connection" arch=amd64 command=create container=319e83e44fa95fec79ebca4c78807096eb2e411dcd5ebafa5cb8f6e016854e23 name=kata-runtime pid=137023 source=virtcontainers subsystem=qmp time="2020-03-31T22:06:33.659406183Z" level=warning msg="sandbox's cgroup won't be updated: cgroup path is empty" arch=amd64 command=create container=319e83e44fa95fec79ebca4c78807096eb2e411dcd5ebafa5cb8f6e016854e23 name=kata-runtime pid=137023 sandbox=319e83e44fa95fec79ebca4c78807096eb2e411dcd5ebafa5cb8f6e016854e23 source=virtcontainers subsystem=sandbox time="2020-03-31T22:06:33.660462989Z" level=info msg="sanner return error: read unix @->/run/vc/vm/319e83e44fa95fec79ebca4c78807096eb2e411dcd5ebafa5cb8f6e016854e23/qmp.sock: use of closed network connection" arch=amd64 command=create container=319e83e44fa95fec79ebca4c78807096eb2e411dcd5ebafa5cb8f6e016854e23 name=kata-runtime pid=137023 source=virtcontainers subsystem=qmp time="2020-03-31T22:06:33.703346611Z" level=warning msg="failed to get sandbox config from old store: open /var/lib/vc/sbs/319e83e44fa95fec79ebca4c78807096eb2e411dcd5ebafa5cb8f6e016854e23/config.json: no such file or directory" arch=amd64 command=state container=319e83e44fa95fec79ebca4c78807096eb2e411dcd5ebafa5cb8f6e016854e23 name=kata-runtime pid=137092 source=virtcontainers time="2020-03-31T22:06:33.70509512Z" level=warning msg="cgroups have not been created and cgroup path is empty" source=virtcontainers/pkg/cgroups time="2020-03-31T22:06:33.744920527Z" level=warning msg="failed to get sandbox config from old store: open /var/lib/vc/sbs/319e83e44fa95fec79ebca4c78807096eb2e411dcd5ebafa5cb8f6e016854e23/config.json: no such file or directory" arch=amd64 command=start container=319e83e44fa95fec79ebca4c78807096eb2e411dcd5ebafa5cb8f6e016854e23 name=kata-runtime pid=137101 source=virtcontainers time="2020-03-31T22:06:33.746623836Z" level=warning msg="cgroups have not been created and cgroup path is empty" source=virtcontainers/pkg/cgroups time="2020-03-31T22:06:33.748626547Z" level=warning msg="failed to get sandbox config from old store: open /var/lib/vc/sbs/319e83e44fa95fec79ebca4c78807096eb2e411dcd5ebafa5cb8f6e016854e23/config.json: no such file or directory" arch=amd64 command=start container=319e83e44fa95fec79ebca4c78807096eb2e411dcd5ebafa5cb8f6e016854e23 name=kata-runtime pid=137101 sandbox=319e83e44fa95fec79ebca4c78807096eb2e411dcd5ebafa5cb8f6e016854e23 source=virtcontainers time="2020-03-31T22:06:33.74936185Z" level=warning msg="cgroups have not been created and cgroup path is empty" source=virtcontainers/pkg/cgroups time="2020-03-31T22:06:33.811739374Z" level=warning msg="failed to get sandbox config from old store: open /var/lib/vc/sbs/319e83e44fa95fec79ebca4c78807096eb2e411dcd5ebafa5cb8f6e016854e23/config.json: no such file or directory" arch=amd64 command=state container=319e83e44fa95fec79ebca4c78807096eb2e411dcd5ebafa5cb8f6e016854e23 name=kata-runtime pid=137112 source=virtcontainers time="2020-03-31T22:06:33.813456083Z" level=warning msg="cgroups have not been created and cgroup path is empty" source=virtcontainers/pkg/cgroups time="2020-03-31T22:06:33.875325605Z" level=warning msg="failed to get sandbox config from old store: open /var/lib/vc/sbs/319e83e44fa95fec79ebca4c78807096eb2e411dcd5ebafa5cb8f6e016854e23/config.json: no such file or directory" arch=amd64 command=kill container=319e83e44fa95fec79ebca4c78807096eb2e411dcd5ebafa5cb8f6e016854e23 name=kata-runtime pid=137124 source=virtcontainers time="2020-03-31T22:06:33.877382415Z" level=warning msg="cgroups have not been created and cgroup path is empty" source=virtcontainers/pkg/cgroups time="2020-03-31T22:06:33.879437426Z" level=warning msg="failed to get sandbox config from old store: open /var/lib/vc/sbs/319e83e44fa95fec79ebca4c78807096eb2e411dcd5ebafa5cb8f6e016854e23/config.json: no such file or directory" arch=amd64 command=kill container=319e83e44fa95fec79ebca4c78807096eb2e411dcd5ebafa5cb8f6e016854e23 name=kata-runtime pid=137124 sandbox=319e83e44fa95fec79ebca4c78807096eb2e411dcd5ebafa5cb8f6e016854e23 source=virtcontainers time="2020-03-31T22:06:33.880494831Z" level=warning msg="cgroups have not been created and cgroup path is empty" source=virtcontainers/pkg/cgroups time="2020-03-31T22:06:33.941538748Z" level=error msg="Could not read qemu pid file" arch=amd64 command=kill container=319e83e44fa95fec79ebca4c78807096eb2e411dcd5ebafa5cb8f6e016854e23 error="open /run/vc/vm/319e83e44fa95fec79ebca4c78807096eb2e411dcd5ebafa5cb8f6e016854e23/pid: no such file or directory" name=kata-runtime pid=137124 sandbox=319e83e44fa95fec79ebca4c78807096eb2e411dcd5ebafa5cb8f6e016854e23 source=virtcontainers subsystem=qemu time="2020-03-31T22:06:33.942498353Z" level=error msg="Could not read qemu pid file" arch=amd64 command=kill container=319e83e44fa95fec79ebca4c78807096eb2e411dcd5ebafa5cb8f6e016854e23 error="open /run/vc/vm/319e83e44fa95fec79ebca4c78807096eb2e411dcd5ebafa5cb8f6e016854e23/pid: no such file or directory" name=kata-runtime pid=137124 sandbox=319e83e44fa95fec79ebca4c78807096eb2e411dcd5ebafa5cb8f6e016854e23 source=virtcontainers subsystem=qemu time="2020-03-31T22:06:33.943048356Z" level=info msg="sanner return error: read unix @->/run/vc/vm/319e83e44fa95fec79ebca4c78807096eb2e411dcd5ebafa5cb8f6e016854e23/qmp.sock: use of closed network connection" arch=amd64 command=kill container=319e83e44fa95fec79ebca4c78807096eb2e411dcd5ebafa5cb8f6e016854e23 name=kata-runtime pid=137124 sandbox=319e83e44fa95fec79ebca4c78807096eb2e411dcd5ebafa5cb8f6e016854e23 source=virtcontainers subsystem=qmp time="2020-03-31T22:06:34.013672523Z" level=warning msg="failed to get sandbox config from old store: open /var/lib/vc/sbs/319e83e44fa95fec79ebca4c78807096eb2e411dcd5ebafa5cb8f6e016854e23/config.json: no such file or directory" arch=amd64 command=state container=319e83e44fa95fec79ebca4c78807096eb2e411dcd5ebafa5cb8f6e016854e23 name=kata-runtime pid=137146 source=virtcontainers time="2020-03-31T22:06:34.015754734Z" level=warning msg="cgroups have not been created and cgroup path is empty" source=virtcontainers/pkg/cgroups time="2020-03-31T22:06:34.062454676Z" level=warning msg="failed to get sandbox config from old store: open /var/lib/vc/sbs/319e83e44fa95fec79ebca4c78807096eb2e411dcd5ebafa5cb8f6e016854e23/config.json: no such file or directory" arch=amd64 command=kill container=319e83e44fa95fec79ebca4c78807096eb2e411dcd5ebafa5cb8f6e016854e23 name=kata-runtime pid=137155 source=virtcontainers time="2020-03-31T22:06:34.064162385Z" level=warning msg="cgroups have not been created and cgroup path is empty" source=virtcontainers/pkg/cgroups time="2020-03-31T22:06:34.066210096Z" level=warning msg="failed to get sandbox config from old store: open /var/lib/vc/sbs/319e83e44fa95fec79ebca4c78807096eb2e411dcd5ebafa5cb8f6e016854e23/config.json: no such file or directory" arch=amd64 command=kill container=319e83e44fa95fec79ebca4c78807096eb2e411dcd5ebafa5cb8f6e016854e23 name=kata-runtime pid=137155 sandbox=319e83e44fa95fec79ebca4c78807096eb2e411dcd5ebafa5cb8f6e016854e23 source=virtcontainers time="2020-03-31T22:06:34.067154201Z" level=warning msg="cgroups have not been created and cgroup path is empty" source=virtcontainers/pkg/cgroups time="2020-03-31T22:06:34.068327307Z" level=error msg="Could not read qemu pid file" arch=amd64 command=kill container=319e83e44fa95fec79ebca4c78807096eb2e411dcd5ebafa5cb8f6e016854e23 error="open /run/vc/vm/319e83e44fa95fec79ebca4c78807096eb2e411dcd5ebafa5cb8f6e016854e23/pid: no such file or directory" name=kata-runtime pid=137155 sandbox=319e83e44fa95fec79ebca4c78807096eb2e411dcd5ebafa5cb8f6e016854e23 source=virtcontainers subsystem=qemu time="2020-03-31T22:06:34.109069418Z" level=warning msg="failed to get sandbox config from old store: open /var/lib/vc/sbs/319e83e44fa95fec79ebca4c78807096eb2e411dcd5ebafa5cb8f6e016854e23/config.json: no such file or directory" arch=amd64 command=delete container=319e83e44fa95fec79ebca4c78807096eb2e411dcd5ebafa5cb8f6e016854e23 name=kata-runtime pid=137164 source=virtcontainers time="2020-03-31T22:06:34.111046829Z" level=warning msg="cgroups have not been created and cgroup path is empty" source=virtcontainers/pkg/cgroups time="2020-03-31T22:06:34.112844938Z" level=warning msg="failed to get sandbox config from old store: open /var/lib/vc/sbs/319e83e44fa95fec79ebca4c78807096eb2e411dcd5ebafa5cb8f6e016854e23/config.json: no such file or directory" arch=amd64 command=delete container=319e83e44fa95fec79ebca4c78807096eb2e411dcd5ebafa5cb8f6e016854e23 name=kata-runtime pid=137164 sandbox=319e83e44fa95fec79ebca4c78807096eb2e411dcd5ebafa5cb8f6e016854e23 source=virtcontainers time="2020-03-31T22:06:34.113667842Z" level=warning msg="cgroups have not been created and cgroup path is empty" source=virtcontainers/pkg/cgroups time="2020-03-31T22:06:34.11509325Z" level=warning msg="failed to get sandbox config from old store: open /var/lib/vc/sbs/319e83e44fa95fec79ebca4c78807096eb2e411dcd5ebafa5cb8f6e016854e23/config.json: no such file or directory" arch=amd64 command=delete container=319e83e44fa95fec79ebca4c78807096eb2e411dcd5ebafa5cb8f6e016854e23 name=kata-runtime pid=137164 sandbox=319e83e44fa95fec79ebca4c78807096eb2e411dcd5ebafa5cb8f6e016854e23 source=virtcontainers time="2020-03-31T22:06:34.115798153Z" level=warning msg="cgroups have not been created and cgroup path is empty" source=virtcontainers/pkg/cgroups time="2020-03-31T22:06:34.120645579Z" level=error msg="Could not read qemu pid file" arch=amd64 command=delete container=319e83e44fa95fec79ebca4c78807096eb2e411dcd5ebafa5cb8f6e016854e23 error="open /run/vc/vm/319e83e44fa95fec79ebca4c78807096eb2e411dcd5ebafa5cb8f6e016854e23/pid: no such file or directory" name=kata-runtime pid=137164 sandbox=319e83e44fa95fec79ebca4c78807096eb2e411dcd5ebafa5cb8f6e016854e23 source=virtcontainers subsystem=qemu time="2020-03-31T22:06:34.12097578Z" level=warning msg="sandbox cgroups path is empty" arch=amd64 command=delete container=319e83e44fa95fec79ebca4c78807096eb2e411dcd5ebafa5cb8f6e016854e23 name=kata-runtime pid=137164 sandbox=319e83e44fa95fec79ebca4c78807096eb2e411dcd5ebafa5cb8f6e016854e23 source=virtcontainers subsystem=sandbox time="2020-03-31T22:06:49.955699114Z" level=info msg="sanner return error: read unix @->/run/vc/vm/c70f07c865a0fc1f8b6bed0addb4bdeefc320a4d24007b2e2c959bca90edf910/qmp.sock: use of closed network connection" arch=amd64 command=create container=c70f07c865a0fc1f8b6bed0addb4bdeefc320a4d24007b2e2c959bca90edf910 name=kata-runtime pid=137670 source=virtcontainers subsystem=qmp time="2020-03-31T22:06:50.709559428Z" level=info msg="{\"error\": {\"class\": \"DeviceNotFound\", \"desc\": \"Device 'virtio-drive-7739ac8a4232285b' not found\"}}" arch=amd64 command=create container=c70f07c865a0fc1f8b6bed0addb4bdeefc320a4d24007b2e2c959bca90edf910 name=kata-runtime pid=137670 source=virtcontainers subsystem=qmp time="2020-03-31T22:06:50.709754629Z" level=error msg="Failed to unplug block device" arch=amd64 command=create container=c70f07c865a0fc1f8b6bed0addb4bdeefc320a4d24007b2e2c959bca90edf910 error="QMP command failed: Device 'virtio-drive-7739ac8a4232285b' not found" name=kata-runtime pid=137670 source=virtcontainers subsystem=device time="2020-03-31T22:06:50.70985293Z" level=error msg="rollback failed detachDevices()" arch=amd64 command=create container=c70f07c865a0fc1f8b6bed0addb4bdeefc320a4d24007b2e2c959bca90edf910 error="QMP command failed: Device 'virtio-drive-7739ac8a4232285b' not found" name=kata-runtime pid=137670 sandbox=c70f07c865a0fc1f8b6bed0addb4bdeefc320a4d24007b2e2c959bca90edf910 source=virtcontainers subsystem=container time="2020-03-31T22:06:50.70993453Z" level=warning msg="Could not umount" arch=amd64 command=create container=c70f07c865a0fc1f8b6bed0addb4bdeefc320a4d24007b2e2c959bca90edf910 error="no such file or directory" host-path=/run/kata-containers/shared/sandboxes/c70f07c865a0fc1f8b6bed0addb4bdeefc320a4d24007b2e2c959bca90edf910/c70f07c865a0fc1f8b6bed0addb4bdeefc320a4d24007b2e2c959bca90edf910-7e2a4e7eb9c8c4a4-resolv.conf name=kata-runtime pid=137670 sandbox=c70f07c865a0fc1f8b6bed0addb4bdeefc320a4d24007b2e2c959bca90edf910 source=virtcontainers subsystem=container time="2020-03-31T22:06:50.709975931Z" level=error msg="rollback failed unmountHostMounts()" arch=amd64 command=create container=c70f07c865a0fc1f8b6bed0addb4bdeefc320a4d24007b2e2c959bca90edf910 error="no such file or directory" name=kata-runtime pid=137670 sandbox=c70f07c865a0fc1f8b6bed0addb4bdeefc320a4d24007b2e2c959bca90edf910 source=virtcontainers subsystem=container time="2020-03-31T22:06:50.710022231Z" level=warning msg="no such file or directory: /run/kata-containers/shared/sandboxes/c70f07c865a0fc1f8b6bed0addb4bdeefc320a4d24007b2e2c959bca90edf910/c70f07c865a0fc1f8b6bed0addb4bdeefc320a4d24007b2e2c959bca90edf910/rootfs" time="2020-03-31T22:06:50.749205534Z" level=warning msg="sandbox cgroups path is empty" arch=amd64 command=create container=c70f07c865a0fc1f8b6bed0addb4bdeefc320a4d24007b2e2c959bca90edf910 name=kata-runtime pid=137670 sandbox=c70f07c865a0fc1f8b6bed0addb4bdeefc320a4d24007b2e2c959bca90edf910 source=virtcontainers subsystem=sandbox time="2020-03-31T22:06:50.749467236Z" level=warning msg="failed to cleanup netns" arch=amd64 command=create container=c70f07c865a0fc1f8b6bed0addb4bdeefc320a4d24007b2e2c959bca90edf910 error="failed to get netns /var/run/netns/cnitest-126d1bad-1319-cc94-e49f-c7a47a204e41: failed to Statfs \"/var/run/netns/cnitest-126d1bad-1319-cc94-e49f-c7a47a204e41\": no such file or directory" name=kata-runtime path=/var/run/netns/cnitest-126d1bad-1319-cc94-e49f-c7a47a204e41 pid=137670 source=katautils time="2020-03-31T22:06:50.749534936Z" level=error msg="rpc error: code = Internal desc = Could not mount /dev/pmem1 to /run/kata-containers/shared/containers/22782a95-e072-40aa-903b-8f2d4c39b3da-test_pmem1: invalid argument" arch=amd64 command=create container=c70f07c865a0fc1f8b6bed0addb4bdeefc320a4d24007b2e2c959bca90edf910 name=kata-runtime pid=137670 source=runtime ``` ## Proxy logs Recent proxy problems found in system journal: ``` time="2020-03-31T21:58:30.191412629Z" level=info msg="time=\"2020-03-31T21:58:30.17733751Z\" level=debug msg=\"request end\" debug_console=false duration=\"14.719µs\" name=kata-agent pid=40 request=/grpc.AgentService/CloseStdin resp=\"&Empty{XXX_unrecognized:[],}\" sandbox=7a2aa54186624e16bfe9626c4d1f606bf14efb6b6cd7616de3a9862778592d75 source=agent\n" name=kata-proxy pid=106448 sandbox=7a2aa54186624e16bfe9626c4d1f606bf14efb6b6cd7616de3a9862778592d75 source=agent time="2020-03-31T21:58:30.295594956Z" level=info msg="time=\"2020-03-31T21:58:30.28207373Z\" level=debug msg=\"request end\" debug_console=false duration=3.87647ms name=kata-agent pid=40 request=/grpc.AgentService/StartContainer resp=\"&Empty{XXX_unrecognized:[],}\" sandbox=7a2aa54186624e16bfe9626c4d1f606bf14efb6b6cd7616de3a9862778592d75 source=agent\n" name=kata-proxy pid=106448 sandbox=7a2aa54186624e16bfe9626c4d1f606bf14efb6b6cd7616de3a9862778592d75 source=agent time="2020-03-31T21:58:30.479116385Z" level=debug msg="Copy stream error" error="read unix /run/vc/sbs/7a2aa54186624e16bfe9626c4d1f606bf14efb6b6cd7616de3a9862778592d75/proxy.sock->@: use of closed network connection" name=kata-proxy pid=106448 sandbox=7a2aa54186624e16bfe9626c4d1f606bf14efb6b6cd7616de3a9862778592d75 source=proxy time="2020-03-31T21:58:30.479889389Z" level=fatal msg="channel error" error="session shutdown" name=kata-proxy pid=106448 sandbox=7a2aa54186624e16bfe9626c4d1f606bf14efb6b6cd7616de3a9862778592d75 source=proxy time="2020-03-31T21:58:44.329106663Z" level=info msg="[ 0.327808] intel_pstate: CPU model not supported\n" name=kata-proxy pid=106990 sandbox=23ba2d5aa58e8802b9587b5e4e0ea1d920be6fd5e944d256841fc95a9b5277f5 source=agent time="2020-03-31T21:58:44.336775302Z" level=info msg="[ 0.335481] EXT4-fs (pmem0p1): DAX enabled. Warning: EXPERIMENTAL, use at your own risk\n" name=kata-proxy pid=106990 sandbox=23ba2d5aa58e8802b9587b5e4e0ea1d920be6fd5e944d256841fc95a9b5277f5 source=agent time="2020-03-31T21:58:44.338043609Z" level=info msg="[ 0.336382] EXT4-fs (pmem0p1): mounted filesystem with ordered data mode. Opts: dax,data=ordered,errors=remount-ro\n" name=kata-proxy pid=106990 sandbox=23ba2d5aa58e8802b9587b5e4e0ea1d920be6fd5e944d256841fc95a9b5277f5 source=agent time="2020-03-31T21:58:44.372632984Z" level=info msg="[ 0.371380] systemd[1]: Failed to stat /etc/localtime, ignoring: No such file or directory\n" name=kata-proxy pid=106990 sandbox=23ba2d5aa58e8802b9587b5e4e0ea1d920be6fd5e944d256841fc95a9b5277f5 source=agent time="2020-03-31T21:58:44.398561115Z" level=info msg="[ 0.397098] systemd[1]: unit_file_build_name_map: normal unit file: /usr/lib/systemd/system/systemd-boot-check-no-failures.service\n" name=kata-proxy pid=106990 sandbox=23ba2d5aa58e8802b9587b5e4e0ea1d920be6fd5e944d256841fc95a9b5277f5 source=agent time="2020-03-31T21:58:44.474827301Z" level=info msg="[ 0.473445] systemd-sysctl[39]: Couldn't write '16' to 'kernel/sysrq', ignoring: No such file or directory\n" name=kata-proxy pid=106990 sandbox=23ba2d5aa58e8802b9587b5e4e0ea1d920be6fd5e944d256841fc95a9b5277f5 source=agent time="2020-03-31T21:58:44.55579001Z" level=debug msg="Copy stream error" error="write unix /run/vc/sbs/23ba2d5aa58e8802b9587b5e4e0ea1d920be6fd5e944d256841fc95a9b5277f5/proxy.sock->@: use of closed network connection" name=kata-proxy pid=106990 sandbox=23ba2d5aa58e8802b9587b5e4e0ea1d920be6fd5e944d256841fc95a9b5277f5 source=proxy time="2020-03-31T21:58:44.566539065Z" level=info msg="time=\"2020-03-31T21:58:44.55183411Z\" level=debug msg=\"request end\" debug_console=false duration=5.572885ms name=kata-agent pid=40 request=/grpc.AgentService/CreateSandbox resp=\"&Empty{XXX_unrecognized:[],}\" sandbox=23ba2d5aa58e8802b9587b5e4e0ea1d920be6fd5e944d256841fc95a9b5277f5 source=agent\n" name=kata-proxy pid=106990 sandbox=23ba2d5aa58e8802b9587b5e4e0ea1d920be6fd5e944d256841fc95a9b5277f5 source=agent time="2020-03-31T21:58:44.681202045Z" level=info msg="time=\"2020-03-31T21:58:44.666454889Z\" level=debug msg=\"request end\" debug_console=false duration=96.115701ms name=kata-agent pid=40 request=/grpc.AgentService/CreateContainer resp=\"&Empty{XXX_unrecognized:[],}\" sandbox=23ba2d5aa58e8802b9587b5e4e0ea1d920be6fd5e944d256841fc95a9b5277f5 source=agent\n" name=kata-proxy pid=106990 sandbox=23ba2d5aa58e8802b9587b5e4e0ea1d920be6fd5e944d256841fc95a9b5277f5 source=agent time="2020-03-31T21:58:44.681885348Z" level=info msg="time=\"2020-03-31T21:58:44.666766712Z\" level=info msg=\"ignoring unexpected signal\" debug_console=false name=kata-agent pid=40 sandbox=23ba2d5aa58e8802b9587b5e4e0ea1d920be6fd5e944d256841fc95a9b5277f5 signal=\"child exited\" source=agent\n" name=kata-proxy pid=106990 sandbox=23ba2d5aa58e8802b9587b5e4e0ea1d920be6fd5e944d256841fc95a9b5277f5 source=agent time="2020-03-31T21:58:44.682335851Z" level=info msg="time=\"2020-03-31T21:58:44.667604213Z\" level=info msg=\"ignoring unexpected signal\" debug_console=false name=kata-agent pid=40 sandbox=23ba2d5aa58e8802b9587b5e4e0ea1d920be6fd5e944d256841fc95a9b5277f5 signal=\"child exited\" source=agent\n" name=kata-proxy pid=106990 sandbox=23ba2d5aa58e8802b9587b5e4e0ea1d920be6fd5e944d256841fc95a9b5277f5 source=agent time="2020-03-31T21:58:44.692615103Z" level=info msg="time=\"2020-03-31T21:58:44.67779861Z\" level=debug msg=\"request end\" debug_console=false duration=\"3.817µs\" name=kata-agent pid=40 request=/grpc.AgentService/OnlineCPUMem resp=\"&Empty{XXX_unrecognized:[],}\" sandbox=23ba2d5aa58e8802b9587b5e4e0ea1d920be6fd5e944d256841fc95a9b5277f5 source=agent\n" name=kata-proxy pid=106990 sandbox=23ba2d5aa58e8802b9587b5e4e0ea1d920be6fd5e944d256841fc95a9b5277f5 source=agent time="2020-03-31T21:58:44.704348862Z" level=info msg="time=\"2020-03-31T21:58:44.689561118Z\" level=debug msg=\"request end\" debug_console=false duration=\"10.76µs\" name=kata-agent pid=40 request=/grpc.AgentService/CloseStdin resp=\"&Empty{XXX_unrecognized:[],}\" sandbox=23ba2d5aa58e8802b9587b5e4e0ea1d920be6fd5e944d256841fc95a9b5277f5 source=agent\n" name=kata-proxy pid=106990 sandbox=23ba2d5aa58e8802b9587b5e4e0ea1d920be6fd5e944d256841fc95a9b5277f5 source=agent time="2020-03-31T21:58:44.808054187Z" level=info msg="time=\"2020-03-31T21:58:44.793103313Z\" level=debug msg=\"request end\" debug_console=false duration=3.980583ms name=kata-agent pid=40 request=/grpc.AgentService/StartContainer resp=\"&Empty{XXX_unrecognized:[],}\" sandbox=23ba2d5aa58e8802b9587b5e4e0ea1d920be6fd5e944d256841fc95a9b5277f5 source=agent\n" name=kata-proxy pid=106990 sandbox=23ba2d5aa58e8802b9587b5e4e0ea1d920be6fd5e944d256841fc95a9b5277f5 source=agent time="2020-03-31T21:58:55.082801275Z" level=info msg="time=\"2020-03-31T21:58:55.068060451Z\" level=debug msg=\"request end\" debug_console=false duration=\"302.981µs\" name=kata-agent pid=40 request=/grpc.AgentService/SignalProcess resp=\"&Empty{XXX_unrecognized:[],}\" sandbox=23ba2d5aa58e8802b9587b5e4e0ea1d920be6fd5e944d256841fc95a9b5277f5 source=agent\n" name=kata-proxy pid=106990 sandbox=23ba2d5aa58e8802b9587b5e4e0ea1d920be6fd5e944d256841fc95a9b5277f5 source=agent time="2020-03-31T21:58:55.088498804Z" level=info msg="time=\"2020-03-31T21:58:55.073737525Z\" level=info msg=\"ignoring unexpected signal\" debug_console=false name=kata-agent pid=40 sandbox=23ba2d5aa58e8802b9587b5e4e0ea1d920be6fd5e944d256841fc95a9b5277f5 signal=\"child exited\" source=agent\n" name=kata-proxy pid=106990 sandbox=23ba2d5aa58e8802b9587b5e4e0ea1d920be6fd5e944d256841fc95a9b5277f5 source=agent time="2020-03-31T21:58:55.189726116Z" level=info msg="time=\"2020-03-31T21:58:55.174945993Z\" level=debug msg=\"request end\" debug_console=false duration=\"223.065µs\" name=kata-agent pid=40 request=/grpc.AgentService/SignalProcess resp=\"&Empty{XXX_unrecognized:[],}\" sandbox=23ba2d5aa58e8802b9587b5e4e0ea1d920be6fd5e944d256841fc95a9b5277f5 source=agent\n" name=kata-proxy pid=106990 sandbox=23ba2d5aa58e8802b9587b5e4e0ea1d920be6fd5e944d256841fc95a9b5277f5 source=agent time="2020-03-31T21:58:55.196195849Z" level=info msg="time=\"2020-03-31T21:58:55.181428284Z\" level=debug msg=\"request end\" debug_console=false duration=\"716.101µs\" name=kata-agent pid=40 request=/grpc.AgentService/RemoveContainer resp=\"&Empty{XXX_unrecognized:[],}\" sandbox=23ba2d5aa58e8802b9587b5e4e0ea1d920be6fd5e944d256841fc95a9b5277f5 source=agent\n" name=kata-proxy pid=106990 sandbox=23ba2d5aa58e8802b9587b5e4e0ea1d920be6fd5e944d256841fc95a9b5277f5 source=agent time="2020-03-31T21:58:55.220463972Z" level=info msg="time=\"2020-03-31T21:58:55.205692616Z\" level=debug msg=\"request end\" debug_console=false duration=13.642225ms name=kata-agent pid=40 request=/grpc.AgentService/DestroySandbox resp=\"&Empty{XXX_unrecognized:[],}\" sandbox=23ba2d5aa58e8802b9587b5e4e0ea1d920be6fd5e944d256841fc95a9b5277f5 source=agent\n" name=kata-proxy pid=106990 sandbox=23ba2d5aa58e8802b9587b5e4e0ea1d920be6fd5e944d256841fc95a9b5277f5 source=agent time="2020-03-31T21:58:55.221526677Z" level=fatal msg="failed to handle exit signal" error="close unix @->/run/vc/vm/23ba2d5aa58e8802b9587b5e4e0ea1d920be6fd5e944d256841fc95a9b5277f5/kata.sock: use of closed network connection" name=kata-proxy pid=106990 sandbox=23ba2d5aa58e8802b9587b5e4e0ea1d920be6fd5e944d256841fc95a9b5277f5 source=proxy time="2020-03-31T21:59:07.670227873Z" level=info msg="[ 0.328960] intel_pstate: CPU model not supported\n" name=kata-proxy pid=107252 sandbox=2b08f7cbb58517f2a7e1005275a25edf271ed4755dc7fb2bb8eae0210e89686b source=agent time="2020-03-31T21:59:07.678674717Z" level=info msg="[ 0.337309] EXT4-fs (pmem0p1): DAX enabled. Warning: EXPERIMENTAL, use at your own risk\n" name=kata-proxy pid=107252 sandbox=2b08f7cbb58517f2a7e1005275a25edf271ed4755dc7fb2bb8eae0210e89686b source=agent time="2020-03-31T21:59:07.679540022Z" level=info msg="[ 0.338067] EXT4-fs (pmem0p1): mounted filesystem with ordered data mode. Opts: dax,data=ordered,errors=remount-ro\n" name=kata-proxy pid=107252 sandbox=2b08f7cbb58517f2a7e1005275a25edf271ed4755dc7fb2bb8eae0210e89686b source=agent time="2020-03-31T21:59:07.721875842Z" level=info msg="[ 0.380406] systemd[1]: Failed to stat /etc/localtime, ignoring: No such file or directory\n" name=kata-proxy pid=107252 sandbox=2b08f7cbb58517f2a7e1005275a25edf271ed4755dc7fb2bb8eae0210e89686b source=agent time="2020-03-31T21:59:07.74657237Z" level=info msg="[ 0.405175] systemd[1]: unit_file_build_name_map: normal unit file: /usr/lib/systemd/system/systemd-boot-check-no-failures.service\n" name=kata-proxy pid=107252 sandbox=2b08f7cbb58517f2a7e1005275a25edf271ed4755dc7fb2bb8eae0210e89686b source=agent time="2020-03-31T21:59:07.826743187Z" level=info msg="[ 0.485339] systemd-sysctl[39]: Couldn't write '16' to 'kernel/sysrq', ignoring: No such file or directory\n" name=kata-proxy pid=107252 sandbox=2b08f7cbb58517f2a7e1005275a25edf271ed4755dc7fb2bb8eae0210e89686b source=agent time="2020-03-31T21:59:07.908547813Z" level=info msg="time=\"2020-03-31T21:59:07.897020328Z\" level=debug msg=\"request end\" debug_console=false duration=5.747369ms name=kata-agent pid=40 request=/grpc.AgentService/CreateSandbox resp=\"&Empty{XXX_unrecognized:[],}\" sandbox=2b08f7cbb58517f2a7e1005275a25edf271ed4755dc7fb2bb8eae0210e89686b source=agent\n" name=kata-proxy pid=107252 sandbox=2b08f7cbb58517f2a7e1005275a25edf271ed4755dc7fb2bb8eae0210e89686b source=agent time="2020-03-31T21:59:08.015210368Z" level=info msg="time=\"2020-03-31T21:59:08.003634555Z\" level=debug msg=\"request end\" debug_console=false duration=91.095998ms name=kata-agent pid=40 request=/grpc.AgentService/CreateContainer resp=\"&Empty{XXX_unrecognized:[],}\" sandbox=2b08f7cbb58517f2a7e1005275a25edf271ed4755dc7fb2bb8eae0210e89686b source=agent\n" name=kata-proxy pid=107252 sandbox=2b08f7cbb58517f2a7e1005275a25edf271ed4755dc7fb2bb8eae0210e89686b source=agent time="2020-03-31T21:59:08.016015572Z" level=info msg="time=\"2020-03-31T21:59:08.004117074Z\" level=info msg=\"ignoring unexpected signal\" debug_console=false name=kata-agent pid=40 sandbox=2b08f7cbb58517f2a7e1005275a25edf271ed4755dc7fb2bb8eae0210e89686b signal=\"child exited\" source=agent\n" name=kata-proxy pid=107252 sandbox=2b08f7cbb58517f2a7e1005275a25edf271ed4755dc7fb2bb8eae0210e89686b source=agent time="2020-03-31T21:59:08.016245473Z" level=info msg="time=\"2020-03-31T21:59:08.004589822Z\" level=info msg=\"ignoring unexpected signal\" debug_console=false name=kata-agent pid=40 sandbox=2b08f7cbb58517f2a7e1005275a25edf271ed4755dc7fb2bb8eae0210e89686b signal=\"child exited\" source=agent\n" name=kata-proxy pid=107252 sandbox=2b08f7cbb58517f2a7e1005275a25edf271ed4755dc7fb2bb8eae0210e89686b source=agent time="2020-03-31T21:59:08.029909744Z" level=info msg="time=\"2020-03-31T21:59:08.018308536Z\" level=debug msg=\"request end\" debug_console=false duration=\"2.892µs\" name=kata-agent pid=40 request=/grpc.AgentService/OnlineCPUMem resp=\"&Empty{XXX_unrecognized:[],}\" sandbox=2b08f7cbb58517f2a7e1005275a25edf271ed4755dc7fb2bb8eae0210e89686b source=agent\n" name=kata-proxy pid=107252 sandbox=2b08f7cbb58517f2a7e1005275a25edf271ed4755dc7fb2bb8eae0210e89686b source=agent time="2020-03-31T21:59:08.042853211Z" level=info msg="time=\"2020-03-31T21:59:08.03123105Z\" level=debug msg=\"request end\" debug_console=false duration=\"29.771µs\" name=kata-agent pid=40 request=/grpc.AgentService/CloseStdin resp=\"&Empty{XXX_unrecognized:[],}\" sandbox=2b08f7cbb58517f2a7e1005275a25edf271ed4755dc7fb2bb8eae0210e89686b source=agent\n" name=kata-proxy pid=107252 sandbox=2b08f7cbb58517f2a7e1005275a25edf271ed4755dc7fb2bb8eae0210e89686b source=agent time="2020-03-31T21:59:08.147779957Z" level=info msg="time=\"2020-03-31T21:59:08.13617897Z\" level=debug msg=\"request end\" debug_console=false duration=3.045014ms name=kata-agent pid=40 request=/grpc.AgentService/StartContainer resp=\"&Empty{XXX_unrecognized:[],}\" sandbox=2b08f7cbb58517f2a7e1005275a25edf271ed4755dc7fb2bb8eae0210e89686b source=agent\n" name=kata-proxy pid=107252 sandbox=2b08f7cbb58517f2a7e1005275a25edf271ed4755dc7fb2bb8eae0210e89686b source=agent time="2020-03-31T21:59:08.161271727Z" level=info msg="time=\"2020-03-31T21:59:08.149662287Z\" level=info msg=\"ignoring unexpected signal\" debug_console=false name=kata-agent pid=40 sandbox=2b08f7cbb58517f2a7e1005275a25edf271ed4755dc7fb2bb8eae0210e89686b signal=\"child exited\" source=agent\n" name=kata-proxy pid=107252 sandbox=2b08f7cbb58517f2a7e1005275a25edf271ed4755dc7fb2bb8eae0210e89686b source=agent time="2020-03-31T21:59:08.201911339Z" level=info msg="time=\"2020-03-31T21:59:08.190285033Z\" level=debug msg=\"request end\" debug_console=false duration=\"318.767µs\" name=kata-agent pid=40 request=/grpc.AgentService/SignalProcess resp=\"&Empty{XXX_unrecognized:[],}\" sandbox=2b08f7cbb58517f2a7e1005275a25edf271ed4755dc7fb2bb8eae0210e89686b source=agent\n" name=kata-proxy pid=107252 sandbox=2b08f7cbb58517f2a7e1005275a25edf271ed4755dc7fb2bb8eae0210e89686b source=agent time="2020-03-31T21:59:08.209121876Z" level=info msg="time=\"2020-03-31T21:59:08.19763548Z\" level=debug msg=\"request end\" debug_console=false duration=\"469.726µs\" name=kata-agent pid=40 request=/grpc.AgentService/RemoveContainer resp=\"&Empty{XXX_unrecognized:[],}\" sandbox=2b08f7cbb58517f2a7e1005275a25edf271ed4755dc7fb2bb8eae0210e89686b source=agent\n" name=kata-proxy pid=107252 sandbox=2b08f7cbb58517f2a7e1005275a25edf271ed4755dc7fb2bb8eae0210e89686b source=agent time="2020-03-31T21:59:08.279942845Z" level=info msg="time=\"2020-03-31T21:59:08.268430958Z\" level=debug msg=\"request end\" debug_console=false duration=10.257606ms name=kata-agent pid=40 request=/grpc.AgentService/DestroySandbox resp=\"&Empty{XXX_unrecognized:[],}\" sandbox=2b08f7cbb58517f2a7e1005275a25edf271ed4755dc7fb2bb8eae0210e89686b source=agent\n" name=kata-proxy pid=107252 sandbox=2b08f7cbb58517f2a7e1005275a25edf271ed4755dc7fb2bb8eae0210e89686b source=agent time="2020-03-31T21:59:08.280482547Z" level=fatal msg="failed to handle exit signal" error="close unix @->/run/vc/vm/2b08f7cbb58517f2a7e1005275a25edf271ed4755dc7fb2bb8eae0210e89686b/kata.sock: use of closed network connection" name=kata-proxy pid=107252 sandbox=2b08f7cbb58517f2a7e1005275a25edf271ed4755dc7fb2bb8eae0210e89686b source=proxy time="2020-03-31T21:59:15.473069456Z" level=fatal msg="channel error" error="accept unix /run/vc/sbs/05303562d14cc20a57e40e93fc4b4e6c24def04a91063c4b6d5e6321d80bc488/proxy.sock: use of closed network connection" name=kata-proxy pid=108187 sandbox=05303562d14cc20a57e40e93fc4b4e6c24def04a91063c4b6d5e6321d80bc488 source=proxy time="2020-03-31T21:59:57.569431903Z" level=fatal msg="failed to handle exit signal" error="close unix @->/run/vc/vm/29be45bad394351448e3b7dcfe0e5a8e3e31ce210c6786671c36c911fb60a949/kata.sock: use of closed network connection" name=kata-proxy pid=110499 sandbox=29be45bad394351448e3b7dcfe0e5a8e3e31ce210c6786671c36c911fb60a949 source=proxy time="2020-03-31T22:00:04.655379157Z" level=fatal msg="channel error" error="accept unix /run/vc/sbs/7e80634a7b9f411cfadc6efa27d1c1405238e5b87914fa8ac48a1806c827b0eb/proxy.sock: use of closed network connection" name=kata-proxy pid=111222 sandbox=7e80634a7b9f411cfadc6efa27d1c1405238e5b87914fa8ac48a1806c827b0eb source=proxy time="2020-03-31T22:06:19.735701974Z" level=fatal msg="failed to handle exit signal" error="close unix @->/run/vc/vm/2f44839a949211840490fe93a4e391891c01c56945e8b1f10d335aa31476c565/kata.sock: use of closed network connection" name=kata-proxy pid=135349 sandbox=2f44839a949211840490fe93a4e391891c01c56945e8b1f10d335aa31476c565 source=proxy time="2020-03-31T22:06:22.982387235Z" level=fatal msg="failed to handle exit signal" error="close unix @->/run/vc/vm/51ddc9492e9cb9fdabc399cde56f1a499a54b62ac7e416b76eeda53bc99c4970/kata.sock: use of closed network connection" name=kata-proxy pid=135573 sandbox=51ddc9492e9cb9fdabc399cde56f1a499a54b62ac7e416b76eeda53bc99c4970 source=proxy time="2020-03-31T22:06:29.365737885Z" level=fatal msg="channel error" error="accept unix /run/vc/sbs/17b84496453e3c540d08bde02707d90adabbe43a34d156cc96ebed67d3e11bef/proxy.sock: use of closed network connection" name=kata-proxy pid=136179 sandbox=17b84496453e3c540d08bde02707d90adabbe43a34d156cc96ebed67d3e11bef source=proxy time="2020-03-31T22:06:33.908253076Z" level=fatal msg="channel error" error="accept unix /run/vc/sbs/319e83e44fa95fec79ebca4c78807096eb2e411dcd5ebafa5cb8f6e016854e23/proxy.sock: use of closed network connection" name=kata-proxy pid=137067 sandbox=319e83e44fa95fec79ebca4c78807096eb2e411dcd5ebafa5cb8f6e016854e23 source=proxy time="2020-03-31T22:06:50.730134335Z" level=fatal msg="channel error" error="accept unix /run/vc/sbs/c70f07c865a0fc1f8b6bed0addb4bdeefc320a4d24007b2e2c959bca90edf910/proxy.sock: use of closed network connection" name=kata-proxy pid=137708 sandbox=c70f07c865a0fc1f8b6bed0addb4bdeefc320a4d24007b2e2c959bca90edf910 source=proxy ``` ## Shim logs Recent shim problems found in system journal: ``` time="2020-03-31T21:56:02.698238569Z" level=info msg="copy stdout failed" container=0dfe2990a2a13706f61f6ba81c8c71d37d9a19b244341007e0b9bc8b9939bd8a error="rpc error: code = Unknown desc = EOF" exec-id=0dfe2990a2a13706f61f6ba81c8c71d37d9a19b244341007e0b9bc8b9939bd8a name=kata-shim pid=1 source=shim time="2020-03-31T21:57:10.852056485Z" level=info msg="copy stdout failed" container=7aaae098c01a4bbce84172dec018f1273f90e0ea04a42cf7d3138b1b47ae50b1 error="rpc error: code = Unknown desc = read /dev/ptmx: input/output error" exec-id=7aaae098c01a4bbce84172dec018f1273f90e0ea04a42cf7d3138b1b47ae50b1 name=kata-shim pid=1 source=shim time="2020-03-31T21:57:10.908703972Z" level=info msg="copy stdout failed" container=a94c853abe41f91e6a91b2773fa0fdf3d270cba50b07baaa18e32dd4b14698da error="rpc error: code = Unknown desc = read /dev/ptmx: input/output error" exec-id=a94c853abe41f91e6a91b2773fa0fdf3d270cba50b07baaa18e32dd4b14698da name=kata-shim pid=1 source=shim time="2020-03-31T21:57:10.912896493Z" level=info msg="copy stdout failed" container=6764a31be73543db873040eea18623329d857c6873e21161d0c96344eedddc4e error="rpc error: code = Unknown desc = read /dev/ptmx: input/output error" exec-id=6764a31be73543db873040eea18623329d857c6873e21161d0c96344eedddc4e name=kata-shim pid=1 source=shim time="2020-03-31T21:57:10.933797099Z" level=info msg="copy stdout failed" container=c1df1a7016a6fad4c55e7d7d8130f0e3e4813b357026f749e77ea9ab9750b7e2 error="rpc error: code = Unknown desc = read /dev/ptmx: input/output error" exec-id=c1df1a7016a6fad4c55e7d7d8130f0e3e4813b357026f749e77ea9ab9750b7e2 name=kata-shim pid=1 source=shim time="2020-03-31T21:57:10.999125229Z" level=info msg="copy stdout failed" container=af0b28257afd4f1086adceb6acea769d891d31b63d8e56dafb75b092f57b0b68 error="rpc error: code = Unknown desc = read /dev/ptmx: input/output error" exec-id=af0b28257afd4f1086adceb6acea769d891d31b63d8e56dafb75b092f57b0b68 name=kata-shim pid=1 source=shim time="2020-03-31T21:57:11.020759339Z" level=info msg="copy stdout failed" container=b20711b71320f81605931ca7ebc15b747e405dd11c11758e40cb1320db4d3883 error="rpc error: code = Unknown desc = read /dev/ptmx: input/output error" exec-id=b20711b71320f81605931ca7ebc15b747e405dd11c11758e40cb1320db4d3883 name=kata-shim pid=1 source=shim time="2020-03-31T21:57:11.030697689Z" level=info msg="copy stdout failed" container=cdd00847716292ec62d7246f79781c0bfbdb644c2feb19098cb961f30694cec0 error="rpc error: code = Unknown desc = read /dev/ptmx: input/output error" exec-id=cdd00847716292ec62d7246f79781c0bfbdb644c2feb19098cb961f30694cec0 name=kata-shim pid=1 source=shim time="2020-03-31T21:57:11.035416013Z" level=info msg="copy stdout failed" container=a20ec9b19ce748f4728763a44bd437d3895f1ecbc1cffce67c7848e4783eb97c error="rpc error: code = Unknown desc = read /dev/ptmx: input/output error" exec-id=a20ec9b19ce748f4728763a44bd437d3895f1ecbc1cffce67c7848e4783eb97c name=kata-shim pid=1 source=shim time="2020-03-31T21:57:11.042177447Z" level=info msg="copy stdout failed" container=cdcb10d894be3341f3879d344287247730dd3cc286617a51ea99c0e134a4da37 error="rpc error: code = Unknown desc = read /dev/ptmx: input/output error" exec-id=cdcb10d894be3341f3879d344287247730dd3cc286617a51ea99c0e134a4da37 name=kata-shim pid=1 source=shim time="2020-03-31T21:57:11.052915101Z" level=info msg="copy stdout failed" container=3801aca142d91fcf73955aba615b27fe691582b0722ecf2edf403f43f8f34cac error="rpc error: code = Unknown desc = read /dev/ptmx: input/output error" exec-id=3801aca142d91fcf73955aba615b27fe691582b0722ecf2edf403f43f8f34cac name=kata-shim pid=1 source=shim time="2020-03-31T21:57:11.077452526Z" level=info msg="copy stdout failed" container=790a8e7dabf62038b29736ee3ad2ea9a6d71ecc9b512960ec91a42cdfd992885 error="rpc error: code = Unknown desc = read /dev/ptmx: input/output error" exec-id=790a8e7dabf62038b29736ee3ad2ea9a6d71ecc9b512960ec91a42cdfd992885 name=kata-shim pid=1 source=shim time="2020-03-31T21:57:11.116044321Z" level=info msg="copy stdout failed" container=ce8603f92a202523ac2cdfd655d4a9926e1a1d3140826562cca6a45cca5474cc error="rpc error: code = Unknown desc = read /dev/ptmx: input/output error" exec-id=ce8603f92a202523ac2cdfd655d4a9926e1a1d3140826562cca6a45cca5474cc name=kata-shim pid=1 source=shim time="2020-03-31T21:57:11.116855825Z" level=info msg="copy stdout failed" container=4574fc81854d771db16e7c6c8828b90da858908ce9c7a7642adfbdeaaffc7c5b error="rpc error: code = Unknown desc = read /dev/ptmx: input/output error" exec-id=4574fc81854d771db16e7c6c8828b90da858908ce9c7a7642adfbdeaaffc7c5b name=kata-shim pid=1 source=shim time="2020-03-31T21:57:11.12571737Z" level=info msg="copy stdout failed" container=879b9d02aa6b8a92ddeb6a2c7d4d4c4d6726415b186023b20214e4ca4ac7333e error="rpc error: code = Unknown desc = read /dev/ptmx: input/output error" exec-id=879b9d02aa6b8a92ddeb6a2c7d4d4c4d6726415b186023b20214e4ca4ac7333e name=kata-shim pid=1 source=shim time="2020-03-31T21:57:11.127874781Z" level=info msg="copy stdout failed" container=9ec9e9635b69ccc7bfa8d0f131cbd059d4a8a5b5056c0ca1dca7bc7258db316d error="rpc error: code = Unknown desc = read /dev/ptmx: input/output error" exec-id=9ec9e9635b69ccc7bfa8d0f131cbd059d4a8a5b5056c0ca1dca7bc7258db316d name=kata-shim pid=1 source=shim time="2020-03-31T21:57:11.165995574Z" level=info msg="copy stdout failed" container=aab08faf1e2d901426422cebf3130cc9efc56eb2fdf5d2d8c65a68ae8413cb67 error="rpc error: code = Unknown desc = read /dev/ptmx: input/output error" exec-id=aab08faf1e2d901426422cebf3130cc9efc56eb2fdf5d2d8c65a68ae8413cb67 name=kata-shim pid=1 source=shim time="2020-03-31T21:57:11.199260442Z" level=info msg="copy stdout failed" container=54f1a16f48f87b231bd08da39a20710fd10c7ab548ec53183cec18ca4669be70 error="rpc error: code = Unknown desc = read /dev/ptmx: input/output error" exec-id=54f1a16f48f87b231bd08da39a20710fd10c7ab548ec53183cec18ca4669be70 name=kata-shim pid=1 source=shim time="2020-03-31T21:57:11.246548681Z" level=info msg="copy stdout failed" container=136736971e4823405e5d73922c685a43206f46f58782dab254b9a0758d73bf66 error="rpc error: code = Unknown desc = read /dev/ptmx: input/output error" exec-id=136736971e4823405e5d73922c685a43206f46f58782dab254b9a0758d73bf66 name=kata-shim pid=1 source=shim time="2020-03-31T21:57:11.33911835Z" level=info msg="copy stdout failed" container=c3b58008f8d2424341e681558738b276981302f7e412427c7a131e85328c697c error="rpc error: code = Unknown desc = read /dev/ptmx: input/output error" exec-id=c3b58008f8d2424341e681558738b276981302f7e412427c7a131e85328c697c name=kata-shim pid=1 source=shim time="2020-03-31T21:57:11.368676399Z" level=info msg="copy stdout failed" container=e35d5434e6150784388a99d47b922764d2494f5c2cfe9edd75c2b2b605df0876 error="rpc error: code = Unknown desc = read /dev/ptmx: input/output error" exec-id=e35d5434e6150784388a99d47b922764d2494f5c2cfe9edd75c2b2b605df0876 name=kata-shim pid=1 source=shim time="2020-03-31T21:58:15.39081344Z" level=info msg="copy stdout failed" container=ebe935455bc68052f74700aa9430991173eba52506ac2dafa547f4744f263985 error="rpc error: code = Unknown desc = read /dev/ptmx: input/output error" exec-id=ebe935455bc68052f74700aa9430991173eba52506ac2dafa547f4744f263985 name=kata-shim pid=1 source=shim time="2020-03-31T21:58:15.398203378Z" level=info msg="copy stdout failed" container=49e5dae7d3b8fdce4f77bd48f33a75bf587219743f3595a2e5a6c896efb79107 error="rpc error: code = Unknown desc = read /dev/ptmx: input/output error" exec-id=49e5dae7d3b8fdce4f77bd48f33a75bf587219743f3595a2e5a6c896efb79107 name=kata-shim pid=1 source=shim time="2020-03-31T21:58:15.414685061Z" level=info msg="copy stdout failed" container=f56acf9a93d5fc4361afa110d6ea260191e9e32ab2e1007770eff636e36a2b7b error="rpc error: code = Unknown desc = read /dev/ptmx: input/output error" exec-id=f56acf9a93d5fc4361afa110d6ea260191e9e32ab2e1007770eff636e36a2b7b name=kata-shim pid=1 source=shim time="2020-03-31T21:58:15.423536006Z" level=info msg="copy stdout failed" container=55675449cf17d620560d5f664ef8ac5fd11660542fd9f2d5c3375fffa5910938 error="rpc error: code = Unknown desc = read /dev/ptmx: input/output error" exec-id=55675449cf17d620560d5f664ef8ac5fd11660542fd9f2d5c3375fffa5910938 name=kata-shim pid=1 source=shim time="2020-03-31T21:58:15.434965064Z" level=info msg="copy stdout failed" container=e6234163dd7c1c1dab6e289bdccb6c2cf2cfc3629631d9629037d0bdf6ced084 error="rpc error: code = Unknown desc = read /dev/ptmx: input/output error" exec-id=e6234163dd7c1c1dab6e289bdccb6c2cf2cfc3629631d9629037d0bdf6ced084 name=kata-shim pid=1 source=shim time="2020-03-31T21:58:15.452577453Z" level=info msg="copy stdout failed" container=3a96c7aff6b455df8a620a8bc7fd5cef1c3eb616e035240066034fac6b56bbf2 error="rpc error: code = Unknown desc = read /dev/ptmx: input/output error" exec-id=3a96c7aff6b455df8a620a8bc7fd5cef1c3eb616e035240066034fac6b56bbf2 name=kata-shim pid=1 source=shim time="2020-03-31T21:58:15.465679219Z" level=info msg="copy stdout failed" container=49fbe4eab4ddfeab51401a8b19c523bd51aee4adbe6d71be87f809fa9cc08393 error="rpc error: code = Unknown desc = read /dev/ptmx: input/output error" exec-id=49fbe4eab4ddfeab51401a8b19c523bd51aee4adbe6d71be87f809fa9cc08393 name=kata-shim pid=1 source=shim time="2020-03-31T21:58:15.473383658Z" level=info msg="copy stdout failed" container=04ecde1e1cef6d99bf3e14bdf6b2db8e874a53ea7288f03ea3b70a922178b7d8 error="rpc error: code = Unknown desc = read /dev/ptmx: input/output error" exec-id=04ecde1e1cef6d99bf3e14bdf6b2db8e874a53ea7288f03ea3b70a922178b7d8 name=kata-shim pid=1 source=shim time="2020-03-31T21:58:15.477885281Z" level=info msg="copy stdout failed" container=77a839122a53527d92ced6d8d9f11dc3594299ffe6a9e915874b0dfd21c574aa error="rpc error: code = Unknown desc = read /dev/ptmx: input/output error" exec-id=77a839122a53527d92ced6d8d9f11dc3594299ffe6a9e915874b0dfd21c574aa name=kata-shim pid=1 source=shim time="2020-03-31T21:58:15.480717395Z" level=info msg="copy stdout failed" container=dba768c2b3843dea2e8577826722913beb26448e287772b13a416aa811446d0f error="rpc error: code = Unknown desc = read /dev/ptmx: input/output error" exec-id=dba768c2b3843dea2e8577826722913beb26448e287772b13a416aa811446d0f name=kata-shim pid=1 source=shim time="2020-03-31T21:58:15.48551592Z" level=info msg="copy stdout failed" container=e3e133f612a35132cb0f3e6b06fd4cc8502b937eb097d73d94e91ab30705d1a3 error="rpc error: code = Unknown desc = read /dev/ptmx: input/output error" exec-id=e3e133f612a35132cb0f3e6b06fd4cc8502b937eb097d73d94e91ab30705d1a3 name=kata-shim pid=1 source=shim time="2020-03-31T21:58:15.487431229Z" level=info msg="copy stdout failed" container=49b22853c06e14d446618bc20f1dafda66302f18211a821a728a0c0dcd4342b3 error="rpc error: code = Unknown desc = read /dev/ptmx: input/output error" exec-id=49b22853c06e14d446618bc20f1dafda66302f18211a821a728a0c0dcd4342b3 name=kata-shim pid=1 source=shim time="2020-03-31T21:58:15.494858467Z" level=info msg="copy stdout failed" container=f6b4fe572a1da34cf6675859c732d986c8cd79ed46912cff6ebe66d218176c39 error="rpc error: code = Unknown desc = read /dev/ptmx: input/output error" exec-id=f6b4fe572a1da34cf6675859c732d986c8cd79ed46912cff6ebe66d218176c39 name=kata-shim pid=1 source=shim time="2020-03-31T21:58:15.499706591Z" level=info msg="copy stdout failed" container=f08656852058e81893e6ef889b6fe9a8659788f7330cb055bd4bab65dac85009 error="rpc error: code = Unknown desc = read /dev/ptmx: input/output error" exec-id=f08656852058e81893e6ef889b6fe9a8659788f7330cb055bd4bab65dac85009 name=kata-shim pid=1 source=shim time="2020-03-31T21:58:15.566936932Z" level=info msg="copy stdout failed" container=622c16759e80153844f040f30c4f2e4de8d2ff49325676794301c623495de432 error="rpc error: code = Unknown desc = read /dev/ptmx: input/output error" exec-id=622c16759e80153844f040f30c4f2e4de8d2ff49325676794301c623495de432 name=kata-shim pid=1 source=shim time="2020-03-31T21:58:15.593210965Z" level=info msg="copy stdout failed" container=afb2f6f6ed0cca820627cd58a9eb76e53fefc166d164016d89f278beeca7c8a4 error="rpc error: code = Unknown desc = read /dev/ptmx: input/output error" exec-id=afb2f6f6ed0cca820627cd58a9eb76e53fefc166d164016d89f278beeca7c8a4 name=kata-shim pid=1 source=shim time="2020-03-31T21:58:15.691283261Z" level=info msg="copy stdout failed" container=3c2681d706f3fc7d76fc57502d8a474a5510af5b1074b1cc2267b298ecdf763d error="rpc error: code = Unknown desc = read /dev/ptmx: input/output error" exec-id=3c2681d706f3fc7d76fc57502d8a474a5510af5b1074b1cc2267b298ecdf763d name=kata-shim pid=1 source=shim time="2020-03-31T21:58:15.706743939Z" level=info msg="copy stdout failed" container=622c3620cc693a93f8d67feea49667c60ae37c524740de8ef41d916a53c300d9 error="rpc error: code = Unknown desc = read /dev/ptmx: input/output error" exec-id=622c3620cc693a93f8d67feea49667c60ae37c524740de8ef41d916a53c300d9 name=kata-shim pid=1 source=shim time="2020-03-31T21:58:15.733395274Z" level=info msg="copy stdout failed" container=e6aa5916bb985a0985679908db17aed640b301b20771efd270c7dc484454ef58 error="rpc error: code = Unknown desc = read /dev/ptmx: input/output error" exec-id=e6aa5916bb985a0985679908db17aed640b301b20771efd270c7dc484454ef58 name=kata-shim pid=1 source=shim time="2020-03-31T21:58:15.777312396Z" level=info msg="copy stdout failed" container=0563d3f1207bdffe383d47ad7bfc0cdf87cb5a5f013f8060ad8c7d5b45a61af8 error="rpc error: code = Unknown desc = read /dev/ptmx: input/output error" exec-id=0563d3f1207bdffe383d47ad7bfc0cdf87cb5a5f013f8060ad8c7d5b45a61af8 name=kata-shim pid=1 source=shim time="2020-03-31T21:58:26.575182732Z" level=info msg="copy stderr failed" container=3e7abbd5d02230dc4830f0e4e914545c03c10e3762cd3518b6f979ce8fa5745e error="rpc error: code = Unknown desc = EOF" exec-id=3e7abbd5d02230dc4830f0e4e914545c03c10e3762cd3518b6f979ce8fa5745e name=kata-shim pid=1 source=shim time="2020-03-31T21:58:26.575896235Z" level=info msg="copy stdout failed" container=3e7abbd5d02230dc4830f0e4e914545c03c10e3762cd3518b6f979ce8fa5745e error="rpc error: code = Unknown desc = EOF" exec-id=3e7abbd5d02230dc4830f0e4e914545c03c10e3762cd3518b6f979ce8fa5745e name=kata-shim pid=1 source=shim time="2020-03-31T21:58:30.479320286Z" level=info msg="copy stderr failed" container=7a2aa54186624e16bfe9626c4d1f606bf14efb6b6cd7616de3a9862778592d75 error="rpc error: code = Unavailable desc = transport is closing" exec-id=7a2aa54186624e16bfe9626c4d1f606bf14efb6b6cd7616de3a9862778592d75 name=kata-shim pid=1 source=shim time="2020-03-31T21:58:30.479365786Z" level=info msg="copy stdout failed" container=7a2aa54186624e16bfe9626c4d1f606bf14efb6b6cd7616de3a9862778592d75 error="rpc error: code = Unavailable desc = transport is closing" exec-id=7a2aa54186624e16bfe9626c4d1f606bf14efb6b6cd7616de3a9862778592d75 name=kata-shim pid=1 source=shim time="2020-03-31T21:58:30.480848394Z" level=error msg="failed waiting for process" container=7a2aa54186624e16bfe9626c4d1f606bf14efb6b6cd7616de3a9862778592d75 error="rpc error: code = Unavailable desc = transport is closing" exec-id=7a2aa54186624e16bfe9626c4d1f606bf14efb6b6cd7616de3a9862778592d75 name=kata-shim pid=1 source=shim time="2020-03-31T21:58:55.086243893Z" level=info msg="copy stderr failed" container=23ba2d5aa58e8802b9587b5e4e0ea1d920be6fd5e944d256841fc95a9b5277f5 error="rpc error: code = Unknown desc = EOF" exec-id=23ba2d5aa58e8802b9587b5e4e0ea1d920be6fd5e944d256841fc95a9b5277f5 name=kata-shim pid=1 source=shim time="2020-03-31T21:58:55.086579495Z" level=info msg="copy stdout failed" container=23ba2d5aa58e8802b9587b5e4e0ea1d920be6fd5e944d256841fc95a9b5277f5 error="rpc error: code = Unknown desc = EOF" exec-id=23ba2d5aa58e8802b9587b5e4e0ea1d920be6fd5e944d256841fc95a9b5277f5 name=kata-shim pid=1 source=shim time="2020-03-31T21:59:08.158584613Z" level=info msg="copy stdout failed" container=2b08f7cbb58517f2a7e1005275a25edf271ed4755dc7fb2bb8eae0210e89686b error="rpc error: code = Unknown desc = EOF" exec-id=2b08f7cbb58517f2a7e1005275a25edf271ed4755dc7fb2bb8eae0210e89686b name=kata-shim pid=1 source=shim time="2020-03-31T21:59:08.158688214Z" level=info msg="copy stderr failed" container=2b08f7cbb58517f2a7e1005275a25edf271ed4755dc7fb2bb8eae0210e89686b error="rpc error: code = Unknown desc = EOF" exec-id=2b08f7cbb58517f2a7e1005275a25edf271ed4755dc7fb2bb8eae0210e89686b name=kata-shim pid=1 source=shim ``` ## Throttler logs No recent throttler problems found in system journal. --- # Container manager details Have `docker` ## Docker Output of "`docker version`": ``` Client: Version: 18.06.3-ce API version: 1.38 Go version: go1.10.3 Git commit: d7080c1 Built: Wed Feb 20 02:27:18 2019 OS/Arch: linux/amd64 Experimental: false Server: Engine: Version: 18.06.3-ce API version: 1.38 (minimum version 1.12) Go version: go1.10.3 Git commit: d7080c1 Built: Wed Feb 20 02:28:44 2019 OS/Arch: linux/amd64 Experimental: false ``` Output of "`docker info`": ``` Containers: 3 Running: 0 Paused: 0 Stopped: 3 Images: 6 Server Version: 18.06.3-ce Storage Driver: overlay2 Backing Filesystem: extfs Supports d_type: true Native Overlay Diff: true Logging Driver: json-file Cgroup Driver: cgroupfs Plugins: Volume: local Network: bridge host macvlan null overlay Log: awslogs fluentd gcplogs gelf journald json-file logentries splunk syslog Swarm: inactive Runtimes: kata-runtime runc Default Runtime: runc Init Binary: docker-init containerd version: 468a545b9edcd5932818eb9de8e72413e616e86e runc version: a592beb5bc4c4092b1b1bac971afed27687340c5 init version: fec3683 Security Options: seccomp Profile: default Kernel Version: 5.3.7-301.fc31.x86_64 Operating System: Fedora 31 (Cloud Edition) OSType: linux Architecture: x86_64 CPUs: 4 Total Memory: 15.61GiB Name: fedora31-92ba70 ID: FG5D:F4V2:UU6J:3SFG:MCNA:X5OC:YA7S:QWJK:UC3U:LTOW:5NLM:YKQZ Docker Root Dir: /var/lib/docker Debug Mode (client): false Debug Mode (server): true File Descriptors: 24 Goroutines: 44 System Time: 2020-03-31T22:07:05.118742859Z EventsListeners: 0 Registry: https://index.docker.io/v1/ Labels: Experimental: false Insecure Registries: 127.0.0.0/8 Live Restore Enabled: false ``` Output of "`systemctl show docker`": ``` Type=notify Restart=on-failure NotifyAccess=main RestartUSec=100ms TimeoutStartUSec=infinity TimeoutStopUSec=1min 30s TimeoutAbortUSec=1min 30s RuntimeMaxUSec=infinity WatchdogUSec=0 WatchdogTimestampMonotonic=0 RootDirectoryStartOnly=no RemainAfterExit=no GuessMainPID=yes MainPID=111330 ControlPID=0 FileDescriptorStoreMax=0 NFileDescriptorStore=0 StatusErrno=0 Result=success ReloadResult=success CleanResult=success UID=[not set] GID=[not set] NRestarts=0 OOMPolicy=continue ExecMainStartTimestamp=Tue 2020-03-31 22:00:04 UTC ExecMainStartTimestampMonotonic=1578821578 ExecMainExitTimestampMonotonic=0 ExecMainPID=111330 ExecMainCode=0 ExecMainStatus=0 ExecStart={ path=/usr/bin/dockerd ; argv[]=/usr/bin/dockerd -D --add-runtime kata-runtime=/usr/local/bin/kata-runtime --default-runtime=runc --storage-driver=overlay2 ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 } ExecStartEx={ path=/usr/bin/dockerd ; argv[]=/usr/bin/dockerd -D --add-runtime kata-runtime=/usr/local/bin/kata-runtime --default-runtime=runc --storage-driver=overlay2 ; flags= ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 } ExecReload={ path=/bin/kill ; argv[]=/bin/kill -s HUP $MAINPID ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 } Slice=system.slice ControlGroup=/system.slice/docker.service MemoryCurrent=374382592 CPUUsageNSec=[not set] TasksCurrent=34 IPIngressBytes=[no data] IPIngressPackets=[no data] IPEgressBytes=[no data] IPEgressPackets=[no data] IOReadBytes=18446744073709551615 IOReadOperations=18446744073709551615 IOWriteBytes=18446744073709551615 IOWriteOperations=18446744073709551615 Delegate=yes DelegateControllers=cpu cpuacct io blkio memory devices pids bpf-firewall bpf-devices CPUAccounting=no CPUWeight=[not set] StartupCPUWeight=[not set] CPUShares=[not set] StartupCPUShares=[not set] CPUQuotaPerSecUSec=infinity CPUQuotaPeriodUSec=infinity IOAccounting=no IOWeight=[not set] StartupIOWeight=[not set] BlockIOAccounting=no BlockIOWeight=[not set] StartupBlockIOWeight=[not set] MemoryAccounting=yes DefaultMemoryLow=0 DefaultMemoryMin=0 MemoryMin=0 MemoryLow=0 MemoryHigh=infinity MemoryMax=infinity MemorySwapMax=infinity MemoryLimit=infinity DevicePolicy=auto TasksAccounting=yes TasksMax=infinity IPAccounting=no UMask=0022 LimitCPU=infinity LimitCPUSoft=infinity LimitFSIZE=infinity LimitFSIZESoft=infinity LimitDATA=infinity LimitDATASoft=infinity LimitSTACK=infinity LimitSTACKSoft=8388608 LimitCORE=infinity LimitCORESoft=infinity LimitRSS=infinity LimitRSSSoft=infinity LimitNOFILE=infinity LimitNOFILESoft=infinity LimitAS=infinity LimitASSoft=infinity LimitNPROC=infinity LimitNPROCSoft=infinity LimitMEMLOCK=65536 LimitMEMLOCKSoft=65536 LimitLOCKS=infinity LimitLOCKSSoft=infinity LimitSIGPENDING=63817 LimitSIGPENDINGSoft=63817 LimitMSGQUEUE=819200 LimitMSGQUEUESoft=819200 LimitNICE=0 LimitNICESoft=0 LimitRTPRIO=0 LimitRTPRIOSoft=0 LimitRTTIME=infinity LimitRTTIMESoft=infinity OOMScoreAdjust=0 Nice=0 IOSchedulingClass=0 IOSchedulingPriority=0 CPUSchedulingPolicy=0 CPUSchedulingPriority=0 CPUAffinity= NUMAPolicy=n/a NUMAMask= TimerSlackNSec=50000 CPUSchedulingResetOnFork=no NonBlocking=no StandardInput=null StandardInputData= StandardOutput=journal StandardError=inherit TTYReset=no TTYVHangup=no TTYVTDisallocate=no SyslogPriority=30 SyslogLevelPrefix=yes SyslogLevel=6 SyslogFacility=3 LogLevelMax=-1 LogRateLimitIntervalUSec=0 LogRateLimitBurst=0 SecureBits=0 CapabilityBoundingSet=cap_chown cap_dac_override cap_dac_read_search cap_fowner cap_fsetid cap_kill cap_setgid cap_setuid cap_setpcap cap_linux_immutable cap_net_bind_service cap_net_broadcast cap_net_admin cap_net_raw cap_ipc_lock cap_ipc_owner cap_sys_module cap_sys_rawio cap_sys_chroot cap_sys_ptrace cap_sys_pacct cap_sys_admin cap_sys_boot cap_sys_nice cap_sys_resource cap_sys_time cap_sys_tty_config cap_mknod cap_lease cap_audit_write cap_audit_control cap_setfcap cap_mac_override cap_mac_admin cap_syslog cap_wake_alarm cap_block_suspend cap_audit_read AmbientCapabilities= DynamicUser=no RemoveIPC=no MountFlags= PrivateTmp=no PrivateDevices=no ProtectKernelTunables=no ProtectKernelModules=no ProtectControlGroups=no PrivateNetwork=no PrivateUsers=no PrivateMounts=no ProtectHome=no ProtectSystem=no SameProcessGroup=no UtmpMode=init IgnoreSIGPIPE=yes NoNewPrivileges=no SystemCallErrorNumber=0 LockPersonality=no RuntimeDirectoryPreserve=no RuntimeDirectoryMode=0755 StateDirectoryMode=0755 CacheDirectoryMode=0755 LogsDirectoryMode=0755 ConfigurationDirectoryMode=0755 TimeoutCleanUSec=infinity MemoryDenyWriteExecute=no RestrictRealtime=no RestrictSUIDSGID=no RestrictNamespaces=no MountAPIVFS=no KeyringMode=private ProtectHostname=no KillMode=process KillSignal=15 FinalKillSignal=9 SendSIGKILL=yes SendSIGHUP=no WatchdogSignal=6 Id=docker.service Names=docker.service Requires=system.slice sysinit.target Wants=network-online.target Conflicts=shutdown.target Before=shutdown.target After=firewalld.service sysinit.target basic.target systemd-journald.socket system.slice network-online.target Documentation=https://docs.docker.com Description=Docker Application Container Engine LoadState=loaded ActiveState=active SubState=running FragmentPath=/usr/lib/systemd/system/docker.service DropInPaths=/etc/systemd/system/docker.service.d/kata-containers.conf UnitFileState=disabled UnitFilePreset=disabled StateChangeTimestamp=Tue 2020-03-31 22:00:06 UTC StateChangeTimestampMonotonic=1580496951 InactiveExitTimestamp=Tue 2020-03-31 22:00:04 UTC InactiveExitTimestampMonotonic=1578822028 ActiveEnterTimestamp=Tue 2020-03-31 22:00:06 UTC ActiveEnterTimestampMonotonic=1580496951 ActiveExitTimestampMonotonic=0 InactiveEnterTimestampMonotonic=0 CanStart=yes CanStop=yes CanReload=yes CanIsolate=no StopWhenUnneeded=no RefuseManualStart=no RefuseManualStop=no AllowIsolate=no DefaultDependencies=yes OnFailureJobMode=replace IgnoreOnIsolate=no NeedDaemonReload=no JobTimeoutUSec=infinity JobRunningTimeoutUSec=infinity JobTimeoutAction=none ConditionResult=yes AssertResult=yes ConditionTimestamp=Tue 2020-03-31 22:00:04 UTC ConditionTimestampMonotonic=1578819213 AssertTimestamp=Tue 2020-03-31 22:00:04 UTC AssertTimestampMonotonic=1578819213 Transient=no Perpetual=no StartLimitIntervalUSec=1min StartLimitBurst=3 StartLimitAction=none FailureAction=none SuccessAction=none InvocationID=0f78eed4e2664bf48788f864acbd0c65 CollectMode=inactive ``` No `kubectl` Have `crio` ## crio Output of "`crio --version`": ``` crio version 1.10.7-dev commit: "6273bea4c9ed788aeb3d051ebf2d030060c05b6c-dirty" ``` Output of "`systemctl show crio`": ``` Type=simple Restart=on-failure NotifyAccess=none RestartUSec=5s TimeoutStartUSec=1min 30s TimeoutStopUSec=1min 30s TimeoutAbortUSec=1min 30s RuntimeMaxUSec=infinity WatchdogUSec=0 WatchdogTimestampMonotonic=0 RootDirectoryStartOnly=no RemainAfterExit=no GuessMainPID=yes MainPID=0 ControlPID=0 FileDescriptorStoreMax=0 NFileDescriptorStore=0 StatusErrno=0 Result=success ReloadResult=success CleanResult=success UID=[not set] GID=[not set] NRestarts=0 OOMPolicy=stop ExecMainStartTimestampMonotonic=0 ExecMainExitTimestampMonotonic=0 ExecMainPID=0 ExecMainCode=0 ExecMainStatus=0 ExecStart={ path=/usr/local/bin/crio ; argv[]=/usr/local/bin/crio --log-level debug ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 } ExecStartEx={ path=/usr/local/bin/crio ; argv[]=/usr/local/bin/crio --log-level debug ; flags= ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 } Slice=system.slice MemoryCurrent=[not set] CPUUsageNSec=[not set] TasksCurrent=[not set] IPIngressBytes=[no data] IPIngressPackets=[no data] IPEgressBytes=[no data] IPEgressPackets=[no data] IOReadBytes=18446744073709551615 IOReadOperations=18446744073709551615 IOWriteBytes=18446744073709551615 IOWriteOperations=18446744073709551615 Delegate=no CPUAccounting=no CPUWeight=[not set] StartupCPUWeight=[not set] CPUShares=[not set] StartupCPUShares=[not set] CPUQuotaPerSecUSec=infinity CPUQuotaPeriodUSec=infinity IOAccounting=no IOWeight=[not set] StartupIOWeight=[not set] BlockIOAccounting=no BlockIOWeight=[not set] StartupBlockIOWeight=[not set] MemoryAccounting=yes DefaultMemoryLow=0 DefaultMemoryMin=0 MemoryMin=0 MemoryLow=0 MemoryHigh=infinity MemoryMax=infinity MemorySwapMax=infinity MemoryLimit=infinity DevicePolicy=auto TasksAccounting=yes TasksMax=19145 IPAccounting=no UMask=0022 LimitCPU=infinity LimitCPUSoft=infinity LimitFSIZE=infinity LimitFSIZESoft=infinity LimitDATA=infinity LimitDATASoft=infinity LimitSTACK=infinity LimitSTACKSoft=8388608 LimitCORE=infinity LimitCORESoft=infinity LimitRSS=infinity LimitRSSSoft=infinity LimitNOFILE=524288 LimitNOFILESoft=1024 LimitAS=infinity LimitASSoft=infinity LimitNPROC=63817 LimitNPROCSoft=63817 LimitMEMLOCK=65536 LimitMEMLOCKSoft=65536 LimitLOCKS=infinity LimitLOCKSSoft=infinity LimitSIGPENDING=63817 LimitSIGPENDINGSoft=63817 LimitMSGQUEUE=819200 LimitMSGQUEUESoft=819200 LimitNICE=0 LimitNICESoft=0 LimitRTPRIO=0 LimitRTPRIOSoft=0 LimitRTTIME=infinity LimitRTTIMESoft=infinity OOMScoreAdjust=0 Nice=0 IOSchedulingClass=0 IOSchedulingPriority=0 CPUSchedulingPolicy=0 CPUSchedulingPriority=0 CPUAffinity= NUMAPolicy=n/a NUMAMask= TimerSlackNSec=50000 CPUSchedulingResetOnFork=no NonBlocking=no StandardInput=null StandardInputData= StandardOutput=journal StandardError=inherit TTYReset=no TTYVHangup=no TTYVTDisallocate=no SyslogPriority=30 SyslogLevelPrefix=yes SyslogLevel=6 SyslogFacility=3 LogLevelMax=-1 LogRateLimitIntervalUSec=0 LogRateLimitBurst=0 SecureBits=0 CapabilityBoundingSet=cap_chown cap_dac_override cap_dac_read_search cap_fowner cap_fsetid cap_kill cap_setgid cap_setuid cap_setpcap cap_linux_immutable cap_net_bind_service cap_net_broadcast cap_net_admin cap_net_raw cap_ipc_lock cap_ipc_owner cap_sys_module cap_sys_rawio cap_sys_chroot cap_sys_ptrace cap_sys_pacct cap_sys_admin cap_sys_boot cap_sys_nice cap_sys_resource cap_sys_time cap_sys_tty_config cap_mknod cap_lease cap_audit_write cap_audit_control cap_setfcap cap_mac_override cap_mac_admin cap_syslog cap_wake_alarm cap_block_suspend cap_audit_read AmbientCapabilities= DynamicUser=no RemoveIPC=no MountFlags= PrivateTmp=no PrivateDevices=no ProtectKernelTunables=no ProtectKernelModules=no ProtectControlGroups=no PrivateNetwork=no PrivateUsers=no PrivateMounts=no ProtectHome=no ProtectSystem=no SameProcessGroup=no UtmpMode=init IgnoreSIGPIPE=yes NoNewPrivileges=no SystemCallErrorNumber=0 LockPersonality=no RuntimeDirectoryPreserve=no RuntimeDirectoryMode=0755 StateDirectoryMode=0755 CacheDirectoryMode=0755 LogsDirectoryMode=0755 ConfigurationDirectoryMode=0755 TimeoutCleanUSec=infinity MemoryDenyWriteExecute=no RestrictRealtime=no RestrictSUIDSGID=no RestrictNamespaces=no MountAPIVFS=no KeyringMode=private ProtectHostname=no KillMode=control-group KillSignal=15 FinalKillSignal=9 SendSIGKILL=yes SendSIGHUP=no WatchdogSignal=6 Id=crio.service Names=crio.service Requires=system.slice sysinit.target Conflicts=shutdown.target Before=shutdown.target After=systemd-journald.socket system.slice basic.target sysinit.target Documentation=https://github.com/cri-o/cri-o Description=CRI-O daemon LoadState=loaded ActiveState=inactive SubState=dead FragmentPath=/etc/systemd/system/crio.service UnitFileState=disabled UnitFilePreset=disabled StateChangeTimestampMonotonic=0 InactiveExitTimestampMonotonic=0 ActiveEnterTimestampMonotonic=0 ActiveExitTimestampMonotonic=0 InactiveEnterTimestampMonotonic=0 CanStart=yes CanStop=yes CanReload=no CanIsolate=no StopWhenUnneeded=no RefuseManualStart=no RefuseManualStop=no AllowIsolate=no DefaultDependencies=yes OnFailureJobMode=replace IgnoreOnIsolate=no NeedDaemonReload=no JobTimeoutUSec=infinity JobRunningTimeoutUSec=infinity JobTimeoutAction=none ConditionResult=no AssertResult=no ConditionTimestampMonotonic=0 AssertTimestampMonotonic=0 Transient=no Perpetual=no StartLimitIntervalUSec=10s StartLimitBurst=5 StartLimitAction=none FailureAction=none SuccessAction=none CollectMode=inactive ``` Output of "`cat /etc/crio/crio.conf`": ``` # The "crio" table contains all of the server options. [crio] # root is a path to the "root directory". CRIO stores all of its data, # including container images, in this directory. root = "/var/lib/containers/storage" # run is a path to the "run directory". CRIO stores all of its state # in this directory. runroot = "/var/run/containers/storage" # storage_driver select which storage driver is used to manage storage # of images and containers. storage_driver = "" # storage_option is used to pass an option to the storage driver. storage_option = [ ] # The "crio.api" table contains settings for the kubelet/gRPC interface. [crio.api] # listen is the path to the AF_LOCAL socket on which crio will listen. listen = "/var/run/crio/crio.sock" # stream_address is the IP address on which the stream server will listen stream_address = "" # stream_port is the port on which the stream server will listen stream_port = "10010" # file_locking is whether file-based locking will be used instead of # in-memory locking file_locking = true # The "crio.runtime" table contains settings pertaining to the OCI # runtime used and options for how to set up and manage the OCI runtime. [crio.runtime] manage_network_ns_lifecycle = true # runtime is the OCI compatible runtime used for trusted container workloads. # This is a mandatory setting as this runtime will be the default one # and will also be used for untrusted container workloads if # runtime_untrusted_workload is not set. runtime = "/usr/local/bin/crio-runc" # runtime_untrusted_workload is the OCI compatible runtime used for untrusted # container workloads. This is an optional setting, except if # default_container_trust is set to "untrusted". runtime_untrusted_workload = "/usr/local/bin/kata-runtime" # default_workload_trust is the default level of trust crio puts in container # workloads. It can either be "trusted" or "untrusted", and the default # is "trusted". # Containers can be run through different container runtimes, depending on # the trust hints we receive from kubelet: # - If kubelet tags a container workload as untrusted, crio will try first to # run it through the untrusted container workload runtime. If it is not set, # crio will use the trusted runtime. # - If kubelet does not provide any information about the container workload trust # level, the selected runtime will depend on the default_container_trust setting. # If it is set to "untrusted", then all containers except for the host privileged # ones, will be run by the runtime_untrusted_workload runtime. Host privileged # containers are by definition trusted and will always use the trusted container # runtime. If default_container_trust is set to "trusted", crio will use the trusted # container runtime for all containers. default_workload_trust = "trusted" # no_pivot instructs the runtime to not use pivot_root, but instead use MS_MOVE no_pivot = false # conmon is the path to conmon binary, used for managing the runtime. conmon = "/usr/local/libexec/crio/conmon" # conmon_env is the environment variable list for conmon process, # used for passing necessary environment variable to conmon or runtime. conmon_env = [ "PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin", ] # selinux indicates whether or not SELinux will be used for pod # separation on the host. If you enable this flag, SELinux must be running # on the host. selinux = true # seccomp_profile is the seccomp json profile path which is used as the # default for the runtime. seccomp_profile = "/etc/crio/seccomp.json" # apparmor_profile is the apparmor profile name which is used as the # default for the runtime. apparmor_profile = "crio-default" # cgroup_manager is the cgroup management implementation to be used # for the runtime. cgroup_manager = "cgroupfs" # hooks_dir_path is the oci hooks directory for automatically executed hooks hooks_dir_path = "/usr/share/containers/oci/hooks.d" # default_mounts is the mounts list to be mounted for the container when created default_mounts = [ ] # pids_limit is the number of processes allowed in a container pids_limit = 1024 # log_size_max is the max limit for the container log size in bytes. # Negative values indicate that no limit is imposed. log_size_max = -1 # The "crio.image" table contains settings pertaining to the # management of OCI images. [crio.image] # default_transport is the prefix we try prepending to an image name if the # image name as we receive it can't be parsed as a valid source reference default_transport = "docker://" # pause_image is the image which we use to instantiate infra containers. pause_image = "k8s.gcr.io/pause" # pause_command is the command to run in a pause_image to have a container just # sit there. If the image contains the necessary information, this value need # not be specified. pause_command = "/pause" # signature_policy is the name of the file which decides what sort of policy we # use when deciding whether or not to trust an image that we've pulled. # Outside of testing situations, it is strongly advised that this be left # unspecified so that the default system-wide policy will be used. signature_policy = "" # image_volumes controls how image volumes are handled. # The valid values are mkdir and ignore. image_volumes = "mkdir" # insecure_registries is used to skip TLS verification when pulling images. insecure_registries = [ ] # registries is used to specify a comma separated list of registries to be used # when pulling an unqualified image (e.g. fedora:rawhide). registries = [ "docker.io" ] # The "crio.network" table contains settings pertaining to the # management of CNI plugins. [crio.network] # network_dir is is where CNI network configuration # files are stored. network_dir = "/etc/cni/net.d/" # plugin_dir is is where CNI plugin binaries are stored. plugin_dir = "/opt/cni/bin/" ``` Have `containerd` ## containerd Output of "`containerd --version`": ``` containerd github.com/containerd/containerd v1.3.2 ff48f57fc83a8c44cf4ad5d672424a98ba37ded6 ``` Output of "`systemctl show containerd`": ``` Type=simple Restart=always NotifyAccess=none RestartUSec=5s TimeoutStartUSec=1min 30s TimeoutStopUSec=1min 30s TimeoutAbortUSec=1min 30s RuntimeMaxUSec=infinity WatchdogUSec=0 WatchdogTimestampMonotonic=0 RootDirectoryStartOnly=no RemainAfterExit=no GuessMainPID=yes MainPID=0 ControlPID=0 FileDescriptorStoreMax=0 NFileDescriptorStore=0 StatusErrno=0 Result=success ReloadResult=success CleanResult=success UID=[not set] GID=[not set] NRestarts=0 OOMPolicy=continue ExecMainStartTimestampMonotonic=0 ExecMainExitTimestampMonotonic=0 ExecMainPID=0 ExecMainCode=0 ExecMainStatus=0 ExecStartPre={ path=/sbin/modprobe ; argv[]=/sbin/modprobe overlay ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 } ExecStartPreEx={ path=/sbin/modprobe ; argv[]=/sbin/modprobe overlay ; flags= ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 } ExecStart={ path=/usr/local/bin/containerd ; argv[]=/usr/local/bin/containerd ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 } ExecStartEx={ path=/usr/local/bin/containerd ; argv[]=/usr/local/bin/containerd ; flags= ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 } Slice=system.slice MemoryCurrent=[not set] CPUUsageNSec=[not set] TasksCurrent=[not set] IPIngressBytes=[no data] IPIngressPackets=[no data] IPEgressBytes=[no data] IPEgressPackets=[no data] IOReadBytes=18446744073709551615 IOReadOperations=18446744073709551615 IOWriteBytes=18446744073709551615 IOWriteOperations=18446744073709551615 Delegate=yes DelegateControllers=cpu cpuacct io blkio memory devices pids bpf-firewall bpf-devices CPUAccounting=no CPUWeight=[not set] StartupCPUWeight=[not set] CPUShares=[not set] StartupCPUShares=[not set] CPUQuotaPerSecUSec=infinity CPUQuotaPeriodUSec=infinity IOAccounting=no IOWeight=[not set] StartupIOWeight=[not set] BlockIOAccounting=no BlockIOWeight=[not set] StartupBlockIOWeight=[not set] MemoryAccounting=yes DefaultMemoryLow=0 DefaultMemoryMin=0 MemoryMin=0 MemoryLow=0 MemoryHigh=infinity MemoryMax=infinity MemorySwapMax=infinity MemoryLimit=infinity DevicePolicy=auto TasksAccounting=yes TasksMax=19145 IPAccounting=no UMask=0022 LimitCPU=infinity LimitCPUSoft=infinity LimitFSIZE=infinity LimitFSIZESoft=infinity LimitDATA=infinity LimitDATASoft=infinity LimitSTACK=infinity LimitSTACKSoft=8388608 LimitCORE=infinity LimitCORESoft=infinity LimitRSS=infinity LimitRSSSoft=infinity LimitNOFILE=1048576 LimitNOFILESoft=1048576 LimitAS=infinity LimitASSoft=infinity LimitNPROC=infinity LimitNPROCSoft=infinity LimitMEMLOCK=65536 LimitMEMLOCKSoft=65536 LimitLOCKS=infinity LimitLOCKSSoft=infinity LimitSIGPENDING=63817 LimitSIGPENDINGSoft=63817 LimitMSGQUEUE=819200 LimitMSGQUEUESoft=819200 LimitNICE=0 LimitNICESoft=0 LimitRTPRIO=0 LimitRTPRIOSoft=0 LimitRTTIME=infinity LimitRTTIMESoft=infinity OOMScoreAdjust=-999 Nice=0 IOSchedulingClass=0 IOSchedulingPriority=0 CPUSchedulingPolicy=0 CPUSchedulingPriority=0 CPUAffinity= NUMAPolicy=n/a NUMAMask= TimerSlackNSec=50000 CPUSchedulingResetOnFork=no NonBlocking=no StandardInput=null StandardInputData= StandardOutput=journal StandardError=inherit TTYReset=no TTYVHangup=no TTYVTDisallocate=no SyslogPriority=30 SyslogLevelPrefix=yes SyslogLevel=6 SyslogFacility=3 LogLevelMax=-1 LogRateLimitIntervalUSec=0 LogRateLimitBurst=0 SecureBits=0 CapabilityBoundingSet=cap_chown cap_dac_override cap_dac_read_search cap_fowner cap_fsetid cap_kill cap_setgid cap_setuid cap_setpcap cap_linux_immutable cap_net_bind_service cap_net_broadcast cap_net_admin cap_net_raw cap_ipc_lock cap_ipc_owner cap_sys_module cap_sys_rawio cap_sys_chroot cap_sys_ptrace cap_sys_pacct cap_sys_admin cap_sys_boot cap_sys_nice cap_sys_resource cap_sys_time cap_sys_tty_config cap_mknod cap_lease cap_audit_write cap_audit_control cap_setfcap cap_mac_override cap_mac_admin cap_syslog cap_wake_alarm cap_block_suspend cap_audit_read AmbientCapabilities= DynamicUser=no RemoveIPC=no MountFlags= PrivateTmp=no PrivateDevices=no ProtectKernelTunables=no ProtectKernelModules=no ProtectControlGroups=no PrivateNetwork=no PrivateUsers=no PrivateMounts=no ProtectHome=no ProtectSystem=no SameProcessGroup=no UtmpMode=init IgnoreSIGPIPE=yes NoNewPrivileges=no SystemCallErrorNumber=0 LockPersonality=no RuntimeDirectoryPreserve=no RuntimeDirectoryMode=0755 StateDirectoryMode=0755 CacheDirectoryMode=0755 LogsDirectoryMode=0755 ConfigurationDirectoryMode=0755 TimeoutCleanUSec=infinity MemoryDenyWriteExecute=no RestrictRealtime=no RestrictSUIDSGID=no RestrictNamespaces=no MountAPIVFS=no KeyringMode=private ProtectHostname=no KillMode=process KillSignal=15 FinalKillSignal=9 SendSIGKILL=yes SendSIGHUP=no WatchdogSignal=6 Id=containerd.service Names=containerd.service Requires=system.slice sysinit.target Conflicts=shutdown.target Before=shutdown.target After=basic.target system.slice systemd-journald.socket network.target sysinit.target Documentation=https://containerd.io Description=containerd container runtime LoadState=loaded ActiveState=inactive SubState=dead FragmentPath=/etc/systemd/system/containerd.service UnitFileState=disabled UnitFilePreset=disabled StateChangeTimestampMonotonic=0 InactiveExitTimestampMonotonic=0 ActiveEnterTimestampMonotonic=0 ActiveExitTimestampMonotonic=0 InactiveEnterTimestampMonotonic=0 CanStart=yes CanStop=yes CanReload=no CanIsolate=no StopWhenUnneeded=no RefuseManualStart=no RefuseManualStop=no AllowIsolate=no DefaultDependencies=yes OnFailureJobMode=replace IgnoreOnIsolate=no NeedDaemonReload=no JobTimeoutUSec=infinity JobRunningTimeoutUSec=infinity JobTimeoutAction=none ConditionResult=no AssertResult=no ConditionTimestampMonotonic=0 AssertTimestampMonotonic=0 Transient=no Perpetual=no StartLimitIntervalUSec=10s StartLimitBurst=5 StartLimitAction=none FailureAction=none SuccessAction=none CollectMode=inactive ``` Output of "`cat /etc/containerd/config.toml`": ``` cat: /etc/containerd/config.toml: No such file or directory ``` --- # Packages No `dpkg` Have `rpm` Output of "`rpm -qa|egrep "(cc-oci-runtimecc-runtimerunv|kata-proxy|kata-runtime|kata-shim|kata-ksm-throttler|kata-containers-image|linux-container|qemu-)"`": ``` ``` ---

[GabyCT]

This is also not working on Centos 8