Show kata-collect-data.sh details

# Meta details Running `kata-collect-data.sh` version `1.12.0 (commit )` at `2020-12-10.22:28:59.643485993+0100`. --- Runtime is `/usr/bin/kata-runtime`. # `kata-env` Output of "`/usr/bin/kata-runtime kata-env`": ```toml [Meta] Version = "1.0.24" [Runtime] Debug = false Trace = false DisableGuestSeccomp = true DisableNewNetNs = false SandboxCgroupOnly = true Path = "/usr/bin/kata-runtime" [Runtime.Version] OCI = "1.0.1-dev" [Runtime.Version.Version] Semver = "1.12.0" Major = 1 Minor = 12 Patch = 0 Commit = "" [Runtime.Config] Path = "/usr/share/kata-containers/defaults/configuration.toml" [Hypervisor] MachineType = "q35" Version = "QEMU emulator version 5.1.0 (qemu-5.1.0-5.fc33)\nCopyright (c) 2003-2020 Fabrice Bellard and the QEMU Project developers" Path = "/usr/bin/qemu-kvm" BlockDeviceDriver = "virtio-scsi" EntropySource = "/dev/urandom" SharedFS = "virtio-fs" VirtioFSDaemon = "/usr/libexec/virtiofsd" Msize9p = 8192 MemorySlots = 10 PCIeRootPort = 0 HotplugVFIOOnRootBus = false Debug = false UseVSock = true [Image] Path = "" [Kernel] Path = "/usr/lib/modules/5.9.13-200.fc33.x86_64/vmlinuz" Parameters = "scsi_mod.scan=none systemd.unified_cgroup_hierarchy=0" [Initrd] Path = "/var/cache/kata-containers/osbuilder-images/5.9.13-200.fc33.x86_64/fedora-kata-5.9.13-200.fc33.x86_64.initrd" [Proxy] Type = "noProxy" Path = "" Debug = false [Proxy.Version] Semver = "" Major = 0 Minor = 0 Patch = 0 Commit = "" [Shim] Type = "kataShim" Path = "/usr/libexec/kata-containers/kata-shim" Debug = false [Shim.Version] Semver = "<>" Major = 0 Minor = 0 Patch = 0 Commit = "<>" [Agent] Type = "kata" Debug = false Trace = false TraceMode = "" TraceType = "" [Host] Kernel = "5.9.13-200.fc33.x86_64" Architecture = "amd64" VMContainerCapable = true SupportVSocks = true [Host.Distro] Name = "Fedora" Version = "33" [Host.CPU] Vendor = "GenuineIntel" Model = "Intel(R) Pentium(R) CPU G4560 @ 3.50GHz" [Netmon] Path = "/usr/libexec/kata-containers/kata-netmon" Debug = false Enable = false [Netmon.Version] Semver = "1.12.0" Major = 1 Minor = 12 Patch = 0 Commit = "<>" ``` --- # Runtime config files ## Runtime default config files ``` /etc/kata-containers/configuration.toml /usr/share/kata-containers/defaults/configuration.toml ``` ## Runtime config file contents Config file `/etc/kata-containers/configuration.toml` not found Config file `/usr/share/defaults/kata-containers/configuration.toml` not found Output of "`cat "/usr/share/kata-containers/defaults/configuration.toml"`": ```toml # Copyright (c) 2017-2019 Intel Corporation # # SPDX-License-Identifier: Apache-2.0 # # XXX: WARNING: this file is auto-generated. # XXX: # XXX: Source file: "cli/config/configuration-qemu.toml.in" # XXX: Project: # XXX: Name: Kata Containers # XXX: Type: kata [hypervisor.qemu] path = "/usr/bin/qemu-kvm" kernel = "/var/cache/kata-containers/vmlinuz.container" initrd = "/var/cache/kata-containers/kata-containers-initrd.img" #image = "/var/cache/kata-containers/kata-containers.img" machine_type = "q35" # List of valid annotation names for the hypervisor # Each member of the list is a regular expression, which is the base name # of the annotation, e.g. "path" for io.katacontainers.config.hypervisor.path" # The default if not set is empty (all annotations rejected.) # Your distribution recommends: [".*"] enable_annotations = [".*"] # List of valid annotation values for the hypervisor path # Each member of the list is a path pattern as described by glob(3). # The default if not set is empty (all annotations rejected.) # Your distribution recommends: ["/usr/bin/qemu-kvm"] valid_hypervisor_paths = ["/usr/bin/qemu-kvm"] # Optional space-separated list of options to pass to the guest kernel. # For example, use `kernel_params = "vsyscall=emulate"` if you are having # trouble running pre-2.15 glibc. # # WARNING: - any parameter specified here will take priority over the default # parameter value of the same name used to start the virtual machine. # Do not set values here unless you understand the impact of doing so as you # may stop the virtual machine from booting. # To see the list of default parameters, enable hypervisor debug, create a # container and look for 'default-kernel-parameters' log entries. kernel_params = "systemd.unified_cgroup_hierarchy=0" # Path to the firmware. # If you want that qemu uses the default firmware leave this option empty firmware = "" # Machine accelerators # comma-separated list of machine accelerators to pass to the hypervisor. # For example, `machine_accelerators = "nosmm,nosmbus,nosata,nopit,static-prt,nofw"` machine_accelerators="" # CPU features # comma-separated list of cpu features to pass to the cpu # For example, `cpu_features = "pmu=off,vmx=off" cpu_features="pmu=off" # Default number of vCPUs per SB/VM: # unspecified or 0 --> will be set to 1 # < 0 --> will be set to the actual number of physical cores # > 0 <= number of physical cores --> will be set to the specified number # > number of physical cores --> will be set to the actual number of physical cores default_vcpus = 1 # Default maximum number of vCPUs per SB/VM: # unspecified or == 0 --> will be set to the actual number of physical cores or to the maximum number # of vCPUs supported by KVM if that number is exceeded # > 0 <= number of physical cores --> will be set to the specified number # > number of physical cores --> will be set to the actual number of physical cores or to the maximum number # of vCPUs supported by KVM if that number is exceeded # WARNING: Depending of the architecture, the maximum number of vCPUs supported by KVM is used when # the actual number of physical cores is greater than it. # WARNING: Be aware that this value impacts the virtual machine's memory footprint and CPU # the hotplug functionality. For example, `default_maxvcpus = 240` specifies that until 240 vCPUs # can be added to a SB/VM, but the memory footprint will be big. Another example, with # `default_maxvcpus = 8` the memory footprint will be small, but 8 will be the maximum number of # vCPUs supported by the SB/VM. In general, we recommend that you do not edit this variable, # unless you know what are you doing. # NOTICE: on arm platform with gicv2 interrupt controller, set it to 8. default_maxvcpus = 0 # Bridges can be used to hot plug devices. # Limitations: # * Currently only pci bridges are supported # * Until 30 devices per bridge can be hot plugged. # * Until 5 PCI bridges can be cold plugged per VM. # This limitation could be a bug in qemu or in the kernel # Default number of bridges per SB/VM: # unspecified or 0 --> will be set to 1 # > 1 <= 5 --> will be set to the specified number # > 5 --> will be set to 5 default_bridges = 1 # Default memory size in MiB for SB/VM. # If unspecified then it will be set 2048 MiB. default_memory = 2048 # # Default memory slots per SB/VM. # If unspecified then it will be set 10. # This is will determine the times that memory will be hotadded to sandbox/VM. #memory_slots = 10 # The size in MiB will be plused to max memory of hypervisor. # It is the memory address space for the NVDIMM devie. # If set block storage driver (block_device_driver) to "nvdimm", # should set memory_offset to the size of block device. # Default 0 #memory_offset = 0 # Specifies virtio-mem will be enabled or not. # Please note that this option should be used with the command # "echo 1 > /proc/sys/vm/overcommit_memory". # Default false #enable_virtio_mem = true # Disable block device from being used for a container's rootfs. # In case of a storage driver like devicemapper where a container's # root file system is backed by a block device, the block device is passed # directly to the hypervisor for performance reasons. # This flag prevents the block device from being passed to the hypervisor, # 9pfs is used instead to pass the rootfs. disable_block_device_use = false # Shared file system type: # - virtio-9p (default) # - virtio-fs shared_fs = "virtio-fs" # Path to vhost-user-fs daemon. virtio_fs_daemon = "/usr/libexec/virtiofsd" # List of valid annotation values for the virtiofs daemon path # Each member of the list is a path pattern as described by glob(3). # The default if not set is empty (all annotations rejected.) # Your distribution recommends: ["/usr/libexec/virtiofsd"] valid_virtio_fs_daemon_paths = ["/usr/libexec/virtiofsd"] # Default size of DAX cache in MiB virtio_fs_cache_size = 0 # Extra args for virtiofsd daemon # # Format example: # ["-o", "arg1=xxx,arg2", "-o", "hello world", "--arg3=yyy"] # # see `virtiofsd -h` for possible options. virtio_fs_extra_args = [] # Cache mode: # # - none # Metadata, data, and pathname lookup are not cached in guest. They are # always fetched from host and any changes are immediately pushed to host. # # - auto # Metadata and pathname lookup cache expires after a configured amount of # time (default is 1 second). Data is cached while the file is open (close # to open consistency). # # - always # Metadata, data, and pathname lookup are cached in guest and never expire. virtio_fs_cache = "auto" # Block storage driver to be used for the hypervisor in case the container # rootfs is backed by a block device. This is virtio-scsi, virtio-blk # or nvdimm. block_device_driver = "virtio-scsi" # Specifies cache-related options will be set to block devices or not. # Default false #block_device_cache_set = true # Specifies cache-related options for block devices. # Denotes whether use of O_DIRECT (bypass the host page cache) is enabled. # Default false #block_device_cache_direct = true # Specifies cache-related options for block devices. # Denotes whether flush requests for the device are ignored. # Default false #block_device_cache_noflush = true # Enable iothreads (data-plane) to be used. This causes IO to be # handled in a separate IO thread. This is currently only implemented # for SCSI. # enable_iothreads = false # Enable pre allocation of VM RAM, default false # Enabling this will result in lower container density # as all of the memory will be allocated and locked # This is useful when you want to reserve all the memory # upfront or in the cases where you want memory latencies # to be very predictable # Default false #enable_mem_prealloc = true # Enable huge pages for VM RAM, default false # Enabling this will result in the VM memory # being allocated using huge pages. # This is useful when you want to use vhost-user network # stacks within the container. This will automatically # result in memory pre allocation #enable_hugepages = true # Enable vhost-user storage device, default false # Enabling this will result in some Linux reserved block type # major range 240-254 being chosen to represent vhost-user devices. enable_vhost_user_store = false # The base directory specifically used for vhost-user devices. # Its sub-path "block" is used for block devices; "block/sockets" is # where we expect vhost-user sockets to live; "block/devices" is where # simulated block device nodes for vhost-user devices to live. vhost_user_store_path = "/var/run/kata-containers/vhost-user" # Enable vIOMMU, default false # Enabling this will result in the VM having a vIOMMU device # This will also add the following options to the kernel's # command line: intel_iommu=on,iommu=pt #enable_iommu = true # Enable IOMMU_PLATFORM, default false # Enabling this will result in the VM device having iommu_platform=on set #enable_iommu_platform = true # List of valid annotation values for the vhost user store path # Each member of the list is a path pattern as described by glob(3). # The default if not set is empty (all annotations rejected.) # Your distribution recommends: ["/var/run/kata-containers/vhost-user"] valid_vhost_user_store_paths = ["/var/run/kata-containers/vhost-user"] # Enable file based guest memory support. The default is an empty string which # will disable this feature. In the case of virtio-fs, this is enabled # automatically and '/dev/shm' is used as the backing folder. # This option will be ignored if VM templating is enabled. #file_mem_backend = "" # List of valid annotation values for the file_mem_backend path # Each member of the list is a path pattern as described by glob(3). # The default if not set is empty (all annotations rejected.) # Your distribution recommends: [""] valid_file_mem_backends = [""] # Enable swap of vm memory. Default false. # The behaviour is undefined if mem_prealloc is also set to true #enable_swap = true # This option changes the default hypervisor and kernel parameters # to enable debug output where available. This extra output is added # to the proxy logs, but only when proxy debug is also enabled. # # Default false #enable_debug = true # Disable the customizations done in the runtime when it detects # that it is running on top a VMM. This will result in the runtime # behaving as it would when running on bare metal. # #disable_nesting_checks = true # This is the msize used for 9p shares. It is the number of bytes # used for 9p packet payload. #msize_9p = 8192 # If true and vsocks are supported, use vsocks to communicate directly # with the agent and no proxy is started, otherwise use unix # sockets and start a proxy to communicate with the agent. # Default false use_vsock = true # If false and nvdimm is supported, use nvdimm device to plug guest image. # Otherwise virtio-block device is used. # Default is false #disable_image_nvdimm = true # VFIO devices are hotplugged on a bridge by default. # Enable hotplugging on root bus. This may be required for devices with # a large PCI bar, as this is a current limitation with hotplugging on # a bridge. This value is valid for "pc" machine type. # Default false #hotplug_vfio_on_root_bus = true # Before hot plugging a PCIe device, you need to add a pcie_root_port device. # Use this parameter when using some large PCI bar devices, such as Nvidia GPU # The value means the number of pcie_root_port # This value is valid when hotplug_vfio_on_root_bus is true and machine_type is "q35" # Default 0 #pcie_root_port = 2 # If vhost-net backend for virtio-net is not desired, set to true. Default is false, which trades off # security (vhost-net runs ring0) for network I/O performance. #disable_vhost_net = true # # Default entropy source. # The path to a host source of entropy (including a real hardware RNG) # /dev/urandom and /dev/random are two main options. # Be aware that /dev/random is a blocking source of entropy. If the host # runs out of entropy, the VMs boot time will increase leading to get startup # timeouts. # The source of entropy /dev/urandom is non-blocking and provides a # generally acceptable source of entropy. It should work well for pretty much # all practical purposes. #entropy_source= "/dev/urandom" # Path to OCI hook binaries in the *guest rootfs*. # This does not affect host-side hooks which must instead be added to # the OCI spec passed to the runtime. # # You can create a rootfs with hooks by customizing the osbuilder scripts: # https://github.com/kata-containers/osbuilder # # Hooks must be stored in a subdirectory of guest_hook_path according to their # hook type, i.e. "guest_hook_path/{prestart,postart,poststop}". # The agent will scan these directories for executable files and add them, in # lexicographical order, to the lifecycle of the guest container. # Hooks are executed in the runtime namespace of the guest. See the official documentation: # https://github.com/opencontainers/runtime-spec/blob/v1.0.1/config.md#posix-platform-hooks # Warnings will be logged if any error is encountered will scanning for hooks, # but it will not abort container execution. #guest_hook_path = "/usr/share/oci/hooks" [factory] # VM templating support. Once enabled, new VMs are created from template # using vm cloning. They will share the same initial kernel, initramfs and # agent memory by mapping it readonly. It helps speeding up new container # creation and saves a lot of memory if there are many kata containers running # on the same host. # # When disabled, new VMs are created from scratch. # # Note: Requires "initrd=" to be set ("image=" is not supported). # # Default false #enable_template = true # Specifies the path of template. # # Default "/run/vc/vm/template" #template_path = "/run/vc/vm/template" # The number of caches of VMCache: # unspecified or == 0 --> VMCache is disabled # > 0 --> will be set to the specified number # # VMCache is a function that creates VMs as caches before using it. # It helps speed up new container creation. # The function consists of a server and some clients communicating # through Unix socket. The protocol is gRPC in protocols/cache/cache.proto. # The VMCache server will create some VMs and cache them by factory cache. # It will convert the VM to gRPC format and transport it when gets # requestion from clients. # Factory grpccache is the VMCache client. It will request gRPC format # VM and convert it back to a VM. If VMCache function is enabled, # kata-runtime will request VM from factory grpccache when it creates # a new sandbox. # # Default 0 #vm_cache_number = 0 # Specify the address of the Unix socket that is used by VMCache. # # Default /var/run/kata-containers/cache.sock #vm_cache_endpoint = "/var/run/kata-containers/cache.sock" [proxy.kata] path = "/usr/libexec/kata-containers/kata-proxy" # If enabled, proxy messages will be sent to the system log # (default: disabled) #enable_debug = true [shim.kata] path = "/usr/libexec/kata-containers/kata-shim" # If enabled, shim messages will be sent to the system log # (default: disabled) #enable_debug = true # If enabled, the shim will create opentracing.io traces and spans. # (See https://www.jaegertracing.io/docs/getting-started). # # Note: By default, the shim runs in a separate network namespace. Therefore, # to allow it to send trace details to the Jaeger agent running on the host, # it is necessary to set 'disable_new_netns=true' so that it runs in the host # network namespace. # # (default: disabled) #enable_tracing = true [agent.kata] # If enabled, make the agent display debug-level messages. # (default: disabled) #enable_debug = true # Enable agent tracing. # # If enabled, the default trace mode is "dynamic" and the # default trace type is "isolated". The trace mode and type are set # explicity with the `trace_type=` and `trace_mode=` options. # # Notes: # # - Tracing is ONLY enabled when `enable_tracing` is set: explicitly # setting `trace_mode=` and/or `trace_type=` without setting `enable_tracing` # will NOT activate agent tracing. # # - See https://github.com/kata-containers/agent/blob/master/TRACING.md for # full details. # # (default: disabled) #enable_tracing = true # #trace_mode = "dynamic" #trace_type = "isolated" # Comma separated list of kernel modules and their parameters. # These modules will be loaded in the guest kernel using modprobe(8). # The following example can be used to load two kernel modules with parameters # - kernel_modules=["e1000e InterruptThrottleRate=3000,3000,3000 EEE=1", "i915 enable_ppgtt=0"] # The first word is considered as the module name and the rest as its parameters. # Container will not be started when: # * A kernel module is specified and the modprobe command is not installed in the guest # or it fails loading the module. # * The module is not available in the guest or it doesn't met the guest kernel # requirements, like architecture and version. # kernel_modules=[] [netmon] # If enabled, the network monitoring process gets started when the # sandbox is created. This allows for the detection of some additional # network being added to the existing network namespace, after the # sandbox has been created. # (default: disabled) #enable_netmon = true # Specify the path to the netmon binary. path = "/usr/libexec/kata-containers/kata-netmon" # If enabled, netmon messages will be sent to the system log # (default: disabled) #enable_debug = true [runtime] # If enabled, the runtime will log additional debug messages to the # system log # (default: disabled) #enable_debug = true # # Internetworking model # Determines how the VM should be connected to the # the container network interface # Options: # # - macvtap # Used when the Container network interface can be bridged using # macvtap. # # - none # Used when customize network. Only creates a tap device. No veth pair. # # - tcfilter # Uses tc filter rules to redirect traffic from the network interface # provided by plugin to a tap interface connected to the VM. # internetworking_model="tcfilter" # disable guest seccomp # Determines whether container seccomp profiles are passed to the virtual # machine and applied by the kata agent. If set to true, seccomp is not applied # within the guest # (default: true) disable_guest_seccomp=true # If enabled, the runtime will create opentracing.io traces and spans. # (See https://www.jaegertracing.io/docs/getting-started). # (default: disabled) #enable_tracing = true # If enabled, the runtime will not create a network namespace for shim and hypervisor processes. # This option may have some potential impacts to your host. It should only be used when you know what you're doing. # `disable_new_netns` conflicts with `enable_netmon` # `disable_new_netns` conflicts with `internetworking_model=tcfilter` and `internetworking_model=macvtap`. It works only # with `internetworking_model=none`. The tap device will be in the host network namespace and can connect to a bridge # (like OVS) directly. # If you are using docker, `disable_new_netns` only works with `docker run --net=none` # (default: false) #disable_new_netns = true # if enabled, the runtime will add all the kata processes inside one dedicated cgroup. # The container cgroups in the host are not created, just one single cgroup per sandbox. # The runtime caller is free to restrict or collect cgroup stats of the overall Kata sandbox. # The sandbox cgroup path is the parent cgroup of a container with the PodSandbox annotation. # The sandbox cgroup is constrained if there is no container type annotation. # See: https://godoc.org/github.com/kata-containers/runtime/virtcontainers#ContainerType sandbox_cgroup_only=true # Enabled experimental feature list, format: ["a", "b"]. # Experimental features are features not stable enough for production, # they may break compatibility, and are prepared for a big version bump. # Supported experimental features: # (default: []) experimental=[] # If enabled, containers are allowed to join the pid namespace of the agent # when the env variable KATA_AGENT_PIDNS is set for a container. # Use this with caution and only when required, as this option allows the container # to access the agent process. It is recommended to enable this option # only in debug scenarios and with containers with lowered priveleges. #enable_agent_pidns = true ``` --- # KSM throttler ## version Output of "` --version`": ``` /usr/bin/kata-collect-data.sh: line 178: --version: command not found ``` ## systemd service # Image details No image --- # Initrd details ```yaml --- osbuilder: url: "https://github.com/kata-containers/osbuilder" version: "fedora-osbuilder-version-unknown" rootfs-creation-time: "2020-12-10T00:32:37.530894792+0000Z" description: "osbuilder rootfs" file-format-version: "0.0.2" architecture: "x86_64" base-distro: name: "" version: "" packages: default: extra: agent: url: "https://github.com/kata-containers/agent" name: "kata-agent" version: "1.12.0-b637f6f8ac590e4b3a5099d29317f61ecbdb3c9b-dirty" agent-is-init-daemon: "no" ``` --- # Logfiles ## Runtime logs Recent runtime problems found in system journal: ``` time="2020-12-10T00:40:09.182903767+01:00" level=error msg="qemu-system-x86_64: -device vhost-vsock-pci,disable-modern=false,vhostfd=3,id=vsock-1694496973,guest-cid=1694496973,romfile=: device is modern-only, use disable-legacy=on\n" arch=amd64 command=create container=c2799cf33b661cd5c4f703b11ec70c6ed10d6acf42cc63688d7147f49cc2bbbe name=kata-runtime pid=39230 source=virtcontainers subsystem=qmp time="2020-12-10T00:40:09.182930678+01:00" level=error msg="failed to launch qemu: qemu-system-x86_64: -device vhost-vsock-pci,disable-modern=false,vhostfd=3,id=vsock-1694496973,guest-cid=1694496973,romfile=: device is modern-only, use disable-legacy=on\n" arch=amd64 command=create container=c2799cf33b661cd5c4f703b11ec70c6ed10d6acf42cc63688d7147f49cc2bbbe error="exit status 1" name=kata-runtime pid=39230 source=virtcontainers subsystem=qemu time="2020-12-10T00:40:09.186327836+01:00" level=error msg="Could not move processes into parent cgroup" arch=amd64 command=create container=c2799cf33b661cd5c4f703b11ec70c6ed10d6acf42cc63688d7147f49cc2bbbe error="write /sys/fs/cgroup/machine.slice/cgroup.procs: device or resource busy" name=kata-runtime pid=39230 source=cgroup-manager time="2020-12-10T00:40:09.232174396+01:00" level=warning msg="Failed to get container, force will not fail: Container ID (c2799cf33b661cd5c4f703b11ec70c6ed10d6acf42cc63688d7147f49cc2bbbe) does not exist" arch=amd64 command=delete container=c2799cf33b661cd5c4f703b11ec70c6ed10d6acf42cc63688d7147f49cc2bbbe name=kata-runtime pid=39265 source=runtime time="2020-12-10T00:44:59.228524121+01:00" level=error msg="Unable to launch /usr/bin/qemu-kvm: exit status 1" arch=amd64 command=create container=14e99c3f9d418557fbe082b947da2b0dafb9ad424dda746e7a4712093abc2c32 name=kata-runtime pid=39579 source=virtcontainers subsystem=qmp time="2020-12-10T00:44:59.228567783+01:00" level=error msg="qemu-system-x86_64: -device vhost-vsock-pci,disable-modern=false,vhostfd=3,id=vsock-4088016220,guest-cid=4088016220,romfile=: device is modern-only, use disable-legacy=on\n" arch=amd64 command=create container=14e99c3f9d418557fbe082b947da2b0dafb9ad424dda746e7a4712093abc2c32 name=kata-runtime pid=39579 source=virtcontainers subsystem=qmp time="2020-12-10T00:44:59.228594961+01:00" level=error msg="failed to launch qemu: qemu-system-x86_64: -device vhost-vsock-pci,disable-modern=false,vhostfd=3,id=vsock-4088016220,guest-cid=4088016220,romfile=: device is modern-only, use disable-legacy=on\n" arch=amd64 command=create container=14e99c3f9d418557fbe082b947da2b0dafb9ad424dda746e7a4712093abc2c32 error="exit status 1" name=kata-runtime pid=39579 source=virtcontainers subsystem=qemu time="2020-12-10T00:44:59.232307465+01:00" level=error msg="Could not move processes into parent cgroup" arch=amd64 command=create container=14e99c3f9d418557fbe082b947da2b0dafb9ad424dda746e7a4712093abc2c32 error="write /sys/fs/cgroup/machine.slice/cgroup.procs: device or resource busy" name=kata-runtime pid=39579 source=cgroup-manager time="2020-12-10T00:44:59.27438345+01:00" level=warning msg="Failed to get container, force will not fail: Container ID (14e99c3f9d418557fbe082b947da2b0dafb9ad424dda746e7a4712093abc2c32) does not exist" arch=amd64 command=delete container=14e99c3f9d418557fbe082b947da2b0dafb9ad424dda746e7a4712093abc2c32 name=kata-runtime pid=39609 source=runtime time="2020-12-10T00:47:29.691469009+01:00" level=error msg="Unable to launch /usr/bin/qemu-kvm: exit status 1" arch=amd64 command=create container=17a3bcead652546a8173fb637f1a4cd1ee106831801ecf81ca8c02dfb363d91b name=kata-runtime pid=39999 source=virtcontainers subsystem=qmp time="2020-12-10T00:47:29.691664904+01:00" level=error msg="qemu-system-x86_64: -device vhost-vsock-pci,disable-modern=false,vhostfd=3,id=vsock-1695942411,guest-cid=1695942411,romfile=: device is modern-only, use disable-legacy=on\n" arch=amd64 command=create container=17a3bcead652546a8173fb637f1a4cd1ee106831801ecf81ca8c02dfb363d91b name=kata-runtime pid=39999 source=virtcontainers subsystem=qmp time="2020-12-10T00:47:29.691690373+01:00" level=error msg="failed to launch qemu: qemu-system-x86_64: -device vhost-vsock-pci,disable-modern=false,vhostfd=3,id=vsock-1695942411,guest-cid=1695942411,romfile=: device is modern-only, use disable-legacy=on\n" arch=amd64 command=create container=17a3bcead652546a8173fb637f1a4cd1ee106831801ecf81ca8c02dfb363d91b error="exit status 1" name=kata-runtime pid=39999 source=virtcontainers subsystem=qemu time="2020-12-10T00:47:29.696291142+01:00" level=error msg="Could not move processes into parent cgroup" arch=amd64 command=create container=17a3bcead652546a8173fb637f1a4cd1ee106831801ecf81ca8c02dfb363d91b error="write /sys/fs/cgroup/machine.slice/cgroup.procs: device or resource busy" name=kata-runtime pid=39999 source=cgroup-manager time="2020-12-10T00:47:29.732831942+01:00" level=warning msg="Failed to get container, force will not fail: Container ID (17a3bcead652546a8173fb637f1a4cd1ee106831801ecf81ca8c02dfb363d91b) does not exist" arch=amd64 command=delete container=17a3bcead652546a8173fb637f1a4cd1ee106831801ecf81ca8c02dfb363d91b name=kata-runtime pid=40030 source=runtime time="2020-12-10T00:47:51.0484913+01:00" level=error msg="Unable to launch /usr/bin/qemu-kvm: exit status 1" arch=amd64 command=create container=1f57d7722fbba66aeb25b35915887f74bcf13f9240e4c8e6334816cad32b0e5d name=kata-runtime pid=40204 source=virtcontainers subsystem=qmp time="2020-12-10T00:47:51.048542748+01:00" level=error msg="qemu-system-x86_64: -device vhost-vsock-pci,disable-modern=false,vhostfd=3,id=vsock-786554896,guest-cid=786554896,romfile=: device is modern-only, use disable-legacy=on\n" arch=amd64 command=create container=1f57d7722fbba66aeb25b35915887f74bcf13f9240e4c8e6334816cad32b0e5d name=kata-runtime pid=40204 source=virtcontainers subsystem=qmp time="2020-12-10T00:47:51.048574666+01:00" level=error msg="failed to launch qemu: qemu-system-x86_64: -device vhost-vsock-pci,disable-modern=false,vhostfd=3,id=vsock-786554896,guest-cid=786554896,romfile=: device is modern-only, use disable-legacy=on\n" arch=amd64 command=create container=1f57d7722fbba66aeb25b35915887f74bcf13f9240e4c8e6334816cad32b0e5d error="exit status 1" name=kata-runtime pid=40204 source=virtcontainers subsystem=qemu time="2020-12-10T00:47:51.053288542+01:00" level=error msg="Could not move processes into parent cgroup" arch=amd64 command=create container=1f57d7722fbba66aeb25b35915887f74bcf13f9240e4c8e6334816cad32b0e5d error="write /sys/fs/cgroup/machine.slice/cgroup.procs: device or resource busy" name=kata-runtime pid=40204 source=cgroup-manager time="2020-12-10T00:47:51.101969922+01:00" level=warning msg="Failed to get container, force will not fail: Container ID (1f57d7722fbba66aeb25b35915887f74bcf13f9240e4c8e6334816cad32b0e5d) does not exist" arch=amd64 command=delete container=1f57d7722fbba66aeb25b35915887f74bcf13f9240e4c8e6334816cad32b0e5d name=kata-runtime pid=40235 source=runtime time="2020-12-10T00:47:51.540964722+01:00" level=warning msg="Failed to get container, force will not fail: Container ID (1f57d7722fbba66aeb25b35915887f74bcf13f9240e4c8e6334816cad32b0e5d) does not exist" arch=amd64 command=delete container=1f57d7722fbba66aeb25b35915887f74bcf13f9240e4c8e6334816cad32b0e5d name=kata-runtime pid=40315 source=runtime time="2020-12-10T00:59:01.881495485+01:00" level=error msg="Unable to launch /usr/bin/qemu-kvm: exit status 1" arch=amd64 command=create container=80fa2261a9b6a5a06ef23bdf108c030c75bc37ebe28f0713a6ffe1cf238d3f8c name=kata-runtime pid=42295 source=virtcontainers subsystem=qmp time="2020-12-10T00:59:01.88154218+01:00" level=error msg="qemu-system-x86_64: -device vhost-vsock-pci,disable-modern=false,vhostfd=3,id=vsock-2440518591,guest-cid=2440518591,romfile=: device is modern-only, use disable-legacy=on\n" arch=amd64 command=create container=80fa2261a9b6a5a06ef23bdf108c030c75bc37ebe28f0713a6ffe1cf238d3f8c name=kata-runtime pid=42295 source=virtcontainers subsystem=qmp time="2020-12-10T00:59:01.88156541+01:00" level=error msg="failed to launch qemu: qemu-system-x86_64: -device vhost-vsock-pci,disable-modern=false,vhostfd=3,id=vsock-2440518591,guest-cid=2440518591,romfile=: device is modern-only, use disable-legacy=on\n" arch=amd64 command=create container=80fa2261a9b6a5a06ef23bdf108c030c75bc37ebe28f0713a6ffe1cf238d3f8c error="exit status 1" name=kata-runtime pid=42295 source=virtcontainers subsystem=qemu time="2020-12-10T00:59:01.890562194+01:00" level=error msg="Could not move processes into parent cgroup" arch=amd64 command=create container=80fa2261a9b6a5a06ef23bdf108c030c75bc37ebe28f0713a6ffe1cf238d3f8c error="write /sys/fs/cgroup/machine.slice/cgroup.procs: device or resource busy" name=kata-runtime pid=42295 source=cgroup-manager time="2020-12-10T00:59:01.938864098+01:00" level=warning msg="Failed to get container, force will not fail: Container ID (80fa2261a9b6a5a06ef23bdf108c030c75bc37ebe28f0713a6ffe1cf238d3f8c) does not exist" arch=amd64 command=delete container=80fa2261a9b6a5a06ef23bdf108c030c75bc37ebe28f0713a6ffe1cf238d3f8c name=kata-runtime pid=42325 source=runtime time="2020-12-10T00:59:02.233067647+01:00" level=warning msg="Failed to get container, force will not fail: Container ID (80fa2261a9b6a5a06ef23bdf108c030c75bc37ebe28f0713a6ffe1cf238d3f8c) does not exist" arch=amd64 command=delete container=80fa2261a9b6a5a06ef23bdf108c030c75bc37ebe28f0713a6ffe1cf238d3f8c name=kata-runtime pid=42407 source=runtime time="2020-12-10T21:04:26.575289067+01:00" level=info msg="sanner return error: read unix @->/run/vc/vm/a2fa83b0cb8713bd8085d1e7d24b1a46a62313eebb36cedbd48787214ce9a260/qmp.sock: read: connection reset by peer" arch=amd64 command=create container=a2fa83b0cb8713bd8085d1e7d24b1a46a62313eebb36cedbd48787214ce9a260 name=kata-runtime pid=49016 source=virtcontainers subsystem=qmp time="2020-12-10T21:04:26.575363818+01:00" level=error msg="Failed to negoatiate QMP capabilities" arch=amd64 command=create container=a2fa83b0cb8713bd8085d1e7d24b1a46a62313eebb36cedbd48787214ce9a260 error="exitting QMP loop, command cancelled" name=kata-runtime pid=49016 source=virtcontainers subsystem=qemu time="2020-12-10T21:04:26.576779323+01:00" level=error msg="Unable to launch /usr/bin/qemu-kvm: exit status 1" arch=amd64 command=create container=a2fa83b0cb8713bd8085d1e7d24b1a46a62313eebb36cedbd48787214ce9a260 name=kata-runtime pid=49016 source=virtcontainers subsystem=qmp time="2020-12-10T21:04:26.576821736+01:00" level=error msg="qemu-system-x86_64: -device vhost-vsock-pci,disable-modern=false,vhostfd=3,id=vsock-1420544146,guest-cid=1420544146,romfile=: device is modern-only, use disable-legacy=on\n" arch=amd64 command=create container=a2fa83b0cb8713bd8085d1e7d24b1a46a62313eebb36cedbd48787214ce9a260 name=kata-runtime pid=49016 source=virtcontainers subsystem=qmp time="2020-12-10T21:04:26.576845451+01:00" level=error msg="failed to launch qemu: qemu-system-x86_64: -device vhost-vsock-pci,disable-modern=false,vhostfd=3,id=vsock-1420544146,guest-cid=1420544146,romfile=: device is modern-only, use disable-legacy=on\n" arch=amd64 command=create container=a2fa83b0cb8713bd8085d1e7d24b1a46a62313eebb36cedbd48787214ce9a260 error="exit status 1" name=kata-runtime pid=49016 source=virtcontainers subsystem=qemu time="2020-12-10T21:04:26.58154729+01:00" level=error msg="Could not move processes into parent cgroup" arch=amd64 command=create container=a2fa83b0cb8713bd8085d1e7d24b1a46a62313eebb36cedbd48787214ce9a260 error="write /sys/fs/cgroup/machine.slice/cgroup.procs: device or resource busy" name=kata-runtime pid=49016 source=cgroup-manager time="2020-12-10T21:04:26.622223549+01:00" level=warning msg="Failed to get container, force will not fail: Container ID (a2fa83b0cb8713bd8085d1e7d24b1a46a62313eebb36cedbd48787214ce9a260) does not exist" arch=amd64 command=delete container=a2fa83b0cb8713bd8085d1e7d24b1a46a62313eebb36cedbd48787214ce9a260 name=kata-runtime pid=49054 source=runtime time="2020-12-10T21:04:26.898168028+01:00" level=warning msg="Failed to get container, force will not fail: Container ID (a2fa83b0cb8713bd8085d1e7d24b1a46a62313eebb36cedbd48787214ce9a260) does not exist" arch=amd64 command=delete container=a2fa83b0cb8713bd8085d1e7d24b1a46a62313eebb36cedbd48787214ce9a260 name=kata-runtime pid=49134 source=runtime time="2020-12-10T21:04:41.048232229+01:00" level=info msg="sanner return error: read unix @->/run/vc/vm/e3a7bc95faf2a210423f0b385def1fcceea79b91a48cd65b12a412eb061677b6/qmp.sock: read: connection reset by peer" arch=amd64 command=create container=e3a7bc95faf2a210423f0b385def1fcceea79b91a48cd65b12a412eb061677b6 name=kata-runtime pid=49239 source=virtcontainers subsystem=qmp time="2020-12-10T21:04:41.048314325+01:00" level=error msg="Failed to negoatiate QMP capabilities" arch=amd64 command=create container=e3a7bc95faf2a210423f0b385def1fcceea79b91a48cd65b12a412eb061677b6 error="exitting QMP loop, command cancelled" name=kata-runtime pid=49239 source=virtcontainers subsystem=qemu time="2020-12-10T21:04:41.0497181+01:00" level=error msg="Unable to launch /usr/bin/qemu-kvm: exit status 1" arch=amd64 command=create container=e3a7bc95faf2a210423f0b385def1fcceea79b91a48cd65b12a412eb061677b6 name=kata-runtime pid=49239 source=virtcontainers subsystem=qmp time="2020-12-10T21:04:41.049759324+01:00" level=error msg="qemu-system-x86_64: -device vhost-vsock-pci,disable-modern=false,vhostfd=3,id=vsock-1696009078,guest-cid=1696009078,romfile=: device is modern-only, use disable-legacy=on\n" arch=amd64 command=create container=e3a7bc95faf2a210423f0b385def1fcceea79b91a48cd65b12a412eb061677b6 name=kata-runtime pid=49239 source=virtcontainers subsystem=qmp time="2020-12-10T21:04:41.049782802+01:00" level=error msg="failed to launch qemu: qemu-system-x86_64: -device vhost-vsock-pci,disable-modern=false,vhostfd=3,id=vsock-1696009078,guest-cid=1696009078,romfile=: device is modern-only, use disable-legacy=on\n" arch=amd64 command=create container=e3a7bc95faf2a210423f0b385def1fcceea79b91a48cd65b12a412eb061677b6 error="exit status 1" name=kata-runtime pid=49239 source=virtcontainers subsystem=qemu time="2020-12-10T21:04:41.053474819+01:00" level=error msg="Could not move processes into parent cgroup" arch=amd64 command=create container=e3a7bc95faf2a210423f0b385def1fcceea79b91a48cd65b12a412eb061677b6 error="write /sys/fs/cgroup/machine.slice/cgroup.procs: device or resource busy" name=kata-runtime pid=49239 source=cgroup-manager time="2020-12-10T21:04:41.108778502+01:00" level=warning msg="Failed to get container, force will not fail: Container ID (e3a7bc95faf2a210423f0b385def1fcceea79b91a48cd65b12a412eb061677b6) does not exist" arch=amd64 command=delete container=e3a7bc95faf2a210423f0b385def1fcceea79b91a48cd65b12a412eb061677b6 name=kata-runtime pid=49272 source=runtime time="2020-12-10T21:04:41.587788485+01:00" level=warning msg="Failed to get container, force will not fail: Container ID (e3a7bc95faf2a210423f0b385def1fcceea79b91a48cd65b12a412eb061677b6) does not exist" arch=amd64 command=delete container=e3a7bc95faf2a210423f0b385def1fcceea79b91a48cd65b12a412eb061677b6 name=kata-runtime pid=49358 source=runtime time="2020-12-10T21:07:10.704644814+01:00" level=info msg="sanner return error: read unix @->/run/vc/vm/d484c917e6e8f77852d3352d38fd1e672fd3633864a6bfa88db19275be1175a7/qmp.sock: read: connection reset by peer" arch=amd64 command=create container=d484c917e6e8f77852d3352d38fd1e672fd3633864a6bfa88db19275be1175a7 name=kata-runtime pid=49461 source=virtcontainers subsystem=qmp time="2020-12-10T21:07:10.704716363+01:00" level=error msg="Failed to negoatiate QMP capabilities" arch=amd64 command=create container=d484c917e6e8f77852d3352d38fd1e672fd3633864a6bfa88db19275be1175a7 error="exitting QMP loop, command cancelled" name=kata-runtime pid=49461 source=virtcontainers subsystem=qemu time="2020-12-10T21:07:10.706202149+01:00" level=error msg="Unable to launch /usr/bin/qemu-kvm: exit status 1" arch=amd64 command=create container=d484c917e6e8f77852d3352d38fd1e672fd3633864a6bfa88db19275be1175a7 name=kata-runtime pid=49461 source=virtcontainers subsystem=qmp time="2020-12-10T21:07:10.706243557+01:00" level=error msg="qemu-system-x86_64: -device vhost-vsock-pci,disable-modern=false,vhostfd=3,id=vsock-204392473,guest-cid=204392473,romfile=: device is modern-only, use disable-legacy=on\n" arch=amd64 command=create container=d484c917e6e8f77852d3352d38fd1e672fd3633864a6bfa88db19275be1175a7 name=kata-runtime pid=49461 source=virtcontainers subsystem=qmp time="2020-12-10T21:07:10.706277443+01:00" level=error msg="failed to launch qemu: qemu-system-x86_64: -device vhost-vsock-pci,disable-modern=false,vhostfd=3,id=vsock-204392473,guest-cid=204392473,romfile=: device is modern-only, use disable-legacy=on\n" arch=amd64 command=create container=d484c917e6e8f77852d3352d38fd1e672fd3633864a6bfa88db19275be1175a7 error="exit status 1" name=kata-runtime pid=49461 source=virtcontainers subsystem=qemu time="2020-12-10T21:07:10.714155915+01:00" level=error msg="Could not move processes into parent cgroup" arch=amd64 command=create container=d484c917e6e8f77852d3352d38fd1e672fd3633864a6bfa88db19275be1175a7 error="write /sys/fs/cgroup/machine.slice/cgroup.procs: device or resource busy" name=kata-runtime pid=49461 source=cgroup-manager time="2020-12-10T21:07:10.749736271+01:00" level=warning msg="Failed to get container, force will not fail: Container ID (d484c917e6e8f77852d3352d38fd1e672fd3633864a6bfa88db19275be1175a7) does not exist" arch=amd64 command=delete container=d484c917e6e8f77852d3352d38fd1e672fd3633864a6bfa88db19275be1175a7 name=kata-runtime pid=49495 source=runtime time="2020-12-10T21:07:11.079249467+01:00" level=warning msg="Failed to get container, force will not fail: Container ID (d484c917e6e8f77852d3352d38fd1e672fd3633864a6bfa88db19275be1175a7) does not exist" arch=amd64 command=delete container=d484c917e6e8f77852d3352d38fd1e672fd3633864a6bfa88db19275be1175a7 name=kata-runtime pid=49555 source=runtime ``` ## Proxy logs Recent proxy problems found in system journal: ``` time="2020-12-03T20:15:57.867585401+01:00" level=fatal msg="failed to handle exit signal" error="close unix @->/run/vc/vm/a187b6f5a87fc598deba1af17fc74cbbe9a3886f34138a9e244b3a3957932f1d/kata.sock: use of closed network connection" name=kata-proxy pid=10431 sandbox=a187b6f5a87fc598deba1af17fc74cbbe9a3886f34138a9e244b3a3957932f1d source=proxy time="2020-12-03T20:20:44.82308422+01:00" level=fatal msg="failed to handle exit signal" error="close unix @->/run/vc/vm/a187b6f5a87fc598deba1af17fc74cbbe9a3886f34138a9e244b3a3957932f1d/kata.sock: use of closed network connection" name=kata-proxy pid=14315 sandbox=a187b6f5a87fc598deba1af17fc74cbbe9a3886f34138a9e244b3a3957932f1d source=proxy time="2020-12-03T20:22:20.273162869+01:00" level=fatal msg="failed to handle exit signal" error="close unix @->/run/vc/vm/a187b6f5a87fc598deba1af17fc74cbbe9a3886f34138a9e244b3a3957932f1d/kata.sock: use of closed network connection" name=kata-proxy pid=14797 sandbox=a187b6f5a87fc598deba1af17fc74cbbe9a3886f34138a9e244b3a3957932f1d source=proxy time="2020-12-03T20:28:31.306379018+01:00" level=fatal msg="failed to handle exit signal" error="close unix @->/run/vc/vm/a187b6f5a87fc598deba1af17fc74cbbe9a3886f34138a9e244b3a3957932f1d/kata.sock: use of closed network connection" name=kata-proxy pid=15617 sandbox=a187b6f5a87fc598deba1af17fc74cbbe9a3886f34138a9e244b3a3957932f1d source=proxy ``` ## Shim logs Recent shim problems found in system journal: ``` time="2020-10-30T06:19:06.174384981+01:00" level=warning msg="close stdin failed" container=caaab1fc99f7c501dea44668fd56ac7db4712fed8a5580048f74bb7c58017a17 error="rpc error: code = NotFound desc = Process d92ec694-99e9-4a1d-a45c-a2a728a3941d not found (container caaab1fc99f7c501dea44668fd56ac7db4712fed8a5580048f74bb7c58017a17)" exec-id=d92ec694-99e9-4a1d-a45c-a2a728a3941d name=kata-shim pid=41412 source=shim time="2020-10-31T18:46:57.196825135+01:00" level=error msg="forward signal failed" container=caaab1fc99f7c501dea44668fd56ac7db4712fed8a5580048f74bb7c58017a17 error="rpc error: code = Unknown desc = no such process" exec-id=0a4536b3-2507-48a2-af60-1a032b54b814 name=kata-shim pid=59386 signal="urgent I/O condition" source=shim time="2020-11-03T13:36:50.887726996+01:00" level=warning msg="copy stdin failed" container=caaab1fc99f7c501dea44668fd56ac7db4712fed8a5580048f74bb7c58017a17 error="read /dev/stdin: input/output error" exec-id=caaab1fc99f7c501dea44668fd56ac7db4712fed8a5580048f74bb7c58017a17 name=kata-shim pid=1 source=shim time="2020-11-04T22:04:49.791699876+01:00" level=warning msg="close stdin failed" container=caaab1fc99f7c501dea44668fd56ac7db4712fed8a5580048f74bb7c58017a17 error="rpc error: code = NotFound desc = Process c9602ef8-c815-453b-ad4c-2a0c61fd7a6e not found (container caaab1fc99f7c501dea44668fd56ac7db4712fed8a5580048f74bb7c58017a17)" exec-id=c9602ef8-c815-453b-ad4c-2a0c61fd7a6e name=kata-shim pid=15881 source=shim time="2020-11-05T07:02:06.860403943+01:00" level=error msg="forward signal failed" container=caaab1fc99f7c501dea44668fd56ac7db4712fed8a5580048f74bb7c58017a17 error="rpc error: code = NotFound desc = Could not signal process: rpc error: code = NotFound desc = Process b5800260-610e-4704-b59e-83e07642b35e not found (container caaab1fc99f7c501dea44668fd56ac7db4712fed8a5580048f74bb7c58017a17)" exec-id=b5800260-610e-4704-b59e-83e07642b35e name=kata-shim pid=20366 signal="urgent I/O condition" source=shim time="2020-11-05T12:36:57.846401694+01:00" level=warning msg="close stdin failed" container=caaab1fc99f7c501dea44668fd56ac7db4712fed8a5580048f74bb7c58017a17 error="rpc error: code = NotFound desc = Process 8e8c31dc-a5e7-412e-8806-b483d0ee0b9f not found (container caaab1fc99f7c501dea44668fd56ac7db4712fed8a5580048f74bb7c58017a17)" exec-id=8e8c31dc-a5e7-412e-8806-b483d0ee0b9f name=kata-shim pid=23147 source=shim time="2020-11-07T12:03:18.811226723+01:00" level=error msg="forward signal failed" container=caaab1fc99f7c501dea44668fd56ac7db4712fed8a5580048f74bb7c58017a17 error="rpc error: code = Unknown desc = no such process" exec-id=21c37ab4-4613-464f-93e5-39ee6881d0d2 name=kata-shim pid=46864 signal="urgent I/O condition" source=shim time="2020-11-08T02:42:28.779200665+01:00" level=warning msg="close stdin failed" container=caaab1fc99f7c501dea44668fd56ac7db4712fed8a5580048f74bb7c58017a17 error="rpc error: code = NotFound desc = Process 62a45e6b-f9b8-4dc7-96e8-0239bd1ea125 not found (container caaab1fc99f7c501dea44668fd56ac7db4712fed8a5580048f74bb7c58017a17)" exec-id=62a45e6b-f9b8-4dc7-96e8-0239bd1ea125 name=kata-shim pid=54166 source=shim time="2020-11-09T05:16:53.823652401+01:00" level=warning msg="close stdin failed" container=caaab1fc99f7c501dea44668fd56ac7db4712fed8a5580048f74bb7c58017a17 error="rpc error: code = NotFound desc = Process 6c29ea2b-c32d-41be-82b5-a9ce81ace949 not found (container caaab1fc99f7c501dea44668fd56ac7db4712fed8a5580048f74bb7c58017a17)" exec-id=6c29ea2b-c32d-41be-82b5-a9ce81ace949 name=kata-shim pid=67503 source=shim time="2020-11-15T09:25:25.659556944+01:00" level=warning msg="close stdin failed" container=caaab1fc99f7c501dea44668fd56ac7db4712fed8a5580048f74bb7c58017a17 error="rpc error: code = NotFound desc = Process 8e0086e6-88ae-46f5-a6a8-163ae77d983d not found (container caaab1fc99f7c501dea44668fd56ac7db4712fed8a5580048f74bb7c58017a17)" exec-id=8e0086e6-88ae-46f5-a6a8-163ae77d983d name=kata-shim pid=34209 source=shim time="2020-11-18T22:39:04.684312737+01:00" level=warning msg="close stdin failed" container=caaab1fc99f7c501dea44668fd56ac7db4712fed8a5580048f74bb7c58017a17 error="rpc error: code = NotFound desc = Process 164bf25e-6f46-47ea-bc35-b0e04a4374c2 not found (container caaab1fc99f7c501dea44668fd56ac7db4712fed8a5580048f74bb7c58017a17)" exec-id=164bf25e-6f46-47ea-bc35-b0e04a4374c2 name=kata-shim pid=76649 source=shim time="2020-11-19T11:52:43.689548364+01:00" level=warning msg="close stdin failed" container=caaab1fc99f7c501dea44668fd56ac7db4712fed8a5580048f74bb7c58017a17 error="rpc error: code = NotFound desc = Process d6bc567c-e59d-4a27-9ee7-c369f3233b46 not found (container caaab1fc99f7c501dea44668fd56ac7db4712fed8a5580048f74bb7c58017a17)" exec-id=d6bc567c-e59d-4a27-9ee7-c369f3233b46 name=kata-shim pid=83165 source=shim time="2020-11-21T00:34:13.574734883+01:00" level=warning msg="close stdin failed" container=caaab1fc99f7c501dea44668fd56ac7db4712fed8a5580048f74bb7c58017a17 error="rpc error: code = NotFound desc = Process d2ed6833-5fda-4136-8346-7fbe60c033cf not found (container caaab1fc99f7c501dea44668fd56ac7db4712fed8a5580048f74bb7c58017a17)" exec-id=d2ed6833-5fda-4136-8346-7fbe60c033cf name=kata-shim pid=12892 source=shim time="2020-11-23T11:38:41.582783958+01:00" level=warning msg="close stdin failed" container=caaab1fc99f7c501dea44668fd56ac7db4712fed8a5580048f74bb7c58017a17 error="rpc error: code = NotFound desc = Process 31c82e97-2652-4276-ba3c-8e3bc042ba44 not found (container caaab1fc99f7c501dea44668fd56ac7db4712fed8a5580048f74bb7c58017a17)" exec-id=31c82e97-2652-4276-ba3c-8e3bc042ba44 name=kata-shim pid=42254 source=shim time="2020-11-26T07:00:24.152446105+01:00" level=warning msg="close stdin failed" container=caaab1fc99f7c501dea44668fd56ac7db4712fed8a5580048f74bb7c58017a17 error="rpc error: code = NotFound desc = Process 06f9f952-4b07-4c12-96cd-bed0bc5bfaeb not found (container caaab1fc99f7c501dea44668fd56ac7db4712fed8a5580048f74bb7c58017a17)" exec-id=06f9f952-4b07-4c12-96cd-bed0bc5bfaeb name=kata-shim pid=5365 source=shim time="2020-11-28T03:12:37.124426751+01:00" level=error msg="forward signal failed" container=caaab1fc99f7c501dea44668fd56ac7db4712fed8a5580048f74bb7c58017a17 error="rpc error: code = Unknown desc = no such process" exec-id=1a7bceca-e9e1-4ad8-81f3-0fac8537db73 name=kata-shim pid=27182 signal="urgent I/O condition" source=shim time="2020-11-28T10:58:51.146528869+01:00" level=warning msg="close stdin failed" container=caaab1fc99f7c501dea44668fd56ac7db4712fed8a5580048f74bb7c58017a17 error="rpc error: code = NotFound desc = Process 8cdfc109-c87d-4a14-b407-fee5454ed142 not found (container caaab1fc99f7c501dea44668fd56ac7db4712fed8a5580048f74bb7c58017a17)" exec-id=8cdfc109-c87d-4a14-b407-fee5454ed142 name=kata-shim pid=31009 source=shim time="2020-11-29T03:29:56.098117386+01:00" level=warning msg="close stdin failed" container=caaab1fc99f7c501dea44668fd56ac7db4712fed8a5580048f74bb7c58017a17 error="rpc error: code = NotFound desc = Process b9c7fe30-2d4d-4cc3-953e-c29e7534b00a not found (container caaab1fc99f7c501dea44668fd56ac7db4712fed8a5580048f74bb7c58017a17)" exec-id=b9c7fe30-2d4d-4cc3-953e-c29e7534b00a name=kata-shim pid=39221 source=shim time="2020-11-30T09:24:57.136810691+01:00" level=error msg="forward signal failed" container=caaab1fc99f7c501dea44668fd56ac7db4712fed8a5580048f74bb7c58017a17 error="rpc error: code = NotFound desc = Could not signal process: rpc error: code = NotFound desc = Process 05a4eb17-172b-45b3-9156-62020ac6765f not found (container caaab1fc99f7c501dea44668fd56ac7db4712fed8a5580048f74bb7c58017a17)" exec-id=05a4eb17-172b-45b3-9156-62020ac6765f name=kata-shim pid=54169 signal="urgent I/O condition" source=shim time="2020-11-30T12:55:35.133667012+01:00" level=warning msg="close stdin failed" container=caaab1fc99f7c501dea44668fd56ac7db4712fed8a5580048f74bb7c58017a17 error="rpc error: code = NotFound desc = Process a5460515-7060-4b04-8a08-0460eec9b4da not found (container caaab1fc99f7c501dea44668fd56ac7db4712fed8a5580048f74bb7c58017a17)" exec-id=a5460515-7060-4b04-8a08-0460eec9b4da name=kata-shim pid=55910 source=shim time="2020-12-02T13:05:09.429365117+01:00" level=warning msg="close stdin failed" container=caaab1fc99f7c501dea44668fd56ac7db4712fed8a5580048f74bb7c58017a17 error="rpc error: code = NotFound desc = Process 99de3f4f-244c-46d1-9364-b319c5f54d0f not found (container caaab1fc99f7c501dea44668fd56ac7db4712fed8a5580048f74bb7c58017a17)" exec-id=99de3f4f-244c-46d1-9364-b319c5f54d0f name=kata-shim pid=79793 source=shim ``` ## Throttler logs No recent throttler problems found in system journal. --- # Container manager details No `docker` No `kubectl` No `crio` No `containerd` --- # Packages No `dpkg` Have `rpm` Output of "`rpm -qa|egrep "(cc-oci-runtimecc-runtimerunv|kata-proxy|kata-runtime|kata-shim|kata-ksm-throttler|kata-containers-image|linux-container|qemu-)"`": ``` qemu-audio-alsa-5.1.0-5.fc33.x86_64 qemu-system-rx-5.1.0-5.fc33.x86_64 qemu-system-sparc-5.1.0-5.fc33.x86_64 qemu-system-m68k-core-5.1.0-5.fc33.x86_64 qemu-block-curl-5.1.0-5.fc33.x86_64 qemu-system-x86-5.1.0-5.fc33.x86_64 qemu-system-riscv-5.1.0-5.fc33.x86_64 qemu-system-moxie-core-5.1.0-5.fc33.x86_64 qemu-block-iscsi-5.1.0-5.fc33.x86_64 qemu-system-m68k-5.1.0-5.fc33.x86_64 qemu-char-baum-5.1.0-5.fc33.x86_64 qemu-system-ppc-core-5.1.0-5.fc33.x86_64 qemu-block-ssh-5.1.0-5.fc33.x86_64 qemu-system-arm-5.1.0-5.fc33.x86_64 qemu-system-xtensa-5.1.0-5.fc33.x86_64 qemu-system-avr-core-5.1.0-5.fc33.x86_64 qemu-system-sh4-core-5.1.0-5.fc33.x86_64 qemu-block-dmg-5.1.0-5.fc33.x86_64 qemu-system-cris-5.1.0-5.fc33.x86_64 qemu-5.1.0-5.fc33.x86_64 qemu-device-usb-redirect-5.1.0-5.fc33.x86_64 qemu-system-nios2-core-5.1.0-5.fc33.x86_64 qemu-device-display-qxl-5.1.0-5.fc33.x86_64 qemu-block-nfs-5.1.0-5.fc33.x86_64 qemu-system-aarch64-5.1.0-5.fc33.x86_64 qemu-system-moxie-5.1.0-5.fc33.x86_64 qemu-system-tricore-5.1.0-5.fc33.x86_64 qemu-audio-pa-5.1.0-5.fc33.x86_64 qemu-system-x86-core-5.1.0-5.fc33.x86_64 qemu-system-lm32-core-5.1.0-5.fc33.x86_64 qemu-system-riscv-core-5.1.0-5.fc33.x86_64 ipxe-roms-qemu-20200823-1.git4bd064de.fc33.noarch qemu-audio-sdl-5.1.0-5.fc33.x86_64 qemu-ui-curses-5.1.0-5.fc33.x86_64 qemu-system-alpha-5.1.0-5.fc33.x86_64 qemu-user-5.1.0-5.fc33.x86_64 qemu-system-nios2-5.1.0-5.fc33.x86_64 qemu-system-unicore32-5.1.0-5.fc33.x86_64 libvirt-daemon-qemu-6.6.0-3.fc33.x86_64 qemu-ui-sdl-5.1.0-5.fc33.x86_64 qemu-system-rx-core-5.1.0-5.fc33.x86_64 qemu-system-mips-core-5.1.0-5.fc33.x86_64 qemu-system-sparc-core-5.1.0-5.fc33.x86_64 qemu-block-gluster-5.1.0-5.fc33.x86_64 qemu-system-avr-5.1.0-5.fc33.x86_64 qemu-system-microblaze-5.1.0-5.fc33.x86_64 qemu-system-sh4-5.1.0-5.fc33.x86_64 qemu-system-arm-core-5.1.0-5.fc33.x86_64 qemu-system-or1k-core-5.1.0-5.fc33.x86_64 qemu-system-xtensa-core-5.1.0-5.fc33.x86_64 qemu-block-rbd-5.1.0-5.fc33.x86_64 qemu-system-mips-5.1.0-5.fc33.x86_64 qemu-system-s390x-core-5.1.0-5.fc33.x86_64 qemu-ui-spice-app-5.1.0-5.fc33.x86_64 qemu-system-lm32-5.1.0-5.fc33.x86_64 qemu-system-aarch64-core-5.1.0-5.fc33.x86_64 qemu-system-tricore-core-5.1.0-5.fc33.x86_64 libvirt-daemon-driver-qemu-6.6.0-3.fc33.x86_64 qemu-common-5.1.0-5.fc33.x86_64 qemu-kvm-5.1.0-5.fc33.x86_64 qemu-system-s390x-5.1.0-5.fc33.x86_64 qemu-system-cris-core-5.1.0-5.fc33.x86_64 qemu-audio-oss-5.1.0-5.fc33.x86_64 qemu-system-or1k-5.1.0-5.fc33.x86_64 qemu-ui-gtk-5.1.0-5.fc33.x86_64 qemu-system-microblaze-core-5.1.0-5.fc33.x86_64 qemu-img-5.1.0-5.fc33.x86_64 qemu-device-usb-smartcard-5.1.0-5.fc33.x86_64 qemu-system-ppc-5.1.0-5.fc33.x86_64 qemu-system-alpha-core-5.1.0-5.fc33.x86_64 qemu-system-unicore32-core-5.1.0-5.fc33.x86_64 qemu-kvm-core-5.1.0-5.fc33.x86_64 kata-shim-1.12.0-1.fc33.x86_64 kata-proxy-1.12.0-1.fc33.x86_64 kata-runtime-1.12.0-2.fc33.x86_64 ``` ---

Journalctr output:

Dec 10 21:04:39 <snip_hostname> unknown: ######Start Contianer#####

Dec 10 21:04:39 <snip_hostname> podman[49160]: 2020-12-10 21:04:39.510212923 +0100 CET m=+0.132783057 container create e3a7bc95faf2a210423f0b385def1fcceea79b91a48cd65b12a412eb061677b6 (image=docker.io/library/alpine:latest, name=musing_faraday)
Dec 10 21:04:39 <snip_hostname> audit[49163]: NETFILTER_CFG table=filter family=7 entries=0 op=xt_register pid=49163 subj=unconfined_u:system_r:container_runtime_t:s0-s0:c0.c1023 comm="podman"
Dec 10 21:04:39 <snip_hostname> NetworkManager[9490]: <info>  [1607630679.5636] manager: (veth72bb2dc4): new Veth device (/org/freedesktop/NetworkManager/Devices/11)
Dec 10 21:04:39 <snip_hostname> kernel: IPv6: ADDRCONF(NETDEV_CHANGE): veth72bb2dc4: link becomes ready
Dec 10 21:04:39 <snip_hostname> kernel: IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready
Dec 10 21:04:39 <snip_hostname> NetworkManager[9490]: <info>  [1607630679.5657] device (veth72bb2dc4): carrier: link connected
Dec 10 21:04:39 <snip_hostname> kernel: cni_dmz_inet: port 1(veth72bb2dc4) entered blocking state
Dec 10 21:04:39 <snip_hostname> kernel: cni_dmz_inet: port 1(veth72bb2dc4) entered disabled state
Dec 10 21:04:39 <snip_hostname> kernel: device veth72bb2dc4 entered promiscuous mode
Dec 10 21:04:39 <snip_hostname> audit: ANOM_PROMISCUOUS dev=veth72bb2dc4 prom=256 old_prom=0 auid=1000 uid=0 gid=0 ses=593
Dec 10 21:04:39 <snip_hostname> kernel: cni_dmz_inet: port 1(veth72bb2dc4) entered blocking state
Dec 10 21:04:39 <snip_hostname> kernel: cni_dmz_inet: port 1(veth72bb2dc4) entered forwarding state
Dec 10 21:04:39 <snip_hostname> NetworkManager[9490]: <info>  [1607630679.5736] device (cni_dmz_inet): carrier: link connected
Dec 10 21:04:39 <snip_hostname> systemd-udevd[49217]: ethtool: autonegotiation is unset or enabled, the speed and duplex are not writable.
Dec 10 21:04:39 <snip_hostname> systemd-udevd[49217]: Using default interface naming scheme 'v245'.
Dec 10 21:04:40 <snip_hostname> dnsmasq[49236]: started, version 2.82 cachesize 150
Dec 10 21:04:40 <snip_hostname> dnsmasq[49236]: compile time options: IPv6 GNU-getopt DBus no-UBus no-i18n IDN2 DHCP DHCPv6 no-Lua TFTP no-conntrack ipset auth DNSSEC loop-detect inotify dumpfile
Dec 10 21:04:40 <snip_hostname> dnsmasq[49236]: using only locally-known addresses for domain dmzi.pods.<snip_mydomain>
Dec 10 21:04:40 <snip_hostname> dnsmasq[49236]: reading /etc/resolv.conf
Dec 10 21:04:40 <snip_hostname> dnsmasq[49236]: using only locally-known addresses for domain dmzi.pods.<snip_mydomain>
Dec 10 21:04:40 <snip_hostname> dnsmasq[49236]: using nameserver 192.168.2.254#53
Dec 10 21:04:40 <snip_hostname> dnsmasq[49236]: using nameserver fe80::56be:53ff:fe7a:d9f9%vnet#53
Dec 10 21:04:40 <snip_hostname> dnsmasq[49236]: read /run/containers/cni/dnsname/cni_dmz_inet/addnhosts - 2 addresses
Dec 10 21:04:40 <snip_hostname> systemd[1]: Started libpod-conmon-e3a7bc95faf2a210423f0b385def1fcceea79b91a48cd65b12a412eb061677b6.scope.
Dec 10 21:04:40 <snip_hostname> kata-runtime[49239]: time="2020-12-10T21:04:40.94194824+01:00" level=info msg="loaded configuration" arch=amd64 command=create file=/usr/share/kata-containers/defaults/configuration.toml format=TOML name=kata-runtime pid=49239 source=katautils
Dec 10 21:04:40 <snip_hostname> kata-runtime[49239]: time="2020-12-10T21:04:40.942240679+01:00" level=info msg="vsock supported" arch=amd64 command=create name=kata-runtime pid=49239 source=katautils
Dec 10 21:04:40 <snip_hostname> kata-runtime[49239]: time="2020-12-10T21:04:40.94226155+01:00" level=info msg="IOMMUPlatform is disabled by default." arch=amd64 command=create name=kata-runtime pid=49239 source=katautils
Dec 10 21:04:40 <snip_hostname> kata-runtime[49239]: time="2020-12-10T21:04:40.942300534+01:00" level=info msg="VSOCK supported, configure to not use proxy" arch=amd64 command=create name=kata-runtime pid=49239 source=katautils
Dec 10 21:04:40 <snip_hostname> kata-runtime[49239]: time="2020-12-10T21:04:40.942324313+01:00" level=info arch=amd64 arguments="\"create --bundle /var/lib/containers/storage/overlay-containers/e3a7bc95faf2a210423f0b385def1fcceea79b91a48cd65b12a412eb061677b6/userdata --pid-file /var/run/containers/storage/overlay-containers/e3a7bc95faf2a210423f0b385def1fcceea79b91a48cd65b12a412eb061677b6/userdata/pidfile --console-socket /tmp/conmon-term.56NXV0 e3a7bc95faf2a210423f0b385def1fcceea79b91a48cd65b12a412eb061677b6\"" command=create commit= name=kata-runtime pid=49239 source=runtime version=1.12.0
Dec 10 21:04:40 <snip_hostname> kata-runtime[49239]: time="2020-12-10T21:04:40.943320213+01:00" level=info msg="shm-size detected: 65536000" arch=amd64 command=create container=e3a7bc95faf2a210423f0b385def1fcceea79b91a48cd65b12a412eb061677b6 name=kata-runtime pid=49239 source=virtcontainers subsystem=oci
Dec 10 21:04:40 <snip_hostname> kata-runtime[49239]: time="2020-12-10T21:04:40.95560129+01:00" level=info msg="adding volume" arch=amd64 command=create container=e3a7bc95faf2a210423f0b385def1fcceea79b91a48cd65b12a412eb061677b6 name=kata-runtime pid=49239 source=virtcontainers subsystem=qemu volume-type=virtio-fs
Dec 10 21:04:40 <snip_hostname> kata-runtime[49239]: time="2020-12-10T21:04:40.956727489+01:00" level=info msg="Endpoints found after scan" arch=amd64 command=create container=e3a7bc95faf2a210423f0b385def1fcceea79b91a48cd65b12a412eb061677b6 endpoints="[0xc0001b0000]" name=kata-runtime pid=49239 source=virtcontainers subsystem=network
Dec 10 21:04:40 <snip_hostname> kata-runtime[49239]: time="2020-12-10T21:04:40.956801928+01:00" level=info msg="Attaching endpoint" arch=amd64 command=create container=e3a7bc95faf2a210423f0b385def1fcceea79b91a48cd65b12a412eb061677b6 endpoint-type=virtual hotplug=false name=kata-runtime pid=49239 source=virtcontainers subsystem=network
Dec 10 21:04:40 <snip_hostname> kernel: eth0: Caught tx_queue_len zero misconfig
Dec 10 21:04:40 <snip_hostname> systemd[1]: Started libcontainer container e3a7bc95faf2a210423f0b385def1fcceea79b91a48cd65b12a412eb061677b6.
Dec 10 21:04:40 <snip_hostname> audit: BPF prog-id=597 op=LOAD
Dec 10 21:04:40 <snip_hostname> kata-runtime[49239]: time="2020-12-10T21:04:40.989795825+01:00" level=info msg="Starting VM" arch=amd64 command=create container=e3a7bc95faf2a210423f0b385def1fcceea79b91a48cd65b12a412eb061677b6 name=kata-runtime pid=49239 sandbox=e3a7bc95faf2a210423f0b385def1fcceea79b91a48cd65b12a412eb061677b6 source=virtcontainers subsystem=sandbox
Dec 10 21:04:40 <snip_hostname> kata-runtime[49239]: time="2020-12-10T21:04:40.990586628+01:00" level=info msg="Adding extra file [0xc0000109b8 0xc000010a90 0xc000010a78]" arch=amd64 command=create container=e3a7bc95faf2a210423f0b385def1fcceea79b91a48cd65b12a412eb061677b6 name=kata-runtime pid=49239 source=virtcontainers subsystem=qmp
Dec 10 21:04:40 <snip_hostname> kata-runtime[49239]: time="2020-12-10T21:04:40.990656344+01:00" level=info msg="launching /usr/bin/qemu-kvm with: [-name sandbox-e3a7bc95faf2a210423f0b385def1fcceea79b91a48cd65b12a412eb061677b6 -uuid 1623d062-9be8-47db-b991-00313c566c9d -machine q35,accel=kvm,kernel_irqchip -cpu host,pmu=off -qmp unix:/run/vc/vm/e3a7bc95faf2a210423f0b385def1fcceea79b91a48cd65b12a412eb061677b6/qmp.sock,server,nowait -m 2048M,slots=10,maxmem=12593M -device pci-bridge,bus=pcie.0,id=pci-bridge-0,chassis_nr=1,shpc=on,addr=2,romfile= -device virtio-serial-pci,disable-modern=false,id=serial0,romfile= -device virtconsole,chardev=charconsole0,id=console0 -chardev socket,id=charconsole0,path=/run/vc/vm/e3a7bc95faf2a210423f0b385def1fcceea79b91a48cd65b12a412eb061677b6/console.sock,server,nowait -device virtio-scsi-pci,id=scsi0,disable-modern=false,romfile= -object rng-random,id=rng0,filename=/dev/urandom -device virtio-rng-pci,rng=rng0,romfile= -device vhost-vsock-pci,disable-modern=false,vhostfd=3,id=vsock-1696009078,guest-cid=1696009078,romfile= -chardev socket,id=char-ce0ea51ead1714e2,path=/run/vc/vm/e3a7bc95faf2a210423f0b385def1fcceea79b91a48cd65b12a412eb061677b6/vhost-fs.sock -device vhost-user-fs-pci,chardev=char-ce0ea51ead1714e2,tag=kataShared,romfile= -netdev tap,id=network-0,vhost=on,vhostfds=4,fds=5 -device driver=virtio-net-pci,netdev=network-0,mac=3e:a5:ad:1b:75:c5,disable-modern=false,mq=on,vectors=4,romfile= -rtc base=utc,driftfix=slew,clock=host -global kvm-pit.lost_tick_policy=discard -vga none -no-user-config -nodefaults -nographic --no-reboot -daemonize -object memory-backend-file,id=dimm1,size=2048M,mem-path=/dev/shm,share=on -numa node,memdev=dimm1 -kernel /usr/lib/modules/5.9.13-200.fc33.x86_64/vmlinuz -initrd /var/cache/kata-containers/osbuilder-images/5.9.13-200.fc33.x86_64/fedora-kata-5.9.13-200.fc33.x86_64.initrd -append tsc=reliable no_timer_check rcupdate.rcu_expedited=1 i8042.direct=1 i8042.dumbkbd=1 i8042.nopnp=1 i8042.noaux=1 noreplace-smp reboot=k console=hvc0 console=hvc1 cryptomgr.notests net.ifnames=0 pci=lastbus=0 iommu=off quiet panic=1 nr_cpus=2 agent.use_vsock=true scsi_mod.scan=none systemd.unified_cgroup_hierarchy=0 -pidfile /run/vc/vm/e3a7bc95faf2a210423f0b385def1fcceea79b91a48cd65b12a412eb061677b6/pid -smp 1,cores=1,threads=1,sockets=2,maxcpus=2]" arch=amd64 command=create container=e3a7bc95faf2a210423f0b385def1fcceea79b91a48cd65b12a412eb061677b6 name=kata-runtime pid=49239 source=virtcontainers subsystem=qmp
Dec 10 21:04:41 <snip_hostname> audit[49249]: AVC avc:  denied  { read } for  pid=49249 comm="qemu-system-x86" name="openssl.cnf" dev="dm-1" ino=9469427 scontext=system_u:system_r:container_kvm_t:s0:c76,c489 tcontext=system_u:object_r:cert_t:s0 tclass=file permissive=0
Dec 10 21:04:41 <snip_hostname> audit[49248]: NETFILTER_CFG table=filter family=7 entries=0 op=xt_register pid=49248 subj=system_u:system_r:container_kvm_t:s0:c76,c489 comm="virtiofsd"
Dec 10 21:04:41 <snip_hostname> audit[49263]: AVC avc:  denied  { write } for  pid=49263 comm="virtiofsd" name="ROOT" dev="dm-1" ino=256 scontext=system_u:system_r:container_kvm_t:s0:c76,c489 tcontext=system_u:object_r:root_t:s0 tclass=dir permissive=0
Dec 10 21:04:41 <snip_hostname> kata-runtime[49239]: time="2020-12-10T21:04:41.02860386+01:00" level=info msg="virtiofsd quits" arch=amd64 command=create container=e3a7bc95faf2a210423f0b385def1fcceea79b91a48cd65b12a412eb061677b6 name=kata-runtime pid=49239 source=virtcontainers subsystem=qemu
Dec 10 21:04:41 <snip_hostname> kata-runtime[49239]: time="2020-12-10T21:04:41.028864822+01:00" level=info msg="Stopping Sandbox" arch=amd64 command=create container=e3a7bc95faf2a210423f0b385def1fcceea79b91a48cd65b12a412eb061677b6 name=kata-runtime pid=49239 source=virtcontainers subsystem=qemu
Dec 10 21:04:41 <snip_hostname> kata-runtime[49239]: time="2020-12-10T21:04:41.029562092+01:00" level=info msg="{\"QMP\": {\"version\": {\"qemu\": {\"micro\": 0, \"minor\": 1, \"major\": 5}, \"package\": \"qemu-5.1.0-5.fc33\"}, \"capabilities\": [\"oob\"]}}" arch=amd64 command=create container=e3a7bc95faf2a210423f0b385def1fcceea79b91a48cd65b12a412eb061677b6 name=kata-runtime pid=49239 source=virtcontainers subsystem=qmp
Dec 10 21:04:41 <snip_hostname> kata-runtime[49239]: time="2020-12-10T21:04:41.02981872+01:00" level=info msg="{\"execute\":\"qmp_capabilities\"}" arch=amd64 command=create container=e3a7bc95faf2a210423f0b385def1fcceea79b91a48cd65b12a412eb061677b6 name=kata-runtime pid=49239 source=virtcontainers subsystem=qmp
Dec 10 21:04:41 <snip_hostname> audit[46877]: NETFILTER_CFG table=filter family=7 entries=0 op=xt_unregister pid=46877 subj=system_u:system_r:kernel_t:s0 comm="kworker/u4:4"
Dec 10 21:04:41 <snip_hostname> kata-runtime[49239]: time="2020-12-10T21:04:41.048232229+01:00" level=info msg="sanner return error: read unix @->/run/vc/vm/e3a7bc95faf2a210423f0b385def1fcceea79b91a48cd65b12a412eb061677b6/qmp.sock: read: connection reset by peer" arch=amd64 command=create container=e3a7bc95faf2a210423f0b385def1fcceea79b91a48cd65b12a412eb061677b6 name=kata-runtime pid=49239 source=virtcontainers subsystem=qmp
Dec 10 21:04:41 <snip_hostname> kata-runtime[49239]: time="2020-12-10T21:04:41.048314325+01:00" level=error msg="Failed to negoatiate QMP capabilities" arch=amd64 command=create container=e3a7bc95faf2a210423f0b385def1fcceea79b91a48cd65b12a412eb061677b6 error="exitting QMP loop, command cancelled" name=kata-runtime pid=49239 source=virtcontainers subsystem=qemu
Dec 10 21:04:41 <snip_hostname> kata-runtime[49239]: time="2020-12-10T21:04:41.04835637+01:00" level=info msg="cleanup vm path" arch=amd64 command=create container=e3a7bc95faf2a210423f0b385def1fcceea79b91a48cd65b12a412eb061677b6 dir=/run/vc/vm/e3a7bc95faf2a210423f0b385def1fcceea79b91a48cd65b12a412eb061677b6 link=/run/vc/vm/e3a7bc95faf2a210423f0b385def1fcceea79b91a48cd65b12a412eb061677b6 name=kata-runtime pid=49239 source=virtcontainers subsystem=qemu
Dec 10 21:04:41 <snip_hostname> kata-runtime[49239]: time="2020-12-10T21:04:41.0497181+01:00" level=error msg="Unable to launch /usr/bin/qemu-kvm: exit status 1" arch=amd64 command=create container=e3a7bc95faf2a210423f0b385def1fcceea79b91a48cd65b12a412eb061677b6 name=kata-runtime pid=49239 source=virtcontainers subsystem=qmp
Dec 10 21:04:41 <snip_hostname> kata-runtime[49239]: time="2020-12-10T21:04:41.049759324+01:00" level=error msg="qemu-system-x86_64: -device vhost-vsock-pci,disable-modern=false,vhostfd=3,id=vsock-1696009078,guest-cid=1696009078,romfile=: device is modern-only, use disable-legacy=on\n" arch=amd64 command=create container=e3a7bc95faf2a210423f0b385def1fcceea79b91a48cd65b12a412eb061677b6 name=kata-runtime pid=49239 source=virtcontainers subsystem=qmp
Dec 10 21:04:41 <snip_hostname> kata-runtime[49239]: time="2020-12-10T21:04:41.049782802+01:00" level=error msg="failed to launch qemu: qemu-system-x86_64: -device vhost-vsock-pci,disable-modern=false,vhostfd=3,id=vsock-1696009078,guest-cid=1696009078,romfile=: device is modern-only, use disable-legacy=on\n" arch=amd64 command=create container=e3a7bc95faf2a210423f0b385def1fcceea79b91a48cd65b12a412eb061677b6 error="exit status 1" name=kata-runtime pid=49239 source=virtcontainers subsystem=qemu
Dec 10 21:04:41 <snip_hostname> kata-runtime[49239]: time="2020-12-10T21:04:41.053383897+01:00" level=info msg="Detaching endpoint" arch=amd64 command=create container=e3a7bc95faf2a210423f0b385def1fcceea79b91a48cd65b12a412eb061677b6 endpoint-type=virtual name=kata-runtime pid=49239 source=virtcontainers subsystem=network
Dec 10 21:04:41 <snip_hostname> kata-runtime[49239]: time="2020-12-10T21:04:41.053474819+01:00" level=error msg="Could not move processes into parent cgroup" arch=amd64 command=create container=e3a7bc95faf2a210423f0b385def1fcceea79b91a48cd65b12a412eb061677b6 error="write /sys/fs/cgroup/machine.slice/cgroup.procs: device or resource busy" name=kata-runtime pid=49239 source=cgroup-manager
Dec 10 21:04:41 <snip_hostname> systemd[1]: Stopping libcontainer container e3a7bc95faf2a210423f0b385def1fcceea79b91a48cd65b12a412eb061677b6.
Dec 10 21:04:41 <snip_hostname> systemd[1]: libpod-e3a7bc95faf2a210423f0b385def1fcceea79b91a48cd65b12a412eb061677b6.scope: Succeeded.
Dec 10 21:04:41 <snip_hostname> systemd[1]: Stopped libcontainer container e3a7bc95faf2a210423f0b385def1fcceea79b91a48cd65b12a412eb061677b6.
Dec 10 21:04:41 <snip_hostname> audit: BPF prog-id=597 op=UNLOAD
Dec 10 21:04:41 <snip_hostname> kata-runtime[49272]: time="2020-12-10T21:04:41.107450811+01:00" level=info msg="loaded configuration" arch=amd64 command=delete file=/usr/share/kata-containers/defaults/configuration.toml format=TOML name=kata-runtime pid=49272 source=katautils
Dec 10 21:04:41 <snip_hostname> kata-runtime[49272]: time="2020-12-10T21:04:41.108161856+01:00" level=info msg="vsock supported" arch=amd64 command=delete name=kata-runtime pid=49272 source=katautils
Dec 10 21:04:41 <snip_hostname> kata-runtime[49272]: time="2020-12-10T21:04:41.108320849+01:00" level=info msg="IOMMUPlatform is disabled by default." arch=amd64 command=delete name=kata-runtime pid=49272 source=katautils
Dec 10 21:04:41 <snip_hostname> kata-runtime[49272]: time="2020-12-10T21:04:41.108458677+01:00" level=info msg="VSOCK supported, configure to not use proxy" arch=amd64 command=delete name=kata-runtime pid=49272 source=katautils
Dec 10 21:04:41 <snip_hostname> kata-runtime[49272]: time="2020-12-10T21:04:41.108572188+01:00" level=info arch=amd64 arguments="\"delete --force e3a7bc95faf2a210423f0b385def1fcceea79b91a48cd65b12a412eb061677b6\"" command=delete commit= name=kata-runtime pid=49272 source=runtime version=1.12.0
Dec 10 21:04:41 <snip_hostname> kata-runtime[49272]: time="2020-12-10T21:04:41.108778502+01:00" level=warning msg="Failed to get container, force will not fail: Container ID (e3a7bc95faf2a210423f0b385def1fcceea79b91a48cd65b12a412eb061677b6) does not exist" arch=amd64 command=delete container=e3a7bc95faf2a210423f0b385def1fcceea79b91a48cd65b12a412eb061677b6 name=kata-runtime pid=49272 source=runtime
Dec 10 21:04:41 <snip_hostname> kernel: cni_dmz_inet: port 1(veth72bb2dc4) entered disabled state
Dec 10 21:04:41 <snip_hostname> kernel: device veth72bb2dc4 left promiscuous mode
Dec 10 21:04:41 <snip_hostname> kernel: cni_dmz_inet: port 1(veth72bb2dc4) entered disabled state
Dec 10 21:04:41 <snip_hostname> audit: ANOM_PROMISCUOUS dev=veth72bb2dc4 prom=0 old_prom=256 auid=1000 uid=0 gid=0 ses=593
Dec 10 21:04:41 <snip_hostname> NetworkManager[9490]: <info>  [1607630681.1358] device (veth72bb2dc4): released from master device cni_dmz_inet
Dec 10 21:04:41 <snip_hostname> systemd[1]: run-netns-cni\x2deb86aa69\x2dbcd1\x2d410f\x2dc591\x2d10ffc55f1872.mount: Succeeded.
Dec 10 21:04:41 <snip_hostname> systemd[48826]: run-netns-cni\x2deb86aa69\x2dbcd1\x2d410f\x2dc591\x2d10ffc55f1872.mount: Succeeded.
Dec 10 21:04:41 <snip_hostname> audit[45198]: NETFILTER_CFG table=filter family=7 entries=0 op=xt_unregister pid=45198 subj=system_u:system_r:kernel_t:s0 comm="kworker/u4:5"
Dec 10 21:04:41 <snip_hostname> systemd[48826]: var-lib-containers-storage-overlay\x2dcontainers-e3a7bc95faf2a210423f0b385def1fcceea79b91a48cd65b12a412eb061677b6-userdata-shm.mount: Succeeded.
Dec 10 21:04:41 <snip_hostname> systemd[1]: var-lib-containers-storage-overlay\x2dcontainers-e3a7bc95faf2a210423f0b385def1fcceea79b91a48cd65b12a412eb061677b6-userdata-shm.mount: Succeeded.
Dec 10 21:04:41 <snip_hostname> systemd[1]: var-lib-containers-storage-overlay-91253345f6986378d16f0310beea760936e55257e3be8c17a7e773ae7d33e27c-merged.mount: Succeeded.
Dec 10 21:04:41 <snip_hostname> systemd[48826]: var-lib-containers-storage-overlay-91253345f6986378d16f0310beea760936e55257e3be8c17a7e773ae7d33e27c-merged.mount: Succeeded.
Dec 10 21:04:41 <snip_hostname> kata-runtime[49358]: time="2020-12-10T21:04:41.587265059+01:00" level=info msg="loaded configuration" arch=amd64 command=delete file=/usr/share/kata-containers/defaults/configuration.toml format=TOML name=kata-runtime pid=49358 source=katautils
Dec 10 21:04:41 <snip_hostname> kata-runtime[49358]: time="2020-12-10T21:04:41.587550712+01:00" level=info msg="vsock supported" arch=amd64 command=delete name=kata-runtime pid=49358 source=katautils
Dec 10 21:04:41 <snip_hostname> kata-runtime[49358]: time="2020-12-10T21:04:41.58758042+01:00" level=info msg="IOMMUPlatform is disabled by default." arch=amd64 command=delete name=kata-runtime pid=49358 source=katautils
Dec 10 21:04:41 <snip_hostname> kata-runtime[49358]: time="2020-12-10T21:04:41.587631991+01:00" level=info msg="VSOCK supported, configure to not use proxy" arch=amd64 command=delete name=kata-runtime pid=49358 source=katautils
Dec 10 21:04:41 <snip_hostname> kata-runtime[49358]: time="2020-12-10T21:04:41.587675685+01:00" level=info arch=amd64 arguments="\"delete --force e3a7bc95faf2a210423f0b385def1fcceea79b91a48cd65b12a412eb061677b6\"" command=delete commit= name=kata-runtime pid=49358 source=runtime version=1.12.0
Dec 10 21:04:41 <snip_hostname> kata-runtime[49358]: time="2020-12-10T21:04:41.587788485+01:00" level=warning msg="Failed to get container, force will not fail: Container ID (e3a7bc95faf2a210423f0b385def1fcceea79b91a48cd65b12a412eb061677b6) does not exist" arch=amd64 command=delete container=e3a7bc95faf2a210423f0b385def1fcceea79b91a48cd65b12a412eb061677b6 name=kata-runtime pid=49358 source=runtime
Dec 10 21:04:41 <snip_hostname> podman[49160]: 2020-12-10 21:04:41.622772193 +0100 CET m=+2.245342337 container remove e3a7bc95faf2a210423f0b385def1fcceea79b91a48cd65b12a412eb061677b6 (image=docker.io/library/alpine:latest, name=musing_faraday)
Dec 10 21:04:41 <snip_hostname> systemd[1]: libpod-conmon-e3a7bc95faf2a210423f0b385def1fcceea79b91a48cd65b12a412eb061677b6.scope: Succeeded.

Dec 10 21:04:41 <snip_hostname> unknown: ######END Contianer#####

Description of problem

After a upgrade of kata-runtime or Podman I am unable to start a container with kata-runtime with F32-server.

I hoped that upgrading to Fedora 33 solve the problem but it did not. I also tried to remove and reinstall kata component, but with the same issues.

Command: podman run --rm --runtime=kata --network=host -it alpine /bin/sh Error: Error: container create failed (no logs from conmon): EOF

This command works fine. podman run --rm --runtime=crun --network=host -it alpine /bin/sh

I wonder what I am doing wrong.

kata-containers / runtime

Fedora 32/33: Can't start a container anymore after update of kata or podman. #3104

Description of problem