search

kata-containers / runtime

Kata Containers version 1.x runtime (for version 2.x see https://github.com/kata-containers/kata-containers).
https://katacontainers.io/
Apache License 2.0
2.1k stars 375 forks source link

Failed to run a container with kata-fc #1689

Closed YLonely closed 3 years ago

YLonely commented 5 years ago

Description of problem

I wanna use kata-fc to run a container, here is the command i use: /opt/kata/bin/kata-fc run test and here is the content of config.json:

{
    "ociVersion": "1.0.1-dev",
    "process": {
        "terminal": true,
        "user": {
            "uid": 0,
            "gid": 0
        },
        "args": [
            "date",
            "-u",
            "+%-s:%-N"
        ],
        "env": [
            "PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin",
            "TERM=xterm"
        ],
        "cwd": "/",
        "capabilities": {
            "bounding": [
                "CAP_AUDIT_WRITE",
                "CAP_KILL",
                "CAP_NET_BIND_SERVICE"
            ],
            "effective": [
                "CAP_AUDIT_WRITE",
                "CAP_KILL",
                "CAP_NET_BIND_SERVICE"
            ],
            "inheritable": [
                "CAP_AUDIT_WRITE",
                "CAP_KILL",
                "CAP_NET_BIND_SERVICE"
            ],
            "permitted": [
                "CAP_AUDIT_WRITE",
                "CAP_KILL",
                "CAP_NET_BIND_SERVICE"
            ],
            "ambient": [
                "CAP_AUDIT_WRITE",
                "CAP_KILL",
                "CAP_NET_BIND_SERVICE"
            ]
        },
        "rlimits": [
            {
                "type": "RLIMIT_NOFILE",
                "hard": 1024,
                "soft": 1024
            }
        ],
        "noNewPrivileges": true
    },
    "root": {
        "path": "rootfs",
        "readonly": true
    },
    "hostname": "runc",
    "mounts": [
        {
            "destination": "/proc",
            "type": "proc",
            "source": "proc"
        },
        {
            "destination": "/dev",
            "type": "tmpfs",
            "source": "tmpfs",
            "options": [
                "nosuid",
                "strictatime",
                "mode=755",
                "size=65536k"
            ]
        },
        {
            "destination": "/dev/pts",
            "type": "devpts",
            "source": "devpts",
            "options": [
                "nosuid",
                "noexec",
                "newinstance",
                "ptmxmode=0666",
                "mode=0620",
                "gid=5"
            ]
        },
        {
            "destination": "/dev/shm",
            "type": "tmpfs",
            "source": "shm",
            "options": [
                "nosuid",
                "noexec",
                "nodev",
                "mode=1777",
                "size=65536k"
            ]
        },
        {
            "destination": "/dev/mqueue",
            "type": "mqueue",
            "source": "mqueue",
            "options": [
                "nosuid",
                "noexec",
                "nodev"
            ]
        },
        {
            "destination": "/sys",
            "type": "sysfs",
            "source": "sysfs",
            "options": [
                "nosuid",
                "noexec",
                "nodev",
                "ro"
            ]
        },
        {
            "destination": "/sys/fs/cgroup",
            "type": "cgroup",
            "source": "cgroup",
            "options": [
                "nosuid",
                "noexec",
                "nodev",
                "relatime",
                "ro"
            ]
        }
    ],
    "linux": {
        "resources": {
            "devices": [
                {
                    "allow": false,
                    "access": "rwm"
                }
            ]
        },
        "namespaces": [
            {
                "type": "pid"
            },
            {
                "type": "network"
            },
            {
                "type": "ipc"
            },
            {
                "type": "uts"
            },
            {
                "type": "mount"
            }
        ],
        "maskedPaths": [
            "/proc/kcore",
            "/proc/latency_stats",
            "/proc/timer_list",
            "/proc/timer_stats",
            "/proc/sched_debug",
            "/sys/firmware"
        ],
        "readonlyPaths": [
            "/proc/asound",
            "/proc/bus",
            "/proc/fs",
            "/proc/irq",
            "/proc/sys",
            "/proc/sysrq-trigger"
        ]
    }
}

Expected result

current time will be printed since i use date as the args in config.json

Actual result

actually it goes like this:

rpc error: code = Unknown desc = rootfs (/run/kata-containers/shared/containers/test/rootfs) does not exist

even though i created a rootfs dir in the path mentioned above(/run/kata-containers/shared/containers/test/)

Show kata-collect-data.sh details

# Meta details Running `kata-collect-data.sh` version `1.7.0-rc1 (commit bce1167c33e3a43cd194e5d06cc125cb053c27b5)` at `2019-05-16.13:02:06.300692778+0800`. --- Runtime is `/usr/local/bin/kata-runtime`. # `kata-env` Output of "`/usr/local/bin/kata-runtime kata-env`": ```toml /etc/kata-containers/configuration.toml: file /root/vm-container/time-test/kata-fc/sr/share/kata-containers/vmlinux-fc-config does not exist ``` --- # Runtime config files ## Runtime default config files ``` /etc/kata-containers/configuration.toml /usr/share/defaults/kata-containers/configuration.toml ``` ## Runtime config file contents Output of "`cat "/etc/kata-containers/configuration.toml"`": ```toml # Copyright (c) 2017-2019 Intel Corporation # # SPDX-License-Identifier: Apache-2.0 # # XXX: WARNING: this file is auto-generated. # XXX: # XXX: Source file: "cli/config/configuration-qemu.toml.in" # XXX: Project: # XXX: Name: Kata Containers # XXX: Type: kata [hypervisor.qemu] path = "/usr/bin/qemu-lite-system-x86_64" kernel = "sr/share/kata-containers/vmlinux-fc-config" initrd = "/usr/share/kata-containers/kata-containers-initrd.img" machine_type = "pc" # Optional space-separated list of options to pass to the guest kernel. # For example, use `kernel_params = "vsyscall=emulate"` if you are having # trouble running pre-2.15 glibc. # # WARNING: - any parameter specified here will take priority over the default # parameter value of the same name used to start the virtual machine. # Do not set values here unless you understand the impact of doing so as you # may stop the virtual machine from booting. # To see the list of default parameters, enable hypervisor debug, create a # container and look for 'default-kernel-parameters' log entries. kernel_params = "ipv6.disable=1 agent.log=debug initcall_debug" # Path to the firmware. # If you want that qemu uses the default firmware leave this option empty firmware = "" # Machine accelerators # comma-separated list of machine accelerators to pass to the hypervisor. # For example, `machine_accelerators = "nosmm,nosmbus,nosata,nopit,static-prt,nofw"` machine_accelerators="" # Default number of vCPUs per SB/VM: # unspecified or 0 --> will be set to 1 # < 0 --> will be set to the actual number of physical cores # > 0 <= number of physical cores --> will be set to the specified number # > number of physical cores --> will be set to the actual number of physical cores default_vcpus = 1 # Default maximum number of vCPUs per SB/VM: # unspecified or == 0 --> will be set to the actual number of physical cores or to the maximum number # of vCPUs supported by KVM if that number is exceeded # > 0 <= number of physical cores --> will be set to the specified number # > number of physical cores --> will be set to the actual number of physical cores or to the maximum number # of vCPUs supported by KVM if that number is exceeded # WARNING: Depending of the architecture, the maximum number of vCPUs supported by KVM is used when # the actual number of physical cores is greater than it. # WARNING: Be aware that this value impacts the virtual machine's memory footprint and CPU # the hotplug functionality. For example, `default_maxvcpus = 240` specifies that until 240 vCPUs # can be added to a SB/VM, but the memory footprint will be big. Another example, with # `default_maxvcpus = 8` the memory footprint will be small, but 8 will be the maximum number of # vCPUs supported by the SB/VM. In general, we recommend that you do not edit this variable, # unless you know what are you doing. default_maxvcpus = 0 # Bridges can be used to hot plug devices. # Limitations: # * Currently only pci bridges are supported # * Until 30 devices per bridge can be hot plugged. # * Until 5 PCI bridges can be cold plugged per VM. # This limitation could be a bug in qemu or in the kernel # Default number of bridges per SB/VM: # unspecified or 0 --> will be set to 1 # > 1 <= 5 --> will be set to the specified number # > 5 --> will be set to 5 default_bridges = 1 # Default memory size in MiB for SB/VM. # If unspecified then it will be set 2048 MiB. default_memory = 2048 # # Default memory slots per SB/VM. # If unspecified then it will be set 10. # This is will determine the times that memory will be hotadded to sandbox/VM. #memory_slots = 10 # The size in MiB will be plused to max memory of hypervisor. # It is the memory address space for the NVDIMM devie. # If set block storage driver (block_device_driver) to "nvdimm", # should set memory_offset to the size of block device. # Default 0 #memory_offset = 0 # Disable block device from being used for a container's rootfs. # In case of a storage driver like devicemapper where a container's # root file system is backed by a block device, the block device is passed # directly to the hypervisor for performance reasons. # This flag prevents the block device from being passed to the hypervisor, # 9pfs is used instead to pass the rootfs. disable_block_device_use = false # Block storage driver to be used for the hypervisor in case the container # rootfs is backed by a block device. This is virtio-scsi, virtio-blk # or nvdimm. block_device_driver = "virtio-scsi" # Specifies cache-related options will be set to block devices or not. # Default false #block_device_cache_set = true # Specifies cache-related options for block devices. # Denotes whether use of O_DIRECT (bypass the host page cache) is enabled. # Default false #block_device_cache_direct = true # Specifies cache-related options for block devices. # Denotes whether flush requests for the device are ignored. # Default false #block_device_cache_noflush = true # Enable iothreads (data-plane) to be used. This causes IO to be # handled in a separate IO thread. This is currently only implemented # for SCSI. # enable_iothreads = false # Enable pre allocation of VM RAM, default false # Enabling this will result in lower container density # as all of the memory will be allocated and locked # This is useful when you want to reserve all the memory # upfront or in the cases where you want memory latencies # to be very predictable # Default false #enable_mem_prealloc = true # Enable huge pages for VM RAM, default false # Enabling this will result in the VM memory # being allocated using huge pages. # This is useful when you want to use vhost-user network # stacks within the container. This will automatically # result in memory pre allocation #enable_hugepages = true # Enable swap of vm memory. Default false. # The behaviour is undefined if mem_prealloc is also set to true #enable_swap = true # This option changes the default hypervisor and kernel parameters # to enable debug output where available. This extra output is added # to the proxy logs, but only when proxy debug is also enabled. # # Default false enable_debug = true # Disable the customizations done in the runtime when it detects # that it is running on top a VMM. This will result in the runtime # behaving as it would when running on bare metal. # #disable_nesting_checks = true # This is the msize used for 9p shares. It is the number of bytes # used for 9p packet payload. #msize_9p = 8192 # If true and vsocks are supported, use vsocks to communicate directly # with the agent and no proxy is started, otherwise use unix # sockets and start a proxy to communicate with the agent. # Default false #use_vsock = true # VFIO devices are hotplugged on a bridge by default. # Enable hotplugging on root bus. This may be required for devices with # a large PCI bar, as this is a current limitation with hotplugging on # a bridge. This value is valid for "pc" machine type. # Default false #hotplug_vfio_on_root_bus = true # If host doesn't support vhost_net, set to true. Thus we won't create vhost fds for nics. # Default false #disable_vhost_net = true # # Default entropy source. # The path to a host source of entropy (including a real hardware RNG) # /dev/urandom and /dev/random are two main options. # Be aware that /dev/random is a blocking source of entropy. If the host # runs out of entropy, the VMs boot time will increase leading to get startup # timeouts. # The source of entropy /dev/urandom is non-blocking and provides a # generally acceptable source of entropy. It should work well for pretty much # all practical purposes. #entropy_source= "/dev/urandom" # Path to OCI hook binaries in the *guest rootfs*. # This does not affect host-side hooks which must instead be added to # the OCI spec passed to the runtime. # # You can create a rootfs with hooks by customizing the osbuilder scripts: # https://github.com/kata-containers/osbuilder # # Hooks must be stored in a subdirectory of guest_hook_path according to their # hook type, i.e. "guest_hook_path/{prestart,postart,poststop}". # The agent will scan these directories for executable files and add them, in # lexicographical order, to the lifecycle of the guest container. # Hooks are executed in the runtime namespace of the guest. See the official documentation: # https://github.com/opencontainers/runtime-spec/blob/v1.0.1/config.md#posix-platform-hooks # Warnings will be logged if any error is encountered will scanning for hooks, # but it will not abort container execution. #guest_hook_path = "/usr/share/oci/hooks" [factory] # VM templating support. Once enabled, new VMs are created from template # using vm cloning. They will share the same initial kernel, initramfs and # agent memory by mapping it readonly. It helps speeding up new container # creation and saves a lot of memory if there are many kata containers running # on the same host. # # When disabled, new VMs are created from scratch. # # Note: Requires "initrd=" to be set ("image=" is not supported). # # Default false enable_template = true # The number of caches of VMCache: # unspecified or == 0 --> VMCache is disabled # > 0 --> will be set to the specified number # # VMCache is a function that creates VMs as caches before using it. # It helps speed up new container creation. # The function consists of a server and some clients communicating # through Unix socket. The protocol is gRPC in protocols/cache/cache.proto. # The VMCache server will create some VMs and cache them by factory cache. # It will convert the VM to gRPC format and transport it when gets # requestion from clients. # Factory grpccache is the VMCache client. It will request gRPC format # VM and convert it back to a VM. If VMCache function is enabled, # kata-runtime will request VM from factory grpccache when it creates # a new sandbox. # # Default 0 #vm_cache_number = 0 # Specify the address of the Unix socket that is used by VMCache. # # Default /var/run/kata-containers/cache.sock #vm_cache_endpoint = "/var/run/kata-containers/cache.sock" [proxy.kata] path = "/usr/libexec/kata-containers/kata-proxy" # If enabled, proxy messages will be sent to the system log # (default: disabled) enable_debug = true [shim.kata] path = "/usr/libexec/kata-containers/kata-shim" # If enabled, shim messages will be sent to the system log # (default: disabled) enable_debug = true # If enabled, the shim will create opentracing.io traces and spans. # (See https://www.jaegertracing.io/docs/getting-started). # # Note: By default, the shim runs in a separate network namespace. Therefore, # to allow it to send trace details to the Jaeger agent running on the host, # it is necessary to set 'disable_new_netns=true' so that it runs in the host # network namespace. # # (default: disabled) #enable_tracing = true [agent.kata] # There is no field for this section. The goal is only to be able to # specify which type of agent the user wants to use. [netmon] # If enabled, the network monitoring process gets started when the # sandbox is created. This allows for the detection of some additional # network being added to the existing network namespace, after the # sandbox has been created. # (default: disabled) #enable_netmon = true # Specify the path to the netmon binary. path = "/usr/libexec/kata-containers/kata-netmon" # If enabled, netmon messages will be sent to the system log # (default: disabled) enable_debug = true [runtime] # If enabled, the runtime will log additional debug messages to the # system log # (default: disabled) enable_debug = true # # Internetworking model # Determines how the VM should be connected to the # the container network interface # Options: # # - bridged # Uses a linux bridge to interconnect the container interface to # the VM. Works for most cases except macvlan and ipvlan. # # - macvtap # Used when the Container network interface can be bridged using # macvtap. # # - none # Used when customize network. Only creates a tap device. No veth pair. # # - tcfilter # Uses tc filter rules to redirect traffic from the network interface # provided by plugin to a tap interface connected to the VM. # internetworking_model="tcfilter" # disable guest seccomp # Determines whether container seccomp profiles are passed to the virtual # machine and applied by the kata agent. If set to true, seccomp is not applied # within the guest # (default: true) disable_guest_seccomp=true # If enabled, the runtime will create opentracing.io traces and spans. # (See https://www.jaegertracing.io/docs/getting-started). # (default: disabled) #enable_tracing = true # If enabled, the runtime will not create a network namespace for shim and hypervisor processes. # This option may have some potential impacts to your host. It should only be used when you know what you're doing. # `disable_new_netns` conflicts with `enable_netmon` # `disable_new_netns` conflicts with `internetworking_model=bridged` and `internetworking_model=macvtap`. It works only # with `internetworking_model=none`. The tap device will be in the host network namespace and can connect to a bridge # (like OVS) directly. # If you are using docker, `disable_new_netns` only works with `docker run --net=none` # (default: false) #disable_new_netns = true # Enabled experimental feature list, format: ["a", "b"]. # Experimental features are features not stable enough for production, # They may break compatibility, and are prepared for a big version bump. # Supported experimental features: # 1. "newstore": new persist storage driver which breaks backward compatibility, # expected to move out of experimental in 2.0.0. # (default: []) experimental=[] ``` Output of "`cat "/opt/kata/share/defaults/kata-containers/configuration.toml"`": ```toml # Copyright (c) 2017-2019 Intel Corporation # # SPDX-License-Identifier: Apache-2.0 # # XXX: WARNING: this file is auto-generated. # XXX: # XXX: Source file: "cli/config/configuration-qemu.toml.in" # XXX: Project: # XXX: Name: Kata Containers # XXX: Type: kata [hypervisor.qemu] path = "/opt/kata/bin/qemu-system-x86_64" kernel = "/opt/kata/share/kata-containers/vmlinuz.container" image = "/opt/kata/share/kata-containers/kata-containers.img" machine_type = "pc" # Optional space-separated list of options to pass to the guest kernel. # For example, use `kernel_params = "vsyscall=emulate"` if you are having # trouble running pre-2.15 glibc. # # WARNING: - any parameter specified here will take priority over the default # parameter value of the same name used to start the virtual machine. # Do not set values here unless you understand the impact of doing so as you # may stop the virtual machine from booting. # To see the list of default parameters, enable hypervisor debug, create a # container and look for 'default-kernel-parameters' log entries. kernel_params = "" # Path to the firmware. # If you want that qemu uses the default firmware leave this option empty firmware = "" # Machine accelerators # comma-separated list of machine accelerators to pass to the hypervisor. # For example, `machine_accelerators = "nosmm,nosmbus,nosata,nopit,static-prt,nofw"` machine_accelerators="" # Default number of vCPUs per SB/VM: # unspecified or 0 --> will be set to 1 # < 0 --> will be set to the actual number of physical cores # > 0 <= number of physical cores --> will be set to the specified number # > number of physical cores --> will be set to the actual number of physical cores default_vcpus = 1 # Default maximum number of vCPUs per SB/VM: # unspecified or == 0 --> will be set to the actual number of physical cores or to the maximum number # of vCPUs supported by KVM if that number is exceeded # > 0 <= number of physical cores --> will be set to the specified number # > number of physical cores --> will be set to the actual number of physical cores or to the maximum number # of vCPUs supported by KVM if that number is exceeded # WARNING: Depending of the architecture, the maximum number of vCPUs supported by KVM is used when # the actual number of physical cores is greater than it. # WARNING: Be aware that this value impacts the virtual machine's memory footprint and CPU # the hotplug functionality. For example, `default_maxvcpus = 240` specifies that until 240 vCPUs # can be added to a SB/VM, but the memory footprint will be big. Another example, with # `default_maxvcpus = 8` the memory footprint will be small, but 8 will be the maximum number of # vCPUs supported by the SB/VM. In general, we recommend that you do not edit this variable, # unless you know what are you doing. default_maxvcpus = 0 # Bridges can be used to hot plug devices. # Limitations: # * Currently only pci bridges are supported # * Until 30 devices per bridge can be hot plugged. # * Until 5 PCI bridges can be cold plugged per VM. # This limitation could be a bug in qemu or in the kernel # Default number of bridges per SB/VM: # unspecified or 0 --> will be set to 1 # > 1 <= 5 --> will be set to the specified number # > 5 --> will be set to 5 default_bridges = 1 # Default memory size in MiB for SB/VM. # If unspecified then it will be set 2048 MiB. default_memory = 2048 # # Default memory slots per SB/VM. # If unspecified then it will be set 10. # This is will determine the times that memory will be hotadded to sandbox/VM. #memory_slots = 10 # The size in MiB will be plused to max memory of hypervisor. # It is the memory address space for the NVDIMM devie. # If set block storage driver (block_device_driver) to "nvdimm", # should set memory_offset to the size of block device. # Default 0 #memory_offset = 0 # Disable block device from being used for a container's rootfs. # In case of a storage driver like devicemapper where a container's # root file system is backed by a block device, the block device is passed # directly to the hypervisor for performance reasons. # This flag prevents the block device from being passed to the hypervisor, # 9pfs is used instead to pass the rootfs. disable_block_device_use = false # Shared file system type: # - virtio-9p (default) # - virtio-fs shared_fs = "virtio-9p" # Path to vhost-user-fs daemon. virtio_fs_daemon = "/opt/kata/bin/virtiofsd" # Default size of DAX cache in MiB virtio_fs_cache_size = 8192 # Cache mode: # # - none # Metadata, data, and pathname lookup are not cached in guest. They are # always fetched from host and any changes are immediately pushed to host. # # - auto # Metadata and pathname lookup cache expires after a configured amount of # time (default is 1 second). Data is cached while the file is open (close # to open consistency). # # - always # Metadata, data, and pathname lookup are cached in guest and never expire. virtio_fs_cache = "always" # Block storage driver to be used for the hypervisor in case the container # rootfs is backed by a block device. This is virtio-scsi, virtio-blk # or nvdimm. block_device_driver = "virtio-scsi" # Specifies cache-related options will be set to block devices or not. # Default false #block_device_cache_set = true # Specifies cache-related options for block devices. # Denotes whether use of O_DIRECT (bypass the host page cache) is enabled. # Default false #block_device_cache_direct = true # Specifies cache-related options for block devices. # Denotes whether flush requests for the device are ignored. # Default false #block_device_cache_noflush = true # Enable iothreads (data-plane) to be used. This causes IO to be # handled in a separate IO thread. This is currently only implemented # for SCSI. # enable_iothreads = false # Enable pre allocation of VM RAM, default false # Enabling this will result in lower container density # as all of the memory will be allocated and locked # This is useful when you want to reserve all the memory # upfront or in the cases where you want memory latencies # to be very predictable # Default false #enable_mem_prealloc = true # Enable huge pages for VM RAM, default false # Enabling this will result in the VM memory # being allocated using huge pages. # This is useful when you want to use vhost-user network # stacks within the container. This will automatically # result in memory pre allocation #enable_hugepages = true # Enable swap of vm memory. Default false. # The behaviour is undefined if mem_prealloc is also set to true #enable_swap = true # This option changes the default hypervisor and kernel parameters # to enable debug output where available. This extra output is added # to the proxy logs, but only when proxy debug is also enabled. # # Default false #enable_debug = true # Disable the customizations done in the runtime when it detects # that it is running on top a VMM. This will result in the runtime # behaving as it would when running on bare metal. # #disable_nesting_checks = true # This is the msize used for 9p shares. It is the number of bytes # used for 9p packet payload. #msize_9p = 8192 # If true and vsocks are supported, use vsocks to communicate directly # with the agent and no proxy is started, otherwise use unix # sockets and start a proxy to communicate with the agent. # Default false #use_vsock = true # VFIO devices are hotplugged on a bridge by default. # Enable hotplugging on root bus. This may be required for devices with # a large PCI bar, as this is a current limitation with hotplugging on # a bridge. This value is valid for "pc" machine type. # Default false #hotplug_vfio_on_root_bus = true # If host doesn't support vhost_net, set to true. Thus we won't create vhost fds for nics. # Default false #disable_vhost_net = true # # Default entropy source. # The path to a host source of entropy (including a real hardware RNG) # /dev/urandom and /dev/random are two main options. # Be aware that /dev/random is a blocking source of entropy. If the host # runs out of entropy, the VMs boot time will increase leading to get startup # timeouts. # The source of entropy /dev/urandom is non-blocking and provides a # generally acceptable source of entropy. It should work well for pretty much # all practical purposes. #entropy_source= "/dev/urandom" # Path to OCI hook binaries in the *guest rootfs*. # This does not affect host-side hooks which must instead be added to # the OCI spec passed to the runtime. # # You can create a rootfs with hooks by customizing the osbuilder scripts: # https://github.com/kata-containers/osbuilder # # Hooks must be stored in a subdirectory of guest_hook_path according to their # hook type, i.e. "guest_hook_path/{prestart,postart,poststop}". # The agent will scan these directories for executable files and add them, in # lexicographical order, to the lifecycle of the guest container. # Hooks are executed in the runtime namespace of the guest. See the official documentation: # https://github.com/opencontainers/runtime-spec/blob/v1.0.1/config.md#posix-platform-hooks # Warnings will be logged if any error is encountered will scanning for hooks, # but it will not abort container execution. #guest_hook_path = "/usr/share/oci/hooks" [factory] # VM templating support. Once enabled, new VMs are created from template # using vm cloning. They will share the same initial kernel, initramfs and # agent memory by mapping it readonly. It helps speeding up new container # creation and saves a lot of memory if there are many kata containers running # on the same host. # # When disabled, new VMs are created from scratch. # # Note: Requires "initrd=" to be set ("image=" is not supported). # # Default false #enable_template = true # Specifies the path of template. # # Default "/run/vc/vm/template" #template_path = "/run/vc/vm/template" # The number of caches of VMCache: # unspecified or == 0 --> VMCache is disabled # > 0 --> will be set to the specified number # # VMCache is a function that creates VMs as caches before using it. # It helps speed up new container creation. # The function consists of a server and some clients communicating # through Unix socket. The protocol is gRPC in protocols/cache/cache.proto. # The VMCache server will create some VMs and cache them by factory cache. # It will convert the VM to gRPC format and transport it when gets # requestion from clients. # Factory grpccache is the VMCache client. It will request gRPC format # VM and convert it back to a VM. If VMCache function is enabled, # kata-runtime will request VM from factory grpccache when it creates # a new sandbox. # # Default 0 #vm_cache_number = 0 # Specify the address of the Unix socket that is used by VMCache. # # Default /var/run/kata-containers/cache.sock #vm_cache_endpoint = "/var/run/kata-containers/cache.sock" [proxy.kata] path = "/opt/kata/libexec/kata-containers/kata-proxy" # If enabled, proxy messages will be sent to the system log # (default: disabled) #enable_debug = true [shim.kata] path = "/opt/kata/libexec/kata-containers/kata-shim" # If enabled, shim messages will be sent to the system log # (default: disabled) #enable_debug = true # If enabled, the shim will create opentracing.io traces and spans. # (See https://www.jaegertracing.io/docs/getting-started). # # Note: By default, the shim runs in a separate network namespace. Therefore, # to allow it to send trace details to the Jaeger agent running on the host, # it is necessary to set 'disable_new_netns=true' so that it runs in the host # network namespace. # # (default: disabled) #enable_tracing = true [agent.kata] # If enabled, make the agent display debug-level messages. # (default: disabled) #enable_debug = true # Enable agent tracing. # # If enabled, the default trace mode is "dynamic" and the # default trace type is "isolated". The trace mode and type are set # explicity with the `trace_type=` and `trace_mode=` options. # # Notes: # # - Tracing is ONLY enabled when `enable_tracing` is set: explicitly # setting `trace_mode=` and/or `trace_type=` without setting `enable_tracing` # will NOT activate agent tracing. # # - See https://github.com/kata-containers/agent/blob/master/TRACING.md for # full details. # # (default: disabled) #enable_tracing = true # #trace_mode = "dynamic" #trace_type = "isolated" [netmon] # If enabled, the network monitoring process gets started when the # sandbox is created. This allows for the detection of some additional # network being added to the existing network namespace, after the # sandbox has been created. # (default: disabled) #enable_netmon = true # Specify the path to the netmon binary. path = "/opt/kata/libexec/kata-containers/kata-netmon" # If enabled, netmon messages will be sent to the system log # (default: disabled) #enable_debug = true [runtime] # If enabled, the runtime will log additional debug messages to the # system log # (default: disabled) #enable_debug = true # # Internetworking model # Determines how the VM should be connected to the # the container network interface # Options: # # - bridged # Uses a linux bridge to interconnect the container interface to # the VM. Works for most cases except macvlan and ipvlan. # # - macvtap # Used when the Container network interface can be bridged using # macvtap. # # - none # Used when customize network. Only creates a tap device. No veth pair. # # - tcfilter # Uses tc filter rules to redirect traffic from the network interface # provided by plugin to a tap interface connected to the VM. # internetworking_model="tcfilter" # disable guest seccomp # Determines whether container seccomp profiles are passed to the virtual # machine and applied by the kata agent. If set to true, seccomp is not applied # within the guest # (default: true) disable_guest_seccomp=true # If enabled, the runtime will create opentracing.io traces and spans. # (See https://www.jaegertracing.io/docs/getting-started). # (default: disabled) #enable_tracing = true # If enabled, the runtime will not create a network namespace for shim and hypervisor processes. # This option may have some potential impacts to your host. It should only be used when you know what you're doing. # `disable_new_netns` conflicts with `enable_netmon` # `disable_new_netns` conflicts with `internetworking_model=bridged` and `internetworking_model=macvtap`. It works only # with `internetworking_model=none`. The tap device will be in the host network namespace and can connect to a bridge # (like OVS) directly. # If you are using docker, `disable_new_netns` only works with `docker run --net=none` # (default: false) #disable_new_netns = true # Enabled experimental feature list, format: ["a", "b"]. # Experimental features are features not stable enough for production, # They may break compatibility, and are prepared for a big version bump. # Supported experimental features: # 1. "newstore": new persist storage driver which breaks backward compatibility, # expected to move out of experimental in 2.0.0. # (default: []) experimental=[] ``` Output of "`cat "/usr/share/defaults/kata-containers/configuration.toml"`": ```toml # Copyright (c) 2017-2019 Intel Corporation # # SPDX-License-Identifier: Apache-2.0 # # XXX: WARNING: this file is auto-generated. # XXX: # XXX: Source file: "cli/config/configuration-qemu.toml.in" # XXX: Project: # XXX: Name: Kata Containers # XXX: Type: kata [hypervisor.qemu] path = "/usr/bin/qemu-lite-system-x86_64" kernel = "sr/share/kata-containers/vmlinux-fc-config" initrd = "/usr/share/kata-containers/kata-containers-initrd.img" machine_type = "pc" # Optional space-separated list of options to pass to the guest kernel. # For example, use `kernel_params = "vsyscall=emulate"` if you are having # trouble running pre-2.15 glibc. # # WARNING: - any parameter specified here will take priority over the default # parameter value of the same name used to start the virtual machine. # Do not set values here unless you understand the impact of doing so as you # may stop the virtual machine from booting. # To see the list of default parameters, enable hypervisor debug, create a # container and look for 'default-kernel-parameters' log entries. kernel_params = "ipv6.disable=1" # Path to the firmware. # If you want that qemu uses the default firmware leave this option empty firmware = "" # Machine accelerators # comma-separated list of machine accelerators to pass to the hypervisor. # For example, `machine_accelerators = "nosmm,nosmbus,nosata,nopit,static-prt,nofw"` machine_accelerators="" # Default number of vCPUs per SB/VM: # unspecified or 0 --> will be set to 1 # < 0 --> will be set to the actual number of physical cores # > 0 <= number of physical cores --> will be set to the specified number # > number of physical cores --> will be set to the actual number of physical cores default_vcpus = 1 # Default maximum number of vCPUs per SB/VM: # unspecified or == 0 --> will be set to the actual number of physical cores or to the maximum number # of vCPUs supported by KVM if that number is exceeded # > 0 <= number of physical cores --> will be set to the specified number # > number of physical cores --> will be set to the actual number of physical cores or to the maximum number # of vCPUs supported by KVM if that number is exceeded # WARNING: Depending of the architecture, the maximum number of vCPUs supported by KVM is used when # the actual number of physical cores is greater than it. # WARNING: Be aware that this value impacts the virtual machine's memory footprint and CPU # the hotplug functionality. For example, `default_maxvcpus = 240` specifies that until 240 vCPUs # can be added to a SB/VM, but the memory footprint will be big. Another example, with # `default_maxvcpus = 8` the memory footprint will be small, but 8 will be the maximum number of # vCPUs supported by the SB/VM. In general, we recommend that you do not edit this variable, # unless you know what are you doing. default_maxvcpus = 0 # Bridges can be used to hot plug devices. # Limitations: # * Currently only pci bridges are supported # * Until 30 devices per bridge can be hot plugged. # * Until 5 PCI bridges can be cold plugged per VM. # This limitation could be a bug in qemu or in the kernel # Default number of bridges per SB/VM: # unspecified or 0 --> will be set to 1 # > 1 <= 5 --> will be set to the specified number # > 5 --> will be set to 5 default_bridges = 1 # Default memory size in MiB for SB/VM. # If unspecified then it will be set 2048 MiB. default_memory = 2048 # # Default memory slots per SB/VM. # If unspecified then it will be set 10. # This is will determine the times that memory will be hotadded to sandbox/VM. #memory_slots = 10 # The size in MiB will be plused to max memory of hypervisor. # It is the memory address space for the NVDIMM devie. # If set block storage driver (block_device_driver) to "nvdimm", # should set memory_offset to the size of block device. # Default 0 #memory_offset = 0 # Disable block device from being used for a container's rootfs. # In case of a storage driver like devicemapper where a container's # root file system is backed by a block device, the block device is passed # directly to the hypervisor for performance reasons. # This flag prevents the block device from being passed to the hypervisor, # 9pfs is used instead to pass the rootfs. disable_block_device_use = false # Block storage driver to be used for the hypervisor in case the container # rootfs is backed by a block device. This is virtio-scsi, virtio-blk # or nvdimm. block_device_driver = "virtio-scsi" # Specifies cache-related options will be set to block devices or not. # Default false #block_device_cache_set = true # Specifies cache-related options for block devices. # Denotes whether use of O_DIRECT (bypass the host page cache) is enabled. # Default false #block_device_cache_direct = true # Specifies cache-related options for block devices. # Denotes whether flush requests for the device are ignored. # Default false #block_device_cache_noflush = true # Enable iothreads (data-plane) to be used. This causes IO to be # handled in a separate IO thread. This is currently only implemented # for SCSI. # enable_iothreads = false # Enable pre allocation of VM RAM, default false # Enabling this will result in lower container density # as all of the memory will be allocated and locked # This is useful when you want to reserve all the memory # upfront or in the cases where you want memory latencies # to be very predictable # Default false #enable_mem_prealloc = true # Enable huge pages for VM RAM, default false # Enabling this will result in the VM memory # being allocated using huge pages. # This is useful when you want to use vhost-user network # stacks within the container. This will automatically # result in memory pre allocation #enable_hugepages = true # Enable swap of vm memory. Default false. # The behaviour is undefined if mem_prealloc is also set to true #enable_swap = true # This option changes the default hypervisor and kernel parameters # to enable debug output where available. This extra output is added # to the proxy logs, but only when proxy debug is also enabled. # # Default false #enable_debug = true # Disable the customizations done in the runtime when it detects # that it is running on top a VMM. This will result in the runtime # behaving as it would when running on bare metal. # #disable_nesting_checks = true # This is the msize used for 9p shares. It is the number of bytes # used for 9p packet payload. #msize_9p = 8192 # If true and vsocks are supported, use vsocks to communicate directly # with the agent and no proxy is started, otherwise use unix # sockets and start a proxy to communicate with the agent. # Default false #use_vsock = true # VFIO devices are hotplugged on a bridge by default. # Enable hotplugging on root bus. This may be required for devices with # a large PCI bar, as this is a current limitation with hotplugging on # a bridge. This value is valid for "pc" machine type. # Default false #hotplug_vfio_on_root_bus = true # If host doesn't support vhost_net, set to true. Thus we won't create vhost fds for nics. # Default false #disable_vhost_net = true # # Default entropy source. # The path to a host source of entropy (including a real hardware RNG) # /dev/urandom and /dev/random are two main options. # Be aware that /dev/random is a blocking source of entropy. If the host # runs out of entropy, the VMs boot time will increase leading to get startup # timeouts. # The source of entropy /dev/urandom is non-blocking and provides a # generally acceptable source of entropy. It should work well for pretty much # all practical purposes. #entropy_source= "/dev/urandom" # Path to OCI hook binaries in the *guest rootfs*. # This does not affect host-side hooks which must instead be added to # the OCI spec passed to the runtime. # # You can create a rootfs with hooks by customizing the osbuilder scripts: # https://github.com/kata-containers/osbuilder # # Hooks must be stored in a subdirectory of guest_hook_path according to their # hook type, i.e. "guest_hook_path/{prestart,postart,poststop}". # The agent will scan these directories for executable files and add them, in # lexicographical order, to the lifecycle of the guest container. # Hooks are executed in the runtime namespace of the guest. See the official documentation: # https://github.com/opencontainers/runtime-spec/blob/v1.0.1/config.md#posix-platform-hooks # Warnings will be logged if any error is encountered will scanning for hooks, # but it will not abort container execution. #guest_hook_path = "/usr/share/oci/hooks" [factory] # VM templating support. Once enabled, new VMs are created from template # using vm cloning. They will share the same initial kernel, initramfs and # agent memory by mapping it readonly. It helps speeding up new container # creation and saves a lot of memory if there are many kata containers running # on the same host. # # When disabled, new VMs are created from scratch. # # Note: Requires "initrd=" to be set ("image=" is not supported). # # Default false enable_template = true # The number of caches of VMCache: # unspecified or == 0 --> VMCache is disabled # > 0 --> will be set to the specified number # # VMCache is a function that creates VMs as caches before using it. # It helps speed up new container creation. # The function consists of a server and some clients communicating # through Unix socket. The protocol is gRPC in protocols/cache/cache.proto. # The VMCache server will create some VMs and cache them by factory cache. # It will convert the VM to gRPC format and transport it when gets # requestion from clients. # Factory grpccache is the VMCache client. It will request gRPC format # VM and convert it back to a VM. If VMCache function is enabled, # kata-runtime will request VM from factory grpccache when it creates # a new sandbox. # # Default 0 #vm_cache_number = 0 # Specify the address of the Unix socket that is used by VMCache. # # Default /var/run/kata-containers/cache.sock #vm_cache_endpoint = "/var/run/kata-containers/cache.sock" [proxy.kata] path = "/usr/libexec/kata-containers/kata-proxy" # If enabled, proxy messages will be sent to the system log # (default: disabled) #enable_debug = true [shim.kata] path = "/usr/libexec/kata-containers/kata-shim" # If enabled, shim messages will be sent to the system log # (default: disabled) #enable_debug = true # If enabled, the shim will create opentracing.io traces and spans. # (See https://www.jaegertracing.io/docs/getting-started). # # Note: By default, the shim runs in a separate network namespace. Therefore, # to allow it to send trace details to the Jaeger agent running on the host, # it is necessary to set 'disable_new_netns=true' so that it runs in the host # network namespace. # # (default: disabled) #enable_tracing = true [agent.kata] # There is no field for this section. The goal is only to be able to # specify which type of agent the user wants to use. [netmon] # If enabled, the network monitoring process gets started when the # sandbox is created. This allows for the detection of some additional # network being added to the existing network namespace, after the # sandbox has been created. # (default: disabled) #enable_netmon = true # Specify the path to the netmon binary. path = "/usr/libexec/kata-containers/kata-netmon" # If enabled, netmon messages will be sent to the system log # (default: disabled) #enable_debug = true [runtime] # If enabled, the runtime will log additional debug messages to the # system log # (default: disabled) #enable_debug = true # # Internetworking model # Determines how the VM should be connected to the # the container network interface # Options: # # - bridged # Uses a linux bridge to interconnect the container interface to # the VM. Works for most cases except macvlan and ipvlan. # # - macvtap # Used when the Container network interface can be bridged using # macvtap. # # - none # Used when customize network. Only creates a tap device. No veth pair. # # - tcfilter # Uses tc filter rules to redirect traffic from the network interface # provided by plugin to a tap interface connected to the VM. # internetworking_model="tcfilter" # disable guest seccomp # Determines whether container seccomp profiles are passed to the virtual # machine and applied by the kata agent. If set to true, seccomp is not applied # within the guest # (default: true) disable_guest_seccomp=true # If enabled, the runtime will create opentracing.io traces and spans. # (See https://www.jaegertracing.io/docs/getting-started). # (default: disabled) #enable_tracing = true # If enabled, the runtime will not create a network namespace for shim and hypervisor processes. # This option may have some potential impacts to your host. It should only be used when you know what you're doing. # `disable_new_netns` conflicts with `enable_netmon` # `disable_new_netns` conflicts with `internetworking_model=bridged` and `internetworking_model=macvtap`. It works only # with `internetworking_model=none`. The tap device will be in the host network namespace and can connect to a bridge # (like OVS) directly. # If you are using docker, `disable_new_netns` only works with `docker run --net=none` # (default: false) #disable_new_netns = true # Enabled experimental feature list, format: ["a", "b"]. # Experimental features are features not stable enough for production, # They may break compatibility, and are prepared for a big version bump. # Supported experimental features: # 1. "newstore": new persist storage driver which breaks backward compatibility, # expected to move out of experimental in 2.0.0. # (default: []) experimental=[] ``` --- # KSM throttler ## version Output of "`/usr/libexec/kata-ksm-throttler/kata-ksm-throttler --version`": ``` kata-ksm-throttler version 1.7.0-alpha1-38c6b40 ``` ## systemd service # Image details No image --- # Initrd details No initrd --- # Logfiles ## Runtime logs Recent runtime problems found in system journal: ``` time="2019-05-16T10:49:13.32147787+08:00" level=warning msg="load sandbox devices failed" arch=amd64 command=run container=test error="open /run/vc/sbs/test/devices.json: no such file or directory" name=kata-runtime pid=13655 sandbox=test sandboxid=test source=virtcontainers subsystem=sandbox time="2019-05-16T10:49:29.622790938+08:00" level=warning msg="VM still running after waiting 15s" arch=amd64 command=run container=test name=kata-runtime pid=13655 source=virtcontainers subsystem=firecracker time="2019-05-16T10:49:29.639533953+08:00" level=warning msg="failed to cleanup netns" arch=amd64 command=run container=test error="failed to get netns /var/run/netns/cni-83291281-81c8-7967-f78f-73e879c1aa08: failed to Statfs \"/var/run/netns/cni-83291281-81c8-7967-f78f-73e879c1aa08\": no such file or directory" name=kata-runtime path=/var/run/netns/cni-83291281-81c8-7967-f78f-73e879c1aa08 pid=13655 source=katautils time="2019-05-16T10:49:29.639592056+08:00" level=error msg="rpc error: code = Unknown desc = rootfs (/run/kata-containers/shared/containers/test/rootfs) does not exist" arch=amd64 command=run container=test name=kata-runtime pid=13655 source=runtime time="2019-05-16T10:51:01.338546735+08:00" level=info msg="No info could be fetched" arch=amd64 command=run container=test error="open /run/vc/sbs/test/hypervisor.json: no such file or directory" function=init name=kata-runtime pid=13984 source=virtcontainers subsystem=firecracker time="2019-05-16T10:51:01.338621547+08:00" level=warning msg="load sandbox devices failed" arch=amd64 command=run container=test error="open /run/vc/sbs/test/devices.json: no such file or directory" name=kata-runtime pid=13984 sandbox=test sandboxid=test source=virtcontainers subsystem=sandbox time="2019-05-16T10:51:17.772198953+08:00" level=warning msg="VM still running after waiting 15s" arch=amd64 command=run container=test name=kata-runtime pid=13984 source=virtcontainers subsystem=firecracker time="2019-05-16T10:51:17.787133021+08:00" level=warning msg="failed to cleanup netns" arch=amd64 command=run container=test error="failed to get netns /var/run/netns/cni-f3a649b9-dfc5-ab28-653a-7398cafa286b: failed to Statfs \"/var/run/netns/cni-f3a649b9-dfc5-ab28-653a-7398cafa286b\": no such file or directory" name=kata-runtime path=/var/run/netns/cni-f3a649b9-dfc5-ab28-653a-7398cafa286b pid=13984 source=katautils time="2019-05-16T10:51:17.787192026+08:00" level=error msg="rpc error: code = Unknown desc = rootfs (/run/kata-containers/shared/containers/test/rootfs) does not exist" arch=amd64 command=run container=test name=kata-runtime pid=13984 source=runtime time="2019-05-16T10:54:45.101411625+08:00" level=error msg="Invalid command \"config\"" arch=amd64 name=kata-runtime pid=14601 source=runtime time="2019-05-16T10:55:43.863188344+08:00" level=info msg="No info could be fetched" arch=amd64 command=run container=test error="open /run/vc/sbs/test/hypervisor.json: no such file or directory" function=init name=kata-runtime pid=14784 source=virtcontainers subsystem=firecracker time="2019-05-16T10:55:43.87167004+08:00" level=warning msg="load sandbox devices failed" arch=amd64 command=run container=test error="open /run/vc/sbs/test/devices.json: no such file or directory" name=kata-runtime pid=14784 sandbox=test sandboxid=test source=virtcontainers subsystem=sandbox time="2019-05-16T10:56:00.406958366+08:00" level=warning msg="VM still running after waiting 15s" arch=amd64 command=run container=test name=kata-runtime pid=14784 source=virtcontainers subsystem=firecracker time="2019-05-16T10:56:00.435854365+08:00" level=warning msg="failed to cleanup netns" arch=amd64 command=run container=test error="failed to get netns /var/run/netns/cni-5fe367f3-799b-3c72-b097-738de6a42827: failed to Statfs \"/var/run/netns/cni-5fe367f3-799b-3c72-b097-738de6a42827\": no such file or directory" name=kata-runtime path=/var/run/netns/cni-5fe367f3-799b-3c72-b097-738de6a42827 pid=14784 source=katautils time="2019-05-16T10:56:00.445230037+08:00" level=error msg="rpc error: code = Unknown desc = rootfs (/run/kata-containers/shared/containers/test/rootfs) does not exist" arch=amd64 command=run container=test name=kata-runtime pid=14784 source=runtime time="2019-05-16T11:03:05.151251877+08:00" level=info msg="No info could be fetched" arch=amd64 command=run container=hhhhhhhhhhhhh error="open /run/vc/sbs/hhhhhhhhhhhhh/hypervisor.json: no such file or directory" function=init name=kata-runtime pid=16087 source=virtcontainers subsystem=firecracker time="2019-05-16T11:03:05.159784947+08:00" level=warning msg="load sandbox devices failed" arch=amd64 command=run container=hhhhhhhhhhhhh error="open /run/vc/sbs/hhhhhhhhhhhhh/devices.json: no such file or directory" name=kata-runtime pid=16087 sandbox=hhhhhhhhhhhhh sandboxid=hhhhhhhhhhhhh source=virtcontainers subsystem=sandbox time="2019-05-16T11:03:21.665461108+08:00" level=warning msg="VM still running after waiting 15s" arch=amd64 command=run container=hhhhhhhhhhhhh name=kata-runtime pid=16087 source=virtcontainers subsystem=firecracker time="2019-05-16T11:03:21.707965588+08:00" level=warning msg="failed to cleanup netns" arch=amd64 command=run container=hhhhhhhhhhhhh error="failed to get netns /var/run/netns/cni-075d74eb-cf13-2117-dabc-4b72d66fd9f2: failed to Statfs \"/var/run/netns/cni-075d74eb-cf13-2117-dabc-4b72d66fd9f2\": no such file or directory" name=kata-runtime path=/var/run/netns/cni-075d74eb-cf13-2117-dabc-4b72d66fd9f2 pid=16087 source=katautils time="2019-05-16T11:03:21.717011321+08:00" level=error msg="rpc error: code = Unknown desc = rootfs (/run/kata-containers/shared/containers/hhhhhhhhhhhhh/rootfs) does not exist" arch=amd64 command=run container=hhhhhhhhhhhhh name=kata-runtime pid=16087 source=runtime time="2019-05-16T11:06:13.118902408+08:00" level=info msg="No info could be fetched" arch=amd64 command=run container=test error="open /run/vc/sbs/test/hypervisor.json: no such file or directory" function=init name=kata-runtime pid=16633 source=virtcontainers subsystem=firecracker time="2019-05-16T11:06:13.118985342+08:00" level=warning msg="load sandbox devices failed" arch=amd64 command=run container=test error="open /run/vc/sbs/test/devices.json: no such file or directory" name=kata-runtime pid=16633 sandbox=test sandboxid=test source=virtcontainers subsystem=sandbox time="2019-05-16T11:06:29.59623307+08:00" level=warning msg="VM still running after waiting 15s" arch=amd64 command=run container=test name=kata-runtime pid=16633 source=virtcontainers subsystem=firecracker time="2019-05-16T11:06:29.61117145+08:00" level=warning msg="failed to cleanup netns" arch=amd64 command=run container=test error="failed to get netns /var/run/netns/cni-aed75fbf-926c-dcfd-8627-a12baaabc6e3: failed to Statfs \"/var/run/netns/cni-aed75fbf-926c-dcfd-8627-a12baaabc6e3\": no such file or directory" name=kata-runtime path=/var/run/netns/cni-aed75fbf-926c-dcfd-8627-a12baaabc6e3 pid=16633 source=katautils time="2019-05-16T11:06:29.61122761+08:00" level=error msg="rpc error: code = Unknown desc = rootfs (/run/kata-containers/shared/containers/test/rootfs) does not exist" arch=amd64 command=run container=test name=kata-runtime pid=16633 source=runtime time="2019-05-16T11:09:50.33687298+08:00" level=info msg="No info could be fetched" arch=amd64 command=run container=test error="open /run/vc/sbs/test/hypervisor.json: no such file or directory" function=init name=kata-runtime pid=17234 source=virtcontainers subsystem=firecracker time="2019-05-16T11:09:50.337062795+08:00" level=warning msg="load sandbox devices failed" arch=amd64 command=run container=test error="open /run/vc/sbs/test/devices.json: no such file or directory" name=kata-runtime pid=17234 sandbox=test sandboxid=test source=virtcontainers subsystem=sandbox time="2019-05-16T11:10:06.724801528+08:00" level=warning msg="VM still running after waiting 15s" arch=amd64 command=run container=test name=kata-runtime pid=17234 source=virtcontainers subsystem=firecracker time="2019-05-16T11:10:06.738700605+08:00" level=warning msg="failed to cleanup netns" arch=amd64 command=run container=test error="failed to get netns /var/run/netns/cni-a28a581d-6f44-eb6b-94b9-3d92f24d8288: failed to Statfs \"/var/run/netns/cni-a28a581d-6f44-eb6b-94b9-3d92f24d8288\": no such file or directory" name=kata-runtime path=/var/run/netns/cni-a28a581d-6f44-eb6b-94b9-3d92f24d8288 pid=17234 source=katautils time="2019-05-16T11:10:06.738752401+08:00" level=error msg="no such file or directory" arch=amd64 command=run container=test name=kata-runtime pid=17234 source=runtime time="2019-05-16T11:10:55.432489278+08:00" level=info msg="No info could be fetched" arch=amd64 command=run container=test error="open /run/vc/sbs/test/hypervisor.json: no such file or directory" function=init name=kata-runtime pid=17652 source=virtcontainers subsystem=firecracker time="2019-05-16T11:10:55.432560143+08:00" level=warning msg="load sandbox devices failed" arch=amd64 command=run container=test error="open /run/vc/sbs/test/devices.json: no such file or directory" name=kata-runtime pid=17652 sandbox=test sandboxid=test source=virtcontainers subsystem=sandbox time="2019-05-16T11:11:11.826923053+08:00" level=warning msg="VM still running after waiting 15s" arch=amd64 command=run container=test name=kata-runtime pid=17652 source=virtcontainers subsystem=firecracker time="2019-05-16T11:11:11.842805124+08:00" level=warning msg="failed to cleanup netns" arch=amd64 command=run container=test error="failed to get netns /var/run/netns/cni-a52300c0-5a6c-4e5b-9fdf-2347f0e660dd: failed to Statfs \"/var/run/netns/cni-a52300c0-5a6c-4e5b-9fdf-2347f0e660dd\": no such file or directory" name=kata-runtime path=/var/run/netns/cni-a52300c0-5a6c-4e5b-9fdf-2347f0e660dd pid=17652 source=katautils time="2019-05-16T11:11:11.84286095+08:00" level=error msg="rpc error: code = Unknown desc = rootfs (/run/kata-containers/shared/containers/test/rootfs) does not exist" arch=amd64 command=run container=test name=kata-runtime pid=17652 source=runtime time="2019-05-16T12:37:31.375031738+08:00" level=info msg="No info could be fetched" arch=amd64 command=run container=test error="open /run/vc/sbs/test/hypervisor.json: no such file or directory" function=init name=kata-runtime pid=31787 source=virtcontainers subsystem=firecracker time="2019-05-16T12:37:31.375122874+08:00" level=warning msg="load sandbox devices failed" arch=amd64 command=run container=test error="open /run/vc/sbs/test/devices.json: no such file or directory" name=kata-runtime pid=31787 sandbox=test sandboxid=test source=virtcontainers subsystem=sandbox time="2019-05-16T12:37:47.764054873+08:00" level=warning msg="VM still running after waiting 15s" arch=amd64 command=run container=test name=kata-runtime pid=31787 source=virtcontainers subsystem=firecracker time="2019-05-16T12:37:47.77863322+08:00" level=warning msg="failed to cleanup netns" arch=amd64 command=run container=test error="failed to get netns /var/run/netns/cni-a7022033-ef4e-c118-9d69-8678f1e8dece: failed to Statfs \"/var/run/netns/cni-a7022033-ef4e-c118-9d69-8678f1e8dece\": no such file or directory" name=kata-runtime path=/var/run/netns/cni-a7022033-ef4e-c118-9d69-8678f1e8dece pid=31787 source=katautils time="2019-05-16T12:37:47.778683715+08:00" level=error msg="no such file or directory" arch=amd64 command=run container=test name=kata-runtime pid=31787 source=runtime time="2019-05-16T12:38:24.060175919+08:00" level=info msg="No info could be fetched" arch=amd64 command=run container=test error="open /run/vc/sbs/test/hypervisor.json: no such file or directory" function=init name=kata-runtime pid=31962 source=virtcontainers subsystem=firecracker time="2019-05-16T12:38:24.06025117+08:00" level=warning msg="load sandbox devices failed" arch=amd64 command=run container=test error="open /run/vc/sbs/test/devices.json: no such file or directory" name=kata-runtime pid=31962 sandbox=test sandboxid=test source=virtcontainers subsystem=sandbox time="2019-05-16T12:38:40.486254268+08:00" level=warning msg="VM still running after waiting 15s" arch=amd64 command=run container=test name=kata-runtime pid=31962 source=virtcontainers subsystem=firecracker time="2019-05-16T12:38:40.512332627+08:00" level=warning msg="failed to cleanup netns" arch=amd64 command=run container=test error="failed to get netns /var/run/netns/cni-0a0e53e2-4e84-4ff5-0701-f46f1a4af976: failed to Statfs \"/var/run/netns/cni-0a0e53e2-4e84-4ff5-0701-f46f1a4af976\": no such file or directory" name=kata-runtime path=/var/run/netns/cni-0a0e53e2-4e84-4ff5-0701-f46f1a4af976 pid=31962 source=katautils time="2019-05-16T12:38:40.512412285+08:00" level=error msg="rpc error: code = Unknown desc = rootfs (/run/kata-containers/shared/containers/test/rootfs) does not exist" arch=amd64 command=run container=test name=kata-runtime pid=31962 source=runtime time="2019-05-16T13:00:10.264867244+08:00" level=info msg="No info could be fetched" arch=amd64 command=run container=test error="open /run/vc/sbs/test/hypervisor.json: no such file or directory" function=init name=kata-runtime pid=5537 source=virtcontainers subsystem=firecracker time="2019-05-16T13:00:10.265167882+08:00" level=warning msg="load sandbox devices failed" arch=amd64 command=run container=test error="open /run/vc/sbs/test/devices.json: no such file or directory" name=kata-runtime pid=5537 sandbox=test sandboxid=test source=virtcontainers subsystem=sandbox time="2019-05-16T13:00:26.69763093+08:00" level=warning msg="VM still running after waiting 15s" arch=amd64 command=run container=test name=kata-runtime pid=5537 source=virtcontainers subsystem=firecracker time="2019-05-16T13:00:26.712874038+08:00" level=warning msg="failed to cleanup netns" arch=amd64 command=run container=test error="failed to get netns /var/run/netns/cni-0fcb344a-f34a-fa81-b9c2-14c9ee227488: failed to Statfs \"/var/run/netns/cni-0fcb344a-f34a-fa81-b9c2-14c9ee227488\": no such file or directory" name=kata-runtime path=/var/run/netns/cni-0fcb344a-f34a-fa81-b9c2-14c9ee227488 pid=5537 source=katautils time="2019-05-16T13:00:26.712929398+08:00" level=error msg="rpc error: code = Unknown desc = rootfs (/run/kata-containers/shared/containers/test/rootfs) does not exist" arch=amd64 command=run container=test name=kata-runtime pid=5537 source=runtime ``` ## Proxy logs Recent proxy problems found in system journal: ``` time="2019-05-15T13:07:01.027314437+08:00" level=fatal msg="channel error" error="accept unix /run/vc/sbs/d0eb8325-f3f0-4603-a1a9-44bd5d420047/proxy.sock: use of closed network connection" name=kata-proxy pid=20981 sandbox=d0eb8325-f3f0-4603-a1a9-44bd5d420047 source=proxy time="2019-05-15T13:23:54.183887578+08:00" level=fatal msg="channel error" error="accept unix /run/vc/sbs/2d0a1493-f137-486a-b7eb-ee63811b526d/proxy.sock: use of closed network connection" name=kata-proxy pid=1922 sandbox=2d0a1493-f137-486a-b7eb-ee63811b526d source=proxy time="2019-05-15T13:23:54.755368296+08:00" level=fatal msg="channel error" error="accept unix /run/vc/sbs/be2dd516-81cc-429b-b6be-83841ce91e86/proxy.sock: use of closed network connection" name=kata-proxy pid=1981 sandbox=be2dd516-81cc-429b-b6be-83841ce91e86 source=proxy time="2019-05-15T13:23:55.321550986+08:00" level=fatal msg="channel error" error="accept unix /run/vc/sbs/b51cefbd-c59f-4dbc-9989-57c1f7df836c/proxy.sock: use of closed network connection" name=kata-proxy pid=2034 sandbox=b51cefbd-c59f-4dbc-9989-57c1f7df836c source=proxy time="2019-05-15T13:23:55.89091785+08:00" level=fatal msg="failed to handle exit signal" error="close unix @->/run/vc/vm/cf030c71-972b-4e4e-a809-6c8d95c18749/kata.sock: use of closed network connection" name=kata-proxy pid=2090 sandbox=cf030c71-972b-4e4e-a809-6c8d95c18749 source=proxy time="2019-05-15T13:23:56.455423715+08:00" level=fatal msg="failed to handle exit signal" error="close unix @->/run/vc/vm/c04f3774-aef6-4fd8-9ad3-583ca302cf36/kata.sock: use of closed network connection" name=kata-proxy pid=2141 sandbox=c04f3774-aef6-4fd8-9ad3-583ca302cf36 source=proxy time="2019-05-15T13:23:57.034560905+08:00" level=fatal msg="channel error" error="accept unix /run/vc/sbs/99074b95-c3ce-4665-82d7-f9e1c2770991/proxy.sock: use of closed network connection" name=kata-proxy pid=2199 sandbox=99074b95-c3ce-4665-82d7-f9e1c2770991 source=proxy time="2019-05-15T13:24:47.804806946+08:00" level=fatal msg="failed to handle exit signal" error="close unix @->/run/vc/vm/9d2eb2f1-aad0-4b2b-a443-ee02deeb6757/kata.sock: use of closed network connection" name=kata-proxy pid=2445 sandbox=9d2eb2f1-aad0-4b2b-a443-ee02deeb6757 source=proxy time="2019-05-15T13:24:48.440042766+08:00" level=fatal msg="failed to handle exit signal" error="close unix @->/run/vc/vm/291a38c7-1bca-42d6-87b4-dd55b45639a0/kata.sock: use of closed network connection" name=kata-proxy pid=2553 sandbox=291a38c7-1bca-42d6-87b4-dd55b45639a0 source=proxy time="2019-05-15T13:24:49.070655841+08:00" level=fatal msg="channel error" error="accept unix /run/vc/sbs/0bd9feb4-55c4-428f-8d0d-4e1496ce6928/proxy.sock: use of closed network connection" name=kata-proxy pid=2706 sandbox=0bd9feb4-55c4-428f-8d0d-4e1496ce6928 source=proxy time="2019-05-15T13:24:49.767589871+08:00" level=fatal msg="failed to handle exit signal" error="close unix @->/run/vc/vm/60582b3b-7a64-4189-940c-68d940afea02/kata.sock: use of closed network connection" name=kata-proxy pid=2815 sandbox=60582b3b-7a64-4189-940c-68d940afea02 source=proxy time="2019-05-15T13:24:50.40081915+08:00" level=fatal msg="failed to handle exit signal" error="close unix @->/run/vc/vm/6d0d527e-f83c-4c6c-a74b-26504e5aaf65/kata.sock: use of closed network connection" name=kata-proxy pid=2935 sandbox=6d0d527e-f83c-4c6c-a74b-26504e5aaf65 source=proxy time="2019-05-15T13:24:51.069731916+08:00" level=fatal msg="failed to handle exit signal" error="close unix @->/run/vc/vm/f76c4075-0e9d-44da-9487-b01e9bca8cc5/kata.sock: use of closed network connection" name=kata-proxy pid=3151 sandbox=f76c4075-0e9d-44da-9487-b01e9bca8cc5 source=proxy time="2019-05-15T13:24:51.703212857+08:00" level=fatal msg="failed to handle exit signal" error="close unix @->/run/vc/vm/5f438bef-5f4e-43c3-846e-79b02f455772/kata.sock: use of closed network connection" name=kata-proxy pid=3263 sandbox=5f438bef-5f4e-43c3-846e-79b02f455772 source=proxy time="2019-05-15T13:24:52.435170515+08:00" level=fatal msg="failed to handle exit signal" error="close unix @->/run/vc/vm/82270279-4332-43e6-bad6-10e093692889/kata.sock: use of closed network connection" name=kata-proxy pid=3359 sandbox=82270279-4332-43e6-bad6-10e093692889 source=proxy time="2019-05-15T13:24:53.078589705+08:00" level=fatal msg="channel error" error="accept unix /run/vc/sbs/b9032d90-3f8b-4dd7-b400-36fea1da3f64/proxy.sock: use of closed network connection" name=kata-proxy pid=3479 sandbox=b9032d90-3f8b-4dd7-b400-36fea1da3f64 source=proxy time="2019-05-15T13:24:53.70219077+08:00" level=fatal msg="channel error" error="accept unix /run/vc/sbs/bd4f4aa8-ced6-4766-9914-5022c67a6d72/proxy.sock: use of closed network connection" name=kata-proxy pid=3570 sandbox=bd4f4aa8-ced6-4766-9914-5022c67a6d72 source=proxy time="2019-05-15T13:25:01.605139345+08:00" level=fatal msg="failed to handle exit signal" error="close unix @->/run/vc/vm/ab359aa0-2b60-47a9-a0a7-77fc9d902f78/kata.sock: use of closed network connection" name=kata-proxy pid=3683 sandbox=ab359aa0-2b60-47a9-a0a7-77fc9d902f78 source=proxy time="2019-05-15T13:25:02.247048553+08:00" level=fatal msg="failed to handle exit signal" error="close unix @->/run/vc/vm/e2e19c98-aef9-4299-852f-f33a5aac96f0/kata.sock: use of closed network connection" name=kata-proxy pid=3766 sandbox=e2e19c98-aef9-4299-852f-f33a5aac96f0 source=proxy time="2019-05-15T13:25:02.94562764+08:00" level=fatal msg="failed to handle exit signal" error="close unix @->/run/vc/vm/e464cc68-0f67-4dd9-80c0-ad58528c7f50/kata.sock: use of closed network connection" name=kata-proxy pid=3859 sandbox=e464cc68-0f67-4dd9-80c0-ad58528c7f50 source=proxy time="2019-05-15T13:25:03.653639223+08:00" level=fatal msg="channel error" error="accept unix /run/vc/sbs/ecca0805-f1dd-4c39-ad76-b9ce10c2320f/proxy.sock: use of closed network connection" name=kata-proxy pid=3961 sandbox=ecca0805-f1dd-4c39-ad76-b9ce10c2320f source=proxy time="2019-05-15T13:25:04.398397759+08:00" level=fatal msg="failed to handle exit signal" error="close unix @->/run/vc/vm/1840d00e-5d1a-4d6a-ac84-a868851787a3/kata.sock: use of closed network connection" name=kata-proxy pid=4057 sandbox=1840d00e-5d1a-4d6a-ac84-a868851787a3 source=proxy time="2019-05-15T13:25:05.06781594+08:00" level=fatal msg="failed to handle exit signal" error="close unix @->/run/vc/vm/8e1f65fd-2ee8-45bb-ae24-b15e70e8c863/kata.sock: use of closed network connection" name=kata-proxy pid=4141 sandbox=8e1f65fd-2ee8-45bb-ae24-b15e70e8c863 source=proxy time="2019-05-15T13:25:05.721778617+08:00" level=fatal msg="channel error" error="accept unix /run/vc/sbs/d1429285-a502-4b65-b0d2-3891f7a2e99b/proxy.sock: use of closed network connection" name=kata-proxy pid=4226 sandbox=d1429285-a502-4b65-b0d2-3891f7a2e99b source=proxy time="2019-05-15T13:25:06.415109616+08:00" level=fatal msg="channel error" error="accept unix /run/vc/sbs/7b8322df-d4ec-47ce-8e2e-cdfb64ea7437/proxy.sock: use of closed network connection" name=kata-proxy pid=4307 sandbox=7b8322df-d4ec-47ce-8e2e-cdfb64ea7437 source=proxy time="2019-05-15T13:25:07.103748292+08:00" level=fatal msg="channel error" error="accept unix /run/vc/sbs/262b0d01-b50a-4efc-9cca-4cc9c8292647/proxy.sock: use of closed network connection" name=kata-proxy pid=4395 sandbox=262b0d01-b50a-4efc-9cca-4cc9c8292647 source=proxy time="2019-05-15T13:25:07.10385137+08:00" level=fatal msg="failed to handle exit signal" error="close unix @->/run/vc/vm/262b0d01-b50a-4efc-9cca-4cc9c8292647/kata.sock: use of closed network connection" name=kata-proxy pid=4395 sandbox=262b0d01-b50a-4efc-9cca-4cc9c8292647 source=proxy time="2019-05-15T13:25:07.817463392+08:00" level=fatal msg="failed to handle exit signal" error="close unix @->/run/vc/vm/7acd12b9-1ab5-4c88-b6bf-b4d69e804de7/kata.sock: use of closed network connection" name=kata-proxy pid=4485 sandbox=7acd12b9-1ab5-4c88-b6bf-b4d69e804de7 source=proxy time="2019-05-15T15:13:14.450118559+08:00" level=fatal msg="failed to handle exit signal" error="close unix @->/run/vc/vm/e380f5c8-f6d3-4be2-b334-25ae0baf6c0e/kata.sock: use of closed network connection" name=kata-proxy pid=21789 sandbox=e380f5c8-f6d3-4be2-b334-25ae0baf6c0e source=proxy time="2019-05-15T15:13:26.597322169+08:00" level=fatal msg="failed to handle exit signal" error="close unix @->/run/vc/vm/28dfd4a7-b367-449a-ba4a-0ba5d70319e6/kata.sock: use of closed network connection" name=kata-proxy pid=21923 sandbox=28dfd4a7-b367-449a-ba4a-0ba5d70319e6 source=proxy time="2019-05-15T15:13:27.279767758+08:00" level=fatal msg="failed to handle exit signal" error="close unix @->/run/vc/vm/00092548-0902-4541-b164-093e2fc9c2f5/kata.sock: use of closed network connection" name=kata-proxy pid=22006 sandbox=00092548-0902-4541-b164-093e2fc9c2f5 source=proxy time="2019-05-15T15:13:27.942970598+08:00" level=fatal msg="failed to handle exit signal" error="close unix @->/run/vc/vm/1261b2b6-9533-4284-b30a-e2301536a96a/kata.sock: use of closed network connection" name=kata-proxy pid=22098 sandbox=1261b2b6-9533-4284-b30a-e2301536a96a source=proxy time="2019-05-15T15:13:28.61307612+08:00" level=fatal msg="channel error" error="accept unix /run/vc/sbs/539d9c6a-4386-489f-b889-a8513ec9c4f4/proxy.sock: use of closed network connection" name=kata-proxy pid=22188 sandbox=539d9c6a-4386-489f-b889-a8513ec9c4f4 source=proxy time="2019-05-15T15:13:29.283865425+08:00" level=fatal msg="channel error" error="accept unix /run/vc/sbs/36012a1e-e513-48f8-b64b-b9edc28fcc63/proxy.sock: use of closed network connection" name=kata-proxy pid=22279 sandbox=36012a1e-e513-48f8-b64b-b9edc28fcc63 source=proxy time="2019-05-15T15:13:29.283947575+08:00" level=fatal msg="failed to handle exit signal" error="close unix @->/run/vc/vm/36012a1e-e513-48f8-b64b-b9edc28fcc63/kata.sock: use of closed network connection" name=kata-proxy pid=22279 sandbox=36012a1e-e513-48f8-b64b-b9edc28fcc63 source=proxy time="2019-05-15T15:13:29.95983582+08:00" level=fatal msg="failed to handle exit signal" error="close unix @->/run/vc/vm/d16d850a-93e1-46c6-bb43-618fdec3bb39/kata.sock: use of closed network connection" name=kata-proxy pid=22373 sandbox=d16d850a-93e1-46c6-bb43-618fdec3bb39 source=proxy time="2019-05-15T15:13:30.622958751+08:00" level=fatal msg="failed to handle exit signal" error="close unix @->/run/vc/vm/21b81f15-2c57-4bda-a615-51a9217f30bb/kata.sock: use of closed network connection" name=kata-proxy pid=22460 sandbox=21b81f15-2c57-4bda-a615-51a9217f30bb source=proxy time="2019-05-15T15:13:31.235071766+08:00" level=fatal msg="channel error" error="accept unix /run/vc/sbs/061e03b2-d397-4a43-8006-640454001b89/proxy.sock: use of closed network connection" name=kata-proxy pid=22553 sandbox=061e03b2-d397-4a43-8006-640454001b89 source=proxy time="2019-05-15T15:13:31.868768335+08:00" level=fatal msg="failed to handle exit signal" error="close unix @->/run/vc/vm/767d90e7-227d-4773-8772-815b4b1f86a0/kata.sock: use of closed network connection" name=kata-proxy pid=22643 sandbox=767d90e7-227d-4773-8772-815b4b1f86a0 source=proxy time="2019-05-15T15:13:32.497642143+08:00" level=fatal msg="channel error" error="accept unix /run/vc/sbs/46abf6d7-b067-4e42-b409-9daba9fe0edf/proxy.sock: use of closed network connection" name=kata-proxy pid=22730 sandbox=46abf6d7-b067-4e42-b409-9daba9fe0edf source=proxy time="2019-05-15T15:13:44.791852682+08:00" level=fatal msg="failed to handle exit signal" error="close unix @->/run/vc/vm/570c63c9-17b2-4f5e-9a13-19bf9e3ecf6c/kata.sock: use of closed network connection" name=kata-proxy pid=22844 sandbox=570c63c9-17b2-4f5e-9a13-19bf9e3ecf6c source=proxy time="2019-05-15T15:13:45.413799009+08:00" level=fatal msg="failed to handle exit signal" error="close unix @->/run/vc/vm/36764ee5-28b0-45c4-acc4-6b73f8e4e8b7/kata.sock: use of closed network connection" name=kata-proxy pid=22936 sandbox=36764ee5-28b0-45c4-acc4-6b73f8e4e8b7 source=proxy time="2019-05-15T15:13:46.052148607+08:00" level=fatal msg="channel error" error="accept unix /run/vc/sbs/6f77170c-44de-460c-9670-ecaec2ff7684/proxy.sock: use of closed network connection" name=kata-proxy pid=23025 sandbox=6f77170c-44de-460c-9670-ecaec2ff7684 source=proxy time="2019-05-15T15:13:46.694810109+08:00" level=fatal msg="channel error" error="accept unix /run/vc/sbs/239b9aef-6419-46ef-95b4-16e12ce3c4e9/proxy.sock: use of closed network connection" name=kata-proxy pid=23115 sandbox=239b9aef-6419-46ef-95b4-16e12ce3c4e9 source=proxy time="2019-05-15T15:13:47.355652732+08:00" level=fatal msg="failed to handle exit signal" error="close unix @->/run/vc/vm/7d3bb744-daeb-4d1b-a841-ebaa2d580370/kata.sock: use of closed network connection" name=kata-proxy pid=23204 sandbox=7d3bb744-daeb-4d1b-a841-ebaa2d580370 source=proxy time="2019-05-15T15:13:48.040975701+08:00" level=fatal msg="failed to handle exit signal" error="close unix @->/run/vc/vm/37bb09f2-c993-4870-b37d-7215079e33f9/kata.sock: use of closed network connection" name=kata-proxy pid=23290 sandbox=37bb09f2-c993-4870-b37d-7215079e33f9 source=proxy time="2019-05-15T15:13:48.708035151+08:00" level=fatal msg="channel error" error="accept unix /run/vc/sbs/dabff8e8-c32d-4872-8bb2-0685ae813626/proxy.sock: use of closed network connection" name=kata-proxy pid=23382 sandbox=dabff8e8-c32d-4872-8bb2-0685ae813626 source=proxy time="2019-05-15T15:13:49.407051599+08:00" level=fatal msg="failed to handle exit signal" error="close unix @->/run/vc/vm/234c0201-0894-4eb0-ba94-5b589e700b39/kata.sock: use of closed network connection" name=kata-proxy pid=23471 sandbox=234c0201-0894-4eb0-ba94-5b589e700b39 source=proxy time="2019-05-15T15:13:50.068814651+08:00" level=fatal msg="channel error" error="accept unix /run/vc/sbs/d1e51392-85b1-4376-9f37-b0bb7290fa88/proxy.sock: use of closed network connection" name=kata-proxy pid=23554 sandbox=d1e51392-85b1-4376-9f37-b0bb7290fa88 source=proxy time="2019-05-15T15:13:50.732468817+08:00" level=fatal msg="failed to handle exit signal" error="close unix @->/run/vc/vm/f69cf365-4fc2-4fdf-815a-10c968b24267/kata.sock: use of closed network connection" name=kata-proxy pid=23640 sandbox=f69cf365-4fc2-4fdf-815a-10c968b24267 source=proxy ``` ## Shim logs No recent shim problems found in system journal. ## Throttler logs No recent throttler problems found in system journal. --- # Container manager details Have `docker` ## Docker Output of "`docker version`": ``` Client: Version: 18.06.3-ce API version: 1.38 Go version: go1.10.3 Git commit: d7080c1 Built: Wed Feb 20 02:28:10 2019 OS/Arch: linux/amd64 Experimental: false Server: Engine: Version: 18.06.3-ce API version: 1.38 (minimum version 1.12) Go version: go1.10.3 Git commit: d7080c1 Built: Wed Feb 20 02:26:34 2019 OS/Arch: linux/amd64 Experimental: false ``` Output of "`docker info`": ``` Containers: 12 Running: 0 Paused: 0 Stopped: 12 Images: 22 Server Version: 18.06.3-ce Storage Driver: devicemapper Pool Name: docker-8:1-3550679-pool Pool Blocksize: 65.54kB Base Device Size: 10.74GB Backing Filesystem: ext4 Udev Sync Supported: true Data file: /dev/loop23 Metadata file: /dev/loop24 Data loop file: /var/lib/docker/devicemapper/devicemapper/data Metadata loop file: /var/lib/docker/devicemapper/devicemapper/metadata Data Space Used: 7.114GB Data Space Total: 107.4GB Data Space Available: 37.91GB Metadata Space Used: 22.24MB Metadata Space Total: 2.147GB Metadata Space Available: 2.125GB Thin Pool Minimum Free Space: 10.74GB Deferred Removal Enabled: true Deferred Deletion Enabled: true Deferred Deleted Device Count: 0 Library Version: 1.02.145 (2017-11-03) Logging Driver: json-file Cgroup Driver: cgroupfs Plugins: Volume: local Network: bridge host macvlan null overlay Log: awslogs fluentd gcplogs gelf journald json-file logentries splunk syslog Swarm: inactive Runtimes: runc runsc kata-fc kata-runtime Default Runtime: runc Init Binary: docker-init containerd version: 468a545b9edcd5932818eb9de8e72413e616e86e runc version: a592beb5bc4c4092b1b1bac971afed27687340c5 init version: fec3683 Security Options: apparmor seccomp Profile: default Kernel Version: 4.18.0-17-generic Operating System: Ubuntu 18.04.2 LTS OSType: linux Architecture: x86_64 CPUs: 4 Total Memory: 3.847GiB Name: ubuntu ID: TIMJ:3CF5:GQRG:RQDH:X5YR:XTS4:4YP2:7ME6:GEZ3:E237:YYJG:CR6H Docker Root Dir: /var/lib/docker Debug Mode (client): false Debug Mode (server): false Registry: https://index.docker.io/v1/ Labels: Experimental: false Insecure Registries: 127.0.0.0/8 Live Restore Enabled: false WARNING: devicemapper: usage of loopback devices is strongly discouraged for production use. Use `--storage-opt dm.thinpooldev` to specify a custom block storage device. WARNING: No swap limit support ``` Output of "`systemctl show docker`": ``` Type=notify Restart=on-failure NotifyAccess=main RestartUSec=100ms TimeoutStartUSec=infinity TimeoutStopUSec=1min 30s RuntimeMaxUSec=infinity WatchdogUSec=0 WatchdogTimestamp=Thu 2019-05-16 12:50:28 CST WatchdogTimestampMonotonic=11626271 PermissionsStartOnly=no RootDirectoryStartOnly=no RemainAfterExit=no GuessMainPID=yes MainPID=1525 ControlPID=0 FileDescriptorStoreMax=0 NFileDescriptorStore=0 StatusErrno=0 Result=success UID=[not set] GID=[not set] NRestarts=0 ExecMainStartTimestamp=Thu 2019-05-16 12:50:24 CST ExecMainStartTimestampMonotonic=7996987 ExecMainExitTimestampMonotonic=0 ExecMainPID=1525 ExecMainCode=0 ExecMainStatus=0 ExecStart={ path=/usr/bin/dockerd ; argv[]=/usr/bin/dockerd -H fd:// ; ignore_errors=no ; start_time=[Thu 2019-05-16 12:50:24 CST] ; stop_time=[n/a] ; pid=1525 ; code=(null) ; status=0/0 } ExecReload={ path=/bin/kill ; argv[]=/bin/kill -s HUP $MAINPID ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 } Slice=system.slice ControlGroup=/system.slice/docker.service MemoryCurrent=[not set] CPUUsageNSec=[not set] TasksCurrent=47 IPIngressBytes=18446744073709551615 IPIngressPackets=18446744073709551615 IPEgressBytes=18446744073709551615 IPEgressPackets=18446744073709551615 Delegate=yes DelegateControllers=cpu cpuacct io blkio memory devices pids CPUAccounting=no CPUWeight=[not set] StartupCPUWeight=[not set] CPUShares=[not set] StartupCPUShares=[not set] CPUQuotaPerSecUSec=infinity IOAccounting=no IOWeight=[not set] StartupIOWeight=[not set] BlockIOAccounting=no BlockIOWeight=[not set] StartupBlockIOWeight=[not set] MemoryAccounting=no MemoryLow=0 MemoryHigh=infinity MemoryMax=infinity MemorySwapMax=infinity MemoryLimit=infinity DevicePolicy=auto TasksAccounting=yes TasksMax=infinity IPAccounting=no UMask=0022 LimitCPU=infinity LimitCPUSoft=infinity LimitFSIZE=infinity LimitFSIZESoft=infinity LimitDATA=infinity LimitDATASoft=infinity LimitSTACK=infinity LimitSTACKSoft=8388608 LimitCORE=infinity LimitCORESoft=infinity LimitRSS=infinity LimitRSSSoft=infinity LimitNOFILE=1048576 LimitNOFILESoft=1048576 LimitAS=infinity LimitASSoft=infinity LimitNPROC=infinity LimitNPROCSoft=infinity LimitMEMLOCK=16777216 LimitMEMLOCKSoft=16777216 LimitLOCKS=infinity LimitLOCKSSoft=infinity LimitSIGPENDING=15501 LimitSIGPENDINGSoft=15501 LimitMSGQUEUE=819200 LimitMSGQUEUESoft=819200 LimitNICE=0 LimitNICESoft=0 LimitRTPRIO=0 LimitRTPRIOSoft=0 LimitRTTIME=infinity LimitRTTIMESoft=infinity OOMScoreAdjust=0 Nice=0 IOSchedulingClass=0 IOSchedulingPriority=0 CPUSchedulingPolicy=0 CPUSchedulingPriority=0 TimerSlackNSec=50000 CPUSchedulingResetOnFork=no NonBlocking=no StandardInput=null StandardInputData= StandardOutput=journal StandardError=inherit TTYReset=no TTYVHangup=no TTYVTDisallocate=no SyslogPriority=30 SyslogLevelPrefix=yes SyslogLevel=6 SyslogFacility=3 LogLevelMax=-1 SecureBits=0 CapabilityBoundingSet=cap_chown cap_dac_override cap_dac_read_search cap_fowner cap_fsetid cap_kill cap_setgid cap_setuid cap_setpcap cap_linux_immutable cap_net_bind_service cap_net_broadcast cap_net_admin cap_net_raw cap_ipc_lock cap_ipc_owner cap_sys_module cap_sys_rawio cap_sys_chroot cap_sys_ptrace cap_sys_pacct cap_sys_admin cap_sys_boot cap_sys_nice cap_sys_resource cap_sys_time cap_sys_tty_config cap_mknod cap_lease cap_audit_write cap_audit_control cap_setfcap cap_mac_override cap_mac_admin cap_syslog cap_wake_alarm cap_block_suspend AmbientCapabilities= DynamicUser=no RemoveIPC=no MountFlags= PrivateTmp=no PrivateDevices=no ProtectKernelTunables=no ProtectKernelModules=no ProtectControlGroups=no PrivateNetwork=no PrivateUsers=no ProtectHome=no ProtectSystem=no SameProcessGroup=no UtmpMode=init IgnoreSIGPIPE=yes NoNewPrivileges=no SystemCallErrorNumber=0 LockPersonality=no RuntimeDirectoryPreserve=no RuntimeDirectoryMode=0755 StateDirectoryMode=0755 CacheDirectoryMode=0755 LogsDirectoryMode=0755 ConfigurationDirectoryMode=0755 MemoryDenyWriteExecute=no RestrictRealtime=no RestrictNamespaces=no MountAPIVFS=no KeyringMode=private KillMode=process KillSignal=15 SendSIGKILL=yes SendSIGHUP=no Id=docker.service Names=docker.service Requires=sysinit.target docker.socket system.slice Wants=network-online.target WantedBy=multi-user.target ConsistsOf=docker.socket Conflicts=shutdown.target Before=multi-user.target shutdown.target After=docker.socket network-online.target system.slice firewalld.service basic.target sysinit.target systemd-journald.socket TriggeredBy=docker.socket Documentation=https://docs.docker.com Description=Docker Application Container Engine LoadState=loaded ActiveState=active SubState=running FragmentPath=/lib/systemd/system/docker.service UnitFileState=enabled UnitFilePreset=enabled StateChangeTimestamp=Thu 2019-05-16 12:50:28 CST StateChangeTimestampMonotonic=11626272 InactiveExitTimestamp=Thu 2019-05-16 12:50:24 CST InactiveExitTimestampMonotonic=7997015 ActiveEnterTimestamp=Thu 2019-05-16 12:50:28 CST ActiveEnterTimestampMonotonic=11626272 ActiveExitTimestampMonotonic=0 InactiveEnterTimestampMonotonic=0 CanStart=yes CanStop=yes CanReload=yes CanIsolate=no StopWhenUnneeded=no RefuseManualStart=no RefuseManualStop=no AllowIsolate=no DefaultDependencies=yes OnFailureJobMode=replace IgnoreOnIsolate=no NeedDaemonReload=no JobTimeoutUSec=infinity JobRunningTimeoutUSec=infinity JobTimeoutAction=none ConditionResult=yes AssertResult=yes ConditionTimestamp=Thu 2019-05-16 12:50:24 CST ConditionTimestampMonotonic=7996386 AssertTimestamp=Thu 2019-05-16 12:50:24 CST AssertTimestampMonotonic=7996386 Transient=no Perpetual=no StartLimitIntervalUSec=1min StartLimitBurst=3 StartLimitAction=none FailureAction=none SuccessAction=none InvocationID=9aef7007607a4369851851da52548c84 CollectMode=inactive ``` No `kubectl` No `crio` Have `containerd` ## containerd Output of "`containerd --version`": ``` containerd github.com/containerd/containerd 1.2.5 bb71b10fd8f58240ca47fbb579b9d1028eea7c84 ``` Output of "`systemctl show containerd`": ``` Type=simple Restart=no NotifyAccess=none RestartUSec=100ms TimeoutStartUSec=1min 30s TimeoutStopUSec=1min 30s RuntimeMaxUSec=infinity WatchdogUSec=0 WatchdogTimestamp=Thu 2019-05-16 12:50:23 CST WatchdogTimestampMonotonic=7095935 PermissionsStartOnly=no RootDirectoryStartOnly=no RemainAfterExit=no GuessMainPID=yes MainPID=1083 ControlPID=0 FileDescriptorStoreMax=0 NFileDescriptorStore=0 StatusErrno=0 Result=success UID=[not set] GID=[not set] NRestarts=0 ExecMainStartTimestamp=Thu 2019-05-16 12:50:23 CST ExecMainStartTimestampMonotonic=7095897 ExecMainExitTimestampMonotonic=0 ExecMainPID=1083 ExecMainCode=0 ExecMainStatus=0 ExecStartPre={ path=/sbin/modprobe ; argv[]=/sbin/modprobe overlay ; ignore_errors=yes ; start_time=[Thu 2019-05-16 12:50:23 CST] ; stop_time=[Thu 2019-05-16 12:50:23 CST] ; pid=1018 ; code=exited ; status=0 } ExecStart={ path=/usr/bin/containerd ; argv[]=/usr/bin/containerd ; ignore_errors=no ; start_time=[Thu 2019-05-16 12:50:23 CST] ; stop_time=[n/a] ; pid=1083 ; code=(null) ; status=0/0 } Slice=system.slice ControlGroup=/system.slice/containerd.service MemoryCurrent=[not set] CPUUsageNSec=[not set] TasksCurrent=13 IPIngressBytes=18446744073709551615 IPIngressPackets=18446744073709551615 IPEgressBytes=18446744073709551615 IPEgressPackets=18446744073709551615 Delegate=yes DelegateControllers=cpu cpuacct io blkio memory devices pids CPUAccounting=no CPUWeight=[not set] StartupCPUWeight=[not set] CPUShares=[not set] StartupCPUShares=[not set] CPUQuotaPerSecUSec=infinity IOAccounting=no IOWeight=[not set] StartupIOWeight=[not set] BlockIOAccounting=no BlockIOWeight=[not set] StartupBlockIOWeight=[not set] MemoryAccounting=no MemoryLow=0 MemoryHigh=infinity MemoryMax=infinity MemorySwapMax=infinity MemoryLimit=infinity DevicePolicy=auto TasksAccounting=yes TasksMax=infinity IPAccounting=no UMask=0022 LimitCPU=infinity LimitCPUSoft=infinity LimitFSIZE=infinity LimitFSIZESoft=infinity LimitDATA=infinity LimitDATASoft=infinity LimitSTACK=infinity LimitSTACKSoft=8388608 LimitCORE=infinity LimitCORESoft=infinity LimitRSS=infinity LimitRSSSoft=infinity LimitNOFILE=1048576 LimitNOFILESoft=1048576 LimitAS=infinity LimitASSoft=infinity LimitNPROC=infinity LimitNPROCSoft=infinity LimitMEMLOCK=16777216 LimitMEMLOCKSoft=16777216 LimitLOCKS=infinity LimitLOCKSSoft=infinity LimitSIGPENDING=15501 LimitSIGPENDINGSoft=15501 LimitMSGQUEUE=819200 LimitMSGQUEUESoft=819200 LimitNICE=0 LimitNICESoft=0 LimitRTPRIO=0 LimitRTPRIOSoft=0 LimitRTTIME=infinity LimitRTTIMESoft=infinity OOMScoreAdjust=0 Nice=0 IOSchedulingClass=0 IOSchedulingPriority=0 CPUSchedulingPolicy=0 CPUSchedulingPriority=0 TimerSlackNSec=50000 CPUSchedulingResetOnFork=no NonBlocking=no StandardInput=null StandardInputData= StandardOutput=journal StandardError=inherit TTYReset=no TTYVHangup=no TTYVTDisallocate=no SyslogPriority=30 SyslogLevelPrefix=yes SyslogLevel=6 SyslogFacility=3 LogLevelMax=-1 SecureBits=0 CapabilityBoundingSet=cap_chown cap_dac_override cap_dac_read_search cap_fowner cap_fsetid cap_kill cap_setgid cap_setuid cap_setpcap cap_linux_immutable cap_net_bind_service cap_net_broadcast cap_net_admin cap_net_raw cap_ipc_lock cap_ipc_owner cap_sys_module cap_sys_rawio cap_sys_chroot cap_sys_ptrace cap_sys_pacct cap_sys_admin cap_sys_boot cap_sys_nice cap_sys_resource cap_sys_time cap_sys_tty_config cap_mknod cap_lease cap_audit_write cap_audit_control cap_setfcap cap_mac_override cap_mac_admin cap_syslog cap_wake_alarm cap_block_suspend AmbientCapabilities= DynamicUser=no RemoveIPC=no MountFlags= PrivateTmp=no PrivateDevices=no ProtectKernelTunables=no ProtectKernelModules=no ProtectControlGroups=no PrivateNetwork=no PrivateUsers=no ProtectHome=no ProtectSystem=no SameProcessGroup=no UtmpMode=init IgnoreSIGPIPE=yes NoNewPrivileges=no SystemCallErrorNumber=0 LockPersonality=no RuntimeDirectoryPreserve=no RuntimeDirectoryMode=0755 StateDirectoryMode=0755 CacheDirectoryMode=0755 LogsDirectoryMode=0755 ConfigurationDirectoryMode=0755 MemoryDenyWriteExecute=no RestrictRealtime=no RestrictNamespaces=no MountAPIVFS=no KeyringMode=private KillMode=process KillSignal=15 SendSIGKILL=yes SendSIGHUP=no Id=containerd.service Names=containerd.service Requires=sysinit.target system.slice WantedBy=multi-user.target Conflicts=shutdown.target Before=multi-user.target shutdown.target After=sysinit.target basic.target system.slice systemd-journald.socket network.target Documentation=https://containerd.io Description=containerd container runtime LoadState=loaded ActiveState=active SubState=running FragmentPath=/lib/systemd/system/containerd.service UnitFileState=enabled UnitFilePreset=enabled StateChangeTimestamp=Thu 2019-05-16 12:50:23 CST StateChangeTimestampMonotonic=7095938 InactiveExitTimestamp=Thu 2019-05-16 12:50:23 CST InactiveExitTimestampMonotonic=6977390 ActiveEnterTimestamp=Thu 2019-05-16 12:50:23 CST ActiveEnterTimestampMonotonic=7095938 ActiveExitTimestampMonotonic=0 InactiveEnterTimestampMonotonic=0 CanStart=yes CanStop=yes CanReload=no CanIsolate=no StopWhenUnneeded=no RefuseManualStart=no RefuseManualStop=no AllowIsolate=no DefaultDependencies=yes OnFailureJobMode=replace IgnoreOnIsolate=no NeedDaemonReload=no JobTimeoutUSec=infinity JobRunningTimeoutUSec=infinity JobTimeoutAction=none ConditionResult=yes AssertResult=yes ConditionTimestamp=Thu 2019-05-16 12:50:23 CST ConditionTimestampMonotonic=6976545 AssertTimestamp=Thu 2019-05-16 12:50:23 CST AssertTimestampMonotonic=6976549 Transient=no Perpetual=no StartLimitIntervalUSec=10s StartLimitBurst=5 StartLimitAction=none FailureAction=none SuccessAction=none InvocationID=402987e742194552b446050c823b13b0 CollectMode=inactive ``` Output of "`cat /etc/containerd/config.toml`": ``` disabled_plugins = ["cri"] #root = "/var/lib/containerd" #state = "/run/containerd" #subreaper = true #oom_score = 0 #[grpc] # address = "/run/containerd/containerd.sock" # uid = 0 # gid = 0 #[debug] # address = "/run/containerd/debug.sock" # uid = 0 # gid = 0 # level = "info" ``` --- # Packages Have `dpkg` Output of "`dpkg -l|egrep "(cc-oci-runtimecc-runtimerunv|kata-proxy|kata-runtime|kata-shim|kata-ksm-throttler|kata-containers-image|linux-container|qemu-)"`": ``` ii ipxe-qemu-256k-compat-efi-roms 1.0.0+git-20150424.a25a16d-0ubuntu2 all PXE boot firmware - Compat EFI ROM images for qemu ii kata-containers-image 1.7.0~alpha1-20 amd64 Kata containers image ii kata-ksm-throttler 1.7.0~alpha1.git+38c6b40-23 amd64 ii kata-linux-container 4.19.28.33-23 amd64 linux kernel optimised for container-like workloads. ii kata-proxy 1.7.0~alpha1+git.ce8ee71-21 amd64 ii kata-runtime 1.7.0~alpha1+git.717a30b-29 amd64 ii kata-shim 1.7.0~alpha1+git.22a8ac5-19 amd64 ii qemu-block-extra:amd64 1:2.11+dfsg-1ubuntu7.12 amd64 extra block backend modules for qemu-system and qemu-utils ii qemu-kvm 1:2.11+dfsg-1ubuntu7.12 amd64 QEMU Full virtualization on x86 hardware ii qemu-lite 2.11.0+git.87517afd72-24 amd64 linux kernel optimised for container-like workloads. ii qemu-slof 20170724+dfsg-1ubuntu1 all Slimline Open Firmware -- QEMU PowerPC version ii qemu-system 1:2.11+dfsg-1ubuntu7.12 amd64 QEMU full system emulation binaries ii qemu-system-arm 1:2.11+dfsg-1ubuntu7.12 amd64 QEMU full system emulation binaries (arm) ii qemu-system-common 1:2.11+dfsg-1ubuntu7.12 amd64 QEMU full system emulation binaries (common files) ii qemu-system-mips 1:2.11+dfsg-1ubuntu7.12 amd64 QEMU full system emulation binaries (mips) ii qemu-system-misc 1:2.11+dfsg-1ubuntu7.12 amd64 QEMU full system emulation binaries (miscellaneous) ii qemu-system-ppc 1:2.11+dfsg-1ubuntu7.12 amd64 QEMU full system emulation binaries (ppc) ii qemu-system-s390x 1:2.11+dfsg-1ubuntu7.12 amd64 QEMU full system emulation binaries (s390x) ii qemu-system-sparc 1:2.11+dfsg-1ubuntu7.12 amd64 QEMU full system emulation binaries (sparc) ii qemu-system-x86 1:2.11+dfsg-1ubuntu7.12 amd64 QEMU full system emulation binaries (x86) ii qemu-utils 1:2.11+dfsg-1ubuntu7.12 amd64 QEMU utilities ii qemu-vanilla 2.11.2+git.0982a56a55-24 amd64 linux kernel optimised for container-like workloads. ``` Have `rpm` Output of "`rpm -qa|egrep "(cc-oci-runtimecc-runtimerunv|kata-proxy|kata-runtime|kata-shim|kata-ksm-throttler|kata-containers-image|linux-container|qemu-)"`": ``` ``` ---

grahamwhaley commented 5 years ago

Thanks for the report @YLonely /cc @egernst @mcastelino for thoughts.

ZYecho commented 5 years ago

I come into the same problem, could you help take a look at this? @egernst @mcastelino

grahamwhaley commented 5 years ago

Hi @ZYecho , thanks for the report... @YLonely - did you find a solution? Recently I added the pre-requisites section to the Kata Firecracker wiki page https://github.com/kata-containers/documentation/wiki/Initial-release-of-Kata-Containers-with-Firecracker-support#pre-requisuites

I'm going to presume running the kata-fc binary directly still requires the use of a block based storage driver (such as devicemapper?). Can somebody confirm /cc @egernst @sboeuf @mcastelino

YLonely commented 5 years ago

Hi @ZYecho , thanks for the report... @YLonely - did you find a solution? Recently I added the pre-requisites section to the Kata Firecracker wiki page https://github.com/kata-containers/documentation/wiki/Initial-release-of-Kata-Containers-with-Firecracker-support#pre-requisuites

I'm going to presume running the kata-fc binary directly still requires the use of a block based storage driver (such as devicemapper?). Can somebody confirm /cc @egernst @sboeuf @mcastelino

@grahamwhaley I think you are right , it failed because of the wrong storage driver. When i use docker to run a container with kata-fc as the run-time and with the overlay2 as the storage driver, it failed again with the same error message. So i presume that's the reason. Cause i don't know much about the storage driver thing, could you please tell me how to set up the devicemapper for kata-fc? thx a lot.

grahamwhaley commented 5 years ago

I've not done this yet myself @YLonely :-) - I think the others I cc'd in earlier can hopefully help us with some details.

mcastelino commented 5 years ago

@YLonely @grahamwhaley yes fc needs a block based graph driver/snapshotter as it does not support 9p or any host to guest file sharing frameworks.

ustiugov commented 5 years ago

@YLonely The guide on Wiki says that you need to modify the /etc/docker/daemon.json, reload the json and restart docker.

{
  "runtimes": {
    "kata-fc": {
      "path": "/opt/kata/bin/kata-fc"
    },
   "kata-qemu": {
      "path": "/opt/kata/bin/kata-qemu"
    }
  },
  "storage-driver": "devicemapper"
}

@mcastelino As this parameter is global for all kata (in fact docker) configs, do you have the intuition regarding its performance implications for non-FS-heavy applications?

mcastelino commented 5 years ago

@ustiugov yes the graph driver unfortunately is a global setting. In terms of performance if you used devicemapper the single biggest performance related gap is that fact that multiple containers no longer share a common shared page cache.

The performance implications are

So for non fs applications it has no impact besides the impact mentioned above. Even for fs heavy apps you get lower performance only if you had shared page cache across two (or more containers)

/cc @grahamwhaley @devimc

YLonely commented 5 years ago

@YLonely The guide on Wiki says that you need to modify the /etc/docker/daemon.json, reload the json and restart docker.

{
  "runtimes": {
    "kata-fc": {
      "path": "/opt/kata/bin/kata-fc"
    },
   "kata-qemu": {
      "path": "/opt/kata/bin/kata-qemu"
    }
  },
  "storage-driver": "devicemapper"
}

@mcastelino As this parameter is global for all kata (in fact docker) configs, do you have the intuition regarding its performance implications for non-FS-heavy applications?

Maybe you have mistaken what I mean, the wiki tells you how to run kata-fc with docker, but i wanna use just the kata-fc to run a container, am i supposed to modify the daemon.json in this scenario ?

egernst commented 5 years ago

@YLonely if you want to do this, you'll want to make sure the rootfs for the container workload you are bundling is block based, then.

havok4u commented 4 years ago

@grahamwhaley @YLonely @mcastelino I understand that devicemapper is the way to go and overlay2 is problematic, but what are you going to do about this statement fro docker

The devicemapper storage driver is deprecated in Docker Engine 18.09, and will be removed in a future release. It is recommended that users of the devicemapper storage driver migrate to overlay2

I'd like to continue using kata-fc, but it is not looking promising with devicemapper being deprecated and won't be compatible in 18.09. Please advise if any updates on fc working with overlay2 is in the works. Currently I see it still broken.

grahamwhaley commented 4 years ago

@egernst @mcastelino - I presume virtiofs works with fc, and would be our current recommended choice?

havok4u commented 4 years ago

@grahamwhaley that would be great, but when I make the change from overlay2 to virtiofs I get the following

dockerd[41474]: failed to start daemon: error initializing graphdriver: driver not supported

Is that supported in the next version?

grahamwhaley commented 4 years ago

/cc @stefanha @dagrh - I'm not up to speed on virtiofs integration specifics...

stefanha commented 4 years ago

I'm not familiar with the Firecracker codebase, but I don't see a vhost-user-fs device in there: https://github.com/firecracker-microvm/firecracker/

I believe @sboeuf had an experimental vhost-user-fs virtio-mmio device at one point. Maybe this was a Firecracker patch before his work focussed on cloud-hypervisor?

sboeuf commented 4 years ago

@stefanha

I believe @sboeuf had an experimental vhost-user-fs virtio-mmio device at one point. Maybe this was a Firecracker patch before his work focussed on cloud-hypervisor?

Yes you guessed it right :) I had started to work on virtio-fs in the context of Firecracker, but since they didn't want the support for it, I dropped the patch. I don't know if someone's interested but Cloud-Hypervisor supports virtio-fs with both transport layers (virtio-pci and virtio-mmio).