fidencio
commented
3 years ago @htlabc, I'm taking a look at your config and it's not going to work.
[crio.runtime.runtimes.kata-runtime]
runtime_path = "/usr/local/bin/kata-runtime"
runtime_type = "oci"
runtime_root = ""This is not what kata-deploy should provide you, and a lot of work has been going on in order to have everything working on the CRI-O side, such as:
- https://github.com/kata-containers/kata-containers/pull/1423
- https://github.com/cri-o/cri-o/pull/4590
- https://github.com/containerd/containerd/pull/5007
While all of those are not closed, I'd strongly recommend you to use the following configuration for CRI-O:
[crio.runtime.runtimes.kata-runtime]
runtime_path = "/usr/local/bin/containerd-shim-kata-v2"
runtime_type = "vm"
runtime_root = "/run/vc"
privileged_without_host_devices = true
htlabc
liubin
i run a pod test in k8s with kata container 1.13.x apiVersion: v1 kind: Pod metadata: name: nginx-untrusted1 annotations: io.kubernetes.cri.untrusted-workload: "true" spec: containers:
At the beginning, the pod can be accessed normally. I can ping the kata container ip through the ping command nginx-untrusted1 1/1 Running 13 15h 10.244.0.107 deep-optiplex-7070
But after a while, the container becomes abnormal ，Then I checked the log of kata and the error I described above appeared root@deep-OptiPlex-7070:~# ping 10.244.0.107 PING 10.244.0.107 (10.244.0.107) 56(84) bytes of data. From 10.244.0.1 icmp_seq=1 Destination Host Unreachable From 10.244.0.1 icmp_seq=2 Destination Host Unreachable From 10.244.0.1 icmp_seq=3 Destination Host Unreachable
From 10.244.0.1 icmp_seq=4 Destination Host Unreachable From 10.244.0.1 icmp_seq=5 Destination Host Unreachable
I replaced the version of kata with 2.0 root@deep-OptiPlex-7070:~# kata-runtime version : 2.0.1 commit : 3df65f4f3a439c8c7b97dd581f8ff4fc1c877a70 OCI specs: 1.0.1-dev
when i apply a new pod in k8s occur new error kubectl describe pod nginx-untrusted5
Events: Type Reason Age From Message
Normal Scheduled default-scheduler Successfully assigned default/nginx-untrusted5 to deep-optiplex-7070 Warning FailedCreatePodSandBox 3m50s (x42 over 13m) kubelet, deep-optiplex-7070 Failed to create pod sandbox: rpc error: code = Unknown desc = failed to create containerd task: OCI runtime create failed: Invalid command "create": unknown
And I found that kata-runtime v2.x does not have the kata list command , So where can I see the running information of the kata container
root@deep-OptiPlex-7070:/tmp# kata-runtime list Invalid command "list"
Show kata-collect-data.sh details
# Meta details Running `kata-collect-data.sh` version `2.0.1 (commit 3df65f4f3a439c8c7b97dd581f8ff4fc1c877a70)` at `2021-02-25.11:46:07.309401818+0800`. ---
Runtime
Runtime is `/usr/bin/kata-runtime`. # `kata-env`
---
/usr/bin/kata-runtime kata-env
```toml [Meta] Version = "1.0.25" [Runtime] Debug = false Trace = false DisableGuestSeccomp = true DisableNewNetNs = false SandboxCgroupOnly = false Path = "/opt/kata/bin/kata-runtime" [Runtime.Version] OCI = "1.0.1-dev" [Runtime.Version.Version] Semver = "2.0.1" Major = 2 Minor = 0 Patch = 1 Commit = "3df65f4f3a439c8c7b97dd581f8ff4fc1c877a70" [Runtime.Config] Path = "/etc/kata-containers/configuration.toml" [Hypervisor] MachineType = "pc" Version = "QEMU emulator version 5.0.0 (kata-static)\nCopyright (c) 2003-2020 Fabrice Bellard and the QEMU Project developers" Path = "/opt/kata/bin/qemu-system-x86_64" BlockDeviceDriver = "virtio-scsi" EntropySource = "/dev/urandom" SharedFS = "virtio-fs" VirtioFSDaemon = "/opt/kata/libexec/kata-qemu/virtiofsd" Msize9p = 8192 MemorySlots = 10 PCIeRootPort = 0 HotplugVFIOOnRootBus = false Debug = false [Image] Path = "/opt/kata/share/kata-containers/kata-containers-image_clearlinux_2.0.1_agent_3df65f4f3a.img" [Kernel] Path = "/opt/kata/share/kata-containers/vmlinux-5.4.71-84" Parameters = "systemd.unit=kata-containers.target systemd.mask=systemd-networkd.service systemd.mask=systemd-networkd.socket scsi_mod.scan=none" [Initrd] Path = "" [Agent] Debug = false Trace = false TraceMode = "" TraceType = "" [Host] Kernel = "5.4.0-65-generic" Architecture = "amd64" VMContainerCapable = true SupportVSocks = true [Host.Distro] Name = "Ubuntu" Version = "18.04" [Host.CPU] Vendor = "GenuineIntel" Model = "Intel(R) Core(TM) i7-9700 CPU @ 3.00GHz" CPUs = 8 [Host.Memory] Total = 32664248 Free = 27396476 Available = 28679996 [Netmon] Path = "/opt/kata/libexec/kata-containers/kata-netmon" Debug = false Enable = false [Netmon.Version] Semver = "2.0.1" Major = 2 Minor = 0 Patch = 1 Commit = "<>"
```
Runtime config files
# Runtime config files ## Runtime default config files ``` /etc/kata-containers/configuration.toml /opt/kata/share/defaults/kata-containers/configuration.toml ``` ## Runtime config file contents
Config file `/usr/share/defaults/kata-containers/configuration.toml` not found
---
cat "/etc/kata-containers/configuration.toml"
```toml # Copyright (c) 2017-2019 Intel Corporation # # SPDX-License-Identifier: Apache-2.0 # # XXX: WARNING: this file is auto-generated. # XXX: # XXX: Source file: "cli/config/configuration-qemu.toml.in" # XXX: Project: # XXX: Name: Kata Containers # XXX: Type: kata [hypervisor.qemu] path = "/opt/kata/bin/qemu-system-x86_64" kernel = "/opt/kata/share/kata-containers/vmlinux.container" image = "/opt/kata/share/kata-containers/kata-containers.img" machine_type = "pc" # List of valid annotation names for the hypervisor # Each member of the list is a regular expression, which is the base name # of the annotation, e.g. "path" for io.katacontainers.config.hypervisor.path" enable_annotations = [] # List of valid annotations values for the hypervisor # Each member of the list is a path pattern as described by glob(3). # The default if not set is empty (all annotations rejected.) # Your distribution recommends: ["/opt/kata/bin/qemu-system-x86_64"] valid_hypervisor_paths = ["/opt/kata/bin/qemu-system-x86_64"] # Optional space-separated list of options to pass to the guest kernel. # For example, use `kernel_params = "vsyscall=emulate"` if you are having # trouble running pre-2.15 glibc. # # WARNING: - any parameter specified here will take priority over the default # parameter value of the same name used to start the virtual machine. # Do not set values here unless you understand the impact of doing so as you # may stop the virtual machine from booting. # To see the list of default parameters, enable hypervisor debug, create a # container and look for 'default-kernel-parameters' log entries. kernel_params = "" # Path to the firmware. # If you want that qemu uses the default firmware leave this option empty firmware = "" # Machine accelerators # comma-separated list of machine accelerators to pass to the hypervisor. # For example, `machine_accelerators = "nosmm,nosmbus,nosata,nopit,static-prt,nofw"` machine_accelerators="" # CPU features # comma-separated list of cpu features to pass to the cpu # For example, `cpu_features = "pmu=off,vmx=off" cpu_features="pmu=off" # Default number of vCPUs per SB/VM: # unspecified or 0 --> will be set to 1 # < 0 --> will be set to the actual number of physical cores # > 0 <= number of physical cores --> will be set to the specified number # > number of physical cores --> will be set to the actual number of physical cores default_vcpus = 1 # Default maximum number of vCPUs per SB/VM: # unspecified or == 0 --> will be set to the actual number of physical cores or to the maximum number # of vCPUs supported by KVM if that number is exceeded # > 0 <= number of physical cores --> will be set to the specified number # > number of physical cores --> will be set to the actual number of physical cores or to the maximum number # of vCPUs supported by KVM if that number is exceeded # WARNING: Depending of the architecture, the maximum number of vCPUs supported by KVM is used when # the actual number of physical cores is greater than it. # WARNING: Be aware that this value impacts the virtual machine's memory footprint and CPU # the hotplug functionality. For example, `default_maxvcpus = 240` specifies that until 240 vCPUs # can be added to a SB/VM, but the memory footprint will be big. Another example, with # `default_maxvcpus = 8` the memory footprint will be small, but 8 will be the maximum number of # vCPUs supported by the SB/VM. In general, we recommend that you do not edit this variable, # unless you know what are you doing. # NOTICE: on arm platform with gicv2 interrupt controller, set it to 8. default_maxvcpus = 0 # Bridges can be used to hot plug devices. # Limitations: # * Currently only pci bridges are supported # * Until 30 devices per bridge can be hot plugged. # * Until 5 PCI bridges can be cold plugged per VM. # This limitation could be a bug in qemu or in the kernel # Default number of bridges per SB/VM: # unspecified or 0 --> will be set to 1 # > 1 <= 5 --> will be set to the specified number # > 5 --> will be set to 5 default_bridges = 1 # Default memory size in MiB for SB/VM. # If unspecified then it will be set 2048 MiB. default_memory = 2048 # # Default memory slots per SB/VM. # If unspecified then it will be set 10. # This is will determine the times that memory will be hotadded to sandbox/VM. #memory_slots = 10 # The size in MiB will be plused to max memory of hypervisor. # It is the memory address space for the NVDIMM devie. # If set block storage driver (block_device_driver) to "nvdimm", # should set memory_offset to the size of block device. # Default 0 #memory_offset = 0 # Specifies virtio-mem will be enabled or not. # Please note that this option should be used with the command # "echo 1 > /proc/sys/vm/overcommit_memory". # Default false #enable_virtio_mem = true # Disable block device from being used for a container's rootfs. # In case of a storage driver like devicemapper where a container's # root file system is backed by a block device, the block device is passed # directly to the hypervisor for performance reasons. # This flag prevents the block device from being passed to the hypervisor, # 9pfs is used instead to pass the rootfs. disable_block_device_use = false # Shared file system type: # - virtio-9p (default) # - virtio-fs shared_fs = "virtio-fs" # Path to vhost-user-fs daemon. virtio_fs_daemon = "/opt/kata/libexec/kata-qemu/virtiofsd" # List of valid annotations values for the virtiofs daemon # The default if not set is empty (all annotations rejected.) # Your distribution recommends: ["/opt/kata/libexec/kata-qemu/virtiofsd"] valid_virtio_fs_daemon_paths = ["/opt/kata/libexec/kata-qemu/virtiofsd"] # Default size of DAX cache in MiB virtio_fs_cache_size = 0 # Extra args for virtiofsd daemon # # Format example: # ["-o", "arg1=xxx,arg2", "-o", "hello world", "--arg3=yyy"] # # see `virtiofsd -h` for possible options. virtio_fs_extra_args = ["--thread-pool-size=1"] # Cache mode: # # - none # Metadata, data, and pathname lookup are not cached in guest. They are # always fetched from host and any changes are immediately pushed to host. # # - auto # Metadata and pathname lookup cache expires after a configured amount of # time (default is 1 second). Data is cached while the file is open (close # to open consistency). # # - always # Metadata, data, and pathname lookup are cached in guest and never expire. virtio_fs_cache = "auto" # Block storage driver to be used for the hypervisor in case the container # rootfs is backed by a block device. This is virtio-scsi, virtio-blk # or nvdimm. block_device_driver = "virtio-scsi" # Specifies cache-related options will be set to block devices or not. # Default false #block_device_cache_set = true # Specifies cache-related options for block devices. # Denotes whether use of O_DIRECT (bypass the host page cache) is enabled. # Default false #block_device_cache_direct = true # Specifies cache-related options for block devices. # Denotes whether flush requests for the device are ignored. # Default false #block_device_cache_noflush = true # Enable iothreads (data-plane) to be used. This causes IO to be # handled in a separate IO thread. This is currently only implemented # for SCSI. # enable_iothreads = false # Enable pre allocation of VM RAM, default false # Enabling this will result in lower container density # as all of the memory will be allocated and locked # This is useful when you want to reserve all the memory # upfront or in the cases where you want memory latencies # to be very predictable # Default false #enable_mem_prealloc = true # Enable huge pages for VM RAM, default false # Enabling this will result in the VM memory # being allocated using huge pages. # This is useful when you want to use vhost-user network # stacks within the container. This will automatically # result in memory pre allocation #enable_hugepages = true # Enable vhost-user storage device, default false # Enabling this will result in some Linux reserved block type # major range 240-254 being chosen to represent vhost-user devices. enable_vhost_user_store = false # The base directory specifically used for vhost-user devices. # Its sub-path "block" is used for block devices; "block/sockets" is # where we expect vhost-user sockets to live; "block/devices" is where # simulated block device nodes for vhost-user devices to live. vhost_user_store_path = "/var/run/kata-containers/vhost-user" # Enable vIOMMU, default false # Enabling this will result in the VM having a vIOMMU device # This will also add the following options to the kernel's # command line: intel_iommu=on,iommu=pt #enable_iommu = true # Enable IOMMU_PLATFORM, default false # Enabling this will result in the VM device having iommu_platform=on set #enable_iommu_platform = true # List of valid annotations values for the vhost user store path # The default if not set is empty (all annotations rejected.) # Your distribution recommends: ["/var/run/kata-containers/vhost-user"] valid_vhost_user_store_paths = ["/var/run/kata-containers/vhost-user"] # Enable file based guest memory support. The default is an empty string which # will disable this feature. In the case of virtio-fs, this is enabled # automatically and '/dev/shm' is used as the backing folder. # This option will be ignored if VM templating is enabled. #file_mem_backend = "" # List of valid annotations values for the file_mem_backend annotation # The default if not set is empty (all annotations rejected.) # Your distribution recommends: [""] valid_file_mem_backends = [""] # Enable swap of vm memory. Default false. # The behaviour is undefined if mem_prealloc is also set to true #enable_swap = true # This option changes the default hypervisor and kernel parameters # to enable debug output where available. # # Default false #enable_debug = true # Disable the customizations done in the runtime when it detects # that it is running on top a VMM. This will result in the runtime # behaving as it would when running on bare metal. # #disable_nesting_checks = true # This is the msize used for 9p shares. It is the number of bytes # used for 9p packet payload. #msize_9p = 8192 # If false and nvdimm is supported, use nvdimm device to plug guest image. # Otherwise virtio-block device is used. # Default is false #disable_image_nvdimm = true # VFIO devices are hotplugged on a bridge by default. # Enable hotplugging on root bus. This may be required for devices with # a large PCI bar, as this is a current limitation with hotplugging on # a bridge. This value is valid for "pc" machine type. # Default false #hotplug_vfio_on_root_bus = true # Before hot plugging a PCIe device, you need to add a pcie_root_port device. # Use this parameter when using some large PCI bar devices, such as Nvidia GPU # The value means the number of pcie_root_port # This value is valid when hotplug_vfio_on_root_bus is true and machine_type is "q35" # Default 0 #pcie_root_port = 2 # If vhost-net backend for virtio-net is not desired, set to true. Default is false, which trades off # security (vhost-net runs ring0) for network I/O performance. #disable_vhost_net = true # # Default entropy source. # The path to a host source of entropy (including a real hardware RNG) # /dev/urandom and /dev/random are two main options. # Be aware that /dev/random is a blocking source of entropy. If the host # runs out of entropy, the VMs boot time will increase leading to get startup # timeouts. # The source of entropy /dev/urandom is non-blocking and provides a # generally acceptable source of entropy. It should work well for pretty much # all practical purposes. #entropy_source= "/dev/urandom" # Path to OCI hook binaries in the *guest rootfs*. # This does not affect host-side hooks which must instead be added to # the OCI spec passed to the runtime. # # You can create a rootfs with hooks by customizing the osbuilder scripts: # https://github.com/kata-containers/osbuilder # # Hooks must be stored in a subdirectory of guest_hook_path according to their # hook type, i.e. "guest_hook_path/{prestart,postart,poststop}". # The agent will scan these directories for executable files and add them, in # lexicographical order, to the lifecycle of the guest container. # Hooks are executed in the runtime namespace of the guest. See the official documentation: # https://github.com/opencontainers/runtime-spec/blob/v1.0.1/config.md#posix-platform-hooks # Warnings will be logged if any error is encountered will scanning for hooks, # but it will not abort container execution. #guest_hook_path = "/usr/share/oci/hooks" # # Use rx Rate Limiter to control network I/O inbound bandwidth(size in bits/sec for SB/VM). # In Qemu, we use classful qdiscs HTB(Hierarchy Token Bucket) to discipline traffic. # Default 0-sized value means unlimited rate. #rx_rate_limiter_max_rate = 0 # Use tx Rate Limiter to control network I/O outbound bandwidth(size in bits/sec for SB/VM). # In Qemu, we use classful qdiscs HTB(Hierarchy Token Bucket) and ifb(Intermediate Functional Block) # to discipline traffic. # Default 0-sized value means unlimited rate. #tx_rate_limiter_max_rate = 0 [factory] # VM templating support. Once enabled, new VMs are created from template # using vm cloning. They will share the same initial kernel, initramfs and # agent memory by mapping it readonly. It helps speeding up new container # creation and saves a lot of memory if there are many kata containers running # on the same host. # # When disabled, new VMs are created from scratch. # # Note: Requires "initrd=" to be set ("image=" is not supported). # # Default false #enable_template = true # Specifies the path of template. # # Default "/run/vc/vm/template" #template_path = "/run/vc/vm/template" # The number of caches of VMCache: # unspecified or == 0 --> VMCache is disabled # > 0 --> will be set to the specified number # # VMCache is a function that creates VMs as caches before using it. # It helps speed up new container creation. # The function consists of a server and some clients communicating # through Unix socket. The protocol is gRPC in protocols/cache/cache.proto. # The VMCache server will create some VMs and cache them by factory cache. # It will convert the VM to gRPC format and transport it when gets # requestion from clients. # Factory grpccache is the VMCache client. It will request gRPC format # VM and convert it back to a VM. If VMCache function is enabled, # kata-runtime will request VM from factory grpccache when it creates # a new sandbox. # # Default 0 #vm_cache_number = 0 # Specify the address of the Unix socket that is used by VMCache. # # Default /var/run/kata-containers/cache.sock #vm_cache_endpoint = "/var/run/kata-containers/cache.sock" [agent.kata] # If enabled, make the agent display debug-level messages. # (default: disabled) #enable_debug = true # Enable agent tracing. # # If enabled, the default trace mode is "dynamic" and the # default trace type is "isolated". The trace mode and type are set # explicity with the `trace_type=` and `trace_mode=` options. # # Notes: # # - Tracing is ONLY enabled when `enable_tracing` is set: explicitly # setting `trace_mode=` and/or `trace_type=` without setting `enable_tracing` # will NOT activate agent tracing. # # - See https://github.com/kata-containers/agent/blob/master/TRACING.md for # full details. # # (default: disabled) #enable_tracing = true # #trace_mode = "dynamic" #trace_type = "isolated" # Comma separated list of kernel modules and their parameters. # These modules will be loaded in the guest kernel using modprobe(8). # The following example can be used to load two kernel modules with parameters # - kernel_modules=["e1000e InterruptThrottleRate=3000,3000,3000 EEE=1", "i915 enable_ppgtt=0"] # The first word is considered as the module name and the rest as its parameters. # Container will not be started when: # * A kernel module is specified and the modprobe command is not installed in the guest # or it fails loading the module. # * The module is not available in the guest or it doesn't met the guest kernel # requirements, like architecture and version. # kernel_modules=[] # Enable debug console. # If enabled, user can connect guest OS running inside hypervisor # through "kata-runtime exec" command
#debug_console_enabled = true
[netmon]
# If enabled, the network monitoring process gets started when the
# sandbox is created. This allows for the detection of some additional
# network being added to the existing network namespace, after the
# sandbox has been created.
# (default: disabled)
#enable_netmon = true
# Specify the path to the netmon binary.
path = "/opt/kata/libexec/kata-containers/kata-netmon"
# If enabled, netmon messages will be sent to the system log
# (default: disabled)
#enable_debug = true
[runtime]
# If enabled, the runtime will log additional debug messages to the
# system log
# (default: disabled)
#enable_debug = true
#
# Internetworking model
# Determines how the VM should be connected to the
# the container network interface
# Options:
#
# - macvtap
# Used when the Container network interface can be bridged using
# macvtap.
#
# - none
# Used when customize network. Only creates a tap device. No veth pair.
#
# - tcfilter
# Uses tc filter rules to redirect traffic from the network interface
# provided by plugin to a tap interface connected to the VM.
#
internetworking_model="tcfilter"
# disable guest seccomp
# Determines whether container seccomp profiles are passed to the virtual
# machine and applied by the kata agent. If set to true, seccomp is not applied
# within the guest
# (default: true)
disable_guest_seccomp=true
# If enabled, the runtime will create opentracing.io traces and spans.
# (See https://www.jaegertracing.io/docs/getting-started).
# (default: disabled)
#enable_tracing = true
# If enabled, the runtime will not create a network namespace for shim and hypervisor processes.
# This option may have some potential impacts to your host. It should only be used when you know what you're doing.
# `disable_new_netns` conflicts with `enable_netmon`
# `disable_new_netns` conflicts with `internetworking_model=tcfilter` and `internetworking_model=macvtap`. It works only
# with `internetworking_model=none`. The tap device will be in the host network namespace and can connect to a bridge
# (like OVS) directly.
# If you are using docker, `disable_new_netns` only works with `docker run --net=none`
# (default: false)
#disable_new_netns = true
# if enabled, the runtime will add all the kata processes inside one dedicated cgroup.
# The container cgroups in the host are not created, just one single cgroup per sandbox.
# The runtime caller is free to restrict or collect cgroup stats of the overall Kata sandbox.
# The sandbox cgroup path is the parent cgroup of a container with the PodSandbox annotation.
# The sandbox cgroup is constrained if there is no container type annotation.
# See: https://godoc.org/github.com/kata-containers/runtime/virtcontainers#ContainerType
sandbox_cgroup_only=false
# Enabled experimental feature list, format: ["a", "b"].
# Experimental features are features not stable enough for production,
# they may break compatibility, and are prepared for a big version bump.
# Supported experimental features:
# (default: [])
experimental=[]
# If enabled, user can run pprof tools with shim v2 process through kata-monitor.
# (default: false)
# EnablePprof = true
```
cat "/opt/kata/share/defaults/kata-containers/configuration.toml"
```toml # Copyright (c) 2017-2019 Intel Corporation # # SPDX-License-Identifier: Apache-2.0 # # XXX: WARNING: this file is auto-generated. # XXX: # XXX: Source file: "cli/config/configuration-qemu.toml.in" # XXX: Project: # XXX: Name: Kata Containers # XXX: Type: kata [hypervisor.qemu] path = "/opt/kata/bin/qemu-system-x86_64" kernel = "/opt/kata/share/kata-containers/vmlinux.container" image = "/opt/kata/share/kata-containers/kata-containers.img" machine_type = "pc" # List of valid annotation names for the hypervisor # Each member of the list is a regular expression, which is the base name # of the annotation, e.g. "path" for io.katacontainers.config.hypervisor.path" enable_annotations = [] # List of valid annotations values for the hypervisor # Each member of the list is a path pattern as described by glob(3). # The default if not set is empty (all annotations rejected.) # Your distribution recommends: ["/opt/kata/bin/qemu-system-x86_64"] valid_hypervisor_paths = ["/opt/kata/bin/qemu-system-x86_64"] # Optional space-separated list of options to pass to the guest kernel. # For example, use `kernel_params = "vsyscall=emulate"` if you are having # trouble running pre-2.15 glibc. # # WARNING: - any parameter specified here will take priority over the default # parameter value of the same name used to start the virtual machine. # Do not set values here unless you understand the impact of doing so as you # may stop the virtual machine from booting. # To see the list of default parameters, enable hypervisor debug, create a # container and look for 'default-kernel-parameters' log entries. kernel_params = "" # Path to the firmware. # If you want that qemu uses the default firmware leave this option empty firmware = "" # Machine accelerators # comma-separated list of machine accelerators to pass to the hypervisor. # For example, `machine_accelerators = "nosmm,nosmbus,nosata,nopit,static-prt,nofw"` machine_accelerators="" # CPU features # comma-separated list of cpu features to pass to the cpu # For example, `cpu_features = "pmu=off,vmx=off" cpu_features="pmu=off" # Default number of vCPUs per SB/VM: # unspecified or 0 --> will be set to 1 # < 0 --> will be set to the actual number of physical cores # > 0 <= number of physical cores --> will be set to the specified number # > number of physical cores --> will be set to the actual number of physical cores default_vcpus = 1 # Default maximum number of vCPUs per SB/VM: # unspecified or == 0 --> will be set to the actual number of physical cores or to the maximum number # of vCPUs supported by KVM if that number is exceeded # > 0 <= number of physical cores --> will be set to the specified number # > number of physical cores --> will be set to the actual number of physical cores or to the maximum number # of vCPUs supported by KVM if that number is exceeded # WARNING: Depending of the architecture, the maximum number of vCPUs supported by KVM is used when # the actual number of physical cores is greater than it. # WARNING: Be aware that this value impacts the virtual machine's memory footprint and CPU # the hotplug functionality. For example, `default_maxvcpus = 240` specifies that until 240 vCPUs # can be added to a SB/VM, but the memory footprint will be big. Another example, with # `default_maxvcpus = 8` the memory footprint will be small, but 8 will be the maximum number of # vCPUs supported by the SB/VM. In general, we recommend that you do not edit this variable, # unless you know what are you doing. # NOTICE: on arm platform with gicv2 interrupt controller, set it to 8. default_maxvcpus = 0 # Bridges can be used to hot plug devices. # Limitations: # * Currently only pci bridges are supported # * Until 30 devices per bridge can be hot plugged. # * Until 5 PCI bridges can be cold plugged per VM. # This limitation could be a bug in qemu or in the kernel # Default number of bridges per SB/VM: # unspecified or 0 --> will be set to 1 # > 1 <= 5 --> will be set to the specified number # > 5 --> will be set to 5 default_bridges = 1 # Default memory size in MiB for SB/VM. # If unspecified then it will be set 2048 MiB. default_memory = 2048 # # Default memory slots per SB/VM. # If unspecified then it will be set 10. # This is will determine the times that memory will be hotadded to sandbox/VM. #memory_slots = 10 # The size in MiB will be plused to max memory of hypervisor. # It is the memory address space for the NVDIMM devie. # If set block storage driver (block_device_driver) to "nvdimm", # should set memory_offset to the size of block device. # Default 0 #memory_offset = 0 # Specifies virtio-mem will be enabled or not. # Please note that this option should be used with the command # "echo 1 > /proc/sys/vm/overcommit_memory". # Default false #enable_virtio_mem = true # Disable block device from being used for a container's rootfs. # In case of a storage driver like devicemapper where a container's # root file system is backed by a block device, the block device is passed # directly to the hypervisor for performance reasons. # This flag prevents the block device from being passed to the hypervisor, # 9pfs is used instead to pass the rootfs. disable_block_device_use = false # Shared file system type: # - virtio-9p (default) # - virtio-fs shared_fs = "virtio-fs" # Path to vhost-user-fs daemon. virtio_fs_daemon = "/opt/kata/libexec/kata-qemu/virtiofsd" # List of valid annotations values for the virtiofs daemon # The default if not set is empty (all annotations rejected.) # Your distribution recommends: ["/opt/kata/libexec/kata-qemu/virtiofsd"] valid_virtio_fs_daemon_paths = ["/opt/kata/libexec/kata-qemu/virtiofsd"] # Default size of DAX cache in MiB virtio_fs_cache_size = 0 # Extra args for virtiofsd daemon # # Format example: # ["-o", "arg1=xxx,arg2", "-o", "hello world", "--arg3=yyy"] # # see `virtiofsd -h` for possible options. virtio_fs_extra_args = ["--thread-pool-size=1"] # Cache mode: # # - none # Metadata, data, and pathname lookup are not cached in guest. They are # always fetched from host and any changes are immediately pushed to host. # # - auto # Metadata and pathname lookup cache expires after a configured amount of # time (default is 1 second). Data is cached while the file is open (close # to open consistency). # # - always # Metadata, data, and pathname lookup are cached in guest and never expire. virtio_fs_cache = "auto" # Block storage driver to be used for the hypervisor in case the container # rootfs is backed by a block device. This is virtio-scsi, virtio-blk # or nvdimm. block_device_driver = "virtio-scsi" # Specifies cache-related options will be set to block devices or not. # Default false #block_device_cache_set = true # Specifies cache-related options for block devices. # Denotes whether use of O_DIRECT (bypass the host page cache) is enabled. # Default false #block_device_cache_direct = true # Specifies cache-related options for block devices. # Denotes whether flush requests for the device are ignored. # Default false #block_device_cache_noflush = true # Enable iothreads (data-plane) to be used. This causes IO to be # handled in a separate IO thread. This is currently only implemented # for SCSI. # enable_iothreads = false # Enable pre allocation of VM RAM, default false # Enabling this will result in lower container density # as all of the memory will be allocated and locked # This is useful when you want to reserve all the memory # upfront or in the cases where you want memory latencies # to be very predictable # Default false #enable_mem_prealloc = true # Enable huge pages for VM RAM, default false # Enabling this will result in the VM memory # being allocated using huge pages. # This is useful when you want to use vhost-user network # stacks within the container. This will automatically # result in memory pre allocation #enable_hugepages = true # Enable vhost-user storage device, default false # Enabling this will result in some Linux reserved block type # major range 240-254 being chosen to represent vhost-user devices. enable_vhost_user_store = false # The base directory specifically used for vhost-user devices. # Its sub-path "block" is used for block devices; "block/sockets" is # where we expect vhost-user sockets to live; "block/devices" is where # simulated block device nodes for vhost-user devices to live. vhost_user_store_path = "/var/run/kata-containers/vhost-user" # Enable vIOMMU, default false # Enabling this will result in the VM having a vIOMMU device # This will also add the following options to the kernel's # command line: intel_iommu=on,iommu=pt #enable_iommu = true # Enable IOMMU_PLATFORM, default false # Enabling this will result in the VM device having iommu_platform=on set #enable_iommu_platform = true # List of valid annotations values for the vhost user store path # The default if not set is empty (all annotations rejected.) # Your distribution recommends: ["/var/run/kata-containers/vhost-user"] valid_vhost_user_store_paths = ["/var/run/kata-containers/vhost-user"] # Enable file based guest memory support. The default is an empty string which # will disable this feature. In the case of virtio-fs, this is enabled # automatically and '/dev/shm' is used as the backing folder. # This option will be ignored if VM templating is enabled. #file_mem_backend = "" # List of valid annotations values for the file_mem_backend annotation # The default if not set is empty (all annotations rejected.) # Your distribution recommends: [""] valid_file_mem_backends = [""] # Enable swap of vm memory. Default false. # The behaviour is undefined if mem_prealloc is also set to true #enable_swap = true # This option changes the default hypervisor and kernel parameters # to enable debug output where available. # # Default false #enable_debug = true # Disable the customizations done in the runtime when it detects # that it is running on top a VMM. This will result in the runtime # behaving as it would when running on bare metal. # #disable_nesting_checks = true # This is the msize used for 9p shares. It is the number of bytes # used for 9p packet payload. #msize_9p = 8192 # If false and nvdimm is supported, use nvdimm device to plug guest image. # Otherwise virtio-block device is used. # Default is false #disable_image_nvdimm = true # VFIO devices are hotplugged on a bridge by default. # Enable hotplugging on root bus. This may be required for devices with # a large PCI bar, as this is a current limitation with hotplugging on # a bridge. This value is valid for "pc" machine type. # Default false #hotplug_vfio_on_root_bus = true # Before hot plugging a PCIe device, you need to add a pcie_root_port device. # Use this parameter when using some large PCI bar devices, such as Nvidia GPU # The value means the number of pcie_root_port # This value is valid when hotplug_vfio_on_root_bus is true and machine_type is "q35" # Default 0 #pcie_root_port = 2 # If vhost-net backend for virtio-net is not desired, set to true. Default is false, which trades off # security (vhost-net runs ring0) for network I/O performance. #disable_vhost_net = true # # Default entropy source. # The path to a host source of entropy (including a real hardware RNG) # /dev/urandom and /dev/random are two main options. # Be aware that /dev/random is a blocking source of entropy. If the host # runs out of entropy, the VMs boot time will increase leading to get startup # timeouts. # The source of entropy /dev/urandom is non-blocking and provides a # generally acceptable source of entropy. It should work well for pretty much # all practical purposes. #entropy_source= "/dev/urandom" # Path to OCI hook binaries in the *guest rootfs*. # This does not affect host-side hooks which must instead be added to # the OCI spec passed to the runtime. # # You can create a rootfs with hooks by customizing the osbuilder scripts: # https://github.com/kata-containers/osbuilder # # Hooks must be stored in a subdirectory of guest_hook_path according to their # hook type, i.e. "guest_hook_path/{prestart,postart,poststop}". # The agent will scan these directories for executable files and add them, in # lexicographical order, to the lifecycle of the guest container. # Hooks are executed in the runtime namespace of the guest. See the official documentation: # https://github.com/opencontainers/runtime-spec/blob/v1.0.1/config.md#posix-platform-hooks # Warnings will be logged if any error is encountered will scanning for hooks, # but it will not abort container execution. #guest_hook_path = "/usr/share/oci/hooks" # # Use rx Rate Limiter to control network I/O inbound bandwidth(size in bits/sec for SB/VM). # In Qemu, we use classful qdiscs HTB(Hierarchy Token Bucket) to discipline traffic. # Default 0-sized value means unlimited rate. #rx_rate_limiter_max_rate = 0 # Use tx Rate Limiter to control network I/O outbound bandwidth(size in bits/sec for SB/VM). # In Qemu, we use classful qdiscs HTB(Hierarchy Token Bucket) and ifb(Intermediate Functional Block) # to discipline traffic. # Default 0-sized value means unlimited rate. #tx_rate_limiter_max_rate = 0 [factory] # VM templating support. Once enabled, new VMs are created from template # using vm cloning. They will share the same initial kernel, initramfs and # agent memory by mapping it readonly. It helps speeding up new container # creation and saves a lot of memory if there are many kata containers running # on the same host. # # When disabled, new VMs are created from scratch. # # Note: Requires "initrd=" to be set ("image=" is not supported). # # Default false #enable_template = true # Specifies the path of template. # # Default "/run/vc/vm/template" #template_path = "/run/vc/vm/template" # The number of caches of VMCache: # unspecified or == 0 --> VMCache is disabled # > 0 --> will be set to the specified number # # VMCache is a function that creates VMs as caches before using it. # It helps speed up new container creation. # The function consists of a server and some clients communicating # through Unix socket. The protocol is gRPC in protocols/cache/cache.proto. # The VMCache server will create some VMs and cache them by factory cache. # It will convert the VM to gRPC format and transport it when gets # requestion from clients. # Factory grpccache is the VMCache client. It will request gRPC format # VM and convert it back to a VM. If VMCache function is enabled, # kata-runtime will request VM from factory grpccache when it creates # a new sandbox. # # Default 0 #vm_cache_number = 0 # Specify the address of the Unix socket that is used by VMCache. # # Default /var/run/kata-containers/cache.sock #vm_cache_endpoint = "/var/run/kata-containers/cache.sock" [agent.kata] # If enabled, make the agent display debug-level messages. # (default: disabled) #enable_debug = true # Enable agent tracing. # # If enabled, the default trace mode is "dynamic" and the # default trace type is "isolated". The trace mode and type are set # explicity with the `trace_type=` and `trace_mode=` options. # # Notes: # # - Tracing is ONLY enabled when `enable_tracing` is set: explicitly # setting `trace_mode=` and/or `trace_type=` without setting `enable_tracing` # will NOT activate agent tracing. # # - See https://github.com/kata-containers/agent/blob/master/TRACING.md for # full details. # # (default: disabled) #enable_tracing = true # #trace_mode = "dynamic" #trace_type = "isolated" # Comma separated list of kernel modules and their parameters. # These modules will be loaded in the guest kernel using modprobe(8). # The following example can be used to load two kernel modules with parameters # - kernel_modules=["e1000e InterruptThrottleRate=3000,3000,3000 EEE=1", "i915 enable_ppgtt=0"] # The first word is considered as the module name and the rest as its parameters. # Container will not be started when: # * A kernel module is specified and the modprobe command is not installed in the guest # or it fails loading the module. # * The module is not available in the guest or it doesn't met the guest kernel # requirements, like architecture and version. # kernel_modules=[] # Enable debug console. # If enabled, user can connect guest OS running inside hypervisor # through "kata-runtime exec" command
#debug_console_enabled = true
[netmon]
# If enabled, the network monitoring process gets started when the
# sandbox is created. This allows for the detection of some additional
# network being added to the existing network namespace, after the
# sandbox has been created.
# (default: disabled)
#enable_netmon = true
# Specify the path to the netmon binary.
path = "/opt/kata/libexec/kata-containers/kata-netmon"
# If enabled, netmon messages will be sent to the system log
# (default: disabled)
#enable_debug = true
[runtime]
# If enabled, the runtime will log additional debug messages to the
# system log
# (default: disabled)
#enable_debug = true
#
# Internetworking model
# Determines how the VM should be connected to the
# the container network interface
# Options:
#
# - macvtap
# Used when the Container network interface can be bridged using
# macvtap.
#
# - none
# Used when customize network. Only creates a tap device. No veth pair.
#
# - tcfilter
# Uses tc filter rules to redirect traffic from the network interface
# provided by plugin to a tap interface connected to the VM.
#
internetworking_model="tcfilter"
# disable guest seccomp
# Determines whether container seccomp profiles are passed to the virtual
# machine and applied by the kata agent. If set to true, seccomp is not applied
# within the guest
# (default: true)
disable_guest_seccomp=true
# If enabled, the runtime will create opentracing.io traces and spans.
# (See https://www.jaegertracing.io/docs/getting-started).
# (default: disabled)
#enable_tracing = true
# If enabled, the runtime will not create a network namespace for shim and hypervisor processes.
# This option may have some potential impacts to your host. It should only be used when you know what you're doing.
# `disable_new_netns` conflicts with `enable_netmon`
# `disable_new_netns` conflicts with `internetworking_model=tcfilter` and `internetworking_model=macvtap`. It works only
# with `internetworking_model=none`. The tap device will be in the host network namespace and can connect to a bridge
# (like OVS) directly.
# If you are using docker, `disable_new_netns` only works with `docker run --net=none`
# (default: false)
#disable_new_netns = true
# if enabled, the runtime will add all the kata processes inside one dedicated cgroup.
# The container cgroups in the host are not created, just one single cgroup per sandbox.
# The runtime caller is free to restrict or collect cgroup stats of the overall Kata sandbox.
# The sandbox cgroup path is the parent cgroup of a container with the PodSandbox annotation.
# The sandbox cgroup is constrained if there is no container type annotation.
# See: https://godoc.org/github.com/kata-containers/runtime/virtcontainers#ContainerType
sandbox_cgroup_only=false
# Enabled experimental feature list, format: ["a", "b"].
# Experimental features are features not stable enough for production,
# they may break compatibility, and are prepared for a big version bump.
# Supported experimental features:
# (default: [])
experimental=[]
# If enabled, user can run pprof tools with shim v2 process through kata-monitor.
# (default: false)
# EnablePprof = true
```
Containerd shim v2
Containerd shim v2 is `/usr/bin/containerd-shim-kata-v2`.
---
containerd-shim-kata-v2 --version
``` Kata Containers containerd shim: id: "io.containerd.kata.v2", version: 2.0.1, commit: 3df65f4f3a439c8c7b97dd581f8ff4fc1c877a70 ```
KSM throttler
# KSM throttler ## version
## systemd service
/usr/libexec/kata-ksm-throttler/kata-ksm-throttler --version
``` kata-ksm-throttler version 1.13.0-alpha0-aaf0821 ```
Image details
# Image details ```yaml --- osbuilder: url: "https://github.com/kata-containers/kata-containers/tools/osbuilder" version: "2.0.1-3df65f4f3a439c8c7b97dd581f8ff4fc1c877a70" rootfs-creation-time: "2021-01-19T20:23:47.239302721+0000Z" description: "osbuilder rootfs" file-format-version: "0.0.2" architecture: "x86_64" base-distro: name: "Clear" version: "34170" packages: default: - "chrony" - "iptables-bin" - "kmod-bin" - "libudev0-shim" - "systemd" - "util-linux-bin" extra: agent: url: "https://github.com/kata-containers/kata-containers" name: "kata-agent" version: "2.0.1" agent-is-init-daemon: "no" ``` ---
Initrd details
# Initrd details No initrd ---
Logfiles
# Logfiles ## Runtime logs
## Throttler logs
## Kata Containerd Shim v2 logs
---
Runtime logs
No recent runtime problems found in system journal.
Throttler logs
No recent throttler problems found in system journal.
Kata Containerd Shim v2
Recent problems found in system journal: ``` time="2021-02-25T11:34:03.135941648+08:00" level=error msg="Invalid command \"create\"" arch=amd64 name= pid=7634 source=runtime time="2021-02-25T11:34:18.265602879+08:00" level=error msg="Invalid command \"create\"" arch=amd64 name= pid=7770 source=runtime time="2021-02-25T11:34:35.173038337+08:00" level=error msg="Invalid command \"create\"" arch=amd64 name= pid=8069 source=runtime time="2021-02-25T11:34:49.182016625+08:00" level=error msg="Invalid command \"create\"" arch=amd64 name= pid=8200 source=runtime time="2021-02-25T11:35:03.173651953+08:00" level=error msg="Invalid command \"create\"" arch=amd64 name= pid=8351 source=runtime time="2021-02-25T11:35:19.083512871+08:00" level=error msg="Invalid command \"create\"" arch=amd64 name= pid=8481 source=runtime time="2021-02-25T11:35:34.120830634+08:00" level=error msg="Invalid command \"create\"" arch=amd64 name= pid=8629 source=runtime time="2021-02-25T11:35:47.142374231+08:00" level=error msg="Invalid command \"create\"" arch=amd64 name= pid=8761 source=runtime time="2021-02-25T11:36:01.301284766+08:00" level=error msg="Invalid command \"create\"" arch=amd64 name= pid=8913 source=runtime time="2021-02-25T11:36:16.185347414+08:00" level=error msg="Invalid command \"create\"" arch=amd64 name= pid=9048 source=runtime time="2021-02-25T11:36:30.262835529+08:00" level=error msg="Invalid command \"create\"" arch=amd64 name= pid=9345 source=runtime time="2021-02-25T11:36:45.188188604+08:00" level=error msg="Invalid command \"create\"" arch=amd64 name= pid=9478 source=runtime time="2021-02-25T11:36:59.104902763+08:00" level=error msg="Invalid command \"create\"" arch=amd64 name= pid=9606 source=runtime time="2021-02-25T11:37:13.148364683+08:00" level=error msg="Invalid command \"create\"" arch=amd64 name= pid=9759 source=runtime time="2021-02-25T11:37:26.62143863+08:00" level=error msg="Invalid command \"create\"" arch=amd64 name= pid=9906 source=runtime time="2021-02-25T11:37:39.187497702+08:00" level=error msg="Invalid command \"create\"" arch=amd64 name= pid=10034 source=runtime time="2021-02-25T11:37:53.179532056+08:00" level=error msg="Invalid command \"create\"" arch=amd64 name= pid=10191 source=runtime time="2021-02-25T11:38:07.079416543+08:00" level=error msg="Invalid command \"create\"" arch=amd64 name= pid=10323 source=runtime time="2021-02-25T11:38:24.198447549+08:00" level=error msg="Invalid command \"create\"" arch=amd64 name= pid=10652 source=runtime time="2021-02-25T11:38:37.139508778+08:00" level=error msg="Invalid command \"create\"" arch=amd64 name= pid=10798 source=runtime time="2021-02-25T11:38:49.214155283+08:00" level=error msg="Invalid command \"create\"" arch=amd64 name= pid=10934 source=runtime time="2021-02-25T11:39:04.139142913+08:00" level=error msg="Invalid command \"create\"" arch=amd64 name= pid=11084 source=runtime time="2021-02-25T11:39:19.206391375+08:00" level=error msg="Invalid command \"create\"" arch=amd64 name= pid=11227 source=runtime time="2021-02-25T11:39:36.17394105+08:00" level=error msg="Invalid command \"create\"" arch=amd64 name= pid=11383 source=runtime time="2021-02-25T11:39:50.165922558+08:00" level=error msg="Invalid command \"create\"" arch=amd64 name= pid=11518 source=runtime time="2021-02-25T11:40:03.123222171+08:00" level=error msg="Invalid command \"create\"" arch=amd64 name= pid=11661 source=runtime time="2021-02-25T11:40:17.132052096+08:00" level=error msg="Invalid command \"create\"" arch=amd64 name= pid=11797 source=runtime time="2021-02-25T11:40:34.292145303+08:00" level=error msg="Invalid command \"create\"" arch=amd64 name= pid=12088 source=runtime time="2021-02-25T11:40:50.067961377+08:00" level=error msg="Invalid command \"create\"" arch=amd64 name= pid=12232 source=runtime time="2021-02-25T11:41:02.025098718+08:00" level=error msg="Invalid command \"create\"" arch=amd64 name= pid=12374 source=runtime time="2021-02-25T11:41:19.117347932+08:00" level=error msg="Invalid command \"create\"" arch=amd64 name= pid=12515 source=runtime time="2021-02-25T11:41:35.360150583+08:00" level=error msg="Invalid command \"create\"" arch=amd64 name= pid=12672 source=runtime time="2021-02-25T11:41:50.134886445+08:00" level=error msg="Invalid command \"create\"" arch=amd64 name= pid=12809 source=runtime time="2021-02-25T11:42:03.092651057+08:00" level=error msg="Invalid command \"create\"" arch=amd64 name= pid=12953 source=runtime time="2021-02-25T11:42:20.193748142+08:00" level=error msg="Invalid command \"create\"" arch=amd64 name= pid=13130 source=runtime time="2021-02-25T11:42:35.003878865+08:00" level=error msg="Invalid command \"create\"" arch=amd64 name= pid=13422 source=runtime time="2021-02-25T11:42:48.070008631+08:00" level=error msg="Invalid command \"create\"" arch=amd64 name= pid=13556 source=runtime time="2021-02-25T11:43:04.062840135+08:00" level=error msg="Invalid command \"create\"" arch=amd64 name= pid=13704 source=runtime time="2021-02-25T11:43:19.121292478+08:00" level=error msg="Invalid command \"create\"" arch=amd64 name= pid=13842 source=runtime time="2021-02-25T11:43:32.16254501+08:00" level=error msg="Invalid command \"create\"" arch=amd64 name= pid=13987 source=runtime time="2021-02-25T11:43:49.338505628+08:00" level=error msg="Invalid command \"create\"" arch=amd64 name= pid=14116 source=runtime time="2021-02-25T11:44:05.34975491+08:00" level=error msg="Invalid command \"create\"" arch=amd64 name= pid=14270 source=runtime time="2021-02-25T11:44:20.216684855+08:00" level=error msg="Invalid command \"create\"" arch=amd64 name= pid=14405 source=runtime time="2021-02-25T11:44:37.035313746+08:00" level=error msg="Invalid command \"create\"" arch=amd64 name= pid=14696 source=runtime time="2021-02-25T11:44:53.460711629+08:00" level=error msg="Invalid command \"create\"" arch=amd64 name= pid=14845 source=runtime time="2021-02-25T11:45:08.052782901+08:00" level=error msg="Invalid command \"create\"" arch=amd64 name= pid=14976 source=runtime time="2021-02-25T11:45:21.169700349+08:00" level=error msg="Invalid command \"create\"" arch=amd64 name= pid=15112 source=runtime time="2021-02-25T11:45:39.005150003+08:00" level=error msg="Invalid command \"create\"" arch=amd64 name= pid=15458 source=runtime time="2021-02-25T11:45:54.131795333+08:00" level=error msg="Invalid command \"create\"" arch=amd64 name= pid=16138 source=runtime time="2021-02-25T11:46:07.115211372+08:00" level=error msg="Invalid command \"create\"" arch=amd64 name= pid=16278 source=runtime ```
Container manager details
# Container manager details
---
Kubernetes
## Kubernetes
kubectl version
``` Client Version: version.Info{Major:"1", Minor:"18", GitVersion:"v1.18.6", GitCommit:"dff82dc0de47299ab66c83c626e08b245ab19037", GitTreeState:"clean", BuildDate:"2020-07-15T16:58:53Z", GoVersion:"go1.13.9", Compiler:"gc", Platform:"linux/amd64"} Server Version: version.Info{Major:"1", Minor:"18", GitVersion:"v1.18.6", GitCommit:"dff82dc0de47299ab66c83c626e08b245ab19037", GitTreeState:"clean", BuildDate:"2020-07-15T16:51:04Z", GoVersion:"go1.13.9", Compiler:"gc", Platform:"linux/amd64"} ```
kubectl config view
``` apiVersion: v1 clusters: - cluster: certificate-authority-data: DATA+OMITTED server: https://192.168.7.113:6443 name: kubernetes contexts: - context: cluster: kubernetes user: kubernetes-admin name: kubernetes-admin@kubernetes current-context: kubernetes-admin@kubernetes kind: Config preferences: {} users: - name: kubernetes-admin user: client-certificate-data: REDACTED client-key-data: REDACTED ```
systemctl show kubelet
``` Type=simple Restart=always NotifyAccess=none RestartUSec=10s TimeoutStartUSec=1min 30s TimeoutStopUSec=1min 30s RuntimeMaxUSec=infinity WatchdogUSec=0 WatchdogTimestamp=Thu 2021-02-25 11:23:01 CST WatchdogTimestampMonotonic=2499884888 PermissionsStartOnly=no RootDirectoryStartOnly=no RemainAfterExit=no GuessMainPID=yes MainPID=31740 ControlPID=0 FileDescriptorStoreMax=0 NFileDescriptorStore=0 StatusErrno=0 Result=success UID=[not set] GID=[not set] NRestarts=0 ExecMainStartTimestamp=Thu 2021-02-25 11:23:01 CST ExecMainStartTimestampMonotonic=2499884863 ExecMainExitTimestampMonotonic=0 ExecMainPID=31740 ExecMainCode=0 ExecMainStatus=0 ExecStartPre={ path=/bin/mkdir ; argv[]=/bin/mkdir -p /sys/fs/cgroup/hugetlb/systemd/system.slice ; ignore_errors=yes ; start_time=[Thu 2021-02-25 11:23:01 CST] ; stop_time=[Thu 2021-02-25 11:23:01 CST] ; pid=31719 ; code=exited ; status=0 } ExecStartPre={ path=/bin/mkdir ; argv[]=/bin/mkdir -p /sys/fs/cgroup/blkio/systemd/system.slice ; ignore_errors=yes ; start_time=[Thu 2021-02-25 11:23:01 CST] ; stop_time=[Thu 2021-02-25 11:23:01 CST] ; pid=31726 ; code=exited ; status=0 } ExecStartPre={ path=/bin/mkdir ; argv[]=/bin/mkdir -p /sys/fs/cgroup/cpuset/systemd/system.slice ; ignore_errors=yes ; start_time=[Thu 2021-02-25 11:23:01 CST] ; stop_time=[Thu 2021-02-25 11:23:01 CST] ; pid=31731 ; code=exited ; status=0 } ExecStartPre={ path=/bin/mkdir ; argv[]=/bin/mkdir -p /sys/fs/cgroup/devices/systemd/system.slice ; ignore_errors=yes ; start_time=[Thu 2021-02-25 11:23:01 CST] ; stop_time=[Thu 2021-02-25 11:23:01 CST] ; pid=31732 ; code=exited ; status=0 } ExecStartPre={ path=/bin/mkdir ; argv[]=/bin/mkdir -p /sys/fs/cgroup/net_cls,net_prio/systemd/system.slice ; ignore_errors=yes ; start_time=[Thu 2021-02-25 11:23:01 CST] ; stop_time=[Thu 2021-02-25 11:23:01 CST] ; pid=31733 ; code=exited ; status=0 } ExecStartPre={ path=/bin/mkdir ; argv[]=/bin/mkdir -p /sys/fs/cgroup/perf_event/systemd/system.slice ; ignore_errors=yes ; start_time=[Thu 2021-02-25 11:23:01 CST] ; stop_time=[Thu 2021-02-25 11:23:01 CST] ; pid=31734 ; code=exited ; status=0 } ExecStartPre={ path=/bin/mkdir ; argv[]=/bin/mkdir -p /sys/fs/cgroup/cpu,cpuacct/systemd/system.slice ; ignore_errors=yes ; start_time=[Thu 2021-02-25 11:23:01 CST] ; stop_time=[Thu 2021-02-25 11:23:01 CST] ; pid=31735 ; code=exited ; status=0 } ExecStartPre={ path=/bin/mkdir ; argv[]=/bin/mkdir -p /sys/fs/cgroup/freezer/systemd/system.slice ; ignore_errors=yes ; start_time=[Thu 2021-02-25 11:23:01 CST] ; stop_time=[Thu 2021-02-25 11:23:01 CST] ; pid=31736 ; code=exited ; status=0 } ExecStartPre={ path=/bin/mkdir ; argv[]=/bin/mkdir -p /sys/fs/cgroup/memory/systemd/system.slice ; ignore_errors=yes ; start_time=[Thu 2021-02-25 11:23:01 CST] ; stop_time=[Thu 2021-02-25 11:23:01 CST] ; pid=31737 ; code=exited ; status=0 } ExecStartPre={ path=/bin/mkdir ; argv[]=/bin/mkdir -p /sys/fs/cgroup/pids/systemd/system.slice ; ignore_errors=yes ; start_time=[Thu 2021-02-25 11:23:01 CST] ; stop_time=[Thu 2021-02-25 11:23:01 CST] ; pid=31738 ; code=exited ; status=0 } ExecStartPre={ path=/bin/mkdir ; argv[]=/bin/mkdir -p /sys/fs/cgroup/systemd/systemd/system.slice ; ignore_errors=yes ; start_time=[Thu 2021-02-25 11:23:01 CST] ; stop_time=[Thu 2021-02-25 11:23:01 CST] ; pid=31739 ; code=exited ; status=0 } ExecStart={ path=/usr/bin/kubelet ; argv[]=/usr/bin/kubelet $KUBELET_KUBECONFIG_ARGS $KUBELET_CONFIG_ARGS $KUBELET_KUBEADM_ARGS $KUBELET_EXTRA_ARGS ; ignore_errors=no ; start_time=[Thu 2021-02-25 11:23:01 CST] ; stop_time=[n/a] ; pid=31740 ; code=(null) ; status=0/0 } Slice=system.slice ControlGroup=/system.slice/kubelet.service MemoryCurrent=[not set] CPUUsageNSec=[not set] TasksCurrent=26 IPIngressBytes=18446744073709551615 IPIngressPackets=18446744073709551615 IPEgressBytes=18446744073709551615 IPEgressPackets=18446744073709551615 Delegate=no CPUAccounting=no CPUWeight=[not set] StartupCPUWeight=[not set] CPUShares=[not set] StartupCPUShares=[not set] CPUQuotaPerSecUSec=infinity IOAccounting=no IOWeight=[not set] StartupIOWeight=[not set] BlockIOAccounting=no BlockIOWeight=[not set] StartupBlockIOWeight=[not set] MemoryAccounting=no MemoryLow=0 MemoryHigh=infinity MemoryMax=infinity MemorySwapMax=infinity MemoryLimit=infinity DevicePolicy=auto TasksAccounting=yes TasksMax=4915 IPAccounting=no Environment=[unprintable] KUBELET_CONFIG_ARGS=--config=/var/lib/kubelet/config.yaml EnvironmentFile=/var/lib/kubelet/kubeadm-flags.env (ignore_errors=yes) EnvironmentFile=/etc/default/kubelet (ignore_errors=yes) UMask=0022 LimitCPU=infinity LimitCPUSoft=infinity LimitFSIZE=infinity LimitFSIZESoft=infinity LimitDATA=infinity LimitDATASoft=infinity LimitSTACK=infinity LimitSTACKSoft=8388608 LimitCORE=infinity LimitCORESoft=0 LimitRSS=infinity LimitRSSSoft=infinity LimitNOFILE=4096 LimitNOFILESoft=1024 LimitAS=infinity LimitASSoft=infinity LimitNPROC=127394 LimitNPROCSoft=127394 LimitMEMLOCK=67108864 LimitMEMLOCKSoft=67108864 LimitLOCKS=infinity LimitLOCKSSoft=infinity LimitSIGPENDING=127394 LimitSIGPENDINGSoft=127394 LimitMSGQUEUE=819200 LimitMSGQUEUESoft=819200 LimitNICE=0 LimitNICESoft=0 LimitRTPRIO=0 LimitRTPRIOSoft=0 LimitRTTIME=infinity LimitRTTIMESoft=infinity OOMScoreAdjust=0 Nice=0 IOSchedulingClass=0 IOSchedulingPriority=0 CPUSchedulingPolicy=0 CPUSchedulingPriority=0 TimerSlackNSec=50000 CPUSchedulingResetOnFork=no NonBlocking=no StandardInput=null StandardInputData= StandardOutput=journal StandardError=inherit TTYReset=no TTYVHangup=no TTYVTDisallocate=no SyslogPriority=30 SyslogLevelPrefix=yes SyslogLevel=6 SyslogFacility=3 LogLevelMax=-1 SecureBits=0 CapabilityBoundingSet=cap_chown cap_dac_override cap_dac_read_search cap_fowner cap_fsetid cap_kill cap_setgid cap_setuid cap_setpcap cap_linux_immutable cap_net_bind_service cap_net_broadcast cap_net_admin cap_net_raw cap_ipc_lock cap_ipc_owner cap_sys_module cap_sys_rawio cap_sys_chroot cap_sys_ptrace cap_sys_pacct cap_sys_admin cap_sys_boot cap_sys_nice cap_sys_resource cap_sys_time cap_sys_tty_config cap_mknod cap_lease cap_audit_write cap_audit_control cap_setfcap cap_mac_override cap_mac_admin cap_syslog cap_wake_alarm cap_block_suspend AmbientCapabilities= DynamicUser=no RemoveIPC=no MountFlags= PrivateTmp=no PrivateDevices=no ProtectKernelTunables=no ProtectKernelModules=no ProtectControlGroups=no PrivateNetwork=no PrivateUsers=no ProtectHome=no ProtectSystem=no SameProcessGroup=no UtmpMode=init IgnoreSIGPIPE=yes NoNewPrivileges=no SystemCallErrorNumber=0 LockPersonality=no RuntimeDirectoryPreserve=no RuntimeDirectoryMode=0755 StateDirectoryMode=0755 CacheDirectoryMode=0755 LogsDirectoryMode=0755 ConfigurationDirectoryMode=0755 MemoryDenyWriteExecute=no RestrictRealtime=no RestrictSUIDSGID=no RestrictNamespaces=no MountAPIVFS=no KeyringMode=private KillMode=control-group KillSignal=15 SendSIGKILL=yes SendSIGHUP=no Id=kubelet.service Names=kubelet.service Requires=sysinit.target system.slice containerd.service Wants=network-online.target WantedBy=multi-user.target Conflicts=shutdown.target Before=shutdown.target multi-user.target After=systemd-journald.socket sysinit.target basic.target network-online.target containerd.service system.slice Description=Kubernetes Kubelet LoadState=loaded ActiveState=active SubState=running FragmentPath=/lib/systemd/system/kubelet.service DropInPaths=/etc/systemd/system/kubelet.service.d/0-containerd.conf /etc/systemd/system/kubelet.service.d/10-kubeadm.conf UnitFileState=enabled UnitFilePreset=enabled StateChangeTimestamp=Thu 2021-02-25 11:23:01 CST StateChangeTimestampMonotonic=2499884889 InactiveExitTimestamp=Thu 2021-02-25 11:23:01 CST InactiveExitTimestampMonotonic=2499862463 ActiveEnterTimestamp=Thu 2021-02-25 11:23:01 CST ActiveEnterTimestampMonotonic=2499884889 ActiveExitTimestamp=Thu 2021-02-25 11:23:00 CST ActiveExitTimestampMonotonic=2499705981 InactiveEnterTimestamp=Thu 2021-02-25 11:23:01 CST InactiveEnterTimestampMonotonic=2499861309 CanStart=yes CanStop=yes CanReload=no CanIsolate=no StopWhenUnneeded=no RefuseManualStart=no RefuseManualStop=no AllowIsolate=no DefaultDependencies=yes OnFailureJobMode=replace IgnoreOnIsolate=no NeedDaemonReload=no JobTimeoutUSec=infinity JobRunningTimeoutUSec=infinity JobTimeoutAction=none ConditionResult=yes AssertResult=yes ConditionTimestamp=Thu 2021-02-25 11:23:01 CST ConditionTimestampMonotonic=2499861829 AssertTimestamp=Thu 2021-02-25 11:23:01 CST AssertTimestampMonotonic=2499861830 Transient=no Perpetual=no StartLimitIntervalUSec=0 StartLimitBurst=5 StartLimitAction=none FailureAction=none SuccessAction=none InvocationID=8b7f66c2841f48e4ab64471730481d01 CollectMode=inactive ```
crio
## crio
crio --version
``` crio version 1.18.4 Version: 1.18.4 GitCommit: be796d1a0acf01c889a404fdbf59ca831cc6eea5 GitTreeState: clean BuildDate: 2021-02-20T04:08:59Z GoVersion: go1.14.6 Compiler: gc Platform: linux/amd64 Linkmode: dynamic ```
systemctl show crio
``` Type=notify Restart=on-abnormal NotifyAccess=main RestartUSec=100ms TimeoutStartUSec=infinity TimeoutStopUSec=1min 30s RuntimeMaxUSec=infinity WatchdogUSec=0 WatchdogTimestampMonotonic=0 PermissionsStartOnly=no RootDirectoryStartOnly=no RemainAfterExit=no GuessMainPID=yes MainPID=0 ControlPID=0 FileDescriptorStoreMax=0 NFileDescriptorStore=0 StatusErrno=0 Result=exit-code UID=[not set] GID=[not set] NRestarts=0 ExecMainStartTimestamp=Thu 2021-02-25 10:41:50 CST ExecMainStartTimestampMonotonic=29650152 ExecMainExitTimestamp=Thu 2021-02-25 10:41:55 CST ExecMainExitTimestampMonotonic=34504239 ExecMainPID=2031 ExecMainCode=1 ExecMainStatus=1 ExecStart={ path=/usr/local/bin/crio ; argv[]=/usr/local/bin/crio $CRIO_CONFIG_OPTIONS $CRIO_RUNTIME_OPTIONS $CRIO_STORAGE_OPTIONS $CRIO_NETWORK_OPTIONS $CRIO_METRICS_OPTIONS ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 } ExecReload={ path=/bin/kill ; argv[]=/bin/kill -s HUP $MAINPID ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 } Slice=system.slice MemoryCurrent=[not set] CPUUsageNSec=[not set] TasksCurrent=[not set] IPIngressBytes=18446744073709551615 IPIngressPackets=18446744073709551615 IPEgressBytes=18446744073709551615 IPEgressPackets=18446744073709551615 Delegate=no CPUAccounting=no CPUWeight=[not set] StartupCPUWeight=[not set] CPUShares=[not set] StartupCPUShares=[not set] CPUQuotaPerSecUSec=infinity IOAccounting=no IOWeight=[not set] StartupIOWeight=[not set] BlockIOAccounting=no BlockIOWeight=[not set] StartupBlockIOWeight=[not set] MemoryAccounting=no MemoryLow=0 MemoryHigh=infinity MemoryMax=infinity MemorySwapMax=infinity MemoryLimit=infinity DevicePolicy=auto TasksAccounting=yes TasksMax=infinity IPAccounting=no Environment=GOTRACEBACK=crash EnvironmentFile=/etc/sysconfig/crio (ignore_errors=yes) UMask=0022 LimitCPU=infinity LimitCPUSoft=infinity LimitFSIZE=infinity LimitFSIZESoft=infinity LimitDATA=infinity LimitDATASoft=infinity LimitSTACK=infinity LimitSTACKSoft=8388608 LimitCORE=infinity LimitCORESoft=infinity LimitRSS=infinity LimitRSSSoft=infinity LimitNOFILE=1048576 LimitNOFILESoft=1048576 LimitAS=infinity LimitASSoft=infinity LimitNPROC=1048576 LimitNPROCSoft=1048576 LimitMEMLOCK=67108864 LimitMEMLOCKSoft=67108864 LimitLOCKS=infinity LimitLOCKSSoft=infinity LimitSIGPENDING=127394 LimitSIGPENDINGSoft=127394 LimitMSGQUEUE=819200 LimitMSGQUEUESoft=819200 LimitNICE=0 LimitNICESoft=0 LimitRTPRIO=0 LimitRTPRIOSoft=0 LimitRTTIME=infinity LimitRTTIMESoft=infinity OOMScoreAdjust=-999 Nice=0 IOSchedulingClass=0 IOSchedulingPriority=0 CPUSchedulingPolicy=0 CPUSchedulingPriority=0 TimerSlackNSec=50000 CPUSchedulingResetOnFork=no NonBlocking=no StandardInput=null StandardInputData= StandardOutput=journal StandardError=inherit TTYReset=no TTYVHangup=no TTYVTDisallocate=no SyslogPriority=30 SyslogLevelPrefix=yes SyslogLevel=6 SyslogFacility=3 LogLevelMax=-1 SecureBits=0 CapabilityBoundingSet=cap_chown cap_dac_override cap_dac_read_search cap_fowner cap_fsetid cap_kill cap_setgid cap_setuid cap_setpcap cap_linux_immutable cap_net_bind_service cap_net_broadcast cap_net_admin cap_net_raw cap_ipc_lock cap_ipc_owner cap_sys_module cap_sys_rawio cap_sys_chroot cap_sys_ptrace cap_sys_pacct cap_sys_admin cap_sys_boot cap_sys_nice cap_sys_resource cap_sys_time cap_sys_tty_config cap_mknod cap_lease cap_audit_write cap_audit_control cap_setfcap cap_mac_override cap_mac_admin cap_syslog cap_wake_alarm cap_block_suspend AmbientCapabilities= DynamicUser=no RemoveIPC=no MountFlags= PrivateTmp=no PrivateDevices=no ProtectKernelTunables=no ProtectKernelModules=no ProtectControlGroups=no PrivateNetwork=no PrivateUsers=no ProtectHome=no ProtectSystem=no SameProcessGroup=no UtmpMode=init IgnoreSIGPIPE=yes NoNewPrivileges=no SystemCallErrorNumber=0 LockPersonality=no RuntimeDirectoryPreserve=no RuntimeDirectoryMode=0755 StateDirectoryMode=0755 CacheDirectoryMode=0755 LogsDirectoryMode=0755 ConfigurationDirectoryMode=0755 MemoryDenyWriteExecute=no RestrictRealtime=no RestrictSUIDSGID=no RestrictNamespaces=no MountAPIVFS=no KeyringMode=private KillMode=control-group KillSignal=15 SendSIGKILL=yes SendSIGHUP=no Id=crio.service Names=crio.service Requires=system.slice sysinit.target Wants=network-online.target WantedBy=multi-user.target Conflicts=shutdown.target Before=shutdown.target multi-user.target After=basic.target system.slice systemd-journald.socket network-online.target sysinit.target Documentation=https://github.com/cri-o/cri-o Description=Container Runtime Interface for OCI (CRI-O) LoadState=loaded ActiveState=failed SubState=failed FragmentPath=/usr/local/lib/systemd/system/crio.service UnitFileState=enabled UnitFilePreset=enabled StateChangeTimestamp=Thu 2021-02-25 10:41:55 CST StateChangeTimestampMonotonic=34504290 InactiveExitTimestamp=Thu 2021-02-25 10:41:50 CST InactiveExitTimestampMonotonic=29650180 ActiveEnterTimestampMonotonic=0 ActiveExitTimestampMonotonic=0 InactiveEnterTimestamp=Thu 2021-02-25 10:41:55 CST InactiveEnterTimestampMonotonic=34504290 CanStart=yes CanStop=yes CanReload=yes CanIsolate=no StopWhenUnneeded=no RefuseManualStart=no RefuseManualStop=no AllowIsolate=no DefaultDependencies=yes OnFailureJobMode=replace IgnoreOnIsolate=no NeedDaemonReload=no JobTimeoutUSec=infinity JobRunningTimeoutUSec=infinity JobTimeoutAction=none ConditionResult=yes AssertResult=yes ConditionTimestamp=Thu 2021-02-25 10:41:50 CST ConditionTimestampMonotonic=29649323 AssertTimestamp=Thu 2021-02-25 10:41:50 CST AssertTimestampMonotonic=29649323 Transient=no Perpetual=no StartLimitIntervalUSec=10s StartLimitBurst=5 StartLimitAction=none FailureAction=none SuccessAction=none InvocationID=99fb6731de6d4b5eafb1feeabd17c3e5 CollectMode=inactive ```
crio config
``` level=info msg="Using default capabilities: CAP_CHOWN, CAP_DAC_OVERRIDE, CAP_FSETID, CAP_FOWNER, CAP_SETGID, CAP_SETUID, CAP_SETPCAP, CAP_NET_BIND_SERVICE, CAP_KILL" # The CRI-O configuration file specifies all of the available configuration # options and command-line flags for the crio(8) OCI Kubernetes Container Runtime # daemon, but in a TOML format that can be more easily modified and versioned. # # Please refer to crio.conf(5) for details of all configuration options. # CRI-O supports partial configuration reload during runtime, which can be # done by sending SIGHUP to the running process. Currently supported options # are explicitly mentioned with: 'This option supports live configuration # reload'. # CRI-O reads its storage defaults from the containers-storage.conf(5) file # located at /etc/containers/storage.conf. Modify this storage configuration if # you want to change the system's defaults. If you want to modify storage just # for CRI-O, you can change the storage configuration options here. [crio] # Path to the "root directory". CRI-O stores all of its data, including # containers images, in this directory. #root = "/var/lib/containers/storage" # Path to the "run directory". CRI-O stores all of its state in this directory. #runroot = "/run/containers/storage" # Storage driver used to manage the storage of images and containers. Please # refer to containers-storage.conf(5) to see all available storage drivers. #storage_driver = "overlay" # List to pass options to the storage driver. Please refer to # containers-storage.conf(5) to see all available storage options. #storage_option = [ # "overlay.mountopt=nodev,metacopy=on", #] # The default log directory where all logs will go unless directly specified by # the kubelet. The log directory specified must be an absolute directory. log_dir = "/var/log/crio/pods" # Location for CRI-O to lay down the temporary version file. # It is used to check if crio wipe should wipe containers, which should # always happen on a node reboot version_file = "/var/run/crio/version" # Location for CRI-O to lay down the persistent version file. # It is used to check if crio wipe should wipe images, which should # only happen when CRI-O has been upgraded version_file_persist = "/var/lib/crio/version" # The crio.api table contains settings for the kubelet/gRPC interface. [crio.api] # Path to AF_LOCAL socket on which CRI-O will listen. listen = "/var/run/crio/crio.sock" # IP address on which the stream server will listen. stream_address = "127.0.0.1" # The port on which the stream server will listen. If the port is set to "0", then # CRI-O will allocate a random free port number. stream_port = "0" # Enable encrypted TLS transport of the stream server. stream_enable_tls = false # Path to the x509 certificate file used to serve the encrypted stream. This # file can change, and CRI-O will automatically pick up the changes within 5 # minutes. stream_tls_cert = "" # Path to the key file used to serve the encrypted stream. This file can # change and CRI-O will automatically pick up the changes within 5 minutes. stream_tls_key = "" # Path to the x509 CA(s) file used to verify and authenticate client # communication with the encrypted stream. This file can change and CRI-O will # automatically pick up the changes within 5 minutes. stream_tls_ca = "" # Maximum grpc send message size in bytes. If not set or <=0, then CRI-O will default to 16 * 1024 * 1024. grpc_max_send_msg_size = 16777216 # Maximum grpc receive message size. If not set or <= 0, then CRI-O will default to 16 * 1024 * 1024. grpc_max_recv_msg_size = 16777216 # The crio.runtime table contains settings pertaining to the OCI runtime used # and options for how to set up and manage the OCI runtime. [crio.runtime] # A list of ulimits to be set in containers by default, specified as # "=:", for example:
# "nofile=1024:2048"
# If nothing is set here, settings will be inherited from the CRI-O daemon
#default_ulimits = [
#]
# default_runtime is the _name_ of the OCI runtime to be used as the default.
# The name is matched against the runtimes map below.
default_runtime = "runc"
# If true, the runtime will not use pivot_root, but instead use MS_MOVE.
no_pivot = false
# decryption_keys_path is the path where the keys required for
# image decryption are stored. This option supports live configuration reload.
decryption_keys_path = "/etc/crio/keys/"
# Path to the conmon binary, used for monitoring the OCI runtime.
# Will be searched for using $PATH if empty.
conmon = ""
# Cgroup setting for conmon
conmon_cgroup = "system.slice"
# Environment variable list for the conmon process, used for passing necessary
# environment variables to conmon or the runtime.
conmon_env = [
"PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin",
]
# Additional environment variables to set for all the
# containers. These are overridden if set in the
# container image spec or in the container runtime configuration.
default_env = [
]
# If true, SELinux will be used for pod separation on the host.
selinux = false
# Path to the seccomp.json profile which is used as the default seccomp profile
# for the runtime. If not specified, then the internal default seccomp profile
# will be used. This option supports live configuration reload.
seccomp_profile = ""
# Used to change the name of the default AppArmor profile of CRI-O. The default
# profile name is "crio-default". This profile only takes effect if the user
# does not specify a profile via the Kubernetes Pod's metadata annotation. If
# the profile is set to "unconfined", then this equals to disabling AppArmor.
# This option supports live configuration reload.
apparmor_profile = "crio-default"
# Cgroup management implementation used for the runtime.
cgroup_manager = "systemd"
# List of default capabilities for containers. If it is empty or commented out,
# only the capabilities defined in the containers json file by the user/kube
# will be added.
default_capabilities = [
"CHOWN",
"DAC_OVERRIDE",
"FSETID",
"FOWNER",
"SETGID",
"SETUID",
"SETPCAP",
"NET_BIND_SERVICE",
"KILL",
]
# List of default sysctls. If it is empty or commented out, only the sysctls
# defined in the container json file by the user/kube will be added.
default_sysctls = [
]
# List of additional devices. specified as
# "::", for example: "--device=/dev/sdc:/dev/xvdc:rwm".
#If it is empty or commented out, only the devices
# defined in the container json file by the user/kube will be added.
additional_devices = [
]
# Path to OCI hooks directories for automatically executed hooks. If one of the
# directories does not exist, then CRI-O will automatically skip them.
hooks_dir = [
"/usr/share/containers/oci/hooks.d",
]
# List of default mounts for each container. **Deprecated:** this option will
# be removed in future versions in favor of default_mounts_file.
default_mounts = [
]
# Path to the file specifying the defaults mounts for each container. The
# format of the config is /SRC:/DST, one mount per line. Notice that CRI-O reads
# its default mounts from the following two files:
#
# 1) /etc/containers/mounts.conf (i.e., default_mounts_file): This is the
# override file, where users can either add in their own default mounts, or
# override the default mounts shipped with the package.
#
# 2) /usr/share/containers/mounts.conf: This is the default file read for
# mounts. If you want CRI-O to read from a different, specific mounts file,
# you can change the default_mounts_file. Note, if this is done, CRI-O will
# only add mounts it finds in this file.
#
#default_mounts_file = ""
# Maximum number of processes allowed in a container.
pids_limit = 1024
# Maximum sized allowed for the container log file. Negative numbers indicate
# that no size limit is imposed. If it is positive, it must be >= 8192 to
# match/exceed conmon's read buffer. The file is truncated and re-opened so the
# limit is never exceeded.
log_size_max = -1
# Whether container output should be logged to journald in addition to the kuberentes log file
log_to_journald = false
# Path to directory in which container exit files are written to by conmon.
container_exits_dir = "/var/run/crio/exits"
# Path to directory for container attach sockets.
container_attach_socket_dir = "/var/run/crio"
# The prefix to use for the source of the bind mounts.
bind_mount_prefix = ""
# If set to true, all containers will run in read-only mode.
read_only = false
# Changes the verbosity of the logs based on the level it is set to. Options
# are fatal, panic, error, warn, info, debug and trace. This option supports
# live configuration reload.
log_level = "info"
# Filter the log messages by the provided regular expression.
# This option supports live configuration reload.
log_filter = ""
# The UID mappings for the user namespace of each container. A range is
# specified in the form containerUID:HostUID:Size. Multiple ranges must be
# separated by comma.
uid_mappings = ""
# The GID mappings for the user namespace of each container. A range is
# specified in the form containerGID:HostGID:Size. Multiple ranges must be
# separated by comma.
gid_mappings = ""
# The minimal amount of time in seconds to wait before issuing a timeout
# regarding the proper termination of the container. The lowest possible
# value is 30s, whereas lower values are not considered by CRI-O.
ctr_stop_timeout = 30
# **DEPRECATED** this option is being replaced by manage_ns_lifecycle, which is described below.
# manage_network_ns_lifecycle = true
# manage_ns_lifecycle determines whether we pin and remove namespaces
# and manage their lifecycle
manage_ns_lifecycle = true
# The directory where the state of the managed namespaces gets tracked.
# Only used when manage_ns_lifecycle is true.
namespaces_dir = "/var/run"
# pinns_path is the path to find the pinns binary, which is needed to manage namespace lifecycle
pinns_path = ""
# The "crio.runtime.runtimes" table defines a list of OCI compatible runtimes.
# The runtime to use is picked based on the runtime_handler provided by the CRI.
# If no runtime_handler is provided, the runtime will be picked based on the level
# of trust of the workload. Each entry in the table should follow the format:
#
#[crio.runtime.runtimes.runtime-handler]
# runtime_path = "/path/to/the/executable"
# runtime_type = "oci"
# runtime_root = "/path/to/the/root"
#
# Where:
# - runtime-handler: name used to identify the runtime
# - runtime_path (optional, string): absolute path to the runtime executable in
# the host filesystem. If omitted, the runtime-handler identifier should match
# the runtime executable name, and the runtime executable should be placed
# in $PATH.
# - runtime_type (optional, string): type of runtime, one of: "oci", "vm". If
# omitted, an "oci" runtime is assumed.
# - runtime_root (optional, string): root directory for storage of containers
# state.
[crio.runtime.runtimes.kata-runtime]
runtime_path = "/usr/local/bin/kata-runtime"
runtime_type = "oci"
runtime_root = ""
[crio.runtime.runtimes.runc]
runtime_path = ""
runtime_type = "oci"
runtime_root = "/run/runc"
# Kata Containers is an OCI runtime, where containers are run inside lightweight
# VMs. Kata provides additional isolation towards the host, minimizing the host attack
# surface and mitigating the consequences of containers breakout.
# Kata Containers with the default configured VMM
#[crio.runtime.runtimes.kata-runtime]
# Kata Containers with the QEMU VMM
#[crio.runtime.runtimes.kata-qemu]
# Kata Containers with the Firecracker VMM
#[crio.runtime.runtimes.kata-fc]
# The crio.image table contains settings pertaining to the management of OCI images.
#
# CRI-O reads its configured registries defaults from the system wide
# containers-registries.conf(5) located in /etc/containers/registries.conf. If
# you want to modify just CRI-O, you can change the registries configuration in
# this file. Otherwise, leave insecure_registries and registries commented out to
# use the system's defaults from /etc/containers/registries.conf.
[crio.image]
# Default transport for pulling images from a remote container storage.
default_transport = "docker://"
# The path to a file containing credentials necessary for pulling images from
# secure registries. The file is similar to that of /var/lib/kubelet/config.json
global_auth_file = ""
# The image used to instantiate infra containers.
# This option supports live configuration reload.
pause_image = "k8s.gcr.io/pause:3.2"
# The path to a file containing credentials specific for pulling the pause_image from
# above. The file is similar to that of /var/lib/kubelet/config.json
# This option supports live configuration reload.
pause_image_auth_file = ""
# The command to run to have a container stay in the paused state.
# When explicitly set to "", it will fallback to the entrypoint and command
# specified in the pause image. When commented out, it will fallback to the
# default: "/pause". This option supports live configuration reload.
pause_command = "/pause"
# Path to the file which decides what sort of policy we use when deciding
# whether or not to trust an image that we've pulled. It is not recommended that
# this option be used, as the default behavior of using the system-wide default
# policy (i.e., /etc/containers/policy.json) is most often preferred. Please
# refer to containers-policy.json(5) for more details.
signature_policy = ""
# List of registries to skip TLS verification for pulling images. Please
# consider configuring the registries via /etc/containers/registries.conf before
# changing them here.
#insecure_registries = "[]"
# Controls how image volumes are handled. The valid values are mkdir, bind and
# ignore; the latter will ignore volumes entirely.
image_volumes = "mkdir"
# List of registries to be used when pulling an unqualified image (e.g.,
# "alpine:latest"). By default, registries is set to "docker.io" for
# compatibility reasons. Depending on your workload and usecase you may add more
# registries (e.g., "quay.io", "registry.fedoraproject.org",
# "registry.opensuse.org", etc.).
#registries = [
# ]
# The crio.network table containers settings pertaining to the management of
# CNI plugins.
[crio.network]
# The default CNI network name to be selected. If not set or "", then
# CRI-O will pick-up the first one found in network_dir.
# cni_default_network = ""
# Path to the directory where CNI configuration files are located.
network_dir = "/etc/cni/net.d/"
# Paths to directories where CNI plugin binaries are located.
plugin_dirs = [
"/opt/cni/bin/",
]
# A necessary configuration for Prometheus based metrics retrieval
[crio.metrics]
# Globally enable or disable metrics support.
enable_metrics = false
# The port on which the metrics server will listen.
metrics_port = 9090
```
containerd
## containerd
containerd --version
``` containerd github.com/containerd/containerd v1.4.3 269548fa27e0089a8b8278fc4fc781d7f65a939b ```
systemctl show containerd
``` Type=simple Restart=always NotifyAccess=none RestartUSec=5s TimeoutStartUSec=infinity TimeoutStopUSec=1min 30s RuntimeMaxUSec=infinity WatchdogUSec=0 WatchdogTimestamp=Thu 2021-02-25 11:22:38 CST WatchdogTimestampMonotonic=2477717420 PermissionsStartOnly=no RootDirectoryStartOnly=no RemainAfterExit=no GuessMainPID=yes MainPID=30987 ControlPID=0 FileDescriptorStoreMax=0 NFileDescriptorStore=0 StatusErrno=0 Result=success UID=[not set] GID=[not set] NRestarts=51 ExecMainStartTimestamp=Thu 2021-02-25 11:22:38 CST ExecMainStartTimestampMonotonic=2477717396 ExecMainExitTimestampMonotonic=0 ExecMainPID=30987 ExecMainCode=0 ExecMainStatus=0 ExecStartPre={ path=/sbin/modprobe ; argv[]=/sbin/modprobe br_netfilter ; ignore_errors=yes ; start_time=[Thu 2021-02-25 11:22:38 CST] ; stop_time=[Thu 2021-02-25 11:22:38 CST] ; pid=30973 ; code=exited ; status=0 } ExecStartPre={ path=/sbin/modprobe ; argv[]=/sbin/modprobe overlay ; ignore_errors=yes ; start_time=[Thu 2021-02-25 11:22:38 CST] ; stop_time=[Thu 2021-02-25 11:22:38 CST] ; pid=30976 ; code=exited ; status=0 } ExecStartPre={ path=/bin/mkdir ; argv[]=/bin/mkdir -p /run/k8s/containerd ; ignore_errors=yes ; start_time=[Thu 2021-02-25 11:22:38 CST] ; stop_time=[Thu 2021-02-25 11:22:38 CST] ; pid=30986 ; code=exited ; status=0 } ExecStart={ path=/soft/containerd/bin/containerd ; argv[]=/soft/containerd/bin/containerd -c /soft/containerd/conf/config.toml -a /run/containerd/containerd.sock --state /soft/containerd/k8s/run/containerd --root /soft/containerd/k8s/containerd ; ignore_errors=no ; start_time=[Thu 2021-02-25 11:22:38 CST] ; stop_time=[n/a] ; pid=30987 ; code=(null) ; status=0/0 } Slice=system.slice ControlGroup=/system.slice/containerd.service MemoryCurrent=[not set] CPUUsageNSec=[not set] TasksCurrent=324 IPIngressBytes=18446744073709551615 IPIngressPackets=18446744073709551615 IPEgressBytes=18446744073709551615 IPEgressPackets=18446744073709551615 Delegate=yes DelegateControllers=cpu cpuacct io blkio memory devices pids CPUAccounting=no CPUWeight=[not set] StartupCPUWeight=[not set] CPUShares=[not set] StartupCPUShares=[not set] CPUQuotaPerSecUSec=infinity IOAccounting=no IOWeight=[not set] StartupIOWeight=[not set] BlockIOAccounting=no BlockIOWeight=[not set] StartupBlockIOWeight=[not set] MemoryAccounting=no MemoryLow=0 MemoryHigh=infinity MemoryMax=infinity MemorySwapMax=infinity MemoryLimit=infinity DevicePolicy=auto TasksAccounting=yes TasksMax=infinity IPAccounting=no UMask=0022 LimitCPU=infinity LimitCPUSoft=infinity LimitFSIZE=infinity LimitFSIZESoft=infinity LimitDATA=infinity LimitDATASoft=infinity LimitSTACK=infinity LimitSTACKSoft=8388608 LimitCORE=infinity LimitCORESoft=infinity LimitRSS=infinity LimitRSSSoft=infinity LimitNOFILE=65535 LimitNOFILESoft=65535 LimitAS=infinity LimitASSoft=infinity LimitNPROC=65535 LimitNPROCSoft=65535 LimitMEMLOCK=67108864 LimitMEMLOCKSoft=67108864 LimitLOCKS=infinity LimitLOCKSSoft=infinity LimitSIGPENDING=127394 LimitSIGPENDINGSoft=127394 LimitMSGQUEUE=819200 LimitMSGQUEUESoft=819200 LimitNICE=0 LimitNICESoft=0 LimitRTPRIO=0 LimitRTPRIOSoft=0 LimitRTTIME=infinity LimitRTTIMESoft=infinity OOMScoreAdjust=-999 Nice=0 IOSchedulingClass=0 IOSchedulingPriority=0 CPUSchedulingPolicy=0 CPUSchedulingPriority=0 TimerSlackNSec=50000 CPUSchedulingResetOnFork=no NonBlocking=no StandardInput=null StandardInputData= StandardOutput=journal StandardError=inherit TTYReset=no TTYVHangup=no TTYVTDisallocate=no SyslogPriority=30 SyslogLevelPrefix=yes SyslogLevel=6 SyslogFacility=3 LogLevelMax=-1 SecureBits=0 CapabilityBoundingSet=cap_chown cap_dac_override cap_dac_read_search cap_fowner cap_fsetid cap_kill cap_setgid cap_setuid cap_setpcap cap_linux_immutable cap_net_bind_service cap_net_broadcast cap_net_admin cap_net_raw cap_ipc_lock cap_ipc_owner cap_sys_module cap_sys_rawio cap_sys_chroot cap_sys_ptrace cap_sys_pacct cap_sys_admin cap_sys_boot cap_sys_nice cap_sys_resource cap_sys_time cap_sys_tty_config cap_mknod cap_lease cap_audit_write cap_audit_control cap_setfcap cap_mac_override cap_mac_admin cap_syslog cap_wake_alarm cap_block_suspend AmbientCapabilities= DynamicUser=no RemoveIPC=no MountFlags= PrivateTmp=no PrivateDevices=no ProtectKernelTunables=no ProtectKernelModules=no ProtectControlGroups=no PrivateNetwork=no PrivateUsers=no ProtectHome=no ProtectSystem=no SameProcessGroup=no UtmpMode=init IgnoreSIGPIPE=yes NoNewPrivileges=no SystemCallErrorNumber=0 LockPersonality=no RuntimeDirectoryPreserve=no RuntimeDirectoryMode=0755 StateDirectoryMode=0755 CacheDirectoryMode=0755 LogsDirectoryMode=0755 ConfigurationDirectoryMode=0755 MemoryDenyWriteExecute=no RestrictRealtime=no RestrictSUIDSGID=no RestrictNamespaces=no MountAPIVFS=no KeyringMode=private KillMode=control-group KillSignal=15 SendSIGKILL=yes SendSIGHUP=no Id=containerd.service Names=containerd.service Requires=system.slice sysinit.target RequiredBy=kubelet.service Conflicts=shutdown.target Before=shutdown.target kubelet.service After=local-fs.target sysinit.target network.target basic.target systemd-journald.socket system.slice Documentation=https://containerd.io Description=containerd container runtime LoadState=loaded ActiveState=active SubState=running FragmentPath=/etc/systemd/system/containerd.service UnitFileState=disabled UnitFilePreset=enabled StateChangeTimestamp=Thu 2021-02-25 11:22:38 CST StateChangeTimestampMonotonic=2477717423 InactiveExitTimestamp=Thu 2021-02-25 11:22:38 CST InactiveExitTimestampMonotonic=2477710861 ActiveEnterTimestamp=Thu 2021-02-25 11:22:38 CST ActiveEnterTimestampMonotonic=2477717423 ActiveExitTimestamp=Thu 2021-02-25 11:22:32 CST ActiveExitTimestampMonotonic=2471613588 InactiveEnterTimestamp=Thu 2021-02-25 11:22:38 CST InactiveEnterTimestampMonotonic=2477710018 CanStart=yes CanStop=yes CanReload=no CanIsolate=no StopWhenUnneeded=no RefuseManualStart=no RefuseManualStop=no AllowIsolate=no DefaultDependencies=yes OnFailureJobMode=replace IgnoreOnIsolate=no NeedDaemonReload=no JobTimeoutUSec=infinity JobRunningTimeoutUSec=infinity JobTimeoutAction=none ConditionResult=yes AssertResult=yes ConditionTimestamp=Thu 2021-02-25 11:22:38 CST ConditionTimestampMonotonic=2477710317 AssertTimestamp=Thu 2021-02-25 11:22:38 CST AssertTimestampMonotonic=2477710317 Transient=no Perpetual=no StartLimitIntervalUSec=10s StartLimitBurst=5 StartLimitAction=none FailureAction=none SuccessAction=none InvocationID=dd0c14eb25194af4832e9456d3dd6941 CollectMode=inactive ```
cat /etc/containerd/config.toml
```toml [plugins.opt] path = "/soft/containerd/bin/containerd" [plugins.cri] stream_server_address = "127.0.0.1" stream_server_port = "10010" sandbox_image = "docker.io/juestnow/pause-amd64:3.2" max_concurrent_downloads = 20 [plugins.cri.containerd] snapshotter = "overlayfs" [plugins.cri.containerd.default_runtime] runtime_type = "io.containerd.runtime.v1.linux" runtime_engine = "" runtime_root = "" [plugins.cri.containerd.untrusted_workload_runtime] runtime_type = "io.containerd.runtime.v1.linux" runtime_engine = "/usr/bin/kata-runtime" runtime_root = "" [plugins.cri.containerd.runtimes.kata-runtime] runtime_type = "io.containerd.runtime.v1.linux" runtime_engine = "/usr/bin/kata-runtime" runtime_root = "" [plugins.cri.cni] bin_dir = "/opt/cni/bin" conf_dir = "/etc/cni/net.d" [plugins."io.containerd.runtime.v1.linux"] shim = "containerd-shim" runtime = "runc" runtime_root = "" no_shim = false shim_debug = false [plugins."io.containerd.runtime.v2.task"] platforms = ["linux/amd64"] [plugins] [plugins.cri] [plugins.cri.containerd] default_runtime_name = "kata" [plugins.cri.containerd.runtimes.kata] runtime_type = "io.containerd.kata.v2" ```
Packages
# Packages Have `dpkg`
No `rpm`
---
dpkg -l|egrep "(cc-oci-runtime|cc-runtime|runv|kata-runtime|kata-ksm-throttler|kata-containers-image|linux-container|qemu-)"
``` ii kata-containers-image 1.13.0~alpha0-49 amd64 Kata containers image ii kata-ksm-throttler 1.13.0~alpha0-52 amd64 ii kata-linux-container 5.4.60.91-52 amd64 linux kernel optimised for container-like workloads. ii qemu-vanilla 5.0.0+git.fdd76fecdd-53 amd64 linux kernel optimised for container-like workloads. ```