kata-containers / runtime

Kata Containers version 1.x runtime (for version 2.x see https://github.com/kata-containers/kata-containers).
https://katacontainers.io/
Apache License 2.0
2.1k stars 375 forks source link

Could not umount bind-mounted volumes #965

Closed luning25 closed 5 years ago

luning25 commented 5 years ago

Description of problem

docker run --rm -d -it --runtime=kata-runtime -v /mnt/tmp:/mnt/tmp:rslave --name bindtest ubuntu mkdir /mnt/tmp/1 mount --bind /home/disk2/ /mnt/tmp/1/ docker stop bindtest

Expected result

Actual result

kata-runtime: level=warning msg="Could not umount" error="device or resource busy"

$ kata-runtime list
stat /var/lib/docker/overlay2/d0c20d0ddcb11fe9425cd41b169534408c66d4dae440a9df5f50f521a422839c/merged: no such file or directory
$ mount|grep kata|grep tmp
rootfs on /run/kata-containers/shared/sandboxes/c41e5e992a59422febdaaac348ce4772970cdbb33b539563949cfce697e6e7b4/c41e5e992a59422febdaaac348ce4772970cdbb33b539563949cfce697e6e7b4-9bdadb79490a7796-tmp type rootfs (rw,size=98644776k,nr_inodes=24661194)
/dev/sdc1 on /run/kata-containers/shared/sandboxes/c41e5e992a59422febdaaac348ce4772970cdbb33b539563949cfce697e6e7b4/c41e5e992a59422febdaaac348ce4772970cdbb33b539563949cfce697e6e7b4-9bdadb79490a7796-tmp/1 type ext4 (rw,noatime,data=ordered)
$ umount /run/kata-containers/shared/sandboxes/c41e5e992a59422febdaaac348ce4772970cdbb33b539563949cfce697e6e7b4/c41e5e992a59422febdaaac348ce4772970cdbb33b539563949cfce697e6e7b4-9bdadb79490a7796-tmp
umount: /run/kata-containers/shared/sandboxes/c41e5e992a59422febdaaac348ce4772970cdbb33b539563949cfce697e6e7b4/c41e5e992a59422febdaaac348ce4772970cdbb33b539563949cfce697e6e7b4-9bdadb79490a7796-tmp: target is busy
        (In some cases useful info about processes that
         use the device is found by lsof(8) or fuser(1).)
$ umount /run/kata-containers/shared/sandboxes/c41e5e992a59422febdaaac348ce4772970cdbb33b539563949cfce697e6e7b4/c41e5e992a59422febdaaac348ce4772970cdbb33b539563949cfce697e6e7b4-9bdadb79490a7796-tmp/1
$ umount /run/kata-containers/shared/sandboxes/c41e5e992a59422febdaaac348ce4772970cdbb33b539563949cfce697e6e7b4/c41e5e992a59422febdaaac348ce4772970cdbb33b539563949cfce697e6e7b4-9bdadb79490a7796-tmp
$

Meta details

Running kata-collect-data.sh version 1.3.1 (commit 258eae0) at 2018-12-03.18:04:52.399496662+0800.


Runtime is /usr/bin/kata-runtime.

kata-env

Output of "/usr/bin/kata-runtime kata-env":

[Meta]
  Version = "1.0.18"

[Runtime]
  Debug = false
  Path = "/usr/bin/kata-runtime"
  [Runtime.Version]
    Semver = "1.3.1"
    Commit = "258eae0"
    OCI = "1.0.1"
  [Runtime.Config]
    Path = "/usr/share/defaults/kata-containers/configuration.toml"
  [Hypervisor]
  MachineType = "pc"
  Version = "QEMU emulator version 2.11.0\nCopyright (c) 2003-2017 Fabrice Bellard and the QEMU Project developers"
  Path = "/usr/bin/qemu-lite-system-x86_64"
  BlockDeviceDriver = "virtio-scsi"
  EntropySource = "/dev/urandom"
  Msize9p = 8192
  MemorySlots = 10
  Debug = false
  UseVSock = false

[Image]
  Path = "/usr/share/kata-containers/kata-containers-image_clearlinux_1.3.1_agent_c7fdd324cda.img"

[Kernel]
  Path = "/usr/share/kata-containers/vmlinuz-4.14.67.16-139.container"
  Parameters = ""

[Initrd]
  Path = ""
 [Proxy]
  Type = "kataProxy"
  Version = "kata-proxy version 1.3.1-d364b2e"
  Path = "/usr/libexec/kata-containers/kata-proxy"
  Debug = false

[Shim]
  Type = "kataShim"
  Version = "kata-shim version 1.3.1-58f757d"
  Path = "/usr/libexec/kata-containers/kata-shim"
  Debug = false

[Agent]
  Type = "kata"

[Host]
  Kernel = "4.4.0-124-generic"
  Architecture = "amd64"
  VMContainerCapable = true
  SupportVSocks = false
  [Host.Distro]
    Name = "Ubuntu"
    Version = "16.04"
  [Host.CPU]
    Vendor = "GenuineIntel"
    Model = "Intel(R) Xeon(R) CPU E5-2650 v4 @ 2.20GHz"
  [Netmon]
  Version = "kata-netmon version 1.3.1"
  Path = "/usr/libexec/kata-containers/kata-netmon"
  Debug = false
  Enable = false

Runtime config files

Runtime default config files

/etc/kata-containers/configuration.toml
/usr/share/defaults/kata-containers/configuration.toml

Runtime config file contents

Config file /etc/kata-containers/configuration.toml not found Output of "cat "/usr/share/defaults/kata-containers/configuration.toml"":

# Copyright (c) 2017-2018 Intel Corporation
#
# SPDX-License-Identifier: Apache-2.0
#

# XXX: WARNING: this file is auto-generated.
# XXX:
# XXX: Source file: "cli/config/configuration.toml.in"
# XXX: Project:
# XXX:   Name: Kata Containers
# XXX:   Type: kata
[hypervisor.qemu]
path = "/usr/bin/qemu-lite-system-x86_64"
kernel = "/usr/share/kata-containers/vmlinuz.container"
image = "/usr/share/kata-containers/kata-containers.img"
machine_type = "pc"

# Optional space-separated list of options to pass to the guest kernel.
# For example, use `kernel_params = "vsyscall=emulate"` if you are having
# trouble running pre-2.15 glibc.
#
# WARNING: - any parameter specified here will take priority over the default
# parameter value of the same name used to start the virtual machine.
# Do not set values here unless you understand the impact of doing so as you
# may stop the virtual machine from booting.
# To see the list of default parameters, enable hypervisor debug, create a
# container and look for 'default-kernel-parameters' log entries.
kernel_params = ""

# Path to the firmware.
# If you want that qemu uses the default firmware leave this option empty
firmware = ""
# Machine accelerators
# comma-separated list of machine accelerators to pass to the hypervisor.
# For example, `machine_accelerators = "nosmm,nosmbus,nosata,nopit,static-prt,nofw"`
machine_accelerators=""

# Default number of vCPUs per SB/VM:
# unspecified or 0                --> will be set to 1
# < 0                             --> will be set to the actual number of physical cores
# > 0 <= number of physical cores --> will be set to the specified number
# > number of physical cores      --> will be set to the actual number of physical cores
default_vcpus = 1

# Default maximum number of vCPUs per SB/VM:
# unspecified or == 0             --> will be set to the actual number of physical cores or to the maximum number
#                                     of vCPUs supported by KVM if that number is exceeded
# > 0 <= number of physical cores --> will be set to the specified number
# > number of physical cores      --> will be set to the actual number of physical cores or to the maximum number
#                                     of vCPUs supported by KVM if that number is exceeded
# WARNING: Depending of the architecture, the maximum number of vCPUs supported by KVM is used when
# the actual number of physical cores is greater than it.
# WARNING: Be aware that this value impacts the virtual machine's memory footprint and CPU
# the hotplug functionality. For example, `default_maxvcpus = 240` specifies that until 240 vCPUs
# can be added to a SB/VM, but the memory footprint will be big. Another example, with
# `default_maxvcpus = 8` the memory footprint will be small, but 8 will be the maximum number of
# vCPUs supported by the SB/VM. In general, we recommend that you do not edit this variable,
# unless you know what are you doing.
default_maxvcpus = 0

# Bridges can be used to hot plug devices.
# Limitations:
# * Currently only pci bridges are supported
# * Until 30 devices per bridge can be hot plugged.
# * Until 5 PCI bridges can be cold plugged per VM.
#   This limitation could be a bug in qemu or in the kernel
# Default number of bridges per SB/VM:
# unspecified or 0   --> will be set to 1
# > 1 <= 5           --> will be set to the specified number
# > 5                --> will be set to 5
default_bridges = 1

# Default memory size in MiB for SB/VM.
# If unspecified then it will be set 2048 MiB.
#default_memory = 2048
#
# Default memory slots per SB/VM.
# If unspecified then it will be set 10.
# This is will determine the times that memory will be hotadded to sandbox/VM.
#memory_slots = 10

# Disable block device from being used for a container's rootfs.
# In case of a storage driver like devicemapper where a container's
# root file system is backed by a block device, the block device is passed
# directly to the hypervisor for performance reasons.
# This flag prevents the block device from being passed to the hypervisor,
# 9pfs is used instead to pass the rootfs.
disable_block_device_use = false

# Block storage driver to be used for the hypervisor in case the container
# rootfs is backed by a block device. This is either virtio-scsi or
# virtio-blk.
block_device_driver = "virtio-scsi"

# Enable iothreads (data-plane) to be used. This causes IO to be
# handled in a separate IO thread. This is currently only implemented
# for SCSI.
#
enable_iothreads = false

# Enable pre allocation of VM RAM, default false
# Enabling this will result in lower container density
# as all of the memory will be allocated and locked
# This is useful when you want to reserve all the memory
# upfront or in the cases where you want memory latencies
# to be very predictable
# Default false
#enable_mem_prealloc = true

# Enable huge pages for VM RAM, default false
# Enabling this will result in the VM memory
# being allocated using huge pages.
# This is useful when you want to use vhost-user network
# stacks within the container. This will automatically
# result in memory pre allocation
#enable_hugepages = true

# Enable swap of vm memory. Default false.
# The behaviour is undefined if mem_prealloc is also set to true
#enable_swap = true

# This option changes the default hypervisor and kernel parameters
# to enable debug output where available. This extra output is added
# to the proxy logs, but only when proxy debug is also enabled.
#
# Default false
#enable_debug = true

# Disable the customizations done in the runtime when it detects
# that it is running on top a VMM. This will result in the runtime
# behaving as it would when running on bare metal.
#
#disable_nesting_checks = true

# This is the msize used for 9p shares. It is the number of bytes
# used for 9p packet payload.
#msize_9p = 8192

# If true and vsocks are supported, use vsocks to communicate directly
# with the agent and no proxy is started, otherwise use unix
# sockets and start a proxy to communicate with the agent.
# Default false
#use_vsock = true

# VFIO devices are hotplugged on a bridge by default.
# Enable hotplugging on root bus. This may be required for devices with
# a large PCI bar, as this is a current limitation with hotplugging on
# a bridge. This value is valid for "pc" machine type.
# Default false
#hotplug_vfio_on_root_bus = true

# If host doesn't support vhost_net, set to true. Thus we won't create vhost fds for nics.
# Default false
#disable_vhost_net = true
#
# Default entropy source.
# The path to a host source of entropy (including a real hardware RNG)
# /dev/urandom and /dev/random are two main options.
# Be aware that /dev/random is a blocking source of entropy.  If the host
# runs out of entropy, the VMs boot time will increase leading to get startup
# timeouts.
# The source of entropy /dev/urandom is non-blocking and provides a
# generally acceptable source of entropy. It should work well for pretty much
# all practical purposes.
#entropy_source= "/dev/urandom"

[factory]
# VM templating support. Once enabled, new VMs are created from template
# using vm cloning. They will share the same initial kernel, initramfs and
# agent memory by mapping it readonly. It helps speeding up new container
# creation and saves a lot of memory if there are many kata containers running
# on the same host.
#
# When disabled, new VMs are created from scratch.
#
# Default false
#enable_template = true

[proxy.kata]
path = "/usr/libexec/kata-containers/kata-proxy"

# If enabled, proxy messages will be sent to the system log
# (default: disabled)
#enable_debug = true

[shim.kata]
path = "/usr/libexec/kata-containers/kata-shim"

# If enabled, shim messages will be sent to the system log
# (default: disabled)
#enable_debug = true

[agent.kata]
# There is no field for this section. The goal is only to be able to
# specify which type of agent the user wants to use.

[netmon]
# If enabled, the network monitoring process gets started when the
# sandbox is created. This allows for the detection of some additional
# network being added to the existing network namespace, after the
# sandbox has been created.
# (default: disabled)
#enable_netmon = true

# Specify the path to the netmon binary.
path = "/usr/libexec/kata-containers/kata-netmon"

# If enabled, netmon messages will be sent to the system log
# (default: disabled)
#enable_debug = true

[runtime]
# If enabled, the runtime will log additional debug messages to the
# system log
# (default: disabled)
#enable_debug = true
#
# Internetworking model
# Determines how the VM should be connected to the
# the container network interface
# Options:
#
#   - bridged
#     Uses a linux bridge to interconnect the container interface to
#     the VM. Works for most cases except macvlan and ipvlan.
#
#   - macvtap
#     Used when the Container network interface can be bridged using
#     macvtap.
internetworking_model="macvtap"

# If enabled, the runtime will create opentracing.io traces and spans.
# (See https://www.jaegertracing.io/docs/getting-started).
# (default: disabled)
#enable_tracing = true

KSM throttler

version

Output of "/usr/libexec/kata-ksm-throttler/kata-ksm-throttler --version":

kata-ksm-throttler version 1.3.0-6e903fb

systemd service

Image details

---
osbuilder:
  url: "https://github.com/kata-containers/osbuilder"
  version: "unknown"
rootfs-creation-time: "2018-10-22T21:13:25.475975441+0000Z"
description: "osbuilder rootfs"
file-format-version: "0.0.2"
architecture: "x86_64"
base-distro:
  name: "Clear"
  version: "25740"
  packages:
    default:
      - "iptables-bin"
      - "libudev0-shim"
      - "systemd"
    extra:

agent:
  url: "https://github.com/kata-containers/agent"
  name: "kata-agent"
  version: "1.3.1-c7fdd324cda8e2ef01203a86d97b03a392e6eb39"
  agent-is-init-daemon: "no"

Initrd details

No initrd


Logfiles

Runtime logs

time="2018-12-03T18:01:46.585237148+08:00" level=warning msg="fetch sandbox device failed" arch=amd64 command=create container=8b0b27204c8270983d9875a1bb3566ffa630de4032301a54d2ad403a5b93fac9 error="open /run/vc/sbs/8b0b27204c8270983d9875a1bb3566ffa630de4032301a54d2ad403a5b93fac9/devices.json: no such file or directory" name=kata-runtime pid=11616 sandbox=8b0b27204c8270983d9875a1bb3566ffa630de4032301a54d2ad403a5b93fac9 sandboxid=8b0b27204c8270983d9875a1bb3566ffa630de4032301a54d2ad403a5b93fac9 source=virtcontainers subsystem=sandbox time="2018-12-03T18:02:54.129607074+08:00" level=warning msg="Could not umount" arch=amd64 command=kill container=8b0b27204c8270983d9875a1bb3566ffa630de4032301a54d2ad403a5b93fac9 error="device or resource busy" host-path=/run/kata-containers/shared/sandboxes/8b0b27204c8270983d9875a1bb3566ffa630de4032301a54d2ad403a5b93fac9/8b0b27204c8270983d9875a1bb3566ffa630de4032301a54d2ad403a5b93fac9-774fa68baf3d006e-tmp name=kata-runtime pid=11847 sandbox=8b0b27204c8270983d9875a1bb3566ffa630de4032301a54d2ad403a5b93fac9 source=virtcontainers subsystem=container time="2018-12-03T18:02:54.13255051+08:00" level=error msg="device or resource busy" arch=amd64 command=kill container=8b0b27204c8270983d9875a1bb3566ffa630de4032301a54d2ad403a5b93fac9 name=kata-runtime pid=11847 sandbox=8b0b27204c8270983d9875a1bb3566ffa630de4032301a54d2ad403a5b93fac9 source=runtime time="2018-12-03T18:02:54.160776473+08:00" level=error msg="rpc error: code = FailedPrecondition desc = Could not signal process : rpc error: code = NotFound desc = Container 8b0b27204c8270983d9875a1bb3566ffa630de4032301a54d2ad403a5b93fac9 not found" arch=amd64 command=kill container=8b0b27204c8270983d9875a1bb3566ffa630de4032301a54d2ad403a5b93fac9 name=kata-runtime pid=11867 sandbox=8b0b27204c8270983d9875a1bb3566ffa630de4032301a54d2ad403a5b93fac9 source=runtime time="2018-12-03T18:03:05.436214231+08:00" level=error msg="stat /var/lib/docker/overlay2/6f4285b36d93966a4e06abc8d0687adad352f01cf3f7965efd1c67bdfe35e028/merged: no such file or directory" arch=amd64 command=list name=kata-runtime pid=11939 source=runtime

## Proxy logs

## Shim logs

No recent shim problems found in system journal.

## Throttler logs

No recent throttler problems found in system journal.

---
# Container manager details

Have `docker`

## Docker

Output of "`docker version`":

Client: Version: 18.06.1-ce API version: 1.38 Go version: go1.10.3 Git commit: e68fc7a Built: Tue Aug 21 17:24:56 2018 OS/Arch: linux/amd64 Experimental: false

Server: Engine: Version: 18.06.1-ce API version: 1.38 (minimum version 1.12) Go version: go1.10.3 Git commit: e68fc7a Built: Tue Aug 21 17:23:21 2018 OS/Arch: linux/amd64 Experimental: false

Output of "`docker info`":

Containers: 0 Running: 0 Paused: 0 Stopped: 0 Images: 2 Server Version: 18.06.1-ce Storage Driver: overlay2 Backing Filesystem: xfs Supports d_type: true Native Overlay Diff: true Logging Driver: json-file Cgroup Driver: cgroupfs Plugins: Volume: local Network: bridge host macvlan null overlay Log: awslogs fluentd gcplogs gelf journald json-file logentries splunk syslog Swarm: inactive Runtimes: runc kata-runtime Default Runtime: runc Init Binary: docker-init containerd version: 468a545b9edcd5932818eb9de8e72413e616e86e runc version: 69663f0bd4b60df09991c08812a60108003fa340 init version: fec3683 Security Options: apparmor seccomp Profile: default Kernel Version: 4.4.0-124-generic Operating System: Ubuntu 16.04.4 LTS OSType: linux Architecture: x86_64 CPUs: 48 Total Memory: 188.8GiB ID: XEXI:7NAA:6AMF:DJNR:VEVY:JMYR:NE4M:TISE:NM3Q:ELKP:MXFD:4MXD Debug Mode (client): false Debug Mode (server): false Registry: https://index.docker.io/v1/ Labels: Experimental: false Insecure Registries: Registry Mirrors: https://registry.docker-cn.com/ Live Restore Enabled: false

Output of "`systemctl show docker`":

Type=notify Restart=on-failure NotifyAccess=main RestartUSec=100ms TimeoutStartUSec=infinity TimeoutStopUSec=1min 30s RuntimeMaxUSec=infinity WatchdogUSec=0 WatchdogTimestamp=Thu 2018-11-22 19:16:40 CST WatchdogTimestampMonotonic=9937759091 FailureAction=none PermissionsStartOnly=no RootDirectoryStartOnly=no RemainAfterExit=no GuessMainPID=yes MainPID=3970 ControlPID=0 FileDescriptorStoreMax=0 NFileDescriptorStore=0 StatusErrno=0 Result=success ExecMainStartTimestamp=Thu 2018-11-22 19:16:40 CST ExecMainStartTimestampMonotonic=9937308176 ExecMainExitTimestampMonotonic=0 ExecMainPID=3970 ExecMainCode=0 ExecMainStatus=0 ExecStart={ path=/usr/bin/dockerd ; argv[]=/usr/bin/dockerd -H fd:// ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 } ExecReload={ path=/bin/kill ; argv[]=/bin/kill -s HUP $MAINPID ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 } Slice=system.slice ControlGroup=/system.slice/docker.service MemoryCurrent=464945152 CPUUsageNSec=23989988186960 TasksCurrent=176 Delegate=yes CPUAccounting=no CPUShares=18446744073709551615 StartupCPUShares=18446744073709551615 CPUQuotaPerSecUSec=infinity BlockIOAccounting=no BlockIOWeight=18446744073709551615 StartupBlockIOWeight=18446744073709551615 MemoryAccounting=no MemoryLimit=18446744073709551615 DevicePolicy=auto TasksAccounting=no TasksMax=18446744073709551615 EnvironmentFile=/etc/default/docker (ignore_errors=no) UMask=0022 LimitCPU=18446744073709551615 LimitCPUSoft=18446744073709551615 LimitFSIZE=18446744073709551615 LimitFSIZESoft=18446744073709551615 LimitDATA=18446744073709551615 LimitDATASoft=18446744073709551615 LimitSTACK=18446744073709551615 LimitSTACKSoft=8388608 LimitCORE=18446744073709551615 LimitCORESoft=18446744073709551615 LimitRSS=18446744073709551615 LimitRSSSoft=18446744073709551615 LimitNOFILE=1048576 LimitNOFILESoft=1048576 LimitAS=18446744073709551615 LimitASSoft=18446744073709551615 LimitNPROC=18446744073709551615 LimitNPROCSoft=18446744073709551615 LimitMEMLOCK=65536 LimitMEMLOCKSoft=65536 LimitLOCKS=18446744073709551615 LimitLOCKSSoft=18446744073709551615 LimitSIGPENDING=770662 LimitSIGPENDINGSoft=770662 LimitMSGQUEUE=819200 LimitMSGQUEUESoft=819200 LimitNICE=0 LimitNICESoft=0 LimitRTPRIO=0 LimitRTPRIOSoft=0 LimitRTTIME=18446744073709551615 LimitRTTIMESoft=18446744073709551615 OOMScoreAdjust=0 Nice=0 IOScheduling=4 CPUSchedulingPolicy=0 CPUSchedulingPriority=0 TimerSlackNSec=50000 CPUSchedulingResetOnFork=no NonBlocking=no StandardInput=null StandardOutput=journal StandardError=inherit TTYReset=no TTYVHangup=no TTYVTDisallocate=no SyslogPriority=30 SyslogLevelPrefix=yes SyslogLevel=6 SyslogFacility=3 SecureBits=0 CapabilityBoundingSet=18446744073709551615 AmbientCapabilities=0 MountFlags=0 PrivateTmp=no PrivateNetwork=no PrivateDevices=no ProtectHome=no ProtectSystem=no SameProcessGroup=no UtmpMode=init IgnoreSIGPIPE=yes NoNewPrivileges=no SystemCallErrorNumber=0 RuntimeDirectoryMode=0755 KillMode=process KillSignal=15 SendSIGKILL=yes SendSIGHUP=no Id=docker.service Names=docker.service Requires=docker.socket sysinit.target system.slice Wants=network-online.target ConsistsOf=docker.socket Conflicts=shutdown.target Before=shutdown.target After=firewalld.service network-online.target basic.target docker.socket systemd-journald.socket system.slice sysinit.target TriggeredBy=docker.socket Documentation=https://docs.docker.com Description=Docker Application Container Engine LoadState=loaded ActiveState=active SubState=running FragmentPath=/lib/systemd/system/docker.service UnitFileState=disabled UnitFilePreset=enabled StateChangeTimestamp=Thu 2018-11-22 19:16:40 CST StateChangeTimestampMonotonic=9937759093 InactiveExitTimestamp=Thu 2018-11-22 19:16:40 CST InactiveExitTimestampMonotonic=9937308244 ActiveEnterTimestamp=Thu 2018-11-22 19:16:40 CST ActiveEnterTimestampMonotonic=9937759093 ActiveExitTimestampMonotonic=0 InactiveEnterTimestampMonotonic=0 CanStart=yes CanStop=yes CanReload=yes CanIsolate=no StopWhenUnneeded=no RefuseManualStart=no RefuseManualStop=no AllowIsolate=no DefaultDependencies=yes OnFailureJobMode=replace IgnoreOnIsolate=no NeedDaemonReload=no JobTimeoutUSec=infinity JobTimeoutAction=none ConditionResult=yes AssertResult=yes ConditionTimestamp=Thu 2018-11-22 19:16:40 CST ConditionTimestampMonotonic=9937290899 AssertTimestamp=Thu 2018-11-22 19:16:40 CST AssertTimestampMonotonic=9937290902 Transient=no StartLimitInterval=60000000 StartLimitBurst=3 StartLimitAction=none

No `kubectl`

---

# Packages

Have `dpkg`
Output of "`dpkg -l|egrep "(cc-oci-runtimecc-runtimerunv|kata-proxy|kata-runtime|kata-shim|kata-ksm-throttler|kata-containers-image|linux-container|qemu-)"`":

ii kata-containers-image 1.3.1-36 amd64 Kata containers image ii kata-ksm-throttler 1.3.1.git+6e903fb-37 amd64 ii kata-linux-container 4.14.67.16-139 amd64 linux kernel optimised for container-like workloads. ii kata-proxy 1.3.1+git.d364b2e-36 amd64 ii kata-runtime 1.3.1+git.258eae0-51 amd64 ii kata-shim 1.3.1+git.58f757d-37 amd64 ii qemu-lite 2.11.0+git.f886228056-52 amd64 linux kernel optimised for container-like workloads. ii qemu-vanilla 2.11.2+git.0982a56a55-46 amd64 linux kernel optimised for container-like workloads.


No `rpm`

---
jodh-intel commented 5 years ago

Hi @running99 - I don't see this behaviour. Please can you confirm what version you are running. Either:

luning25 commented 5 years ago

The output of kata-collect-data.sh has been pasted.

jodh-intel commented 5 years ago

Thanks @running99. That shows you are not using 1.4.0. Please could you try upgrading and see if the behaviour is resolved for you?

luning25 commented 5 years ago

I got the same error.

kata-runtime  : 1.4.0
   commit   : 0a7a4379dc15a7690df83333e3bc2eef882bf302
   OCI specs: 1.0.1-dev
zhiminghufighting commented 5 years ago

@jodh-intel, i can reproduce it stably in my two centos7.3 hosts with different os minor version + kata 1.3 $ 1.4. The basic step is the same as @running99's cmd and no error is report in my screen. If you run the cmd "ps -aux| grep kata" after "docker stop bindtest", you can see qemu-lite is still running and it will not exit. At the same time, you can check the mount list in system with "cat /etc/mtab | grep run", then you can see the volume /mnt/tmp being mounted into bindtest container is not be umounted normally. 183389149 390560046

zhiminghufighting commented 5 years ago

@jodh-intel, It can be reproduced in running99's debian and in my centos 7.3. I attach the screenshot from my one of centos7.3. Could you please help verify it in your environment according to my method? Thanks a lot!

devimc commented 5 years ago

same issue :worried:

zhiminghufighting commented 5 years ago

@devimc thanks for your inputs! It should be a real issue.

zhiminghufighting commented 5 years ago

@jodh-intel, can we confirm this issue is a real issue now?