Closed luning25 closed 5 years ago
Hi @running99 - I don't see this behaviour. Please can you confirm what version you are running. Either:
kata-runtime --version
.Or (much more helpful) paste the output of sudo kata-collect-data.sh
directly into a comment here as requested in the template.
Remember to check to ensure the output doesn't contain any personal / sensitive information first!
The output of kata-collect-data.sh
has been pasted.
Thanks @running99. That shows you are not using 1.4.0. Please could you try upgrading and see if the behaviour is resolved for you?
I got the same error.
kata-runtime : 1.4.0
commit : 0a7a4379dc15a7690df83333e3bc2eef882bf302
OCI specs: 1.0.1-dev
@jodh-intel, i can reproduce it stably in my two centos7.3 hosts with different os minor version + kata 1.3 $ 1.4. The basic step is the same as @running99's cmd and no error is report in my screen. If you run the cmd "ps -aux| grep kata" after "docker stop bindtest", you can see qemu-lite is still running and it will not exit. At the same time, you can check the mount list in system with "cat /etc/mtab | grep run", then you can see the volume /mnt/tmp being mounted into bindtest container is not be umounted normally.
@jodh-intel, It can be reproduced in running99's debian and in my centos 7.3. I attach the screenshot from my one of centos7.3. Could you please help verify it in your environment according to my method? Thanks a lot!
same issue :worried:
@devimc thanks for your inputs! It should be a real issue.
@jodh-intel, can we confirm this issue is a real issue now?
Description of problem
docker run --rm -d -it --runtime=kata-runtime -v /mnt/tmp:/mnt/tmp:rslave --name bindtest ubuntu mkdir /mnt/tmp/1 mount --bind /home/disk2/ /mnt/tmp/1/ docker stop bindtest
Expected result
Actual result
kata-runtime: level=warning msg="Could not umount" error="device or resource busy"
Meta details
Running
kata-collect-data.sh
version1.3.1 (commit 258eae0)
at2018-12-03.18:04:52.399496662+0800
.Runtime is
/usr/bin/kata-runtime
.kata-env
Output of "
/usr/bin/kata-runtime kata-env
":Runtime config files
Runtime default config files
Runtime config file contents
Config file
/etc/kata-containers/configuration.toml
not found Output of "cat "/usr/share/defaults/kata-containers/configuration.toml"
":KSM throttler
version
Output of "
/usr/libexec/kata-ksm-throttler/kata-ksm-throttler --version
":systemd service
Image details
Initrd details
No initrd
Logfiles
Runtime logs
time="2018-12-03T18:01:46.585237148+08:00" level=warning msg="fetch sandbox device failed" arch=amd64 command=create container=8b0b27204c8270983d9875a1bb3566ffa630de4032301a54d2ad403a5b93fac9 error="open /run/vc/sbs/8b0b27204c8270983d9875a1bb3566ffa630de4032301a54d2ad403a5b93fac9/devices.json: no such file or directory" name=kata-runtime pid=11616 sandbox=8b0b27204c8270983d9875a1bb3566ffa630de4032301a54d2ad403a5b93fac9 sandboxid=8b0b27204c8270983d9875a1bb3566ffa630de4032301a54d2ad403a5b93fac9 source=virtcontainers subsystem=sandbox time="2018-12-03T18:02:54.129607074+08:00" level=warning msg="Could not umount" arch=amd64 command=kill container=8b0b27204c8270983d9875a1bb3566ffa630de4032301a54d2ad403a5b93fac9 error="device or resource busy" host-path=/run/kata-containers/shared/sandboxes/8b0b27204c8270983d9875a1bb3566ffa630de4032301a54d2ad403a5b93fac9/8b0b27204c8270983d9875a1bb3566ffa630de4032301a54d2ad403a5b93fac9-774fa68baf3d006e-tmp name=kata-runtime pid=11847 sandbox=8b0b27204c8270983d9875a1bb3566ffa630de4032301a54d2ad403a5b93fac9 source=virtcontainers subsystem=container time="2018-12-03T18:02:54.13255051+08:00" level=error msg="device or resource busy" arch=amd64 command=kill container=8b0b27204c8270983d9875a1bb3566ffa630de4032301a54d2ad403a5b93fac9 name=kata-runtime pid=11847 sandbox=8b0b27204c8270983d9875a1bb3566ffa630de4032301a54d2ad403a5b93fac9 source=runtime time="2018-12-03T18:02:54.160776473+08:00" level=error msg="rpc error: code = FailedPrecondition desc = Could not signal process : rpc error: code = NotFound desc = Container 8b0b27204c8270983d9875a1bb3566ffa630de4032301a54d2ad403a5b93fac9 not found" arch=amd64 command=kill container=8b0b27204c8270983d9875a1bb3566ffa630de4032301a54d2ad403a5b93fac9 name=kata-runtime pid=11867 sandbox=8b0b27204c8270983d9875a1bb3566ffa630de4032301a54d2ad403a5b93fac9 source=runtime time="2018-12-03T18:03:05.436214231+08:00" level=error msg="stat /var/lib/docker/overlay2/6f4285b36d93966a4e06abc8d0687adad352f01cf3f7965efd1c67bdfe35e028/merged: no such file or directory" arch=amd64 command=list name=kata-runtime pid=11939 source=runtime
Client: Version: 18.06.1-ce API version: 1.38 Go version: go1.10.3 Git commit: e68fc7a Built: Tue Aug 21 17:24:56 2018 OS/Arch: linux/amd64 Experimental: false
Server: Engine: Version: 18.06.1-ce API version: 1.38 (minimum version 1.12) Go version: go1.10.3 Git commit: e68fc7a Built: Tue Aug 21 17:23:21 2018 OS/Arch: linux/amd64 Experimental: false
Containers: 0 Running: 0 Paused: 0 Stopped: 0 Images: 2 Server Version: 18.06.1-ce Storage Driver: overlay2 Backing Filesystem: xfs Supports d_type: true Native Overlay Diff: true Logging Driver: json-file Cgroup Driver: cgroupfs Plugins: Volume: local Network: bridge host macvlan null overlay Log: awslogs fluentd gcplogs gelf journald json-file logentries splunk syslog Swarm: inactive Runtimes: runc kata-runtime Default Runtime: runc Init Binary: docker-init containerd version: 468a545b9edcd5932818eb9de8e72413e616e86e runc version: 69663f0bd4b60df09991c08812a60108003fa340 init version: fec3683 Security Options: apparmor seccomp Profile: default Kernel Version: 4.4.0-124-generic Operating System: Ubuntu 16.04.4 LTS OSType: linux Architecture: x86_64 CPUs: 48 Total Memory: 188.8GiB ID: XEXI:7NAA:6AMF:DJNR:VEVY:JMYR:NE4M:TISE:NM3Q:ELKP:MXFD:4MXD Debug Mode (client): false Debug Mode (server): false Registry: https://index.docker.io/v1/ Labels: Experimental: false Insecure Registries: Registry Mirrors: https://registry.docker-cn.com/ Live Restore Enabled: false
Type=notify Restart=on-failure NotifyAccess=main RestartUSec=100ms TimeoutStartUSec=infinity TimeoutStopUSec=1min 30s RuntimeMaxUSec=infinity WatchdogUSec=0 WatchdogTimestamp=Thu 2018-11-22 19:16:40 CST WatchdogTimestampMonotonic=9937759091 FailureAction=none PermissionsStartOnly=no RootDirectoryStartOnly=no RemainAfterExit=no GuessMainPID=yes MainPID=3970 ControlPID=0 FileDescriptorStoreMax=0 NFileDescriptorStore=0 StatusErrno=0 Result=success ExecMainStartTimestamp=Thu 2018-11-22 19:16:40 CST ExecMainStartTimestampMonotonic=9937308176 ExecMainExitTimestampMonotonic=0 ExecMainPID=3970 ExecMainCode=0 ExecMainStatus=0 ExecStart={ path=/usr/bin/dockerd ; argv[]=/usr/bin/dockerd -H fd:// ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 } ExecReload={ path=/bin/kill ; argv[]=/bin/kill -s HUP $MAINPID ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 } Slice=system.slice ControlGroup=/system.slice/docker.service MemoryCurrent=464945152 CPUUsageNSec=23989988186960 TasksCurrent=176 Delegate=yes CPUAccounting=no CPUShares=18446744073709551615 StartupCPUShares=18446744073709551615 CPUQuotaPerSecUSec=infinity BlockIOAccounting=no BlockIOWeight=18446744073709551615 StartupBlockIOWeight=18446744073709551615 MemoryAccounting=no MemoryLimit=18446744073709551615 DevicePolicy=auto TasksAccounting=no TasksMax=18446744073709551615 EnvironmentFile=/etc/default/docker (ignore_errors=no) UMask=0022 LimitCPU=18446744073709551615 LimitCPUSoft=18446744073709551615 LimitFSIZE=18446744073709551615 LimitFSIZESoft=18446744073709551615 LimitDATA=18446744073709551615 LimitDATASoft=18446744073709551615 LimitSTACK=18446744073709551615 LimitSTACKSoft=8388608 LimitCORE=18446744073709551615 LimitCORESoft=18446744073709551615 LimitRSS=18446744073709551615 LimitRSSSoft=18446744073709551615 LimitNOFILE=1048576 LimitNOFILESoft=1048576 LimitAS=18446744073709551615 LimitASSoft=18446744073709551615 LimitNPROC=18446744073709551615 LimitNPROCSoft=18446744073709551615 LimitMEMLOCK=65536 LimitMEMLOCKSoft=65536 LimitLOCKS=18446744073709551615 LimitLOCKSSoft=18446744073709551615 LimitSIGPENDING=770662 LimitSIGPENDINGSoft=770662 LimitMSGQUEUE=819200 LimitMSGQUEUESoft=819200 LimitNICE=0 LimitNICESoft=0 LimitRTPRIO=0 LimitRTPRIOSoft=0 LimitRTTIME=18446744073709551615 LimitRTTIMESoft=18446744073709551615 OOMScoreAdjust=0 Nice=0 IOScheduling=4 CPUSchedulingPolicy=0 CPUSchedulingPriority=0 TimerSlackNSec=50000 CPUSchedulingResetOnFork=no NonBlocking=no StandardInput=null StandardOutput=journal StandardError=inherit TTYReset=no TTYVHangup=no TTYVTDisallocate=no SyslogPriority=30 SyslogLevelPrefix=yes SyslogLevel=6 SyslogFacility=3 SecureBits=0 CapabilityBoundingSet=18446744073709551615 AmbientCapabilities=0 MountFlags=0 PrivateTmp=no PrivateNetwork=no PrivateDevices=no ProtectHome=no ProtectSystem=no SameProcessGroup=no UtmpMode=init IgnoreSIGPIPE=yes NoNewPrivileges=no SystemCallErrorNumber=0 RuntimeDirectoryMode=0755 KillMode=process KillSignal=15 SendSIGKILL=yes SendSIGHUP=no Id=docker.service Names=docker.service Requires=docker.socket sysinit.target system.slice Wants=network-online.target ConsistsOf=docker.socket Conflicts=shutdown.target Before=shutdown.target After=firewalld.service network-online.target basic.target docker.socket systemd-journald.socket system.slice sysinit.target TriggeredBy=docker.socket Documentation=https://docs.docker.com Description=Docker Application Container Engine LoadState=loaded ActiveState=active SubState=running FragmentPath=/lib/systemd/system/docker.service UnitFileState=disabled UnitFilePreset=enabled StateChangeTimestamp=Thu 2018-11-22 19:16:40 CST StateChangeTimestampMonotonic=9937759093 InactiveExitTimestamp=Thu 2018-11-22 19:16:40 CST InactiveExitTimestampMonotonic=9937308244 ActiveEnterTimestamp=Thu 2018-11-22 19:16:40 CST ActiveEnterTimestampMonotonic=9937759093 ActiveExitTimestampMonotonic=0 InactiveEnterTimestampMonotonic=0 CanStart=yes CanStop=yes CanReload=yes CanIsolate=no StopWhenUnneeded=no RefuseManualStart=no RefuseManualStop=no AllowIsolate=no DefaultDependencies=yes OnFailureJobMode=replace IgnoreOnIsolate=no NeedDaemonReload=no JobTimeoutUSec=infinity JobTimeoutAction=none ConditionResult=yes AssertResult=yes ConditionTimestamp=Thu 2018-11-22 19:16:40 CST ConditionTimestampMonotonic=9937290899 AssertTimestamp=Thu 2018-11-22 19:16:40 CST AssertTimestampMonotonic=9937290902 Transient=no StartLimitInterval=60000000 StartLimitBurst=3 StartLimitAction=none
ii kata-containers-image 1.3.1-36 amd64 Kata containers image ii kata-ksm-throttler 1.3.1.git+6e903fb-37 amd64 ii kata-linux-container 4.14.67.16-139 amd64 linux kernel optimised for container-like workloads. ii kata-proxy 1.3.1+git.d364b2e-36 amd64 ii kata-runtime 1.3.1+git.258eae0-51 amd64 ii kata-shim 1.3.1+git.58f757d-37 amd64 ii qemu-lite 2.11.0+git.f886228056-52 amd64 linux kernel optimised for container-like workloads. ii qemu-vanilla 2.11.2+git.0982a56a55-46 amd64 linux kernel optimised for container-like workloads.