This was originally seen on CAHC but it can also be reproduced using vanilla CentOS and the docker package from the virt7-candidate-common-candidaterepo
When trying to build a simple httpd container, an error from devmapper is observed.
devmapper: Error mounting '/dev/mapper/docker-253:1-25165909-d910cb56f603bf9819b87742c683ad088c33873d6f73f0e42b83c5d0b9d891f3' on '/var/lib/docker/devicemapper/mnt/d910cb56f603bf9819b87742c683ad088c33873d6f73f0e
42b83c5d0b9d891f3': invalid argument
Note this is only observable when using the devicemapper storage driver. When I switch to the overlay2 driver, this is NOT observed.
See conditions below:
# rpm -q docker
docker-1.13.1-35.git8fd0ebb.el7.x86_64
# cat centos_httpd_Dockerfile
FROM registry.centos.org/centos/centos:latest
MAINTAINER Micah Abbott <micah@redhat.com>
LABEL Version=1.2
LABEL RUN="docker run -d --name NAME -p 80:80 IMAGE"
ENV container docker
ADD makecache.sh /
RUN /makecache.sh && \
yum -y install httpd && \
yum clean all
RUN echo "SUCCESS centos_httpd" > /var/www/html/index.html
EXPOSE 80
ENTRYPOINT [ "/usr/sbin/httpd" ]
CMD [ "-D", "FOREGROUND" ]
# cat makecache.sh
#!/bin/bash
set -xeou pipefail
retries=5
while [ $retries -gt 0 ]; do
if yum makecache; then
break
fi
retries=$((retries - 1))
done
# docker build -t centos_httpd -f centos_httpd_Dockerfile .
Sending build context to Docker daemon 3.072 kB
Step 1/11 : FROM registry.centos.org/centos/centos:latest
Trying to pull repository registry.centos.org/centos/centos ...
sha256:8a11240d5dec9248d678e75b800c8f6c8f0676973708b118be2a8a533872df88: Pulling from registry.centos.org/centos/centos
66881a2148ed: Pull complete
Digest: sha256:8a11240d5dec9248d678e75b800c8f6c8f0676973708b118be2a8a533872df88
Status: Downloaded newer image for registry.centos.org/centos/centos:latest
---> 12bb732d09c5
Step 2/11 : MAINTAINER Micah Abbott <micah@redhat.com>
---> Running in f1b8bf63b595
---> fce3b0e3cf65
Removing intermediate container f1b8bf63b595
Step 3/11 : LABEL Version 1.2
---> Running in a624dfa6e855
---> 3f36c312a945
Removing intermediate container a624dfa6e855
Step 4/11 : LABEL RUN "docker run -d --name NAME -p 80:80 IMAGE"
---> Running in 8318d108bbef
---> 6f2cb050d843
Removing intermediate container 8318d108bbef
Step 5/11 : ENV container docker
---> Running in 4f527bd1b7fa
---> a67c8f6c57d9
Removing intermediate container 4f527bd1b7fa
Step 6/11 : ADD makecache.sh /
---> 81fc2e2700f6
Removing intermediate container 0ecc51c4b1e9
Step 7/11 : RUN /makecache.sh && yum -y install httpd && yum clean all
---> Running in b7a17ac27593
+ retries=5
+ '[' 5 -gt 0 ']'
+ yum makecache
Loaded plugins: fastestmirror, ovl
Determining fastest mirrors
* base: repo1.ash.innoscale.net
* extras: mirror.wdc1.us.leaseweb.net
* updates: mirror.vtti.vt.edu
Metadata Cache Created
+ break
Loaded plugins: fastestmirror, ovl
Loading mirror speeds from cached hostfile
* base: repo1.ash.innoscale.net
* extras: mirror.wdc1.us.leaseweb.net
* updates: mirror.vtti.vt.edu
Resolving Dependencies
--> Running transaction check
---> Package httpd.x86_64 0:2.4.6-67.el7.centos.6 will be installed
--> Processing Dependency: httpd-tools = 2.4.6-67.el7.centos.6 for package: httpd-2.4.6-67.el7.centos.6.x86_64
--> Processing Dependency: system-logos >= 7.92.1-1 for package: httpd-2.4.6-67.el7.centos.6.x86_64
--> Processing Dependency: /etc/mime.types for package: httpd-2.4.6-67.el7.centos.6.x86_64
--> Processing Dependency: libaprutil-1.so.0()(64bit) for package: httpd-2.4.6-67.el7.centos.6.x86_64
--> Processing Dependency: libapr-1.so.0()(64bit) for package: httpd-2.4.6-67.el7.centos.6.x86_64
--> Running transaction check
---> Package apr.x86_64 0:1.4.8-3.el7 will be installed
---> Package apr-util.x86_64 0:1.5.2-6.el7 will be installed
---> Package centos-logos.noarch 0:70.0.6-3.el7.centos will be installed
---> Package httpd-tools.x86_64 0:2.4.6-67.el7.centos.6 will be installed
---> Package mailcap.noarch 0:2.1.41-2.el7 will be installed
--> Finished Dependency Resolution
Dependencies Resolved
================================================================================
Package Arch Version Repository Size
================================================================================
Installing:
httpd x86_64 2.4.6-67.el7.centos.6 updates 2.7 M
Installing for dependencies:
apr x86_64 1.4.8-3.el7 base 103 k
apr-util x86_64 1.5.2-6.el7 base 92 k
centos-logos noarch 70.0.6-3.el7.centos base 21 M
httpd-tools x86_64 2.4.6-67.el7.centos.6 updates 88 k
mailcap noarch 2.1.41-2.el7 base 31 k
Transaction Summary
================================================================================
Install 1 Package (+5 Dependent packages)
Total download size: 24 M
Installed size: 32 M
Downloading packages:
warning: /var/cache/yum/x86_64/7/updates/packages/httpd-tools-2.4.6-67.el7.centos.6.x86_64.rpm: Header V3 RSA/SHA256 Signature, key ID f4a80eb5: NOKEY
Public key for httpd-tools-2.4.6-67.el7.centos.6.x86_64.rpm is not installed
Public key for apr-util-1.5.2-6.el7.x86_64.rpm is not installed
--------------------------------------------------------------------------------
Total 5.7 MB/s | 24 MB 00:04
Retrieving key from file:///etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-7
Importing GPG key 0xF4A80EB5:
Userid : "CentOS-7 Key (CentOS 7 Official Signing Key) <security@centos.org>"
Fingerprint: 6341 ab27 53d7 8a78 a7c2 7bb1 24c6 a8a7 f4a8 0eb5
Package : centos-release-7-4.1708.el7.centos.x86_64 (@CentOS)
From : /etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-7
Running transaction check
Running transaction test
Transaction test succeeded
Running transaction
Installing : apr-1.4.8-3.el7.x86_64 1/6
Installing : apr-util-1.5.2-6.el7.x86_64 2/6
Installing : httpd-tools-2.4.6-67.el7.centos.6.x86_64 3/6
Installing : centos-logos-70.0.6-3.el7.centos.noarch 4/6
Installing : mailcap-2.1.41-2.el7.noarch 5/6
Installing : httpd-2.4.6-67.el7.centos.6.x86_64 6/6
Verifying : apr-1.4.8-3.el7.x86_64 1/6
Verifying : mailcap-2.1.41-2.el7.noarch 2/6
Verifying : httpd-2.4.6-67.el7.centos.6.x86_64 3/6
Verifying : apr-util-1.5.2-6.el7.x86_64 4/6
Verifying : httpd-tools-2.4.6-67.el7.centos.6.x86_64 5/6
Verifying : centos-logos-70.0.6-3.el7.centos.noarch 6/6
Installed:
httpd.x86_64 0:2.4.6-67.el7.centos.6
Dependency Installed:
apr.x86_64 0:1.4.8-3.el7
apr-util.x86_64 0:1.5.2-6.el7
centos-logos.noarch 0:70.0.6-3.el7.centos
httpd-tools.x86_64 0:2.4.6-67.el7.centos.6
mailcap.noarch 0:2.1.41-2.el7
Complete!
Loaded plugins: fastestmirror, ovl
Cleaning repos: base extras updates
Cleaning up everything
Maybe you want: rm -rf /var/cache/yum, to also free up space taken by orphaned data from disabled or removed repos
Cleaning up list of fastest mirrors
devmapper: Error mounting '/dev/mapper/docker-253:1-25165909-d910cb56f603bf9819b87742c683ad088c33873d6f73f0e42b83c5d0b9d891f3' on '/var/lib/docker/devicemapper/mnt/d910cb56f603bf9819b87742c683ad088c33873d6f73f0e
42b83c5d0b9d891f3': invalid argument
# journalctl -b --since 15:33:45
-- Logs begin at Wed 2017-11-08 15:11:19 UTC, end at Wed 2017-11-08 15:35:33 UTC. --
Nov 08 15:33:45 micah-centos-vm1108a dockerd-current[11672]: time="2017-11-08T15:33:45.918568560Z" level=info msg="{Action=_ping, Username=cloud-user, LoginUID=1000, PID=11749}"
Nov 08 15:33:45 micah-centos-vm1108a dockerd-current[11672]: time="2017-11-08T15:33:45.920770815Z" level=info msg="{Action=build, Username=cloud-user, LoginUID=1000, PID=11749}"
Nov 08 15:33:47 micah-centos-vm1108a kernel: XFS (dm-3): Mounting V5 Filesystem
Nov 08 15:33:48 micah-centos-vm1108a kernel: XFS (dm-3): Ending clean mount
Nov 08 15:34:01 micah-centos-vm1108a kernel: XFS (dm-3): Unmounting Filesystem
Nov 08 15:34:01 micah-centos-vm1108a kernel: XFS (dm-3): Mounting V5 Filesystem
Nov 08 15:34:01 micah-centos-vm1108a kernel: XFS (dm-3): Ending clean mount
Nov 08 15:34:02 micah-centos-vm1108a kernel: XFS (dm-3): Unmounting Filesystem
Nov 08 15:34:02 micah-centos-vm1108a kernel: XFS (dm-3): Mounting V5 Filesystem
Nov 08 15:34:02 micah-centos-vm1108a kernel: XFS (dm-3): Ending clean mount
Nov 08 15:34:02 micah-centos-vm1108a kernel: XFS (dm-3): Unmounting Filesystem
Nov 08 15:34:02 micah-centos-vm1108a kernel: XFS (dm-3): Mounting V5 Filesystem
Nov 08 15:34:02 micah-centos-vm1108a kernel: XFS (dm-3): Ending clean mount
Nov 08 15:34:02 micah-centos-vm1108a kernel: XFS (dm-4): Mounting V5 Filesystem
Nov 08 15:34:02 micah-centos-vm1108a kernel: XFS (dm-4): Ending clean mount
Nov 08 15:34:04 micah-centos-vm1108a kernel: XFS (dm-4): Unmounting Filesystem Nov 08 15:34:05 micah-centos-vm1108a kernel: XFS (dm-4): Mounting V5 Filesystem
Nov 08 15:34:05 micah-centos-vm1108a kernel: XFS (dm-4): Ending clean mount
Nov 08 15:34:05 micah-centos-vm1108a kernel: XFS (dm-4): Unmounting Filesystem
Nov 08 15:34:06 micah-centos-vm1108a dockerd-current[11672]: time="2017-11-08T15:34:06.148615311Z" level=info msg="Layer sha256:c27beeb315f6ed4830cdc23257775a1ddb2a944ea512e8da4a0f986de65671d7 cleaned up"
Nov 08 15:34:06 micah-centos-vm1108a kernel: XFS (dm-3): Unmounting Filesystem Nov 08 15:34:06 micah-centos-vm1108a kernel: XFS (dm-3): Mounting V5 Filesystem
Nov 08 15:34:07 micah-centos-vm1108a kernel: XFS (dm-3): Ending clean mount
Nov 08 15:34:07 micah-centos-vm1108a kernel: XFS (dm-3): Unmounting Filesystem
Nov 08 15:34:08 micah-centos-vm1108a kernel: XFS (dm-3): Mounting V5 Filesystem
Nov 08 15:34:08 micah-centos-vm1108a kernel: XFS (dm-3): Ending clean mount
Nov 08 15:34:08 micah-centos-vm1108a kernel: XFS (dm-3): Unmounting Filesystem
Nov 08 15:34:08 micah-centos-vm1108a kernel: XFS (dm-3): Mounting V5 Filesystem
Nov 08 15:34:08 micah-centos-vm1108a kernel: XFS (dm-3): Ending clean mount
Nov 08 15:34:08 micah-centos-vm1108a kernel: XFS (dm-4): Mounting V5 Filesystem
Nov 08 15:34:08 micah-centos-vm1108a kernel: XFS (dm-4): Ending clean mount
Nov 08 15:34:09 micah-centos-vm1108a kernel: XFS (dm-4): Unmounting Filesystem
Nov 08 15:34:11 micah-centos-vm1108a kernel: XFS (dm-4): Mounting V5 Filesystem
Nov 08 15:34:11 micah-centos-vm1108a kernel: XFS (dm-4): Ending clean mount
Nov 08 15:34:11 micah-centos-vm1108a kernel: XFS (dm-4): Unmounting Filesystem
Nov 08 15:34:11 micah-centos-vm1108a dockerd-current[11672]: time="2017-11-08T15:34:11.845338377Z" level=info msg="Layer sha256:c27beeb315f6ed4830cdc23257775a1ddb2a944ea512e8da4a0f986de65671d7 cleaned up"
Nov 08 15:34:11 micah-centos-vm1108a kernel: XFS (dm-3): Unmounting Filesystem
Nov 08 15:34:12 micah-centos-vm1108a kernel: XFS (dm-3): Mounting V5 Filesystem
Nov 08 15:34:12 micah-centos-vm1108a kernel: XFS (dm-3): Ending clean mount
Nov 08 15:34:12 micah-centos-vm1108a kernel: XFS (dm-3): Unmounting Filesystem
Nov 08 15:34:12 micah-centos-vm1108a kernel: XFS (dm-3): Mounting V5 Filesystem
Nov 08 15:34:13 micah-centos-vm1108a kernel: XFS (dm-3): Ending clean mount
Nov 08 15:34:13 micah-centos-vm1108a kernel: XFS (dm-3): Unmounting Filesystem
Nov 08 15:34:13 micah-centos-vm1108a kernel: XFS (dm-3): Mounting V5 Filesystem
Nov 08 15:34:13 micah-centos-vm1108a kernel: XFS (dm-3): Ending clean mount
Nov 08 15:34:13 micah-centos-vm1108a kernel: XFS (dm-4): Mounting V5 Filesystem
Nov 08 15:34:13 micah-centos-vm1108a kernel: XFS (dm-4): Ending clean mount
Nov 08 15:34:14 micah-centos-vm1108a kernel: XFS (dm-4): Unmounting Filesystem
Nov 08 15:34:15 micah-centos-vm1108a kernel: XFS (dm-4): Mounting V5 Filesystem
Nov 08 15:34:15 micah-centos-vm1108a kernel: XFS (dm-4): Ending clean mount
Nov 08 15:34:15 micah-centos-vm1108a kernel: XFS (dm-4): Unmounting Filesystem
Nov 08 15:34:16 micah-centos-vm1108a dockerd-current[11672]: time="2017-11-08T15:34:16.163424906Z" level=info msg="Layer sha256:c27beeb315f6ed4830cdc23257775a1ddb2a944ea512e8da4a0f986de65671d7 cleaned up"
Nov 08 15:34:16 micah-centos-vm1108a kernel: XFS (dm-3): Unmounting Filesystem
Nov 08 15:34:17 micah-centos-vm1108a kernel: XFS (dm-3): Mounting V5 Filesystem
Nov 08 15:34:19 micah-centos-vm1108a kernel: XFS (dm-3): Ending clean mount
Nov 08 15:34:19 micah-centos-vm1108a kernel: XFS (dm-3): Unmounting Filesystem
Nov 08 15:34:19 micah-centos-vm1108a kernel: XFS (dm-3): Mounting V5 Filesystem
Nov 08 15:34:19 micah-centos-vm1108a kernel: XFS (dm-3): Ending clean mount
Nov 08 15:34:19 micah-centos-vm1108a kernel: XFS (dm-3): Unmounting Filesystem
Nov 08 15:34:19 micah-centos-vm1108a kernel: XFS (dm-3): Mounting V5 Filesystem
Nov 08 15:34:19 micah-centos-vm1108a kernel: XFS (dm-3): Ending clean mount
Nov 08 15:34:19 micah-centos-vm1108a kernel: XFS (dm-4): Mounting V5 Filesystem
Nov 08 15:34:19 micah-centos-vm1108a kernel: XFS (dm-4): Ending clean mount
Nov 08 15:34:22 micah-centos-vm1108a kernel: XFS (dm-4): Unmounting Filesystem
Nov 08 15:34:25 micah-centos-vm1108a kernel: XFS (dm-4): Mounting V5 Filesystem
Nov 08 15:34:25 micah-centos-vm1108a kernel: XFS (dm-4): Ending clean mount
Nov 08 15:34:25 micah-centos-vm1108a kernel: XFS (dm-4): Unmounting Filesystem
Nov 08 15:34:25 micah-centos-vm1108a dockerd-current[11672]: time="2017-11-08T15:34:25.494801703Z" level=info msg="Layer sha256:c27beeb315f6ed4830cdc23257775a1ddb2a944ea512e8da4a0f986de65671d7 cleaned up"
Nov 08 15:34:25 micah-centos-vm1108a kernel: XFS (dm-3): Unmounting Filesystem
Nov 08 15:34:26 micah-centos-vm1108a kernel: XFS (dm-3): Mounting V5 Filesystem
Nov 08 15:34:26 micah-centos-vm1108a kernel: XFS (dm-3): Ending clean mount
Nov 08 15:34:26 micah-centos-vm1108a kernel: XFS (dm-3): Unmounting Filesystem
Nov 08 15:34:27 micah-centos-vm1108a kernel: XFS (dm-3): Mounting V5 Filesystem
Nov 08 15:34:27 micah-centos-vm1108a kernel: XFS (dm-3): Ending clean mount
Nov 08 15:34:27 micah-centos-vm1108a kernel: XFS (dm-3): Unmounting Filesystem
Nov 08 15:34:27 micah-centos-vm1108a kernel: XFS (dm-3): Mounting V5 Filesystem
Nov 08 15:34:27 micah-centos-vm1108a kernel: XFS (dm-3): Ending clean mount
Nov 08 15:34:27 micah-centos-vm1108a kernel: XFS (dm-3): Unmounting Filesystem
Nov 08 15:34:27 micah-centos-vm1108a kernel: XFS (dm-3): Mounting V5 Filesystem
Nov 08 15:34:27 micah-centos-vm1108a kernel: XFS (dm-3): Ending clean mount
Nov 08 15:34:27 micah-centos-vm1108a kernel: XFS (dm-4): Mounting V5 Filesystem
Nov 08 15:34:27 micah-centos-vm1108a kernel: XFS (dm-4): Ending clean mount
Nov 08 15:34:28 micah-centos-vm1108a kernel: XFS (dm-4): Unmounting Filesystem
Nov 08 15:34:30 micah-centos-vm1108a kernel: XFS (dm-4): Mounting V5 Filesystem
Nov 08 15:34:30 micah-centos-vm1108a kernel: XFS (dm-4): Ending clean mount
Nov 08 15:34:30 micah-centos-vm1108a kernel: XFS (dm-4): Unmounting Filesystem
Nov 08 15:34:30 micah-centos-vm1108a kernel: XFS (dm-3): Unmounting Filesystem
Nov 08 15:34:31 micah-centos-vm1108a kernel: XFS (dm-3): Mounting V5 Filesystem
Nov 08 15:34:32 micah-centos-vm1108a kernel: XFS (dm-3): Ending clean mount
Nov 08 15:34:32 micah-centos-vm1108a kernel: XFS (dm-3): Unmounting Filesystem
Nov 08 15:34:32 micah-centos-vm1108a kernel: XFS (dm-3): Mounting V5 Filesystem
Nov 08 15:34:32 micah-centos-vm1108a kernel: XFS (dm-3): Ending clean mount
Nov 08 15:34:32 micah-centos-vm1108a kernel: XFS (dm-3): Unmounting Filesystem
Nov 08 15:34:32 micah-centos-vm1108a kernel: XFS (dm-3): Mounting V5 Filesystem
Nov 08 15:34:32 micah-centos-vm1108a kernel: XFS (dm-3): Ending clean mount
Nov 08 15:34:32 micah-centos-vm1108a kernel: docker0: port 1(vethbb25335) entered blocking state
Nov 08 15:34:32 micah-centos-vm1108a kernel: docker0: port 1(vethbb25335) entered disabled state
Nov 08 15:34:32 micah-centos-vm1108a kernel: device vethbb25335 entered promiscuous mode
Nov 08 15:34:32 micah-centos-vm1108a kernel: IPv6: ADDRCONF(NETDEV_UP): vethbb25335: link is not ready
Nov 08 15:34:32 micah-centos-vm1108a kernel: docker0: port 1(vethbb25335) entered blocking state
Nov 08 15:34:32 micah-centos-vm1108a kernel: docker0: port 1(vethbb25335) entered forwarding state
Nov 08 15:34:32 micah-centos-vm1108a kernel: docker0: port 1(vethbb25335) entered disabled state
Nov 08 15:34:32 micah-centos-vm1108a systemd[1]: Started libcontainer container b7a17ac2759319055628225b1d850c1f248ac182edf0ef099d816630b2a05119.
Nov 08 15:34:32 micah-centos-vm1108a systemd[1]: Starting libcontainer container b7a17ac2759319055628225b1d850c1f248ac182edf0ef099d816630b2a05119.
Nov 08 15:34:33 micah-centos-vm1108a kernel: SELinux: mount invalid. Same superblock, different security settings for (dev mqueue, type mqueue)
Nov 08 15:34:33 micah-centos-vm1108a kernel: IPv6: ADDRCONF(NETDEV_CHANGE): vethbb25335: link becomes ready
Nov 08 15:34:33 micah-centos-vm1108a kernel: docker0: port 1(vethbb25335) entered blocking state
Nov 08 15:34:33 micah-centos-vm1108a kernel: docker0: port 1(vethbb25335) entered forwarding state
Nov 08 15:34:33 micah-centos-vm1108a oci-register-machine[12219]: 2017/11/08 15:34:33 Register machine: prestart b7a17ac2759319055628225b1d850c1f248ac182edf0ef099d816630b2a05119 12208 /var/lib/docker/devicemappe
r/mnt/d910cb56f603bf9819b87742c683ad088c33873d6f73f0e42b83c5d0b9d891f3/rootfs
Nov 08 15:34:33 micah-centos-vm1108a dbus[548]: [system] Activating via systemd: service name='org.freedesktop.machine1' unit='dbus-org.freedesktop.machine1.service'
Nov 08 15:34:33 micah-centos-vm1108a dbus-daemon[548]: dbus[548]: [system] Activating via systemd: service name='org.freedesktop.machine1' unit='dbus-org.freedesktop.machine1.service'
Nov 08 15:34:33 micah-centos-vm1108a systemd[1]: Starting Virtual Machine and Container Registration Service...
Nov 08 15:34:33 micah-centos-vm1108a dbus[548]: [system] Successfully activated service 'org.freedesktop.machine1'
Nov 08 15:34:33 micah-centos-vm1108a dbus-daemon[548]: dbus[548]: [system] Successfully activated service 'org.freedesktop.machine1'
Nov 08 15:34:33 micah-centos-vm1108a systemd[1]: Started Virtual Machine and Container Registration Service.
Nov 08 15:34:33 micah-centos-vm1108a systemd-machined[12222]: New machine b7a17ac2759319055628225b1d850c1f.
Nov 08 15:34:33 micah-centos-vm1108a oci-systemd-hook[12223]: systemdhook <debug>: Skipping as container command is /bin/sh, not init or systemd
Nov 08 15:34:33 micah-centos-vm1108a oci-umount[12224]: umounthook <info>: prestart /var/lib/docker/devicemapper/mnt/d910cb56f603bf9819b87742c683ad088c33873d6f73f0e42b83c5d0b9d891f3/rootfs
Nov 08 15:34:33 micah-centos-vm1108a oci-umount[12224]: umounthook <info>: Failed to canonicalize path [/var/lib/docker/overlay2]: No such file or directory. Skipping.
Nov 08 15:34:33 micah-centos-vm1108a oci-umount[12224]: umounthook <info>: Failed to canonicalize path [/var/lib/docker/overlay]: No such file or directory. Skipping.
Nov 08 15:34:33 micah-centos-vm1108a oci-umount[12224]: umounthook <info>: Failed to canonicalize path [/var/lib/docker-latest/overlay2]: No such file or directory. Skipping.
Nov 08 15:34:33 micah-centos-vm1108a oci-umount[12224]: umounthook <info>: Failed to canonicalize path [/var/lib/docker-latest/overlay]: No such file or directory. Skipping.
Nov 08 15:34:33 micah-centos-vm1108a oci-umount[12224]: umounthook <info>: Failed to canonicalize path [/var/lib/docker-latest/devicemapper]: No such file or directory. Skipping.
Nov 08 15:34:33 micah-centos-vm1108a oci-umount[12224]: umounthook <info>: Failed to canonicalize path [/var/lib/docker-latest/containers/]: No such file or directory. Skipping.
Nov 08 15:34:33 micah-centos-vm1108a oci-umount[12224]: umounthook <info>: Failed to canonicalize path [/var/lib/container/storage/lvm]: No such file or directory. Skipping.
Nov 08 15:34:33 micah-centos-vm1108a oci-umount[12224]: umounthook <info>: Failed to canonicalize path [/var/lib/container/storage/devicemapper]: No such file or directory. Skipping.
Nov 08 15:34:33 micah-centos-vm1108a oci-umount[12224]: umounthook <info>: Failed to canonicalize path [/var/lib/container/storage/overlay]: No such file or directory. Skipping.
Nov 08 15:34:33 micah-centos-vm1108a oci-umount[12224]: umounthook <info>: Could not find mapping for mount [/var/lib/docker/devicemapper] from host to conatiner. Skipping.
Nov 08 15:34:33 micah-centos-vm1108a oci-umount[12224]: umounthook <info>: Could not find mapping for mount [/var/lib/docker/containers] from host to conatiner. Skipping.
Nov 08 15:34:33 micah-centos-vm1108a dockerd-current[11672]: + retries=5
Nov 08 15:34:33 micah-centos-vm1108a dockerd-current[11672]: + '[' 5 -gt 0 ']'
Nov 08 15:34:33 micah-centos-vm1108a dockerd-current[11672]: + yum makecache
Nov 08 15:34:33 micah-centos-vm1108a dockerd-current[11672]: Loaded plugins: fastestmirror, ovl
Nov 08 15:34:38 micah-centos-vm1108a dockerd-current[11672]: Determining fastest mirrors
Nov 08 15:34:39 micah-centos-vm1108a dockerd-current[11672]: * base: repo1.ash.innoscale.net
Nov 08 15:34:39 micah-centos-vm1108a dockerd-current[11672]: * extras: mirror.wdc1.us.leaseweb.net
Nov 08 15:34:39 micah-centos-vm1108a dockerd-current[11672]: * updates: mirror.vtti.vt.edu
Nov 08 15:34:42 micah-centos-vm1108a dockerd-current[11672]: Metadata Cache Created
Nov 08 15:34:42 micah-centos-vm1108a dockerd-current[11672]: + break
Nov 08 15:34:42 micah-centos-vm1108a dockerd-current[11672]: Loaded plugins: fastestmirror, ovl
Nov 08 15:34:42 micah-centos-vm1108a dockerd-current[11672]: Loading mirror speeds from cached hostfile
Nov 08 15:34:42 micah-centos-vm1108a dockerd-current[11672]: * base: repo1.ash.innoscale.net
Nov 08 15:34:42 micah-centos-vm1108a dockerd-current[11672]: * extras: mirror.wdc1.us.leaseweb.net
Nov 08 15:34:42 micah-centos-vm1108a dockerd-current[11672]: * updates: mirror.vtti.vt.edu
Nov 08 15:34:43 micah-centos-vm1108a dockerd-current[11672]: Resolving Dependencies
Nov 08 15:34:43 micah-centos-vm1108a dockerd-current[11672]: --> Running transaction check
Nov 08 15:34:43 micah-centos-vm1108a dockerd-current[11672]: ---> Package httpd.x86_64 0:2.4.6-67.el7.centos.6 will be installed
Nov 08 15:34:43 micah-centos-vm1108a dockerd-current[11672]: --> Processing Dependency: httpd-tools = 2.4.6-67.el7.centos.6 for package: httpd-2.4.6-67.el7.centos.6.x86_64
Nov 08 15:34:43 micah-centos-vm1108a dockerd-current[11672]: --> Processing Dependency: system-logos >= 7.92.1-1 for package: httpd-2.4.6-67.el7.centos.6.x86_64
Nov 08 15:34:43 micah-centos-vm1108a dockerd-current[11672]: --> Processing Dependency: /etc/mime.types for package: httpd-2.4.6-67.el7.centos.6.x86_64
Nov 08 15:34:43 micah-centos-vm1108a dockerd-current[11672]: --> Processing Dependency: libaprutil-1.so.0()(64bit) for package: httpd-2.4.6-67.el7.centos.6.x86_64
Nov 08 15:34:43 micah-centos-vm1108a dockerd-current[11672]: --> Processing Dependency: libapr-1.so.0()(64bit) for package: httpd-2.4.6-67.el7.centos.6.x86_64
Nov 08 15:34:43 micah-centos-vm1108a dockerd-current[11672]: --> Running transaction check
Nov 08 15:34:43 micah-centos-vm1108a dockerd-current[11672]: ---> Package apr.x86_64 0:1.4.8-3.el7 will be installed
Nov 08 15:34:43 micah-centos-vm1108a dockerd-current[11672]: ---> Package apr-util.x86_64 0:1.5.2-6.el7 will be installed
Nov 08 15:34:43 micah-centos-vm1108a dockerd-current[11672]: ---> Package centos-logos.noarch 0:70.0.6-3.el7.centos will be installed
Nov 08 15:34:43 micah-centos-vm1108a dockerd-current[11672]: ---> Package httpd-tools.x86_64 0:2.4.6-67.el7.centos.6 will be installed
Nov 08 15:34:43 micah-centos-vm1108a dockerd-current[11672]: ---> Package mailcap.noarch 0:2.1.41-2.el7 will be installed
Nov 08 15:34:43 micah-centos-vm1108a dockerd-current[11672]: --> Finished Dependency Resolution
Nov 08 15:34:43 micah-centos-vm1108a dockerd-current[11672]:
Nov 08 15:34:43 micah-centos-vm1108a dockerd-current[11672]: Dependencies Resolved
Nov 08 15:34:43 micah-centos-vm1108a dockerd-current[11672]:
Nov 08 15:34:43 micah-centos-vm1108a dockerd-current[11672]: ================================================================================
Nov 08 15:34:43 micah-centos-vm1108a dockerd-current[11672]: Package Arch Version Repository Size
Nov 08 15:34:43 micah-centos-vm1108a dockerd-current[11672]: ================================================================================
Nov 08 15:34:43 micah-centos-vm1108a dockerd-current[11672]: Installing:
Nov 08 15:34:43 micah-centos-vm1108a dockerd-current[11672]: httpd x86_64 2.4.6-67.el7.centos.6 updates 2.7 M
Nov 08 15:34:43 micah-centos-vm1108a dockerd-current[11672]: Installing for dependencies:
Nov 08 15:34:43 micah-centos-vm1108a dockerd-current[11672]: apr x86_64 1.4.8-3.el7 base 103 k
Nov 08 15:34:43 micah-centos-vm1108a dockerd-current[11672]: apr-util x86_64 1.5.2-6.el7 base 92 k
Nov 08 15:34:43 micah-centos-vm1108a dockerd-current[11672]: centos-logos noarch 70.0.6-3.el7.centos base 21 M
Nov 08 15:34:43 micah-centos-vm1108a dockerd-current[11672]: httpd-tools x86_64 2.4.6-67.el7.centos.6 updates 88 k
Nov 08 15:34:43 micah-centos-vm1108a dockerd-current[11672]: mailcap noarch 2.1.41-2.el7 base 31 k
Nov 08 15:34:43 micah-centos-vm1108a dockerd-current[11672]:
Nov 08 15:34:43 micah-centos-vm1108a dockerd-current[11672]: Transaction Summary
Nov 08 15:34:43 micah-centos-vm1108a dockerd-current[11672]: ================================================================================
Nov 08 15:34:43 micah-centos-vm1108a dockerd-current[11672]: Install 1 Package (+5 Dependent packages)
Nov 08 15:34:43 micah-centos-vm1108a dockerd-current[11672]:
Nov 08 15:34:43 micah-centos-vm1108a dockerd-current[11672]: Total download size: 24 M
Nov 08 15:34:43 micah-centos-vm1108a dockerd-current[11672]: Installed size: 32 M
Nov 08 15:34:43 micah-centos-vm1108a dockerd-current[11672]: Downloading packages:
Nov 08 15:34:43 micah-centos-vm1108a dockerd-current[11672]: Public key for httpd-tools-2.4.6-67.el7.centos.6.x86_64.rpm is not installed
Nov 08 15:34:43 micah-centos-vm1108a dockerd-current[11672]: warning: /var/cache/yum/x86_64/7/updates/packages/httpd-tools-2.4.6-67.el7.centos.6.x86_64.rpm: Header V3 RSA/SHA256 Signature, key ID f4a80eb5: NOKEY
Nov 08 15:34:43 micah-centos-vm1108a dockerd-current[11672]: Public key for apr-util-1.5.2-6.el7.x86_64.rpm is not installed
Nov 08 15:34:47 micah-centos-vm1108a dockerd-current[11672]: --------------------------------------------------------------------------------
Nov 08 15:34:47 micah-centos-vm1108a dockerd-current[11672]: Total 5.7 MB/s | 24 MB 00:04
Nov 08 15:34:47 micah-centos-vm1108a dockerd-current[11672]: Retrieving key from file:///etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-7
Nov 08 15:34:47 micah-centos-vm1108a dockerd-current[11672]: Importing GPG key 0xF4A80EB5:
Nov 08 15:34:47 micah-centos-vm1108a dockerd-current[11672]: Userid : "CentOS-7 Key (CentOS 7 Official Signing Key) <security@centos.org>"
Nov 08 15:34:47 micah-centos-vm1108a dockerd-current[11672]: Fingerprint: 6341 ab27 53d7 8a78 a7c2 7bb1 24c6 a8a7 f4a8 0eb5
Nov 08 15:34:47 micah-centos-vm1108a dockerd-current[11672]: Package : centos-release-7-4.1708.el7.centos.x86_64 (@CentOS)
Nov 08 15:34:47 micah-centos-vm1108a dockerd-current[11672]: From : /etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-7
Nov 08 15:34:48 micah-centos-vm1108a dockerd-current[11672]: Running transaction check
Nov 08 15:34:48 micah-centos-vm1108a dockerd-current[11672]: Running transaction test
Nov 08 15:34:48 micah-centos-vm1108a dockerd-current[11672]: Transaction test succeeded
Nov 08 15:34:48 micah-centos-vm1108a dockerd-current[11672]: Running transaction
Nov 08 15:34:50 micah-centos-vm1108a dockerd-current[11672]: Installing : apr-1.4.8-3.el7.x86_64 1/6
Nov 08 15:34:52 micah-centos-vm1108a dockerd-current[11672]: Installing : apr-util-1.5.2-6.el7.x86_64 2/6
Nov 08 15:34:53 micah-centos-vm1108a dockerd-current[11672]: Installing : httpd-tools-2.4.6-67.el7.centos.6.x86_64 3/6
Nov 08 15:34:54 micah-centos-vm1108a dockerd-current[11672]: Installing : centos-logos-70.0.6-3.el7.centos.noarch 4/6
Nov 08 15:34:56 micah-centos-vm1108a dockerd-current[11672]: Installing : mailcap-2.1.41-2.el7.noarch 5/6
Nov 08 15:34:58 micah-centos-vm1108a dockerd-current[11672]: Installing : httpd-2.4.6-67.el7.centos.6.x86_64 6/6
Nov 08 15:34:58 micah-centos-vm1108a dockerd-current[11672]: Verifying : apr-1.4.8-3.el7.x86_64 1/6
Nov 08 15:34:59 micah-centos-vm1108a dockerd-current[11672]: Verifying : mailcap-2.1.41-2.el7.noarch 2/6
Nov 08 15:34:59 micah-centos-vm1108a dockerd-current[11672]: Verifying : httpd-2.4.6-67.el7.centos.6.x86_64 3/6
Nov 08 15:35:00 micah-centos-vm1108a dockerd-current[11672]: Verifying : apr-util-1.5.2-6.el7.x86_64 4/6
Nov 08 15:35:00 micah-centos-vm1108a dockerd-current[11672]: Verifying : httpd-tools-2.4.6-67.el7.centos.6.x86_64 5/6
Nov 08 15:35:02 micah-centos-vm1108a dockerd-current[11672]: Verifying : centos-logos-70.0.6-3.el7.centos.noarch 6/6
Nov 08 15:35:02 micah-centos-vm1108a dockerd-current[11672]:
Nov 08 15:35:02 micah-centos-vm1108a dockerd-current[11672]: Installed:
Nov 08 15:35:02 micah-centos-vm1108a dockerd-current[11672]: httpd.x86_64 0:2.4.6-67.el7.centos.6
Nov 08 15:35:02 micah-centos-vm1108a dockerd-current[11672]:
Nov 08 15:35:02 micah-centos-vm1108a dockerd-current[11672]: Dependency Installed:
Nov 08 15:35:02 micah-centos-vm1108a dockerd-current[11672]: apr.x86_64 0:1.4.8-3.el7
Nov 08 15:35:02 micah-centos-vm1108a dockerd-current[11672]: apr-util.x86_64 0:1.5.2-6.el7
Nov 08 15:35:02 micah-centos-vm1108a dockerd-current[11672]: centos-logos.noarch 0:70.0.6-3.el7.centos
Nov 08 15:35:02 micah-centos-vm1108a dockerd-current[11672]: httpd-tools.x86_64 0:2.4.6-67.el7.centos.6
Nov 08 15:35:02 micah-centos-vm1108a dockerd-current[11672]: mailcap.noarch 0:2.1.41-2.el7
Nov 08 15:35:02 micah-centos-vm1108a dockerd-current[11672]:
Nov 08 15:35:02 micah-centos-vm1108a dockerd-current[11672]: Complete!
Nov 08 15:35:02 micah-centos-vm1108a dockerd-current[11672]: Loaded plugins: fastestmirror, ovl
Nov 08 15:35:02 micah-centos-vm1108a dockerd-current[11672]: Cleaning repos: base extras updates
Nov 08 15:35:02 micah-centos-vm1108a dockerd-current[11672]: Cleaning up everything
Nov 08 15:35:02 micah-centos-vm1108a dockerd-current[11672]: Maybe you want: rm -rf /var/cache/yum, to also free up space taken by orphaned data from disabled or removed repos
Nov 08 15:35:02 micah-centos-vm1108a dockerd-current[11672]: Cleaning up list of fastest mirrors
Nov 08 15:35:02 micah-centos-vm1108a systemd[1]: Stopped libcontainer container b7a17ac2759319055628225b1d850c1f248ac182edf0ef099d816630b2a05119.
Nov 08 15:35:02 micah-centos-vm1108a oci-register-machine[12326]: 2017/11/08 15:35:02 Register machine: poststop b7a17ac2759319055628225b1d850c1f248ac182edf0ef099d816630b2a05119 0 /var/lib/docker/devicemapper/mn
t/d910cb56f603bf9819b87742c683ad088c33873d6f73f0e42b83c5d0b9d891f3/rootfs
Nov 08 15:35:02 micah-centos-vm1108a systemd-machined[12222]: Machine b7a17ac2759319055628225b1d850c1f terminated.
Nov 08 15:35:02 micah-centos-vm1108a oci-systemd-hook[12332]: systemdhook <debug>: Skipping as container command is /bin/sh, not init or systemd
Nov 08 15:35:02 micah-centos-vm1108a oci-umount[12333]: umounthook <debug>: poststop ignored
Nov 08 15:35:02 micah-centos-vm1108a docker-containerd-current[10558]: time="2017-11-08T15:35:02.663175481Z" level=error msg="containerd: deleting container" error="exit status 1: \"container b7a17ac275931905562
8225b1d850c1f248ac182edf0ef099d816630b2a05119 does not exist\\none or more of the container deletions failed\\n\""
Nov 08 15:35:02 micah-centos-vm1108a kernel: docker0: port 1(vethbb25335) entered disabled state
Nov 08 15:35:02 micah-centos-vm1108a kernel: docker0: port 1(vethbb25335) entered disabled state
Nov 08 15:35:02 micah-centos-vm1108a kernel: device vethbb25335 left promiscuous mode
Nov 08 15:35:02 micah-centos-vm1108a kernel: docker0: port 1(vethbb25335) entered disabled state
Nov 08 15:35:02 micah-centos-vm1108a kernel: SELinux: mount invalid. Same superblock, different security settings for (dev dm-3, type xfs)
Nov 08 15:35:33 micah-centos-vm1108a kernel: XFS (dm-3): Unmounting Filesystem
runcom
commented
6 years ago
This was originally seen on CAHC but it can also be reproduced using vanilla CentOS and the
dockerpackage from thevirt7-candidate-common-candidaterepoWhen trying to build a simple
httpdcontainer, an error from devmapper is observed.Note this is only observable when using the
devicemapperstorage driver. When I switch to theoverlay2driver, this is NOT observed.See conditions below: