Open olalonde opened 8 years ago
i'd be interested to know if this still fails in CoreOS 991.0.0, whose kernel has some overlay changes.
@mischief I confirmed it still fails in CoreOS 1000.0.0
$ docker info
docker info
Containers: 0
Running: 0
Paused: 0
Stopped: 0
Images: 0
Server Version: 1.10.3
Storage Driver: overlay
Backing Filesystem: extfs
Execution Driver: native-0.2
Logging Driver: json-file
Plugins:
Volume: local
Network: null host bridge
Kernel Version: 4.4.6-coreos
Operating System: CoreOS 1000.0.0 (MoreOS)
OSType: linux
Architecture: x86_64
CPUs: 1
Total Memory: 1.958 GiB
Interesting. I am getting this error sometimes for Docker hub automatic builds for my Meteor apps in exactly the same step, when install Meteor inside, which uses tar to extract files. But what is interesting is that sometimes it happens and sometimes it does not. Does Docker hub build images on different underlying configurations?
I am seeing the same issue on docker hub with our meteor build image. https://hub.docker.com/r/onmodulus/build-meteor/builds/bxnluvw33haxrgvhlylfmpv/
Judging from the relevant code in tar, the ancestor directory must have undergone a copy_up in the middle of the extraction, resulting in its st_dev/st_ino changing which tar is using as a heuristic for rename detection:
839 if (check_for_renamed_directories)
840 {
841 if (fstatat (chdir_fd, data->file_name, &st, data->atflag) != 0)
842 {
843 stat_error (data->file_name);
844 skip_this_one = 1;
845 }
846 else
847 {
848 current_mode = st.st_mode;
849 current_mode_mask = ALL_MODE_BITS;
850 if (! (st.st_dev == data->dev && st.st_ino == data->ino))
851 {
852 ERROR ((0, 0,
853 _("%s: Directory renamed before its status could be extracted"),
854 quotearg_colon (data->file_name)));
855 skip_this_one = 1;
856 }
857 }
858 }
That's unfortunate, and I'm not yet clear on if any recent overlayfs developments have improved on this situation. I also don't see an obvious flag to turn off this heuristic in tar.
An immediate workaround would be to use btrfs instead of ext4+overlayfs, which CoreOS still supports. You can automate the switch to btrfs on first boot using Ignition as well.
Addendum: Upon further study of the overlayfs implementation, it doesn't seem like the st_dev/st_ino values should be changing spuriously, and I've only managed to reproduce this once out of ~50 attempts in a simplified tar extraction of the meteor release tar on a plain overlayfs mount. I'll keep digging, but it's very likely to be overlayfs-related considering the nature of the failure.
Addendum II: These inode numbers are being generated on-the-fly by overlayfs: https://github.com/coreos/linux/blob/v4.6-coreos/fs/inode.c#L848 https://github.com/coreos/linux/blob/v4.6-coreos/fs/overlayfs/inode.c#L408 https://github.com/coreos/linux/blob/v4.6-coreos/fs/overlayfs/super.c#L559
So the persistence of the inode number is entirely bound to the cache, which explains why this can be frustrating to reproduce.
Here's a simple test one can use to observe the problem tar
is complaining about:
overlay # mkdir u l w mnt
overlay # mount -t overlay overlay -o upperdir=u,lowerdir=l,workdir=w mnt
overlay # mkdir mnt/foo
overlay # stat mnt/foo | grep Inode
Device: 31h/49d Inode: 75030743 Links: 2
overlay # stat mnt/foo | grep Inode
Device: 31h/49d Inode: 75030743 Links: 2
overlay # echo 2 > /proc/sys/vm/drop_caches
overlay # stat mnt/foo | grep Inode
Device: 31h/49d Inode: 75030163 Links: 2
overlay # stat mnt/foo | grep Inode
Device: 31h/49d Inode: 75030163 Links: 2
I see no immediate solution to this without significant changes to overlayfs.
Definitely another overlayfs issue, for the upstream discussion see http://thread.gmane.org/gmane.linux.file-systems.union/879/focus=879
we re experiencing this as well here: https://hub.docker.com/r/tzapu/development-meteor/builds/biszzqx9oawewfe8hnrzm4c/
some builds make it through, some fail, even if it s the same thing...
Same happens when using docker cloud to build... All 4 builds have failed when extracting the tar.
Docker Repo: https://cloud.docker.com/app/monbelle/repository/docker/monbelle/meteor/builds Github: https://github.com/gsc-dev/docker-meteor
I'm trying to install meteor in my gitlab ci/cd so that it can bundle and deploy the project. I'm getting the same error when the tar is being extracted. For me it happens every single time.
@idmontie Same here for me, fails every time, except in my case only when i add the --delete
flag which i do in my on_stop
for my review apps setup.
I was getting the same errors on Mac and on Docker Hub.
A workaround was to use bsdtar
instead of tar
in the guest OS. I was using Ubuntu 14.04 in the Docker image. You may need to install bsdtar
using the command apt-get install -y --no-install-recommends bsdtar
and then use bsdtar
in place of tar
.
I think kernel 4.13's OVERLAY_FS_INDEX=y config should fix this. See https://lwn.net/Articles/725276/
You can install and replace tar
with bsdtar
in the Dockerfile, right before installing meteor:
RUN apt-get install -y bsdtar && ln -sf $(which bsdtar) $(which tar)
RUN curl "https://install.meteor.com/?release=1.5.2.2" | sh
@povesteam your fix works fine, but making other apt-get install to fail :(
@Jeyanthinath How about putting the original tar
back after meteor installation?
# meteor installer doesn't work with the default tar binary
RUN apt-get install -y bsdtar \
&& cp $(which tar) $(which tar)~ \
&& ln -sf $(which bsdtar) $(which tar)
# install Meteor forcing its progress
RUN curl "https://install.meteor.com/?release=1.6.0.1" \
| sed 's/VERBOSITY="--silent"/VERBOSITY="--progress-bar"/' \
| sh
# put back the original tar
RUN mv $(which tar)~ $(which tar)
Thanks @xmjiao for the idea!
apt-get clean
before tar
, it works for me .
First reported here: https://github.com/docker/docker/issues/19647
Command to reproduce:
On Docker v1.8.1, aufs storage driver, the command completes successfully:
On Docker v1.8.3, overfayfs storage driver, the command fails:
More logs:
Related issues: https://github.com/deis/deis/issues/4867 https://github.com/meteorhacks/meteord/issues/64 https://github.com/meteor/meteor/issues/5762