mischief
commented
8 years ago i'd be interested to know if this still fails in CoreOS 991.0.0, whose kernel has some overlay changes.
Open olalonde opened 8 years ago
mischief
commented
8 years ago i'd be interested to know if this still fails in CoreOS 991.0.0, whose kernel has some overlay changes.
AkihiroSuda
commented
8 years ago @mischief I confirmed it still fails in CoreOS 1000.0.0
$ docker info
docker info
Containers: 0
Running: 0
Paused: 0
Stopped: 0
Images: 0
Server Version: 1.10.3
Storage Driver: overlay
Backing Filesystem: extfs
Execution Driver: native-0.2
Logging Driver: json-file
Plugins:
Volume: local
Network: null host bridge
Kernel Version: 4.4.6-coreos
Operating System: CoreOS 1000.0.0 (MoreOS)
OSType: linux
Architecture: x86_64
CPUs: 1
Total Memory: 1.958 GiB
mitar
commented
8 years ago Interesting. I am getting this error sometimes for Docker hub automatic builds for my Meteor apps in exactly the same step, when install Meteor inside, which uses tar to extract files. But what is interesting is that sometimes it happens and sometimes it does not. Does Docker hub build images on different underlying configurations?
kenXengineering
commented
8 years ago I am seeing the same issue on docker hub with our meteor build image. https://hub.docker.com/r/onmodulus/build-meteor/builds/bxnluvw33haxrgvhlylfmpv/
vcaputo
commented
8 years ago Judging from the relevant code in tar, the ancestor directory must have undergone a copy_up in the middle of the extraction, resulting in its st_dev/st_ino changing which tar is using as a heuristic for rename detection:
839 if (check_for_renamed_directories)
840 {
841 if (fstatat (chdir_fd, data->file_name, &st, data->atflag) != 0)
842 {
843 stat_error (data->file_name);
844 skip_this_one = 1;
845 }
846 else
847 {
848 current_mode = st.st_mode;
849 current_mode_mask = ALL_MODE_BITS;
850 if (! (st.st_dev == data->dev && st.st_ino == data->ino))
851 {
852 ERROR ((0, 0,
853 _("%s: Directory renamed before its status could be extracted"),
854 quotearg_colon (data->file_name)));
855 skip_this_one = 1;
856 }
857 }
858 }That's unfortunate, and I'm not yet clear on if any recent overlayfs developments have improved on this situation. I also don't see an obvious flag to turn off this heuristic in tar.
An immediate workaround would be to use btrfs instead of ext4+overlayfs, which CoreOS still supports. You can automate the switch to btrfs on first boot using Ignition as well.
Addendum: Upon further study of the overlayfs implementation, it doesn't seem like the st_dev/st_ino values should be changing spuriously, and I've only managed to reproduce this once out of ~50 attempts in a simplified tar extraction of the meteor release tar on a plain overlayfs mount. I'll keep digging, but it's very likely to be overlayfs-related considering the nature of the failure.
Addendum II: These inode numbers are being generated on-the-fly by overlayfs: https://github.com/coreos/linux/blob/v4.6-coreos/fs/inode.c#L848 https://github.com/coreos/linux/blob/v4.6-coreos/fs/overlayfs/inode.c#L408 https://github.com/coreos/linux/blob/v4.6-coreos/fs/overlayfs/super.c#L559
So the persistence of the inode number is entirely bound to the cache, which explains why this can be frustrating to reproduce.
Here's a simple test one can use to observe the problem tar is complaining about:
overlay # mkdir u l w mnt
overlay # mount -t overlay overlay -o upperdir=u,lowerdir=l,workdir=w mnt
overlay # mkdir mnt/foo
overlay # stat mnt/foo | grep Inode
Device: 31h/49d Inode: 75030743 Links: 2
overlay # stat mnt/foo | grep Inode
Device: 31h/49d Inode: 75030743 Links: 2
overlay # echo 2 > /proc/sys/vm/drop_caches
overlay # stat mnt/foo | grep Inode
Device: 31h/49d Inode: 75030163 Links: 2
overlay # stat mnt/foo | grep Inode
Device: 31h/49d Inode: 75030163 Links: 2I see no immediate solution to this without significant changes to overlayfs.
vcaputo
commented
8 years ago Definitely another overlayfs issue, for the upstream discussion see http://thread.gmane.org/gmane.linux.file-systems.union/879/focus=879
tzapu
commented
8 years ago we re experiencing this as well here: https://hub.docker.com/r/tzapu/development-meteor/builds/biszzqx9oawewfe8hnrzm4c/
some builds make it through, some fail, even if it s the same thing...
gsc-dev
commented
7 years ago Same happens when using docker cloud to build... All 4 builds have failed when extracting the tar.
Docker Repo: https://cloud.docker.com/app/monbelle/repository/docker/monbelle/meteor/builds Github: https://github.com/gsc-dev/docker-meteor
idmontie
commented
7 years ago I'm trying to install meteor in my gitlab ci/cd so that it can bundle and deploy the project. I'm getting the same error when the tar is being extracted. For me it happens every single time.
Koleok
commented
7 years ago @idmontie Same here for me, fails every time, except in my case only when i add the --delete flag which i do in my on_stop for my review apps setup.
xmjiao
commented
7 years ago I was getting the same errors on Mac and on Docker Hub.
A workaround was to use bsdtar instead of tar in the guest OS. I was using Ubuntu 14.04 in the Docker image. You may need to install bsdtar using the command apt-get install -y --no-install-recommends bsdtar and then use bsdtar in place of tar.
euank
commented
7 years ago I think kernel 4.13's OVERLAY_FS_INDEX=y config should fix this. See https://lwn.net/Articles/725276/
povesteam
commented
6 years ago You can install and replace tar with bsdtar in the Dockerfile, right before installing meteor:
RUN apt-get install -y bsdtar && ln -sf $(which bsdtar) $(which tar)
RUN curl "https://install.meteor.com/?release=1.5.2.2" | sh
Jeyanthinath
commented
6 years ago @povesteam your fix works fine, but making other apt-get install to fail :(
povesteam
commented
6 years ago @Jeyanthinath How about putting the original tar back after meteor installation?
# meteor installer doesn't work with the default tar binary
RUN apt-get install -y bsdtar \
&& cp $(which tar) $(which tar)~ \
&& ln -sf $(which bsdtar) $(which tar)
# install Meteor forcing its progress
RUN curl "https://install.meteor.com/?release=1.6.0.1" \
| sed 's/VERBOSITY="--silent"/VERBOSITY="--progress-bar"/' \
| sh
# put back the original tar
RUN mv $(which tar)~ $(which tar)Thanks @xmjiao for the idea!
kgbook
commented
4 years ago apt-get clean before tar, it works for me .
First reported here: https://github.com/docker/docker/issues/19647
Command to reproduce:
On Docker v1.8.1, aufs storage driver, the command completes successfully:
On Docker v1.8.3, overfayfs storage driver, the command fails:
More logs:
Related issues: https://github.com/deis/deis/issues/4867 https://github.com/meteorhacks/meteord/issues/64 https://github.com/meteor/meteor/issues/5762