building containers fails with "rm: can't remove '/usr': Invalid argument"

[f0]

Hi, running opensuse thumbleweed wit rkt 1.9.1 , try to built the dummy container (also happen with the containers from the example directory)

+ export ACI_HOME=/dgr/aci-home
+ /dgr/builder/stage2/builder.sh
+ export SYSTEMD_LOG_LEVEL=err
+ export ROOTFS=/opt/stage2/aci-dummy/rootfs
+ chmod 755 /opt/stage2
+ chmod 755 /opt/stage2/aci-dummy
+ [ -z /dgr/aci-home ]
+ ls -A /usr/bin/
+ [ !  ]
+ isLevelEnabled debug
+ levelFromString debug
+ echo debug
+ awk {print toupper($0)}
+ echo 5
+ return 0
+ l=5
+ [ 5 -le 5 ]
+ return 0
+ echo_purple Nothing as builder dependency, mapping / to /dgr
+ echo -e \033[0;35mNothing as builder dependency, mapping / to /dgr\033[0m
Nothing as builder dependency, mapping / to /dgr
+ cp /etc/resolv.conf /dgr/etc/resolv.conf
+ rm -Rf /usr /etc /lib64 /lib /bin
rm: can't remove '/usr': Invalid argument
12:25:50 FATAL  d/aci-builder/bin-run/main.go:26  Build failed                                
                       d/a/bin-run/builder.go:223 Builder run failed                           aci=aci-dummy
                                                  exit status 1
12:25:50 DEBUG     dgr/bin-dgr/common/exec.go:16  Running external command                     command=rkt   --debug --insecure-options=ondisk,image rm --uuid-file /home/fkoch/source/centos/target/builder.uuid
12:25:50 DEBUG     dgr/bin-dgr/common/exec.go:16  Running external command                     command=rkt   --debug --insecure-options=ondisk,image image rm sha512-637c6f5b42de63c9924439b542e5ffae174a23245f168d4bfdd9b3bc6b3a0232
12:25:50 DEBUG     dgr/bin-dgr/common/exec.go:56  Running external command                     command=chown   -R 1000:100 /home/fkoch/source/centos/target
12:25:50 FATAL         dgr/bin-dgr/command.go:120 Build command failed                        
                     dgr/bin-dgr/aci-build.go:72  Builder container return with failed status  aci=aci.example.com/aci-dummy:1
               d/bin-dgr/common/rkt-client.go:210 Run failed                                   config={Path: InsecureOptions:[ondisk image] dir: LocalConfig: SystemConfig: UserConfig: TrustKeysFromHttps:false NoStore:false StoreOnly:false}
                                                  exit status 1
~/source/centos % sudo /home/fkoch/bin/dgr -L debug build

[n0rad]

Thanks for the issue. @sinfomicien that is also running opensuse had the issue few weeks ago.

It seems to come from overlayfs that is not able to remove a non overlayed directory on this distrib. I still don't know what is causing this, so I'm not able to fix it. Also I don't have an opensuse to play with it and find the problem.

I will think about a solution to get around this by not removing the directory, but I still would like to find the problem since it can cause problems somewhere else.

[f0]

@n0rad maybe btrfs comes into play.... overlayfs ontop of btrfs is a bit "buggy" e.g https://bugzilla.redhat.com/show_bug.cgi?id=1255512

[f0]

on fedora 24 , works without problems. (with rkt 1.10, selinux disabled)

[n0rad]

fixed in v72 by mass linking files instead of removing the directory and linking it.