buildah bud fails due to missing file system in the container

[gireeshpunathil]

Failing case: missing mount binding to the parent container

#docker run -it --privileged quay.io/buildah/stable
[root@e8c64993a35b /]# echo 'FROM alpine' > Dockerfile
[root@e8c64993a35b /]# echo 'RUN whoami' >> Dockerfile
[root@e8c64993a35b /]# buildah bud -t foo .
STEP 1: FROM alpine
Getting image source signatures
Copying blob 9d48c3bd43c5 done
Copying config 9617696764 done
Writing manifest to image destination
Storing signatures
STEP 2: RUN whoami
process exited with error: fork/exec /bin/sh: no such file or directorysubprocess exited with status 1
                                                                                                      error building at STEP "RUN whoami": exit status 1
[root@e8c64993a35b /]# 

Passing case: mount binding enabled to the parent container

#docker run -it --privileged -v containers:/var/lib/containers quay.io/buildah/stable
[root@c4975399f13e /]# echo 'FROM alpine' > Dockerfile
[root@c4975399f13e /]# echo 'RUN whoami' >> Dockerfile
[root@c4975399f13e /]# buildah bud -t foo .
STEP 1: FROM alpine
STEP 2: RUN whoami
root
STEP 3: COMMIT foo
Getting image source signatures
Copying blob 03901b4a2ea8 skipped: already exists
Copying blob 652252567dcf done
Copying config bbf36d740b done
Writing manifest to image destination
Storing signatures
bbf36d740bc9b1defd3efb7cce0900b70f7a532863b7bc49d89c8a0c44a4d1f6
[root@c4975399f13e /]# 

So far so good. Now I converted this cli invocation into a tekton pipeline as follows (only the task steps are shown that is most relevant here)

  steps:
    - name: assemble-extract-step
      image: gireeshpunathil/foo
      command: ["/bin/bash"]
      args:
        - -c
        - "somescript.sh"
      env:
      securityContext:
        privileged: true
      volumeMounts:
        - name: varlibcontainers
          mountPath: /var/lib/containers
  volumes:
    - name: varlibcontainers

The image gireeshpunathil/foo is essentially a thin layer on top of quay.io/buildah/stable.

This works as expected with minikube. (just like second case above)

This fails in minishift (just like first case above)

With my limited experience with minishift, I believe this has something to do with either:

• the mount bind is not properly assigned processed with the container.
• sufficient privilege has not yet been provided

As I am not very sure about the root cause, and as the issue is manifesting as a buildah command failure, I am raising it here. Please advise, thanks in advance!

Additional info - queried from the faulty container:

# mount | grep container
/dev/mapper/rhel-root on /var/lib/containers type xfs (rw,relatime,attr2,inode64,noquota)

# env | grep BUILDAH
_BUILDAH_STARTED_IN_USERNS=
BUILDAH_ISOLATION=chroot

excerpt from strace:

newfstatat(AT_FDCWD, "/var/lib/containers/storage/overlay/bd2ae87e47f432e463e519a14a27376510799b51d5a1be793297f22a760a2bf2/merged/bin/bash", 0xc000785d78, AT_SYMLINK_NOFOLLOW) = -1 ENOENT (No such file or directory)

# ls -lrt /var/lib/containers/storage/overlay/bd2ae87e47f432e463e519a14a27376510799b51d5a1be793297f22a760a2bf2/merged/
total 0
drwx------ 2 root root  6 Aug 27 04:03 sys
drwx------ 2 root root  6 Aug 27 04:03 proc
drwx------ 2 root root  6 Aug 27 04:03 dev
d--x--x--x 3 root root 21 Aug 27 04:03 run
d--x--x--x 2 root root 38 Aug 27 04:03 etc
#

So! /usr and and many other system folders are missing! how this could be!

All other required info for the bug tracker (system information etc.) are precisely the same as in https://github.com/containers/buildah/issues/1814, so not duplicating here.

[rhatdan]

If you look on the hosts, do you see the /usr directory there? I don't see how this would be Buidlah's fault. All of the directories in the contianer, are setup automatically other then /etc? Although I would bet the only content in /etc/ is the /etc/resolv.conf. This really looks like their is no image at all.

[gireeshpunathil]

@rhatdan - thanks for the comment. Yes, I can see the host has everything alright. Here is the full dump:

# ls -lrt /
total 56
drwxr-xr-x   2 root root 4096 Feb 11  2019 srv
lrwxrwxrwx   1 root root    8 Feb 11  2019 sbin -> usr/sbin
drwxr-xr-x   2 root root 4096 Feb 11  2019 opt
drwxr-xr-x   2 root root 4096 Feb 11  2019 mnt
drwxr-xr-x   2 root root 4096 Feb 11  2019 media
lrwxrwxrwx   1 root root    9 Feb 11  2019 lib64 -> usr/lib64
lrwxrwxrwx   1 root root    7 Feb 11  2019 lib -> usr/lib
drwxr-xr-x   2 root root 4096 Feb 11  2019 home
dr-xr-xr-x   2 root root 4096 Feb 11  2019 boot
lrwxrwxrwx   1 root root    7 Feb 11  2019 bin -> usr/bin
drwx------   2 root root 4096 Aug  1 07:47 lost+found
drwxr-xr-x   1 root root 4096 Aug  1 07:48 usr
dr-xr-x---   2 root root 4096 Aug  1 07:48 root
drwxr-xr-x   1 root root 4096 Aug  9 13:47 var
drwxr-xr-x   1 root root 4096 Aug 27 11:59 etc
dr-xr-xr-x 182 root root    0 Aug 27 11:59 proc
drwxr-xr-x  10 root root 2920 Aug 27 11:59 dev
dr-xr-xr-x  13 root root    0 Aug 27 12:00 sys
drwxr-xr-x   1 root root 4096 Aug 27 12:00 run
drwxrwxrwt   1 root root 4096 Aug 27 12:00 tmp

# pwd
/var/lib/containers/storage/overlay/bd2ae87e47f432e463e519a14a27376510799b51d5a1be793297f22a760a2bf2/merged/

# ls -lRt
.:
total 20
drwx------ 2 root root 4096 Aug 27 12:00 dev
d--x--x--x 2 root root 4096 Aug 27 12:00 etc
drwx------ 2 root root 4096 Aug 27 12:00 proc
d--x--x--x 3 root root 4096 Aug 27 12:00 run
drwx------ 2 root root 4096 Aug 27 12:00 sys

./dev:
total 0

./etc:
total 0
---------- 1 root root 0 Aug 27 12:00 hosts
---------- 1 root root 0 Aug 27 12:00 resolv.conf

./proc:
total 0

./run:
total 4
d--x--x--x 2 root root 4096 Aug 27 12:00 secrets

./run/secrets:
total 0

./sys:
total 0
# 

[rhatdan]

What does the mount command show as the lower directory for the contianer? Is there any content in it. It looks to me like the lowerdir is a scratch directory.

[gireeshpunathil]

@rhatdan - lower is empty. In fact except merged, all others - lower, work, diff and link are all empty!

Again, without knowing too much internals of buildah, I can say that this behavior is consistent with the scenario when the parent container (quay.io/biuldah/stable) was invoked without the volume mount of /var/lib/containers. If you can assert that this is expected behavior under such circumstances then it is fine by me, and will have to deal it with minishift as to why the prescribed volume mount is not being honored. At this point, I am unsure of it. Please advise!

[rhatdan]

No what is supposed to happen, is the alpine image is supposed to populate the /var/lib/containers that you volume mounted in and then these other directories are mounton .

[gireeshpunathil]

@rhatdan - thanks for your help as always! I have further done some work on the failing system, and have this observation. Though it is not a direct continuation of the previous diagnosis, I hope this brings some clues!

Did these in two identical boxes:

buildah --debug pull alpine 2> 1
buildah --debug from --name foo alpine 2> 2
buildah --debug mount foo 2> 3

passing system:

/var/lib/containers/storage/overlay/b788d08d19e2088bf63f52e2960ac6f41070210d28bea92624b90124a72adb14/merged
[root@o1 foo]# ls /var/lib/containers/storage/overlay/b788d08d19e2088bf63f52e2960ac6f41070210d28bea92624b90124a72adb14/merged
bin  dev  etc  home  lib  media  mnt  opt  proc  root  run  sbin  srv  sys  tmp  usr  var

failing system:

/var/lib/containers/storage/overlay/e74a02a55ea5c8708a6d051e5ac79d32db8c3875a3a93a635144dff40350e86d/merged
[root@g1 foo] ls /var/lib/containers/storage/overlay/e74a02a55ea5c8708a6d051e5ac79d32db8c3875a3a93a635144dff40350e86d/merged
[root@g1 foo]

nothing!

the debug log of the mount command in the failing case shows an additional section as follows:

time="2019-08-28T14:43:22Z" level=debug msg="overlay: mount_data=nodev,metacopy=on,lowerdir=/var/lib/containers/storage/overlay/l/BDHSVFIXVDKG4RPQGFRBQKFXLY,upperdir=/var/lib/containers/storage/overlay/e74a02a55ea5c8708a6d051e5ac79d32db8c3875a3a93a635144dff40350e86d/diff,workdir=/var/lib/containers/storage/overlay/e74a02a55ea5c8708a6d051e5ac79d32db8c3875a3a93a635144dff40350e86d/work"

and its lowerdir contain everything what we wanted!

[root@g1 foo]# ls /var/lib/containers/storage/overlay/l/BDHSVFIXVDKG4RPQGFRBQKFXLY
bin  dev  etc  home  lib  media  mnt  opt  proc  root  run  sbin  srv  sys  tmp  usr  var
[root@g1 foo]# 

Does this sound anything interesting, and does it provide additional clues? thanks!

[gireeshpunathil]

btw, @rhatdan - just wanted to update you on my latest finding: with quay.io/buildah/stable:v1.9.0 I see both the above issues are resolved, issue is with the latest (I did not bisect to see precisely the breaking version though).

thanks for your help!

[rhatdan]

Ok so this is with the quay.io/buildah/latest? @TomSweeneyRedHat PTAL

[TomSweeneyRedHat]

@gireeshpunathil Can you try:

# mkdir /var/lib/mycontainer
# podman run -it --privileged --device /dev/fuse:rw -v /var/lib/mycontainer:/var/lib/containers:Z quay.io/buildah/stable[root@417094881c8e /]# echo 'FROM alpine' > Dockerfile
[root@417094881c8e /]# echo 'RUN whoami' >> Dockerfile
[root@417094881c8e /]# buildah bud -t foo .
STEP 1: FROM alpine
STEP 2: RUN whoami
root
STEP 3: COMMIT foo
Getting image source signatures
Copying blob 03901b4a2ea8 skipped: already exists
Copying blob 57e39d0247a1 done
Copying config 8894a30566 done
Writing manifest to image destination
Storing signatures
8894a305663faf6ecb66fa42c8595b04ad0498c9d3329df2fb26cd10ab203d88

I think the only change here over yours is I went with Podman and added the `--device /dev/fuse:rw' With that change in play, I'm seeing the /usr dir in my container. Holler if I missed a step in your testing.

[rhatdan]

@TomSweeneyRedHat Nice catch, I forget about /dev/fuse, I wonder if we should make this a default?

[gireeshpunathil]

@TomSweeneyRedHat - thanks. But it does not work!

[root@minishift appsody]# docker run -it --privileged --device /dev/fuse:rw -v /var/lib/mycontainer:/var/lib/containers:Z quay.io/buildah/stable
[root@3b676b13d647 /]# echo 'FROM alpine' > Dockerfile
[root@3b676b13d647 /]# echo 'RUN whoami' >> Dockerfile
[root@3b676b13d647 /]# buildah bud -t foo .
STEP 1: FROM alpine
Getting image source signatures
Copying blob 9d48c3bd43c5 done
Copying config 9617696764 done
Writing manifest to image destination
Storing signatures
STEP 2: RUN whoami
process exited with error: fork/exec /bin/sh: no such file or directorysubprocess exited with status 1
                                                                                                      error building at STEP "RUN whoami": exit status 1
[root@3b676b13d647 /]# 

[rhatdan]

Oh, yes I keep forgetting the seccomp rules in docker are stricter then in podman, basically Docker blocks all mount commands, even though bind mounts are safe. Although --privileged should have removed that.

This works for me

#  docker run --security-opt seccomp:unconfined -it --device /dev/fuse:rw -v /var/lib/mycontainer:/var/lib/containers:Z quay.io/buildah/stable
[root@3bbaa213c7a6 /]# echo 'FROM alpine' > Dockerfile
[root@3bbaa213c7a6 /]#  echo 'RUN whoami' >> Dockerfile
[root@3bbaa213c7a6 /]#  buildah bud -t foo .
STEP 1: FROM alpine
STEP 2: RUN whoami
root
STEP 3: COMMIT foo
Getting image source signatures
Copying blob 256a7af3acb1 skipped: already exists
Copying blob 5f70bf18a086 skipped: already exists
Copying config 3684c127b9 done
Writing manifest to image destination
Storing signatures
3684c127b9f3789656cd7c2180af9e6d1f7f19fe8c9c5c045fd1a42d40616ae4

Or this:

#  docker run --security-opt seccomp:/usr/share/containers/seccomp.json -it --device /dev/fuse:rw -v /var/lib/mycontainer:/var/lib/containers:Z quay.io/buildah/stable
[root@62e2c444fa1e /]# echo 'FROM alpine' > Dockerfile
[root@62e2c444fa1e /]# echo 'RUN whoami' >> Dockerfile
[root@62e2c444fa1e /]# buildah bud -t foo .
STEP 1: FROM alpine
STEP 2: RUN whoami
root
STEP 3: COMMIT foo
Getting image source signatures
Copying blob 256a7af3acb1 skipped: already exists
Copying blob 5f70bf18a086 skipped: already exists
Copying config 2a4dff05ea done
Writing manifest to image destination
Storing signatures
2a4dff05ea3bac9ddbcb9c9b5bc989d45d8e95a25149ce86cd366a66e1b0edc4
[root@62e2c444fa1e /]# exit
exit

[rhatdan]

Or the best

sh-5.0# podman run -it --device /dev/fuse:rw -v /var/lib/mycontainer:/var/lib/containers:Z quay.io/buildah/stable
[root@582a7110adaf /]# echo 'FROM alpine' > Dockerfile
[root@582a7110adaf /]# echo 'RUN whoami' >> Dockerfile
[root@582a7110adaf /]# buildah bud -t foo .
STEP 1: FROM alpine
STEP 2: RUN whoami
root
STEP 3: COMMIT foo
Getting image source signatures
Copying blob 256a7af3acb1 skipped: already exists
Copying blob 5f70bf18a086 skipped: already exists
Copying config c14641cbf7 done
Writing manifest to image destination
Storing signatures
c14641cbf79d1bd346e0ba344257b1e030b4ab1f7efd868212e413cbe484d3af

[gireeshpunathil]

@rhatdan - unfortunately, these options also do not help, threw a different error!

[root@minishift apd]# docker run --security-opt seccomp:unconfined -it --device /dev/fuse:rw -v /var/lib/mycontainer:/var/lib/containers:Z quay.io/buildah/stable
Unable to find image 'quay.io/buildah/stable:latest' locally
Trying to pull repository quay.io/buildah/stable ... 
sha256:df351f55df51d3527ad4c22e29f03c3bd71860753adbff1a2e74c625e3aa24cc: Pulling from quay.io/buildah/stable
5a915a173fbc: Pull complete 
b726a44e32dd: Pull complete 
a6f41e56544a: Pull complete 
c0b738707d1f: Pull complete 
Digest: sha256:df351f55df51d3527ad4c22e29f03c3bd71860753adbff1a2e74c625e3aa24cc
Status: Downloaded newer image for quay.io/buildah/stable:latest
[root@ea3e00f743ff /]# echo 'FROM alpine' > Dockerfile && echo 'RUN whoami' >> Dockerfile && buildah bud -t foo .
STEP 1: FROM alpine
ERRO[0000] error unmounting /var/lib/containers/storage/overlay/7c87c3e76e2db4cfc3cbf1981b90d5727db012b8440b8b912e7f1f798699ebb1/merged: invalid argument 
error mounting new container: error mounting build container "e98b788b5538e1f461641936a9a420b3ae836aee2422006d5c60abd7e79ce5a4": error creating overlay mount to /var/lib/containers/storage/overlay/7c87c3e76e2db4cfc3cbf1981b90d5727db012b8440b8b912e7f1f798699ebb1/merged: using mount program /usr/bin/fuse-overlayfs: fuse: failed to exec fusermount3: No such file or directory
fuse-overlayfs: cannot mount: Operation not permitted
: exit status 1
ERRO[0000] exit status 1                                
[root@ea3e00f743ff /]# exit
exit

[root@minishift apd]# docker run --security-opt seccomp:/usr/share/containers/seccomp.json -it --device /dev/fuse:rw -v /var/lib/mycontainer:/var/lib/containers:Z quay.io/buildah/stable
[root@d58a862284f2 /]# echo 'FROM alpine' > Dockerfile && echo 'RUN whoami' >> Dockerfile && buildah bud -t foo .
STEP 1: FROM alpine
ERRO[0000] error unmounting /var/lib/containers/storage/overlay/9ed70333b82dc604679b1e830a335724e79aa6657a3017d21a2b15fe58a4bc14/merged: invalid argument 
error mounting new container: error mounting build container "9d70e8663f051fdedf818e2dca50bd1995e8b330c7c577445392add65e2869e4": error creating overlay mount to /var/lib/containers/storage/overlay/9ed70333b82dc604679b1e830a335724e79aa6657a3017d21a2b15fe58a4bc14/merged: using mount program /usr/bin/fuse-overlayfs: fuse: failed to exec fusermount3: No such file or directory
fuse-overlayfs: cannot mount: Operation not permitted
: exit status 1
ERRO[0000] exit status 1                                
[root@d58a862284f2 /]# 

[rhatdan]

Weird these all work fine for me. @giuseppe Any ideas? Is fusermount3 supposed to be in the container?

[gireeshpunathil]

further info: last good build: v1.9.0, first bad build: v1.9.1. I could do a git bisect, but unfortunately bit difficult to follow the build procedure. Is it documented somewhere, or can someone share it for me to follow? thanks!

[TomSweeneyRedHat]

@gireeshpnathil I'm not 100% sure of the build procedure that you're looking for? I use ./release.sh to create the initial commit then massage things by hand from there.

@rhatdan actually did both. 1.9.0 at: https://github.com/containers/buildah/pull/1676 1.9.1 at https://github.com/containers/buildah/pull/1716

if you search the closed PR's for the word "bump" you should see all of them. Is that what you're looking for?

Tagging @lsm5 in case he's thoughts.

[gireeshpunathil]

@TomSweeneyRedHat - sorry if I confused you! my idea is:

• setup a buildah build locally (able to do make install or whatever)
• test it on my system - I suspect the latest one will fail (show above result)
• then I can unwind commits by doing git bisect until I get a good build.
• this will work, as we know we have a good build in v1.9.0

So, I am looking for a procedure on how to build the buildah execubale. I tried to follow the steps in the travis CI, but looks like it has a lot of steps due to the fact that it has to spawn a container, download secrets ... etc.

[TomSweeneyRedHat]

@gireeshpunathil the confusion is probably all on my end, I've only had one cup of tea today, I'm way behind schedule!

After you've used git bisect to determine the commit id that you want, can you do a bunch of git hackery? Setup a sandbox ala: https://github.com/containers/buildah/blob/master/install.md#installation-from-github and then:

git checkout -b test_branch {commitid}
make
make install 

[gireeshpunathil]

Apologies for the delay, was actually working to circumvent this issue back in my project. But now I have some updates:

I wasn't greatly successful with git bisect: the image version v1.9.0 is just a month old, but in github commits, even if I traverse months together, I am uable to find a one that works on this machine!

then I took a different approach. I placed latest buildah binary into v1.9.0 quay image and tested. It worked! Similarly, I placed the buildahfrom v1.9.0 into the latest quay image, and it failed!

So the bottom line for me is that it is not buildah, but something else in the image that is decisive.

And I zeroed down onto fuse-overlayfs. If I manually upgrade it from a working image, it starts to fail. Here is the full dump:

[root@h1 ~]# docker run -it --rm --privileged -v containers:/var/lib/containers --network=host quay.io/buildah/stable:v1.9.0 bash
81f7db1a27fd: Pull complete 
Digest: sha256:90c9f9a81e1ec67d8d69764bcef78f123fb741325d1654c91270b2980f501198
Status: Downloaded newer image for quay.io/buildah/stable:v1.9.0
[root@h1 /]# 
[root@h1 /]# echo 'FROM alpine' > Dockerfile && echo 'RUN whoami' >> Dockerfile && buildah bud -t foo .
STEP 1: FROM alpine
...
STEP 2: RUN whoami
root
STEP 3: COMMIT foo
...
Storing signatures
6ce61864831be61d597744c0fa34a2b4ec48e9aa4baba47e2a761ee14fcd59ea
[root@h1 /]# yum list installed | fuse-overlayfs
fuse-overlayfs: upperdir not specified
[root@h1 /]# yum list installed | grep fuse-overlayfs
fuse-overlayfs.x86_64              0.3-10.dev.gita7c8295.fc30      @updates        

[root@h1 /]# yum update fuse-overlayfs.x86_64
Fedora 30 - x86_64                                                 5.9 MB/s |  70 MB     00:11    
Last metadata expiration check: 0:00:32 ago on Sun 08 Sep 2019 12:00:34 PM UTC.
Dependencies resolved.
===================================================================================================
 Package                 Architecture    Version                            Repository        Size
===================================================================================================
Upgrading:
 fuse-overlayfs          x86_64          0.6.1-2.gitc548530.fc30            updates           56 k

Transaction Summary
===================================================================================================
Upgrade  1 Package

Total download size: 56 k
Is this ok [y/N]: y
Downloading Packages:
fuse-overlayfs-0.6.1-2.gitc548530.fc30.x86_64.rpm                   30 kB/s |  56 kB     00:01    
---------------------------------------------------------------------------------------------------
Total                                                               14 kB/s |  56 kB     00:03     
Running transaction check
Transaction check succeeded.
Running transaction test
Transaction test succeeded.
Running transaction
  Preparing        :                                                                           1/1 
  Upgrading        : fuse-overlayfs-0.6.1-2.gitc548530.fc30.x86_64                             1/2 
  Running scriptlet: fuse-overlayfs-0.6.1-2.gitc548530.fc30.x86_64                             1/2 
  Cleanup          : fuse-overlayfs-0.3-10.dev.gita7c8295.fc30.x86_64                          2/2 
  Running scriptlet: fuse-overlayfs-0.3-10.dev.gita7c8295.fc30.x86_64                          2/2 
  Verifying        : fuse-overlayfs-0.6.1-2.gitc548530.fc30.x86_64                             1/2 
  Verifying        : fuse-overlayfs-0.3-10.dev.gita7c8295.fc30.x86_64                          2/2 

Upgraded:
  fuse-overlayfs-0.6.1-2.gitc548530.fc30.x86_64                                                    

Complete!
[root@h1 /]# yum list installed | grep fuse-overlayfs
fuse-overlayfs.x86_64              0.6.1-2.gitc548530.fc30         @updates        
[root@h1 /]# echo 'FROM alpine' > Dockerfile && echo 'RUN whoami' >> Dockerfile && buildah bud -t foo .
STEP 1: FROM alpine
STEP 2: RUN whoami
process exited with error: fork/exec /bin/sh: no such file or directorysubprocess exited with status 1
   error building at STEP "RUN whoami": exit status 1
[root@h1 /]#

Is there anything I can do? are there any debug options for this tool? thanks again!

[gireeshpunathil]

ok, more findings: this time I am able to do git bisect successfully, on the fuse-overlayfs component. I hope we can continue conversation here, as these are very closely related projects? Let me know if that is not the case.

This commit https://github.com/containers/fuse-overlayfs/commit/35949df0bbbf507937f0ddafdb3557234531211e is the point of failure. Without knowing anything about the fuse logic I cannot make any comments; but a wild guess is that fuse_buf_copy did not seem to have happen under some unforeseen circumstances, due to which we see the merged folder empty.

I have the full build environment and the failing system handy with me; so if you have any proposals on debug statements, patches, printfs etc. I am happy to follow it!

Copying @giuseppe to in case if he knows.

[gireeshpunathil]

I did further debugging, and found that

https://github.com/containers/fuse-overlayfs/blob/35949df0bbbf507937f0ddafdb3557234531211e/main.c#L500-L502

makes the difference. If I comment that, the issue is resolved!

the documentation for this flag says:

Indicates that writeback caching should be enabled. This means that individual write request may be buffered and merged in the kernel before they are send to the filesystem.
This feature is disabled by default.

[gireeshpunathil]

how do we know my kernel has the support for the writeback cache, and if yes, it works well?

[gireeshpunathil]

If I were to suspect the libfuse module on the host, this is what I have:

good system:

[root@g1 ~]# yum list installed | grep fuse
fuse.x86_64                     2.9.2-11.el7               @Local-Base          
fuse-libs.x86_64                2.9.2-11.el7               @Local-Base          
glusterfs-fuse.x86_64           3.12.2-47.2.el7            installed 

bad system:

[root@h1 mount]# yum list installed | grep fuse
fuse.x86_64                        2.9.2-11.el7                        @base    
fuse-libs.x86_64                   2.9.2-11.el7                        @base    
glusterfs-fuse.x86_64              3.12.2-18.el7                       @base    

The only difference is in the glusterfs, that too in the minor versios.

And that, did not yield any updates:

[root@h1 mount]# yum update glusterfs-fuse.x86_64
Loaded plugins: fastestmirror
Loading mirror speeds from cached hostfile
 * base: centos.excellmedia.net
 * extras: centos.excellmedia.net
 * updates: centos.excellmedia.net
base                                                               | 3.6 kB  00:00:00     
centos-openshift-origin311                                         | 2.9 kB  00:00:00     
extras                                                             | 3.4 kB  00:00:00     
updates                                                            | 3.4 kB  00:00:00     
No packages marked for update

so!

[giuseppe]

could you try disabling the writeback cache with?

[storage]
  [storage.options]
    mountopt = "writeback=0"

in your ~/.config/containers/storage.conf file if running as rootless, or /etc/containers/storage.conf when running as root in a container as in your use case?

[giuseppe]

alternatively, could you check if this fuse-overlayfs patch solves the issue:

diff --git a/main.c b/main.c
index 1abc464..f591a05 100644
--- a/main.c
+++ b/main.c
@@ -536,6 +536,9 @@ ovl_init (void *userdata, struct fuse_conn_info *conn)
 {
   struct ovl_data *lo = (struct ovl_data *) userdata;

+  if ((conn->capable & FUSE_CAP_WRITEBACK_CACHE) == 0)
+    lo->writeback = 0;
+
   conn->want |= FUSE_CAP_DONT_MASK | FUSE_CAP_SPLICE_READ | FUSE_CAP_SPLICE_WRITE | FUSE_CAP_SPLICE_MOVE;
   if (lo->writeback)
     conn->want |= FUSE_CAP_WRITEBACK_CACHE;

[gireeshpunathil]

thanks @giuseppe ! will test and get back.

[gireeshpunathil]

# storage.conf is the configuration file for all tools
# that share the containers/storage libraries
# See man 5 containers-storage.conf for more information
# The "container storage" table contains all of the server options.
[storage]

# Default Storage Driver
driver = "overlay"

# Temporary storage location
runroot = "/var/run/containers/storage"

# Primary Read/Write location of container storage 
graphroot = "/var/lib/containers/storage"

[storage.options]
  mountopt = "writeback=0"
# Storage options to be passed to underlying storage drivers 

@giuseppe - this is the head of my /etc/containers/storage.conf, the last line is my addition, based on your advice. Pasting it here to see if the syntax looks good to you or not as it did not work as expected. Also, do we need to do any refresh? like `sysctl stop | start on some services?

the patch your provided works, many thanks!!! so that means the underlying host does not need any changes, if this change is going upstream? If so, what could have happened wrong with these systems? could you be so kind to please explain a bit, for my understanding?

thanks again!

[giuseppe]

@gireeshpunathil yes the conf file looks correctly.

libfuse checks whether some features are available and informs the FUSE program (fuse-overlayfs in this case) what can be requested. fuse-overlayfs didn't check for it, and assumed writeback is always supported, which from your bug report seems to not be the case.

PR opened here: https://github.com/containers/fuse-overlayfs/pull/117

[gireeshpunathil]

thanks all for the help, really appreciated!!

I will watch out for a release being cut out of #117 , any rough estimate on the dates?

[gireeshpunathil]

can some one advise me which buildah release this fix was incorporated into? the reason why I ask is, I don't see the intended behavior with latest buildah releases. cc @rhatdan @TomSweeneyRedHat @giuseppe

[nalind]

Based on the log of differences between 0.6 and 0.7 (https://github.com/containers/fuse-overlayfs/compare/v0.6...v0.7), it should have been in 0.7.

[TomSweeneyRedHat]

@gireeshpunathil as @nalind pointed out, it's in the latest release. That being said, other than cloning and building fuse-overlayfs from GitHub, I'm not sure where you could get a rpm yet. @giuseppe or @lsm5 do you have any pointers?

[gireeshpunathil]

Based on the log of differences between 0.6 and 0.7 (v0.6...v0.7), it should have been in 0.7.

@nalind - thanks, will check on that.

I actually meant buildah release or buildah container image that has this (good) version of fuse-overlay

[giuseppe]

@gireeshpunathil Buildah doesn't ship fuse-overlayfs. It is shipped separately.

What distro are you using? fuse-overlayfs 0.7 is already on Fedora 30/31 stable

[gireeshpunathil]

@giuseppe - ah, thanks for the information. Will check on that.

So basically we should be good if I do:

yum update && yum install fuse-overlayfs prior to the sequences in my first comment I guess?

[TomSweeneyRedHat]

@gireeshpunathil I'd go with yum update && yum update fuse-overlayfs, then rpm -qa | grep overlayfs, and verify the version.

[gireeshpunathil]

@TomSweeneyRedHat - thanks. My main question (I acknowledge that did not come good) was whether we update fuse-overlayfs on the host, or inside the (buildah) container?

[TomSweeneyRedHat]

@gireeshpunathil ah! Should only be on the host side that you will need to update fuse-overlayfs on, the container should be fine.

[gireeshpunathil]

thanks @TomSweeneyRedHat for the clarification!