Some files in a glusterfs mount do not appear.

[Reisen]

Unsure of the cause yet but will try and find a reproducible example to update the issue with.

I have a glusterfs disk (mounted with mount -t glusterfs node:/gv0 /mnt) which is then mounted into a container with krun and podman: podman --runtime=/usr/local/bin/krun -v /mnt:/mnt which for some reason does not display some files, of a rough 30,000 files, about 10% of them simple do not appear in the VM. All permissions are correct and running with crun or podman machine does not have this problem.

I notice that this doesn't seem to be a file based problem though, but more that no files appear in certain folders, I can touch /mnt/foo/test.txt for example where /mnt/foo is one such affected folder and the file will appear on the host, but still not show in the VM. Any files placed, or created in this folder whether by the VM or the host are invisible to the container.

[slp]

Hm... this is weird, because what the virtiofs device in libkrun does is simply call SYS_getdents64, just as any conventional userspace application.

Reproducing this one is going to be complicated, but I'll try to set up a test environment with glusterfs soon(ish).

[Reisen]

@slp I haven't found a minimal reproduction yet but there is some additional behaviour that actually I didn't spot before and now I have a theory, here I notice the folder appears if I create enough files:

[root@5c392b7da779 /mnt/foo]# ls
[root@5c392b7da779 /mnt/foo]# touch a.txt; ls
[root@5c392b7da779 /mnt/foo]# touch b.txt; ls
'Arch Linux (64bit).iso'   a.txt   b.txt
[root@5c392b7da779 /mnt/foo]# rm a.txt; ls
'Arch Linux (64bit).iso'   b.txt
[root@5c392b7da779 /mnt/foo]# rm b.txt; ls
[root@5c392b7da779 /mnt/foo]# touch b.txt; ls
'Arch Linux (64bit).iso'   b.txt

So what I'm noticing is I have a two node gluster setup. a.txt gets placed on node A alongside the existing iso, but b.txt gets placed on node B. As soon as that happens the whole folder appears, here's the gluster brick view:

$ ssh brick1 ls -l /mnt/foo
'Arch Linux (64bit).iso'   a.txt 
$ ssh brick2 ls -l /mnt/foo
b.txt

It just so happens brick2 is the gluster mount: mount -t glusterfs brick2:/gv0 /mnt so this seems to be likely what's causing it, I.E, while the folder on B is empty the whole folder is seen as empty.

I am wondering if there is some bind propogation problem occurring here where gluster is using submounts to unify the various connected bricks, I tried using -v /mnt:/mnt:shared but it didn't help. docker's default bind propogation is rprivate and I'm still not seeing this behaviour there so I'm not 100% convinced this is the exact cause, but it definitely seems to be at least a large hint at what the cause might be.

(Note: mount --bind /mnt /mnt && mount --make-shared /mnt on the host has no effect)