Users lose access to filesystem after 2.33.?

[LotusGuy]

Following an upgrade from mergerfs-2.32.5-1.el7.x86_64 -> mergerfs-2.33.3-1.el7.x86_64 on Centos 7.9 users could only see the contents of the mergerfs mount point. All directories at this point appeared empty.

For root: [root@Nethserver ~]# ls -l /MyPool/ total 636608 drwxrwx--- 5 root fileshare@BogoLinux.net 4096 Jan 25 21:17 Backup -rw------- 1 root root 651839138 Jan 27 05:02 content -rw------- 1 root root 0 Jan 27 16:14 content.lock drwxrws--- 5 root fileshare@BogoLinux.net 197 Jan 26 16:00 DVR drwxrws--- 2 root fileshare@BogoLinux.net 6 Apr 8 2021 Games drwxrwx--- 4 root fileshare@BogoLinux.net 50 Sep 21 10:57 iTunes drwxrws--- 2 root fileshare@BogoLinux.net 6 Aug 20 2019 Lotus Stuff drwxrws--- 2 root fileshare@BogoLinux.net 8192 Dec 3 14:33 Movies drwxrws--- 19 root fileshare@BogoLinux.net 4096 Oct 5 2020 Original Movies drwxrws---+ 2 root fileshare@BogoLinux.net 6 Mar 15 2015 Swapsies drwxrwx--- 16 root fileshare@BogoLinux.net 4096 Nov 19 16:53 TV Shows [root@Nethserver ~]# ls -l /MyPool/iTunes/ total 24 drwxrws---+ 2 root fileshare@BogoLinux.net 6 Dec 7 2016 Automatically Add to iTunes -rw-rw---- 1 root fileshare@BogoLinux.net 4953 Nov 26 18:41 cd3o.m3u drwxr-sr-x 2 root fileshare@BogoLinux.net 6 Nov 21 2019 Downloads drwxrws--- 3 root fileshare@BogoLinux.net 28 Jun 27 2019 LossLess drwxrws---+ 2 root fileshare@BogoLinux.net 87 Dec 7 2016 Mobile Applications drwxrwx---+ 11 root fileshare@BogoLinux.net 4096 Oct 13 14:38 Music drwxrws---+ 5 root fileshare@BogoLinux.net 90 Nov 21 2019 Podcasts drwxrwx--- 2 root fileshare@BogoLinux.net 41 Sep 21 10:57 Voice Memos [root@Nethserver ~]#

But for a user: [eddie@BogoLinux.net@Nethserver ~]$ ls -l /MyPool/ total 636608 drwxrwx--- 5 root fileshare@BogoLinux.net 4096 Jan 25 21:17 Backup -rw------- 1 root root 651839138 Jan 27 05:02 content -rw------- 1 root root 0 Jan 27 16:14 content.lock drwxrws--- 5 root fileshare@BogoLinux.net 197 Jan 26 16:00 DVR drwxrws--- 2 root fileshare@BogoLinux.net 6 Apr 8 2021 Games drwxrwx--- 4 root fileshare@BogoLinux.net 50 Sep 21 10:57 iTunes drwxrws--- 2 root fileshare@BogoLinux.net 6 Aug 20 2019 Lotus Stuff drwxrws--- 2 root fileshare@BogoLinux.net 8192 Dec 3 14:33 Movies drwxrws--- 19 root fileshare@BogoLinux.net 4096 Oct 5 2020 Original Movies drwxrws---+ 2 root fileshare@BogoLinux.net 6 Mar 15 2015 Swapsies drwxrwx--- 16 root fileshare@BogoLinux.net 4096 Nov 19 16:53 TV Shows [eddie@BogoLinux.net@Nethserver ~]$ ls -l /MyPool/iTunes/ total 0 [eddie@BogoLinux.net@Nethserver ~]$

No other changes other than the mergerfs upgrade took place. Erasing the 2.33.3 version and reinstalling 2.32.5 resolved the issue.

The mount command used: /poolmount/Data? /MyPool fuse.mergerfs defaults,allow_other,direct_io,use_ino,minfreespace=50G,func.getattr=newest,category.create=mfs,fsname=mergerfs 0 0

As an aside, the update from 2.32.? -> 2.33.? also throws a conflict error:

[root@Nethserver ~]# yum update mergerfs-2.33.1-1.el7.x86_64.rpm Loaded plugins: changelog, elrepo, fastestmirror, nethserver_events Examining mergerfs-2.33.1-1.el7.x86_64.rpm: mergerfs-2.33.1-1.el7.x86_64 Marking mergerfs-2.33.1-1.el7.x86_64.rpm as an update to mergerfs-2.32.5-1.el7.x86_64 Resolving Dependencies --> Running transaction check ---> Package mergerfs.x86_64 0:2.32.5-1.el7 will be updated ---> Package mergerfs.x86_64 0:2.33.1-1.el7 will be an update --> Finished Dependency Resolution --> Finding unneeded leftover dependencies Found and removing 0 unneeded dependencies

Dependencies Resolved

================================================================================ Package Arch Version Repository Size

Updating: mergerfs x86_64 2.33.1-1.el7 /mergerfs-2.33.1-1.el7.x86_64 617 k

Transaction Summary

Upgrade 1 Package

Total size: 617 k Is this ok [y/d/N]: y Downloading packages: Running transaction check Running transaction test

Transaction check error: file /usr/share/man/man1/mergerfs.1.gz from install of mergerfs-2.33.1-1.el7.x86_64 conflicts with file from package mergerfs-2.32.5-1.el7.x86_64

Error Summary

[root@Nethserver ~]#

Cheers.

[trapexit]

Can you provide the details requested in the ticket template? I really can't help debug issues if you don't provide a minimal reproducible example and / or the details requested.

[LotusGuy]

System information:

[eddie@BogoLinux.net@Nethserver ~]$ uname -a Linux Nethserver.BogoLinux.net 3.10.0-1160.53.1.el7.x86_64 #1 SMP Fri Jan 14 13:59:45 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux [eddie@BogoLinux.net@Nethserver ~]$ mergerfs -V mergerfs version: 2.33.3 [eddie@BogoLinux.net@Nethserver ~]$ [eddie@BogoLinux.net@Nethserver ~]$ df -h Filesystem Size Used Avail Use% Mounted on devtmpfs 7.8G 0 7.8G 0% /dev tmpfs 7.8G 4.0K 7.8G 1% /dev/shm tmpfs 7.8G 13M 7.8G 1% /run tmpfs 7.8G 0 7.8G 0% /sys/fs/cgroup /dev/mapper/cl-root 32G 8.4G 24G 27% / /dev/sdb 3.7T 1.6T 2.1T 44% /poolmount/Parity1 /dev/sdh 1.9T 1.4T 520G 73% /poolmount/Data1 /dev/sdj 1.9T 1.1T 791G 58% /poolmount/Data6 /dev/sda 3.7T 2.9T 762G 80% /poolmount/DataB /dev/sdl 1.9T 969G 895G 52% /poolmount/Data2 /dev/sdd 1.9T 942G 921G 51% /poolmount/Data8 /dev/sdc 1.9T 950G 913G 51% /poolmount/Data9 /dev/sdg 1.9T 1.4T 514G 73% /poolmount/Data3 /dev/sdk 3.7T 1.9T 1.8T 53% /poolmount/Data4 /dev/sdf 1.9T 926G 937G 50% /poolmount/Data5 /dev/sde 1.9T 648G 1.2T 35% /poolmount/DataA /dev/sdi 1.9T 1.2T 660G 65% /poolmount/Data7 /dev/sdm1 1014M 247M 768M 25% /boot /dev/mapper/cl-PlexLib 20G 4.9G 16G 25% /var/lib/plexmediaserver tmpfs 1.6G 0 1.6G 0% /run/user/1568801106 mergerfs 24T 14T 9.7T 60% /MyPool [eddie@BogoLinux.net@Nethserver ~]$ lsblk -f NAME FSTYPE LABEL UUID MOUNTPOINT sda xfs DataB 2c3641a0-9ec5-470d-bd9d-27d2e54d4fd6 /poolmount/DataB sdb xfs Parity1 c4dfcb52-a2fe-4571-8274-8638aff36cca /poolmount/Parity sdc xfs Data9 9c3c0609-2c09-47c5-915d-8e8520a2301b /poolmount/Data9 sdd xfs Data8 f225a749-93eb-447b-9e3d-aed1d5dc33da /poolmount/Data8 sde xfs DataA b0ed33c5-7e34-4a91-b05f-0bf5041cda33 /poolmount/DataA sdf xfs Data5 4b4b4377-e98b-4fd2-b2f7-948829308f38 /poolmount/Data5 sdg xfs Data3 76a707b5-89fe-488c-9721-5285381b17d4 /poolmount/Data3 sdh xfs Data1 9f12fc75-405f-439e-8a68-f08bb6402341 /poolmount/Data1 sdi xfs Data7 41e428c3-1b67-4e9f-a9d9-71b981f4b59f /poolmount/Data7 sdj xfs Data6 5ef6630d-b91c-40c2-b423-cb38fd8d91c4 /poolmount/Data6 sdk xfs Data4 2f01e9a8-5222-4f24-8420-983c4fae5453 /poolmount/Data4 sdl xfs Data2 9bbe4400-d1b0-418d-baf3-72970e2b91c2 /poolmount/Data2 sdm ├─sdm1 xfs 24f31a2d-cecb-4238-aa6b-00ed87db1db4 /boot └─sdm2 LVM2_mem PTVeke-5Gpq-kV0N-7ud3-omlj-eZJn-EKZZjd ├─cl-root xfs 55901af1-c334-4fe7-9210-a6584e4b9b32 / ├─cl-swap swap 0f4d5c66-91d9-4d95-af43-f8e65ac6addd [SWAP] └─cl-PlexLib xfs PlexLib 38959cdf-bc70-40a3-8580-f388764b3657 /var/lib/plexmedi [eddie@BogoLinux.net@Nethserver ~]$

The mergerfs mount is in the initial report.

For me, to reproduce is simply upgrade from version 2.32.5 to any 2.33 version. (By removing 2.32.5 and installing 2.33.? because of the merge conflict mentioned above). To "fix" the issue, I just revert back to 2.32.5.

Not sure what else I can give that might be useful.

[trapexit]

The docs explain what is useful:

https://github.com/trapexit/mergerfs/blob/master/.github/ISSUE_TEMPLATE/bug_report.md https://github.com/trapexit/mergerfs#support

A completely stripped down version of the situation. Does it happen with only one branch? or is it only multiple branches? Do other commands work or is it just readdir that fails? Have you taken straces? Is it different with different permissions.

[trapexit]

Many people, including myself, use the latest versions without issue so I need the most simplest version of the issue to attempt to reproduce. I don't have a CentOS system. I don't have your setup. I don't have those filesystems with those files and permissions. I need something as simple as possible that replicates the issue and all the relevant data to debug without having access to your machine. That includes all the details I've requested in the docs.

[LotusGuy]

Still working on setting up a minimal environment, but in the meantime, hoping it helps, strace's of a simple "ls" command on both my working system and the failing one. Files are actually tgz, but can't upload that extension.

2.33.3.gz 2.32.5.gz .

[trapexit]

If this is still an issue with the latest release please reopen. I was never able to reproduce even on a centos 7 system.