search

trapexit / mergerfs

a featureful union filesystem
http://spawn.link
Other
4.29k stars 174 forks source link

xattrs do not work on nfs mounted mergerfs pool #1132

Open Tomaszal opened 1 year ago

Tomaszal commented 1 year ago

Describe the bug

nfs supports xattrs since Linux 5.9 and nfs 4.2. I have tested it on my setup by exporting a regular folder on an ext4 filesystem over nfs and attempting to use xattrs. However, when exporting a mergerfs pool over nfs, trying to use xattrs errors with Operation not supported. I have tried using the nfsopenhack=all option, but it does not help. From the nfsopenhack description it's unrelated anyway, so I did not expect it to help, but it was the only relevant option I could try.

To Reproduce

  1. Create a mergerfs pool on a server host, f.e. at /mnt/storage/pool
  2. Export the pool on the server host using nfs 4.2 or newer and Linux 5.9 or newer, f.e. with the following in /etc/exports: 
    /mnt/storage/pool 192.168.1.0/24(rw,fsid=0,no_root_squash)
  3. Mount the nfs directory on a client host, f.e.: 
    mount -t nfs -o vers=4 servername:/ /mnt/storage
  4. Create a test file on the client host nfs directory, f.e.: 
    touch /mnt/storage/test
  5. Try setting an xattr on the test file on the client host, f.e.: 
    attr -s foo -V bar /mnt/storage/test

Expected behavior

The xattr gets set successfully on the client host:

Attribute "foo" set to a 4 byte value for /mnt/storage/test:
bar

And can be accessed afterwards on the client host:

attr -l /mnt/storage/test 
Attribute "foo" has a 4 byte value for /mnt/storage/test

System information:

trapexit commented 1 year ago

Hm... not sure why it wouldn't work if you have xattrs enabled in mergerfs. Maybe there is something funny with how FUSE and NFS work together in this regard. I'll setup a NFS share and check it out.

Tomaszal commented 1 year ago

Yeah I forgot to mention that xattrs work locally in the mergerfs pool with these configs, they only don't work when accessing the pool through nfs.

trapexit commented 1 year ago

My guess is that it is a kernel thing. I'll ask the FUSE authors.

trapexit commented 1 year ago

I'm almost positive I tested this and had the issues you mention but I just tested again and it is working fine. Kernel 5.15 on Ubuntu and Mint systems.

Can you double check the setup and instructions to reproduce?

Tomaszal commented 1 year ago

@trapexit I've tested it again on Arch with kernel 6.1.12 and got the same results. This time I set up a MergerFS pool on the system drive and NFS server and client on the same machine as I don't have the original setup. Seeing how that's enough to reproduce the bug, I've made a script to make it easier to test:

#!/bin/bash
set -euo pipefail

# Set test root folder
TEST_ROOT="/mnt/mergerfs-xattrs-test"

# Create required directories
mkdir -p $TEST_ROOT/{disks/{1,2,3},pool,client}

# Mount MergerFS pool
mergerfs -o defaults,nfsopenhack=all,xattr=passthrough "$TEST_ROOT/disks/*" "$TEST_ROOT/pool"

# Setup NFS exports
NFS_EXPORT_LINE="$TEST_ROOT/pool 127.0.0.1(rw,fsid=0,no_root_squash,no_subtree_check)"
NFS_EXPORT_FILE="/etc/exports"
grep -qF -- "$NFS_EXPORT_LINE" "$NFS_EXPORT_FILE" || echo "$NFS_EXPORT_LINE" >>"$NFS_EXPORT_FILE"

# Start NFS server and refresh exports
systemctl start nfsv4-server.service
exportfs -ar

# Setup NFS client
mount -t nfs -o vers=4 127.0.0.1:/ "$TEST_ROOT/client"

# Perform test
touch "$TEST_ROOT/client/test"
if attr -s foo -V bar "$TEST_ROOT/client/test"; then
  RESULT="PASS"
else
  RESULT="FAIL"
fi

# Cleanup
awk -i inplace -v line="$NFS_EXPORT_LINE" '!index($0,line)' "$NFS_EXPORT_FILE" 
exportfs -ar
sleep 1s # wait for mounts to not be busy
umount "$TEST_ROOT/client"
umount "$TEST_ROOT/pool"
rm -rf "$TEST_ROOT"

# Print result
echo "MergerFS xattrs test: $RESULT"

Here's the output I get on my system (running the script with sudo):

attr_set: Operation not supported
Could not set "foo" for /mnt/mergerfs-xattrs-test/client/test
MergerFS xattrs test: FAIL
trapexit commented 1 year ago

Thanks. Will take a look and get back to the kernel folks.

trapexit commented 1 year ago

Have you tried setting fsid and not using the mounting of root path like that? Because besides that I simply can't reproduce this on 5.15 on Ubuntu. It works fine if I change the fsid and mount the full path.

I don't know why that would matter but it appears to and for me if you just make those changes to your script it works as desired.

Tomaszal commented 1 year ago

You're right, after doing the following changes it works as expected:

 # Setup NFS exports
-NFS_EXPORT_LINE="$TEST_ROOT/pool 127.0.0.1(rw,fsid=0,no_root_squash,no_subtree_check)"
+NFS_EXPORT_LINE="$TEST_ROOT/pool 127.0.0.1(rw,fsid=1,no_root_squash,no_subtree_check)"
 # Setup NFS client
-mount -t nfs -o vers=4 127.0.0.1:/ "$TEST_ROOT/client"
+mount -t nfs -o vers=4 "127.0.0.1:$TEST_ROOT/pool" "$TEST_ROOT/client

It is very odd and I'd still consider it a bug, as I see no reason why it shouldn't work in the first scenario. But I guess the issue is probably with NFS and at least there's this workaround now. Though it still might have something to do with how MergerFS interacts with NFS, as running the first case (with fsid=0) without creating the MergerFS pool works just fine (simply comment out the mergerfs and umount "$TEST_ROOT/pool" lines to try).

trapexit commented 1 year ago

Not sure but at most it is a FUSE <-> NFS issue as mergerfs is not behaving any differently between these two situations. I'll do some more investigation and talk to the kernel devs.