xrdfs ls not working with POSIX system #2069

[lucalavezzo]

Hi,

We have xrootd (v5.6.1) set up with the xrootd-multiuser plugin (2.1.2-1.osg36.el7), and are experiencing some problems with the xrdfs ls command. For context, we have it running with xrootd-privileged and cmsd-privileged as a service on our cluster.

While we are able to chmod, mkdir, touch etc. on xrdfs correctly, the ls for a directory that according to the POSIX permissions I should be able to view (and can view in a shell) does not seem to work; e.g. if I (lavezzo) want to cat a file that has 700 permissions I can, but if I want to ls a directory with the same permissions, I cannot:

[lavezzo@XXX]$ ll
drwx--S---  2 lavezzo lavezzo 4096 Aug  7 18:04 test
[lavezzo@XXX lavezzo]$ ll test/file.txt
-rwx------ 1 lavezzo lavezzo 13 Aug  7 18:04 test/file.txt
[lavezzo@XXX]$ xrdfs root://<redirector>/
[<redirector>:1094] / > cd /store/user/lavezzo
[<redirector>:1094] /store/user/lavezzo > ls -l
d--- 2023-08-07 22:04:49        4096 /store/user/lavezzo/test
[<redirector>:1094] /store/user/lavezzo > ls test
[<redirector>:1094] /store/user/lavezzo > ls test/file.txt
[ERROR] Server responded with an error: [3005] Unable to open directory /store/user/lavezzo/test/file.txt; not a directory

[<redirector>:1094] /store/user/lavezzo > cat test/file.txt
hello world!

In the logs, I see that multiuser is doing something, but then just hits me with permission denied:

230807 18:13:19 2363453 multiuser_UserSentry: Switching FS uid for user lavezzo
230807 18:13:19 2363453 multiuser_UserSentry: Switching FS uid for user lavezzo
230807 18:13:19 2363453 ofs_readdir: lavezzo.3189625:38@XXX Unable to read directory /store/user/lavezzo/test; permission denied

The problem happens both if I make a directory and make it readable only by me via a normal shell as well as via xrdfs.

When I make it via bash,

[lavezzo@XXX]$ mkdir testing_permissions
[lavezzo@XXX]$ echo "hello world!" > testing_permissions/file.txt
[lavezzo@XXX]$ chmod -R 700 testing_permissions/
[lavezzo@XXX]$ ll
total 24
drwx--S---  2 lavezzo lavezzo 4096 Aug  8 06:56 testing_permissions
[lavezzo@XXX]$ ll testing_permissions/
total 2
-rwx------ 1 lavezzo lavezzo 12 Aug  8 06:56 file.txt
[lavezzo@XXX]$ ls testing_permissions/
file.txt
[lavezzo@XXX]$ cat testing_permissions/file.txt
hello world
[lavezzo@XXX]$ xrdfs root://<REDIRECTOR>/
[<REDIRECTOR>:1094] / > cd /store/user/lavezzo
[<REDIRECTOR>:1094] /store/user/lavezzo > ls -l
d--- 2023-08-08 10:56:44        4096 /store/user/lavezzo/testing_permissions
[<REDIRECTOR>:1094] /store/user/lavezzo > ls testing_permissions
[<REDIRECTOR>:1094] /store/user/lavezzo > cat testing_permissions/file.txt
hello world

and via xrdfs,

[REDIRECTOR:1094] /store/user/lavezzo > mkdir testing_permissions2
[REDIRECTOR:1094] /store/user/lavezzo > ls -l
d--- 2023-08-08 10:56:44        4096 /store/user/lavezzo/testing_permissions
d--- 2023-08-08 11:00:26        4096 /store/user/lavezzo/testing_permissions2
[REDIRECTOR:1094] /store/user/lavezzo > ls testing_permissions2
[lavezzo@XXX]$ echo "hello world" > testing_permissions2/file.txt
[lavezzo@XXX]$ xrdfs root://REDIRECTOR/
[REDIRECTOR:1094] / > cd /store/user/lavezzo
[REDIRECTOR:1094] /store/user/lavezzo > ls testing_permissions2
[lavezzo@XXX]$ ls testing_permissions2
file.txt

Here you can see clearly in both examples that on the normal shell I can ls the directories without problem, but via xrdfds I cannot.

This is particularly damaging for us because by default when a directory is created through xrdfs mkdir, the permissions are set as such that other users don't have read access, and we haven't been able to successfully change this either.

Any clue whether this is an issue of xrootd-multiuser or of our setup? In either case I would really appreciate some help in debugging this!

[djw8605]

Interesting, for a readdir, it doesn't look like we are "wrapping" the call with a change in permissions. See MultiuserDirectory.hh below: https://github.com/opensciencegrid/xrootd-multiuser/blob/4ca61896df0108afe8adee14613c328639e24052/src/MultiuserDirectory.hh#L31-L34

Compare with the opendir above. I wonder if that's the issue. @jthiltges and @abh3 , I would like your opinions. I don't know the code flow, but it seems logical that the readdir goes through this code path.

[lucalavezzo]

Interesting, we might try to test the Readdir modified with your suggestion, i.e. wrapping it with the change in permissions.

In the meanwhile, is there a way to default a different permissions for newly-created directories via xrdfs mkdir?

Thanks again.

[djw8605]

Are you familiar with umasks? There is the option of multiuser.umask in the config to specify a umask which may solve the issue with newly created directories.

[abh3]

In xrootd 5.6 you can specify the minimum and maximum permission allowed. The specified permissions are enforced. See,

https://xrootd.slac.stanford.edu/doc/dev56/ofs_config.htm#_Toc136617291

These permission are applied before the multi-user plugin is called.

Andy

On Wed, 9 Aug 2023, lucalavezzo wrote:

Interesting, we might try to test the Readdir modified with your suggestion, i.e. wrapping it with the change in permissions.

In the meanwhile, is there a way to default a different permissions for newly-created directories via xrdfs mkdir?

Thanks again.

-- Reply to this email directly or view it on GitHub: https://github.com/opensciencegrid/xrootd-multiuser/issues/49#issuecomment-1671980376 You are receiving this because you were mentioned.

Message ID: @.***>

[lucalavezzo]

@djw8605 yea, indeed we have it set like:

ofs.osslib ++ libXrdMultiuser.so
ofs.ckslib * libXrdMultiuser.so
multiuser.umask 0002
multiuser.checksumonwrite off

But as the example before showed, it doesn't seem to be working.

@abh3 Thank you for the suggestion! I have set up the minimum permissions to be quite high

ofs.crmode dirs 0644:0644 files 0644:0644

and indeed, I can now see this working, i.e. if I make xrdfs mkdir I can also xrdfs ls it. However, I'm not sure whether this resolved the underlying problem with xrdfs ls, as this leaves the issue that an user cannot read their own files, if they want to make them private.

[jthiltges]

Hi @lucalavezzo,

I worked on replicating the issue and ran across an interesting difference in behavior: xrdfs ... ls seems to works, while ls -l does not.

In the ls case, XRootD opens the directory, multiuser drops permissions, then XRootD lists the contents from the already open file descriptor.

In the ls -l case, XRootD opens the directory, multiuser drops permissions, XRootD attempts to fstat the contents, and the call fails due to permissions.

As to the best solution, I don't yet have an opinion without diving in further.

Regards, John

strace details

I ran the following command to list a directory with 700 permissions: ``` $ xrdfs xrootd.example.edu:1094 ls /store/user/jthiltge/owner.cmsuser/child ``` And the strace with annotations: ``` # xrootd-multiuser switches to client's uid/gid [pid 2606100] setfsuid(7343) = 991 [pid 2606100] setfsgid(11265) = 987 # xrootd opens fd on target directory [pid 2606100] openat(AT_FDCWD, "/mnt/t2ceph/cms/store/user/jthiltge/owner.cmsuser/child", O_RDONLY|O_CLOEXEC) = 40 [pid 2606100] fstat(40, {st_mode=S_IFDIR|0700, st_size=2, ...}) = 0 [pid 2606100] fcntl(40, F_GETFL) = 0x8000 (flags O_RDONLY|O_LARGEFILE) [pid 2606100] fcntl(40, F_SETFD, FD_CLOEXEC) = 0 # xrootd-multiuser switches back to xrootd service uid/gid [pid 2606100] setfsuid(991) = 7343 [pid 2606100] setfsgid(987) = 11265 # xrootd lists directory using open fd [pid 2606100] getdents64(40, 0xe317c0 /* 4 entries */, 65536) = 112 [pid 2606100] getdents64(40, 0xe317c0 /* 0 entries */, 65536) = 0 ``` Running the same command, except with `ls -l`: ``` # xrootd-multiuser switches to client's uid/gid [pid 2606100] setfsuid(7343) = 991 [pid 2606100] setfsgid(11265) = 987 # xrootd opens fd on target directory [pid 2606100] openat(AT_FDCWD, "/mnt/t2ceph/cms/store/user/jthiltge/owner.cmsuser/child", O_RDONLY|O_CLOEXEC) = 40 [pid 2606100] fstat(40, {st_mode=S_IFDIR|0700, st_size=2, ...}) = 0 [pid 2606100] fcntl(40, F_GETFL) = 0x8000 (flags O_RDONLY|O_LARGEFILE) [pid 2606100] fcntl(40, F_SETFD, FD_CLOEXEC) = 0 # xrootd-multiuser switches back to xrootd service uid/gid [pid 2606100] setfsuid(991) = 7343 [pid 2606100] setfsgid(987) = 11265 # xrootd lists directory using open fd [pid 2606100] getdents64(40, 0xe317c0 /* 4 entries */, 65536) = 112 # xrootd calls fstat("."), which fails as the permissions have already dropped [pid 2606100] newfstatat(40, ".", 0x7f669a0b1070, 0) = -1 EACCES (Permission denied) ```

[jthiltges]

Interesting, for a readdir, it doesn't look like we are "wrapping" the call with a change in permissions. See MultiuserDirectory.hh below:

https://github.com/opensciencegrid/xrootd-multiuser/blob/4ca61896df0108afe8adee14613c328639e24052/src/MultiuserDirectory.hh#L31-L34

Compare with the opendir above. I wonder if that's the issue. @jthiltges and @abh3 , I would like your opinions. I don't know the code flow, but it seems logical that the readdir goes through this code path.

Looking like you nailed it, Derek.

Grabbing a backtrace of the denied fstatat call, we find XrdOssDir::Readdir up the chain. I'll open a PR shortly to include the id switching there too. A disadvantage is that will it cause an id switch per entry, but I'm not sure of a better immediate solution.

(gdb) bt
#0  0x00007f692bc1363d in __fxstatat64 () from /lib64/libc.so.6
#1  0x00007f692d1cdc25 in fstatat64 (__flag=0, __statbuf=<optimized out>, __filename=0x1c7c223 ".", __fd=<optimized out>) at /usr/include/sys/stat.h:477
#2  XrdOssDir::Readdir (blen=255, buff=0x1bfa164 ".", this=0x1cd70e0) at /usr/src/debug/xrootd-5.5.5-1.2.osg36.el8.x86_64/xrootd/src/XrdOss/XrdOssApi.cc:598
#3  XrdOssDir::Readdir (this=0x1cd70e0, buff=0x1bfa164 ".", blen=255) at /usr/src/debug/xrootd-5.5.5-1.2.osg36.el8.x86_64/xrootd/src/XrdOss/XrdOssApi.cc:583
#4  0x00007f692d1a833e in XrdOfsDirectory::nextEntry (this=0x1bfa130) at /usr/src/debug/xrootd-5.5.5-1.2.osg36.el8.x86_64/xrootd/src/XrdOfs/XrdOfs.cc:334
#5  0x00007f692d1a2983 in XrdXrootdProtocol::do_DirStat (this=0x1be6630, dp=0x1bfa130, pbuff=0x7f6912581bb0 "", opaque=0x0) at /usr/src/debug/xrootd-5.5.5-1.2.osg36.el8.x86_64/xrootd/src/XrdXrootd/XrdXrootdXeq.cc:748
#6  0x00007f692d1a3062 in XrdXrootdProtocol::do_Dirlist (this=this@entry=0x1be6630) at /usr/src/debug/xrootd-5.5.5-1.2.osg36.el8.x86_64/xrootd/src/XrdXrootd/XrdXrootdXeq.cc:650
#7  0x00007f692d18f4de in XrdXrootdProtocol::Process2 (this=0x1be6630) at /usr/src/debug/xrootd-5.5.5-1.2.osg36.el8.x86_64/xrootd/src/XrdXrootd/XrdXrootdProtocol.cc:585
#8  0x00007f692ced6460 in XrdLinkXeq::DoIt (this=<optimized out>) at /usr/src/debug/xrootd-5.5.5-1.2.osg36.el8.x86_64/xrootd/src/Xrd/XrdLinkXeq.cc:320
#9  XrdLinkXeq::DoIt (this=0x7f6918002098) at /usr/src/debug/xrootd-5.5.5-1.2.osg36.el8.x86_64/xrootd/src/Xrd/XrdLinkXeq.cc:308
#10 0x00007f692ced2d92 in XrdLink::setProtocol (this=0x7f6918002098, pp=<optimized out>, runit=<optimized out>, push=<optimized out>) at /usr/src/debug/xrootd-5.5.5-1.2.osg36.el8.x86_64/xrootd/src/Xrd/XrdLink.cc:435
#11 0x00007f692ced9707 in XrdScheduler::Run (this=0x615740 <XrdGlobal::Sched>) at /usr/src/debug/xrootd-5.5.5-1.2.osg36.el8.x86_64/xrootd/src/Xrd/XrdScheduler.cc:406
#12 0x00007f692ced9829 in XrdStartWorking (carg=<optimized out>) at /usr/src/debug/xrootd-5.5.5-1.2.osg36.el8.x86_64/xrootd/src/Xrd/XrdScheduler.cc:89
#13 0x00007f692ce695b7 in XrdSysThread_Xeq (myargs=0x7f69180222d0) at /usr/src/debug/xrootd-5.5.5-1.2.osg36.el8.x86_64/xrootd/src/XrdSys/XrdSysPthread.cc:86
#14 0x00007f692bec11cf in start_thread () from /lib64/libpthread.so.0
#15 0x00007f692bb2ce73 in clone () from /lib64/libc.so.6

[lucalavezzo]

Thank you for the solution. When will this be available through yum install?

[djw8605]

Sure, what OS do you need it for? EL8?

[djw8605]

• EL7: https://kojihub0000.chtc.wisc.edu/kojifiles/work/tasks/503/380503/xrootd-multiuser-2.1.4-1.osg36.el7.x86_64.rpm
• EL8: https://kojihub0000.chtc.wisc.edu/kojifiles/work/tasks/501/380501/xrootd-multiuser-2.1.4-1.osg36.el8.x86_64.rpm
• EL9: https://kojihub0000.chtc.wisc.edu/kojifiles/work/tasks/499/380499/xrootd-multiuser-2.1.4-1.osg36.el9.x86_64.rpm

Please test and let us know if it works for you.

[lucalavezzo]

Seems to be working great, thanks a lot!