search

opensciencegrid / xrootd-multiuser

A filesystem plugin to allow Xrootd write as a different Unix user
Apache License 2.0
2 stars 12 forks source link

Segfault in MultiuserChecksum::Calc with gfal-sum and macaroons #38

Closed jthiltges closed 7 months ago

jthiltges commented 1 year ago

When it doesn't have a credential, gfal-sum appears to retrieve a macaroon, then uses it to contact the server. This causes a segfault.

If the incoming request doesn't have a username, the sentryPtr is returned as a nullptr https://github.com/opensciencegrid/xrootd-multiuser/blob/c50f8cb33921907fb32e4e8f1dda96da1261017a/src/multiuser.cpp#L246-L249

Then sentryPtr->IsValid() hits the nullptr and segfaults https://github.com/opensciencegrid/xrootd-multiuser/blob/c50f8cb33921907fb32e4e8f1dda96da1261017a/src/multiuser.cpp#L259-L260

Backtrace segfault ``` Thread 5 "xrootd" received signal SIGSEGV, Segmentation fault. MultiuserChecksum::Calc (this=, Xfn=, Cks=..., doSet=) at /usr/src/debug/xrootd-multiuser-2.1.2-1.osg36.el8.x86_64/src/multiuser.cpp:260 260 if (!sentryPtr->IsValid()) return -EACCES; (gdb) bt #0 MultiuserChecksum::Calc (this=, Xfn=, Cks=..., doSet=) at /usr/src/debug/xrootd-multiuser-2.1.2-1.osg36.el8.x86_64/src/multiuser.cpp:260 #1 0x00007f33fd75da07 in XrdOfs::chksum (this=0x7f33fd9dec40 , Func=XrdSfsFileSystem::csCalc, csName=0x7f33e40174c0 "adler32", Path=0x7f33e4012f70 "/store/hello_world.txt", einfo=..., client=0x0, opaque=0x0) at /usr/src/debug/xrootd-5.5.0-1.1.osg36.el8.x86_64/xrootd/src/XrdOfs/XrdOfs.cc:1847 #2 0x00007f33fd741711 in XrdXrootdProtocol::CheckSum (Stream=0x7f33e4011090, argv=0x7f33f90a6b50, argc=) at /usr/src/debug/xrootd-5.5.0-1.1.osg36.el8.x86_64/xrootd/src/XrdXrootd/XrdXrootdProtocol.cc:1023 #3 0x00007f33fd449c97 in XrdOucProg::Run (this=0x254db70, Sp=0x7f33e4011090, argV=, argC=3, envV=) at /usr/src/debug/xrootd-5.5.0-1.1.osg36.el8.x86_64/xrootd/src/XrdOuc/XrdOucProg.cc:130 #4 0x00007f33fd449f7e in XrdOucProg::Run (this=, Sp=Sp@entry=0x7f33e4011090, arg1=, arg2=, arg3=, arg4=) at /usr/src/debug/xrootd-5.5.0-1.1.osg36.el8.x86_64/xrootd/src/XrdOuc/XrdOucProg.cc:186 #5 0x00007f33fd738764 in XrdXrootdJob2Do::DoIt (this=0x7f33e4010fe0) at /usr/src/debug/xrootd-5.5.0-1.1.osg36.el8.x86_64/xrootd/src/XrdXrootd/XrdXrootdJob.cc:179 #6 0x00007f33fd48d977 in XrdScheduler::Run (this=0x615740 ) at /usr/src/debug/xrootd-5.5.0-1.1.osg36.el8.x86_64/xrootd/src/Xrd/XrdScheduler.cc:406 #7 0x00007f33fd48da99 in XrdStartWorking (carg=) at /usr/src/debug/xrootd-5.5.0-1.1.osg36.el8.x86_64/xrootd/src/Xrd/XrdScheduler.cc:89 #8 0x00007f33fd41e047 in XrdSysThread_Xeq (myargs=0x2511660) at /usr/src/debug/xrootd-5.5.0-1.1.osg36.el8.x86_64/xrootd/src/XrdSys/XrdSysPthread.cc:86 #9 0x00007f33fc4761cf in start_thread () from /lib64/libpthread.so.0 #10 0x00007f33fc0e2dd3 in clone () from /lib64/libc.so.6 ```

Reproducing

To reproduce, make an unauthenticated request with gfal-sum

$ docker run --rm=true -it opensciencegrid/software-base:3.6-el7-release /bin/bash
[root@65cb5394c895 /]# ulimit -n 16364
[root@65cb5394c895 /]# yum install -y gfal2-all gfal2-util-scripts python2-gfal2-util
[root@65cb5394c895 /]# gfal-sum https://red-xfer13.unl.edu:1094/store/hello_world.txt ADLER32
gfal-sum error: 112 (Host is down) - (Neon): Could not read status line: Secure connection truncated

Environment

Further debugging

Setting a breakpoint on MultiuserChecksum::GenerateUserSentry and submitting an unauthenticated checksum request with gfal-sum, we get two breakpoint hits. The first time, env->secEnv() is defined, m_is_anonymous gets set, and the sentryPtr is assigned as expected. https://github.com/opensciencegrid/xrootd-multiuser/blob/c50f8cb33921907fb32e4e8f1dda96da1261017a/src/UserSentry.hh#L77-L78

Backtrace 1 ``` (gdb) bt #0 MultiuserChecksum::GenerateUserSentry (env=0x7f87b4891d40, this=0x150b040) at /usr/src/debug/xrootd-multiuser-2.1.2-1.osg36.el8.x86_64/src/multiuser.cpp:242 #1 MultiuserChecksum::Get (this=0x150b040, Xfn=0x7f87a0008000 "/store/hello_world.txt", Cks=...) at /usr/src/debug/xrootd-multiuser-2.1.2-1.osg36.el8.x86_64/src/multiuser.cpp:282 #2 0x00007f87b8f4a9b4 in XrdOfs::chksum (this=0x7f87b91cbc40 , Func=XrdSfsFileSystem::csGet, csName=0x13b0af0 "adler32", Path=0x7f87a0008000 "/store/hello_world.txt", einfo=..., client=0x7f87a0000de8, opaque=0x7f87a0008017 "authz=Bearer%20MDAxY2xvY2F0aW9uIFQyX1VTX05lYnJhc2thCjAwMzRpZGVudGlmaWVyIGVkN2U3OTAxLTA5ZDYtNDQxNC1iMzdlLWNmMDM0MTBlNTY3YwowMDFmY2lkIGFjdGl2aXR5OlJFQURfTUVUQURBVEEKMDAxZmNpZCBhY3Rpdml0eTpMSVNULERPV05MT"...) at /usr/src/debug/xrootd-5.5.0-1.1.osg36.el8.x86_64/xrootd/src/XrdOfs/XrdOfs.cc:1848 #3 0x00007f87b8f39ac2 in XrdXrootdProtocol::do_CKsum (this=0x7f87a00183c8, algT=0x13b0af0 "adler32", Path=0x7f87a0008000 "/store/hello_world.txt", Opaque=0x7f87a0008017 "authz=Bearer%20MDAxY2xvY2F0aW9uIFQyX1VTX05lYnJhc2thCjAwMzRpZGVudGlmaWVyIGVkN2U3OTAxLTA5ZDYtNDQxNC1iMzdlLWNmMDM0MTBlNTY3YwowMDFmY2lkIGFjdGl2aXR5OlJFQURfTUVUQURBVEEKMDAxZmNpZCBhY3Rpdml0eTpMSVNULERPV05MT"...) at /usr/src/debug/xrootd-5.5.0-1.1.osg36.el8.x86_64/xrootd/src/XrdXrootd/XrdXrootdXeq.cc:486 #4 0x00007f87b8f43f8a in XrdXrootdProtocol::do_CKsum (this=0x7f87a00183c8, canit=) at /usr/include/c++/8/ext/new_allocator.h:116 #5 0x00007f87b8f30378 in XrdXrootdProtocol::Process2 (this=0x7f87a00183c8) at /usr/src/debug/xrootd-5.5.0-1.1.osg36.el8.x86_64/xrootd/src/XrdXrootd/XrdXrootdProtocol.cc:590 #6 0x00007f87b8f34395 in XrdXrootdTransit::Process (this=0x7f87a00183c0, lp=0x7f87a4007880) at /usr/src/debug/xrootd-5.5.0-1.1.osg36.el8.x86_64/xrootd/src/XrdXrootd/XrdXrootdTransit.cc:397 #7 0x00007f87b8c77a40 in XrdLinkXeq::DoIt (this=) at /usr/src/debug/xrootd-5.5.0-1.1.osg36.el8.x86_64/xrootd/src/Xrd/XrdLinkXeq.cc:320 #8 XrdLinkXeq::DoIt (this=0x7f87a4007880) at /usr/src/debug/xrootd-5.5.0-1.1.osg36.el8.x86_64/xrootd/src/Xrd/XrdLinkXeq.cc:308 #9 0x00007f87b8c74392 in XrdLink::setProtocol (this=0x7f87a4007880, pp=, runit=, push=) at /usr/src/debug/xrootd-5.5.0-1.1.osg36.el8.x86_64/xrootd/src/Xrd/XrdLink.cc:435 #10 0x00007f87b8c7a977 in XrdScheduler::Run (this=0x615740 ) at /usr/src/debug/xrootd-5.5.0-1.1.osg36.el8.x86_64/xrootd/src/Xrd/XrdScheduler.cc:406 #11 0x00007f87b8c7aa99 in XrdStartWorking (carg=) at /usr/src/debug/xrootd-5.5.0-1.1.osg36.el8.x86_64/xrootd/src/Xrd/XrdScheduler.cc:89 #12 0x00007f87b8c0b047 in XrdSysThread_Xeq (myargs=0x1374660) at /usr/src/debug/xrootd-5.5.0-1.1.osg36.el8.x86_64/xrootd/src/XrdSys/XrdSysPthread.cc:86 #13 0x00007f87b7c631cf in start_thread () from /lib64/libpthread.so.0 #14 0x00007f87b78cfdd3 in clone () from /lib64/libc.so.6 (gdb) print *client $1 = {prot = "https\000\000", prox = "\000\000\000\000\000\000\000", name = 0x0, host = 0x7f87a00012c0 "[::ffff:129.93.227.111]", vorg = 0x0, role = 0x0, grps = 0x0, caps = 0x0, endorsements = 0x0, moninfo = 0x0, creds = 0x0, credslen = 0, ueid = 1, addrInfo = 0x7f87a40079c0, tident = 0x7f87a4007a7a "unknown.1:35@hcc-pki-vpn", pident = 0x7f87a4007a7a "unknown.1:35@hcc-pki-vpn", sessvar = 0x0, uid = 0, gid = 0, future = {0x0, 0x0, 0x0}, eaAPI = 0x7f87a00011e0} ```

But then there's a second breakpoint hit--from the same client request--and the env hash is empty, leading to the segfault.

Backtrace 2 ``` #0 MultiuserChecksum::GenerateUserSentry (env=0x7f87b4893290, this=0x150b040) at /usr/src/debug/xrootd-multiuser-2.1.2-1.osg36.el8.x86_64/src/multiuser.cpp:241 #1 MultiuserChecksum::Calc (this=0x150b040, Xfn=0x7f87a0012f60 "/store/hello_world.txt", Cks=..., doSet=1) at /usr/src/debug/xrootd-multiuser-2.1.2-1.osg36.el8.x86_64/src/multiuser.cpp:259 #2 0x00007f87b8f4aa07 in XrdOfs::chksum (this=0x7f87b91cbc40 , Func=XrdSfsFileSystem::csCalc, csName=0x7f87a00174b0 "adler32", Path=0x7f87a0012f60 "/store/hello_world.txt", einfo=..., client=0x0, opaque=0x0) at /usr/src/debug/xrootd-5.5.0-1.1.osg36.el8.x86_64/xrootd/src/XrdOfs/XrdOfs.cc:1847 #3 0x00007f87b8f2e711 in XrdXrootdProtocol::CheckSum (Stream=0x7f87a0011080, argv=0x7f87b4893b50, argc=) at /usr/src/debug/xrootd-5.5.0-1.1.osg36.el8.x86_64/xrootd/src/XrdXrootd/XrdXrootdProtocol.cc:1023 #4 0x00007f87b8c36c97 in XrdOucProg::Run (this=0x13b0b70, Sp=0x7f87a0011080, argV=, argC=3, envV=) at /usr/src/debug/xrootd-5.5.0-1.1.osg36.el8.x86_64/xrootd/src/XrdOuc/XrdOucProg.cc:130 #5 0x00007f87b8c36f7e in XrdOucProg::Run (this=, Sp=Sp@entry=0x7f87a0011080, arg1=, arg2=, arg3=, arg4=) at /usr/src/debug/xrootd-5.5.0-1.1.osg36.el8.x86_64/xrootd/src/XrdOuc/XrdOucProg.cc:186 #6 0x00007f87b8f25764 in XrdXrootdJob2Do::DoIt (this=0x7f87a0010fd0) at /usr/src/debug/xrootd-5.5.0-1.1.osg36.el8.x86_64/xrootd/src/XrdXrootd/XrdXrootdJob.cc:179 #7 0x00007f87b8c7a977 in XrdScheduler::Run (this=0x615740 ) at /usr/src/debug/xrootd-5.5.0-1.1.osg36.el8.x86_64/xrootd/src/Xrd/XrdScheduler.cc:406 #8 0x00007f87b8c7aa99 in XrdStartWorking (carg=) at /usr/src/debug/xrootd-5.5.0-1.1.osg36.el8.x86_64/xrootd/src/Xrd/XrdScheduler.cc:89 #9 0x00007f87b8c0b047 in XrdSysThread_Xeq (myargs=0x1374660) at /usr/src/debug/xrootd-5.5.0-1.1.osg36.el8.x86_64/xrootd/src/XrdSys/XrdSysPthread.cc:86 #10 0x00007f87b7c631cf in start_thread () from /lib64/libpthread.so.0 #11 0x00007f87b78cfdd3 in clone () from /lib64/libc.so.6 (gdb) print client $4 = (const XrdSecEntity *) 0x0 ```
djw8605 commented 1 year ago

Great issue description. Since you did all this work already, do you have a suggested fix?

jthiltges commented 1 year ago

Fair question! I didn't... but spent a little more time poking and might now. I suspect the issue is that the anonymous user gets a macaroon without a username. Then nothing is set for the request.name.

https://github.com/xrootd/xrootd/blob/8e1e528232d8ead08468eefcbf991737eb5751da/src/XrdMacaroons/XrdMacaroonsAuthz.cc#L260-L264

If we have no secEnv and the request.name is unset, we need to treat it as anonymous. I'm not sure of a clean way to handle this. We could extend UserSentry(const std::string username to treat an empty username string as anonymous.

https://github.com/opensciencegrid/xrootd-multiuser/blob/c50f8cb33921907fb32e4e8f1dda96da1261017a/src/UserSentry.hh#L90-L94

following the pattern of UserSentry(const XrdSecEntity *client above

https://github.com/opensciencegrid/xrootd-multiuser/blob/c50f8cb33921907fb32e4e8f1dda96da1261017a/src/UserSentry.hh#L76-L78