search

dCache / dcache

dCache - a system for storing and retrieving huge amounts of data, distributed among a large number of heterogenous server nodes, under a single virtual filesystem tree with a variety of standard access methods
https://dcache.org
277 stars 132 forks source link

User can sever directories from dCache namespace by renaming them #7559

Closed XMol closed 2 months ago

XMol commented 2 months ago

Hello dCache.org,

as reported on dCache RT #10608, a CMS user managed to render his/her data inaccessible with a basic rename operation. Admins can recover from that, no data is lost, but the same incident is reproducible at will:

  1. Create a new directory and rename it to a subdirectory of itself 
    $ xrdfs ppsdcache-kit.gridka.de mkdir /pnfs/gridka.de/dteam/disk-only/top
$ xrdfs ppsdcache-kit.gridka.de mv /pnfs/gridka.de/dteam/disk-only/top{,/sub}

  2. The subdirectory "sub" now has itself as parent directory, not "disk-only", and "top" is gone.

    chimera=> select * from t_dirs where iname = 'sub' or iname = 'top';
 iparent | ichild | iname
---------+--------+-------
  842029 | 842029 | sub
(1 row)

chimera=> select * from t_dirs where iparent = path2inumber(pnfsid2inumber('000000000000000000000000000000000000'), 'pnfs/gridka.de/dteam/disk-only') order by iname;
 iparent | ichild |                      iname
---------+--------+--------------------------------------------------
       7 | 837288 | gftptest-ppsgridftp-kit.gridka.de-1710230461.tmp
       7 | 837292 | gftptest-ppsgridftp-kit.gridka.de-1710231361.tmp
       7 | 837289 | httptest-ppswebdav-kit.gridka.de-1710230582.tmp
       7 | 841026 | httptest-ppswebdav-kit.gridka.de-1713051181.tmp
       7 | 385424 | readtest.dat
       7 | 837291 | srmtest-ppssrm-kit.gridka.de-1710230701.tmp
       7 | 704537 | test
       7 | 834672 | test-perf-marker-samuel
       7 | 834677 | test-perf-marker-samuel-1GB
       7 | 834683 | test-perf-marker-samuel-5GB
       7 | 838323 | test-samu
(11 rows)
  3. This is not possible with the NFS mounted namespace (thanks to POSIX?) 
    $ mkdir /pnfs/gridka.de/dteam/disk-only/top
$ mv /pnfs/gridka.de/dteam/disk-only/top/sub
mv: missing destination file operand after '/pnfs/gridka.de/dteam/disk-only/top/sub'
Try 'mv --help' for more information.

I wasn't able to find any safe-guard against cyclical directory structure in the code of JdbcFs. The rename operation is implemented through FsSqlDriver.rename(), which sets the new parent id and iname, but has no sanity checks beyond counting the number of rows changed by the SQL statement (if more than one, then raise JdbcUpdateAffectedIncorrectNumberOfRowsException).

Even though the user did something he/she didn't anticipate the consequences of, it is still a change that dCache - in my humble opinion - must prevent. Considering the possible consequences - depending on the access level of the client, the entire namespace can be destroyed by renaming the highest directory - this should be fixed with high priority!

Kind regards,
Xavier.

/cc @samuambroj

XMol commented 2 months ago

On a side note, this exact problem has occurred in the past, which we reported through dCache RT #9964. This time, we know what the user did, which is why analysis went so much faster and we can tell how to prevent it going forward.

XMol commented 2 months ago

Thinking about this challenge for some hours, I thought one could forego the check for cyclic relations in t_dirs if rename started a transaction and validates that the updated ichild still has the root directory as an ancestor before committing.

The benefit of the cyclic-check however would be, that it may be reused whenever (soft) links are created.

Tagging @DmitryLitvintsev, since he also offered guidance on our past issue.

paulmillar commented 2 months ago

Just to add, I was able to reproduce this problem using system-test

I added xrootd.authz.user = 1000:1000,0 to the unauthenticated xrootd door:

[dCacheDomain/xrootd]
xrootd.cell.name=Xrootd-${host.name}
xrootd.net.port=1094
xrootd.authz.write-paths=/
xrootd.authz.user = 1000:1000,0

With this configuration, I'm able to use xrdfs without GSI authentication.

paul@celebrimbor:~/git/dCache (master)$ xrdfs localhost ls -l /public
paul@celebrimbor:~/git/dCache (master)$ xrdfs localhost mkdir /public/test-1
paul@celebrimbor:~/git/dCache (master)$ xrdfs localhost ls -l /public
drwx 2024-04-26 09:46:05         512 /public/test-1
paul@celebrimbor:~/git/dCache (master)$ xrdfs localhost mv /public/test-1{,/sub}
paul@celebrimbor:~/git/dCache (master)$ echo $?
0
paul@celebrimbor:~/git/dCache (master)$ xrdfs localhost ls -l /public
paul@celebrimbor:~/git/dCache (master)$

Here is the offending command in the xrootd's access log file:

level=INFO ts=2024-04-26T11:46:14.051+0200 event=org.dcache.xrootd.request session=door:Xrootd-celebrimbor@dCacheDomain:AAYW/MY6hvA request=mv source=/public/test-1 target=/public/test-1/sub response=ok

Here are the corresponding message delivery events for this interaction:

paul@celebrimbor:~/git/dCache (master)$ grep AAYW/MY6hvA packages/system-test/target/dcache/var/log/dCacheDomain.events 
ts=2024-04-26T11:46:14.041+0200 event=org.dcache.cells.send.begin uoid=<-3801126463757269391:-7769129997317332006> lastuoid=<-3801126463757269391:-7769129997317332006> session=door:Xrootd-celebrimbor@dCacheDomain:AAYW/MY6hvA mode=callback message=PnfsResolveSymlinksMessage source=[>Xrootd-celebrimbor@dCacheDomain] destination=[>PnfsManager@local]
ts=2024-04-26T11:46:14.042+0200 event=org.dcache.cells.queue.begin uoid=<-3801126463757269391:-7769129997317332006> lastuoid=<-3801126463757269391:-7769129997317332006> session=door:Xrootd-celebrimbor@dCacheDomain:AAYW/MY6hvA source=[>Xrootd-celebrimbor@dCacheDomain] destination=[>PnfsManager@local]
ts=2024-04-26T11:46:14.042+0200 event=org.dcache.cells.queue.end uoid=<-3801126463757269391:-7769129997317332006> session=door:Xrootd-celebrimbor@dCacheDomain:AAYW/MY6hvA
ts=2024-04-26T11:46:14.042+0200 event=org.dcache.cells.deliver.begin uoid=<-3801126463757269391:-7769129997317332006> lastuoid=<-3801126463757269391:-7769129997317332006> session=door:Xrootd-celebrimbor@dCacheDomain:AAYW/MY6hvA message=PnfsResolveSymlinksMessage source=[>Xrootd-celebrimbor@dCacheDomain] destination=[PnfsManager@local]
ts=2024-04-26T11:46:14.042+0200 event=org.dcache.cells.deliver.end uoid=<-3801126463757269391:-7769129997317332006> session=door:Xrootd-celebrimbor@dCacheDomain:AAYW/MY6hvA
ts=2024-04-26T11:46:14.043+0200 event=org.dcache.cells.send.begin uoid=<5471928183860774607:-8349757380756507862> lastuoid=<-3801126463757269391:-7769129997317332006> session=door:Xrootd-celebrimbor@dCacheDomain:AAYW/MY6hvA mode=async message=PnfsResolveSymlinksMessage source=[>PnfsManager@dCacheDomain] destination=[>Xrootd-celebrimbor@dCacheDomain]
ts=2024-04-26T11:46:14.043+0200 event=org.dcache.cells.send.end uoid=<-3801126463757269391:-7769129997317332006> session=door:Xrootd-celebrimbor@dCacheDomain:AAYW/MY6hvA
ts=2024-04-26T11:46:14.043+0200 event=org.dcache.cells.send.end uoid=<5471928183860774607:-8349757380756507862> session=door:Xrootd-celebrimbor@dCacheDomain:AAYW/MY6hvA
ts=2024-04-26T11:46:14.043+0200 event=org.dcache.cells.send.begin uoid=<-9181860208406345979:-6921212834270286819> lastuoid=<-9181860208406345979:-6921212834270286819> session=door:Xrootd-celebrimbor@dCacheDomain:AAYW/MY6hvA mode=callback message=PnfsResolveSymlinksMessage source=[>Xrootd-celebrimbor@dCacheDomain] destination=[>PnfsManager@local]
ts=2024-04-26T11:46:14.043+0200 event=org.dcache.cells.queue.begin uoid=<-9181860208406345979:-6921212834270286819> lastuoid=<-9181860208406345979:-6921212834270286819> session=door:Xrootd-celebrimbor@dCacheDomain:AAYW/MY6hvA source=[>Xrootd-celebrimbor@dCacheDomain] destination=[>PnfsManager@local]
ts=2024-04-26T11:46:14.043+0200 event=org.dcache.cells.queue.end uoid=<-9181860208406345979:-6921212834270286819> session=door:Xrootd-celebrimbor@dCacheDomain:AAYW/MY6hvA
ts=2024-04-26T11:46:14.043+0200 event=org.dcache.cells.deliver.begin uoid=<-9181860208406345979:-6921212834270286819> lastuoid=<-9181860208406345979:-6921212834270286819> session=door:Xrootd-celebrimbor@dCacheDomain:AAYW/MY6hvA message=PnfsResolveSymlinksMessage source=[>Xrootd-celebrimbor@dCacheDomain] destination=[PnfsManager@local]
ts=2024-04-26T11:46:14.043+0200 event=org.dcache.cells.deliver.end uoid=<-9181860208406345979:-6921212834270286819> session=door:Xrootd-celebrimbor@dCacheDomain:AAYW/MY6hvA
ts=2024-04-26T11:46:14.044+0200 event=org.dcache.cells.send.begin uoid=<-3370225016636224167:-4733538726325168102> lastuoid=<-9181860208406345979:-6921212834270286819> session=door:Xrootd-celebrimbor@dCacheDomain:AAYW/MY6hvA mode=async message=PnfsResolveSymlinksMessage source=[>PnfsManager@dCacheDomain] destination=[>Xrootd-celebrimbor@dCacheDomain]
ts=2024-04-26T11:46:14.044+0200 event=org.dcache.cells.send.end uoid=<-9181860208406345979:-6921212834270286819> session=door:Xrootd-celebrimbor@dCacheDomain:AAYW/MY6hvA
ts=2024-04-26T11:46:14.044+0200 event=org.dcache.cells.send.end uoid=<-3370225016636224167:-4733538726325168102> session=door:Xrootd-celebrimbor@dCacheDomain:AAYW/MY6hvA
ts=2024-04-26T11:46:14.044+0200 event=org.dcache.cells.send.begin uoid=<-5610325841158912743:-8695396045043399668> lastuoid=<-5610325841158912743:-8695396045043399668> session=door:Xrootd-celebrimbor@dCacheDomain:AAYW/MY6hvA mode=callback message=PnfsRenameMessage source=[>Xrootd-celebrimbor@dCacheDomain] destination=[>PnfsManager@local]
ts=2024-04-26T11:46:14.046+0200 event=org.dcache.cells.queue.begin uoid=<-5610325841158912743:-8695396045043399668> lastuoid=<-5610325841158912743:-8695396045043399668> session=door:Xrootd-celebrimbor@dCacheDomain:AAYW/MY6hvA source=[>Xrootd-celebrimbor@dCacheDomain] destination=[>PnfsManager@local]
ts=2024-04-26T11:46:14.046+0200 event=org.dcache.cells.queue.end uoid=<-5610325841158912743:-8695396045043399668> session=door:Xrootd-celebrimbor@dCacheDomain:AAYW/MY6hvA
ts=2024-04-26T11:46:14.046+0200 event=org.dcache.cells.deliver.begin uoid=<-5610325841158912743:-8695396045043399668> lastuoid=<-5610325841158912743:-8695396045043399668> session=door:Xrootd-celebrimbor@dCacheDomain:AAYW/MY6hvA message=PnfsRenameMessage source=[>Xrootd-celebrimbor@dCacheDomain] destination=[PnfsManager@local]
ts=2024-04-26T11:46:14.046+0200 event=org.dcache.cells.deliver.end uoid=<-5610325841158912743:-8695396045043399668> session=door:Xrootd-celebrimbor@dCacheDomain:AAYW/MY6hvA
ts=2024-04-26T11:46:14.051+0200 event=org.dcache.cells.send.begin uoid=<7621137282627881479:-5777959526666910572> lastuoid=<-5610325841158912743:-8695396045043399668> session=door:Xrootd-celebrimbor@dCacheDomain:AAYW/MY6hvA mode=async message=PnfsRenameMessage source=[>PnfsManager@dCacheDomain] destination=[>Xrootd-celebrimbor@dCacheDomain]
ts=2024-04-26T11:46:14.051+0200 event=org.dcache.cells.send.end uoid=<-5610325841158912743:-8695396045043399668> session=door:Xrootd-celebrimbor@dCacheDomain:AAYW/MY6hvA
ts=2024-04-26T11:46:14.051+0200 event=org.dcache.cells.send.end uoid=<7621137282627881479:-5777959526666910572> session=door:Xrootd-celebrimbor@dCacheDomain:AAYW/MY6hvA

Boiling down to the messages, this is

paul@celebrimbor:~/git/dCache (master)$ grep AAYW/MY6hvA packages/system-test/target/dcache/var/log/dCacheDomain.events | grep event=org.dcache.cells.deliver.begin | sed -n 's/.* \(message=[^ ]*\) .*/\1/p'
message=PnfsResolveSymlinksMessage
message=PnfsResolveSymlinksMessage
message=PnfsRenameMessage
paul@celebrimbor:~/git/dCache (master)$

Ignoring the PnfsResolveSymlinksMessage messages, there is a single PnfsRenameMessage is causing this problem.

DmitryLitvintsev commented 2 months ago

@XMol Question, this:

xrdfs ppsdcache-kit.gridka.de mv /pnfs/gridka.de/dteam/disk-only/top{,/sub}

Is the same as calling:

xrdfs ppsdcache-kit.gridka.de mv /pnfs/gridka.de/dteam/disk-only/top  /pnfs/gridka.de/dteam/disk-only/top/sub

And xrdfs failing to safeguard? (I am just niot used to the syntax)

# mv some5 some5/sub 
mv: cannot move ‘some5’ to a subdirectory of itself, ‘some5/sub’

Is this correct?

P.S.: yes, looks like it:

$ xrdfs root://fndcaitb4.fnal.gov mv  /pnfs/fnal.gov/usr/test/litvinse/some5  /pnfs/fnal.gov/usr/test/litvinse/some5/sub 
$ echo $? 
0
$ xrdfs root://fndcaitb4.fnal.gov ls   /pnfs/fnal.gov/usr/test/litvinse/some5/sub
[ERROR] Server responded with an error: [3011] No such file
DmitryLitvintsev commented 2 months ago

yeah, and the same can be done with FTP:

$ uberftp  -rename gsiftp://fndcaitb4.fnal.gov/pnfs/fnal.gov/usr/test/litvinse/some4  /pnfs/fnal.gov/usr/test/litvinse/some4/sub 
$ echo $? 
0
paulmillar commented 2 months ago

Just to add a few comments:

First, (in case this is in doubt) the behaviour is a bug.

Second, the bug is in Chimera.

One of the design decisions for Chimera was that the database should enforce the filesystem consistency. When following this policy, it isn't surprising that there's no protection against this problem in the doors. There should also be no need for protection in PnfsManager itself or the PnfsManager's adapter to support Chimera (ChimeraNameSpaceProvider).

Instead, Chimera itself should have rejected the requested change coming from a door.

XMol commented 2 months ago

I agree with Paul's comments. Especially that a fix should be done for Chimera, rather than the individual doors, which is why I was looking for something in that regards in dCache's database interface code.