DIRACGrid / DIRAC

DIRAC Grid
http://diracgrid.org
GNU General Public License v3.0
114 stars 176 forks source link

Is it possible to add a last chance option for file transfers via FTS ? #7680

Closed marianne013 closed 1 month ago

marianne013 commented 4 months ago

I'm currently seeing the following problem: An FTS transfer fails:

(base) lx05:move_moedal > dirac-rms-request 800512 | grep /vo.moedal.org/user/a/aditya.upreti/BeamPipe/gen/SchGen_GRID_80GeV_50gD_5000ev_10611_May20.mac
    [469] ID=7552636 LFN='/vo.moedal.org/user/a/aditya.upreti/BeamPipe/gen/SchGen_GRID_80GeV_50gD_5000ev_10611_May20.mac' Status='Failed' Checksum='2e4b1535'

looking at the log file on the FTS server:

https://fts01.grid.hep.ph.ic.ac.uk:8449/var/log/fts3/transfers/2024-06-17/se03.esc.qmul.ac.uk__gfe02.grid.hep.ph.ic.ac.uk/2024-06-17-1324__se03.esc.qmul.ac.uk__gfe02.grid.hep.ph.ic.ac.uk__52380956__33752f0c-2ca9-11ef-8adf-525400e091e6

It ends in:

INFO    Mon, 17 Jun 2024 14:24:02 +0100; DESTINATION Source and destination file size matching
INFO    Mon, 17 Jun 2024 14:24:02 +0100; Transfer finished successfully
INFO    Mon, 17 Jun 2024 14:24:07 +0100; Finishing copy process...

But the "global" message is:

Error reason: No FTS server has updated the transfer status the last 900 seconds. Probably stalled

Now the file is at Imperial (correct size and all):

[lt2-moedal001@gfe02 moedal]$ pwd
/pnfs/hep.ph.ic.ac.uk/data/moedal
[lt2-moedal001@gfe02 moedal]$ ls -l vo.moedal.org/user/a/aditya.upreti/BeamPipe/gen/SchGen_GRID_80GeV_50gD_5000ev_10611_May20.mac
-rw-r-----. 1 lt2-moedal001 lt2-moedal 19758 Jun 17 15:50 vo.moedal.org/user/a/aditya.upreti/BeamPipe/gen/SchGen_GRID_80GeV_50gD_5000ev_10611_May20.mac

But not registered in the file catalogue:

(base) lx05:move_moedal > dirac-dms-lfn-replicas /vo.moedal.org/user/a/aditya.upreti/BeamPipe/gen/SchGen_GRID_80GeV_50gD_5000ev_10611_May20.mac
Replica Lookup Time: 3.65 seconds 
LFN                                                                                            StorageElement     URL
=====================================================================================================================
/vo.moedal.org/user/a/aditya.upreti/BeamPipe/gen/SchGen_GRID_80GeV_50gD_5000ev_10611_May20.mac UKI-LT2-QMUL2-disk srm://se03.esc.qmul.ac.uk:8444/srm/managerv2?SFN=/vo.moedal.org/vo.moedal.org/user/a/aditya.upreti/BeamPipe/gen/SchGen_GRID_80GeV_50gD_5000ev_10611_May20.mac

Is there any way for the transfer software to do one last ditch check to see if the file has actually transferred and the register it ?

Probably a question for @chaen and and FYI for @sfayer

chaen commented 4 months ago

As discussed live: