Closed JBlack93 closed 4 years ago
The right thing would be to check which protocol works and use that one instead of randomizing. Also, 1800s is a long-ish timeout for just checking for the existence of a file I would say...
Agreed.
The protocol is only used for gfal-ls
, so all the protocols currently used (xroot
, gsiftp
, srm
) should work unless there is a problem with the corresponding service at the time.
We could simply use the protocol used for the copy, which would cycle it as need be, and avoids an unnecessary import random
.
@JBlack93 - were you been running with the new copy log feature enabled - and if so, would you mind posting the log?
Unfortunately not (I'll ensure to have this enabled in future runs).
However, it was clear in this case, through the stderr
(and confirmations through sysadmins) that the GSIFTP service went down over the weekend, highlighting this particular issue.
Pity as that would be really useful.
Do you know if the gfal-ls
command exited with an error while the gsiftp
service was down?
From stderr
:
gfal-sum error: 110 (Connection timed out) - Operation timed out
gfal-sum error: 110 (Connection timed out) - Operation timed out
gfal-sum error: 110 (Connection timed out) - Operation timed out
gfal-ls error: 110 (Connection timed out) - Operation timed out
Command timed out after 1800 seconds!
gfal-ls error: 110 (Connection timed out) - Operation timed out
gfal-ls error: 110 (Connection timed out) - Operation timed out
gfal-ls error: 110 (Connection timed out) - Operation timed out
Command timed out after 1800 seconds!
gfal-ls error: 110 (Connection timed out) - Operation timed out
gfal-ls error: 110 (Connection timed out) - Operation timed out
gfal-ls error: 110 (Connection timed out) - Operation timed out
Command timed out after 1800 seconds!
gfal-ls error: 110 (Connection timed out) - Operation timed out
gfal-ls error: 110 (Connection timed out) - Operation timed out
gfal-ls error: 110 (Connection timed out) - Operation timed out
Command timed out after 1800 seconds!
gfal-ls error: 110 (Connection timed out) - Operation timed out
gfal-ls error: 110 (Connection timed out) - Operation timed out
gfal-ls error: 110 (Connection timed out) - Operation timed out
Command timed out after 1800 seconds!
gfal-ls error: 110 (Connection timed out) - Operation timed out
gfal-ls error: 110 (Connection timed out) - Operation timed out
gfal-ls error: 110 (Connection timed out) - Operation timed out
Command timed out after 1800 seconds!
gfal-ls error: 110 (Connection timed out) - Operation timed out
gfal-ls error: 110 (Connection timed out) - Operation timed out
gfal-ls error: 110 (Connection timed out) - Operation timed out
Command timed out after 1800 seconds!
gfal-ls error: 110 (Connection timed out) - Operation timed out
gfal-ls error: 110 (Connection timed out) - Operation timed out
gfal-ls error: 110 (Connection timed out) - Operation timed out
Command timed out after 1800 seconds!
gfal-ls error: 110 (Connection timed out) - Operation timed out
gfal-copy error: 70 (Communication error on send) - DESTINATION SRM_PUT_TURL srm-ifce err: Communication error on send, err: [SE][PrepareToPut][] httpg://se01.dur.scotgrid.ac.uk/srm/managerv2: CGSI-gSOAP running on n127.dur.scotgrid.ac.uk reports Error reading token data: Connection closed
gfal-ls error: 110 (Connection timed out) - Operation timed out
gfal-ls error: 110 (Connection timed out) - Operation timed out
Command timed out after 1800 seconds!
gfal-ls error: 110 (Connection timed out) - Operation timed out
gfal-ls error: 110 (Connection timed out) - Operation timed out
gfal-ls error: 110 (Connection timed out) - Operation timed out
Hi James, Marian and I have just had a look at this - is there further log output you chopped off the beginning? It'd be good to see the first errors that arose.
I only chopped the python version number: stderr.log stdout.log
Cheers @JBlack93 , it was the gfal-sum
errors at the top that threw me. @marianheil and I went through the both the standard error and the standard output and caught a few other possible bugs. Pull request #55 should fix them.
We have a dependence on gsiftp which results in large copy times and non-zero exit status if GSIFTP is down for any reason.
After
gfal-copy
cycles through appropriate protocols we check for a successful copy throughgfal-ls
. Currently this is hard coded to use thegsiftp
protocol. If the gsiftp server is down this causes a huge delay (currently we gfal-ls timeout is 1800s). This resulted in a "copy-time" of 1day 7hours for HEJ over this weekend.Two-fold solution:
test_file_presence()
This issue exists through all "run" files.