Tune error output

[bbockelm]

When stashcp has errors, we get a really generic "All methods failed" -- but when you enable debug output, you get all debug output. See below:

[bbockelm@login04 osg-flock]$ ./stashcp/stashcp /osgconnect/public/dweitzel/stashcp/test.file.2 /tmp/baz
ERROR[2021-09-22T15:59:35-05:00] All methods failed! Unable to download file. 
[bbockelm@login04 osg-flock]$ ./stashcp/stashcp --debug /osgconnect/public/dweitzel/stashcp/test.file.2 /tmp/baz
DEBUG[2021-09-22T16:00:14-05:00] Len of source: 2                             
DEBUG[2021-09-22T16:00:14-05:00] Sources: [/osgconnect/public/dweitzel/stashcp/test.file.2] 
DEBUG[2021-09-22T16:00:14-05:00] Destination: /tmp/baz                        
INFO[2021-09-22T16:00:14-05:00] Trying CVMFS...                              
DEBUG[2021-09-22T16:00:14-05:00] Checking if the CVMFS file exists: /cvmfs/stash.osgstorage.org/osgconnect/public/dweitzel/stashcp/test.file.2 
DEBUG[2021-09-22T16:00:14-05:00] CVMFS File does not exist                    
INFO[2021-09-22T16:00:14-05:00] Trying XROOTD...                             
DEBUG[2021-09-22T16:00:14-05:00] Running the command to check of xrdcp existance: xrdcp -V 2>&1 
DEBUG[2021-09-22T16:00:14-05:00] exec: "xrdcp -V 2>&1": executable file not found in $PATH 
INFO[2021-09-22T16:00:14-05:00] Trying HTTP...                               
DEBUG[2021-09-22T16:00:14-05:00] Trying server site of wlcg-wpad.fnal.gov     
DEBUG[2021-09-22T16:00:14-05:00] Querying http://131.225.188.246/stashservers.dat 
DEBUG[2021-09-22T16:00:14-05:00] Got OK code 200 from wlcg-wpad.fnal.gov      
DEBUG[2021-09-22T16:00:14-05:00] Ordered list of caches: [5 2 7 4 8 3 6 10 1 11 13 12 15 14 9] 
DEBUG[2021-09-22T16:00:14-05:00] Hashed: eadeb0d44329927064b3b6c214854cbb2bbd96c7 From CVMFS: eadeb0d44329927064b3b6c214854cbb2bbd96c7 
DEBUG[2021-09-22T16:00:14-05:00] Got a *rsa.PublicKey, with remaining data: "\n" 
DEBUG[2021-09-22T16:00:14-05:00] Signature Matched                            
DEBUG[2021-09-22T16:00:14-05:00] Cache list: xroot=stashcache.t2.ucsd.edu,mwt2-stashcache.campuscluster.illinois.edu,its-condor-xrootd1.syr.edu,osg-kansas-city-stashcache.nrp.internet2.edu,osg-chicago-stashcache.nrp.internet2.edu,osg-new-york-stashcache.nrp.internet2.edu,stash-cache.osg.chtc.io,osg-gftp2.pace.gatech.edu,dtn2-daejeon.kreonet.net,osg-houston-stashcache.nrp.internet2.edu,osg-sunnyvale-stashcache.nrp.internet2.edu,fiona.uvalight.net,stashcache.gravity.cf.ac.uk,xcache.cr.cnaf.infn.it,xcachevirgo.pic.es;xroots=xrootd-local.unl.edu,stashcache.t2.ucsd.edu,its-condor-xrootd1.syr.edu,osg-kansas-city-stashcache.nrp.internet2.edu,osg-chicago-stashcache.nrp.internet2.edu,osg-new-york-stashcache.nrp.internet2.edu,osg-gftp2.pace.gatech.edu,dtn2-daejeon.kreonet.net,osg-houston-stashcache.nrp.internet2.edu,osg-sunnyvale-stashcache.nrp.internet2.edu,ligo.hpc.swin.edu.au,fiona.uvalight.net,stashcache.gravity.cf.ac.uk,xcache.cr.cnaf.infn.it,xcachevirgo.pic.es;cms=xrootd.t2.ucsd.edu,cms-xcache.rcac.purdue.edu,red-xcache1.unl.edu;hash=-sha1 
DEBUG[2021-09-22T16:00:14-05:00] Cache list: [stashcache.t2.ucsd.edu mwt2-stashcache.campuscluster.illinois.edu its-condor-xrootd1.syr.edu osg-kansas-city-stashcache.nrp.internet2.edu osg-chicago-stashcache.nrp.internet2.edu osg-new-york-stashcache.nrp.internet2.edu stash-cache.osg.chtc.io osg-gftp2.pace.gatech.edu dtn2-daejeon.kreonet.net osg-houston-stashcache.nrp.internet2.edu osg-sunnyvale-stashcache.nrp.internet2.edu fiona.uvalight.net stashcache.gravity.cf.ac.uk xcache.cr.cnaf.infn.it xcachevirgo.pic.es] 
DEBUG[2021-09-22T16:00:14-05:00] Closest cache: root://osg-chicago-stashcache.nrp.internet2.edu 
DEBUG[2021-09-22T16:00:14-05:00] Returning closest cache: root://osg-chicago-stashcache.nrp.internet2.edu 
DEBUG[2021-09-22T16:00:14-05:00] Ordered list of nearest caches: [root://osg-chicago-stashcache.nrp.internet2.edu root://mwt2-stashcache.campuscluster.illinois.edu root://stash-cache.osg.chtc.io root://osg-kansas-city-stashcache.nrp.internet2.edu root://osg-gftp2.pace.gatech.edu root://its-condor-xrootd1.syr.edu root://osg-new-york-stashcache.nrp.internet2.edu root://osg-houston-stashcache.nrp.internet2.edu root://stashcache.t2.ucsd.edu root://osg-sunnyvale-stashcache.nrp.internet2.edu root://stashcache.gravity.cf.ac.uk root://fiona.uvalight.net root://xcachevirgo.pic.es root://xcache.cr.cnaf.infn.it root://dtn2-daejeon.kreonet.net] 
DEBUG[2021-09-22T16:00:14-05:00] Trying the caches: [root://osg-chicago-stashcache.nrp.internet2.edu root://mwt2-stashcache.campuscluster.illinois.edu root://stash-cache.osg.chtc.io] 
DEBUG[2021-09-22T16:00:14-05:00] Constructed URL: http://osg-chicago-stashcache.nrp.internet2.edu:8000/osgconnect/public/dweitzel/stashcp/test.file.2 
DEBUG[2021-09-22T16:00:15-05:00] Got error from HTTP download server returned 404 Not Found 
DEBUG[2021-09-22T16:00:15-05:00] Error downloading from HTTP: server returned 404 Not Found 
DEBUG[2021-09-22T16:00:15-05:00] Constructed URL: http://mwt2-stashcache.campuscluster.illinois.edu:8000/osgconnect/public/dweitzel/stashcp/test.file.2 
DEBUG[2021-09-22T16:02:23-05:00] Error downloading from HTTP: Cancelled transfer, too slow 
DEBUG[2021-09-22T16:02:23-05:00] Constructed URL: http://stash-cache.osg.chtc.io:8000/osgconnect/public/dweitzel/stashcp/test.file.2 
DEBUG[2021-09-22T16:02:23-05:00] Got error from HTTP download server returned 404 Not Found 
DEBUG[2021-09-22T16:02:23-05:00] Error downloading from HTTP: server returned 404 Not Found 
DEBUG[2021-09-22T16:02:23-05:00] Failed to download with HTTP                 
ERROR[2021-09-22T16:02:23-05:00] All methods failed! Unable to download file. 
DEBUG[2021-09-22T16:02:23-05:00] Returned from collector.atlas-ml.org: ok     

Can we pick through the above messages and escalate a few to reasonable error levels such that we could give them to users?

[djw8605]

Almost all errors are recoverable. It's not until we try all methods do we know it will ultimately fail. I'm open to suggestions. Should we send error messages to a user if it fails an HTTP download?

[djw8605]

This is fixed in #13 and released in 6.4.0