search

cern-fts / gfal2-python

python2 and 3 bindings for gfal2
https://dmc-docs.web.cern.ch/dmc-docs/gfal2-python.html
Apache License 2.0
5 stars 3 forks source link

Human readable mapping between GError codes and messages #3

Open todor-ivanov opened 2 years ago

todor-ivanov commented 2 years ago

Dear Experts,

In one of our cleanup services we are using gfal2-python for remote deletions at the sites. It is working well, but we have troubles accounting the errors. And more precisely: we are doing ctx.unlink([list of pfns]) and once we have the returned list of GErrors we iterate through it and try to preserve them in an external database. The database we use is MongoDB. And as simple it sounds it still leads us to a problem with inserting the records in the external database, because some of the error messages are having way too complicate format to be used as key values in the database. Which is kind of a blocker for us. Here we have some more details about the errors we are fighting with: [1]. There could be plenty of solutions to the problem and one way out of the situation is a well defined mapping between error codes and human readable strings/variables, enums etc... anything of the sort would do. Do you already have such mapping or a place (eventually in the documentation or in a header file), where we can take a look and create it ourselves?

Thanks in advance!

[1] https://github.com/dmwm/WMCore/pull/10687#issuecomment-1017986865

todor-ivanov commented 2 years ago

As a current solution we are mapping them to the possix error codes. So far this works well. But we need a confirmation from you that this is actually the best approach. And those error codes do indeed follow the possix standard.

But since gfal supports plenty of protocols we need to double check with you. As an example, this GError (message && code) seems to be having its origins from gsiftp :

 GError('globus_ftp_client: the server responded with an error 451 451-GlobusError: v=1 c=INTERNAL_ERROR  451-GridFTP-Errno: 5  451-GridFTP-Reason: System error in unlink (host=g27n19.hep.wi
sc.edu, user=cmsprod, path=/store/unmerged/RunIISummer20UL16RECOAPV/ST_tW_Dilept_5f_DR_TuneCP5_13TeV-amcatnlo-pythia8/AODSIM/106X_mcRun2_asymptotic_preVFP_v8-v2/2550000/EF9A4B74-9D49-EA47-B3
0E-8225B6E45E74.root)  451-GridFTP-Error-String: Input/output error  451 End.   ', 70)

and if we try to map it to the possix error code we get:

In [61]: os.strerror(70)
Out[61]: 'Communication error on send'

Which is more or less similar, but actually the most precise interpretation we get when we try to use the gfal-rm tool itself:

$ gfal-rm gsiftp://cms-lvs-gridftp.hep.wisc.edu:2811/hdfs/store/unmerged/RunIISummer20UL16RECOAPV/ST_tW_Dilept_5f_DR_TuneCP5_13TeV-amcatnlo-pythia8/AODSIM/106X_mcRun2_asymptotic_preVFP_v8-v2/2550000/EF9A4B74-9D49-EA47-B30E-8225B6E45E74.root
gsiftp://cms-lvs-gridftp.hep.wisc.edu:2811/hdfs/store/unmerged/RunIISummer20UL16RECOAPV/ST_tW_Dilept_5f_DR_TuneCP5_13TeV-amcatnlo-pythia8/AODSIM/106X_mcRun2_asymptotic_preVFP_v8-v2/2550000/EF9A4B74-9D49-EA47-B30E-8225B6E45E74.root  MISSING

So it leads to the impression there is an intermediate layer for purifying things by including the protocol in use into the picture or something else. Any clarification would be of great help.

Thanks again!