Phedex api crashes when chain=direct is specified in pfn-to-lfn element of the tfc file

[DMWMBot]

Here is the script to test this:

---

from WMCore.Services.PhEDEx.PhEDEx import PhEDEx from WMCore.Storage.TrivialFileCatalog import readTFC

phedexApi = PhEDEx( responseType='xml') sites = ['T2_AT_Vienna', 'T2_BE_IIHE', 'T2_BE_UCL', 'T2_BR_SPRACE', 'T2_BR_UERJ', 'T2_CH_CAF', 'T2_CH_CSCS', 'T2_CN_Beijing', 'T2_DE_DESY', 'T2_DE_RWTH', 'T2_EE_Estonia', 'T2_ES_CIEMAT', 'T2_ES_IFCA', 'T2_FI_HIP', 'T2_FR_CCIN2P3', 'T2_FR_GRIF_IRFU', 'T2_FR_GRIF_LLR', 'T2_FR_IPHC', 'T2_HU_Budapest', 'T2_IN_TIFR', 'T2_IT_Bari', 'T2_IT_Legnaro', 'T2_IT_Pisa', 'T2_IT_Rome', 'T2_KR_KNU', 'T2_PK_NCP', 'T2_PL_Cracow', 'T2_PL_Warsaw', 'T2_PT_LIP_Lisbon', 'T2_PT_NCG_Lisbon', 'T2_RU_IHEP', 'T2_RU_INR', 'T2_RU_ITEP', 'T2_RU_JINR', 'T2_RU_PNPI', 'T2_RU_RRC_KI', 'T2_RU_SINP', 'T2_TR_METU', 'T2_TW_Taiwan', 'T2_UA_KIPT', 'T2_UK_London_Brunel', 'T2_UK_London_IC', 'T2_UK_SGrid_Bristol', 'T2_UK_SGrid_RALPP', 'T2_US_Caltech', 'T2_US_Florida', 'T2_US_MIT', 'T2_US_Nebraska', 'T2_US_Purdue', 'T2_US_UCSD', 'T2_US_Wisconsin']

for node in sites:

tfc_file = phedexApi.cacheFileName('tfc', inputdata={'node': node})
tfc = readTFC(tfc_file)
print tfc
lfn = '/store/user'
print "orig lfn %s" %lfn
pfn = tfc.matchLFN('srmv2', lfn)
print "pfn is %s" %pfn
lfn = tfc.matchPFN('srmv2', pfn)
print "again lfn %s" %lfn

[sfoulkes]

sfoulkes: Can you turn this into a unit test?

[DMWMBot]

riahi: the unitest for phedex was crashing last week...let's try again now.

[ticoann]

sryu: Hi Hassen, Could you add stack trace for the errors? The way PhEDEx service encoding and decoding scheme has to be changed (instead response type it will use accept_type in request header - I will change that). But I am not sure whether that is related to this. Is there problem when you use json response type?

[DMWMBot]

riahi: Hi seanghcan,

Here is the stack trace:

orig lfn /store/user pfn is srm://maite.iihe.ac.be:8443/srm/managerv2?SFN=/pnfs/iihe/cms/ph/sc4/store/user srm://maite.iihe.ac.be:8443/srm/managerv2?SFN=/pnfs/iihe/cms/ph/sc4/store/user Traceback (most recent call last): File "ConvertSite.py", line 39, in lfn = tfc.matchPFN('srmv2', pfn) File "/home/crab/WMAgent/install/WMCORE/src/python/WMCore/Storage/TrivialFileCatalog.py", line 140, in matchPFN return self.doMatch(protocol, pfn, 'pfn-to-lfn') File "/home/crab/WMAgent/install/WMCORE/src/python/WMCore/Storage/TrivialFileCatalog.py", line 101, in doMatch splitPath = mapping['path-match'].split(path, 1)8e09c9da98331a5fcdbf0b1e9b52e7f2ba11a7cc TypeError: expected string or buffer

[ticoann]

sryu: Hi Hassen, What PhEDEx service call are you using? cacheFileName('tfc', inputdata={'node': node}) doesn't return data it only generates cacheFileName which probably empty if you didn't make any remote call. The only supported apis for PhEDEx REST calls are here. https://svnweb.cern.ch/trac/CMSDMWM/browser/WMCore/trunk/src/python/WMCore/Services/PhEDEx/PhEDEx.py So if you are not using getNodeTFC(node) function, try with that first. If you are using that function, could you print out the result of that function. I will further investigate.

[DMWMBot]

riahi: I'm using the getNodeTFC(node) method

Here is the result: <?xml version='1.0' encoding='ISO-8859-1'?>

Here the tfc: lfn-to-pfn: dcap /+(.) dcap://maite.iihe.ac.be/$1 chain=direct lfn-to-pfn: direct ./LoadTest07BelgiumIIHE(.).__.* /pnfs/iihe/cms/ph/sc4/LoadTest/LoadTest07_IIHEBelgium$1 lfn-to-pfn: direct /becms/(.) /pnfs/iihe/becms/$1 lfn-to-pfn: direct /store/user/(.) /pnfs/iihe/cms/store/user/$1 lfn-to-pfn: direct /store/caf/user/(.) /pnfs/iihe/cms/store/user/$1 lfn-to-pfn: direct /+(.) /pnfs/iihe/cms/ph/sc4/$1 lfn-to-pfn: srm /+(.) srm://maite.iihe.ac.be:8443/srm/managerv1?SFN=/$1 chain=direct lfn-to-pfn: srmv2 /+(.) srm://maite.iihe.ac.be:8443/srm/managerv2?SFN=/$1 chain=direct pfn-to-lfn: dcap dcap:/+maite.iihe.ac.be/(.) $1 chain=direct pfn-to-lfn: direct /+pnfs/iihe/becms/+(.) /becms/$1 pfn-to-lfn: direct /+pnfs/iihe/cms/store/user/+(.) /store/user/$1 pfn-to-lfn: direct /+pnfs/iihe/cms/ph/sc4/+(.) /$1 pfn-to-lfn: srm .\?SFN=(.) $1 chain=direct pfn-to-lfn: srmv2 .\?SFN=(.) $1 chain=direct

and here is the error:

orig lfn /store/user pfn is srm://maite.iihe.ac.be:8443/srm/managerv2?SFN=/pnfs/iihe/cms/ph/sc4/store/user srm://maite.iihe.ac.be:8443/srm/managerv2?SFN=/pnfs/iihe/cms/ph/sc4/store/user Traceback (most recent call last): File "ConvertSite.py", line 40, in lfn = tfc.matchPFN('srmv2', pfn) File "/home/crab/WMAgent/install/WMCORE/src/python/WMCore/Storage/TrivialFileCatalog.py", line 140, in matchPFN return self.doMatch(protocol, pfn, 'pfn-to-lfn') File "/home/crab/WMAgent/install/WMCORE/src/python/WMCore/Storage/TrivialFileCatalog.py", line 101, in doMatch splitPath = mapping['path-match'].split(path, 1)8e09c9da98331a5fcdbf0b1e9b52e7f2ba11a7cc TypeError: expected string or buffer

[ticoann]

sryu: Steve, could you review this?

[DMWMBot]

riahi: Hi Seangchan,

You can find enclosed the unitests of phedex api where I add a new method to test getNodeNames. I don't know if the patch that you submitted fixes that too. This is the error (*) that I had when trying to call getNodeNames.

(*)

ERROR: testSiteLookup (main.PhEDExTest)

Traceback (most recent call last): File "PhEDEx_t.py", line 40, in testSiteLookup CMSsite = phedexApi.getNodeNames('storm-se-01.ba.infn.it') File "/home/crab/WMAgent/install/WMCORE/src/python/WMCore/Services/PhEDEx/PhEDEx.py", line 282, in getNodeNames nodeList = output['phedex']['node'] TypeError: string indices must be integers, not str

---

[ticoann]

sryu: Hi Hassen,

Actually getNodeNames is remote call, so you need to pass the endpoint when you instantiate PhEDEx service unless you want to test that against production server - which probably we shouldn't.

dict = {} dict['endpoint'] = self.phedexTestDS phedexApi = PhEDEx(dict)

responseType is irrelevant here. It will be json. (hard coded in the unit test). If you want xml type back, change 'json' to 'xml' from self.phedexTestDS value. You need also right cert to run the test and many need ssh tunnel. I update the instruction #648.

Let me know if you still have a trouble.