Closed DMWMBot closed 12 years ago
sfoulkes: Can you turn this into a unit test?
riahi: the unitest for phedex was crashing last week...let's try again now.
sryu: Hi Hassen, Could you add stack trace for the errors? The way PhEDEx service encoding and decoding scheme has to be changed (instead response type it will use accept_type in request header - I will change that). But I am not sure whether that is related to this. Is there problem when you use json response type?
riahi: Hi seanghcan,
Here is the stack trace:
orig lfn /store/user
pfn is srm://maite.iihe.ac.be:8443/srm/managerv2?SFN=/pnfs/iihe/cms/ph/sc4/store/user
srm://maite.iihe.ac.be:8443/srm/managerv2?SFN=/pnfs/iihe/cms/ph/sc4/store/user
Traceback (most recent call last):
File "ConvertSite.py", line 39, in
sryu: Hi Hassen, What PhEDEx service call are you using? cacheFileName('tfc', inputdata={'node': node}) doesn't return data it only generates cacheFileName which probably empty if you didn't make any remote call. The only supported apis for PhEDEx REST calls are here. https://svnweb.cern.ch/trac/CMSDMWM/browser/WMCore/trunk/src/python/WMCore/Services/PhEDEx/PhEDEx.py So if you are not using getNodeTFC(node) function, try with that first. If you are using that function, could you print out the result of that function. I will further investigate.
riahi: I'm using the getNodeTFC(node) method
Here is the result:
<?xml version='1.0' encoding='ISO-8859-1'?>
Here the tfc: lfn-to-pfn: dcap /+(.) dcap://maite.iihe.ac.be/$1 chain=direct lfn-to-pfn: direct ./LoadTest07BelgiumIIHE(.).__.* /pnfs/iihe/cms/ph/sc4/LoadTest/LoadTest07_IIHEBelgium$1 lfn-to-pfn: direct /becms/(.) /pnfs/iihe/becms/$1 lfn-to-pfn: direct /store/user/(.) /pnfs/iihe/cms/store/user/$1 lfn-to-pfn: direct /store/caf/user/(.) /pnfs/iihe/cms/store/user/$1 lfn-to-pfn: direct /+(.) /pnfs/iihe/cms/ph/sc4/$1 lfn-to-pfn: srm /+(.) srm://maite.iihe.ac.be:8443/srm/managerv1?SFN=/$1 chain=direct lfn-to-pfn: srmv2 /+(.) srm://maite.iihe.ac.be:8443/srm/managerv2?SFN=/$1 chain=direct pfn-to-lfn: dcap dcap:/+maite.iihe.ac.be/(.) $1 chain=direct pfn-to-lfn: direct /+pnfs/iihe/becms/+(.) /becms/$1 pfn-to-lfn: direct /+pnfs/iihe/cms/store/user/+(.) /store/user/$1 pfn-to-lfn: direct /+pnfs/iihe/cms/ph/sc4/+(.) /$1 pfn-to-lfn: srm .\?SFN=(.) $1 chain=direct pfn-to-lfn: srmv2 .\?SFN=(.) $1 chain=direct
and here is the error:
orig lfn /store/user
pfn is srm://maite.iihe.ac.be:8443/srm/managerv2?SFN=/pnfs/iihe/cms/ph/sc4/store/user
srm://maite.iihe.ac.be:8443/srm/managerv2?SFN=/pnfs/iihe/cms/ph/sc4/store/user
Traceback (most recent call last):
File "ConvertSite.py", line 40, in
sryu: Steve, could you review this?
riahi: Hi Seangchan,
You can find enclosed the unitests of phedex api where I add a new method to test getNodeNames. I don't know if the patch that you submitted fixes that too. This is the error (*) that I had when trying to call getNodeNames.
(*)
Traceback (most recent call last): File "PhEDEx_t.py", line 40, in testSiteLookup CMSsite = phedexApi.getNodeNames('storm-se-01.ba.infn.it') File "/home/crab/WMAgent/install/WMCORE/src/python/WMCore/Services/PhEDEx/PhEDEx.py", line 282, in getNodeNames nodeList = output['phedex']['node'] TypeError: string indices must be integers, not str
sryu: Hi Hassen,
Actually getNodeNames is remote call, so you need to pass the endpoint when you instantiate PhEDEx service unless you want to test that against production server - which probably we shouldn't.
dict = {} dict['endpoint'] = self.phedexTestDS phedexApi = PhEDEx(dict)
responseType is irrelevant here. It will be json. (hard coded in the unit test). If you want xml type back, change 'json' to 'xml' from self.phedexTestDS value. You need also right cert to run the test and many need ssh tunnel. I update the instruction #648.
Let me know if you still have a trouble.
Here is the script to test this:
from WMCore.Services.PhEDEx.PhEDEx import PhEDEx from WMCore.Storage.TrivialFileCatalog import readTFC
phedexApi = PhEDEx( responseType='xml') sites = ['T2_AT_Vienna', 'T2_BE_IIHE', 'T2_BE_UCL', 'T2_BR_SPRACE', 'T2_BR_UERJ', 'T2_CH_CAF', 'T2_CH_CSCS', 'T2_CN_Beijing', 'T2_DE_DESY', 'T2_DE_RWTH', 'T2_EE_Estonia', 'T2_ES_CIEMAT', 'T2_ES_IFCA', 'T2_FI_HIP', 'T2_FR_CCIN2P3', 'T2_FR_GRIF_IRFU', 'T2_FR_GRIF_LLR', 'T2_FR_IPHC', 'T2_HU_Budapest', 'T2_IN_TIFR', 'T2_IT_Bari', 'T2_IT_Legnaro', 'T2_IT_Pisa', 'T2_IT_Rome', 'T2_KR_KNU', 'T2_PK_NCP', 'T2_PL_Cracow', 'T2_PL_Warsaw', 'T2_PT_LIP_Lisbon', 'T2_PT_NCG_Lisbon', 'T2_RU_IHEP', 'T2_RU_INR', 'T2_RU_ITEP', 'T2_RU_JINR', 'T2_RU_PNPI', 'T2_RU_RRC_KI', 'T2_RU_SINP', 'T2_TR_METU', 'T2_TW_Taiwan', 'T2_UA_KIPT', 'T2_UK_London_Brunel', 'T2_UK_London_IC', 'T2_UK_SGrid_Bristol', 'T2_UK_SGrid_RALPP', 'T2_US_Caltech', 'T2_US_Florida', 'T2_US_MIT', 'T2_US_Nebraska', 'T2_US_Purdue', 'T2_US_UCSD', 'T2_US_Wisconsin']
for node in sites: