Dataservice "tfc" call returning the path for a given lfn at a given node

ericvaandering commented 11 years ago

Original Savannah ticket 37025 reported by egeland on Tue May 27 11:33:54 2008.

I'd like to request that the "tfc" call of dataservice :

https://cmsweb.cern.ch/phedex/datasvc/xml/prod/tfc?node=T2_XY_ZZZ

could return the path for a given lfn at a given node. So with:

inputs: node, lfn output: full path endpoint

something like what Brian's tool provide (*).

thanks, Ale

(*) Brian's tool: For example: http://t2.unl.edu/phedex/tfc/map/T2_IT_Legnaro?lfn=/store/user/afanfani&protocol=srmv2 return srm://t2-srm-02.lnl.infn.it:8443/srm/managerv2?SFN=/pnfs/lnl.infn.it/data/cms/store/user/afanfani

ericvaandering commented 11 years ago

Comment by egeland on Tue Jun 24 04:38:06 2008

First version done, awaiting testing feedback.

ericvaandering commented 11 years ago

Comment by afanfani on Tue Jun 24 09:26:27 2008

I'm testing it. It's seems ok for CRAB usecase. A couple of comments:

1) asking for a not existing protocol, i.e. protocol=whatever https://cmsweb.cern.ch/phedex/test/datasvc/xml/prod/lfn2pfn?lfn=/store/user/afanfani&node=T1_CH_CERN_Buffer&protocol=whatever there is no error message and the returned pfn is empty. I guess that's because you don't have a list of valid protocols like for the node names. Anyway it's fine to check in CRAB that the returned pfn is not empty. I might need to check that "srmv2" gives correct results for all the sites.

2) The url : https://cmsweb.cern.ch/phedex/test/datasvc/xml/prod/lfn2pfn is a "test" url that will be replaced with the real one once the testing phase is over?

thanks, Ale

ericvaandering commented 11 years ago

Comment by afanfani on Tue Jun 24 15:25:13 2008

I tried to use the lfn2tfc in a python script /afs/cern.ch/user/a/afanfani/public/CRAB/PhEDExDatasvc/getPFN.py , that can be made part of CRAB, with all the T2 names from siteDB and in most of the cases I get back a PFN. There are few problematic sites in a) and b) but that's not fault of the call. ciao, Ale

a) two sites are not known with the name I provided: Error message: Error when making call 'lfn2pfn': unknown node 'T2_FR_CCIN2P3' Error message: Error when making call 'lfn2pfn': unknown node 'T2_RU_IHEP'

b) for four sites I could not get the PFN

Unable to get stageout path (PFN) from https://cmsweb.cern.ch/phedex/test/datasvc/xml/prod/lf n2pfn?node=T2_FR_IPHC&lfn=/store/user/&protocol=srmv2

Unable to get stageout path (PFN) from https://cmsweb.cern.ch/phedex/test/datasvc/xml/prod/lf n2pfn?node=T2_RU_INR&lfn=/store/user/&protocol=srmv2

Unable to get stageout path (PFN) from https://cmsweb.cern.ch/phedex/test/datasvc/xml/prod/lf n2pfn?node=T2_RU_PNPI&lfn=/store/user/&protocol=srmv2

Unable to get stageout path (PFN) from https://cmsweb.cern.ch/phedex/test/datasvc/xml/prod/lf n2pfn?node=T2_TR_ULAKBIM&lfn=/store/user/&protocol=srmv2

ericvaandering commented 11 years ago

Comment by afanfani on Wed Jun 25 07:16:39 2008

With CRAB-like call as in CRAB/python/PhEDExDatasvcInfo.py I've tested it for all the T2 nodes and srmv2 protocol, extracting the information and then trying to use the result to perform an "lcg-ls". Results are below (*). As far as the PhEDEx Data service call is concerned I think it's fine.

(*) a) OK for 28 sites: T2_AT_Vienna, T2_BE_IIHE, T2_CH_CAF, T2_CH_CSCS, T2_CN_Beijing, T2_DE_DESY, T2_DE_RWTH, T2_ES_IFCA, T2_FR_GRIF_IRFU, T2_FR_GRIF_LAL T2_FR_GRIF_LLR, T2_FR_GRIF_LPNHE, T2_HU_Budapest,T2_IN_TIFR, T2_IT_Pisa, T2_IT_Rome, T2_KR_KNU, T2_RU_ITEP, T2_RU_JINR, T2_RU_RRC_KI, T2_RU_SINP, T2_TW_Taiwan, T2_UK_London_Brunel, T2_UK_London_IC, T2_UK_London_QMUL T2_UK_London_RHUL,T2_UK_SGrid_Bristol,T2_UK_SGrid_RALPP b) Wrong information: == Error when making call 'lfn2pfn': unknown node at sites T2_FR_CCIN2P3 T2_RU_IHEP

== Empty PFN for /store/user/ at sites T2_FR_IPHC T2_RU_INR T2_RU_PNPI T2_TR_ULAKBIM

== wrong PFN at site T2_TR_METU for protocol srmv2 returned PFN /dpm/grid.metu.edu.tr/home/cms/phedexsrm://eymir.grid.metu.edu.tr:8445/srm/managerv1?SFN=/dpm/grid.metu.edu.tr/hom$is missing "srm:"

c) Failing lcg-ls == no host found in BDII T2_BE_UCL (mantainance?) T2_PL_Warsaw (mantainance?) T2_IT_Legnaro (known to be in mantainance) T2_US_Caltech,T2_US_UCSD (mantainance?)

== permission denied on:

OSG sites due to my crappy certificate: T2_US_Florida,T2_US_MIT,T2_US_Nebraska,T2_US_Purdue,T2_US_Wisconsin T2_BR_SPRACE,T2_BR_UERJ
other sites T2_EE_Estonia (mantainance or not permissions to me?) T2_ES_CIEMAT (known to be in mantainance) T2_IT_Bari (mantainance?) T2_PT_LIP_Lisbon (mantainance or not permissions to me?) T2_FI_HIP (mantainance or not permissions to me?)

ericvaandering commented 11 years ago

Comment by egeland on Thu Jun 26 04:38:30 2008

Yes, the http://cmsweb.cern.ch/phedex/test URL is a testing one. If everything is fine we will deploy it into production.

We could check in the dataservice for null PFNs and return an <error/>, but that might be problematic should you want to fetch PFNs in bulk for many nodes.

I looked into some of the "wrong information" cases. For some of them it looks fine to me. For T2_RU_IHEP, they actually have 2 nodes, T2_RU_IHEP_Disk and T2_RU_IHEP_MSS. Only the _Disk node has a TFC published. T2_FR_CCIN2P3 is not a known node to PhEDEx.

https://cmsweb.cern.ch/phedex/test/datasvc/xml/prod/lfn2pfn?node=T2_RU_IHEP_Disk&node=T2_TR_ULAKBIM&node=T2_RU_PNPI&node=T2_FR_IPHC&node=T2_RU_INR&lfn=/store/user&protocol=srm

ericvaandering commented 11 years ago

Comment by afanfani on Thu Jun 26 11:27:56 2008

Allowing the user to specify more sites for stageout, thus fetching PFN in bulk, is not supported now in CRAB so it's not urgent to have it.

T2_FRCCIN2P3 is CMS name taken from SiteDB, I guess is not a PhEDEx node name since at IN2P3 they use the T1 node since they share the same storage between T1 and T2. For the users is going to be confusing to specify T2* for all the sites but T1_FR_CCIN2P3 for IN2P3 but we can probably live with it

ciao, A.

ericvaandering commented 11 years ago

Comment by afanfani on Fri Jun 27 04:28:07 2008

Since the intervention yesterday I nolonger manage to access the test URL: https://cmsweb.cern.ch/phedex/test/datasvc/xml/prod/lfn2pfn?node=T2_RU_IHEP_Disk&node=T2_TR_ULAKBIM&node=T2_RU_PNPI&node=T2_FR_IPHC&node=T2_RU_INR&lfn=/store/user&protocol=srm

recieving a:

Bad Gateway The proxy server received an invalid response from an upstream server.

Is that known or intentional? I can't check now but I suspect what you says is fine for some sites is ok for "srm" protocol, while I'm asking for "srmv2".

ciao, A.

ericvaandering commented 11 years ago

Comment by egeland on Tue Sep 2 03:30:27 2008

lfn2pfn was deployed in PHEDEX-datasvc 1.1.

ericvaandering commented 11 years ago

Closed by egeland on Tue Sep 2 03:30:27 2008

dmwm / PHEDEX

Dataservice "tfc" call returning the path for a given lfn at a given node #383

dmwm / PHEDEX

Dataservice &quot;tfc&quot; call returning the path for a given lfn at a given node #383

Dataservice "tfc" call returning the path for a given lfn at a given node #383