search

NDCMS / lobster

A userspace workflow management tool for harnessing non-dedicated resources for high-throughput workloads.
MIT License
3 stars 14 forks source link

SITECONF parsing to account for multiple options (i.e. fallback) #354

Open matz-e opened 8 years ago

matz-e commented 8 years ago

We should have the SITECONF parsing account for multiple resolutions per protocol, and also properly use these on the worker…

matz-e commented 8 years ago

Drawback: This will not work, anymore:

>>> s.expand_site("srm://T3_US_NotreDame/store/user/foo")
u'srm://deepthought.crc.nd.edu:8443/srm/v2/server?SFN=/hadoop/store/user/foo'

@khurtado: can URL matching be done via SiteDB or other WMCore means? This would be nice to retain.

klannon commented 8 years ago

Why doesn't that work?

klannon commented 8 years ago

I'm trying to understand what the actual problem is here (i.e. which protocol(s) do you not think are being correctly resolved). The one you posted above looks correct to me.

matz-e commented 8 years ago

I meant that the SiteDB methods don't allow to get the path manipulation the same way that the SITECONF files do… With the above, there's no worries what the exact path prefix for the SRM instance is, and you can just swap out T3_US_NotreDame and T2_CH_CERN without any further action (like removing the hadoop/ part).

klannon commented 8 years ago

What's wrong with SITECONF?

matz-e commented 8 years ago

The XrootD protocol resolves to xrootd.unl.edu, but I don't think we're registered in the re-director.

klannon commented 8 years ago

Right. That's not a bug. That's what we've decided to do with our SITECONF. If we want to fix it, we just need to add more xrootd lines to that file.

Be aware that protocol mappings in SITECONF are not one-to-one because SITECONF allows the concept off fallback. We should probably have three (or maybe two) lines in our SITECONF for xrootd:

  1. deepthought
  2. The cache (unless the deepthought redirector includes the cache)
  3. global (maybe changing Nebraska to the FNAL redirector?)
klannon commented 8 years ago

Do we still ship a custom SITECONF with Lobster jobs? If so, you can hack that one to try out different options.

matz-e commented 8 years ago

This is done on the master to resolve CMS names you pass in the StorageConfiguration constructor…

matz-e commented 8 years ago

Multiple matches we could probably treat within Lobster, next week or later.

klannon commented 8 years ago

Right. We just need to update our local SITECONF. (We should do that anyway.) And you'll defintely want to update the Lobster code to account for the fact that if you ask for the xrootd protocol, you should expect a list, not a single entry. I agree this doesn't need to happen this week as we can just manually set xrootd to deepthought in the storage list. (That's actually how I've always done it when I wanted access to our local storage.)

klannon commented 8 years ago

Anyway, to fill out the remainder of the issues discussed, I would advise AGAINST using SiteDB as I don't believe any other CMS service extracts any information from SiteDB. WMCore uses the following url to resolve requests (same info as SITECONF):

https://cmsweb.cern.ch/phedex/datasvc/xml/prod/lfn2pfn?node=T3_US_NotreDame&lfn=/store/user/lannon&protocol=srmv2

The drawback there is that the URL above extracts its info from a DB that's only updated when we give a new storage.xml file to the folks operating the central PhEDEx service for T3 sites.

SITECONF is really the best place to get this from.

klannon commented 8 years ago

I suppose I'm proposing changing the name of this issue to update SITECONF parsing to account for multiple options (i.e. fallback) for each protocol. :)

And, we need to bug Sergei and Kenyi to help us get ND's SITECONF updated. We should probably do this anyway to account for the cache.