Fix base lfn in stdspecs

sfoulkes commented 14 years ago

It's currently so weird combination of the T0 and PA conventions.

sfoulkes commented 14 years ago

sfoulkes: Dave, review please? Specifically that I have the convention correct.

drsm79 commented 14 years ago

metson: The convention isn't quite right:

https://twiki.cern.ch/twiki/bin/viewauth/CMS/DMWMPG_Namespace#Top_level_namespace

There should be an lfn counter (to deal with large blocks) and a GUID for a file name at the end (though maybe you're making a base path?), e.g.

/store/data/09_Spring/single_electron/AOD/v2/0132/somerandomhexnumber.root

hufnagel commented 14 years ago

hufnagel: The Tie0follows that convention, except our lfn counter is run based: <r/u/n>, every element being padded to be 3 characters. For MC this is likely not very useful though because we have one run with potentially many many files per run (this will change somewhat with run based MC though).

For data reprocessing it remains a viable option though, although I am not sure if it's feasible/worthwhile to implement something like this or to just say that the Tie0does it's own thing and anything coming later just uses lfn counters.

In any case, I am pretty certain a single 4 digit number will not be sufficient.

Also, having the run number encoded in the LFN is convenient and can even be useful on the technical level (at least for the Tie0 where you can in theory play games with the TFC and do per run access paths, would have allowed us to recover the runs broken due to the latest xrootd bug for instance).

sfoulkes commented 14 years ago

sfoulkes: This is just making the base path. The LFN counter and GUID file name is added at runtime during stage out.

hufnagel commented 14 years ago

hufnagel: Btw, on second look, the twiki is slightly wrong (at least for the Tie0. Twiki lists

/store/data/acquisition_era/primary-dataset/data_tier/processing_version/lfn_counter/filename.root

but it really is

/store/data/acquisition_era/primary-dataset/data_tier/[processing_string-]processing_version/lfn_counter/filename.root

Only if processingString is None does it collapse to the syntax mentioned on the twiki.

For instance, PromptReco files would go to

/store/hidata/HIRun2010/HIAllPhysics/RECO/PromptReco-v3/000/152/625/E2B84CEA-74FA-DF11-B7CF-0030487CD17C.root

This was recently changed by request of DataOps. I think originally we thought this would all be encoded in the processingVersion, but it wasn't used this way and we ended up with LFN space collission between different samples (PromptReco and ReReco).

drsm79 commented 14 years ago

metson: Replying to [comment:5 hufnagel]:

This was recently changed by request of DataOps. I think originally we thought this would all be encoded in the processingVersion, but it wasn't used this way and we ended up with LFN space collission between different samples (PromptReco and ReReco).

Why wasn't it done as agreed?

hufnagel commented 14 years ago

hufnagel: PromptReco and ReReco we could have kept separate this way. The real problem came in when we looked at skims though. To keep separate LFN namespace for a full Reco pass and the corresponding RECO skims (or full AOD and skim AOD samples respectively), we would have had to assign different processingVersions for full processing passes and each skim run after. Mix in multiple full processing passes and out of order skims and you end up with a complete mess.