NDCMS / lobster

A userspace workflow management tool for harnessing non-dedicated resources for high-throughput workloads.
MIT License
3 stars 14 forks source link

Remove trailing slash in lfn2pfn method return #645

Closed Andrew42 closed 4 years ago

Andrew42 commented 4 years ago

This change allows the user to specify individual input files to run over using the XrootD storage.input method, rather then an entire directory.

I tested the changes with other storage input methods tying to access both individual files and directories. Specifically, I tested the XrootD, HDFS, and File input methods.

For XrootD, I needed to make some adjustments to how I specified the storage.input method as well as how I passed in the path name depending on if I was specifying a single file path or a directory. For accessing a single file the following was sufficient:

storage = StorageConfiguration(output=[...])
samp = "/store/user/kmohrman/FullProduction/Round6/Batch1/postLHE_step/v1/mAOD_step_ttHJet_HanV4ttXJetStartPtChecks_run2/HIG-RunIIFall17MiniAOD-00821ND_298532.root"
storage.input = ["root://deepthought.crc.nd.edu/"]
ds = Dataset(files=[samp])

However, for accessing a directory using the XrootD method, I had to modify things to look like this:

storage = StorageConfiguration(output=[...])
samp = "user/kmohrman/FullProduction/Round6/Batch1/postLHE_step/v1/mAOD_step_ttHJet_HanV4ttXJetStartPtChecks_run2"
storage.input = ["root://deepthought.crc.nd.edu//store/"]
ds = Dataset(files=[samp])

For the local file access method a similar modification was needed, for example the following works for accessing a single file:

storage = StorageConfiguration(output=[...])
samp = "/afs/crc.nd.edu/user/a/awightma/CMSSW_Releases/from_govner/CMSSW_9_4_6/src/EFTMultilepton/TemplateMakers/test/test_dir/HIG-RunIIFall17MiniAOD-00821ND_298299.root"
storage.input = ["file://"]
ds = Dataset(files=[samp])

For accessing a directory you have to do the following:

storage = StorageConfiguration(output=[...])
samp = "crc.nd.edu/user/a/awightma/CMSSW_Releases/from_govner/CMSSW_9_4_6/src/EFTMultilepton/TemplateMakers/test/test_dir"
storage.input = ["file:///afs/"]
ds = Dataset(files=[samp])