search

NDCMS / lobster

A userspace workflow management tool for harnessing non-dedicated resources for high-throughput workloads.
MIT License
3 stars 14 forks source link

Allow XrootD stage-out. #638

Closed khurtado closed 5 years ago

khurtado commented 5 years ago

Fixes #161. Briefly tested with a simple task. People wanting to test are welcome!

For context, at present, the configuration below will just skip XRootD for stage-out with some message like:

>>>> skipping unhandled stage-out method: root://deepthought.crc.nd.edu//store/user/... @ Sat Feb 23 08:50:36 2019

and go directly to gsiftp when hdfs is not available. This PR should fix that.

    output=[
        "hdfs://eddie.crc.nd.edu:19000/store/user/$USER/lobster_test_" + version,
        "root://deepthought.crc.nd.edu//store/user/$USER/lobster_test_" + version,
        "gsiftp://T3_US_NotreDame/store/user/$USER/lobster_test_" + version,
    ]
klannon commented 5 years ago

I don't have time to test this, but if you did at least basic testing, I'm happy to merge this pull request. Just to verify: you actually tested this code by running some jobs and staging out the output via XRootD?

khurtado commented 5 years ago

@klannon Yes, I ran some jobs on crcfe02 and verified the stage out happened via XRootD