search

riga / law

Build large-scale task workflows: luigi + job submission + remote targets + environment sandboxing using Docker/Singularity
http://law.readthedocs.io
BSD 3-Clause "New" or "Revised" License
98 stars 41 forks source link

Add spool option for HTCondor #186

Closed TheRealLoliges486 closed 2 months ago

TheRealLoliges486 commented 2 months ago

Description

Hi,

Currently as far as CERN is concerned, the submission of HTCondor jobs while on EOS is not allowed. The usual workaround is using the -spool option while submitting (so basically condor_submit -spool job.sub). Consider implementing that to law.

Thanks :-)

riga commented 2 months ago

Added in e9408bfa.

The option defaults to False, but on task level you can define

class MyTask(MyBaseTasks, HTCondorWorkflow):

    htcondor_job_kwargs_submit = {"spool": True}

    ....

to activate it.