facebookincubator / submitit

Python 3.8+ toolbox for submitting jobs to Slurm
MIT License
1.3k stars 125 forks source link

Add optional setup step for local executor #1754

Closed jrapin closed 1 year ago

jrapin commented 1 year ago

Allows to add a setup step for local cluster. When using a setup step we need (?) to use shell=True though

This allows changing the conda env before triggering the job with something like:

string = subprocess.check_output("conda info --base".split(), shell=False.decode("utf8")
cfile = Path(string.strip()) / "etc/profile.d/conda.sh"
cmds = [
    f". {cfile.absolute()}",
    "conda deactivate",
    f"conda activate {self.conda_env}",
]
executor.update_parameters(local_setup=cmds, slurm_setup=cmds)
jrapin commented 1 year ago

I'm not an expert with subprocesses, I could not make it work without changing to shell=True when there is a setup step, I expect that may not be possible? + probably would fail in many cases unexpected by the user