search

facebookincubator / submitit

Python 3.8+ toolbox for submitting jobs to Slurm
MIT License
1.28k stars 121 forks source link

Include script directory to the search path? #12

Open jrapin opened 4 years ago

jrapin commented 4 years ago

Feedback received:

It would be great if submitit also includes the script directory into its search path.
So the script could be running like non-submitit cases.

submitit is expected to work is if it were run locally, so adding the script path can definitely make sense if it has no border effect (anything to fear?)

gwenzek commented 4 years ago

I'm a bit mitigated. For me it seems that it only helps with code which is not in modules. How much should we bend backward for this use case ?

gwenzek commented 3 years ago

I have changed my mind on this.

The problem is that when running the python schedule_exp.py python will automatically add the current folder to sys.path. When the job starts it's started with python -m submitit and doesn't add "." which leads to confusing behavior.

gwenzek commented 3 years ago

Alternative implementation of #1596:

process = subprocess.Popen(
        [sys.executable, "-m", "submitit.local._local", str(folder)],
        shell=False,
        env=env,
        cwd=sys.path[0],  # <--- set the cwd to be the same than the one used for launching the script. 
    )

this need to be adapted for the SLURM executor. And it may lead to other errors if you have a custom manipulation of sys.path (eg that prepend something to the path and changes sys.path[0])

I think #1596 is more robust