search

TUM-DAML / seml

SEML: Slurm Experiment Management Library
Other
165 stars 29 forks source link

TMPDIR is not configurable #94

Closed jan-engelmann closed 2 years ago

jan-engelmann commented 2 years ago

Expected Behavior

It should be possible to set the TMPDIR from where the jobs are run. Otherwise jobs fail if there is not sufficient space on /tmp.

Actual Behavior

The temporary directory is created in this line here.

/tmp is hardcoded.

Error message:

Starting job 5944119
SLURM assigned me the node(s): supergpu08
/var/spool/slurmd/job5944119/slurm_script: line 34: cannot create temp file for here-document: No space left on device
mkdir: cannot create directory ‘/tmp/8e303830-9b96-4a67-87a7-c16a400f6705’: No space left on device
Experiments are running under the following process IDs:

Specifications

Run on a scientific cluster where /tmp should not be used but /localscratch instead

n-gao commented 2 years ago

Fixed in bcc9e3cdd11a514bf1f9ee7d36eb5dde4acd5117