themattinthehatt / behavenet

Toolbox for analyzing behavioral videos and neural activity
https://behavenet.readthedocs.io/
MIT License
57 stars 15 forks source link

Slurm's srun command passes hyperparameters as command line args instead of config files #12

Closed yifr closed 3 years ago

yifr commented 4 years ago

I'm trying to run some decoding trials using slurm. I have a slurm config file that looks like this:

#!/bin/bash
#SBATCH --job-name=decoder_grid_search    # Job name
#SBATCH --mail-type=END,FAIL          # Mail events (NONE, BEGIN, END, FAIL, ALL)
#SBATCH --mail-user=yoniifriedman@gmail.com   # Where to send mail
#SBATCH --time=00:05:00               # Time limit hrs:min:sec
#SBATCH --account=zi
#SBATCH --nodes 1
#SBATCH -c 1

The job fails, with this message:

Screen Shot 2020-06-23 at 6 49 04 PM

Here's the srun command where the arguments are coming from:

Screen Shot 2020-06-23 at 6 48 51 PM

Should the srun command contain the typical config command line arguments instead of the hyperparameters themselves?

The __get_run_command function for context:

def __get_run_command(self, trial, slurm_cmd_script_path, timestamp, exp_i, on_gpu):
        trial_args = self._SlurmCluster__get_hopt_params(trial)
        trial_args = '{} --{} {} --{} {}'.format(trial_args,
                                                 HyperOptArgumentParser.SLURM_CMD_PATH,
                                                 slurm_cmd_script_path,
                                                 HyperOptArgumentParser.SLURM_EXP_CMD,
                                                 exp_i)

        cmd = 'srun {} {} {}'.format(self.python_cmd, self.script_name, trial_args)
        return cmd
themattinthehatt commented 4 years ago

@ebatty any input on this? I haven't used behavenet with slurm before