Local serial execution of test pipeline

[danisven]

Modified the test pipeline in an attempt to run the pipeline locally on the KAW server.

Modified batch_execution.py:

from doepipeline.generator import PipelineGenerator
#from doepipeline.executor import SSHExecutor
from doepipeline.executor import LocalExecutor
import os
import yaml

if __name__ == '__main__':

    generator = PipelineGenerator.from_yaml('/media/data/daniel/doe/doepipeline_testscript/pipeline.yaml')
    designer = generator.new_designer_from_config()
    design = designer.new_design()
    pipeline = generator.new_pipeline_collection(design)

    executor = LocalExecutor(workdir='test_pipeline', execution_type='serial', base_command='nohup {script}')
    executor.execute_command('cd test_pipeline; ls | grep "[0-9]" | xargs rm -r')
    results = executor.run_pipeline_collection(pipeline)
    optimum = designer.update_factors_from_response(results)
    pass

$ (doe_pipeline)daniel@kaw:/media/data/daniel/doe/doepipeline_testscript$ python batch_execution.py
Traceback (most recent call last):
  File "/media/data/db/data/anaconda/envs/doe_pipeline/lib/python3.5/site-packages/doepipeline-0.1-py3.5.egg/doepipeline/generator.py", line 22, in __init__
    self._validate_config(config)
  File "/media/data/db/data/anaconda/envs/doe_pipeline/lib/python3.5/site-packages/doepipeline-0.1-py3.5.egg/doepipeline/generator.py", line 221, in _validate_config
    'job specified with SLURM but SLURM project-name is missing'
AssertionError: job specified with SLURM but SLURM project-name is missing

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "batch_execution.py", line 10, in <module>
    generator = PipelineGenerator.from_yaml('/media/data/daniel/doe/doepipeline_testscript/pipeline.yaml')
  File "/media/data/db/data/anaconda/envs/doe_pipeline/lib/python3.5/site-packages/doepipeline-0.1-py3.5.egg/doepipeline/generator.py", line 55, in from_yaml
    return cls(config, *args, **kwargs)
  File "/media/data/db/data/anaconda/envs/doe_pipeline/lib/python3.5/site-packages/doepipeline-0.1-py3.5.egg/doepipeline/generator.py", line 24, in __init__
    raise ValueError('Invalid config: ' + str(e))
ValueError: Invalid config: job specified with SLURM but SLURM project-name is missing

Does this boil down to line 220 in generator.py?

assert (any('SLURM' in job.keys() for job in jobs) and 'SLURM' in config_dict),\
            'job specified with SLURM but SLURM project-name is missing'

The YAML file I use does not contain 'SLURM'.

[RickardSjogren]

A SLURM entry is required in the config file. So far the stand-alone SLURM entry only contains one key namely account_name. Since, according to the SLURM user guide under required job parameters:

These are the things you typically need to specify for each job (required parameters might differ sligthly depending on which other parameters are set):

Which project should be accounted for the running time (Format: -A [project name])
    For example, if you project is named p2010999, you specify -A p2010999
    You can find your current projects (and other projects that you have run jobs in) with the program projinfo.

[danisven]

Even for a config file that specifies a pipeline that is to be run locally, without SLURM?

[danisven]

Line 220, generator.py:

assert (any('SLURM' in job.keys() for job in jobs) and 'SLURM' in config_dict),\
            'job specified with SLURM but SLURM project-name is missing'

The assert will only be True if at least one of the jobs contain 'SLURM' and the config file contain 'SLURM', no?

Shouldn't the above be changed to something like:

if any('SLURM' in job.keys() for job in jobs):  # If any job has been specified to use SLURM
    assert 'SLURM' in config_dict, "job specified with SLURM but SLURM entry is missing"  # Make sure that there is a SLURM entry in the config file

Because that way we first check if any of the jobs should be run using SLURM, and if so, check that the config file has the SLURM entry somewhere.

Or am I missing something obvious?