search

cgat-developers / cgat-core

Core functionality of the CGAT code
MIT License
32 stars 13 forks source link

drmaa no suitable queues error #36

Closed logust79 closed 6 years ago

logust79 commented 6 years ago

First of all, great work! I downloaded the core code and ran the tests and found the following errors. tldr: all of those related to drmaa.errors.DeniedByDrmException: code 17: error: no suitable queues

I tested on a SGE cluster system. How can I solve it? Thanks, Jing

======================================================================
FAIL: test_shell_log_is_created_at_absolute_path (test_pipeline_control.TestExecutionControl)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/cluster/project8/vyp/JingYu/git/cgat/cgat-core/tests/test_pipeline_control.py", line 83, in test_shell_log_is_created_at_absolute_path
    "python {}/template_pipeline.py make all --shell-logfile={}".format(ROOT, shell_file))
  File "/cluster/project8/vyp/JingYu/git/cgat/cgat-core/tests/test_pipeline_control.py", line 37, in run_command
    self.assertEqual(proc.returncode, 0, msg="stderr = {}".format(stderr))
AssertionError: 1 != 0 : stderr = Traceback (most recent call last):
  File "/cluster/project8/vyp/JingYu/git/cgat/cgat-core/CGATCore/Pipeline/Control.py", line 1164, in run_workflow
    one_second_per_job=False,
  File "/cluster/project8/vyp/JingYu/git/cgat/conda-install/envs/cgat-c/lib/python3.6/site-packages/ruffus/task.py", line 5938, in pipeline_run
    raise job_errors
ruffus.ruffus_exceptions.RethrownJobError:

Original exceptions:

    Exception #1
      'drmaa.errors.DeniedByDrmException(code 17: error: no suitable queues)' raised in ...
       Task = def compute_mean(...):
       Job  = [sample_00.txt -> sample_00.mean]

    Traceback (most recent call last):
      File "/cluster/project8/vyp/JingYu/git/cgat/conda-install/envs/cgat-c/lib/python3.6/site-packages/ruffus/task.py", line 751, in run_pooled_job_without_exceptions
        register_cleanup, touch_files_only)
      File "/cluster/project8/vyp/JingYu/git/cgat/conda-install/envs/cgat-c/lib/python3.6/site-packages/ruffus/task.py", line 567, in job_wrapper_io_files
        ret_val = user_defined_work_func(*params)
      File "/cluster/project8/vyp/JingYu/git/cgat/cgat-core/tests/template_pipeline.py", line 67, in compute_mean
        P.run(statement)
      File "/cluster/project8/vyp/JingYu/git/cgat/cgat-core/CGATCore/Pipeline/Execution.py", line 1328, in run
        benchmark_data = r.run(statement_list)
      File "/cluster/project8/vyp/JingYu/git/cgat/cgat-core/CGATCore/Pipeline/Execution.py", line 910, in run
        job_id = self.session.runJob(jt)
      File "/cluster/project8/vyp/JingYu/git/cgat/conda-install/envs/cgat-c/lib/python3.6/site-packages/drmaa/session.py", line 314, in runJob
        c(drmaa_run_job, jid, sizeof(jid), jobTemplate)
      File "/cluster/project8/vyp/JingYu/git/cgat/conda-install/envs/cgat-c/lib/python3.6/site-packages/drmaa/helpers.py", line 303, in c
        return f(*(args + (error_buffer, sizeof(error_buffer))))
      File "/cluster/project8/vyp/JingYu/git/cgat/conda-install/envs/cgat-c/lib/python3.6/site-packages/drmaa/errors.py", line 151, in error_check
        raise _ERRORS[code - 1](error_string)
    drmaa.errors.DeniedByDrmException: code 17: error: no suitable queues

    Exception #2
      'drmaa.errors.DeniedByDrmException(code 17: error: no suitable queues)' raised in ...
       Task = def compute_mean(...):
       Job  = [sample_01.txt -> sample_01.mean]

    Traceback (most recent call last):
      File "/cluster/project8/vyp/JingYu/git/cgat/conda-install/envs/cgat-c/lib/python3.6/site-packages/ruffus/task.py", line 751, in run_pooled_job_without_exceptions
        register_cleanup, touch_files_only)
      File "/cluster/project8/vyp/JingYu/git/cgat/conda-install/envs/cgat-c/lib/python3.6/site-packages/ruffus/task.py", line 567, in job_wrapper_io_files
        ret_val = user_defined_work_func(*params)
      File "/cluster/project8/vyp/JingYu/git/cgat/cgat-core/tests/template_pipeline.py", line 67, in compute_mean
        P.run(statement)
      File "/cluster/project8/vyp/JingYu/git/cgat/cgat-core/CGATCore/Pipeline/Execution.py", line 1328, in run
        benchmark_data = r.run(statement_list)
      File "/cluster/project8/vyp/JingYu/git/cgat/cgat-core/CGATCore/Pipeline/Execution.py", line 910, in run
        job_id = self.session.runJob(jt)
      File "/cluster/project8/vyp/JingYu/git/cgat/conda-install/envs/cgat-c/lib/python3.6/site-packages/drmaa/session.py", line 314, in runJob
        c(drmaa_run_job, jid, sizeof(jid), jobTemplate)
      File "/cluster/project8/vyp/JingYu/git/cgat/conda-install/envs/cgat-c/lib/python3.6/site-packages/drmaa/helpers.py", line 303, in c
        return f(*(args + (error_buffer, sizeof(error_buffer))))
      File "/cluster/project8/vyp/JingYu/git/cgat/conda-install/envs/cgat-c/lib/python3.6/site-packages/drmaa/errors.py", line 151, in error_check
        raise _ERRORS[code - 1](error_string)
    drmaa.errors.DeniedByDrmException: code 17: error: no suitable queues

    Exception #3
      'drmaa.errors.DeniedByDrmException(code 17: error: no suitable queues)' raised in ...
       Task = def compute_mean(...):
       Job  = [sample_02.txt -> sample_02.mean]

    Traceback (most recent call last):
      File "/cluster/project8/vyp/JingYu/git/cgat/conda-install/envs/cgat-c/lib/python3.6/site-packages/ruffus/task.py", line 751, in run_pooled_job_without_exceptions
        register_cleanup, touch_files_only)
      File "/cluster/project8/vyp/JingYu/git/cgat/conda-install/envs/cgat-c/lib/python3.6/site-packages/ruffus/task.py", line 567, in job_wrapper_io_files
        ret_val = user_defined_work_func(*params)
      File "/cluster/project8/vyp/JingYu/git/cgat/cgat-core/tests/template_pipeline.py", line 67, in compute_mean
        P.run(statement)
      File "/cluster/project8/vyp/JingYu/git/cgat/cgat-core/CGATCore/Pipeline/Execution.py", line 1328, in run
        benchmark_data = r.run(statement_list)
      File "/cluster/project8/vyp/JingYu/git/cgat/cgat-core/CGATCore/Pipeline/Execution.py", line 910, in run
        job_id = self.session.runJob(jt)
      File "/cluster/project8/vyp/JingYu/git/cgat/conda-install/envs/cgat-c/lib/python3.6/site-packages/drmaa/session.py", line 314, in runJob
        c(drmaa_run_job, jid, sizeof(jid), jobTemplate)
      File "/cluster/project8/vyp/JingYu/git/cgat/conda-install/envs/cgat-c/lib/python3.6/site-packages/drmaa/helpers.py", line 303, in c
        return f(*(args + (error_buffer, sizeof(error_buffer))))
      File "/cluster/project8/vyp/JingYu/git/cgat/conda-install/envs/cgat-c/lib/python3.6/site-packages/drmaa/errors.py", line 151, in error_check
        raise _ERRORS[code - 1](error_string)
    drmaa.errors.DeniedByDrmException: code 17: error: no suitable queues

    Exception #4
      'drmaa.errors.DeniedByDrmException(code 17: error: no suitable queues)' raised in ...
       Task = def compute_mean(...):
       Job  = [sample_03.txt -> sample_03.mean]

    Traceback (most recent call last):
      File "/cluster/project8/vyp/JingYu/git/cgat/conda-install/envs/cgat-c/lib/python3.6/site-packages/ruffus/task.py", line 751, in run_pooled_job_without_exceptions
        register_cleanup, touch_files_only)
      File "/cluster/project8/vyp/JingYu/git/cgat/conda-install/envs/cgat-c/lib/python3.6/site-packages/ruffus/task.py", line 567, in job_wrapper_io_files
        ret_val = user_defined_work_func(*params)
      File "/cluster/project8/vyp/JingYu/git/cgat/cgat-core/tests/template_pipeline.py", line 67, in compute_mean
        P.run(statement)
      File "/cluster/project8/vyp/JingYu/git/cgat/cgat-core/CGATCore/Pipeline/Execution.py", line 1328, in run
        benchmark_data = r.run(statement_list)
      File "/cluster/project8/vyp/JingYu/git/cgat/cgat-core/CGATCore/Pipeline/Execution.py", line 910, in run
        job_id = self.session.runJob(jt)
      File "/cluster/project8/vyp/JingYu/git/cgat/conda-install/envs/cgat-c/lib/python3.6/site-packages/drmaa/session.py", line 314, in runJob
        c(drmaa_run_job, jid, sizeof(jid), jobTemplate)
      File "/cluster/project8/vyp/JingYu/git/cgat/conda-install/envs/cgat-c/lib/python3.6/site-packages/drmaa/helpers.py", line 303, in c
        return f(*(args + (error_buffer, sizeof(error_buffer))))
      File "/cluster/project8/vyp/JingYu/git/cgat/conda-install/envs/cgat-c/lib/python3.6/site-packages/drmaa/errors.py", line 151, in error_check
        raise _ERRORS[code - 1](error_string)
    drmaa.errors.DeniedByDrmException: code 17: error: no suitable queues

    Exception #5
      'drmaa.errors.DeniedByDrmException(code 17: error: no suitable queues)' raised in ...
       Task = def compute_mean(...):
       Job  = [sample_04.txt -> sample_04.mean]

    Traceback (most recent call last):
      File "/cluster/project8/vyp/JingYu/git/cgat/conda-install/envs/cgat-c/lib/python3.6/site-packages/ruffus/task.py", line 751, in run_pooled_job_without_exceptions
        register_cleanup, touch_files_only)
      File "/cluster/project8/vyp/JingYu/git/cgat/conda-install/envs/cgat-c/lib/python3.6/site-packages/ruffus/task.py", line 567, in job_wrapper_io_files
        ret_val = user_defined_work_func(*params)
      File "/cluster/project8/vyp/JingYu/git/cgat/cgat-core/tests/template_pipeline.py", line 67, in compute_mean
        P.run(statement)
      File "/cluster/project8/vyp/JingYu/git/cgat/cgat-core/CGATCore/Pipeline/Execution.py", line 1328, in run
        benchmark_data = r.run(statement_list)
      File "/cluster/project8/vyp/JingYu/git/cgat/cgat-core/CGATCore/Pipeline/Execution.py", line 910, in run
        job_id = self.session.runJob(jt)
      File "/cluster/project8/vyp/JingYu/git/cgat/conda-install/envs/cgat-c/lib/python3.6/site-packages/drmaa/session.py", line 314, in runJob
        c(drmaa_run_job, jid, sizeof(jid), jobTemplate)
      File "/cluster/project8/vyp/JingYu/git/cgat/conda-install/envs/cgat-c/lib/python3.6/site-packages/drmaa/helpers.py", line 303, in c
        return f(*(args + (error_buffer, sizeof(error_buffer))))
      File "/cluster/project8/vyp/JingYu/git/cgat/conda-install/envs/cgat-c/lib/python3.6/site-packages/drmaa/errors.py", line 151, in error_check
        raise _ERRORS[code - 1](error_string)
    drmaa.errors.DeniedByDrmException: code 17: error: no suitable queues

    Exception #6
      'drmaa.errors.DeniedByDrmException(code 17: error: no suitable queues)' raised in ...
       Task = def compute_mean(...):
       Job  = [sample_05.txt -> sample_05.mean]

    Traceback (most recent call last):
      File "/cluster/project8/vyp/JingYu/git/cgat/conda-install/envs/cgat-c/lib/python3.6/site-packages/ruffus/task.py", line 751, in run_pooled_job_without_exceptions
        register_cleanup, touch_files_only)
      File "/cluster/project8/vyp/JingYu/git/cgat/conda-install/envs/cgat-c/lib/python3.6/site-packages/ruffus/task.py", line 567, in job_wrapper_io_files
        ret_val = user_defined_work_func(*params)
      File "/cluster/project8/vyp/JingYu/git/cgat/cgat-core/tests/template_pipeline.py", line 67, in compute_mean
        P.run(statement)
      File "/cluster/project8/vyp/JingYu/git/cgat/cgat-core/CGATCore/Pipeline/Execution.py", line 1328, in run
        benchmark_data = r.run(statement_list)
      File "/cluster/project8/vyp/JingYu/git/cgat/cgat-core/CGATCore/Pipeline/Execution.py", line 910, in run
        job_id = self.session.runJob(jt)
      File "/cluster/project8/vyp/JingYu/git/cgat/conda-install/envs/cgat-c/lib/python3.6/site-packages/drmaa/session.py", line 314, in runJob
        c(drmaa_run_job, jid, sizeof(jid), jobTemplate)
      File "/cluster/project8/vyp/JingYu/git/cgat/conda-install/envs/cgat-c/lib/python3.6/site-packages/drmaa/helpers.py", line 303, in c
        return f(*(args + (error_buffer, sizeof(error_buffer))))
      File "/cluster/project8/vyp/JingYu/git/cgat/conda-install/envs/cgat-c/lib/python3.6/site-packages/drmaa/errors.py", line 151, in error_check
        raise _ERRORS[code - 1](error_string)
    drmaa.errors.DeniedByDrmException: code 17: error: no suitable queues

    Exception #7
      'drmaa.errors.DeniedByDrmException(code 17: error: no suitable queues)' raised in ...
       Task = def compute_mean(...):
       Job  = [sample_06.txt -> sample_06.mean]

    Traceback (most recent call last):
      File "/cluster/project8/vyp/JingYu/git/cgat/conda-install/envs/cgat-c/lib/python3.6/site-packages/ruffus/task.py", line 751, in run_pooled_job_without_exceptions
        register_cleanup, touch_files_only)
      File "/cluster/project8/vyp/JingYu/git/cgat/conda-install/envs/cgat-c/lib/python3.6/site-packages/ruffus/task.py", line 567, in job_wrapper_io_files
        ret_val = user_defined_work_func(*params)
      File "/cluster/project8/vyp/JingYu/git/cgat/cgat-core/tests/template_pipeline.py", line 67, in compute_mean
        P.run(statement)
      File "/cluster/project8/vyp/JingYu/git/cgat/cgat-core/CGATCore/Pipeline/Execution.py", line 1328, in run
        benchmark_data = r.run(statement_list)
      File "/cluster/project8/vyp/JingYu/git/cgat/cgat-core/CGATCore/Pipeline/Execution.py", line 910, in run
        job_id = self.session.runJob(jt)
      File "/cluster/project8/vyp/JingYu/git/cgat/conda-install/envs/cgat-c/lib/python3.6/site-packages/drmaa/session.py", line 314, in runJob
        c(drmaa_run_job, jid, sizeof(jid), jobTemplate)
      File "/cluster/project8/vyp/JingYu/git/cgat/conda-install/envs/cgat-c/lib/python3.6/site-packages/drmaa/helpers.py", line 303, in c
        return f(*(args + (error_buffer, sizeof(error_buffer))))
      File "/cluster/project8/vyp/JingYu/git/cgat/conda-install/envs/cgat-c/lib/python3.6/site-packages/drmaa/errors.py", line 151, in error_check
        raise _ERRORS[code - 1](error_string)
    drmaa.errors.DeniedByDrmException: code 17: error: no suitable queues

    Exception #8
      'drmaa.errors.DeniedByDrmException(code 17: error: no suitable queues)' raised in ...
       Task = def compute_mean(...):
       Job  = [sample_07.txt -> sample_07.mean]

    Traceback (most recent call last):
      File "/cluster/project8/vyp/JingYu/git/cgat/conda-install/envs/cgat-c/lib/python3.6/site-packages/ruffus/task.py", line 751, in run_pooled_job_without_exceptions
        register_cleanup, touch_files_only)
      File "/cluster/project8/vyp/JingYu/git/cgat/conda-install/envs/cgat-c/lib/python3.6/site-packages/ruffus/task.py", line 567, in job_wrapper_io_files
        ret_val = user_defined_work_func(*params)
      File "/cluster/project8/vyp/JingYu/git/cgat/cgat-core/tests/template_pipeline.py", line 67, in compute_mean
        P.run(statement)
      File "/cluster/project8/vyp/JingYu/git/cgat/cgat-core/CGATCore/Pipeline/Execution.py", line 1328, in run
        benchmark_data = r.run(statement_list)
      File "/cluster/project8/vyp/JingYu/git/cgat/cgat-core/CGATCore/Pipeline/Execution.py", line 910, in run
        job_id = self.session.runJob(jt)
      File "/cluster/project8/vyp/JingYu/git/cgat/conda-install/envs/cgat-c/lib/python3.6/site-packages/drmaa/session.py", line 314, in runJob
        c(drmaa_run_job, jid, sizeof(jid), jobTemplate)
      File "/cluster/project8/vyp/JingYu/git/cgat/conda-install/envs/cgat-c/lib/python3.6/site-packages/drmaa/helpers.py", line 303, in c
        return f(*(args + (error_buffer, sizeof(error_buffer))))
      File "/cluster/project8/vyp/JingYu/git/cgat/conda-install/envs/cgat-c/lib/python3.6/site-packages/drmaa/errors.py", line 151, in error_check
        raise _ERRORS[code - 1](error_string)
    drmaa.errors.DeniedByDrmException: code 17: error: no suitable queues

    Exception #9
      'drmaa.errors.DeniedByDrmException(code 17: error: no suitable queues)' raised in ...
       Task = def compute_mean(...):
       Job  = [sample_08.txt -> sample_08.mean]

    Traceback (most recent call last):
      File "/cluster/project8/vyp/JingYu/git/cgat/conda-install/envs/cgat-c/lib/python3.6/site-packages/ruffus/task.py", line 751, in run_pooled_job_without_exceptions
        register_cleanup, touch_files_only)
      File "/cluster/project8/vyp/JingYu/git/cgat/conda-install/envs/cgat-c/lib/python3.6/site-packages/ruffus/task.py", line 567, in job_wrapper_io_files
        ret_val = user_defined_work_func(*params)
      File "/cluster/project8/vyp/JingYu/git/cgat/cgat-core/tests/template_pipeline.py", line 67, in compute_mean
        P.run(statement)
      File "/cluster/project8/vyp/JingYu/git/cgat/cgat-core/CGATCore/Pipeline/Execution.py", line 1328, in run
        benchmark_data = r.run(statement_list)
      File "/cluster/project8/vyp/JingYu/git/cgat/cgat-core/CGATCore/Pipeline/Execution.py", line 910, in run
        job_id = self.session.runJob(jt)
      File "/cluster/project8/vyp/JingYu/git/cgat/conda-install/envs/cgat-c/lib/python3.6/site-packages/drmaa/session.py", line 314, in runJob
        c(drmaa_run_job, jid, sizeof(jid), jobTemplate)
      File "/cluster/project8/vyp/JingYu/git/cgat/conda-install/envs/cgat-c/lib/python3.6/site-packages/drmaa/helpers.py", line 303, in c
        return f(*(args + (error_buffer, sizeof(error_buffer))))
      File "/cluster/project8/vyp/JingYu/git/cgat/conda-install/envs/cgat-c/lib/python3.6/site-packages/drmaa/errors.py", line 151, in error_check
        raise _ERRORS[code - 1](error_string)
    drmaa.errors.DeniedByDrmException: code 17: error: no suitable queues

    Exception #10
      'drmaa.errors.DeniedByDrmException(code 17: error: no suitable queues)' raised in ...
       Task = def compute_mean(...):
       Job  = [sample_09.txt -> sample_09.mean]

    Traceback (most recent call last):
      File "/cluster/project8/vyp/JingYu/git/cgat/conda-install/envs/cgat-c/lib/python3.6/site-packages/ruffus/task.py", line 751, in run_pooled_job_without_exceptions
        register_cleanup, touch_files_only)
      File "/cluster/project8/vyp/JingYu/git/cgat/conda-install/envs/cgat-c/lib/python3.6/site-packages/ruffus/task.py", line 567, in job_wrapper_io_files
        ret_val = user_defined_work_func(*params)
      File "/cluster/project8/vyp/JingYu/git/cgat/cgat-core/tests/template_pipeline.py", line 67, in compute_mean
        P.run(statement)
      File "/cluster/project8/vyp/JingYu/git/cgat/cgat-core/CGATCore/Pipeline/Execution.py", line 1328, in run
        benchmark_data = r.run(statement_list)
      File "/cluster/project8/vyp/JingYu/git/cgat/cgat-core/CGATCore/Pipeline/Execution.py", line 910, in run
        job_id = self.session.runJob(jt)
      File "/cluster/project8/vyp/JingYu/git/cgat/conda-install/envs/cgat-c/lib/python3.6/site-packages/drmaa/session.py", line 314, in runJob
        c(drmaa_run_job, jid, sizeof(jid), jobTemplate)
      File "/cluster/project8/vyp/JingYu/git/cgat/conda-install/envs/cgat-c/lib/python3.6/site-packages/drmaa/helpers.py", line 303, in c
        return f(*(args + (error_buffer, sizeof(error_buffer))))
      File "/cluster/project8/vyp/JingYu/git/cgat/conda-install/envs/cgat-c/lib/python3.6/site-packages/drmaa/errors.py", line 151, in error_check
        raise _ERRORS[code - 1](error_string)
    drmaa.errors.DeniedByDrmException: code 17: error: no suitable queues

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/cluster/project8/vyp/JingYu/git/cgat/cgat-core/tests/template_pipeline.py", line 126, in <module>
    sys.exit(main(sys.argv))
  File "/cluster/project8/vyp/JingYu/git/cgat/cgat-core/tests/template_pipeline.py", line 122, in main
    return P.run_workflow(options, args)
  File "/cluster/project8/vyp/JingYu/git/cgat/cgat-core/CGATCore/Pipeline/Control.py", line 1252, in run_workflow
    raise ValueError("pipeline failed with %i errors" % len(ex.args)) from ex
ValueError: pipeline failed with 10 errors

======================================================================
FAIL: test_shell_log_is_created_in_workdir (test_pipeline_control.TestExecutionControl)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/cluster/project8/vyp/JingYu/git/cgat/cgat-core/tests/test_pipeline_control.py", line 73, in test_shell_log_is_created_in_workdir
    "python {}/template_pipeline.py make all --shell-logfile=shell.log".format(ROOT))
  File "/cluster/project8/vyp/JingYu/git/cgat/cgat-core/tests/test_pipeline_control.py", line 37, in run_command
    self.assertEqual(proc.returncode, 0, msg="stderr = {}".format(stderr))
AssertionError: 1 != 0 : stderr = Traceback (most recent call last):
  File "/cluster/project8/vyp/JingYu/git/cgat/cgat-core/CGATCore/Pipeline/Control.py", line 1164, in run_workflow
    one_second_per_job=False,
  File "/cluster/project8/vyp/JingYu/git/cgat/conda-install/envs/cgat-c/lib/python3.6/site-packages/ruffus/task.py", line 5938, in pipeline_run
    raise job_errors
ruffus.ruffus_exceptions.RethrownJobError:

Original exceptions:

    Exception #1
      'drmaa.errors.DeniedByDrmException(code 17: error: no suitable queues)' raised in ...
       Task = def compute_mean(...):
       Job  = [sample_00.txt -> sample_00.mean]

    Traceback (most recent call last):
      File "/cluster/project8/vyp/JingYu/git/cgat/conda-install/envs/cgat-c/lib/python3.6/site-packages/ruffus/task.py", line 751, in run_pooled_job_without_exceptions
        register_cleanup, touch_files_only)
      File "/cluster/project8/vyp/JingYu/git/cgat/conda-install/envs/cgat-c/lib/python3.6/site-packages/ruffus/task.py", line 567, in job_wrapper_io_files
        ret_val = user_defined_work_func(*params)
      File "/cluster/project8/vyp/JingYu/git/cgat/cgat-core/tests/template_pipeline.py", line 67, in compute_mean
        P.run(statement)
      File "/cluster/project8/vyp/JingYu/git/cgat/cgat-core/CGATCore/Pipeline/Execution.py", line 1328, in run
        benchmark_data = r.run(statement_list)
      File "/cluster/project8/vyp/JingYu/git/cgat/cgat-core/CGATCore/Pipeline/Execution.py", line 910, in run
        job_id = self.session.runJob(jt)
      File "/cluster/project8/vyp/JingYu/git/cgat/conda-install/envs/cgat-c/lib/python3.6/site-packages/drmaa/session.py", line 314, in runJob
        c(drmaa_run_job, jid, sizeof(jid), jobTemplate)
      File "/cluster/project8/vyp/JingYu/git/cgat/conda-install/envs/cgat-c/lib/python3.6/site-packages/drmaa/helpers.py", line 303, in c
        return f(*(args + (error_buffer, sizeof(error_buffer))))
      File "/cluster/project8/vyp/JingYu/git/cgat/conda-install/envs/cgat-c/lib/python3.6/site-packages/drmaa/errors.py", line 151, in error_check
        raise _ERRORS[code - 1](error_string)
    drmaa.errors.DeniedByDrmException: code 17: error: no suitable queues

    Exception #2
      'drmaa.errors.DeniedByDrmException(code 17: error: no suitable queues)' raised in ...
       Task = def compute_mean(...):
       Job  = [sample_01.txt -> sample_01.mean]

    Traceback (most recent call last):
      File "/cluster/project8/vyp/JingYu/git/cgat/conda-install/envs/cgat-c/lib/python3.6/site-packages/ruffus/task.py", line 751, in run_pooled_job_without_exceptions
        register_cleanup, touch_files_only)
      File "/cluster/project8/vyp/JingYu/git/cgat/conda-install/envs/cgat-c/lib/python3.6/site-packages/ruffus/task.py", line 567, in job_wrapper_io_files
        ret_val = user_defined_work_func(*params)
      File "/cluster/project8/vyp/JingYu/git/cgat/cgat-core/tests/template_pipeline.py", line 67, in compute_mean
        P.run(statement)
      File "/cluster/project8/vyp/JingYu/git/cgat/cgat-core/CGATCore/Pipeline/Execution.py", line 1328, in run
        benchmark_data = r.run(statement_list)
      File "/cluster/project8/vyp/JingYu/git/cgat/cgat-core/CGATCore/Pipeline/Execution.py", line 910, in run
        job_id = self.session.runJob(jt)
      File "/cluster/project8/vyp/JingYu/git/cgat/conda-install/envs/cgat-c/lib/python3.6/site-packages/drmaa/session.py", line 314, in runJob
        c(drmaa_run_job, jid, sizeof(jid), jobTemplate)
      File "/cluster/project8/vyp/JingYu/git/cgat/conda-install/envs/cgat-c/lib/python3.6/site-packages/drmaa/helpers.py", line 303, in c
        return f(*(args + (error_buffer, sizeof(error_buffer))))
      File "/cluster/project8/vyp/JingYu/git/cgat/conda-install/envs/cgat-c/lib/python3.6/site-packages/drmaa/errors.py", line 151, in error_check
        raise _ERRORS[code - 1](error_string)
    drmaa.errors.DeniedByDrmException: code 17: error: no suitable queues

    Exception #3
      'drmaa.errors.DeniedByDrmException(code 17: error: no suitable queues)' raised in ...
       Task = def compute_mean(...):
       Job  = [sample_02.txt -> sample_02.mean]

    Traceback (most recent call last):
      File "/cluster/project8/vyp/JingYu/git/cgat/conda-install/envs/cgat-c/lib/python3.6/site-packages/ruffus/task.py", line 751, in run_pooled_job_without_exceptions
        register_cleanup, touch_files_only)
      File "/cluster/project8/vyp/JingYu/git/cgat/conda-install/envs/cgat-c/lib/python3.6/site-packages/ruffus/task.py", line 567, in job_wrapper_io_files
        ret_val = user_defined_work_func(*params)
      File "/cluster/project8/vyp/JingYu/git/cgat/cgat-core/tests/template_pipeline.py", line 67, in compute_mean
        P.run(statement)
      File "/cluster/project8/vyp/JingYu/git/cgat/cgat-core/CGATCore/Pipeline/Execution.py", line 1328, in run
        benchmark_data = r.run(statement_list)
      File "/cluster/project8/vyp/JingYu/git/cgat/cgat-core/CGATCore/Pipeline/Execution.py", line 910, in run
        job_id = self.session.runJob(jt)
      File "/cluster/project8/vyp/JingYu/git/cgat/conda-install/envs/cgat-c/lib/python3.6/site-packages/drmaa/session.py", line 314, in runJob
        c(drmaa_run_job, jid, sizeof(jid), jobTemplate)
      File "/cluster/project8/vyp/JingYu/git/cgat/conda-install/envs/cgat-c/lib/python3.6/site-packages/drmaa/helpers.py", line 303, in c
        return f(*(args + (error_buffer, sizeof(error_buffer))))
      File "/cluster/project8/vyp/JingYu/git/cgat/conda-install/envs/cgat-c/lib/python3.6/site-packages/drmaa/errors.py", line 151, in error_check
        raise _ERRORS[code - 1](error_string)
    drmaa.errors.DeniedByDrmException: code 17: error: no suitable queues

    Exception #4
      'drmaa.errors.DeniedByDrmException(code 17: error: no suitable queues)' raised in ...
       Task = def compute_mean(...):
       Job  = [sample_03.txt -> sample_03.mean]

    Traceback (most recent call last):
      File "/cluster/project8/vyp/JingYu/git/cgat/conda-install/envs/cgat-c/lib/python3.6/site-packages/ruffus/task.py", line 751, in run_pooled_job_without_exceptions
        register_cleanup, touch_files_only)
      File "/cluster/project8/vyp/JingYu/git/cgat/conda-install/envs/cgat-c/lib/python3.6/site-packages/ruffus/task.py", line 567, in job_wrapper_io_files
        ret_val = user_defined_work_func(*params)
      File "/cluster/project8/vyp/JingYu/git/cgat/cgat-core/tests/template_pipeline.py", line 67, in compute_mean
        P.run(statement)
      File "/cluster/project8/vyp/JingYu/git/cgat/cgat-core/CGATCore/Pipeline/Execution.py", line 1328, in run
        benchmark_data = r.run(statement_list)
      File "/cluster/project8/vyp/JingYu/git/cgat/cgat-core/CGATCore/Pipeline/Execution.py", line 910, in run
        job_id = self.session.runJob(jt)
      File "/cluster/project8/vyp/JingYu/git/cgat/conda-install/envs/cgat-c/lib/python3.6/site-packages/drmaa/session.py", line 314, in runJob
        c(drmaa_run_job, jid, sizeof(jid), jobTemplate)
      File "/cluster/project8/vyp/JingYu/git/cgat/conda-install/envs/cgat-c/lib/python3.6/site-packages/drmaa/helpers.py", line 303, in c
        return f(*(args + (error_buffer, sizeof(error_buffer))))
      File "/cluster/project8/vyp/JingYu/git/cgat/conda-install/envs/cgat-c/lib/python3.6/site-packages/drmaa/errors.py", line 151, in error_check
        raise _ERRORS[code - 1](error_string)
    drmaa.errors.DeniedByDrmException: code 17: error: no suitable queues

    Exception #5
      'drmaa.errors.DeniedByDrmException(code 17: error: no suitable queues)' raised in ...
       Task = def compute_mean(...):
       Job  = [sample_04.txt -> sample_04.mean]

    Traceback (most recent call last):
      File "/cluster/project8/vyp/JingYu/git/cgat/conda-install/envs/cgat-c/lib/python3.6/site-packages/ruffus/task.py", line 751, in run_pooled_job_without_exceptions
        register_cleanup, touch_files_only)
      File "/cluster/project8/vyp/JingYu/git/cgat/conda-install/envs/cgat-c/lib/python3.6/site-packages/ruffus/task.py", line 567, in job_wrapper_io_files
        ret_val = user_defined_work_func(*params)
      File "/cluster/project8/vyp/JingYu/git/cgat/cgat-core/tests/template_pipeline.py", line 67, in compute_mean
        P.run(statement)
      File "/cluster/project8/vyp/JingYu/git/cgat/cgat-core/CGATCore/Pipeline/Execution.py", line 1328, in run
        benchmark_data = r.run(statement_list)
      File "/cluster/project8/vyp/JingYu/git/cgat/cgat-core/CGATCore/Pipeline/Execution.py", line 910, in run
        job_id = self.session.runJob(jt)
      File "/cluster/project8/vyp/JingYu/git/cgat/conda-install/envs/cgat-c/lib/python3.6/site-packages/drmaa/session.py", line 314, in runJob
        c(drmaa_run_job, jid, sizeof(jid), jobTemplate)
      File "/cluster/project8/vyp/JingYu/git/cgat/conda-install/envs/cgat-c/lib/python3.6/site-packages/drmaa/helpers.py", line 303, in c
        return f(*(args + (error_buffer, sizeof(error_buffer))))
      File "/cluster/project8/vyp/JingYu/git/cgat/conda-install/envs/cgat-c/lib/python3.6/site-packages/drmaa/errors.py", line 151, in error_check
        raise _ERRORS[code - 1](error_string)
    drmaa.errors.DeniedByDrmException: code 17: error: no suitable queues

    Exception #6
      'drmaa.errors.DeniedByDrmException(code 17: error: no suitable queues)' raised in ...
       Task = def compute_mean(...):
       Job  = [sample_05.txt -> sample_05.mean]

    Traceback (most recent call last):
      File "/cluster/project8/vyp/JingYu/git/cgat/conda-install/envs/cgat-c/lib/python3.6/site-packages/ruffus/task.py", line 751, in run_pooled_job_without_exceptions
        register_cleanup, touch_files_only)
      File "/cluster/project8/vyp/JingYu/git/cgat/conda-install/envs/cgat-c/lib/python3.6/site-packages/ruffus/task.py", line 567, in job_wrapper_io_files
        ret_val = user_defined_work_func(*params)
      File "/cluster/project8/vyp/JingYu/git/cgat/cgat-core/tests/template_pipeline.py", line 67, in compute_mean
        P.run(statement)
      File "/cluster/project8/vyp/JingYu/git/cgat/cgat-core/CGATCore/Pipeline/Execution.py", line 1328, in run
        benchmark_data = r.run(statement_list)
      File "/cluster/project8/vyp/JingYu/git/cgat/cgat-core/CGATCore/Pipeline/Execution.py", line 910, in run
        job_id = self.session.runJob(jt)
      File "/cluster/project8/vyp/JingYu/git/cgat/conda-install/envs/cgat-c/lib/python3.6/site-packages/drmaa/session.py", line 314, in runJob
        c(drmaa_run_job, jid, sizeof(jid), jobTemplate)
      File "/cluster/project8/vyp/JingYu/git/cgat/conda-install/envs/cgat-c/lib/python3.6/site-packages/drmaa/helpers.py", line 303, in c
        return f(*(args + (error_buffer, sizeof(error_buffer))))
      File "/cluster/project8/vyp/JingYu/git/cgat/conda-install/envs/cgat-c/lib/python3.6/site-packages/drmaa/errors.py", line 151, in error_check
        raise _ERRORS[code - 1](error_string)
    drmaa.errors.DeniedByDrmException: code 17: error: no suitable queues

    Exception #7
      'drmaa.errors.DeniedByDrmException(code 17: error: no suitable queues)' raised in ...
       Task = def compute_mean(...):
       Job  = [sample_06.txt -> sample_06.mean]

    Traceback (most recent call last):
      File "/cluster/project8/vyp/JingYu/git/cgat/conda-install/envs/cgat-c/lib/python3.6/site-packages/ruffus/task.py", line 751, in run_pooled_job_without_exceptions
        register_cleanup, touch_files_only)
      File "/cluster/project8/vyp/JingYu/git/cgat/conda-install/envs/cgat-c/lib/python3.6/site-packages/ruffus/task.py", line 567, in job_wrapper_io_files
        ret_val = user_defined_work_func(*params)
      File "/cluster/project8/vyp/JingYu/git/cgat/cgat-core/tests/template_pipeline.py", line 67, in compute_mean
        P.run(statement)
      File "/cluster/project8/vyp/JingYu/git/cgat/cgat-core/CGATCore/Pipeline/Execution.py", line 1328, in run
        benchmark_data = r.run(statement_list)
      File "/cluster/project8/vyp/JingYu/git/cgat/cgat-core/CGATCore/Pipeline/Execution.py", line 910, in run
        job_id = self.session.runJob(jt)
      File "/cluster/project8/vyp/JingYu/git/cgat/conda-install/envs/cgat-c/lib/python3.6/site-packages/drmaa/session.py", line 314, in runJob
        c(drmaa_run_job, jid, sizeof(jid), jobTemplate)
      File "/cluster/project8/vyp/JingYu/git/cgat/conda-install/envs/cgat-c/lib/python3.6/site-packages/drmaa/helpers.py", line 303, in c
        return f(*(args + (error_buffer, sizeof(error_buffer))))
      File "/cluster/project8/vyp/JingYu/git/cgat/conda-install/envs/cgat-c/lib/python3.6/site-packages/drmaa/errors.py", line 151, in error_check
        raise _ERRORS[code - 1](error_string)
    drmaa.errors.DeniedByDrmException: code 17: error: no suitable queues

    Exception #8
      'drmaa.errors.DeniedByDrmException(code 17: error: no suitable queues)' raised in ...
       Task = def compute_mean(...):
       Job  = [sample_07.txt -> sample_07.mean]

    Traceback (most recent call last):
      File "/cluster/project8/vyp/JingYu/git/cgat/conda-install/envs/cgat-c/lib/python3.6/site-packages/ruffus/task.py", line 751, in run_pooled_job_without_exceptions
        register_cleanup, touch_files_only)
      File "/cluster/project8/vyp/JingYu/git/cgat/conda-install/envs/cgat-c/lib/python3.6/site-packages/ruffus/task.py", line 567, in job_wrapper_io_files
        ret_val = user_defined_work_func(*params)
      File "/cluster/project8/vyp/JingYu/git/cgat/cgat-core/tests/template_pipeline.py", line 67, in compute_mean
        P.run(statement)
      File "/cluster/project8/vyp/JingYu/git/cgat/cgat-core/CGATCore/Pipeline/Execution.py", line 1328, in run
        benchmark_data = r.run(statement_list)
      File "/cluster/project8/vyp/JingYu/git/cgat/cgat-core/CGATCore/Pipeline/Execution.py", line 910, in run
        job_id = self.session.runJob(jt)
      File "/cluster/project8/vyp/JingYu/git/cgat/conda-install/envs/cgat-c/lib/python3.6/site-packages/drmaa/session.py", line 314, in runJob
        c(drmaa_run_job, jid, sizeof(jid), jobTemplate)
      File "/cluster/project8/vyp/JingYu/git/cgat/conda-install/envs/cgat-c/lib/python3.6/site-packages/drmaa/helpers.py", line 303, in c
        return f(*(args + (error_buffer, sizeof(error_buffer))))
      File "/cluster/project8/vyp/JingYu/git/cgat/conda-install/envs/cgat-c/lib/python3.6/site-packages/drmaa/errors.py", line 151, in error_check
        raise _ERRORS[code - 1](error_string)
    drmaa.errors.DeniedByDrmException: code 17: error: no suitable queues

    Exception #9
      'drmaa.errors.DeniedByDrmException(code 17: error: no suitable queues)' raised in ...
       Task = def compute_mean(...):
       Job  = [sample_08.txt -> sample_08.mean]

    Traceback (most recent call last):
      File "/cluster/project8/vyp/JingYu/git/cgat/conda-install/envs/cgat-c/lib/python3.6/site-packages/ruffus/task.py", line 751, in run_pooled_job_without_exceptions
        register_cleanup, touch_files_only)
      File "/cluster/project8/vyp/JingYu/git/cgat/conda-install/envs/cgat-c/lib/python3.6/site-packages/ruffus/task.py", line 567, in job_wrapper_io_files
        ret_val = user_defined_work_func(*params)
      File "/cluster/project8/vyp/JingYu/git/cgat/cgat-core/tests/template_pipeline.py", line 67, in compute_mean
        P.run(statement)
      File "/cluster/project8/vyp/JingYu/git/cgat/cgat-core/CGATCore/Pipeline/Execution.py", line 1328, in run
        benchmark_data = r.run(statement_list)
      File "/cluster/project8/vyp/JingYu/git/cgat/cgat-core/CGATCore/Pipeline/Execution.py", line 910, in run
        job_id = self.session.runJob(jt)
      File "/cluster/project8/vyp/JingYu/git/cgat/conda-install/envs/cgat-c/lib/python3.6/site-packages/drmaa/session.py", line 314, in runJob
        c(drmaa_run_job, jid, sizeof(jid), jobTemplate)
      File "/cluster/project8/vyp/JingYu/git/cgat/conda-install/envs/cgat-c/lib/python3.6/site-packages/drmaa/helpers.py", line 303, in c
        return f(*(args + (error_buffer, sizeof(error_buffer))))
      File "/cluster/project8/vyp/JingYu/git/cgat/conda-install/envs/cgat-c/lib/python3.6/site-packages/drmaa/errors.py", line 151, in error_check
        raise _ERRORS[code - 1](error_string)
    drmaa.errors.DeniedByDrmException: code 17: error: no suitable queues

    Exception #10
      'drmaa.errors.DeniedByDrmException(code 17: error: no suitable queues)' raised in ...
       Task = def compute_mean(...):
       Job  = [sample_09.txt -> sample_09.mean]

    Traceback (most recent call last):
      File "/cluster/project8/vyp/JingYu/git/cgat/conda-install/envs/cgat-c/lib/python3.6/site-packages/ruffus/task.py", line 751, in run_pooled_job_without_exceptions
        register_cleanup, touch_files_only)
      File "/cluster/project8/vyp/JingYu/git/cgat/conda-install/envs/cgat-c/lib/python3.6/site-packages/ruffus/task.py", line 567, in job_wrapper_io_files
        ret_val = user_defined_work_func(*params)
      File "/cluster/project8/vyp/JingYu/git/cgat/cgat-core/tests/template_pipeline.py", line 67, in compute_mean
        P.run(statement)
      File "/cluster/project8/vyp/JingYu/git/cgat/cgat-core/CGATCore/Pipeline/Execution.py", line 1328, in run
        benchmark_data = r.run(statement_list)
      File "/cluster/project8/vyp/JingYu/git/cgat/cgat-core/CGATCore/Pipeline/Execution.py", line 910, in run
        job_id = self.session.runJob(jt)
      File "/cluster/project8/vyp/JingYu/git/cgat/conda-install/envs/cgat-c/lib/python3.6/site-packages/drmaa/session.py", line 314, in runJob
        c(drmaa_run_job, jid, sizeof(jid), jobTemplate)
      File "/cluster/project8/vyp/JingYu/git/cgat/conda-install/envs/cgat-c/lib/python3.6/site-packages/drmaa/helpers.py", line 303, in c
        return f(*(args + (error_buffer, sizeof(error_buffer))))
      File "/cluster/project8/vyp/JingYu/git/cgat/conda-install/envs/cgat-c/lib/python3.6/site-packages/drmaa/errors.py", line 151, in error_check
        raise _ERRORS[code - 1](error_string)
    drmaa.errors.DeniedByDrmException: code 17: error: no suitable queues

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/cluster/project8/vyp/JingYu/git/cgat/cgat-core/tests/template_pipeline.py", line 126, in <module>
    sys.exit(main(sys.argv))
  File "/cluster/project8/vyp/JingYu/git/cgat/cgat-core/tests/template_pipeline.py", line 122, in main
    return P.run_workflow(options, args)
  File "/cluster/project8/vyp/JingYu/git/cgat/cgat-core/CGATCore/Pipeline/Control.py", line 1252, in run_workflow
    raise ValueError("pipeline failed with %i errors" % len(ex.args)) from ex
ValueError: pipeline failed with 10 errors

----------------------------------------------------------------------
Ran 42 tests in 165.684s

FAILED (failures=4, errors=14)
sebastian-luna-valero commented 6 years ago

Hi @logust79

Thanks for your feedback.

I guess this is related to the hard-coded parameters in: https://github.com/cgat-developers/cgat-core/blob/master/CGATCore/Pipeline/Parameters.py#L67

We use several defaults that work in our SGE but won't in yours:

Here are the commands you need to get the correct value for those parameters in your SGE install:

# get list of submission queues
qconf -sql

# get list of parallel environments
qconf -spl

# get a list of all consumable resources
qconf -sc

You would need to create your own pipeline.yml and configure those values correctly.

Best regards, Sebastian

PD: This should all be documented but it is not currently. I will be updating the docs after the code is ready for production.

logust79 commented 6 years ago

Hi @sebastian-luna-valero After adding the following line to pipeline.yml it now works!

cluster:
    memory_resource: tmem
    parallel_environment: mpi

Many thanks! Jing