Error at first stage of pipeline

[jasonshiller]

Hi,

We are having a bit of trouble getting Red dog to run on our cluster. At this stage we can't seem to get past the first step. We keep getting the error below.

Thanks, Jason

Starting pipeline... 427 jobs to be executed in total stage = makeDir, jobID = 1692894 Traceback (most recent call last): File "/usr/local/bin/rubra", line 9, in load_entry_point('Rubra==0.1.5', 'console_scripts', 'rubra')() File "/usr/local/lib/python2.7/site-packages/Rubra-0.1.5-py2.7.egg/rubra/rubra.py", line 66, in main gnu_make_maximal_rebuild_mode=rebuildMode) File "/usr/local/lib/python2.7/site-packages/ruffus-2.2-py2.7.egg/ruffus/task.py", line 2680, in pipeline_run raise errt ruffus.ruffus_exceptions.RethrownJobError:

Exceptions running jobs for

'def RedDog.makeDir(...):'

Original exception:

Exception #1
exceptions.IOError([Errno 2] No such file or directory: '/group/pathogens/Ralstonia/pseudomonas/temp/success/dir.makeDir.Success'):
for RedDog.makeDir.Job = [False -> dir.makeDir.Success]

Traceback (most recent call last):
  File "/usr/local/lib/python2.7/site-packages/ruffus-2.2-py2.7.egg/ruffus/task.py", line 517, in run_pooled_job_without_exceptions
    return_value =  job_wrapper(param, user_defined_work_func, register_cleanup, touch_files_only)
  File "/usr/local/lib/python2.7/site-packages/ruffus-2.2-py2.7.egg/ruffus/task.py", line 447, in job_wrapper_io_files
    ret_val = user_defined_work_func(*param)
  File "RedDog.py", line 953, in makeDir
    runStageCheck('makeDir', flagFile, outPrefix, full_sequence_list_string)
  File "/usr/local/lib/python2.7/site-packages/Rubra-0.1.5-py2.7.egg/rubra/utils.py", line 130, in runStageCheck
    open(flag_file, 'w').close()
IOError: [Errno 2] No such file or directory: '/group/pathogens/Ralstonia/pseudomonas/temp/success/dir.makeDir.Success'

[katholt]

I just got this too, with the latest version cloned from the repo. Must be something that was introduced in the last fix David?

[jasonshiller]

Sorry should have mentioned the version - V1beta.5 020915

[katholt]

I was using V1beta.7 (BlackCat)

[katholt]

Actually I'm not sure if it's the same error. With me it is:

Starting pipeline... 19 jobs to be executed in total stage = makeDir Error: command failed: python makeDir.py /vlsci/VR0082/shared/data/moraxella/catarrhalis/CAS_deepseq1/RDv1b7_231115/ stage = makeRef Error: command failed: python convertGenbankToFasta.py /vlsci/VR0082/shared/data/moraxella/catarrhalis/genbank/NC_014147.gbk /vlsci/VR0082/shared/data/moraxella/catarrhalis/CAS_deepseq1/RDv1b7_231115/temp/NC_014147.fasta Traceback (most recent call last): File "/usr/local/python/2.7.5-gcc/bin/rubra", line 9, in load_entry_point('Rubra==0.1.5', 'console_scripts', 'rubra')() File "/usr/local/python/2.7.5-gcc/lib/python2.7/site-packages/rubra/rubra.py", line 66, in main gnu_make_maximal_rebuild_mode=rebuildMode) File "build/bdist.linux-x86_64/egg/ruffus/task.py", line 2671, in pipeline_run File "build/bdist.linux-x86_64/egg/ruffus/task.py", line 2406, in fill_queue_with_job_parameters File "build/bdist.linux-x86_64/egg/ruffus/task.py", line 2345, in parameter_generator ruffus.ruffus_exceptions.RethrownJobError:

Exceptions generating parameters for

'def RedDog.indexRef(...):'

Original exception:

Exception #1
ruffus.ruffus_exceptions.MissingInputFileError(    

    No way to run job: Input file ['/vlsci/VR0082/shared/data/moraxella/catarrhalis/CAS_deepseq1/RDv1b7_231115/temp/NC_014147.fasta'] does not exist):
for RedDog.indexRef.

Traceback (most recent call last):
  File "build/bdist.linux-x86_64/egg/ruffus/task.py", line 2279, in parameter_generator
    check_input_files_exist (*param)
  File "build/bdist.linux-x86_64/egg/ruffus/file_name_parameters.py", line 191, in check_input_files_exist
    "Input file ['%s'] does not exist" % f)
MissingInputFileError:     

    No way to run job: Input file ['/vlsci/VR0082/shared/data/moraxella/catarrhalis/CAS_deepseq1/RDv1b7_231115/temp/NC_014147.fasta'] does not exist

[d-j-e]

Hi Jason, Are you running on Barcoo or any other VLSCI-hosted system?

cheers, David

[jasonshiller]

Hi David,

No it is running on our cluster at AgriBio. Cheers, Jason

[d-j-e]

In the log folder, is there a err file for the makeDir stage?

[jasonshiller]

yes there is but it is empty

[d-j-e]

Okay - I should ask first, are you using a distributed system (i.e you have to queue jobs) or a dedicated server? If its a distributed system, which queuing system are you using (E.G. slurm, torque, or something else...)?

[jasonshiller]

We are running torque

[d-j-e]

Which version of rubra did you install?

[jasonshiller]

It is version 0.1.5

[d-j-e]

Can you send me your config file via davidje@student(dot)unimelb(dot)edu(dot)au ?

[jasonshiller]

Thanks I will send that now