ComparativeGenomicsToolkit / Comparative-Annotation-Toolkit

Apache License 2.0
157 stars 49 forks source link

Error with luigi #163

Open YPGG1234 opened 4 years ago

YPGG1234 commented 4 years ago

Hi,

Recently I have installing cat, I think I had installed everthing except little bit wrong with Augustus (I got it from bioconda, version 3.3.2, but I think it cannot config as you said). When I run :

luigi --module cat RunCat --hal=test_data/vertebrates.hal --ref-genome=mm10 --workers=10 --config=test_data/test.config --work-dir test_install --out-dir test_install --local-scheduler --augustus --augustus-cgp --augustus-pb --assembly-hub

I got :

03-30 10:10:19 luigi-interface WARNING Will not run Task: RunCat or any dependencies due to error in complete() method:
Traceback (most recent call last):
  File "/home/liran/Software/miniconda2/envs/C-A-T/lib/python2.7/site-packages/luigi/worker.py", line 409, in check_complete
    is_complete = task.complete()
  File "/home/liran/Software/miniconda2/envs/C-A-T/lib/python2.7/site-packages/luigi/task.py", line 826, in complete
    return all(r.complete() for r in flatten(self.requires()))
  File "/home/liran/Software/miniconda2/envs/C-A-T/lib/python2.7/site-packages/luigi/task.py", line 887, in flatten
    for result in iterator:
  File "/home/liran/Software/C-A-T/Comparative-Annotation-Toolkit/cat/__init__.py", line 624, in requires
    pipeline_args = self.get_pipeline_args()
  File "/home/liran/Software/C-A-T/Comparative-Annotation-Toolkit/cat/__init__.py", line 211, in get_pipeline_args
    if not tools.misc.is_exec('docker'):
  File "/home/liran/Software/C-A-T/Comparative-Annotation-Toolkit/tools/misc.py", line 80, in is_exec
    raise ToolMissingException("{0} not found. Either install {0}, or install CAT's dependencies and use --binary-mode local.".format(binary_mode))
ToolMissingException: docker not found. Either install docker, or install CAT's dependencies and use --binary-mode local.

03-30 10:10:19 luigi-interface INFO  Informed scheduler that task   RunCat_False_True_True_a22c0dbb99   has status   UNKNOWN
03-30 10:10:19 luigi-interface INFO  Done scheduling tasks
03-30 10:10:19 luigi-interface INFO  Running Worker with 10 processes
03-30 10:10:19 luigi-interface INFO  Worker Worker(salt=976387333, workers=10, host=subnet190-2.nwsuaf.edu.cn, username=liran, pid=31552) was stopped. Shutting down Keep-Alive thread
03-30 10:10:19 luigi-interface INFO
===== Luigi Execution Summary =====

Scheduled 1 tasks of which:
* 1 failed scheduling:
    - 1 RunCat(...)

Did not run any tasks
This progress looks :( because there were tasks whose scheduling failed

===== Luigi Execution Summary =====

When I check out the log dir, I got :

luigi-server.log 2020-03-30 09:52:08,921 luigi.scheduler[22095] INFO: No prior state file exists at /var/lib/luigi-server/state.pickle. Starting with empty state

luigi-server-2020-03-30.err socket.error: [Errno 98] Address already in use

Can you help me to sovle it? Thank you

ifiddes commented 4 years ago

If you are compiling binaries yourself and not using docker, you need to run CAT with --binary-mode local.

Ian Fiddes, PhD

On Sun, Mar 29, 2020 at 7:28 PM YPGG1234 notifications@github.com wrote:

Hi,

Recently I have installing cat, I think I had installed everthing except little bit wrong with Augustus (I got it from bioconda, version 3.3.2, but I think it cannot config as you said). When I run :

luigi --module cat RunCat --hal=test_data/vertebrates.hal --ref-genome=mm10 --workers=10 --config=test_data/test.config --work-dir test_install --out-dir test_install --local-scheduler --augustus --augustus-cgp --augustus-pb --assembly-hub

I got :

03-30 10:10:19 luigi-interface WARNING Will not run Task: RunCat or any dependencies due to error in complete() method: Traceback (most recent call last): File "/home/liran/Software/miniconda2/envs/C-A-T/lib/python2.7/site-packages/luigi/worker.py", line 409, in check_complete is_complete = task.complete() File "/home/liran/Software/miniconda2/envs/C-A-T/lib/python2.7/site-packages/luigi/task.py", line 826, in complete return all(r.complete() for r in flatten(self.requires())) File "/home/liran/Software/miniconda2/envs/C-A-T/lib/python2.7/site-packages/luigi/task.py", line 887, in flatten for result in iterator: File "/home/liran/Software/C-A-T/Comparative-Annotation-Toolkit/cat/init.py", line 624, in requires pipeline_args = self.get_pipeline_args() File "/home/liran/Software/C-A-T/Comparative-Annotation-Toolkit/cat/init.py", line 211, in get_pipeline_args if not tools.misc.is_exec('docker'): File "/home/liran/Software/C-A-T/Comparative-Annotation-Toolkit/tools/misc.py", line 80, in is_exec raise ToolMissingException("{0} not found. Either install {0}, or install CAT's dependencies and use --binary-mode local.".format(binary_mode)) ToolMissingException: docker not found. Either install docker, or install CAT's dependencies and use --binary-mode local.

03-30 10:10:19 luigi-interface INFO Informed scheduler that task RunCat_False_True_True_a22c0dbb99 has status UNKNOWN 03-30 10:10:19 luigi-interface INFO Done scheduling tasks 03-30 10:10:19 luigi-interface INFO Running Worker with 10 processes 03-30 10:10:19 luigi-interface INFO Worker Worker(salt=976387333, workers=10, host=subnet190-2.nwsuaf.edu.cn, username=liran, pid=31552) was stopped. Shutting down Keep-Alive thread 03-30 10:10:19 luigi-interface INFO ===== Luigi Execution Summary =====

Scheduled 1 tasks of which:

  • 1 failed scheduling:
    • 1 RunCat(...)

Did not run any tasks This progress looks :( because there were tasks whose scheduling failed

===== Luigi Execution Summary =====

When I check out the log dir, I got :

luigi-server.log 2020-03-30 09:52:08,921 luigi.scheduler[22095] INFO: No prior state file exists at /var/lib/luigi-server/state.pickle. Starting with empty state

luigi-server-2020-03-30.err socket.error: [Errno 98] Address already in use

Can you help me to sovle it? Thank you

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/ComparativeGenomicsToolkit/Comparative-Annotation-Toolkit/issues/163, or unsubscribe https://github.com/notifications/unsubscribe-auth/AB3UWXLTOBYH2HPURE3DPMLRJ77TJANCNFSM4LWHG7NQ .

YPGG1234 commented 4 years ago

Thanks for your mention, I have tried it , but I got :

03-30 10:33:48 luigi-interface INFO  Informed scheduler that task   RunCat_False_True_True_a22c0dbb99   has status   PENDING
03-30 10:33:48 luigi-interface INFO  Informed scheduler that task   ReportStats_False_True_True_a22c0dbb99   has status   PENDING
03-30 10:33:48 luigi-interface INFO  Informed scheduler that task   AssemblyHub_False_True_True_a22c0dbb99   has status   PENDING
03-30 10:33:48 luigi-interface INFO  Informed scheduler that task   CreateTracks_False_True_True_a22c0dbb99   has status   PENDING
03-30 10:33:48 luigi-interface INFO  Informed scheduler that task   CreateTrackDbs_False_True_True_2bb603043f   has status   PENDING
03-30 10:33:48 luigi-interface INFO  Informed scheduler that task   CreateTracksDriverTask_False_True_True_2bb603043f   has status   PENDING
03-30 10:33:49 luigi-interface INFO  Informed scheduler that task   SpliceTrack_False_True_True_414df3c083   has status   PENDING
03-30 10:33:49 luigi-interface INFO  Informed scheduler that task   ReferenceFiles_False_True_True_a22c0dbb99   has status   DONE
03-30 10:33:49 luigi-interface INFO  Informed scheduler that task   BgpTrack_False_True_True_ef8b57a4b3   has status   PENDING
03-30 10:33:49 luigi-interface INFO  Informed scheduler that task   CreateTrackDbs_False_True_True_962ce509b6   has status   PENDING
03-30 10:33:49 luigi-interface INFO  Informed scheduler that task   CreateTracksDriverTask_False_True_True_962ce509b6   has status   PENDING
03-30 10:33:49 luigi-interface INFO  Informed scheduler that task   SpliceTrack_False_True_True_0ba10fdc5e   has status   PENDING
03-30 10:33:49 luigi-interface INFO  Informed scheduler that task   AugustusTrack_False_True_True_fe5b130f82   has status   PENDING
03-30 10:33:49 luigi-interface INFO  Informed scheduler that task   BgpTrack_False_True_True_b3c6e38954   has status   PENDING
03-30 10:33:49 luigi-interface INFO  Informed scheduler that task   TransMapTrack_False_True_True_d057e76fa7   has status   PENDING
03-30 10:33:49 luigi-interface INFO  Informed scheduler that task   EvaluationTrack_False_True_True_e09c287d51   has status   PENDING
03-30 10:33:49 luigi-interface INFO  Informed scheduler that task   ConsensusTrack_False_True_True_899aaff4b5   has status   PENDING
03-30 10:33:49 luigi-interface INFO  Informed scheduler that task   DenovoTrack_False_True_True_19d83a84cd   has status   PENDING
03-30 10:33:49 luigi-interface INFO  Informed scheduler that task   CreateTrackDbs_False_True_True_a56ac842ed   has status   PENDING
03-30 10:33:49 luigi-interface INFO  Informed scheduler that task   CreateTracksDriverTask_False_True_True_a56ac842ed   has status   PENDING
03-30 10:33:49 luigi-interface INFO  Informed scheduler that task   BgpTrack_False_True_True_a045b3eed3   has status   PENDING
03-30 10:33:49 luigi-interface INFO  Informed scheduler that task   TransMapTrack_False_True_True_f8c1c7c8f1   has status   PENDING
03-30 10:33:49 luigi-interface INFO  Informed scheduler that task   EvaluationTrack_False_True_True_1369572bbd   has status   PENDING
03-30 10:33:49 luigi-interface INFO  Informed scheduler that task   ConsensusTrack_False_True_True_2e26a53515   has status   PENDING
03-30 10:33:49 luigi-interface INFO  Informed scheduler that task   DenovoTrack_False_True_True_4964d40230   has status   PENDING
03-30 10:33:49 luigi-interface INFO  Informed scheduler that task   CreateTrackDbs_False_True_True_5e66b93a27   has status   PENDING
03-30 10:33:49 luigi-interface INFO  Informed scheduler that task   CreateTracksDriverTask_False_True_True_5e66b93a27   has status   PENDING
03-30 10:33:49 luigi-interface INFO  Informed scheduler that task   BgpTrack_False_True_True_06e82f2455   has status   PENDING
03-30 10:33:49 luigi-interface INFO  Informed scheduler that task   TransMapTrack_False_True_True_247bc669c6   has status   PENDING
03-30 10:33:49 luigi-interface INFO  Informed scheduler that task   EvaluationTrack_False_True_True_1fb03ca6f2   has status   PENDING
03-30 10:33:49 luigi-interface INFO  Informed scheduler that task   ConsensusTrack_False_True_True_22451c1436   has status   PENDING
03-30 10:33:49 luigi-interface INFO  Informed scheduler that task   DenovoTrack_False_True_True_07fbf61039   has status   PENDING
03-30 10:33:49 luigi-interface INFO  Informed scheduler that task   CreateTrackDbs_False_True_True_ffd7733252   has status   PENDING
03-30 10:33:49 luigi-interface INFO  Informed scheduler that task   CreateTracksDriverTask_False_True_True_ffd7733252   has status   PENDING
03-30 10:33:49 luigi-interface INFO  Informed scheduler that task   BgpTrack_False_True_True_4e47ec61fc   has status   PENDING
03-30 10:33:49 luigi-interface INFO  Informed scheduler that task   TransMapTrack_False_True_True_d1c0ce3423   has status   PENDING
03-30 10:33:49 luigi-interface INFO  Informed scheduler that task   EvaluationTrack_False_True_True_0df56c9568   has status   PENDING
03-30 10:33:49 luigi-interface INFO  Informed scheduler that task   ConsensusTrack_False_True_True_16101cfad8   has status   PENDING
03-30 10:33:49 luigi-interface INFO  Informed scheduler that task   DenovoTrack_False_True_True_ac89a6e910   has status   PENDING
03-30 10:33:49 luigi-interface INFO  Informed scheduler that task   CreateTrackDbs_False_True_True_b2ea75e80d   has status   PENDING
03-30 10:33:49 luigi-interface INFO  Informed scheduler that task   CreateTracksDriverTask_False_True_True_b2ea75e80d   has status   PENDING
03-30 10:33:49 luigi-interface INFO  Informed scheduler that task   SpliceTrack_False_True_True_a2c639d7d0   has status   PENDING
03-30 10:33:49 luigi-interface INFO  Informed scheduler that task   AugustusTrack_False_True_True_eaa4ef7f4c   has status   PENDING
03-30 10:33:49 luigi-interface INFO  Informed scheduler that task   BgpTrack_False_True_True_6bb68024a2   has status   PENDING
03-30 10:33:49 luigi-interface INFO  Informed scheduler that task   TransMapTrack_False_True_True_26942e186f   has status   PENDING
03-30 10:33:49 luigi-interface INFO  Informed scheduler that task   EvaluationTrack_False_True_True_d4b3522d4f   has status   PENDING
03-30 10:33:49 luigi-interface INFO  Informed scheduler that task   ConsensusTrack_False_True_True_3e54a7a8a3   has status   PENDING
03-30 10:33:49 luigi-interface INFO  Informed scheduler that task   DenovoTrack_False_True_True_20652ea98b   has status   PENDING
03-30 10:33:49 luigi-interface INFO  Informed scheduler that task   CreateTrackDbs_False_True_True_df60fe5db8   has status   PENDING
03-30 10:33:49 luigi-interface INFO  Informed scheduler that task   CreateTracksDriverTask_False_True_True_df60fe5db8   has status   PENDING
03-30 10:33:49 luigi-interface INFO  Informed scheduler that task   SpliceTrack_False_True_True_c06a29233f   has status   PENDING
03-30 10:33:49 luigi-interface INFO  Informed scheduler that task   IsoSeqBamTrack_False_True_True_174d8e5f0a   has status   PENDING
03-30 10:33:49 luigi-interface INFO  Informed scheduler that task   AugustusTrack_False_True_True_94fd6fc914   has status   PENDING
03-30 10:33:49 luigi-interface INFO  Informed scheduler that task   BgpTrack_False_True_True_44ade36bc4   has status   PENDING
03-30 10:33:49 luigi-interface INFO  Informed scheduler that task   TransMapTrack_False_True_True_c7fa399adc   has status   PENDING
03-30 10:33:49 luigi-interface INFO  Informed scheduler that task   EvaluationTrack_False_True_True_3e752a8281   has status   PENDING
03-30 10:33:49 luigi-interface INFO  Informed scheduler that task   ConsensusTrack_False_True_True_df65d3091f   has status   PENDING
03-30 10:33:49 luigi-interface INFO  Informed scheduler that task   DenovoTrack_False_True_True_bbb549ac8d   has status   PENDING
03-30 10:33:49 luigi-interface INFO  Informed scheduler that task   DenovoTrack_False_True_True_6fa8e23eac   has status   PENDING
03-30 10:33:49 luigi-interface INFO  Informed scheduler that task   CreateTrackDbs_False_True_True_185cafd88d   has status   PENDING
03-30 10:33:49 luigi-interface INFO  Informed scheduler that task   CreateTracksDriverTask_False_True_True_185cafd88d   has status   PENDING
03-30 10:33:49 luigi-interface INFO  Informed scheduler that task   BgpTrack_False_True_True_a52a90a819   has status   PENDING
03-30 10:33:49 luigi-interface INFO  Informed scheduler that task   TransMapTrack_False_True_True_28e168baa8   has status   PENDING
03-30 10:33:49 luigi-interface INFO  Informed scheduler that task   EvaluationTrack_False_True_True_44b3b1b596   has status   PENDING
03-30 10:33:49 luigi-interface INFO  Informed scheduler that task   ConsensusTrack_False_True_True_3d71bb2e15   has status   PENDING
03-30 10:33:49 luigi-interface INFO  Informed scheduler that task   DenovoTrack_False_True_True_efbd446679   has status   PENDING
03-30 10:33:49 luigi-interface INFO  Informed scheduler that task   CreateDirectoryStructure_False_True_True_a22c0dbb99   has status   PENDING
03-30 10:33:49 luigi-interface INFO  Informed scheduler that task   Plots_False_True_True_a22c0dbb99   has status   PENDING
03-30 10:33:49 luigi-interface INFO  Informed scheduler that task   Consensus_False_True_True_a22c0dbb99   has status   PENDING
03-30 10:33:49 luigi-interface INFO  Informed scheduler that task   ConsensusDriverTask_False_True_True_962ce509b6   has status   PENDING
03-30 10:33:49 luigi-interface INFO  Informed scheduler that task   ConsensusDriverTask_False_True_True_a56ac842ed   has status   PENDING
03-30 10:33:49 luigi-interface INFO  Informed scheduler that task   ConsensusDriverTask_False_True_True_5e66b93a27   has status   PENDING
03-30 10:33:49 luigi-interface INFO  Informed scheduler that task   ConsensusDriverTask_False_True_True_ffd7733252   has status   PENDING
03-30 10:33:49 luigi-interface INFO  Informed scheduler that task   ConsensusDriverTask_False_True_True_b2ea75e80d   has status   PENDING
03-30 10:33:49 luigi-interface INFO  Informed scheduler that task   ConsensusDriverTask_False_True_True_df60fe5db8   has status   PENDING
03-30 10:33:49 luigi-interface INFO  Informed scheduler that task   ConsensusDriverTask_False_True_True_185cafd88d   has status   PENDING
03-30 10:33:49 luigi-interface INFO  Informed scheduler that task   EvaluateTranscripts_False_True_True_a22c0dbb99   has status   PENDING
03-30 10:33:49 luigi-interface INFO  Informed scheduler that task   EvaluateDriverTask_False_True_True_962ce509b6   has status   PENDING
03-30 10:33:49 luigi-interface INFO  Informed scheduler that task   EvaluateDriverTask_False_True_True_a56ac842ed   has status   PENDING
03-30 10:33:49 luigi-interface INFO  Informed scheduler that task   EvaluateDriverTask_False_True_True_5e66b93a27   has status   PENDING
03-30 10:33:49 luigi-interface INFO  Informed scheduler that task   EvaluateDriverTask_False_True_True_ffd7733252   has status   PENDING
03-30 10:33:49 luigi-interface INFO  Informed scheduler that task   EvaluateDriverTask_False_True_True_b2ea75e80d   has status   PENDING
03-30 10:33:49 luigi-interface INFO  Informed scheduler that task   EvaluateDriverTask_False_True_True_df60fe5db8   has status   PENDING
03-30 10:33:49 luigi-interface INFO  Informed scheduler that task   EvaluateDriverTask_False_True_True_185cafd88d   has status   PENDING
03-30 10:33:50 luigi-interface INFO  Informed scheduler that task   AlignTranscripts_False_True_True_a22c0dbb99   has status   PENDING
03-30 10:33:50 luigi-interface INFO  Informed scheduler that task   AlignTranscriptDriverTask_False_True_True_962ce509b6   has status   PENDING
03-30 10:33:50 luigi-interface INFO  Informed scheduler that task   GenomeFiles_False_True_True_a22c0dbb99   has status   DONE
03-30 10:33:50 luigi-interface INFO  Informed scheduler that task   AlignTranscriptDriverTask_False_True_True_a56ac842ed   has status   PENDING
03-30 10:33:50 luigi-interface INFO  Informed scheduler that task   AlignTranscriptDriverTask_False_True_True_5e66b93a27   has status   PENDING
03-30 10:33:50 luigi-interface INFO  Informed scheduler that task   AlignTranscriptDriverTask_False_True_True_ffd7733252   has status   PENDING
03-30 10:33:50 luigi-interface INFO  Informed scheduler that task   AlignTranscriptDriverTask_False_True_True_b2ea75e80d   has status   PENDING
03-30 10:33:50 luigi-interface INFO  Informed scheduler that task   AlignTranscriptDriverTask_False_True_True_df60fe5db8   has status   PENDING
03-30 10:33:50 luigi-interface INFO  Informed scheduler that task   AlignTranscriptDriverTask_False_True_True_185cafd88d   has status   PENDING
03-30 10:33:50 luigi-interface INFO  Informed scheduler that task   Hgm_False_True_True_a22c0dbb99   has status   PENDING
03-30 10:33:50 luigi-interface INFO  Informed scheduler that task   HgmDriverTask_False_True_True_e079ff495e   has status   PENDING
03-30 10:33:50 luigi-interface INFO  Informed scheduler that task   HgmDriverTask_False_True_True_2f9793b986   has status   PENDING
03-30 10:33:50 luigi-interface INFO  Informed scheduler that task   HgmDriverTask_False_True_True_3324565358   has status   PENDING
03-30 10:33:50 luigi-interface INFO  Informed scheduler that task   HgmDriverTask_False_True_True_6cb70a33d5   has status   PENDING
03-30 10:33:50 luigi-interface INFO  Informed scheduler that task   HgmDriverTask_False_True_True_a7f6ffd80e   has status   PENDING
03-30 10:33:50 luigi-interface INFO  Informed scheduler that task   IsoSeqTranscripts_False_True_True_a22c0dbb99   has status   PENDING
03-30 10:33:50 luigi-interface INFO  Informed scheduler that task   IsoSeqTranscriptsDriverTask_False_True_True_df60fe5db8   has status   PENDING
03-30 10:33:50 luigi-interface INFO  Informed scheduler that task   FindDenovoParents_False_True_True_e079ff495e   has status   PENDING
03-30 10:33:50 luigi-interface INFO  Informed scheduler that task   AugustusPb_False_True_True_a22c0dbb99   has status   PENDING
03-30 10:33:50 luigi-interface INFO  Informed scheduler that task   AugustusPbDriverTask_False_True_True_df60fe5db8   has status   PENDING
03-30 10:33:50 luigi-interface INFO  Informed scheduler that task   FindDenovoParents_False_True_True_6cb70a33d5   has status   PENDING
03-30 10:33:50 luigi-interface INFO  Informed scheduler that task   AugustusCgp_False_True_True_a22c0dbb99   has status   PENDING
03-30 10:33:50 luigi-interface INFO  Informed scheduler that task   Augustus_False_True_True_a22c0dbb99   has status   PENDING
03-30 10:33:50 luigi-interface INFO  Informed scheduler that task   AugustusDriverTask_False_True_True_962ce509b6   has status   PENDING
03-30 10:33:50 luigi-interface INFO  Informed scheduler that task   AugustusDriverTask_False_True_True_a56ac842ed   has status   PENDING
03-30 10:33:50 luigi-interface INFO  Informed scheduler that task   AugustusDriverTask_False_True_True_5e66b93a27   has status   PENDING
03-30 10:33:50 luigi-interface INFO  Informed scheduler that task   AugustusDriverTask_False_True_True_ffd7733252   has status   PENDING
03-30 10:33:50 luigi-interface INFO  Informed scheduler that task   AugustusDriverTask_False_True_True_b2ea75e80d   has status   PENDING
03-30 10:33:50 luigi-interface INFO  Informed scheduler that task   AugustusDriverTask_False_True_True_df60fe5db8   has status   PENDING
03-30 10:33:50 luigi-interface INFO  Informed scheduler that task   AugustusDriverTask_False_True_True_185cafd88d   has status   PENDING
03-30 10:33:50 luigi-interface INFO  Informed scheduler that task   EvaluateTransMap_False_True_True_a22c0dbb99   has status   DONE
03-30 10:33:50 luigi-interface INFO  Informed scheduler that task   TransMap_False_True_True_a22c0dbb99   has status   DONE
03-30 10:33:50 luigi-interface INFO  Informed scheduler that task   Chaining_False_True_True_a22c0dbb99   has status   DONE
03-30 10:33:50 luigi-interface INFO  Informed scheduler that task   BuildDb_False_True_True_a22c0dbb99   has status   PENDING
03-30 10:33:50 luigi-interface INFO  Informed scheduler that task   GenerateHints_False_True_True_062470fd8f   has status   PENDING
03-30 10:33:50 luigi-interface INFO  Informed scheduler that task   GenerateHints_False_True_True_7ca55844cb   has status   PENDING
03-30 10:33:50 luigi-interface INFO  Informed scheduler that task   GenerateHints_False_True_True_38092cda82   has status   DONE
03-30 10:33:50 luigi-interface INFO  Informed scheduler that task   GenerateHints_False_True_True_59d3d5afbf   has status   DONE
03-30 10:33:50 luigi-interface INFO  Informed scheduler that task   GenerateHints_False_True_True_d94badbc23   has status   DONE
03-30 10:33:50 luigi-interface INFO  Informed scheduler that task   GenerateHints_False_True_True_0ccd06bf5f   has status   PENDING
03-30 10:33:50 luigi-interface INFO  Informed scheduler that task   GenerateHints_False_True_True_aa7607109d   has status   PENDING
03-30 10:33:50 luigi-interface INFO  Informed scheduler that task   GenerateHints_False_True_True_baf78e6f16   has status   DONE
03-30 10:33:50 luigi-interface INFO  Informed scheduler that task   PrepareFiles_False_True_True_a22c0dbb99   has status   DONE
03-30 10:33:50 luigi-interface INFO  Done scheduling tasks
03-30 10:33:50 luigi-interface INFO  Running Worker with 10 processes
03-30 10:33:50 luigi-interface INFO  [pid 35764] Worker Worker(salt=724431596, workers=10, host=subnet190-2.nwsuaf.edu.cn, username=liran, pid=35207) running   ToilTask: GenerateHints for rheMac3 using batchSystem singleMachine
03-30 10:33:50 luigi-interface INFO  [pid 35765] Worker Worker(salt=724431596, workers=10, host=subnet190-2.nwsuaf.edu.cn, username=liran, pid=35207) running   ToilTask: GenerateHints for hg38 using batchSystem singleMachine
03-30 10:33:50 luigi-interface INFO  [pid 35767] Worker Worker(salt=724431596, workers=10, host=subnet190-2.nwsuaf.edu.cn, username=liran, pid=35207) running   ToilTask: GenerateHints for galGal4 using batchSystem singleMachine
03-30 10:33:50 luigi-interface INFO  [pid 35770] Worker Worker(salt=724431596, workers=10, host=subnet190-2.nwsuaf.edu.cn, username=liran, pid=35207) running   ToilTask: GenerateHints for mm10 using batchSystem singleMachine
03-30 10:33:51 cat   INFO  Beginning GenerateHints Toil pipeline for rheMac3.
03-30 10:33:51 cat   INFO  Beginning GenerateHints Toil pipeline for hg38.
03-30 10:33:51 cat   INFO  Beginning GenerateHints Toil pipeline for galGal4.
03-30 10:33:51 cat.hints_db INFO  Restarting Toil hints pipeline for rheMac3.
03-30 10:33:51 cat   INFO  Beginning GenerateHints Toil pipeline for mm10.
03-30 10:33:51 cat.hints_db INFO  Restarting Toil hints pipeline for hg38.
03-30 10:33:51 toil.jobStores.abstractJobStore CRITICAL Repairing job: S/f/jobWsVBMf
03-30 10:33:51 toil.jobStores.abstractJobStore CRITICAL Repairing job: S/f/jobWsVBMf
03-30 10:33:51 cat.hints_db INFO  Restarting Toil hints pipeline for galGal4.
03-30 10:33:51 cat.hints_db INFO  Restarting Toil hints pipeline for mm10.
03-30 10:33:51 toil.jobStores.abstractJobStore CRITICAL Repairing job: t/Z/jobbxMOiF
03-30 10:33:51 toil.jobStores.abstractJobStore CRITICAL Repairing job: t/Z/jobbxMOiF
03-30 10:33:51 toil.jobStores.abstractJobStore CRITICAL Repairing job: I/E/jobpE4Exm
03-30 10:33:51 toil.jobStores.abstractJobStore CRITICAL Repairing job: I/E/jobpE4Exm
03-30 10:33:51 toil.jobStores.abstractJobStore CRITICAL Repairing job: 8/V/jobc4vH8x
03-30 10:33:51 toil.jobStores.abstractJobStore CRITICAL Repairing job: 8/V/jobc4vH8x
03-30 10:33:51 toil.jobStores.abstractJobStore CRITICAL Repairing job: r/j/jobHUG8TV
03-30 10:33:51 toil.jobStores.abstractJobStore CRITICAL Repairing job: r/j/jobHUG8TV
03-30 10:33:51 toil.jobStores.abstractJobStore CRITICAL Repairing job: X/R/jobbpn764
03-30 10:33:51 toil.jobStores.abstractJobStore CRITICAL Repairing job: X/R/jobbpn764
03-30 10:33:51 toil.jobStores.abstractJobStore CRITICAL Repairing job: V/4/job6X8beL
03-30 10:33:51 toil.jobStores.abstractJobStore CRITICAL Repairing job: V/4/job6X8beL
Exception in thread Thread-1:
Traceback (most recent call last):
  File "/home/liran/Software/miniconda2/envs/C-A-T/lib/python2.7/threading.py", line 801, in __bootstrap_inner
    self.run()
  File "/home/liran/Software/miniconda2/envs/C-A-T/lib/python2.7/threading.py", line 754, in run
    self.__target(*self.__args, **self.__kwargs)
  File "/home/liran/Software/miniconda2/envs/C-A-T/lib/python2.7/site-packages/toil/fileStore.py", line 1461, in asyncWrite
    raise RuntimeError("The termination flag is set, exiting")
RuntimeError: The termination flag is set, exiting
Exception in thread Thread-2:
Traceback (most recent call last):
  File "/home/liran/Software/miniconda2/envs/C-A-T/lib/python2.7/threading.py", line 801, in __bootstrap_inner
    self.run()
  File "/home/liran/Software/miniconda2/envs/C-A-T/lib/python2.7/threading.py", line 754, in run
    self.__target(*self.__args, **self.__kwargs)
  File "/home/liran/Software/miniconda2/envs/C-A-T/lib/python2.7/site-packages/toil/fileStore.py", line 1461, in asyncWrite
    raise RuntimeError("The termination flag is set, exiting")
RuntimeError: The termination flag is set, exiting

Exception RuntimeError: RuntimeError('cannot join current thread',) in <bound method CachingFileStore.__del__ of <toil.fileStore.CachingFileStore object at 0x7fb7de2da650>> ignored
Exception in thread Thread-1:
Traceback (most recent call last):
  File "/home/liran/Software/miniconda2/envs/C-A-T/lib/python2.7/threading.py", line 801, in __bootstrap_inner
    self.run()
  File "/home/liran/Software/miniconda2/envs/C-A-T/lib/python2.7/threading.py", line 754, in run
    self.__target(*self.__args, **self.__kwargs)
  File "/home/liran/Software/miniconda2/envs/C-A-T/lib/python2.7/site-packages/toil/fileStore.py", line 1461, in asyncWrite
    raise RuntimeError("The termination flag is set, exiting")
RuntimeError: The termination flag is set, exiting

Exception in thread Thread-2:
Traceback (most recent call last):
  File "/home/liran/Software/miniconda2/envs/C-A-T/lib/python2.7/threading.py", line 801, in __bootstrap_inner
    self.run()
  File "/home/liran/Software/miniconda2/envs/C-A-T/lib/python2.7/threading.py", line 754, in run
    self.__target(*self.__args, **self.__kwargs)
  File "/home/liran/Software/miniconda2/envs/C-A-T/lib/python2.7/site-packages/toil/fileStore.py", line 1461, in asyncWrite
    raise RuntimeError("The termination flag is set, exiting")
RuntimeError: The termination flag is set, exiting

Exception RuntimeError: RuntimeError('cannot join current thread',) in <bound method CachingFileStore.__del__ of <toil.fileStore.CachingFileStore object at 0x7f288681d650>> ignored
Exception in thread Thread-1:
Traceback (most recent call last):
  File "/home/liran/Software/miniconda2/envs/C-A-T/lib/python2.7/threading.py", line 801, in __bootstrap_inner
    self.run()
  File "/home/liran/Software/miniconda2/envs/C-A-T/lib/python2.7/threading.py", line 754, in run
    self.__target(*self.__args, **self.__kwargs)
  File "/home/liran/Software/miniconda2/envs/C-A-T/lib/python2.7/site-packages/toil/fileStore.py", line 1461, in asyncWrite
    raise RuntimeError("The termination flag is set, exiting")
RuntimeError: The termination flag is set, exiting
Exception in thread Thread-2:
Traceback (most recent call last):
  File "/home/liran/Software/miniconda2/envs/C-A-T/lib/python2.7/threading.py", line 801, in __bootstrap_inner
    self.run()
  File "/home/liran/Software/miniconda2/envs/C-A-T/lib/python2.7/threading.py", line 754, in run
    self.__target(*self.__args, **self.__kwargs)
  File "/home/liran/Software/miniconda2/envs/C-A-T/lib/python2.7/site-packages/toil/fileStore.py", line 1461, in asyncWrite
    raise RuntimeError("The termination flag is set, exiting")
RuntimeError: The termination flag is set, exiting

Exception RuntimeError: RuntimeError('cannot join current thread',) in <bound method CachingFileStore.__del__ of <toil.fileStore.CachingFileStore object at 0x7f945367b650>> ignored
Exception in thread Thread-1:
Traceback (most recent call last):
  File "/home/liran/Software/miniconda2/envs/C-A-T/lib/python2.7/threading.py", line 801, in __bootstrap_inner
    self.run()
  File "/home/liran/Software/miniconda2/envs/C-A-T/lib/python2.7/threading.py", line 754, in run
    self.__target(*self.__args, **self.__kwargs)
  File "/home/liran/Software/miniconda2/envs/C-A-T/lib/python2.7/site-packages/toil/fileStore.py", line 1461, in asyncWrite
    raise RuntimeError("The termination flag is set, exiting")
RuntimeError: The termination flag is set, exiting

Exception in thread Thread-2:
Traceback (most recent call last):
  File "/home/liran/Software/miniconda2/envs/C-A-T/lib/python2.7/threading.py", line 801, in __bootstrap_inner
    self.run()
  File "/home/liran/Software/miniconda2/envs/C-A-T/lib/python2.7/threading.py", line 754, in run
    self.__target(*self.__args, **self.__kwargs)
  File "/home/liran/Software/miniconda2/envs/C-A-T/lib/python2.7/site-packages/toil/fileStore.py", line 1461, in asyncWrite
    raise RuntimeError("The termination flag is set, exiting")
RuntimeError: The termination flag is set, exiting

Exception RuntimeError: RuntimeError('cannot join current thread',) in <bound method CachingFileStore.__del__ of <toil.fileStore.CachingFileStore object at 0x7fb2f60e3650>> ignored
Exception in thread Thread-1:
Traceback (most recent call last):
  File "/home/liran/Software/miniconda2/envs/C-A-T/lib/python2.7/threading.py", line 801, in __bootstrap_inner
    self.run()
  File "/home/liran/Software/miniconda2/envs/C-A-T/lib/python2.7/threading.py", line 754, in run
    self.__target(*self.__args, **self.__kwargs)
  File "/home/liran/Software/miniconda2/envs/C-A-T/lib/python2.7/site-packages/toil/fileStore.py", line 1461, in asyncWrite
    raise RuntimeError("The termination flag is set, exiting")
RuntimeError: The termination flag is set, exiting

Exception in thread Thread-2:
Traceback (most recent call last):
  File "/home/liran/Software/miniconda2/envs/C-A-T/lib/python2.7/threading.py", line 801, in __bootstrap_inner
    self.run()
  File "/home/liran/Software/miniconda2/envs/C-A-T/lib/python2.7/threading.py", line 754, in run
    self.__target(*self.__args, **self.__kwargs)
  File "/home/liran/Software/miniconda2/envs/C-A-T/lib/python2.7/site-packages/toil/fileStore.py", line 1461, in asyncWrite
    raise RuntimeError("The termination flag is set, exiting")
RuntimeError: The termination flag is set, exiting

Exception RuntimeError: RuntimeError('cannot join current thread',) in <bound method CachingFileStore.__del__ of <toil.fileStore.CachingFileStore object at 0x7f779b04f650>> ignored
Exception in thread Thread-1:
Traceback (most recent call last):
  File "/home/liran/Software/miniconda2/envs/C-A-T/lib/python2.7/threading.py", line 801, in __bootstrap_inner
    self.run()
  File "/home/liran/Software/miniconda2/envs/C-A-T/lib/python2.7/threading.py", line 754, in run
    self.__target(*self.__args, **self.__kwargs)
  File "/home/liran/Software/miniconda2/envs/C-A-T/lib/python2.7/site-packages/toil/fileStore.py", line 1461, in asyncWrite
    raise RuntimeError("The termination flag is set, exiting")
RuntimeError: The termination flag is set, exiting

Exception in thread Thread-2:
Traceback (most recent call last):
  File "/home/liran/Software/miniconda2/envs/C-A-T/lib/python2.7/threading.py", line 801, in __bootstrap_inner
    self.run()
  File "/home/liran/Software/miniconda2/envs/C-A-T/lib/python2.7/threading.py", line 754, in run
    self.__target(*self.__args, **self.__kwargs)
  File "/home/liran/Software/miniconda2/envs/C-A-T/lib/python2.7/site-packages/toil/fileStore.py", line 1461, in asyncWrite
    raise RuntimeError("The termination flag is set, exiting")
RuntimeError: The termination flag is set, exiting

Exception RuntimeError: RuntimeError('cannot join current thread',) in <bound method CachingFileStore.__del__ of <toil.fileStore.CachingFileStore object at 0x7f4fd5ef2650>> ignored
03-30 10:33:54 toil.leader WARNING The job seems to have left a log file, indicating failure: 'namesort_bam' I/E/jobpE4Exm
---TOIL WORKER OUTPUT LOG---
I/E/jobpE4Exm    INFO:toil:Running Toil version 3.5.0-7f5bc1cab176d99d6996076ae5b8bc2453fa04a4.
I/E/jobpE4Exm    WARNING:toil.resource:Can't find resource for leader path '/home/liran/Software/C-A-T/Comparative-Annotation-Toolkit/cat'
I/E/jobpE4Exm    WARNING:toil.resource:Can't localize module ModuleDescriptor(dirPath='/home/liran/Software/C-A-T/Comparative-Annotation-Toolkit', name='cat.hints_db', fromVirtualEnv=False)
I/E/jobpE4Exm    INFO:luigi-interface:Loaded ['luigi.cfg']
I/E/jobpE4Exm    INFO:toil.fileStore:Starting job ('namesort_bam' I/E/jobpE4Exm) with ID (4c007acbed9b807693aea63673f27db95e3d1d50).
I/E/jobpE4Exm    WARNING:toil.resource:Can't find resource for leader path '/home/liran/Software/C-A-T/Comparative-Annotation-Toolkit/cat'
I/E/jobpE4Exm    WARNING:toil.resource:Can't localize module ModuleDescriptor(dirPath='/home/liran/Software/C-A-T/Comparative-Annotation-Toolkit', name='cat.hints_db', fromVirtualEnv=False)
I/E/jobpE4Exm    samtools: error while loading shared libraries: libcrypto.so.1.0.0: cannot open shared object file: No such file or directory
I/E/jobpE4Exm    sambamba-sort: not enough data in stream
I/E/jobpE4Exm    Traceback (most recent call last):
I/E/jobpE4Exm      File "/home/liran/Software/miniconda2/envs/C-A-T/lib/python2.7/site-packages/toil/worker.py", line 340, in main
I/E/jobpE4Exm        job._runner(jobGraph=jobGraph, jobStore=jobStore, fileStore=fileStore)
I/E/jobpE4Exm      File "/home/liran/Software/miniconda2/envs/C-A-T/lib/python2.7/site-packages/toil/job.py", line 1270, in _runner
I/E/jobpE4Exm        returnValues = self._run(jobGraph, fileStore)
I/E/jobpE4Exm      File "/home/liran/Software/miniconda2/envs/C-A-T/lib/python2.7/site-packages/toil/job.py", line 1217, in _run
I/E/jobpE4Exm        return self.run(fileStore)
I/E/jobpE4Exm      File "/home/liran/Software/miniconda2/envs/C-A-T/lib/python2.7/site-packages/toil/job.py", line 1383, in run
I/E/jobpE4Exm        rValue = userFunction(*((self,) + tuple(self._args)), **self._kwargs)
I/E/jobpE4Exm      File "/home/liran/Software/C-A-T/Comparative-Annotation-Toolkit/cat/hints_db.py", line 161, in namesort_bam
I/E/jobpE4Exm        tools.procOps.run_proc(cmd, stdout=name_sorted)
I/E/jobpE4Exm      File "/home/liran/Software/C-A-T/Comparative-Annotation-Toolkit/tools/procOps.py", line 68, in run_proc
I/E/jobpE4Exm        pl.wait()
I/E/jobpE4Exm      File "/home/liran/Software/C-A-T/Comparative-Annotation-Toolkit/tools/pipeline.py", line 1127, in wait
I/E/jobpE4Exm        self.raiseIfExcept()
I/E/jobpE4Exm      File "/home/liran/Software/C-A-T/Comparative-Annotation-Toolkit/tools/pipeline.py", line 1085, in raiseIfExcept
I/E/jobpE4Exm        p.raiseIfExcept()
I/E/jobpE4Exm      File "/home/liran/Software/C-A-T/Comparative-Annotation-Toolkit/tools/pipeline.py", line 749, in raiseIfExcept
I/E/jobpE4Exm        raise self.exceptInfo[0], self.exceptInfo[1], self.exceptInfo[2]
I/E/jobpE4Exm    ProcException: process exited 127: samtools view -b /tmp/toil-3140b0cd-fef4-45e2-9576-8ad3d48ca251/tmpJRrVMj/2e8bffff-c008-4483-aa7c-c8092d9ee649/tmptxrYF3.tmp chr16
I/E/jobpE4Exm    ERROR:toil.worker:Exiting the worker because of a failed job on host subnet190-2.nwsuaf.edu.cn
I/E/jobpE4Exm    WARNING:toil.jobGraph:Due to failure we are reducing the remaining retry count of job 'namesort_bam' I/E/jobpE4Exm with ID I/E/jobpE4Exm to 0
03-30 10:33:54 toil.leader WARNING Job 'namesort_bam' I/E/jobpE4Exm with ID I/E/jobpE4Exm is completely failed
03-30 10:33:54 toil.leader WARNING The job seems to have left a log file, indicating failure: 'namesort_bam' 8/V/jobc4vH8x
---TOIL WORKER OUTPUT LOG---
8/V/jobc4vH8x    INFO:toil:Running Toil version 3.5.0-7f5bc1cab176d99d6996076ae5b8bc2453fa04a4.
8/V/jobc4vH8x    WARNING:toil.resource:Can't find resource for leader path '/home/liran/Software/C-A-T/Comparative-Annotation-Toolkit/cat'
8/V/jobc4vH8x    WARNING:toil.resource:Can't localize module ModuleDescriptor(dirPath='/home/liran/Software/C-A-T/Comparative-Annotation-Toolkit', name='cat.hints_db', fromVirtualEnv=False)
8/V/jobc4vH8x    INFO:luigi-interface:Loaded ['luigi.cfg']
8/V/jobc4vH8x    INFO:toil.fileStore:Starting job ('namesort_bam' 8/V/jobc4vH8x) with ID (87279c8647989aceb657e93f65389bbd4c04219c).
8/V/jobc4vH8x    WARNING:toil.resource:Can't find resource for leader path '/home/liran/Software/C-A-T/Comparative-Annotation-Toolkit/cat'
8/V/jobc4vH8x    WARNING:toil.resource:Can't localize module ModuleDescriptor(dirPath='/home/liran/Software/C-A-T/Comparative-Annotation-Toolkit', name='cat.hints_db', fromVirtualEnv=False)
8/V/jobc4vH8x    samtools: error while loading shared libraries: libcrypto.so.1.0.0: cannot open shared object file: No such file or directory
8/V/jobc4vH8x    sambamba-sort: not enough data in stream
8/V/jobc4vH8x    Traceback (most recent call last):
8/V/jobc4vH8x      File "/home/liran/Software/miniconda2/envs/C-A-T/lib/python2.7/site-packages/toil/worker.py", line 340, in main
8/V/jobc4vH8x        job._runner(jobGraph=jobGraph, jobStore=jobStore, fileStore=fileStore)
8/V/jobc4vH8x      File "/home/liran/Software/miniconda2/envs/C-A-T/lib/python2.7/site-packages/toil/job.py", line 1270, in _runner
8/V/jobc4vH8x        returnValues = self._run(jobGraph, fileStore)
8/V/jobc4vH8x      File "/home/liran/Software/miniconda2/envs/C-A-T/lib/python2.7/site-packages/toil/job.py", line 1217, in _run
8/V/jobc4vH8x        return self.run(fileStore)
8/V/jobc4vH8x      File "/home/liran/Software/miniconda2/envs/C-A-T/lib/python2.7/site-packages/toil/job.py", line 1383, in run
8/V/jobc4vH8x        rValue = userFunction(*((self,) + tuple(self._args)), **self._kwargs)
8/V/jobc4vH8x      File "/home/liran/Software/C-A-T/Comparative-Annotation-Toolkit/cat/hints_db.py", line 161, in namesort_bam
8/V/jobc4vH8x        tools.procOps.run_proc(cmd, stdout=name_sorted)
8/V/jobc4vH8x      File "/home/liran/Software/C-A-T/Comparative-Annotation-Toolkit/tools/procOps.py", line 68, in run_proc
8/V/jobc4vH8x        pl.wait()
8/V/jobc4vH8x      File "/home/liran/Software/C-A-T/Comparative-Annotation-Toolkit/tools/pipeline.py", line 1127, in wait
8/V/jobc4vH8x        self.raiseIfExcept()
8/V/jobc4vH8x      File "/home/liran/Software/C-A-T/Comparative-Annotation-Toolkit/tools/pipeline.py", line 1085, in raiseIfExcept
8/V/jobc4vH8x        p.raiseIfExcept()
8/V/jobc4vH8x      File "/home/liran/Software/C-A-T/Comparative-Annotation-Toolkit/tools/pipeline.py", line 749, in raiseIfExcept
8/V/jobc4vH8x        raise self.exceptInfo[0], self.exceptInfo[1], self.exceptInfo[2]
8/V/jobc4vH8x    ProcException: process exited 127: samtools view -b /tmp/toil-3fcac38b-8337-485e-a366-851a45df2826/tmpuIJTun/0ef8bc56-4882-431c-9ad2-3da22248ee5d/tmpXltoEJ.tmp chr14
8/V/jobc4vH8x    ERROR:toil.worker:Exiting the worker because of a failed job on host subnet190-2.nwsuaf.edu.cn
8/V/jobc4vH8x    WARNING:toil.jobGraph:Due to failure we are reducing the remaining retry count of job 'namesort_bam' 8/V/jobc4vH8x with ID 8/V/jobc4vH8x to 0
03-30 10:33:54 toil.leader WARNING Job 'namesort_bam' 8/V/jobc4vH8x with ID 8/V/jobc4vH8x is completely failed
03-30 10:33:54 toil.leader WARNING The job seems to have left a log file, indicating failure: 'namesort_bam' S/f/jobWsVBMf
---TOIL WORKER OUTPUT LOG---
S/f/jobWsVBMf    INFO:toil:Running Toil version 3.5.0-7f5bc1cab176d99d6996076ae5b8bc2453fa04a4.
S/f/jobWsVBMf    WARNING:toil.resource:Can't find resource for leader path '/home/liran/Software/C-A-T/Comparative-Annotation-Toolkit/cat'
S/f/jobWsVBMf    WARNING:toil.resource:Can't localize module ModuleDescriptor(dirPath='/home/liran/Software/C-A-T/Comparative-Annotation-Toolkit', name='cat.hints_db', fromVirtualEnv=False)
S/f/jobWsVBMf    INFO:luigi-interface:Loaded ['luigi.cfg']
S/f/jobWsVBMf    INFO:toil.fileStore:Starting job ('namesort_bam' S/f/jobWsVBMf) with ID (4dce1ee2b02d7cfbce2741de810659fa32dfcea7).
S/f/jobWsVBMf    WARNING:toil.resource:Can't find resource for leader path '/home/liran/Software/C-A-T/Comparative-Annotation-Toolkit/cat'
S/f/jobWsVBMf    WARNING:toil.resource:Can't localize module ModuleDescriptor(dirPath='/home/liran/Software/C-A-T/Comparative-Annotation-Toolkit', name='cat.hints_db', fromVirtualEnv=False)
S/f/jobWsVBMf    samtools: error while loading shared libraries: libcrypto.so.1.0.0: cannot open shared object file: No such file or directory
S/f/jobWsVBMf    sambamba-sort: not enough data in stream
S/f/jobWsVBMf    Traceback (most recent call last):
S/f/jobWsVBMf      File "/home/liran/Software/miniconda2/envs/C-A-T/lib/python2.7/site-packages/toil/worker.py", line 340, in main
S/f/jobWsVBMf        job._runner(jobGraph=jobGraph, jobStore=jobStore, fileStore=fileStore)
S/f/jobWsVBMf      File "/home/liran/Software/miniconda2/envs/C-A-T/lib/python2.7/site-packages/toil/job.py", line 1270, in _runner
S/f/jobWsVBMf        returnValues = self._run(jobGraph, fileStore)
S/f/jobWsVBMf      File "/home/liran/Software/miniconda2/envs/C-A-T/lib/python2.7/site-packages/toil/job.py", line 1217, in _run
S/f/jobWsVBMf        return self.run(fileStore)
S/f/jobWsVBMf      File "/home/liran/Software/miniconda2/envs/C-A-T/lib/python2.7/site-packages/toil/job.py", line 1383, in run
S/f/jobWsVBMf        rValue = userFunction(*((self,) + tuple(self._args)), **self._kwargs)
S/f/jobWsVBMf      File "/home/liran/Software/C-A-T/Comparative-Annotation-Toolkit/cat/hints_db.py", line 161, in namesort_bam
S/f/jobWsVBMf        tools.procOps.run_proc(cmd, stdout=name_sorted)
S/f/jobWsVBMf      File "/home/liran/Software/C-A-T/Comparative-Annotation-Toolkit/tools/procOps.py", line 68, in run_proc
S/f/jobWsVBMf        pl.wait()
S/f/jobWsVBMf      File "/home/liran/Software/C-A-T/Comparative-Annotation-Toolkit/tools/pipeline.py", line 1127, in wait
S/f/jobWsVBMf        self.raiseIfExcept()
S/f/jobWsVBMf      File "/home/liran/Software/C-A-T/Comparative-Annotation-Toolkit/tools/pipeline.py", line 1085, in raiseIfExcept
S/f/jobWsVBMf        p.raiseIfExcept()
S/f/jobWsVBMf      File "/home/liran/Software/C-A-T/Comparative-Annotation-Toolkit/tools/pipeline.py", line 749, in raiseIfExcept
S/f/jobWsVBMf        raise self.exceptInfo[0], self.exceptInfo[1], self.exceptInfo[2]
S/f/jobWsVBMf    ProcException: process exited 127: samtools view -b /tmp/toil-3d7f7e23-af68-431b-9a1b-27ac0c6bd62d/tmpsmFNMz/375989f7-b60c-402d-9ac9-1e71ab9ff27d/tmpFmm0Bq.tmp chr20
S/f/jobWsVBMf    ERROR:toil.worker:Exiting the worker because of a failed job on host subnet190-2.nwsuaf.edu.cn
S/f/jobWsVBMf    WARNING:toil.jobGraph:Due to failure we are reducing the remaining retry count of job 'namesort_bam' S/f/jobWsVBMf with ID S/f/jobWsVBMf to 0
03-30 10:33:54 toil.leader WARNING Job 'namesort_bam' S/f/jobWsVBMf with ID S/f/jobWsVBMf is completely failed
03-30 10:33:55 toil.leader WARNING The job seems to have left a log file, indicating failure: 'generate_iso_seq_hints' t/Z/jobbxMOiF
---TOIL WORKER OUTPUT LOG---
t/Z/jobbxMOiF    INFO:toil:Running Toil version 3.5.0-7f5bc1cab176d99d6996076ae5b8bc2453fa04a4.
t/Z/jobbxMOiF    WARNING:toil.resource:Can't find resource for leader path '/home/liran/Software/C-A-T/Comparative-Annotation-Toolkit/cat'
t/Z/jobbxMOiF    WARNING:toil.resource:Can't localize module ModuleDescriptor(dirPath='/home/liran/Software/C-A-T/Comparative-Annotation-Toolkit', name='cat.hints_db', fromVirtualEnv=False)
t/Z/jobbxMOiF    INFO:luigi-interface:Loaded ['luigi.cfg']
t/Z/jobbxMOiF    INFO:toil.fileStore:Starting job ('generate_iso_seq_hints' t/Z/jobbxMOiF) with ID (8fabd729abe37ab3990e0ba45052b4b16daddbdc).
t/Z/jobbxMOiF    WARNING:toil.resource:Can't find resource for leader path '/home/liran/Software/C-A-T/Comparative-Annotation-Toolkit/cat'
t/Z/jobbxMOiF    WARNING:toil.resource:Can't localize module ModuleDescriptor(dirPath='/home/liran/Software/C-A-T/Comparative-Annotation-Toolkit', name='cat.hints_db', fromVirtualEnv=False)
t/Z/jobbxMOiF    samtools: error while loading shared libraries: libcrypto.so.1.0.0: cannot open shared object file: No such file or directory
t/Z/jobbxMOiF
t/Z/jobbxMOiF    Traceback (most recent call last):
t/Z/jobbxMOiF      File "/home/liran/Software/miniconda2/envs/C-A-T/lib/python2.7/site-packages/toil/worker.py", line 340, in main
t/Z/jobbxMOiF        job._runner(jobGraph=jobGraph, jobStore=jobStore, fileStore=fileStore)
t/Z/jobbxMOiF      File "/home/liran/Software/miniconda2/envs/C-A-T/lib/python2.7/site-packages/toil/job.py", line 1270, in _runner
t/Z/jobbxMOiF        returnValues = self._run(jobGraph, fileStore)
t/Z/jobbxMOiF      File "/home/liran/Software/miniconda2/envs/C-A-T/lib/python2.7/site-packages/toil/job.py", line 1217, in _run
t/Z/jobbxMOiF        return self.run(fileStore)
t/Z/jobbxMOiF      File "/home/liran/Software/miniconda2/envs/C-A-T/lib/python2.7/site-packages/toil/job.py", line 1383, in run
t/Z/jobbxMOiF        rValue = userFunction(*((self,) + tuple(self._args)), **self._kwargs)
t/Z/jobbxMOiF      File "/home/liran/Software/C-A-T/Comparative-Annotation-Toolkit/cat/hints_db.py", line 391, in generate_iso_seq_hints
t/Z/jobbxMOiF        tools.procOps.run_proc(cmd)
t/Z/jobbxMOiF      File "/home/liran/Software/C-A-T/Comparative-Annotation-Toolkit/tools/procOps.py", line 68, in run_proc
t/Z/jobbxMOiF        pl.wait()
t/Z/jobbxMOiF      File "/home/liran/Software/C-A-T/Comparative-Annotation-Toolkit/tools/pipeline.py", line 1127, in wait
t/Z/jobbxMOiF        self.raiseIfExcept()
t/Z/jobbxMOiF      File "/home/liran/Software/C-A-T/Comparative-Annotation-Toolkit/tools/pipeline.py", line 1085, in raiseIfExcept
t/Z/jobbxMOiF        p.raiseIfExcept()
t/Z/jobbxMOiF      File "/home/liran/Software/C-A-T/Comparative-Annotation-Toolkit/tools/pipeline.py", line 749, in raiseIfExcept
t/Z/jobbxMOiF        raise self.exceptInfo[0], self.exceptInfo[1], self.exceptInfo[2]
t/Z/jobbxMOiF    ProcException: process exited 127: samtools view -b -F 4 /tmp/toil-3140b0cd-fef4-45e2-9576-8ad3d48ca251/tmpAvOHNn/6d114774-f138-4425-aac9-19a919ed47a4/tmpKn5D35.tmp
t/Z/jobbxMOiF    ERROR:toil.worker:Exiting the worker because of a failed job on host subnet190-2.nwsuaf.edu.cn
t/Z/jobbxMOiF    WARNING:toil.jobGraph:Due to failure we are reducing the remaining retry count of job 'generate_iso_seq_hints' t/Z/jobbxMOiF with ID t/Z/jobbxMOiF to 0
03-30 10:33:55 toil.leader WARNING Job 'generate_iso_seq_hints' t/Z/jobbxMOiF with ID t/Z/jobbxMOiF is completely failed
03-30 10:33:55 toil.leader WARNING The job seems to have left a log file, indicating failure: 'namesort_bam' r/j/jobHUG8TV
---TOIL WORKER OUTPUT LOG---
r/j/jobHUG8TV    INFO:toil:Running Toil version 3.5.0-7f5bc1cab176d99d6996076ae5b8bc2453fa04a4.
r/j/jobHUG8TV    WARNING:toil.resource:Can't find resource for leader path '/home/liran/Software/C-A-T/Comparative-Annotation-Toolkit/cat'
r/j/jobHUG8TV    WARNING:toil.resource:Can't localize module ModuleDescriptor(dirPath='/home/liran/Software/C-A-T/Comparative-Annotation-Toolkit', name='cat.hints_db', fromVirtualEnv=False)
r/j/jobHUG8TV    INFO:luigi-interface:Loaded ['luigi.cfg']
r/j/jobHUG8TV    INFO:toil.fileStore:Starting job ('namesort_bam' r/j/jobHUG8TV) with ID (b776329fa019fddcf731411725fdcd084fc41346).
r/j/jobHUG8TV    WARNING:toil.resource:Can't find resource for leader path '/home/liran/Software/C-A-T/Comparative-Annotation-Toolkit/cat'
r/j/jobHUG8TV    WARNING:toil.resource:Can't localize module ModuleDescriptor(dirPath='/home/liran/Software/C-A-T/Comparative-Annotation-Toolkit', name='cat.hints_db', fromVirtualEnv=False)
r/j/jobHUG8TV    samtools: error while loading shared libraries: libcrypto.so.1.0.0: cannot open shared object file: No such file or directory
r/j/jobHUG8TV    sambamba-sort: not enough data in stream
r/j/jobHUG8TV    Traceback (most recent call last):
r/j/jobHUG8TV      File "/home/liran/Software/miniconda2/envs/C-A-T/lib/python2.7/site-packages/toil/worker.py", line 340, in main
r/j/jobHUG8TV        job._runner(jobGraph=jobGraph, jobStore=jobStore, fileStore=fileStore)
r/j/jobHUG8TV      File "/home/liran/Software/miniconda2/envs/C-A-T/lib/python2.7/site-packages/toil/job.py", line 1270, in _runner
r/j/jobHUG8TV        returnValues = self._run(jobGraph, fileStore)
r/j/jobHUG8TV      File "/home/liran/Software/miniconda2/envs/C-A-T/lib/python2.7/site-packages/toil/job.py", line 1217, in _run
r/j/jobHUG8TV        return self.run(fileStore)
r/j/jobHUG8TV      File "/home/liran/Software/miniconda2/envs/C-A-T/lib/python2.7/site-packages/toil/job.py", line 1383, in run
r/j/jobHUG8TV        rValue = userFunction(*((self,) + tuple(self._args)), **self._kwargs)
r/j/jobHUG8TV      File "/home/liran/Software/C-A-T/Comparative-Annotation-Toolkit/cat/hints_db.py", line 161, in namesort_bam
r/j/jobHUG8TV        tools.procOps.run_proc(cmd, stdout=name_sorted)
r/j/jobHUG8TV      File "/home/liran/Software/C-A-T/Comparative-Annotation-Toolkit/tools/procOps.py", line 68, in run_proc
r/j/jobHUG8TV        pl.wait()
r/j/jobHUG8TV      File "/home/liran/Software/C-A-T/Comparative-Annotation-Toolkit/tools/pipeline.py", line 1127, in wait
r/j/jobHUG8TV        self.raiseIfExcept()
r/j/jobHUG8TV      File "/home/liran/Software/C-A-T/Comparative-Annotation-Toolkit/tools/pipeline.py", line 1085, in raiseIfExcept
r/j/jobHUG8TV        p.raiseIfExcept()
r/j/jobHUG8TV      File "/home/liran/Software/C-A-T/Comparative-Annotation-Toolkit/tools/pipeline.py", line 749, in raiseIfExcept
r/j/jobHUG8TV        raise self.exceptInfo[0], self.exceptInfo[1], self.exceptInfo[2]
r/j/jobHUG8TV    ProcException: process exited 127: samtools view -b /tmp/toil-3a71830d-df1e-460d-b3df-50f0633d456e/tmpakvgey/04d45eaf-4719-488b-989e-a9424a58a4ed/tmpdYlxSD.tmp chr17
r/j/jobHUG8TV    ERROR:toil.worker:Exiting the worker because of a failed job on host subnet190-2.nwsuaf.edu.cn
r/j/jobHUG8TV    WARNING:toil.jobGraph:Due to failure we are reducing the remaining retry count of job 'namesort_bam' r/j/jobHUG8TV with ID r/j/jobHUG8TV to 0
03-30 10:33:55 toil.leader WARNING Job 'namesort_bam' r/j/jobHUG8TV with ID r/j/jobHUG8TV is completely failed
03-30 10:33:55 toil.leader WARNING The job seems to have left a log file, indicating failure: 'namesort_bam' X/R/jobbpn764
---TOIL WORKER OUTPUT LOG---
X/R/jobbpn764    INFO:toil:Running Toil version 3.5.0-7f5bc1cab176d99d6996076ae5b8bc2453fa04a4.
X/R/jobbpn764    WARNING:toil.resource:Can't find resource for leader path '/home/liran/Software/C-A-T/Comparative-Annotation-Toolkit/cat'
X/R/jobbpn764    WARNING:toil.resource:Can't localize module ModuleDescriptor(dirPath='/home/liran/Software/C-A-T/Comparative-Annotation-Toolkit', name='cat.hints_db', fromVirtualEnv=False)
X/R/jobbpn764    INFO:luigi-interface:Loaded ['luigi.cfg']
X/R/jobbpn764    INFO:toil.fileStore:Starting job ('namesort_bam' X/R/jobbpn764) with ID (cca09013a4e0d74a60cb5fd59d573d52bae1f6ca).
X/R/jobbpn764    WARNING:toil.resource:Can't find resource for leader path '/home/liran/Software/C-A-T/Comparative-Annotation-Toolkit/cat'
X/R/jobbpn764    WARNING:toil.resource:Can't localize module ModuleDescriptor(dirPath='/home/liran/Software/C-A-T/Comparative-Annotation-Toolkit', name='cat.hints_db', fromVirtualEnv=False)
X/R/jobbpn764    samtools: error while loading shared libraries: libcrypto.so.1.0.0: cannot open shared object file: No such file or directory
X/R/jobbpn764    sambamba-sort: not enough data in stream
X/R/jobbpn764    Traceback (most recent call last):
X/R/jobbpn764      File "/home/liran/Software/miniconda2/envs/C-A-T/lib/python2.7/site-packages/toil/worker.py", line 340, in main
X/R/jobbpn764        job._runner(jobGraph=jobGraph, jobStore=jobStore, fileStore=fileStore)
X/R/jobbpn764      File "/home/liran/Software/miniconda2/envs/C-A-T/lib/python2.7/site-packages/toil/job.py", line 1270, in _runner
X/R/jobbpn764        returnValues = self._run(jobGraph, fileStore)
X/R/jobbpn764      File "/home/liran/Software/miniconda2/envs/C-A-T/lib/python2.7/site-packages/toil/job.py", line 1217, in _run
X/R/jobbpn764        return self.run(fileStore)
X/R/jobbpn764      File "/home/liran/Software/miniconda2/envs/C-A-T/lib/python2.7/site-packages/toil/job.py", line 1383, in run
X/R/jobbpn764        rValue = userFunction(*((self,) + tuple(self._args)), **self._kwargs)
X/R/jobbpn764      File "/home/liran/Software/C-A-T/Comparative-Annotation-Toolkit/cat/hints_db.py", line 161, in namesort_bam
X/R/jobbpn764        tools.procOps.run_proc(cmd, stdout=name_sorted)
X/R/jobbpn764      File "/home/liran/Software/C-A-T/Comparative-Annotation-Toolkit/tools/procOps.py", line 68, in run_proc
X/R/jobbpn764        pl.wait()
X/R/jobbpn764      File "/home/liran/Software/C-A-T/Comparative-Annotation-Toolkit/tools/pipeline.py", line 1127, in wait
X/R/jobbpn764        self.raiseIfExcept()
X/R/jobbpn764      File "/home/liran/Software/C-A-T/Comparative-Annotation-Toolkit/tools/pipeline.py", line 1085, in raiseIfExcept
X/R/jobbpn764        p.raiseIfExcept()
X/R/jobbpn764      File "/home/liran/Software/C-A-T/Comparative-Annotation-Toolkit/tools/pipeline.py", line 749, in raiseIfExcept
X/R/jobbpn764        raise self.exceptInfo[0], self.exceptInfo[1], self.exceptInfo[2]
X/R/jobbpn764    ProcException: process exited 127: samtools view -b /tmp/toil-3a71830d-df1e-460d-b3df-50f0633d456e/tmpKHYgxy/ae5fdc4d-6925-4bc7-ba9e-219e6fd42641/tmpT43J37.tmp chr17
X/R/jobbpn764    ERROR:toil.worker:Exiting the worker because of a failed job on host subnet190-2.nwsuaf.edu.cn
X/R/jobbpn764    WARNING:toil.jobGraph:Due to failure we are reducing the remaining retry count of job 'namesort_bam' X/R/jobbpn764 with ID X/R/jobbpn764 to 0
03-30 10:33:55 toil.leader WARNING Job 'namesort_bam' X/R/jobbpn764 with ID X/R/jobbpn764 is completely failed
Exception in thread Thread-1:
Traceback (most recent call last):
  File "/home/liran/Software/miniconda2/envs/C-A-T/lib/python2.7/threading.py", line 801, in __bootstrap_inner
    self.run()
  File "/home/liran/Software/miniconda2/envs/C-A-T/lib/python2.7/threading.py", line 754, in run
    self.__target(*self.__args, **self.__kwargs)
  File "/home/liran/Software/miniconda2/envs/C-A-T/lib/python2.7/site-packages/toil/fileStore.py", line 1461, in asyncWrite
    raise RuntimeError("The termination flag is set, exiting")
RuntimeError: The termination flag is set, exiting
Exception in thread Thread-2:
Traceback (most recent call last):
  File "/home/liran/Software/miniconda2/envs/C-A-T/lib/python2.7/threading.py", line 801, in __bootstrap_inner
    self.run()
  File "/home/liran/Software/miniconda2/envs/C-A-T/lib/python2.7/threading.py", line 754, in run
    self.__target(*self.__args, **self.__kwargs)
  File "/home/liran/Software/miniconda2/envs/C-A-T/lib/python2.7/site-packages/toil/fileStore.py", line 1461, in asyncWrite
    raise RuntimeError("The termination flag is set, exiting")
RuntimeError: The termination flag is set, exiting

Exception RuntimeError: RuntimeError('cannot join current thread',) in <bound method CachingFileStore.__del__ of <toil.fileStore.CachingFileStore object at 0x7f53c76a2650>> ignored
03-30 10:33:58 toil.leader WARNING The job seems to have left a log file, indicating failure: 'namesort_bam' V/4/job6X8beL
---TOIL WORKER OUTPUT LOG---
V/4/job6X8beL    INFO:toil:Running Toil version 3.5.0-7f5bc1cab176d99d6996076ae5b8bc2453fa04a4.
V/4/job6X8beL    WARNING:toil.resource:Can't find resource for leader path '/home/liran/Software/C-A-T/Comparative-Annotation-Toolkit/cat'
V/4/job6X8beL    WARNING:toil.resource:Can't localize module ModuleDescriptor(dirPath='/home/liran/Software/C-A-T/Comparative-Annotation-Toolkit', name='cat.hints_db', fromVirtualEnv=False)
V/4/job6X8beL    INFO:luigi-interface:Loaded ['luigi.cfg']
V/4/job6X8beL    INFO:toil.fileStore:Starting job ('namesort_bam' V/4/job6X8beL) with ID (58da24787b046b0440994ecffb59a24b137a50d2).
V/4/job6X8beL    WARNING:toil.resource:Can't find resource for leader path '/home/liran/Software/C-A-T/Comparative-Annotation-Toolkit/cat'
V/4/job6X8beL    WARNING:toil.resource:Can't localize module ModuleDescriptor(dirPath='/home/liran/Software/C-A-T/Comparative-Annotation-Toolkit', name='cat.hints_db', fromVirtualEnv=False)
V/4/job6X8beL    samtools: error while loading shared libraries: libcrypto.so.1.0.0: cannot open shared object file: No such file or directory
V/4/job6X8beL    sambamba-sort: not enough data in stream
V/4/job6X8beL    Traceback (most recent call last):
V/4/job6X8beL      File "/home/liran/Software/miniconda2/envs/C-A-T/lib/python2.7/site-packages/toil/worker.py", line 340, in main
V/4/job6X8beL        job._runner(jobGraph=jobGraph, jobStore=jobStore, fileStore=fileStore)
V/4/job6X8beL      File "/home/liran/Software/miniconda2/envs/C-A-T/lib/python2.7/site-packages/toil/job.py", line 1270, in _runner
V/4/job6X8beL        returnValues = self._run(jobGraph, fileStore)
V/4/job6X8beL      File "/home/liran/Software/miniconda2/envs/C-A-T/lib/python2.7/site-packages/toil/job.py", line 1217, in _run
V/4/job6X8beL        return self.run(fileStore)
V/4/job6X8beL      File "/home/liran/Software/miniconda2/envs/C-A-T/lib/python2.7/site-packages/toil/job.py", line 1383, in run
V/4/job6X8beL        rValue = userFunction(*((self,) + tuple(self._args)), **self._kwargs)
V/4/job6X8beL      File "/home/liran/Software/C-A-T/Comparative-Annotation-Toolkit/cat/hints_db.py", line 161, in namesort_bam
V/4/job6X8beL        tools.procOps.run_proc(cmd, stdout=name_sorted)
V/4/job6X8beL      File "/home/liran/Software/C-A-T/Comparative-Annotation-Toolkit/tools/procOps.py", line 68, in run_proc
V/4/job6X8beL        pl.wait()
V/4/job6X8beL      File "/home/liran/Software/C-A-T/Comparative-Annotation-Toolkit/tools/pipeline.py", line 1127, in wait
V/4/job6X8beL        self.raiseIfExcept()
V/4/job6X8beL      File "/home/liran/Software/C-A-T/Comparative-Annotation-Toolkit/tools/pipeline.py", line 1085, in raiseIfExcept
V/4/job6X8beL        p.raiseIfExcept()
V/4/job6X8beL      File "/home/liran/Software/C-A-T/Comparative-Annotation-Toolkit/tools/pipeline.py", line 749, in raiseIfExcept
V/4/job6X8beL        raise self.exceptInfo[0], self.exceptInfo[1], self.exceptInfo[2]
V/4/job6X8beL    ProcException: process exited 127: samtools view -b /tmp/toil-3a71830d-df1e-460d-b3df-50f0633d456e/tmpSZI7pC/778fec04-5972-4eb0-8ba0-6b0842da5065/tmpxeYvLw.tmp chr17
V/4/job6X8beL    ERROR:toil.worker:Exiting the worker because of a failed job on host subnet190-2.nwsuaf.edu.cn
V/4/job6X8beL    WARNING:toil.jobGraph:Due to failure we are reducing the remaining retry count of job 'namesort_bam' V/4/job6X8beL with ID V/4/job6X8beL to 0
03-30 10:33:58 toil.leader WARNING Job 'namesort_bam' V/4/job6X8beL with ID V/4/job6X8beL is completely failed
03-30 10:33:59 luigi-interface ERROR [pid 35764] Worker Worker(salt=724431596, workers=10, host=subnet190-2.nwsuaf.edu.cn, username=liran, pid=35207) failed    ToilTask: GenerateHints for rheMac3 using batchSystem singleMachine
Traceback (most recent call last):
  File "/home/liran/Software/miniconda2/envs/C-A-T/lib/python2.7/site-packages/luigi/worker.py", line 199, in run
    new_deps = self._run_get_new_deps()
  File "/home/liran/Software/miniconda2/envs/C-A-T/lib/python2.7/site-packages/luigi/worker.py", line 141, in _run_get_new_deps
    task_gen = self.task.run()
  File "/home/liran/Software/C-A-T/Comparative-Annotation-Toolkit/cat/__init__.py", line 1053, in run
    hints_db(self.hints_args, toil_options)
  File "/home/liran/Software/C-A-T/Comparative-Annotation-Toolkit/cat/hints_db.py", line 84, in hints_db
    combined_hints = t.restart()
  File "/home/liran/Software/miniconda2/envs/C-A-T/lib/python2.7/site-packages/toil/common.py", line 700, in restart
    return self._runMainLoop(rootJobGraph)
  File "/home/liran/Software/miniconda2/envs/C-A-T/lib/python2.7/site-packages/toil/common.py", line 984, in _runMainLoop
    jobCache=self._jobCache).run()
  File "/home/liran/Software/miniconda2/envs/C-A-T/lib/python2.7/site-packages/toil/leader.py", line 200, in run
    raise FailedJobsException(self.config.jobStore, self.toilState.totalFailedJobs, self.jobStore)
FailedJobsException: The job store 'file:/home/liran/Software/C-A-T/Comparative-Annotation-Toolkit/test_install/toil/hints_db/rheMac3/jobStore' contains 2 failed jobs: 'setup_hints' m/h/jobJLngJF, 'namesort_bam' S/f/jobWsVBMf
=========> Failed job 'namesort_bam' S/f/jobWsVBMf
---TOIL WORKER OUTPUT LOG---
INFO:toil:Running Toil version 3.5.0-7f5bc1cab176d99d6996076ae5b8bc2453fa04a4.
WARNING:toil.resource:Can't find resource for leader path '/home/liran/Software/C-A-T/Comparative-Annotation-Toolkit/cat'
WARNING:toil.resource:Can't localize module ModuleDescriptor(dirPath='/home/liran/Software/C-A-T/Comparative-Annotation-Toolkit', name='cat.hints_db', fromVirtualEnv=False)
INFO:luigi-interface:Loaded ['luigi.cfg']
INFO:toil.fileStore:Starting job ('namesort_bam' S/f/jobWsVBMf) with ID (4dce1ee2b02d7cfbce2741de810659fa32dfcea7).
WARNING:toil.resource:Can't find resource for leader path '/home/liran/Software/C-A-T/Comparative-Annotation-Toolkit/cat'
WARNING:toil.resource:Can't localize module ModuleDescriptor(dirPath='/home/liran/Software/C-A-T/Comparative-Annotation-Toolkit', name='cat.hints_db', fromVirtualEnv=False)
samtools: error while loading shared libraries: libcrypto.so.1.0.0: cannot open shared object file: No such file or directory
sambamba-sort: not enough data in stream
Traceback (most recent call last):
  File "/home/liran/Software/miniconda2/envs/C-A-T/lib/python2.7/site-packages/toil/worker.py", line 340, in main
    job._runner(jobGraph=jobGraph, jobStore=jobStore, fileStore=fileStore)
  File "/home/liran/Software/miniconda2/envs/C-A-T/lib/python2.7/site-packages/toil/job.py", line 1270, in _runner
    returnValues = self._run(jobGraph, fileStore)
  File "/home/liran/Software/miniconda2/envs/C-A-T/lib/python2.7/site-packages/toil/job.py", line 1217, in _run
    return self.run(fileStore)
  File "/home/liran/Software/miniconda2/envs/C-A-T/lib/python2.7/site-packages/toil/job.py", line 1383, in run
    rValue = userFunction(*((self,) + tuple(self._args)), **self._kwargs)
  File "/home/liran/Software/C-A-T/Comparative-Annotation-Toolkit/cat/hints_db.py", line 161, in namesort_bam
    tools.procOps.run_proc(cmd, stdout=name_sorted)
  File "/home/liran/Software/C-A-T/Comparative-Annotation-Toolkit/tools/procOps.py", line 68, in run_proc
    pl.wait()
  File "/home/liran/Software/C-A-T/Comparative-Annotation-Toolkit/tools/pipeline.py", line 1127, in wait
    self.raiseIfExcept()
  File "/home/liran/Software/C-A-T/Comparative-Annotation-Toolkit/tools/pipeline.py", line 1085, in raiseIfExcept
    p.raiseIfExcept()
  File "/home/liran/Software/C-A-T/Comparative-Annotation-Toolkit/tools/pipeline.py", line 749, in raiseIfExcept
    raise self.exceptInfo[0], self.exceptInfo[1], self.exceptInfo[2]
ProcException: process exited 127: samtools view -b /tmp/toil-3d7f7e23-af68-431b-9a1b-27ac0c6bd62d/tmpsmFNMz/375989f7-b60c-402d-9ac9-1e71ab9ff27d/tmpFmm0Bq.tmp chr20
ERROR:toil.worker:Exiting the worker because of a failed job on host subnet190-2.nwsuaf.edu.cn
WARNING:toil.jobGraph:Due to failure we are reducing the remaining retry count of job 'namesort_bam' S/f/jobWsVBMf with ID S/f/jobWsVBMf to 0
<=========

03-30 10:33:59 luigi-interface INFO  Informed scheduler that task   GenerateHints_False_True_True_0ccd06bf5f   has status   FAILED
03-30 10:33:59 luigi-interface ERROR [pid 35765] Worker Worker(salt=724431596, workers=10, host=subnet190-2.nwsuaf.edu.cn, username=liran, pid=35207) failed    ToilTask: GenerateHints for hg38 using batchSystem singleMachine
Traceback (most recent call last):
  File "/home/liran/Software/miniconda2/envs/C-A-T/lib/python2.7/site-packages/luigi/worker.py", line 199, in run
    new_deps = self._run_get_new_deps()
  File "/home/liran/Software/miniconda2/envs/C-A-T/lib/python2.7/site-packages/luigi/worker.py", line 141, in _run_get_new_deps
    task_gen = self.task.run()
  File "/home/liran/Software/C-A-T/Comparative-Annotation-Toolkit/cat/__init__.py", line 1053, in run
    hints_db(self.hints_args, toil_options)
  File "/home/liran/Software/C-A-T/Comparative-Annotation-Toolkit/cat/hints_db.py", line 84, in hints_db
    combined_hints = t.restart()
  File "/home/liran/Software/miniconda2/envs/C-A-T/lib/python2.7/site-packages/toil/common.py", line 700, in restart
    return self._runMainLoop(rootJobGraph)
  File "/home/liran/Software/miniconda2/envs/C-A-T/lib/python2.7/site-packages/toil/common.py", line 984, in _runMainLoop
    jobCache=self._jobCache).run()
  File "/home/liran/Software/miniconda2/envs/C-A-T/lib/python2.7/site-packages/toil/leader.py", line 200, in run
    raise FailedJobsException(self.config.jobStore, self.toilState.totalFailedJobs, self.jobStore)
FailedJobsException: The job store 'file:/home/liran/Software/C-A-T/Comparative-Annotation-Toolkit/test_install/toil/hints_db/hg38/jobStore' contains 3 failed jobs: 'namesort_bam' I/E/jobpE4Exm, 'generate_iso_seq_hints' t/Z/jobbxMOiF, 'setup_hints' t/L/jobKfhBXL
=========> Failed job 'namesort_bam' I/E/jobpE4Exm
---TOIL WORKER OUTPUT LOG---
INFO:toil:Running Toil version 3.5.0-7f5bc1cab176d99d6996076ae5b8bc2453fa04a4.
WARNING:toil.resource:Can't find resource for leader path '/home/liran/Software/C-A-T/Comparative-Annotation-Toolkit/cat'
WARNING:toil.resource:Can't localize module ModuleDescriptor(dirPath='/home/liran/Software/C-A-T/Comparative-Annotation-Toolkit', name='cat.hints_db', fromVirtualEnv=False)
INFO:luigi-interface:Loaded ['luigi.cfg']
INFO:toil.fileStore:Starting job ('namesort_bam' I/E/jobpE4Exm) with ID (4c007acbed9b807693aea63673f27db95e3d1d50).
WARNING:toil.resource:Can't find resource for leader path '/home/liran/Software/C-A-T/Comparative-Annotation-Toolkit/cat'
WARNING:toil.resource:Can't localize module ModuleDescriptor(dirPath='/home/liran/Software/C-A-T/Comparative-Annotation-Toolkit', name='cat.hints_db', fromVirtualEnv=False)
samtools: error while loading shared libraries: libcrypto.so.1.0.0: cannot open shared object file: No such file or directory
sambamba-sort: not enough data in stream
Traceback (most recent call last):
  File "/home/liran/Software/miniconda2/envs/C-A-T/lib/python2.7/site-packages/toil/worker.py", line 340, in main
    job._runner(jobGraph=jobGraph, jobStore=jobStore, fileStore=fileStore)
  File "/home/liran/Software/miniconda2/envs/C-A-T/lib/python2.7/site-packages/toil/job.py", line 1270, in _runner
    returnValues = self._run(jobGraph, fileStore)
  File "/home/liran/Software/miniconda2/envs/C-A-T/lib/python2.7/site-packages/toil/job.py", line 1217, in _run
    return self.run(fileStore)
  File "/home/liran/Software/miniconda2/envs/C-A-T/lib/python2.7/site-packages/toil/job.py", line 1383, in run
    rValue = userFunction(*((self,) + tuple(self._args)), **self._kwargs)
  File "/home/liran/Software/C-A-T/Comparative-Annotation-Toolkit/cat/hints_db.py", line 161, in namesort_bam
    tools.procOps.run_proc(cmd, stdout=name_sorted)
  File "/home/liran/Software/C-A-T/Comparative-Annotation-Toolkit/tools/procOps.py", line 68, in run_proc
    pl.wait()
  File "/home/liran/Software/C-A-T/Comparative-Annotation-Toolkit/tools/pipeline.py", line 1127, in wait
    self.raiseIfExcept()
  File "/home/liran/Software/C-A-T/Comparative-Annotation-Toolkit/tools/pipeline.py", line 1085, in raiseIfExcept
    p.raiseIfExcept()
  File "/home/liran/Software/C-A-T/Comparative-Annotation-Toolkit/tools/pipeline.py", line 749, in raiseIfExcept
    raise self.exceptInfo[0], self.exceptInfo[1], self.exceptInfo[2]
ProcException: process exited 127: samtools view -b /tmp/toil-3140b0cd-fef4-45e2-9576-8ad3d48ca251/tmpJRrVMj/2e8bffff-c008-4483-aa7c-c8092d9ee649/tmptxrYF3.tmp chr16
ERROR:toil.worker:Exiting the worker because of a failed job on host subnet190-2.nwsuaf.edu.cn
WARNING:toil.jobGraph:Due to failure we are reducing the remaining retry count of job 'namesort_bam' I/E/jobpE4Exm with ID I/E/jobpE4Exm to 0
<=========

=========> Failed job 'generate_iso_seq_hints' t/Z/jobbxMOiF
---TOIL WORKER OUTPUT LOG---
INFO:toil:Running Toil version 3.5.0-7f5bc1cab176d99d6996076ae5b8bc2453fa04a4.
WARNING:toil.resource:Can't find resource for leader path '/home/liran/Software/C-A-T/Comparative-Annotation-Toolkit/cat'
WARNING:toil.resource:Can't localize module ModuleDescriptor(dirPath='/home/liran/Software/C-A-T/Comparative-Annotation-Toolkit', name='cat.hints_db', fromVirtualEnv=False)
INFO:luigi-interface:Loaded ['luigi.cfg']
INFO:toil.fileStore:Starting job ('generate_iso_seq_hints' t/Z/jobbxMOiF) with ID (8fabd729abe37ab3990e0ba45052b4b16daddbdc).
WARNING:toil.resource:Can't find resource for leader path '/home/liran/Software/C-A-T/Comparative-Annotation-Toolkit/cat'
WARNING:toil.resource:Can't localize module ModuleDescriptor(dirPath='/home/liran/Software/C-A-T/Comparative-Annotation-Toolkit', name='cat.hints_db', fromVirtualEnv=False)
samtools: error while loading shared libraries: libcrypto.so.1.0.0: cannot open shared object file: No such file or directory

Traceback (most recent call last):
  File "/home/liran/Software/miniconda2/envs/C-A-T/lib/python2.7/site-packages/toil/worker.py", line 340, in main
    job._runner(jobGraph=jobGraph, jobStore=jobStore, fileStore=fileStore)
  File "/home/liran/Software/miniconda2/envs/C-A-T/lib/python2.7/site-packages/toil/job.py", line 1270, in _runner
    returnValues = self._run(jobGraph, fileStore)
  File "/home/liran/Software/miniconda2/envs/C-A-T/lib/python2.7/site-packages/toil/job.py", line 1217, in _run
    return self.run(fileStore)
  File "/home/liran/Software/miniconda2/envs/C-A-T/lib/python2.7/site-packages/toil/job.py", line 1383, in run
    rValue = userFunction(*((self,) + tuple(self._args)), **self._kwargs)
  File "/home/liran/Software/C-A-T/Comparative-Annotation-Toolkit/cat/hints_db.py", line 391, in generate_iso_seq_hints
    tools.procOps.run_proc(cmd)
  File "/home/liran/Software/C-A-T/Comparative-Annotation-Toolkit/tools/procOps.py", line 68, in run_proc
    pl.wait()
  File "/home/liran/Software/C-A-T/Comparative-Annotation-Toolkit/tools/pipeline.py", line 1127, in wait
    self.raiseIfExcept()
  File "/home/liran/Software/C-A-T/Comparative-Annotation-Toolkit/tools/pipeline.py", line 1085, in raiseIfExcept
    p.raiseIfExcept()
  File "/home/liran/Software/C-A-T/Comparative-Annotation-Toolkit/tools/pipeline.py", line 749, in raiseIfExcept
    raise self.exceptInfo[0], self.exceptInfo[1], self.exceptInfo[2]
ProcException: process exited 127: samtools view -b -F 4 /tmp/toil-3140b0cd-fef4-45e2-9576-8ad3d48ca251/tmpAvOHNn/6d114774-f138-4425-aac9-19a919ed47a4/tmpKn5D35.tmp
ERROR:toil.worker:Exiting the worker because of a failed job on host subnet190-2.nwsuaf.edu.cn
WARNING:toil.jobGraph:Due to failure we are reducing the remaining retry count of job 'generate_iso_seq_hints' t/Z/jobbxMOiF with ID t/Z/jobbxMOiF to 0
<=========

03-30 10:33:59 luigi-interface INFO  Informed scheduler that task   GenerateHints_False_True_True_aa7607109d   has status   FAILED
03-30 10:33:59 luigi-interface ERROR [pid 35767] Worker Worker(salt=724431596, workers=10, host=subnet190-2.nwsuaf.edu.cn, username=liran, pid=35207) failed    ToilTask: GenerateHints for galGal4 using batchSystem singleMachine
Traceback (most recent call last):
  File "/home/liran/Software/miniconda2/envs/C-A-T/lib/python2.7/site-packages/luigi/worker.py", line 199, in run
    new_deps = self._run_get_new_deps()
  File "/home/liran/Software/miniconda2/envs/C-A-T/lib/python2.7/site-packages/luigi/worker.py", line 141, in _run_get_new_deps
    task_gen = self.task.run()
  File "/home/liran/Software/C-A-T/Comparative-Annotation-Toolkit/cat/__init__.py", line 1053, in run
    hints_db(self.hints_args, toil_options)
  File "/home/liran/Software/C-A-T/Comparative-Annotation-Toolkit/cat/hints_db.py", line 84, in hints_db
    combined_hints = t.restart()
  File "/home/liran/Software/miniconda2/envs/C-A-T/lib/python2.7/site-packages/toil/common.py", line 700, in restart
    return self._runMainLoop(rootJobGraph)
  File "/home/liran/Software/miniconda2/envs/C-A-T/lib/python2.7/site-packages/toil/common.py", line 984, in _runMainLoop
    jobCache=self._jobCache).run()
  File "/home/liran/Software/miniconda2/envs/C-A-T/lib/python2.7/site-packages/toil/leader.py", line 200, in run
    raise FailedJobsException(self.config.jobStore, self.toilState.totalFailedJobs, self.jobStore)
FailedJobsException: The job store 'file:/home/liran/Software/C-A-T/Comparative-Annotation-Toolkit/test_install/toil/hints_db/galGal4/jobStore' contains 2 failed jobs: 'namesort_bam' 8/V/jobc4vH8x, 'setup_hints' u/L/job0lS851
=========> Failed job 'namesort_bam' 8/V/jobc4vH8x
---TOIL WORKER OUTPUT LOG---
INFO:toil:Running Toil version 3.5.0-7f5bc1cab176d99d6996076ae5b8bc2453fa04a4.
WARNING:toil.resource:Can't find resource for leader path '/home/liran/Software/C-A-T/Comparative-Annotation-Toolkit/cat'
WARNING:toil.resource:Can't localize module ModuleDescriptor(dirPath='/home/liran/Software/C-A-T/Comparative-Annotation-Toolkit', name='cat.hints_db', fromVirtualEnv=False)
INFO:luigi-interface:Loaded ['luigi.cfg']
INFO:toil.fileStore:Starting job ('namesort_bam' 8/V/jobc4vH8x) with ID (87279c8647989aceb657e93f65389bbd4c04219c).
WARNING:toil.resource:Can't find resource for leader path '/home/liran/Software/C-A-T/Comparative-Annotation-Toolkit/cat'
WARNING:toil.resource:Can't localize module ModuleDescriptor(dirPath='/home/liran/Software/C-A-T/Comparative-Annotation-Toolkit', name='cat.hints_db', fromVirtualEnv=False)
samtools: error while loading shared libraries: libcrypto.so.1.0.0: cannot open shared object file: No such file or directory
sambamba-sort: not enough data in stream
Traceback (most recent call last):
  File "/home/liran/Software/miniconda2/envs/C-A-T/lib/python2.7/site-packages/toil/worker.py", line 340, in main
    job._runner(jobGraph=jobGraph, jobStore=jobStore, fileStore=fileStore)
  File "/home/liran/Software/miniconda2/envs/C-A-T/lib/python2.7/site-packages/toil/job.py", line 1270, in _runner
    returnValues = self._run(jobGraph, fileStore)
  File "/home/liran/Software/miniconda2/envs/C-A-T/lib/python2.7/site-packages/toil/job.py", line 1217, in _run
    return self.run(fileStore)
  File "/home/liran/Software/miniconda2/envs/C-A-T/lib/python2.7/site-packages/toil/job.py", line 1383, in run
    rValue = userFunction(*((self,) + tuple(self._args)), **self._kwargs)
  File "/home/liran/Software/C-A-T/Comparative-Annotation-Toolkit/cat/hints_db.py", line 161, in namesort_bam
    tools.procOps.run_proc(cmd, stdout=name_sorted)
  File "/home/liran/Software/C-A-T/Comparative-Annotation-Toolkit/tools/procOps.py", line 68, in run_proc
    pl.wait()
  File "/home/liran/Software/C-A-T/Comparative-Annotation-Toolkit/tools/pipeline.py", line 1127, in wait
    self.raiseIfExcept()
  File "/home/liran/Software/C-A-T/Comparative-Annotation-Toolkit/tools/pipeline.py", line 1085, in raiseIfExcept
    p.raiseIfExcept()
  File "/home/liran/Software/C-A-T/Comparative-Annotation-Toolkit/tools/pipeline.py", line 749, in raiseIfExcept
    raise self.exceptInfo[0], self.exceptInfo[1], self.exceptInfo[2]
ProcException: process exited 127: samtools view -b /tmp/toil-3fcac38b-8337-485e-a366-851a45df2826/tmpuIJTun/0ef8bc56-4882-431c-9ad2-3da22248ee5d/tmpXltoEJ.tmp chr14
ERROR:toil.worker:Exiting the worker because of a failed job on host subnet190-2.nwsuaf.edu.cn
WARNING:toil.jobGraph:Due to failure we are reducing the remaining retry count of job 'namesort_bam' 8/V/jobc4vH8x with ID 8/V/jobc4vH8x to 0
<=========

03-30 10:33:59 luigi-interface INFO  Informed scheduler that task   GenerateHints_False_True_True_7ca55844cb   has status   FAILED
03-30 10:34:03 luigi-interface ERROR [pid 35770] Worker Worker(salt=724431596, workers=10, host=subnet190-2.nwsuaf.edu.cn, username=liran, pid=35207) failed    ToilTask: GenerateHints for mm10 using batchSystem singleMachine
Traceback (most recent call last):
  File "/home/liran/Software/miniconda2/envs/C-A-T/lib/python2.7/site-packages/luigi/worker.py", line 199, in run
    new_deps = self._run_get_new_deps()
  File "/home/liran/Software/miniconda2/envs/C-A-T/lib/python2.7/site-packages/luigi/worker.py", line 141, in _run_get_new_deps
    task_gen = self.task.run()
  File "/home/liran/Software/C-A-T/Comparative-Annotation-Toolkit/cat/__init__.py", line 1053, in run
    hints_db(self.hints_args, toil_options)
  File "/home/liran/Software/C-A-T/Comparative-Annotation-Toolkit/cat/hints_db.py", line 84, in hints_db
    combined_hints = t.restart()
  File "/home/liran/Software/miniconda2/envs/C-A-T/lib/python2.7/site-packages/toil/common.py", line 700, in restart
    return self._runMainLoop(rootJobGraph)
  File "/home/liran/Software/miniconda2/envs/C-A-T/lib/python2.7/site-packages/toil/common.py", line 984, in _runMainLoop
    jobCache=self._jobCache).run()
  File "/home/liran/Software/miniconda2/envs/C-A-T/lib/python2.7/site-packages/toil/leader.py", line 200, in run
    raise FailedJobsException(self.config.jobStore, self.toilState.totalFailedJobs, self.jobStore)
FailedJobsException: The job store 'file:/home/liran/Software/C-A-T/Comparative-Annotation-Toolkit/test_install/toil/hints_db/mm10/jobStore' contains 4 failed jobs: 'namesort_bam' r/j/jobHUG8TV, 'namesort_bam' X/R/jobbpn764, 'namesort_bam' V/4/job6X8beL, 'setup_hints' 5/e/jobs0eWrH
=========> Failed job 'namesort_bam' r/j/jobHUG8TV
---TOIL WORKER OUTPUT LOG---
INFO:toil:Running Toil version 3.5.0-7f5bc1cab176d99d6996076ae5b8bc2453fa04a4.
WARNING:toil.resource:Can't find resource for leader path '/home/liran/Software/C-A-T/Comparative-Annotation-Toolkit/cat'
WARNING:toil.resource:Can't localize module ModuleDescriptor(dirPath='/home/liran/Software/C-A-T/Comparative-Annotation-Toolkit', name='cat.hints_db', fromVirtualEnv=False)
INFO:luigi-interface:Loaded ['luigi.cfg']
INFO:toil.fileStore:Starting job ('namesort_bam' r/j/jobHUG8TV) with ID (b776329fa019fddcf731411725fdcd084fc41346).
WARNING:toil.resource:Can't find resource for leader path '/home/liran/Software/C-A-T/Comparative-Annotation-Toolkit/cat'
WARNING:toil.resource:Can't localize module ModuleDescriptor(dirPath='/home/liran/Software/C-A-T/Comparative-Annotation-Toolkit', name='cat.hints_db', fromVirtualEnv=False)
samtools: error while loading shared libraries: libcrypto.so.1.0.0: cannot open shared object file: No such file or directory
sambamba-sort: not enough data in stream
Traceback (most recent call last):
  File "/home/liran/Software/miniconda2/envs/C-A-T/lib/python2.7/site-packages/toil/worker.py", line 340, in main
    job._runner(jobGraph=jobGraph, jobStore=jobStore, fileStore=fileStore)
  File "/home/liran/Software/miniconda2/envs/C-A-T/lib/python2.7/site-packages/toil/job.py", line 1270, in _runner
    returnValues = self._run(jobGraph, fileStore)
  File "/home/liran/Software/miniconda2/envs/C-A-T/lib/python2.7/site-packages/toil/job.py", line 1217, in _run
    return self.run(fileStore)
  File "/home/liran/Software/miniconda2/envs/C-A-T/lib/python2.7/site-packages/toil/job.py", line 1383, in run
    rValue = userFunction(*((self,) + tuple(self._args)), **self._kwargs)
  File "/home/liran/Software/C-A-T/Comparative-Annotation-Toolkit/cat/hints_db.py", line 161, in namesort_bam
    tools.procOps.run_proc(cmd, stdout=name_sorted)
  File "/home/liran/Software/C-A-T/Comparative-Annotation-Toolkit/tools/procOps.py", line 68, in run_proc
    pl.wait()
  File "/home/liran/Software/C-A-T/Comparative-Annotation-Toolkit/tools/pipeline.py", line 1127, in wait
    self.raiseIfExcept()
  File "/home/liran/Software/C-A-T/Comparative-Annotation-Toolkit/tools/pipeline.py", line 1085, in raiseIfExcept
    p.raiseIfExcept()
  File "/home/liran/Software/C-A-T/Comparative-Annotation-Toolkit/tools/pipeline.py", line 749, in raiseIfExcept
    raise self.exceptInfo[0], self.exceptInfo[1], self.exceptInfo[2]
ProcException: process exited 127: samtools view -b /tmp/toil-3a71830d-df1e-460d-b3df-50f0633d456e/tmpakvgey/04d45eaf-4719-488b-989e-a9424a58a4ed/tmpdYlxSD.tmp chr17
ERROR:toil.worker:Exiting the worker because of a failed job on host subnet190-2.nwsuaf.edu.cn
WARNING:toil.jobGraph:Due to failure we are reducing the remaining retry count of job 'namesort_bam' r/j/jobHUG8TV with ID r/j/jobHUG8TV to 0
<=========

=========> Failed job 'namesort_bam' X/R/jobbpn764
---TOIL WORKER OUTPUT LOG---
INFO:toil:Running Toil version 3.5.0-7f5bc1cab176d99d6996076ae5b8bc2453fa04a4.
WARNING:toil.resource:Can't find resource for leader path '/home/liran/Software/C-A-T/Comparative-Annotation-Toolkit/cat'
WARNING:toil.resource:Can't localize module ModuleDescriptor(dirPath='/home/liran/Software/C-A-T/Comparative-Annotation-Toolkit', name='cat.hints_db', fromVirtualEnv=False)
INFO:luigi-interface:Loaded ['luigi.cfg']
INFO:toil.fileStore:Starting job ('namesort_bam' X/R/jobbpn764) with ID (cca09013a4e0d74a60cb5fd59d573d52bae1f6ca).
WARNING:toil.resource:Can't find resource for leader path '/home/liran/Software/C-A-T/Comparative-Annotation-Toolkit/cat'
WARNING:toil.resource:Can't localize module ModuleDescriptor(dirPath='/home/liran/Software/C-A-T/Comparative-Annotation-Toolkit', name='cat.hints_db', fromVirtualEnv=False)
samtools: error while loading shared libraries: libcrypto.so.1.0.0: cannot open shared object file: No such file or directory
sambamba-sort: not enough data in stream
Traceback (most recent call last):
  File "/home/liran/Software/miniconda2/envs/C-A-T/lib/python2.7/site-packages/toil/worker.py", line 340, in main
    job._runner(jobGraph=jobGraph, jobStore=jobStore, fileStore=fileStore)
  File "/home/liran/Software/miniconda2/envs/C-A-T/lib/python2.7/site-packages/toil/job.py", line 1270, in _runner
    returnValues = self._run(jobGraph, fileStore)
  File "/home/liran/Software/miniconda2/envs/C-A-T/lib/python2.7/site-packages/toil/job.py", line 1217, in _run
    return self.run(fileStore)
  File "/home/liran/Software/miniconda2/envs/C-A-T/lib/python2.7/site-packages/toil/job.py", line 1383, in run
    rValue = userFunction(*((self,) + tuple(self._args)), **self._kwargs)
  File "/home/liran/Software/C-A-T/Comparative-Annotation-Toolkit/cat/hints_db.py", line 161, in namesort_bam
    tools.procOps.run_proc(cmd, stdout=name_sorted)
  File "/home/liran/Software/C-A-T/Comparative-Annotation-Toolkit/tools/procOps.py", line 68, in run_proc
    pl.wait()
  File "/home/liran/Software/C-A-T/Comparative-Annotation-Toolkit/tools/pipeline.py", line 1127, in wait
    self.raiseIfExcept()
  File "/home/liran/Software/C-A-T/Comparative-Annotation-Toolkit/tools/pipeline.py", line 1085, in raiseIfExcept
    p.raiseIfExcept()
  File "/home/liran/Software/C-A-T/Comparative-Annotation-Toolkit/tools/pipeline.py", line 749, in raiseIfExcept
    raise self.exceptInfo[0], self.exceptInfo[1], self.exceptInfo[2]
ProcException: process exited 127: samtools view -b /tmp/toil-3a71830d-df1e-460d-b3df-50f0633d456e/tmpKHYgxy/ae5fdc4d-6925-4bc7-ba9e-219e6fd42641/tmpT43J37.tmp chr17
ERROR:toil.worker:Exiting the worker because of a failed job on host subnet190-2.nwsuaf.edu.cn
WARNING:toil.jobGraph:Due to failure we are reducing the remaining retry count of job 'namesort_bam' X/R/jobbpn764 with ID X/R/jobbpn764 to 0
<=========

=========> Failed job 'namesort_bam' V/4/job6X8beL
---TOIL WORKER OUTPUT LOG---
INFO:toil:Running Toil version 3.5.0-7f5bc1cab176d99d6996076ae5b8bc2453fa04a4.
WARNING:toil.resource:Can't find resource for leader path '/home/liran/Software/C-A-T/Comparative-Annotation-Toolkit/cat'
WARNING:toil.resource:Can't localize module ModuleDescriptor(dirPath='/home/liran/Software/C-A-T/Comparative-Annotation-Toolkit', name='cat.hints_db', fromVirtualEnv=False)
INFO:luigi-interface:Loaded ['luigi.cfg']
INFO:toil.fileStore:Starting job ('namesort_bam' V/4/job6X8beL) with ID (58da24787b046b0440994ecffb59a24b137a50d2).
WARNING:toil.resource:Can't find resource for leader path '/home/liran/Software/C-A-T/Comparative-Annotation-Toolkit/cat'
WARNING:toil.resource:Can't localize module ModuleDescriptor(dirPath='/home/liran/Software/C-A-T/Comparative-Annotation-Toolkit', name='cat.hints_db', fromVirtualEnv=False)
samtools: error while loading shared libraries: libcrypto.so.1.0.0: cannot open shared object file: No such file or directory
sambamba-sort: not enough data in stream
Traceback (most recent call last):
  File "/home/liran/Software/miniconda2/envs/C-A-T/lib/python2.7/site-packages/toil/worker.py", line 340, in main
    job._runner(jobGraph=jobGraph, jobStore=jobStore, fileStore=fileStore)
  File "/home/liran/Software/miniconda2/envs/C-A-T/lib/python2.7/site-packages/toil/job.py", line 1270, in _runner
    returnValues = self._run(jobGraph, fileStore)
  File "/home/liran/Software/miniconda2/envs/C-A-T/lib/python2.7/site-packages/toil/job.py", line 1217, in _run
    return self.run(fileStore)
  File "/home/liran/Software/miniconda2/envs/C-A-T/lib/python2.7/site-packages/toil/job.py", line 1383, in run
    rValue = userFunction(*((self,) + tuple(self._args)), **self._kwargs)
  File "/home/liran/Software/C-A-T/Comparative-Annotation-Toolkit/cat/hints_db.py", line 161, in namesort_bam
    tools.procOps.run_proc(cmd, stdout=name_sorted)
  File "/home/liran/Software/C-A-T/Comparative-Annotation-Toolkit/tools/procOps.py", line 68, in run_proc
    pl.wait()
  File "/home/liran/Software/C-A-T/Comparative-Annotation-Toolkit/tools/pipeline.py", line 1127, in wait
    self.raiseIfExcept()
  File "/home/liran/Software/C-A-T/Comparative-Annotation-Toolkit/tools/pipeline.py", line 1085, in raiseIfExcept
    p.raiseIfExcept()
  File "/home/liran/Software/C-A-T/Comparative-Annotation-Toolkit/tools/pipeline.py", line 749, in raiseIfExcept
    raise self.exceptInfo[0], self.exceptInfo[1], self.exceptInfo[2]
ProcException: process exited 127: samtools view -b /tmp/toil-3a71830d-df1e-460d-b3df-50f0633d456e/tmpSZI7pC/778fec04-5972-4eb0-8ba0-6b0842da5065/tmpxeYvLw.tmp chr17
ERROR:toil.worker:Exiting the worker because of a failed job on host subnet190-2.nwsuaf.edu.cn
WARNING:toil.jobGraph:Due to failure we are reducing the remaining retry count of job 'namesort_bam' V/4/job6X8beL with ID V/4/job6X8beL to 0
<=========

03-30 10:34:03 luigi-interface INFO  Informed scheduler that task   GenerateHints_False_True_True_062470fd8f   has status   FAILED
03-30 10:34:03 luigi-interface INFO  Worker Worker(salt=724431596, workers=10, host=subnet190-2.nwsuaf.edu.cn, username=liran, pid=35207) was stopped. Shutting down Keep-Alive thread
03-30 10:34:05 luigi-interface INFO
===== Luigi Execution Summary =====

Scheduled 127 tasks of which:
* 10 complete ones were encountered:
    - 1 Chaining(...)
    - 1 EvaluateTransMap(...)
    - 4 GenerateHints(...)
    - 1 GenomeFiles(...)
    - 1 PrepareFiles(...)
    ...
* 4 failed:
    - 4 GenerateHints(...)
* 113 were left pending, among these:
    * 113 had failed dependencies:
        - 7 AlignTranscriptDriverTask(...)
        - 1 AlignTranscripts(...)
        - 1 AssemblyHub(...)
        - 1 Augustus(...)
        - 1 AugustusCgp(...)
        ...

This progress looks :( because there were failed tasks

===== Luigi Execution Summary =====
ifiddes commented 4 years ago

Something is wrong with the library linking of your samtools binary. This is the relevant error message:

samtools: error while loading shared libraries: libcrypto.so.1.0.0: cannot open shared object file: No such file or directory

Ian Fiddes, PhD

On Sun, Mar 29, 2020 at 7:37 PM YPGG1234 notifications@github.com wrote:

Thanks for your mention, I have tried it , but I got :

03-30 10:33:48 luigi-interface INFO Informed scheduler that task RunCat_False_True_True_a22c0dbb99 has status PENDING 03-30 10:33:48 luigi-interface INFO Informed scheduler that task ReportStats_False_True_True_a22c0dbb99 has status PENDING 03-30 10:33:48 luigi-interface INFO Informed scheduler that task AssemblyHub_False_True_True_a22c0dbb99 has status PENDING 03-30 10:33:48 luigi-interface INFO Informed scheduler that task CreateTracks_False_True_True_a22c0dbb99 has status PENDING 03-30 10:33:48 luigi-interface INFO Informed scheduler that task CreateTrackDbs_False_True_True_2bb603043f has status PENDING 03-30 10:33:48 luigi-interface INFO Informed scheduler that task CreateTracksDriverTask_False_True_True_2bb603043f has status PENDING 03-30 10:33:49 luigi-interface INFO Informed scheduler that task SpliceTrack_False_True_True_414df3c083 has status PENDING 03-30 10:33:49 luigi-interface INFO Informed scheduler that task ReferenceFiles_False_True_True_a22c0dbb99 has status DONE 03-30 10:33:49 luigi-interface INFO Informed scheduler that task BgpTrack_False_True_True_ef8b57a4b3 has status PENDING 03-30 10:33:49 luigi-interface INFO Informed scheduler that task CreateTrackDbs_False_True_True_962ce509b6 has status PENDING 03-30 10:33:49 luigi-interface INFO Informed scheduler that task CreateTracksDriverTask_False_True_True_962ce509b6 has status PENDING 03-30 10:33:49 luigi-interface INFO Informed scheduler that task SpliceTrack_False_True_True_0ba10fdc5e has status PENDING 03-30 10:33:49 luigi-interface INFO Informed scheduler that task AugustusTrack_False_True_True_fe5b130f82 has status PENDING 03-30 10:33:49 luigi-interface INFO Informed scheduler that task BgpTrack_False_True_True_b3c6e38954 has status PENDING 03-30 10:33:49 luigi-interface INFO Informed scheduler that task TransMapTrack_False_True_True_d057e76fa7 has status PENDING 03-30 10:33:49 luigi-interface INFO Informed scheduler that task EvaluationTrack_False_True_True_e09c287d51 has status PENDING 03-30 10:33:49 luigi-interface INFO Informed scheduler that task ConsensusTrack_False_True_True_899aaff4b5 has status PENDING 03-30 10:33:49 luigi-interface INFO Informed scheduler that task DenovoTrack_False_True_True_19d83a84cd has status PENDING 03-30 10:33:49 luigi-interface INFO Informed scheduler that task CreateTrackDbs_False_True_True_a56ac842ed has status PENDING 03-30 10:33:49 luigi-interface INFO Informed scheduler that task CreateTracksDriverTask_False_True_True_a56ac842ed has status PENDING 03-30 10:33:49 luigi-interface INFO Informed scheduler that task BgpTrack_False_True_True_a045b3eed3 has status PENDING 03-30 10:33:49 luigi-interface INFO Informed scheduler that task TransMapTrack_False_True_True_f8c1c7c8f1 has status PENDING 03-30 10:33:49 luigi-interface INFO Informed scheduler that task EvaluationTrack_False_True_True_1369572bbd has status PENDING 03-30 10:33:49 luigi-interface INFO Informed scheduler that task ConsensusTrack_False_True_True_2e26a53515 has status PENDING 03-30 10:33:49 luigi-interface INFO Informed scheduler that task DenovoTrack_False_True_True_4964d40230 has status PENDING 03-30 10:33:49 luigi-interface INFO Informed scheduler that task CreateTrackDbs_False_True_True_5e66b93a27 has status PENDING 03-30 10:33:49 luigi-interface INFO Informed scheduler that task CreateTracksDriverTask_False_True_True_5e66b93a27 has status PENDING 03-30 10:33:49 luigi-interface INFO Informed scheduler that task BgpTrack_False_True_True_06e82f2455 has status PENDING 03-30 10:33:49 luigi-interface INFO Informed scheduler that task TransMapTrack_False_True_True_247bc669c6 has status PENDING 03-30 10:33:49 luigi-interface INFO Informed scheduler that task EvaluationTrack_False_True_True_1fb03ca6f2 has status PENDING 03-30 10:33:49 luigi-interface INFO Informed scheduler that task ConsensusTrack_False_True_True_22451c1436 has status PENDING 03-30 10:33:49 luigi-interface INFO Informed scheduler that task DenovoTrack_False_True_True_07fbf61039 has status PENDING 03-30 10:33:49 luigi-interface INFO Informed scheduler that task CreateTrackDbs_False_True_True_ffd7733252 has status PENDING 03-30 10:33:49 luigi-interface INFO Informed scheduler that task CreateTracksDriverTask_False_True_True_ffd7733252 has status PENDING 03-30 10:33:49 luigi-interface INFO Informed scheduler that task BgpTrack_False_True_True_4e47ec61fc has status PENDING 03-30 10:33:49 luigi-interface INFO Informed scheduler that task TransMapTrack_False_True_True_d1c0ce3423 has status PENDING 03-30 10:33:49 luigi-interface INFO Informed scheduler that task EvaluationTrack_False_True_True_0df56c9568 has status PENDING 03-30 10:33:49 luigi-interface INFO Informed scheduler that task ConsensusTrack_False_True_True_16101cfad8 has status PENDING 03-30 10:33:49 luigi-interface INFO Informed scheduler that task DenovoTrack_False_True_True_ac89a6e910 has status PENDING 03-30 10:33:49 luigi-interface INFO Informed scheduler that task CreateTrackDbs_False_True_True_b2ea75e80d has status PENDING 03-30 10:33:49 luigi-interface INFO Informed scheduler that task CreateTracksDriverTask_False_True_True_b2ea75e80d has status PENDING 03-30 10:33:49 luigi-interface INFO Informed scheduler that task SpliceTrack_False_True_True_a2c639d7d0 has status PENDING 03-30 10:33:49 luigi-interface INFO Informed scheduler that task AugustusTrack_False_True_True_eaa4ef7f4c has status PENDING 03-30 10:33:49 luigi-interface INFO Informed scheduler that task BgpTrack_False_True_True_6bb68024a2 has status PENDING 03-30 10:33:49 luigi-interface INFO Informed scheduler that task TransMapTrack_False_True_True_26942e186f has status PENDING 03-30 10:33:49 luigi-interface INFO Informed scheduler that task EvaluationTrack_False_True_True_d4b3522d4f has status PENDING 03-30 10:33:49 luigi-interface INFO Informed scheduler that task ConsensusTrack_False_True_True_3e54a7a8a3 has status PENDING 03-30 10:33:49 luigi-interface INFO Informed scheduler that task DenovoTrack_False_True_True_20652ea98b has status PENDING 03-30 10:33:49 luigi-interface INFO Informed scheduler that task CreateTrackDbs_False_True_True_df60fe5db8 has status PENDING 03-30 10:33:49 luigi-interface INFO Informed scheduler that task CreateTracksDriverTask_False_True_True_df60fe5db8 has status PENDING 03-30 10:33:49 luigi-interface INFO Informed scheduler that task SpliceTrack_False_True_True_c06a29233f has status PENDING 03-30 10:33:49 luigi-interface INFO Informed scheduler that task IsoSeqBamTrack_False_True_True_174d8e5f0a has status PENDING 03-30 10:33:49 luigi-interface INFO Informed scheduler that task AugustusTrack_False_True_True_94fd6fc914 has status PENDING 03-30 10:33:49 luigi-interface INFO Informed scheduler that task BgpTrack_False_True_True_44ade36bc4 has status PENDING 03-30 10:33:49 luigi-interface INFO Informed scheduler that task TransMapTrack_False_True_True_c7fa399adc has status PENDING 03-30 10:33:49 luigi-interface INFO Informed scheduler that task EvaluationTrack_False_True_True_3e752a8281 has status PENDING 03-30 10:33:49 luigi-interface INFO Informed scheduler that task ConsensusTrack_False_True_True_df65d3091f has status PENDING 03-30 10:33:49 luigi-interface INFO Informed scheduler that task DenovoTrack_False_True_True_bbb549ac8d has status PENDING 03-30 10:33:49 luigi-interface INFO Informed scheduler that task DenovoTrack_False_True_True_6fa8e23eac has status PENDING 03-30 10:33:49 luigi-interface INFO Informed scheduler that task CreateTrackDbs_False_True_True_185cafd88d has status PENDING 03-30 10:33:49 luigi-interface INFO Informed scheduler that task CreateTracksDriverTask_False_True_True_185cafd88d has status PENDING 03-30 10:33:49 luigi-interface INFO Informed scheduler that task BgpTrack_False_True_True_a52a90a819 has status PENDING 03-30 10:33:49 luigi-interface INFO Informed scheduler that task TransMapTrack_False_True_True_28e168baa8 has status PENDING 03-30 10:33:49 luigi-interface INFO Informed scheduler that task EvaluationTrack_False_True_True_44b3b1b596 has status PENDING 03-30 10:33:49 luigi-interface INFO Informed scheduler that task ConsensusTrack_False_True_True_3d71bb2e15 has status PENDING 03-30 10:33:49 luigi-interface INFO Informed scheduler that task DenovoTrack_False_True_True_efbd446679 has status PENDING 03-30 10:33:49 luigi-interface INFO Informed scheduler that task CreateDirectoryStructure_False_True_True_a22c0dbb99 has status PENDING 03-30 10:33:49 luigi-interface INFO Informed scheduler that task Plots_False_True_True_a22c0dbb99 has status PENDING 03-30 10:33:49 luigi-interface INFO Informed scheduler that task Consensus_False_True_True_a22c0dbb99 has status PENDING 03-30 10:33:49 luigi-interface INFO Informed scheduler that task ConsensusDriverTask_False_True_True_962ce509b6 has status PENDING 03-30 10:33:49 luigi-interface INFO Informed scheduler that task ConsensusDriverTask_False_True_True_a56ac842ed has status PENDING 03-30 10:33:49 luigi-interface INFO Informed scheduler that task ConsensusDriverTask_False_True_True_5e66b93a27 has status PENDING 03-30 10:33:49 luigi-interface INFO Informed scheduler that task ConsensusDriverTask_False_True_True_ffd7733252 has status PENDING 03-30 10:33:49 luigi-interface INFO Informed scheduler that task ConsensusDriverTask_False_True_True_b2ea75e80d has status PENDING 03-30 10:33:49 luigi-interface INFO Informed scheduler that task ConsensusDriverTask_False_True_True_df60fe5db8 has status PENDING 03-30 10:33:49 luigi-interface INFO Informed scheduler that task ConsensusDriverTask_False_True_True_185cafd88d has status PENDING 03-30 10:33:49 luigi-interface INFO Informed scheduler that task EvaluateTranscripts_False_True_True_a22c0dbb99 has status PENDING 03-30 10:33:49 luigi-interface INFO Informed scheduler that task EvaluateDriverTask_False_True_True_962ce509b6 has status PENDING 03-30 10:33:49 luigi-interface INFO Informed scheduler that task EvaluateDriverTask_False_True_True_a56ac842ed has status PENDING 03-30 10:33:49 luigi-interface INFO Informed scheduler that task EvaluateDriverTask_False_True_True_5e66b93a27 has status PENDING 03-30 10:33:49 luigi-interface INFO Informed scheduler that task EvaluateDriverTask_False_True_True_ffd7733252 has status PENDING 03-30 10:33:49 luigi-interface INFO Informed scheduler that task EvaluateDriverTask_False_True_True_b2ea75e80d has status PENDING 03-30 10:33:49 luigi-interface INFO Informed scheduler that task EvaluateDriverTask_False_True_True_df60fe5db8 has status PENDING 03-30 10:33:49 luigi-interface INFO Informed scheduler that task EvaluateDriverTask_False_True_True_185cafd88d has status PENDING 03-30 10:33:50 luigi-interface INFO Informed scheduler that task AlignTranscripts_False_True_True_a22c0dbb99 has status PENDING 03-30 10:33:50 luigi-interface INFO Informed scheduler that task AlignTranscriptDriverTask_False_True_True_962ce509b6 has status PENDING 03-30 10:33:50 luigi-interface INFO Informed scheduler that task GenomeFiles_False_True_True_a22c0dbb99 has status DONE 03-30 10:33:50 luigi-interface INFO Informed scheduler that task AlignTranscriptDriverTask_False_True_True_a56ac842ed has status PENDING 03-30 10:33:50 luigi-interface INFO Informed scheduler that task AlignTranscriptDriverTask_False_True_True_5e66b93a27 has status PENDING 03-30 10:33:50 luigi-interface INFO Informed scheduler that task AlignTranscriptDriverTask_False_True_True_ffd7733252 has status PENDING 03-30 10:33:50 luigi-interface INFO Informed scheduler that task AlignTranscriptDriverTask_False_True_True_b2ea75e80d has status PENDING 03-30 10:33:50 luigi-interface INFO Informed scheduler that task AlignTranscriptDriverTask_False_True_True_df60fe5db8 has status PENDING 03-30 10:33:50 luigi-interface INFO Informed scheduler that task AlignTranscriptDriverTask_False_True_True_185cafd88d has status PENDING 03-30 10:33:50 luigi-interface INFO Informed scheduler that task Hgm_False_True_True_a22c0dbb99 has status PENDING 03-30 10:33:50 luigi-interface INFO Informed scheduler that task HgmDriverTask_False_True_True_e079ff495e has status PENDING 03-30 10:33:50 luigi-interface INFO Informed scheduler that task HgmDriverTask_False_True_True_2f9793b986 has status PENDING 03-30 10:33:50 luigi-interface INFO Informed scheduler that task HgmDriverTask_False_True_True_3324565358 has status PENDING 03-30 10:33:50 luigi-interface INFO Informed scheduler that task HgmDriverTask_False_True_True_6cb70a33d5 has status PENDING 03-30 10:33:50 luigi-interface INFO Informed scheduler that task HgmDriverTask_False_True_True_a7f6ffd80e has status PENDING 03-30 10:33:50 luigi-interface INFO Informed scheduler that task IsoSeqTranscripts_False_True_True_a22c0dbb99 has status PENDING 03-30 10:33:50 luigi-interface INFO Informed scheduler that task IsoSeqTranscriptsDriverTask_False_True_True_df60fe5db8 has status PENDING 03-30 10:33:50 luigi-interface INFO Informed scheduler that task FindDenovoParents_False_True_True_e079ff495e has status PENDING 03-30 10:33:50 luigi-interface INFO Informed scheduler that task AugustusPb_False_True_True_a22c0dbb99 has status PENDING 03-30 10:33:50 luigi-interface INFO Informed scheduler that task AugustusPbDriverTask_False_True_True_df60fe5db8 has status PENDING 03-30 10:33:50 luigi-interface INFO Informed scheduler that task FindDenovoParents_False_True_True_6cb70a33d5 has status PENDING 03-30 10:33:50 luigi-interface INFO Informed scheduler that task AugustusCgp_False_True_True_a22c0dbb99 has status PENDING 03-30 10:33:50 luigi-interface INFO Informed scheduler that task Augustus_False_True_True_a22c0dbb99 has status PENDING 03-30 10:33:50 luigi-interface INFO Informed scheduler that task AugustusDriverTask_False_True_True_962ce509b6 has status PENDING 03-30 10:33:50 luigi-interface INFO Informed scheduler that task AugustusDriverTask_False_True_True_a56ac842ed has status PENDING 03-30 10:33:50 luigi-interface INFO Informed scheduler that task AugustusDriverTask_False_True_True_5e66b93a27 has status PENDING 03-30 10:33:50 luigi-interface INFO Informed scheduler that task AugustusDriverTask_False_True_True_ffd7733252 has status PENDING 03-30 10:33:50 luigi-interface INFO Informed scheduler that task AugustusDriverTask_False_True_True_b2ea75e80d has status PENDING 03-30 10:33:50 luigi-interface INFO Informed scheduler that task AugustusDriverTask_False_True_True_df60fe5db8 has status PENDING 03-30 10:33:50 luigi-interface INFO Informed scheduler that task AugustusDriverTask_False_True_True_185cafd88d has status PENDING 03-30 10:33:50 luigi-interface INFO Informed scheduler that task EvaluateTransMap_False_True_True_a22c0dbb99 has status DONE 03-30 10:33:50 luigi-interface INFO Informed scheduler that task TransMap_False_True_True_a22c0dbb99 has status DONE 03-30 10:33:50 luigi-interface INFO Informed scheduler that task Chaining_False_True_True_a22c0dbb99 has status DONE 03-30 10:33:50 luigi-interface INFO Informed scheduler that task BuildDb_False_True_True_a22c0dbb99 has status PENDING 03-30 10:33:50 luigi-interface INFO Informed scheduler that task GenerateHints_False_True_True_062470fd8f has status PENDING 03-30 10:33:50 luigi-interface INFO Informed scheduler that task GenerateHints_False_True_True_7ca55844cb has status PENDING 03-30 10:33:50 luigi-interface INFO Informed scheduler that task GenerateHints_False_True_True_38092cda82 has status DONE 03-30 10:33:50 luigi-interface INFO Informed scheduler that task GenerateHints_False_True_True_59d3d5afbf has status DONE 03-30 10:33:50 luigi-interface INFO Informed scheduler that task GenerateHints_False_True_True_d94badbc23 has status DONE 03-30 10:33:50 luigi-interface INFO Informed scheduler that task GenerateHints_False_True_True_0ccd06bf5f has status PENDING 03-30 10:33:50 luigi-interface INFO Informed scheduler that task GenerateHints_False_True_True_aa7607109d has status PENDING 03-30 10:33:50 luigi-interface INFO Informed scheduler that task GenerateHints_False_True_True_baf78e6f16 has status DONE 03-30 10:33:50 luigi-interface INFO Informed scheduler that task PrepareFiles_False_True_True_a22c0dbb99 has status DONE 03-30 10:33:50 luigi-interface INFO Done scheduling tasks 03-30 10:33:50 luigi-interface INFO Running Worker with 10 processes 03-30 10:33:50 luigi-interface INFO [pid 35764] Worker Worker(salt=724431596, workers=10, host=subnet190-2.nwsuaf.edu.cn, username=liran, pid=35207) running ToilTask: GenerateHints for rheMac3 using batchSystem singleMachine 03-30 10:33:50 luigi-interface INFO [pid 35765] Worker Worker(salt=724431596, workers=10, host=subnet190-2.nwsuaf.edu.cn, username=liran, pid=35207) running ToilTask: GenerateHints for hg38 using batchSystem singleMachine 03-30 10:33:50 luigi-interface INFO [pid 35767] Worker Worker(salt=724431596, workers=10, host=subnet190-2.nwsuaf.edu.cn, username=liran, pid=35207) running ToilTask: GenerateHints for galGal4 using batchSystem singleMachine 03-30 10:33:50 luigi-interface INFO [pid 35770] Worker Worker(salt=724431596, workers=10, host=subnet190-2.nwsuaf.edu.cn, username=liran, pid=35207) running ToilTask: GenerateHints for mm10 using batchSystem singleMachine 03-30 10:33:51 cat INFO Beginning GenerateHints Toil pipeline for rheMac3. 03-30 10:33:51 cat INFO Beginning GenerateHints Toil pipeline for hg38. 03-30 10:33:51 cat INFO Beginning GenerateHints Toil pipeline for galGal4. 03-30 10:33:51 cat.hints_db INFO Restarting Toil hints pipeline for rheMac3. 03-30 10:33:51 cat INFO Beginning GenerateHints Toil pipeline for mm10. 03-30 10:33:51 cat.hints_db INFO Restarting Toil hints pipeline for hg38. 03-30 10:33:51 toil.jobStores.abstractJobStore CRITICAL Repairing job: S/f/jobWsVBMf 03-30 10:33:51 toil.jobStores.abstractJobStore CRITICAL Repairing job: S/f/jobWsVBMf 03-30 10:33:51 cat.hints_db INFO Restarting Toil hints pipeline for galGal4. 03-30 10:33:51 cat.hints_db INFO Restarting Toil hints pipeline for mm10. 03-30 10:33:51 toil.jobStores.abstractJobStore CRITICAL Repairing job: t/Z/jobbxMOiF 03-30 10:33:51 toil.jobStores.abstractJobStore CRITICAL Repairing job: t/Z/jobbxMOiF 03-30 10:33:51 toil.jobStores.abstractJobStore CRITICAL Repairing job: I/E/jobpE4Exm 03-30 10:33:51 toil.jobStores.abstractJobStore CRITICAL Repairing job: I/E/jobpE4Exm 03-30 10:33:51 toil.jobStores.abstractJobStore CRITICAL Repairing job: 8/V/jobc4vH8x 03-30 10:33:51 toil.jobStores.abstractJobStore CRITICAL Repairing job: 8/V/jobc4vH8x 03-30 10:33:51 toil.jobStores.abstractJobStore CRITICAL Repairing job: r/j/jobHUG8TV 03-30 10:33:51 toil.jobStores.abstractJobStore CRITICAL Repairing job: r/j/jobHUG8TV 03-30 10:33:51 toil.jobStores.abstractJobStore CRITICAL Repairing job: X/R/jobbpn764 03-30 10:33:51 toil.jobStores.abstractJobStore CRITICAL Repairing job: X/R/jobbpn764 03-30 10:33:51 toil.jobStores.abstractJobStore CRITICAL Repairing job: V/4/job6X8beL 03-30 10:33:51 toil.jobStores.abstractJobStore CRITICAL Repairing job: V/4/job6X8beL Exception in thread Thread-1: Traceback (most recent call last): File "/home/liran/Software/miniconda2/envs/C-A-T/lib/python2.7/threading.py", line 801, in bootstrap_inner self.run() File "/home/liran/Software/miniconda2/envs/C-A-T/lib/python2.7/threading.py", line 754, in run self.target(*self.args, **self.kwargs) File "/home/liran/Software/miniconda2/envs/C-A-T/lib/python2.7/site-packages/toil/fileStore.py", line 1461, in asyncWrite raise RuntimeError("The termination flag is set, exiting") RuntimeError: The termination flag is set, exiting Exception in thread Thread-2: Traceback (most recent call last): File "/home/liran/Software/miniconda2/envs/C-A-T/lib/python2.7/threading.py", line 801, in bootstrap_inner self.run() File "/home/liran/Software/miniconda2/envs/C-A-T/lib/python2.7/threading.py", line 754, in run self.target(*self.args, **self.kwargs) File "/home/liran/Software/miniconda2/envs/C-A-T/lib/python2.7/site-packages/toil/fileStore.py", line 1461, in asyncWrite raise RuntimeError("The termination flag is set, exiting") RuntimeError: The termination flag is set, exiting

Exception RuntimeError: RuntimeError('cannot join current thread',) in <bound method CachingFileStore.del of <toil.fileStore.CachingFileStore object at 0x7fb7de2da650>> ignored Exception in thread Thread-1: Traceback (most recent call last): File "/home/liran/Software/miniconda2/envs/C-A-T/lib/python2.7/threading.py", line 801, in __bootstrap_inner self.run() File "/home/liran/Software/miniconda2/envs/C-A-T/lib/python2.7/threading.py", line 754, in run self.target(*self.args, **self.__kwargs) File "/home/liran/Software/miniconda2/envs/C-A-T/lib/python2.7/site-packages/toil/fileStore.py", line 1461, in asyncWrite raise RuntimeError("The termination flag is set, exiting") RuntimeError: The termination flag is set, exiting

Exception in thread Thread-2: Traceback (most recent call last): File "/home/liran/Software/miniconda2/envs/C-A-T/lib/python2.7/threading.py", line 801, in __bootstrap_inner self.run() File "/home/liran/Software/miniconda2/envs/C-A-T/lib/python2.7/threading.py", line 754, in run self.target(*self.args, **self.__kwargs) File "/home/liran/Software/miniconda2/envs/C-A-T/lib/python2.7/site-packages/toil/fileStore.py", line 1461, in asyncWrite raise RuntimeError("The termination flag is set, exiting") RuntimeError: The termination flag is set, exiting

Exception RuntimeError: RuntimeError('cannot join current thread',) in <bound method CachingFileStore.del of <toil.fileStore.CachingFileStore object at 0x7f288681d650>> ignored Exception in thread Thread-1: Traceback (most recent call last): File "/home/liran/Software/miniconda2/envs/C-A-T/lib/python2.7/threading.py", line 801, in bootstrap_inner self.run() File "/home/liran/Software/miniconda2/envs/C-A-T/lib/python2.7/threading.py", line 754, in run self.target(*self.args, **self.kwargs) File "/home/liran/Software/miniconda2/envs/C-A-T/lib/python2.7/site-packages/toil/fileStore.py", line 1461, in asyncWrite raise RuntimeError("The termination flag is set, exiting") RuntimeError: The termination flag is set, exiting Exception in thread Thread-2: Traceback (most recent call last): File "/home/liran/Software/miniconda2/envs/C-A-T/lib/python2.7/threading.py", line 801, in bootstrap_inner self.run() File "/home/liran/Software/miniconda2/envs/C-A-T/lib/python2.7/threading.py", line 754, in run self.target(*self.args, **self.kwargs) File "/home/liran/Software/miniconda2/envs/C-A-T/lib/python2.7/site-packages/toil/fileStore.py", line 1461, in asyncWrite raise RuntimeError("The termination flag is set, exiting") RuntimeError: The termination flag is set, exiting

Exception RuntimeError: RuntimeError('cannot join current thread',) in <bound method CachingFileStore.del of <toil.fileStore.CachingFileStore object at 0x7f945367b650>> ignored Exception in thread Thread-1: Traceback (most recent call last): File "/home/liran/Software/miniconda2/envs/C-A-T/lib/python2.7/threading.py", line 801, in __bootstrap_inner self.run() File "/home/liran/Software/miniconda2/envs/C-A-T/lib/python2.7/threading.py", line 754, in run self.target(*self.args, **self.__kwargs) File "/home/liran/Software/miniconda2/envs/C-A-T/lib/python2.7/site-packages/toil/fileStore.py", line 1461, in asyncWrite raise RuntimeError("The termination flag is set, exiting") RuntimeError: The termination flag is set, exiting

Exception in thread Thread-2: Traceback (most recent call last): File "/home/liran/Software/miniconda2/envs/C-A-T/lib/python2.7/threading.py", line 801, in __bootstrap_inner self.run() File "/home/liran/Software/miniconda2/envs/C-A-T/lib/python2.7/threading.py", line 754, in run self.target(*self.args, **self.__kwargs) File "/home/liran/Software/miniconda2/envs/C-A-T/lib/python2.7/site-packages/toil/fileStore.py", line 1461, in asyncWrite raise RuntimeError("The termination flag is set, exiting") RuntimeError: The termination flag is set, exiting

Exception RuntimeError: RuntimeError('cannot join current thread',) in <bound method CachingFileStore.del of <toil.fileStore.CachingFileStore object at 0x7fb2f60e3650>> ignored Exception in thread Thread-1: Traceback (most recent call last): File "/home/liran/Software/miniconda2/envs/C-A-T/lib/python2.7/threading.py", line 801, in __bootstrap_inner self.run() File "/home/liran/Software/miniconda2/envs/C-A-T/lib/python2.7/threading.py", line 754, in run self.target(*self.args, **self.__kwargs) File "/home/liran/Software/miniconda2/envs/C-A-T/lib/python2.7/site-packages/toil/fileStore.py", line 1461, in asyncWrite raise RuntimeError("The termination flag is set, exiting") RuntimeError: The termination flag is set, exiting

Exception in thread Thread-2: Traceback (most recent call last): File "/home/liran/Software/miniconda2/envs/C-A-T/lib/python2.7/threading.py", line 801, in __bootstrap_inner self.run() File "/home/liran/Software/miniconda2/envs/C-A-T/lib/python2.7/threading.py", line 754, in run self.target(*self.args, **self.__kwargs) File "/home/liran/Software/miniconda2/envs/C-A-T/lib/python2.7/site-packages/toil/fileStore.py", line 1461, in asyncWrite raise RuntimeError("The termination flag is set, exiting") RuntimeError: The termination flag is set, exiting

Exception RuntimeError: RuntimeError('cannot join current thread',) in <bound method CachingFileStore.del of <toil.fileStore.CachingFileStore object at 0x7f779b04f650>> ignored Exception in thread Thread-1: Traceback (most recent call last): File "/home/liran/Software/miniconda2/envs/C-A-T/lib/python2.7/threading.py", line 801, in __bootstrap_inner self.run() File "/home/liran/Software/miniconda2/envs/C-A-T/lib/python2.7/threading.py", line 754, in run self.target(*self.args, **self.__kwargs) File "/home/liran/Software/miniconda2/envs/C-A-T/lib/python2.7/site-packages/toil/fileStore.py", line 1461, in asyncWrite raise RuntimeError("The termination flag is set, exiting") RuntimeError: The termination flag is set, exiting

Exception in thread Thread-2: Traceback (most recent call last): File "/home/liran/Software/miniconda2/envs/C-A-T/lib/python2.7/threading.py", line 801, in __bootstrap_inner self.run() File "/home/liran/Software/miniconda2/envs/C-A-T/lib/python2.7/threading.py", line 754, in run self.target(*self.args, **self.__kwargs) File "/home/liran/Software/miniconda2/envs/C-A-T/lib/python2.7/site-packages/toil/fileStore.py", line 1461, in asyncWrite raise RuntimeError("The termination flag is set, exiting") RuntimeError: The termination flag is set, exiting

Exception RuntimeError: RuntimeError('cannot join current thread',) in <bound method CachingFileStore.del of <toil.fileStore.CachingFileStore object at 0x7f4fd5ef2650>> ignored 03-30 10:33:54 toil.leader WARNING The job seems to have left a log file, indicating failure: 'namesort_bam' I/E/jobpE4Exm ---TOIL WORKER OUTPUT LOG--- I/E/jobpE4Exm INFO:toil:Running Toil version 3.5.0-7f5bc1cab176d99d6996076ae5b8bc2453fa04a4. I/E/jobpE4Exm WARNING:toil.resource:Can't find resource for leader path '/home/liran/Software/C-A-T/Comparative-Annotation-Toolkit/cat' I/E/jobpE4Exm WARNING:toil.resource:Can't localize module ModuleDescriptor(dirPath='/home/liran/Software/C-A-T/Comparative-Annotation-Toolkit', name='cat.hints_db', fromVirtualEnv=False) I/E/jobpE4Exm INFO:luigi-interface:Loaded ['luigi.cfg'] I/E/jobpE4Exm INFO:toil.fileStore:Starting job ('namesort_bam' I/E/jobpE4Exm) with ID (4c007acbed9b807693aea63673f27db95e3d1d50). I/E/jobpE4Exm WARNING:toil.resource:Can't find resource for leader path '/home/liran/Software/C-A-T/Comparative-Annotation-Toolkit/cat' I/E/jobpE4Exm WARNING:toil.resource:Can't localize module ModuleDescriptor(dirPath='/home/liran/Software/C-A-T/Comparative-Annotation-Toolkit', name='cat.hints_db', fromVirtualEnv=False) I/E/jobpE4Exm samtools: error while loading shared libraries: libcrypto.so.1.0.0: cannot open shared object file: No such file or directory I/E/jobpE4Exm sambamba-sort: not enough data in stream I/E/jobpE4Exm Traceback (most recent call last): I/E/jobpE4Exm File "/home/liran/Software/miniconda2/envs/C-A-T/lib/python2.7/site-packages/toil/worker.py", line 340, in main I/E/jobpE4Exm job._runner(jobGraph=jobGraph, jobStore=jobStore, fileStore=fileStore) I/E/jobpE4Exm File "/home/liran/Software/miniconda2/envs/C-A-T/lib/python2.7/site-packages/toil/job.py", line 1270, in _runner I/E/jobpE4Exm returnValues = self._run(jobGraph, fileStore) I/E/jobpE4Exm File "/home/liran/Software/miniconda2/envs/C-A-T/lib/python2.7/site-packages/toil/job.py", line 1217, in _run I/E/jobpE4Exm return self.run(fileStore) I/E/jobpE4Exm File "/home/liran/Software/miniconda2/envs/C-A-T/lib/python2.7/site-packages/toil/job.py", line 1383, in run I/E/jobpE4Exm rValue = userFunction(*((self,) + tuple(self._args)), self._kwargs) I/E/jobpE4Exm File "/home/liran/Software/C-A-T/Comparative-Annotation-Toolkit/cat/hints_db.py", line 161, in namesort_bam I/E/jobpE4Exm tools.procOps.run_proc(cmd, stdout=name_sorted) I/E/jobpE4Exm File "/home/liran/Software/C-A-T/Comparative-Annotation-Toolkit/tools/procOps.py", line 68, in run_proc I/E/jobpE4Exm pl.wait() I/E/jobpE4Exm File "/home/liran/Software/C-A-T/Comparative-Annotation-Toolkit/tools/pipeline.py", line 1127, in wait I/E/jobpE4Exm self.raiseIfExcept() I/E/jobpE4Exm File "/home/liran/Software/C-A-T/Comparative-Annotation-Toolkit/tools/pipeline.py", line 1085, in raiseIfExcept I/E/jobpE4Exm p.raiseIfExcept() I/E/jobpE4Exm File "/home/liran/Software/C-A-T/Comparative-Annotation-Toolkit/tools/pipeline.py", line 749, in raiseIfExcept I/E/jobpE4Exm raise self.exceptInfo[0], self.exceptInfo[1], self.exceptInfo[2] I/E/jobpE4Exm ProcException: process exited 127: samtools view -b /tmp/toil-3140b0cd-fef4-45e2-9576-8ad3d48ca251/tmpJRrVMj/2e8bffff-c008-4483-aa7c-c8092d9ee649/tmptxrYF3.tmp chr16 I/E/jobpE4Exm ERROR:toil.worker:Exiting the worker because of a failed job on host subnet190-2.nwsuaf.edu.cn I/E/jobpE4Exm WARNING:toil.jobGraph:Due to failure we are reducing the remaining retry count of job 'namesort_bam' I/E/jobpE4Exm with ID I/E/jobpE4Exm to 0 03-30 10:33:54 toil.leader WARNING Job 'namesort_bam' I/E/jobpE4Exm with ID I/E/jobpE4Exm is completely failed 03-30 10:33:54 toil.leader WARNING The job seems to have left a log file, indicating failure: 'namesort_bam' 8/V/jobc4vH8x ---TOIL WORKER OUTPUT LOG--- 8/V/jobc4vH8x INFO:toil:Running Toil version 3.5.0-7f5bc1cab176d99d6996076ae5b8bc2453fa04a4. 8/V/jobc4vH8x WARNING:toil.resource:Can't find resource for leader path '/home/liran/Software/C-A-T/Comparative-Annotation-Toolkit/cat' 8/V/jobc4vH8x WARNING:toil.resource:Can't localize module ModuleDescriptor(dirPath='/home/liran/Software/C-A-T/Comparative-Annotation-Toolkit', name='cat.hints_db', fromVirtualEnv=False) 8/V/jobc4vH8x INFO:luigi-interface:Loaded ['luigi.cfg'] 8/V/jobc4vH8x INFO:toil.fileStore:Starting job ('namesort_bam' 8/V/jobc4vH8x) with ID (87279c8647989aceb657e93f65389bbd4c04219c). 8/V/jobc4vH8x WARNING:toil.resource:Can't find resource for leader path '/home/liran/Software/C-A-T/Comparative-Annotation-Toolkit/cat' 8/V/jobc4vH8x WARNING:toil.resource:Can't localize module ModuleDescriptor(dirPath='/home/liran/Software/C-A-T/Comparative-Annotation-Toolkit', name='cat.hints_db', fromVirtualEnv=False) 8/V/jobc4vH8x samtools: error while loading shared libraries: libcrypto.so.1.0.0: cannot open shared object file: No such file or directory 8/V/jobc4vH8x sambamba-sort: not enough data in stream 8/V/jobc4vH8x Traceback (most recent call last): 8/V/jobc4vH8x File "/home/liran/Software/miniconda2/envs/C-A-T/lib/python2.7/site-packages/toil/worker.py", line 340, in main 8/V/jobc4vH8x job._runner(jobGraph=jobGraph, jobStore=jobStore, fileStore=fileStore) 8/V/jobc4vH8x File "/home/liran/Software/miniconda2/envs/C-A-T/lib/python2.7/site-packages/toil/job.py", line 1270, in _runner 8/V/jobc4vH8x returnValues = self._run(jobGraph, fileStore) 8/V/jobc4vH8x File "/home/liran/Software/miniconda2/envs/C-A-T/lib/python2.7/site-packages/toil/job.py", line 1217, in _run 8/V/jobc4vH8x return self.run(fileStore) 8/V/jobc4vH8x File "/home/liran/Software/miniconda2/envs/C-A-T/lib/python2.7/site-packages/toil/job.py", line 1383, in run 8/V/jobc4vH8x rValue = userFunction(*((self,) + tuple(self._args)), *self._kwargs) 8/V/jobc4vH8x File "/home/liran/Software/C-A-T/Comparative-Annotation-Toolkit/cat/hints_db.py", line 161, in namesort_bam 8/V/jobc4vH8x tools.procOps.run_proc(cmd, stdout=name_sorted) 8/V/jobc4vH8x File "/home/liran/Software/C-A-T/Comparative-Annotation-Toolkit/tools/procOps.py", line 68, in run_proc 8/V/jobc4vH8x pl.wait() 8/V/jobc4vH8x File "/home/liran/Software/C-A-T/Comparative-Annotation-Toolkit/tools/pipeline.py", line 1127, in wait 8/V/jobc4vH8x self.raiseIfExcept() 8/V/jobc4vH8x File "/home/liran/Software/C-A-T/Comparative-Annotation-Toolkit/tools/pipeline.py", line 1085, in raiseIfExcept 8/V/jobc4vH8x p.raiseIfExcept() 8/V/jobc4vH8x File "/home/liran/Software/C-A-T/Comparative-Annotation-Toolkit/tools/pipeline.py", line 749, in raiseIfExcept 8/V/jobc4vH8x raise self.exceptInfo[0], self.exceptInfo[1], self.exceptInfo[2] 8/V/jobc4vH8x ProcException: process exited 127: samtools view -b /tmp/toil-3fcac38b-8337-485e-a366-851a45df2826/tmpuIJTun/0ef8bc56-4882-431c-9ad2-3da22248ee5d/tmpXltoEJ.tmp chr14 8/V/jobc4vH8x ERROR:toil.worker:Exiting the worker because of a failed job on host subnet190-2.nwsuaf.edu.cn 8/V/jobc4vH8x WARNING:toil.jobGraph:Due to failure we are reducing the remaining retry count of job 'namesort_bam' 8/V/jobc4vH8x with ID 8/V/jobc4vH8x to 0 03-30 10:33:54 toil.leader WARNING Job 'namesort_bam' 8/V/jobc4vH8x with ID 8/V/jobc4vH8x is completely failed 03-30 10:33:54 toil.leader WARNING The job seems to have left a log file, indicating failure: 'namesort_bam' S/f/jobWsVBMf ---TOIL WORKER OUTPUT LOG--- S/f/jobWsVBMf INFO:toil:Running Toil version 3.5.0-7f5bc1cab176d99d6996076ae5b8bc2453fa04a4. S/f/jobWsVBMf WARNING:toil.resource:Can't find resource for leader path '/home/liran/Software/C-A-T/Comparative-Annotation-Toolkit/cat' S/f/jobWsVBMf WARNING:toil.resource:Can't localize module ModuleDescriptor(dirPath='/home/liran/Software/C-A-T/Comparative-Annotation-Toolkit', name='cat.hints_db', fromVirtualEnv=False) S/f/jobWsVBMf INFO:luigi-interface:Loaded ['luigi.cfg'] S/f/jobWsVBMf INFO:toil.fileStore:Starting job ('namesort_bam' S/f/jobWsVBMf) with ID (4dce1ee2b02d7cfbce2741de810659fa32dfcea7). S/f/jobWsVBMf WARNING:toil.resource:Can't find resource for leader path '/home/liran/Software/C-A-T/Comparative-Annotation-Toolkit/cat' S/f/jobWsVBMf WARNING:toil.resource:Can't localize module ModuleDescriptor(dirPath='/home/liran/Software/C-A-T/Comparative-Annotation-Toolkit', name='cat.hints_db', fromVirtualEnv=False) S/f/jobWsVBMf samtools: error while loading shared libraries: libcrypto.so.1.0.0: cannot open shared object file: No such file or directory S/f/jobWsVBMf sambamba-sort: not enough data in stream S/f/jobWsVBMf Traceback (most recent call last): S/f/jobWsVBMf File "/home/liran/Software/miniconda2/envs/C-A-T/lib/python2.7/site-packages/toil/worker.py", line 340, in main S/f/jobWsVBMf job._runner(jobGraph=jobGraph, jobStore=jobStore, fileStore=fileStore) S/f/jobWsVBMf File "/home/liran/Software/miniconda2/envs/C-A-T/lib/python2.7/site-packages/toil/job.py", line 1270, in _runner S/f/jobWsVBMf returnValues = self._run(jobGraph, fileStore) S/f/jobWsVBMf File "/home/liran/Software/miniconda2/envs/C-A-T/lib/python2.7/site-packages/toil/job.py", line 1217, in _run S/f/jobWsVBMf return self.run(fileStore) S/f/jobWsVBMf File "/home/liran/Software/miniconda2/envs/C-A-T/lib/python2.7/site-packages/toil/job.py", line 1383, in run S/f/jobWsVBMf rValue = userFunction(((self,) + tuple(self._args)), self._kwargs) S/f/jobWsVBMf File "/home/liran/Software/C-A-T/Comparative-Annotation-Toolkit/cat/hints_db.py", line 161, in namesort_bam S/f/jobWsVBMf tools.procOps.run_proc(cmd, stdout=name_sorted) S/f/jobWsVBMf File "/home/liran/Software/C-A-T/Comparative-Annotation-Toolkit/tools/procOps.py", line 68, in run_proc S/f/jobWsVBMf pl.wait() S/f/jobWsVBMf File "/home/liran/Software/C-A-T/Comparative-Annotation-Toolkit/tools/pipeline.py", line 1127, in wait S/f/jobWsVBMf self.raiseIfExcept() S/f/jobWsVBMf File "/home/liran/Software/C-A-T/Comparative-Annotation-Toolkit/tools/pipeline.py", line 1085, in raiseIfExcept S/f/jobWsVBMf p.raiseIfExcept() S/f/jobWsVBMf File "/home/liran/Software/C-A-T/Comparative-Annotation-Toolkit/tools/pipeline.py", line 749, in raiseIfExcept S/f/jobWsVBMf raise self.exceptInfo[0], self.exceptInfo[1], self.exceptInfo[2] S/f/jobWsVBMf ProcException: process exited 127: samtools view -b /tmp/toil-3d7f7e23-af68-431b-9a1b-27ac0c6bd62d/tmpsmFNMz/375989f7-b60c-402d-9ac9-1e71ab9ff27d/tmpFmm0Bq.tmp chr20 S/f/jobWsVBMf ERROR:toil.worker:Exiting the worker because of a failed job on host subnet190-2.nwsuaf.edu.cn S/f/jobWsVBMf WARNING:toil.jobGraph:Due to failure we are reducing the remaining retry count of job 'namesort_bam' S/f/jobWsVBMf with ID S/f/jobWsVBMf to 0 03-30 10:33:54 toil.leader WARNING Job 'namesort_bam' S/f/jobWsVBMf with ID S/f/jobWsVBMf is completely failed 03-30 10:33:55 toil.leader WARNING The job seems to have left a log file, indicating failure: 'generate_iso_seq_hints' t/Z/jobbxMOiF ---TOIL WORKER OUTPUT LOG--- t/Z/jobbxMOiF INFO:toil:Running Toil version 3.5.0-7f5bc1cab176d99d6996076ae5b8bc2453fa04a4. t/Z/jobbxMOiF WARNING:toil.resource:Can't find resource for leader path '/home/liran/Software/C-A-T/Comparative-Annotation-Toolkit/cat' t/Z/jobbxMOiF WARNING:toil.resource:Can't localize module ModuleDescriptor(dirPath='/home/liran/Software/C-A-T/Comparative-Annotation-Toolkit', name='cat.hints_db', fromVirtualEnv=False) t/Z/jobbxMOiF INFO:luigi-interface:Loaded ['luigi.cfg'] t/Z/jobbxMOiF INFO:toil.fileStore:Starting job ('generate_iso_seq_hints' t/Z/jobbxMOiF) with ID (8fabd729abe37ab3990e0ba45052b4b16daddbdc). t/Z/jobbxMOiF WARNING:toil.resource:Can't find resource for leader path '/home/liran/Software/C-A-T/Comparative-Annotation-Toolkit/cat' t/Z/jobbxMOiF WARNING:toil.resource:Can't localize module ModuleDescriptor(dirPath='/home/liran/Software/C-A-T/Comparative-Annotation-Toolkit', name='cat.hints_db', fromVirtualEnv=False) t/Z/jobbxMOiF samtools: error while loading shared libraries: libcrypto.so.1.0.0: cannot open shared object file: No such file or directory t/Z/jobbxMOiF t/Z/jobbxMOiF Traceback (most recent call last): t/Z/jobbxMOiF File "/home/liran/Software/miniconda2/envs/C-A-T/lib/python2.7/site-packages/toil/worker.py", line 340, in main t/Z/jobbxMOiF job._runner(jobGraph=jobGraph, jobStore=jobStore, fileStore=fileStore) t/Z/jobbxMOiF File "/home/liran/Software/miniconda2/envs/C-A-T/lib/python2.7/site-packages/toil/job.py", line 1270, in _runner t/Z/jobbxMOiF returnValues = self._run(jobGraph, fileStore) t/Z/jobbxMOiF File "/home/liran/Software/miniconda2/envs/C-A-T/lib/python2.7/site-packages/toil/job.py", line 1217, in _run t/Z/jobbxMOiF return self.run(fileStore) t/Z/jobbxMOiF File "/home/liran/Software/miniconda2/envs/C-A-T/lib/python2.7/site-packages/toil/job.py", line 1383, in run t/Z/jobbxMOiF rValue = userFunction(*((self,) + tuple(self._args)), self._kwargs) t/Z/jobbxMOiF File "/home/liran/Software/C-A-T/Comparative-Annotation-Toolkit/cat/hints_db.py", line 391, in generate_iso_seq_hints t/Z/jobbxMOiF tools.procOps.run_proc(cmd) t/Z/jobbxMOiF File "/home/liran/Software/C-A-T/Comparative-Annotation-Toolkit/tools/procOps.py", line 68, in run_proc t/Z/jobbxMOiF pl.wait() t/Z/jobbxMOiF File "/home/liran/Software/C-A-T/Comparative-Annotation-Toolkit/tools/pipeline.py", line 1127, in wait t/Z/jobbxMOiF self.raiseIfExcept() t/Z/jobbxMOiF File "/home/liran/Software/C-A-T/Comparative-Annotation-Toolkit/tools/pipeline.py", line 1085, in raiseIfExcept t/Z/jobbxMOiF p.raiseIfExcept() t/Z/jobbxMOiF File "/home/liran/Software/C-A-T/Comparative-Annotation-Toolkit/tools/pipeline.py", line 749, in raiseIfExcept t/Z/jobbxMOiF raise self.exceptInfo[0], self.exceptInfo[1], self.exceptInfo[2] t/Z/jobbxMOiF ProcException: process exited 127: samtools view -b -F 4 /tmp/toil-3140b0cd-fef4-45e2-9576-8ad3d48ca251/tmpAvOHNn/6d114774-f138-4425-aac9-19a919ed47a4/tmpKn5D35.tmp t/Z/jobbxMOiF ERROR:toil.worker:Exiting the worker because of a failed job on host subnet190-2.nwsuaf.edu.cn t/Z/jobbxMOiF WARNING:toil.jobGraph:Due to failure we are reducing the remaining retry count of job 'generate_iso_seq_hints' t/Z/jobbxMOiF with ID t/Z/jobbxMOiF to 0 03-30 10:33:55 toil.leader WARNING Job 'generate_iso_seq_hints' t/Z/jobbxMOiF with ID t/Z/jobbxMOiF is completely failed 03-30 10:33:55 toil.leader WARNING The job seems to have left a log file, indicating failure: 'namesort_bam' r/j/jobHUG8TV ---TOIL WORKER OUTPUT LOG--- r/j/jobHUG8TV INFO:toil:Running Toil version 3.5.0-7f5bc1cab176d99d6996076ae5b8bc2453fa04a4. r/j/jobHUG8TV WARNING:toil.resource:Can't find resource for leader path '/home/liran/Software/C-A-T/Comparative-Annotation-Toolkit/cat' r/j/jobHUG8TV WARNING:toil.resource:Can't localize module ModuleDescriptor(dirPath='/home/liran/Software/C-A-T/Comparative-Annotation-Toolkit', name='cat.hints_db', fromVirtualEnv=False) r/j/jobHUG8TV INFO:luigi-interface:Loaded ['luigi.cfg'] r/j/jobHUG8TV INFO:toil.fileStore:Starting job ('namesort_bam' r/j/jobHUG8TV) with ID (b776329fa019fddcf731411725fdcd084fc41346). r/j/jobHUG8TV WARNING:toil.resource:Can't find resource for leader path '/home/liran/Software/C-A-T/Comparative-Annotation-Toolkit/cat' r/j/jobHUG8TV WARNING:toil.resource:Can't localize module ModuleDescriptor(dirPath='/home/liran/Software/C-A-T/Comparative-Annotation-Toolkit', name='cat.hints_db', fromVirtualEnv=False) r/j/jobHUG8TV samtools: error while loading shared libraries: libcrypto.so.1.0.0: cannot open shared object file: No such file or directory r/j/jobHUG8TV sambamba-sort: not enough data in stream r/j/jobHUG8TV Traceback (most recent call last): r/j/jobHUG8TV File "/home/liran/Software/miniconda2/envs/C-A-T/lib/python2.7/site-packages/toil/worker.py", line 340, in main r/j/jobHUG8TV job._runner(jobGraph=jobGraph, jobStore=jobStore, fileStore=fileStore) r/j/jobHUG8TV File "/home/liran/Software/miniconda2/envs/C-A-T/lib/python2.7/site-packages/toil/job.py", line 1270, in _runner r/j/jobHUG8TV returnValues = self._run(jobGraph, fileStore) r/j/jobHUG8TV File "/home/liran/Software/miniconda2/envs/C-A-T/lib/python2.7/site-packages/toil/job.py", line 1217, in _run r/j/jobHUG8TV return self.run(fileStore) r/j/jobHUG8TV File "/home/liran/Software/miniconda2/envs/C-A-T/lib/python2.7/site-packages/toil/job.py", line 1383, in run r/j/jobHUG8TV rValue = userFunction(*((self,) + tuple(self._args)), *self._kwargs) r/j/jobHUG8TV File "/home/liran/Software/C-A-T/Comparative-Annotation-Toolkit/cat/hints_db.py", line 161, in namesort_bam r/j/jobHUG8TV tools.procOps.run_proc(cmd, stdout=name_sorted) r/j/jobHUG8TV File "/home/liran/Software/C-A-T/Comparative-Annotation-Toolkit/tools/procOps.py", line 68, in run_proc r/j/jobHUG8TV pl.wait() r/j/jobHUG8TV File "/home/liran/Software/C-A-T/Comparative-Annotation-Toolkit/tools/pipeline.py", line 1127, in wait r/j/jobHUG8TV self.raiseIfExcept() r/j/jobHUG8TV File "/home/liran/Software/C-A-T/Comparative-Annotation-Toolkit/tools/pipeline.py", line 1085, in raiseIfExcept r/j/jobHUG8TV p.raiseIfExcept() r/j/jobHUG8TV File "/home/liran/Software/C-A-T/Comparative-Annotation-Toolkit/tools/pipeline.py", line 749, in raiseIfExcept r/j/jobHUG8TV raise self.exceptInfo[0], self.exceptInfo[1], self.exceptInfo[2] r/j/jobHUG8TV ProcException: process exited 127: samtools view -b /tmp/toil-3a71830d-df1e-460d-b3df-50f0633d456e/tmpakvgey/04d45eaf-4719-488b-989e-a9424a58a4ed/tmpdYlxSD.tmp chr17 r/j/jobHUG8TV ERROR:toil.worker:Exiting the worker because of a failed job on host subnet190-2.nwsuaf.edu.cn r/j/jobHUG8TV WARNING:toil.jobGraph:Due to failure we are reducing the remaining retry count of job 'namesort_bam' r/j/jobHUG8TV with ID r/j/jobHUG8TV to 0 03-30 10:33:55 toil.leader WARNING Job 'namesort_bam' r/j/jobHUG8TV with ID r/j/jobHUG8TV is completely failed 03-30 10:33:55 toil.leader WARNING The job seems to have left a log file, indicating failure: 'namesort_bam' X/R/jobbpn764 ---TOIL WORKER OUTPUT LOG--- X/R/jobbpn764 INFO:toil:Running Toil version 3.5.0-7f5bc1cab176d99d6996076ae5b8bc2453fa04a4. X/R/jobbpn764 WARNING:toil.resource:Can't find resource for leader path '/home/liran/Software/C-A-T/Comparative-Annotation-Toolkit/cat' X/R/jobbpn764 WARNING:toil.resource:Can't localize module ModuleDescriptor(dirPath='/home/liran/Software/C-A-T/Comparative-Annotation-Toolkit', name='cat.hints_db', fromVirtualEnv=False) X/R/jobbpn764 INFO:luigi-interface:Loaded ['luigi.cfg'] X/R/jobbpn764 INFO:toil.fileStore:Starting job ('namesort_bam' X/R/jobbpn764) with ID (cca09013a4e0d74a60cb5fd59d573d52bae1f6ca). X/R/jobbpn764 WARNING:toil.resource:Can't find resource for leader path '/home/liran/Software/C-A-T/Comparative-Annotation-Toolkit/cat' X/R/jobbpn764 WARNING:toil.resource:Can't localize module ModuleDescriptor(dirPath='/home/liran/Software/C-A-T/Comparative-Annotation-Toolkit', name='cat.hints_db', fromVirtualEnv=False) X/R/jobbpn764 samtools: error while loading shared libraries: libcrypto.so.1.0.0: cannot open shared object file: No such file or directory X/R/jobbpn764 sambamba-sort: not enough data in stream X/R/jobbpn764 Traceback (most recent call last): X/R/jobbpn764 File "/home/liran/Software/miniconda2/envs/C-A-T/lib/python2.7/site-packages/toil/worker.py", line 340, in main X/R/jobbpn764 job._runner(jobGraph=jobGraph, jobStore=jobStore, fileStore=fileStore) X/R/jobbpn764 File "/home/liran/Software/miniconda2/envs/C-A-T/lib/python2.7/site-packages/toil/job.py", line 1270, in _runner X/R/jobbpn764 returnValues = self._run(jobGraph, fileStore) X/R/jobbpn764 File "/home/liran/Software/miniconda2/envs/C-A-T/lib/python2.7/site-packages/toil/job.py", line 1217, in _run X/R/jobbpn764 return self.run(fileStore) X/R/jobbpn764 File "/home/liran/Software/miniconda2/envs/C-A-T/lib/python2.7/site-packages/toil/job.py", line 1383, in run X/R/jobbpn764 rValue = userFunction(((self,) + tuple(self._args)), self._kwargs) X/R/jobbpn764 File "/home/liran/Software/C-A-T/Comparative-Annotation-Toolkit/cat/hints_db.py", line 161, in namesort_bam X/R/jobbpn764 tools.procOps.run_proc(cmd, stdout=name_sorted) X/R/jobbpn764 File "/home/liran/Software/C-A-T/Comparative-Annotation-Toolkit/tools/procOps.py", line 68, in run_proc X/R/jobbpn764 pl.wait() X/R/jobbpn764 File "/home/liran/Software/C-A-T/Comparative-Annotation-Toolkit/tools/pipeline.py", line 1127, in wait X/R/jobbpn764 self.raiseIfExcept() X/R/jobbpn764 File "/home/liran/Software/C-A-T/Comparative-Annotation-Toolkit/tools/pipeline.py", line 1085, in raiseIfExcept X/R/jobbpn764 p.raiseIfExcept() X/R/jobbpn764 File "/home/liran/Software/C-A-T/Comparative-Annotation-Toolkit/tools/pipeline.py", line 749, in raiseIfExcept X/R/jobbpn764 raise self.exceptInfo[0], self.exceptInfo[1], self.exceptInfo[2] X/R/jobbpn764 ProcException: process exited 127: samtools view -b /tmp/toil-3a71830d-df1e-460d-b3df-50f0633d456e/tmpKHYgxy/ae5fdc4d-6925-4bc7-ba9e-219e6fd42641/tmpT43J37.tmp chr17 X/R/jobbpn764 ERROR:toil.worker:Exiting the worker because of a failed job on host subnet190-2.nwsuaf.edu.cn X/R/jobbpn764 WARNING:toil.jobGraph:Due to failure we are reducing the remaining retry count of job 'namesort_bam' X/R/jobbpn764 with ID X/R/jobbpn764 to 0 03-30 10:33:55 toil.leader WARNING Job 'namesort_bam' X/R/jobbpn764 with ID X/R/jobbpn764 is completely failed Exception in thread Thread-1: Traceback (most recent call last): File "/home/liran/Software/miniconda2/envs/C-A-T/lib/python2.7/threading.py", line 801, in bootstrap_inner self.run() File "/home/liran/Software/miniconda2/envs/C-A-T/lib/python2.7/threading.py", line 754, in run self.target(*self.args, **self.kwargs) File "/home/liran/Software/miniconda2/envs/C-A-T/lib/python2.7/site-packages/toil/fileStore.py", line 1461, in asyncWrite raise RuntimeError("The termination flag is set, exiting") RuntimeError: The termination flag is set, exiting Exception in thread Thread-2: Traceback (most recent call last): File "/home/liran/Software/miniconda2/envs/C-A-T/lib/python2.7/threading.py", line 801, in bootstrap_inner self.run() File "/home/liran/Software/miniconda2/envs/C-A-T/lib/python2.7/threading.py", line 754, in run self.target(*self.args, **self.kwargs) File "/home/liran/Software/miniconda2/envs/C-A-T/lib/python2.7/site-packages/toil/fileStore.py", line 1461, in asyncWrite raise RuntimeError("The termination flag is set, exiting") RuntimeError: The termination flag is set, exiting

Exception RuntimeError: RuntimeError('cannot join current thread',) in <bound method CachingFileStore.del of <toil.fileStore.CachingFileStore object at 0x7f53c76a2650>> ignored 03-30 10:33:58 toil.leader WARNING The job seems to have left a log file, indicating failure: 'namesort_bam' V/4/job6X8beL ---TOIL WORKER OUTPUT LOG--- V/4/job6X8beL INFO:toil:Running Toil version 3.5.0-7f5bc1cab176d99d6996076ae5b8bc2453fa04a4. V/4/job6X8beL WARNING:toil.resource:Can't find resource for leader path '/home/liran/Software/C-A-T/Comparative-Annotation-Toolkit/cat' V/4/job6X8beL WARNING:toil.resource:Can't localize module ModuleDescriptor(dirPath='/home/liran/Software/C-A-T/Comparative-Annotation-Toolkit', name='cat.hints_db', fromVirtualEnv=False) V/4/job6X8beL INFO:luigi-interface:Loaded ['luigi.cfg'] V/4/job6X8beL INFO:toil.fileStore:Starting job ('namesort_bam' V/4/job6X8beL) with ID (58da24787b046b0440994ecffb59a24b137a50d2). V/4/job6X8beL WARNING:toil.resource:Can't find resource for leader path '/home/liran/Software/C-A-T/Comparative-Annotation-Toolkit/cat' V/4/job6X8beL WARNING:toil.resource:Can't localize module ModuleDescriptor(dirPath='/home/liran/Software/C-A-T/Comparative-Annotation-Toolkit', name='cat.hints_db', fromVirtualEnv=False) V/4/job6X8beL samtools: error while loading shared libraries: libcrypto.so.1.0.0: cannot open shared object file: No such file or directory V/4/job6X8beL sambamba-sort: not enough data in stream V/4/job6X8beL Traceback (most recent call last): V/4/job6X8beL File "/home/liran/Software/miniconda2/envs/C-A-T/lib/python2.7/site-packages/toil/worker.py", line 340, in main V/4/job6X8beL job._runner(jobGraph=jobGraph, jobStore=jobStore, fileStore=fileStore) V/4/job6X8beL File "/home/liran/Software/miniconda2/envs/C-A-T/lib/python2.7/site-packages/toil/job.py", line 1270, in _runner V/4/job6X8beL returnValues = self._run(jobGraph, fileStore) V/4/job6X8beL File "/home/liran/Software/miniconda2/envs/C-A-T/lib/python2.7/site-packages/toil/job.py", line 1217, in _run V/4/job6X8beL return self.run(fileStore) V/4/job6X8beL File "/home/liran/Software/miniconda2/envs/C-A-T/lib/python2.7/site-packages/toil/job.py", line 1383, in run V/4/job6X8beL rValue = userFunction(*((self,) + tuple(self._args)), *self._kwargs) V/4/job6X8beL File "/home/liran/Software/C-A-T/Comparative-Annotation-Toolkit/cat/hints_db.py", line 161, in namesort_bam V/4/job6X8beL tools.procOps.run_proc(cmd, stdout=name_sorted) V/4/job6X8beL File "/home/liran/Software/C-A-T/Comparative-Annotation-Toolkit/tools/procOps.py", line 68, in run_proc V/4/job6X8beL pl.wait() V/4/job6X8beL File "/home/liran/Software/C-A-T/Comparative-Annotation-Toolkit/tools/pipeline.py", line 1127, in wait V/4/job6X8beL self.raiseIfExcept() V/4/job6X8beL File "/home/liran/Software/C-A-T/Comparative-Annotation-Toolkit/tools/pipeline.py", line 1085, in raiseIfExcept V/4/job6X8beL p.raiseIfExcept() V/4/job6X8beL File "/home/liran/Software/C-A-T/Comparative-Annotation-Toolkit/tools/pipeline.py", line 749, in raiseIfExcept V/4/job6X8beL raise self.exceptInfo[0], self.exceptInfo[1], self.exceptInfo[2] V/4/job6X8beL ProcException: process exited 127: samtools view -b /tmp/toil-3a71830d-df1e-460d-b3df-50f0633d456e/tmpSZI7pC/778fec04-5972-4eb0-8ba0-6b0842da5065/tmpxeYvLw.tmp chr17 V/4/job6X8beL ERROR:toil.worker:Exiting the worker because of a failed job on host subnet190-2.nwsuaf.edu.cn V/4/job6X8beL WARNING:toil.jobGraph:Due to failure we are reducing the remaining retry count of job 'namesort_bam' V/4/job6X8beL with ID V/4/job6X8beL to 0 03-30 10:33:58 toil.leader WARNING Job 'namesort_bam' V/4/job6X8beL with ID V/4/job6X8beL is completely failed 03-30 10:33:59 luigi-interface ERROR [pid 35764] Worker Worker(salt=724431596, workers=10, host=subnet190-2.nwsuaf.edu.cn, username=liran, pid=35207) failed ToilTask: GenerateHints for rheMac3 using batchSystem singleMachine Traceback (most recent call last): File "/home/liran/Software/miniconda2/envs/C-A-T/lib/python2.7/site-packages/luigi/worker.py", line 199, in run new_deps = self._run_get_new_deps() File "/home/liran/Software/miniconda2/envs/C-A-T/lib/python2.7/site-packages/luigi/worker.py", line 141, in _run_get_new_deps task_gen = self.task.run() File "/home/liran/Software/C-A-T/Comparative-Annotation-Toolkit/cat/init.py", line 1053, in run hints_db(self.hints_args, toil_options) File "/home/liran/Software/C-A-T/Comparative-Annotation-Toolkit/cat/hints_db.py", line 84, in hints_db combined_hints = t.restart() File "/home/liran/Software/miniconda2/envs/C-A-T/lib/python2.7/site-packages/toil/common.py", line 700, in restart return self._runMainLoop(rootJobGraph) File "/home/liran/Software/miniconda2/envs/C-A-T/lib/python2.7/site-packages/toil/common.py", line 984, in _runMainLoop jobCache=self._jobCache).run() File "/home/liran/Software/miniconda2/envs/C-A-T/lib/python2.7/site-packages/toil/leader.py", line 200, in run raise FailedJobsException(self.config.jobStore, self.toilState.totalFailedJobs, self.jobStore) FailedJobsException: The job store 'file:/home/liran/Software/C-A-T/Comparative-Annotation-Toolkit/test_install/toil/hints_db/rheMac3/jobStore' contains 2 failed jobs: 'setup_hints' m/h/jobJLngJF, 'namesort_bam' S/f/jobWsVBMf =========> Failed job 'namesort_bam' S/f/jobWsVBMf ---TOIL WORKER OUTPUT LOG--- INFO:toil:Running Toil version 3.5.0-7f5bc1cab176d99d6996076ae5b8bc2453fa04a4. WARNING:toil.resource:Can't find resource for leader path '/home/liran/Software/C-A-T/Comparative-Annotation-Toolkit/cat' WARNING:toil.resource:Can't localize module ModuleDescriptor(dirPath='/home/liran/Software/C-A-T/Comparative-Annotation-Toolkit', name='cat.hints_db', fromVirtualEnv=False) INFO:luigi-interface:Loaded ['luigi.cfg'] INFO:toil.fileStore:Starting job ('namesort_bam' S/f/jobWsVBMf) with ID (4dce1ee2b02d7cfbce2741de810659fa32dfcea7). WARNING:toil.resource:Can't find resource for leader path '/home/liran/Software/C-A-T/Comparative-Annotation-Toolkit/cat' WARNING:toil.resource:Can't localize module ModuleDescriptor(dirPath='/home/liran/Software/C-A-T/Comparative-Annotation-Toolkit', name='cat.hints_db', fromVirtualEnv=False) samtools: error while loading shared libraries: libcrypto.so.1.0.0: cannot open shared object file: No such file or directory sambamba-sort: not enough data in stream Traceback (most recent call last): File "/home/liran/Software/miniconda2/envs/C-A-T/lib/python2.7/site-packages/toil/worker.py", line 340, in main job._runner(jobGraph=jobGraph, jobStore=jobStore, fileStore=fileStore) File "/home/liran/Software/miniconda2/envs/C-A-T/lib/python2.7/site-packages/toil/job.py", line 1270, in _runner returnValues = self._run(jobGraph, fileStore) File "/home/liran/Software/miniconda2/envs/C-A-T/lib/python2.7/site-packages/toil/job.py", line 1217, in _run return self.run(fileStore) File "/home/liran/Software/miniconda2/envs/C-A-T/lib/python2.7/site-packages/toil/job.py", line 1383, in run rValue = userFunction(((self,) + tuple(self._args)), **self._kwargs) File "/home/liran/Software/C-A-T/Comparative-Annotation-Toolkit/cat/hints_db.py", line 161, in namesort_bam tools.procOps.run_proc(cmd, stdout=name_sorted) File "/home/liran/Software/C-A-T/Comparative-Annotation-Toolkit/tools/procOps.py", line 68, in run_proc pl.wait() File "/home/liran/Software/C-A-T/Comparative-Annotation-Toolkit/tools/pipeline.py", line 1127, in wait self.raiseIfExcept() File "/home/liran/Software/C-A-T/Comparative-Annotation-Toolkit/tools/pipeline.py", line 1085, in raiseIfExcept p.raiseIfExcept() File "/home/liran/Software/C-A-T/Comparative-Annotation-Toolkit/tools/pipeline.py", line 749, in raiseIfExcept raise self.exceptInfo[0], self.exceptInfo[1], self.exceptInfo[2] ProcException: process exited 127: samtools view -b /tmp/toil-3d7f7e23-af68-431b-9a1b-27ac0c6bd62d/tmpsmFNMz/375989f7-b60c-402d-9ac9-1e71ab9ff27d/tmpFmm0Bq.tmp chr20 ERROR:toil.worker:Exiting the worker because of a failed job on host subnet190-2.nwsuaf.edu.cn WARNING:toil.jobGraph:Due to failure we are reducing the remaining retry count of job 'namesort_bam' S/f/jobWsVBMf with ID S/f/jobWsVBMf to 0 <=========

03-30 10:33:59 luigi-interface INFO Informed scheduler that task GenerateHints_False_True_True_0ccd06bf5f has status FAILED 03-30 10:33:59 luigi-interface ERROR [pid 35765] Worker Worker(salt=724431596, workers=10, host=subnet190-2.nwsuaf.edu.cn, username=liran, pid=35207) failed ToilTask: GenerateHints for hg38 using batchSystem singleMachine Traceback (most recent call last): File "/home/liran/Software/miniconda2/envs/C-A-T/lib/python2.7/site-packages/luigi/worker.py", line 199, in run new_deps = self._run_get_new_deps() File "/home/liran/Software/miniconda2/envs/C-A-T/lib/python2.7/site-packages/luigi/worker.py", line 141, in _run_get_new_deps task_gen = self.task.run() File "/home/liran/Software/C-A-T/Comparative-Annotation-Toolkit/cat/init.py", line 1053, in run hints_db(self.hints_args, toil_options) File "/home/liran/Software/C-A-T/Comparative-Annotation-Toolkit/cat/hints_db.py", line 84, in hints_db combined_hints = t.restart() File "/home/liran/Software/miniconda2/envs/C-A-T/lib/python2.7/site-packages/toil/common.py", line 700, in restart return self._runMainLoop(rootJobGraph) File "/home/liran/Software/miniconda2/envs/C-A-T/lib/python2.7/site-packages/toil/common.py", line 984, in _runMainLoop jobCache=self._jobCache).run() File "/home/liran/Software/miniconda2/envs/C-A-T/lib/python2.7/site-packages/toil/leader.py", line 200, in run raise FailedJobsException(self.config.jobStore, self.toilState.totalFailedJobs, self.jobStore) FailedJobsException: The job store 'file:/home/liran/Software/C-A-T/Comparative-Annotation-Toolkit/test_install/toil/hints_db/hg38/jobStore' contains 3 failed jobs: 'namesort_bam' I/E/jobpE4Exm, 'generate_iso_seq_hints' t/Z/jobbxMOiF, 'setup_hints' t/L/jobKfhBXL =========> Failed job 'namesort_bam' I/E/jobpE4Exm ---TOIL WORKER OUTPUT LOG--- INFO:toil:Running Toil version 3.5.0-7f5bc1cab176d99d6996076ae5b8bc2453fa04a4. WARNING:toil.resource:Can't find resource for leader path '/home/liran/Software/C-A-T/Comparative-Annotation-Toolkit/cat' WARNING:toil.resource:Can't localize module ModuleDescriptor(dirPath='/home/liran/Software/C-A-T/Comparative-Annotation-Toolkit', name='cat.hints_db', fromVirtualEnv=False) INFO:luigi-interface:Loaded ['luigi.cfg'] INFO:toil.fileStore:Starting job ('namesort_bam' I/E/jobpE4Exm) with ID (4c007acbed9b807693aea63673f27db95e3d1d50). WARNING:toil.resource:Can't find resource for leader path '/home/liran/Software/C-A-T/Comparative-Annotation-Toolkit/cat' WARNING:toil.resource:Can't localize module ModuleDescriptor(dirPath='/home/liran/Software/C-A-T/Comparative-Annotation-Toolkit', name='cat.hints_db', fromVirtualEnv=False) samtools: error while loading shared libraries: libcrypto.so.1.0.0: cannot open shared object file: No such file or directory sambamba-sort: not enough data in stream Traceback (most recent call last): File "/home/liran/Software/miniconda2/envs/C-A-T/lib/python2.7/site-packages/toil/worker.py", line 340, in main job._runner(jobGraph=jobGraph, jobStore=jobStore, fileStore=fileStore) File "/home/liran/Software/miniconda2/envs/C-A-T/lib/python2.7/site-packages/toil/job.py", line 1270, in _runner returnValues = self._run(jobGraph, fileStore) File "/home/liran/Software/miniconda2/envs/C-A-T/lib/python2.7/site-packages/toil/job.py", line 1217, in _run return self.run(fileStore) File "/home/liran/Software/miniconda2/envs/C-A-T/lib/python2.7/site-packages/toil/job.py", line 1383, in run rValue = userFunction(*((self,) + tuple(self._args)), **self._kwargs) File "/home/liran/Software/C-A-T/Comparative-Annotation-Toolkit/cat/hints_db.py", line 161, in namesort_bam tools.procOps.run_proc(cmd, stdout=name_sorted) File "/home/liran/Software/C-A-T/Comparative-Annotation-Toolkit/tools/procOps.py", line 68, in run_proc pl.wait() File "/home/liran/Software/C-A-T/Comparative-Annotation-Toolkit/tools/pipeline.py", line 1127, in wait self.raiseIfExcept() File "/home/liran/Software/C-A-T/Comparative-Annotation-Toolkit/tools/pipeline.py", line 1085, in raiseIfExcept p.raiseIfExcept() File "/home/liran/Software/C-A-T/Comparative-Annotation-Toolkit/tools/pipeline.py", line 749, in raiseIfExcept raise self.exceptInfo[0], self.exceptInfo[1], self.exceptInfo[2] ProcException: process exited 127: samtools view -b /tmp/toil-3140b0cd-fef4-45e2-9576-8ad3d48ca251/tmpJRrVMj/2e8bffff-c008-4483-aa7c-c8092d9ee649/tmptxrYF3.tmp chr16 ERROR:toil.worker:Exiting the worker because of a failed job on host subnet190-2.nwsuaf.edu.cn WARNING:toil.jobGraph:Due to failure we are reducing the remaining retry count of job 'namesort_bam' I/E/jobpE4Exm with ID I/E/jobpE4Exm to 0 <=========

=========> Failed job 'generate_iso_seq_hints' t

ifiddes commented 4 years ago

You ran it without --binary-mode local. I only see --binary mode in your command invocation.

Ian Fiddes, PhD

On Sun, Mar 29, 2020 at 9:08 PM YPGG1234 notifications@github.com wrote:

Thanks for your quickly reply, I fixed samtools and re-run the command :

luigi --module cat RunCat --hal=test_data/vertebrates.hal --ref-genome=mm10 --workers=10 --config=test_data/test.config --work-dir test_install --out-dir test_install --local-scheduler --augustus --augustus-cgp --augustus-pb --assembly-hub --binary-mode

but this time I got :

03-30 12:05:13 luigi-interface WARNING Will not run Task: RunCat or any dependencies due to error in complete() method:

Traceback (most recent call last):

File "/home/liran/Software/miniconda2/envs/C-A-T/lib/python2.7/site-packages/luigi/worker.py", line 409, in check_complete

is_complete = task.complete()

File "/home/liran/Software/miniconda2/envs/C-A-T/lib/python2.7/site-packages/luigi/task.py", line 826, in complete

return all(r.complete() for r in flatten(self.requires()))

File "/home/liran/Software/miniconda2/envs/C-A-T/lib/python2.7/site-packages/luigi/task.py", line 887, in flatten

for result in iterator:

File "/home/liran/Software/C-A-T/Comparative-Annotation-Toolkit/cat/init.py", line 624, in requires

pipeline_args = self.get_pipeline_args()

File "/home/liran/Software/C-A-T/Comparative-Annotation-Toolkit/cat/init.py", line 211, in get_pipeline_args

if not tools.misc.is_exec('docker'):

File "/home/liran/Software/C-A-T/Comparative-Annotation-Toolkit/tools/misc.py", line 80, in is_exec

raise ToolMissingException("{0} not found. Either install {0}, or install CAT's dependencies and use --binary-mode local.".format(binary_mode))

ToolMissingException: docker not found. Either install docker, or install CAT's dependencies and use --binary-mode local.

03-30 12:05:13 luigi-interface INFO Informed scheduler that task RunCat_False_True_True_a22c0dbb99 has status UNKNOWN

03-30 12:05:13 luigi-interface INFO Done scheduling tasks

03-30 12:05:13 luigi-interface INFO Running Worker with 10 processes

03-30 12:05:13 luigi-interface INFO Worker Worker(salt=748797917, workers=10, host=subnet190-2.nwsuaf.edu.cn, username=liran, pid=42227) was stopped. Shutting down Keep-Alive thread

03-30 12:05:13 luigi-interface INFO

===== Luigi Execution Summary =====

Scheduled 1 tasks of which:

  • 1 failed scheduling:

    • 1 RunCat(...)

Did not run any tasks

This progress looks :( because there were tasks whose scheduling failed

===== Luigi Execution Summary =====

I got C-A-T by git clone and pip install -e. But actually most of the dependencies of cat (such as luigi, toil, pyfasta and so on) I have got from bioconda earlier. All of these satisfy the version requirements.

Another problem is the first command line:

luigid --background --logdir luigi_logs

No handlers could be found for logger "luigi"

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/ComparativeGenomicsToolkit/Comparative-Annotation-Toolkit/issues/163#issuecomment-605771118, or unsubscribe https://github.com/notifications/unsubscribe-auth/AB3UWXM2NTKLQG4AYFF62L3RKALK5ANCNFSM4LWHG7NQ .

YPGG1234 commented 4 years ago

Hi, I've tested two data ecently, one with the gff3 looks pretty ,

Found 116 transcripts and 54 genes
Found the following gene biotypes: Counter({'protein_coding': 50, 'misc_RNA': 2, 'tRNA': 2})
Found the following transcript biotypes: Counter({'protein_coding': 111, 'misc_RNA': 3, 'tRNA': 2})
Some example database rows:
                             GeneId                                     TranscriptName      GeneName     GeneBiotype TranscriptBiotype
TranscriptId
XM_012176316.3         gene-TMBIM7P  transmembrane BAX inhibitor motif containing 7...       TMBIM7P  protein_coding    protein_coding
NM_001082596.1            gene-TAC1                             tachykinin precursor 1          TAC1  protein_coding    protein_coding
XM_004007752.4           gene-NXPH1                                    neurexophilin 1         NXPH1  protein_coding    protein_coding
XM_027968459.1         gene-PPP1R9A  protein phosphatase 1 regulatory subunit 9A, t...       PPP1R9A  protein_coding    protein_coding
NM_001126348.1            gene-PON2                                      paraoxonase 2          PON2  protein_coding    protein_coding
XM_027968480.1            gene-RPA3                             replication protein A3          RPA3  protein_coding    protein_coding
XM_012176373.3            gene-ASNS  asparagine synthetase (glutamine-hydrolyzing),...          ASNS  protein_coding    protein_coding
rna-TRNAG-CCC-43  gene-TRNAG-CCC-43                                           tRNA-Gly     TRNAG-CCC            tRNA              tRNA
XM_027968482.1           gene-UMAD1  UBAP1-MVB12-associated (UMA) domain containing...         UMAD1  protein_coding    protein_coding
XM_012176369.3          gene-SDHAF3  succinate dehydrogenase complex assembly facto...        SDHAF3  protein_coding    protein_coding
XM_012176340.3           gene-GNGT1  G protein subunit gamma transducin 1, transcri...         GNGT1  protein_coding    protein_coding
XM_027968458.1            gene-SGCE         sarcoglycan epsilon, transcript variant X2          SGCE  protein_coding    protein_coding
XM_004007726.4          gene-COL1A2                      collagen type I alpha 2 chain        COL1A2  protein_coding    protein_coding
XM_012176297.2           gene-AKAP9  A-kinase anchoring protein 9, transcript varia...         AKAP9  protein_coding    protein_coding
XM_027968449.1        gene-HEPACAM2     HEPACAM family member 2, transcript variant X1      HEPACAM2  protein_coding    protein_coding
XM_012176324.3    gene-LOC101114349  EF-hand calcium-binding domain-containing prot...  LOC101114349  protein_coding    protein_coding
XM_015095183.2           gene-KRIT1  KRIT1 ankyrin repeat containing, transcript va...         KRIT1  protein_coding    protein_coding
XM_012177413.3            gene-CDK6   cyclin dependent kinase 6, transcript variant X1          CDK6  protein_coding    protein_coding
XM_027968493.1            gene-ICA1   islet cell autoantigen 1, transcript variant X11          ICA1  protein_coding    protein_coding
XM_027968473.1            gene-SEM1                SEM1 26S proteasome complex subunit          SEM1  protein_coding    protein_coding
Some example gene rows:
                                                 TranscriptName GeneName     GeneBiotype TranscriptBiotype
GeneId
gene-AKAP9    A-kinase anchoring protein 9, transcript varia...    AKAP9  protein_coding    protein_coding
gene-ANKIB1   ankyrin repeat and IBR domain containing 1, tr...   ANKIB1  protein_coding    protein_coding
gene-ASB4              ankyrin repeat and SOCS box containing 4     ASB4  protein_coding    protein_coding
gene-ASNS     asparagine synthetase (glutamine-hydrolyzing),...     ASNS  protein_coding    protein_coding
gene-BET1     Bet1 golgi vesicular membrane trafficking protein     BET1  protein_coding    protein_coding
gene-C1GALT1  core 1 synthase, glycoprotein-N-acetylgalactos...  C1GALT1  protein_coding    protein_coding
gene-CALCR                                  calcitonin receptor    CALCR  protein_coding    protein_coding
gene-CASD1      CAS1 domain containing 1, transcript variant X2    CASD1  protein_coding    protein_coding
gene-CDK6      cyclin dependent kinase 6, transcript variant X1     CDK6  protein_coding    protein_coding
gene-COL1A2                       collagen type I alpha 2 chain   COL1A2  protein_coding    protein_coding

but when I run : luigi --module cat RunCat --hal=Hu_Oar_ARS.hal --target-genomes='("Hu",)' --ref-genome=Oar --workers=1 --config=Oar.cfg --work-dir Oar_test --out-dir Oar_test --assembly-hub --local-scheduler --binary-mode local &>>Hu.log

I got

......
......
INFO: [pid 2664] Worker Worker(salt=792176316, workers=1, host=subnet190-2.nwsuaf.edu.cn, username=liran, pid=2664) running   Task: BgpTrack for Oar
subnet190-2.nwsuaf.edu.cn 2020-04-08 21:48:20,738 MainThread INFO luigi-interface: [pid 2664] Worker Worker(salt=792176316, workers=1, host=subnet190-2.nwsuaf.edu.cn, username=liran, pid=2664) running   Task: BgpTrack for Oar
ERROR: [pid 2664] Worker Worker(salt=792176316, workers=1, host=subnet190-2.nwsuaf.edu.cn, username=liran, pid=2664) failed    Task: BgpTrack for Oar
Traceback (most recent call last):
  File "/home/liran/Software/miniconda2/envs/C-A-T/lib/python2.7/site-packages/luigi/worker.py", line 199, in run
    new_deps = self._run_get_new_deps()
  File "/home/liran/Software/miniconda2/envs/C-A-T/lib/python2.7/site-packages/luigi/worker.py", line 141, in _run_get_new_deps
    task_gen = self.task.run()
  File "/home/liran/Software/C-A-T/Comparative-Annotation-Toolkit/cat/__init__.py", line 2533, in run
    s = annotation_info.ix[tools.nameConversions.strip_alignment_numbers(tx.name)]
  File "/home/liran/Software/miniconda2/envs/C-A-T/lib/python2.7/site-packages/pandas/core/indexing.py", line 126, in __getitem__
    return self._getitem_axis(key, axis=axis)
  File "/home/liran/Software/miniconda2/envs/C-A-T/lib/python2.7/site-packages/pandas/core/indexing.py", line 1106, in _getitem_axis
    return self._get_label(key, axis=axis)
  File "/home/liran/Software/miniconda2/envs/C-A-T/lib/python2.7/site-packages/pandas/core/indexing.py", line 141, in _get_label
    return self.obj._xs(label, axis=axis)
  File "/home/liran/Software/miniconda2/envs/C-A-T/lib/python2.7/site-packages/pandas/core/generic.py", line 3585, in xs
    loc = self.index.get_loc(key)
  File "/home/liran/Software/miniconda2/envs/C-A-T/lib/python2.7/site-packages/pandas/core/indexes/base.py", line 2659, in get_loc
    return self._engine.get_loc(self._maybe_cast_indexer(key))
  File "pandas/_libs/index.pyx", line 108, in pandas._libs.index.IndexEngine.get_loc
  File "pandas/_libs/index.pyx", line 132, in pandas._libs.index.IndexEngine.get_loc
  File "pandas/_libs/hashtable_class_helper.pxi", line 1601, in pandas._libs.hashtable.PyObjectHashTable.get_item
  File "pandas/_libs/hashtable_class_helper.pxi", line 1608, in pandas._libs.hashtable.PyObjectHashTable.get_item
KeyError: 'rna-TRNAE-UUC'
subnet190-2.nwsuaf.edu.cn 2020-04-08 21:48:21,463 MainThread ERROR luigi-interface: [pid 2664] Worker Worker(salt=792176316, workers=1, host=subnet190-2.nwsuaf.edu.cn, username=liran, pid=2664) failed    Task: BgpTrack for Oar
Traceback (most recent call last):
  File "/home/liran/Software/miniconda2/envs/C-A-T/lib/python2.7/site-packages/luigi/worker.py", line 199, in run
    new_deps = self._run_get_new_deps()
  File "/home/liran/Software/miniconda2/envs/C-A-T/lib/python2.7/site-packages/luigi/worker.py", line 141, in _run_get_new_deps
    task_gen = self.task.run()
  File "/home/liran/Software/C-A-T/Comparative-Annotation-Toolkit/cat/__init__.py", line 2533, in run
    s = annotation_info.ix[tools.nameConversions.strip_alignment_numbers(tx.name)]
  File "/home/liran/Software/miniconda2/envs/C-A-T/lib/python2.7/site-packages/pandas/core/indexing.py", line 126, in __getitem__
    return self._getitem_axis(key, axis=axis)
  File "/home/liran/Software/miniconda2/envs/C-A-T/lib/python2.7/site-packages/pandas/core/indexing.py", line 1106, in _getitem_axis
    return self._get_label(key, axis=axis)
  File "/home/liran/Software/miniconda2/envs/C-A-T/lib/python2.7/site-packages/pandas/core/indexing.py", line 141, in _get_label
    return self.obj._xs(label, axis=axis)
  File "/home/liran/Software/miniconda2/envs/C-A-T/lib/python2.7/site-packages/pandas/core/generic.py", line 3585, in xs
    loc = self.index.get_loc(key)
  File "/home/liran/Software/miniconda2/envs/C-A-T/lib/python2.7/site-packages/pandas/core/indexes/base.py", line 2659, in get_loc
    return self._engine.get_loc(self._maybe_cast_indexer(key))
  File "pandas/_libs/index.pyx", line 108, in pandas._libs.index.IndexEngine.get_loc
  File "pandas/_libs/index.pyx", line 132, in pandas._libs.index.IndexEngine.get_loc
  File "pandas/_libs/hashtable_class_helper.pxi", line 1601, in pandas._libs.hashtable.PyObjectHashTable.get_item
  File "pandas/_libs/hashtable_class_helper.pxi", line 1608, in pandas._libs.hashtable.PyObjectHashTable.get_item
KeyError: 'rna-TRNAE-UUC'
DEBUG: 1 running tasks, waiting for next task to finish
subnet190-2.nwsuaf.edu.cn 2020-04-08 21:48:21,470 MainThread DEBUG luigi-interface: 1 running tasks, waiting for next task to finish
INFO: Informed scheduler that task   BgpTrack_False_True_False_51b53a242c   has status   FAILED
subnet190-2.nwsuaf.edu.cn 2020-04-08 21:48:21,477 MainThread INFO luigi-interface: Informed scheduler that task   BgpTrack_False_True_False_51b53a242c   has status   FAILED
DEBUG: Asking scheduler for work...
subnet190-2.nwsuaf.edu.cn 2020-04-08 21:48:21,477 MainThread DEBUG luigi-interface: Asking scheduler for work...
DEBUG: Pending tasks: 13
subnet190-2.nwsuaf.edu.cn 2020-04-08 21:48:21,478 MainThread DEBUG luigi-interface: Pending tasks: 13
INFO: [pid 2664] Worker Worker(salt=792176316, workers=1, host=subnet190-2.nwsuaf.edu.cn, username=liran, pid=2664) running   Task: ConsensusTrack for Hu
subnet190-2.nwsuaf.edu.cn 2020-04-08 21:48:21,478 MainThread INFO luigi-interface: [pid 2664] Worker Worker(salt=792176316, workers=1, host=subnet190-2.nwsuaf.edu.cn, username=liran, pid=2664) running   Task: ConsensusTrack for Hu
INFO: [pid 2664] Worker Worker(salt=792176316, workers=1, host=subnet190-2.nwsuaf.edu.cn, username=liran, pid=2664) done      Task: ConsensusTrack for Hu
subnet190-2.nwsuaf.edu.cn 2020-04-08 21:48:22,437 MainThread INFO luigi-interface: [pid 2664] Worker Worker(salt=792176316, workers=1, host=subnet190-2.nwsuaf.edu.cn, username=liran, pid=2664) done      Task: ConsensusTrack for Hu
DEBUG: 1 running tasks, waiting for next task to finish
subnet190-2.nwsuaf.edu.cn 2020-04-08 21:48:22,438 MainThread DEBUG luigi-interface: 1 running tasks, waiting for next task to finish
INFO: Informed scheduler that task   ConsensusTrack_False_True_False_d1b33b3f25   has status   DONE
subnet190-2.nwsuaf.edu.cn 2020-04-08 21:48:22,440 MainThread INFO luigi-interface: Informed scheduler that task   ConsensusTrack_False_True_False_d1b33b3f25   has status   DONE
DEBUG: Asking scheduler for work...
subnet190-2.nwsuaf.edu.cn 2020-04-08 21:48:22,440 MainThread DEBUG luigi-interface: Asking scheduler for work...
DEBUG: Pending tasks: 12
subnet190-2.nwsuaf.edu.cn 2020-04-08 21:48:22,441 MainThread DEBUG luigi-interface: Pending tasks: 12
INFO: [pid 2664] Worker Worker(salt=792176316, workers=1, host=subnet190-2.nwsuaf.edu.cn, username=liran, pid=2664) running   Task: TransMapTrack for Hu
subnet190-2.nwsuaf.edu.cn 2020-04-08 21:48:22,441 MainThread INFO luigi-interface: [pid 2664] Worker Worker(salt=792176316, workers=1, host=subnet190-2.nwsuaf.edu.cn, username=liran, pid=2664) running   Task: TransMapTrack for Hu
INFO: [pid 2664] Worker Worker(salt=792176316, workers=1, host=subnet190-2.nwsuaf.edu.cn, username=liran, pid=2664) done      Task: TransMapTrack for Hu
subnet190-2.nwsuaf.edu.cn 2020-04-08 21:48:23,414 MainThread INFO luigi-interface: [pid 2664] Worker Worker(salt=792176316, workers=1, host=subnet190-2.nwsuaf.edu.cn, username=liran, pid=2664) done      Task: TransMapTrack for Hu
DEBUG: 1 running tasks, waiting for next task to finish
subnet190-2.nwsuaf.edu.cn 2020-04-08 21:48:23,415 MainThread DEBUG luigi-interface: 1 running tasks, waiting for next task to finish
INFO: Informed scheduler that task   TransMapTrack_False_True_False_da9dbb94af   has status   DONE
subnet190-2.nwsuaf.edu.cn 2020-04-08 21:48:23,417 MainThread INFO luigi-interface: Informed scheduler that task   TransMapTrack_False_True_False_da9dbb94af   has status   DONE
DEBUG: Asking scheduler for work...
subnet190-2.nwsuaf.edu.cn 2020-04-08 21:48:23,417 MainThread DEBUG luigi-interface: Asking scheduler for work...
DEBUG: Pending tasks: 11
subnet190-2.nwsuaf.edu.cn 2020-04-08 21:48:23,418 MainThread DEBUG luigi-interface: Pending tasks: 11
INFO: [pid 2664] Worker Worker(salt=792176316, workers=1, host=subnet190-2.nwsuaf.edu.cn, username=liran, pid=2664) running   Task: EvaluationTrack for Hu
subnet190-2.nwsuaf.edu.cn 2020-04-08 21:48:23,418 MainThread INFO luigi-interface: [pid 2664] Worker Worker(salt=792176316, workers=1, host=subnet190-2.nwsuaf.edu.cn, username=liran, pid=2664) running   Task: EvaluationTrack for Hu
INFO: [pid 2664] Worker Worker(salt=792176316, workers=1, host=subnet190-2.nwsuaf.edu.cn, username=liran, pid=2664) done      Task: EvaluationTrack for Hu
subnet190-2.nwsuaf.edu.cn 2020-04-08 21:48:24,350 MainThread INFO luigi-interface: [pid 2664] Worker Worker(salt=792176316, workers=1, host=subnet190-2.nwsuaf.edu.cn, username=liran, pid=2664) done      Task: EvaluationTrack for Hu
DEBUG: 1 running tasks, waiting for next task to finish
subnet190-2.nwsuaf.edu.cn 2020-04-08 21:48:24,351 MainThread DEBUG luigi-interface: 1 running tasks, waiting for next task to finish
INFO: Informed scheduler that task   EvaluationTrack_False_True_False_b905f3d972   has status   DONE
subnet190-2.nwsuaf.edu.cn 2020-04-08 21:48:24,353 MainThread INFO luigi-interface: Informed scheduler that task   EvaluationTrack_False_True_False_b905f3d972   has status   DONE
DEBUG: Asking scheduler for work...
subnet190-2.nwsuaf.edu.cn 2020-04-08 21:48:24,354 MainThread DEBUG luigi-interface: Asking scheduler for work...
DEBUG: Pending tasks: 10
subnet190-2.nwsuaf.edu.cn 2020-04-08 21:48:24,355 MainThread DEBUG luigi-interface: Pending tasks: 10
INFO: [pid 2664] Worker Worker(salt=792176316, workers=1, host=subnet190-2.nwsuaf.edu.cn, username=liran, pid=2664) running   Task: BgpTrack for Hu
subnet190-2.nwsuaf.edu.cn 2020-04-08 21:48:24,355 MainThread INFO luigi-interface: [pid 2664] Worker Worker(salt=792176316, workers=1, host=subnet190-2.nwsuaf.edu.cn, username=liran, pid=2664) running   Task: BgpTrack for Hu
INFO: [pid 2664] Worker Worker(salt=792176316, workers=1, host=subnet190-2.nwsuaf.edu.cn, username=liran, pid=2664) done      Task: BgpTrack for Hu
subnet190-2.nwsuaf.edu.cn 2020-04-08 21:48:25,191 MainThread INFO luigi-interface: [pid 2664] Worker Worker(salt=792176316, workers=1, host=subnet190-2.nwsuaf.edu.cn, username=liran, pid=2664) done      Task: BgpTrack for Hu
DEBUG: 1 running tasks, waiting for next task to finish
subnet190-2.nwsuaf.edu.cn 2020-04-08 21:48:25,192 MainThread DEBUG luigi-interface: 1 running tasks, waiting for next task to finish
INFO: Informed scheduler that task   BgpTrack_False_True_False_71dcc029b3   has status   DONE
subnet190-2.nwsuaf.edu.cn 2020-04-08 21:48:25,193 MainThread INFO luigi-interface: Informed scheduler that task   BgpTrack_False_True_False_71dcc029b3   has status   DONE
DEBUG: Asking scheduler for work...
subnet190-2.nwsuaf.edu.cn 2020-04-08 21:48:25,193 MainThread DEBUG luigi-interface: Asking scheduler for work...
DEBUG: Pending tasks: 9
subnet190-2.nwsuaf.edu.cn 2020-04-08 21:48:25,194 MainThread DEBUG luigi-interface: Pending tasks: 9
INFO: [pid 2664] Worker Worker(salt=792176316, workers=1, host=subnet190-2.nwsuaf.edu.cn, username=liran, pid=2664) running   Task: CreateTracksDriverTask for Hu
subnet190-2.nwsuaf.edu.cn 2020-04-08 21:48:25,194 MainThread INFO luigi-interface: [pid 2664] Worker Worker(salt=792176316, workers=1, host=subnet190-2.nwsuaf.edu.cn, username=liran, pid=2664) running   Task: CreateTracksDriverTask for Hu
INFO: [pid 2664] Worker Worker(salt=792176316, workers=1, host=subnet190-2.nwsuaf.edu.cn, username=liran, pid=2664) done      Task: CreateTracksDriverTask for Hu
subnet190-2.nwsuaf.edu.cn 2020-04-08 21:48:25,240 MainThread INFO luigi-interface: [pid 2664] Worker Worker(salt=792176316, workers=1, host=subnet190-2.nwsuaf.edu.cn, username=liran, pid=2664) done      Task: CreateTracksDriverTask for Hu
DEBUG: 1 running tasks, waiting for next task to finish
subnet190-2.nwsuaf.edu.cn 2020-04-08 21:48:25,241 MainThread DEBUG luigi-interface: 1 running tasks, waiting for next task to finish
INFO: Informed scheduler that task   CreateTracksDriverTask_False_True_False_26f9f66c19   has status   DONE
subnet190-2.nwsuaf.edu.cn 2020-04-08 21:48:25,242 MainThread INFO luigi-interface: Informed scheduler that task   CreateTracksDriverTask_False_True_False_26f9f66c19   has status   DONE
DEBUG: Asking scheduler for work...
subnet190-2.nwsuaf.edu.cn 2020-04-08 21:48:25,242 MainThread DEBUG luigi-interface: Asking scheduler for work...
DEBUG: Pending tasks: 8
subnet190-2.nwsuaf.edu.cn 2020-04-08 21:48:25,243 MainThread DEBUG luigi-interface: Pending tasks: 8
INFO: [pid 2664] Worker Worker(salt=792176316, workers=1, host=subnet190-2.nwsuaf.edu.cn, username=liran, pid=2664) running   Task: CreateTrackDbs for Hu
subnet190-2.nwsuaf.edu.cn 2020-04-08 21:48:25,243 MainThread INFO luigi-interface: [pid 2664] Worker Worker(salt=792176316, workers=1, host=subnet190-2.nwsuaf.edu.cn, username=liran, pid=2664) running   Task: CreateTrackDbs for Hu
INFO: [pid 2664] Worker Worker(salt=792176316, workers=1, host=subnet190-2.nwsuaf.edu.cn, username=liran, pid=2664) done      Task: CreateTrackDbs for Hu
subnet190-2.nwsuaf.edu.cn 2020-04-08 21:48:25,329 MainThread INFO luigi-interface: [pid 2664] Worker Worker(salt=792176316, workers=1, host=subnet190-2.nwsuaf.edu.cn, username=liran, pid=2664) done      Task: CreateTrackDbs for Hu
DEBUG: 1 running tasks, waiting for next task to finish
subnet190-2.nwsuaf.edu.cn 2020-04-08 21:48:25,330 MainThread DEBUG luigi-interface: 1 running tasks, waiting for next task to finish
INFO: Informed scheduler that task   CreateTrackDbs_False_True_False_26f9f66c19   has status   DONE
subnet190-2.nwsuaf.edu.cn 2020-04-08 21:48:25,331 MainThread INFO luigi-interface: Informed scheduler that task   CreateTrackDbs_False_True_False_26f9f66c19   has status   DONE
DEBUG: Asking scheduler for work...
subnet190-2.nwsuaf.edu.cn 2020-04-08 21:48:25,332 MainThread DEBUG luigi-interface: Asking scheduler for work...
DEBUG: Done
subnet190-2.nwsuaf.edu.cn 2020-04-08 21:48:25,332 MainThread DEBUG luigi-interface: Done
DEBUG: There are no more tasks to run at this time
subnet190-2.nwsuaf.edu.cn 2020-04-08 21:48:25,332 MainThread DEBUG luigi-interface: There are no more tasks to run at this time
DEBUG: There are 7 pending tasks possibly being run by other workers
subnet190-2.nwsuaf.edu.cn 2020-04-08 21:48:25,332 MainThread DEBUG luigi-interface: There are 7 pending tasks possibly being run by other workers
DEBUG: There are 7 pending tasks unique to this worker
subnet190-2.nwsuaf.edu.cn 2020-04-08 21:48:25,332 MainThread DEBUG luigi-interface: There are 7 pending tasks unique to this worker
DEBUG: There are 7 pending tasks last scheduled by this worker
subnet190-2.nwsuaf.edu.cn 2020-04-08 21:48:25,332 MainThread DEBUG luigi-interface: There are 7 pending tasks last scheduled by this worker
INFO: Worker Worker(salt=792176316, workers=1, host=subnet190-2.nwsuaf.edu.cn, username=liran, pid=2664) was stopped. Shutting down Keep-Alive thread
subnet190-2.nwsuaf.edu.cn 2020-04-08 21:48:25,336 Thread-1 INFO luigi-interface: Worker Worker(salt=792176316, workers=1, host=subnet190-2.nwsuaf.edu.cn, username=liran, pid=2664) was stopped. Shutting down Keep-Alive thread
subnet190-2.nwsuaf.edu.cn 2020-04-08 21:48:25,358 MainThread WARNING cat: No extrinsic data found in config. Will load genomes and annotation only.
INFO:
===== Luigi Execution Summary =====

Scheduled 51 tasks of which:
* 44 ran successfully:
    - 1 AlignTranscriptDriverTask(...)
    - 1 AlignTranscripts(...)
    - 1 BgpTrack(...)
    - 1 BuildDb(...)
    - 1 Chaining(...)
    ...
* 1 failed:
    - 1 BgpTrack(...)
* 6 were left pending, among these:
    * 6 had failed dependencies:
        - 1 AssemblyHub(...)
        - 1 CreateTrackDbs(...)
        - 1 CreateTracks(...)
        - 1 CreateTracksDriverTask(...)
        - 1 ReportStats(...)
        ...

This progress looks :( because there were failed tasks

===== Luigi Execution Summary =====

subnet190-2.nwsuaf.edu.cn 2020-04-08 21:48:25,614 MainThread INFO luigi-interface:
===== Luigi Execution Summary =====

Scheduled 51 tasks of which:
* 44 ran successfully:
    - 1 AlignTranscriptDriverTask(...)
    - 1 AlignTranscripts(...)
    - 1 BgpTrack(...)
    - 1 BuildDb(...)
    - 1 Chaining(...)
    ...
* 1 failed:
    - 1 BgpTrack(...)
* 6 were left pending, among these:
    * 6 had failed dependencies:
        - 1 AssemblyHub(...)
        - 1 CreateTrackDbs(...)
        - 1 CreateTracks(...)
        - 1 CreateTracksDriverTask(...)
        - 1 ReportStats(...)
        ...

This progress looks :( because there were failed tasks

===== Luigi Execution Summary =====

subnet190-2.nwsuaf.edu.cn 2020-04-08 21:48:26,224 MainThread WARNING cat: No extrinsic data found in config. Will load genomes and annotation only.

Could you tell me what cause this inconsistent?

The second gff3 looks a little bit wired

Found 46486 transcripts and 23308 genes
Found the following gene biotypes: Counter({'protein_coding': 20343, 'tRNA': 1792, 'misc_RNA': 745, 'precursor_RNA': 264, 'V_segment': 146, 'C_region': 16, 'rRNA': 2})
Found the following transcript biotypes: Counter({'protein_coding': 42687, 'tRNA': 1792, 'misc_RNA': 1579, 'precursor_RNA': 264, 'V_segment': 146, 'C_region': 16, 'rRNA': 2})
Some example database rows:
                           GeneId                                     TranscriptName      GeneName     GeneBiotype TranscriptBiotype
TranscriptId
XM_018059664.1           gene-CBL          Cbl proto-oncogene, transcript variant X1           CBL  protein_coding    protein_coding
XM_018038520.1        gene-ZNF165                            zinc finger protein 165        ZNF165  protein_coding    protein_coding
XM_013974243.2  gene-LOC102174890  zinc finger and SCAN domain-containing protein 30  LOC102174890  protein_coding    protein_coding
XM_018043726.1  gene-LOC108634324                melanoma-associated antigen B2-like  LOC108634324  protein_coding    protein_coding
XM_018039398.1         gene-KCNG2  potassium voltage-gated channel modifier subfa...         KCNG2  protein_coding    protein_coding
XM_005694577.1        gene-SMIM23                 small integral membrane protein 23        SMIM23  protein_coding    protein_coding
XM_018061103.1           gene-MVK           mevalonate kinase, transcript variant X2           MVK  protein_coding    protein_coding
XM_018051671.1       gene-TGFBR3L    transforming growth factor beta receptor 3 like       TGFBR3L  protein_coding    protein_coding
XM_018050040.1       gene-FAM172A  family with sequence similarity 172 member A, ...       FAM172A  protein_coding    protein_coding
XM_005690178.2  gene-LOC102175320                        olfactory receptor 4B1-like  LOC102175320  protein_coding    protein_coding
XM_018065508.1         gene-CENPK        centromere protein K, transcript variant X2         CENPK  protein_coding    protein_coding
XM_018051752.1  gene-LOC106502257                       uncharacterized LOC106502257  LOC106502257  protein_coding    protein_coding
XR_001919467.1  gene-LOC102178101  protein-serine O-palmitoleoyltransferase porcu...  LOC102178101        misc_RNA          misc_RNA
XM_018060880.1         gene-CSRP1  cysteine and glycine rich protein 1, transcrip...         CSRP1  protein_coding    protein_coding
XM_005693143.3         gene-BCAS3  BCAS3, microtubule associated cell migration f...         BCAS3  protein_coding    protein_coding
XM_018064833.1         gene-MBTD1     mbt domain containing 1, transcript variant X3         MBTD1  protein_coding    protein_coding
XM_018051140.1        gene-ZNF358     zinc finger protein 358, transcript variant X1        ZNF358  protein_coding    protein_coding
XM_018044954.1  gene-LOC108634752                     collagen alpha-1(I) chain-like  LOC108634752  protein_coding    protein_coding
XM_005692309.3          gene-DMKN                   dermokine, transcript variant X1          DMKN  protein_coding    protein_coding
XM_018059266.1         gene-HIPK3  homeodomain interacting protein kinase 3, tran...         HIPK3  protein_coding    protein_coding
Some example gene rows:
                                                 TranscriptName GeneName     GeneBiotype TranscriptBiotype
GeneId
gene-3BHSD    3beta-hydroxysteroid dehydrogenase/isomerase, ...    3BHSD  protein_coding    protein_coding
gene-A1BG                                alpha-1-B glycoprotein     A1BG  protein_coding    protein_coding
gene-A1CF     APOBEC1 complementation factor, transcript var...     A1CF  protein_coding    protein_coding
gene-A2ML1    alpha-2-macroglobulin like 1, transcript varia...    A2ML1  protein_coding    protein_coding
gene-A3GALT2                  alpha 1,3-galactosyltransferase 2  A3GALT2  protein_coding    protein_coding
gene-A4GALT                     alpha 1,4-galactosyltransferase   A4GALT  protein_coding    protein_coding
gene-A4GNT    alpha-1,4-N-acetylglucosaminyltransferase, tra...    A4GNT  protein_coding    protein_coding
gene-AAAS                          aladin WD repeat nucleoporin     AAAS  protein_coding    protein_coding
gene-AACS                            acetoacetyl-CoA synthetase     AACS  protein_coding    protein_coding
gene-AADACL3  arylacetamide deacetylase like 3, transcript v...  AADACL3  protein_coding    protein_coding
Found 59 genes where one gene name maps to multiple gene IDs:
TRNAC-ACA: gene-TRNAC-ACA-86,gene-TRNAC-ACA-87,gene-TRNAC-ACA-84,gene-TRNAC-ACA-85,gene-TRNAC-ACA-82,gene-TRNAC-ACA-83,gene-TRNAC-ACA-80,gene-TRNAC-ACA-81,gene-TRNAC-ACA-88,gene-TRNAC-ACA-89,gene-TRNAC-ACA-33,gene-TRNAC-ACA-32,gene-TRNAC-ACA-31,gene-TRNAC-ACA-30,gene-TRNAC-ACA-37,gene-TRNAC-ACA-36,gene-TRNAC-ACA-35,gene-TRNAC-ACA-34,gene-TRNAC-ACA-39,gene-TRNAC-ACA-38,gene-TRNAC-ACA-24,gene-TRNAC-ACA-25,gene-TRNAC-ACA-26,gene-TRNAC-ACA-27,gene-TRNAC-ACA-20,gene-TRNAC-ACA-21,gene-TRNAC-ACA-22,gene-TRNAC-ACA-23,gene-TRNAC-ACA-28,gene-TRNAC-ACA-29,gene-TRNAC-ACA-5,gene-TRNAC-ACA-4,gene-TRNAC-ACA-7,gene-TRNAC-ACA-105,gene-TRNAC-ACA-102,gene-TRNAC-ACA-103,gene-TRNAC-ACA-100,gene-TRNAC-ACA-2,gene-TRNAC-ACA-9,gene-TRNAC-ACA-8,gene-TRNAC-ACA-108,gene-TRNAC-ACA-109,gene-TRNAC-ACA,gene-TRNAC-ACA-50,gene-TRNAC-ACA-53,gene-TRNAC-ACA-52,gene-TRNAC-ACA-55,gene-TRNAC-ACA-54,gene-TRNAC-ACA-57,gene-TRNAC-ACA-56,gene-T
......
......

I dont know whether it can work right, becasue the hal file correspond to this gff3 file is still running. Another question is the input hal file must be generated from Cactus? It costs too many times and resources for large genome. Hal file converted from maf file seems work poorly.

ifiddes commented 4 years ago

Your GFF3 error looks like a bug to me. Are you able to share the GFF3 file with me?

edit: Where did the GFF3 come from? I can try to add a script that modifies it to fit the CAT expectations, I have been populating the programs/ folder with scripts like that.

Cactus: CAT is not tested with alignments produced from maf2hal. I suspect it will not work well, because that code is very old. @diekhans is there any updates to that? I have wanted to enable other types of alignments in CAT for a while; maybe we should look into that. CAT really operates on the level of pairwise chain files, but the idea of having a HAL to get the genome names from is pretty baked in, but could be worked around.

diekhans commented 4 years ago

maf2hal has never worked reliably

Ian Fiddes notifications@github.com writes:

Your GFF3 error looks like a bug to me. Are you able to share the GFF3 file with me?

Cactus: CAT is not tested with alignments produced from maf2hal. I suspect it will not work well, because that code is very old. @diekhans is there any updates to that? I have wanted to enable other types of alignments in CAT for a while; maybe we should look into that. CAT really operates on the level of pairwise chain files, but the idea of having a HAL to get the genome names from is pretty baked in, but could be worked around.

YPGG1234 commented 4 years ago

Your GFF3 error looks like a bug to me. Are you able to share the GFF3 file with me?

edit: Where did the GFF3 come from? I can try to add a script that modifies it to fit the CAT expectations, I have been populating the programs/ folder with scripts like that.

Cactus: CAT is not tested with alignments produced from maf2hal. I suspect it will not work well, because that code is very old. @diekhans is there any updates to that? I have wanted to enable other types of alignments in CAT for a while; maybe we should look into that. CAT really operates on the level of pairwise chain files, but the idea of having a HAL to get the genome names from is pretty baked in, but could be worked around.

Of cource. This gff3 file is modified from current sheep reference genome Oar_rambouillet_v1.0 gff3 to fit my test reference data (chromosome 4, region: 10,000,000 - 20,000,000). I put the tree ((mytestdata,sheep)goat); to the Cactus input file, and when I used halStats I got ((mytestdata:1,sheep:1)Anc1:1,goat:1)Anc0;, actually I want to use sheep and goat to annotate my real data separately, and I wish I can do it with one hal file (since the generating of hal file is very pain) , does this tree right? If the gff3 is 'right', it can work done?

Does Found 59 genes where one gene name maps to multiple gene IDs: in gff3 files affect CAT works?

sheep4.gff3.txt

ifiddes commented 4 years ago

Hi,

I think you are using an older version of CAT. These issues with GFF3 files are why, with CAT2.0, I require that the GFF3 fit my format. Your GFF3 fails the CAT2.0 parser. I have produced a new script, https://github.com/ComparativeGenomicsToolkit/Comparative-Annotation-Toolkit/blob/test_ncbi_gff3/programs/cat_parse_ncbi_gff3 that converts your GFF3 into one CAT likes.

Can you try switching to CAT 2.0 and using this?

Your other questions:

Does Found 59 genes where one gene name maps to multiple gene IDs: in gff3 files affect CAT works?

No. This warning is just letting you know that there may be ambiguity. CAT operates on the level of gene ID, gene name is just for convenience.

I put the tree ((mytestdata,sheep)goat); to the Cactus input file, and when I used halStats I got ((mytestdata:1,sheep:1)Anc1:1,goat:1)Anc0;, actually I want to use sheep and goat to annotate my real data separately, and I wish I can do it with one hal file (since the generating of hal file is very pain) , does this tree right? If the gff3 is 'right', it can work done?

You should produce a binary tree. Something like this:

((mytestdata,sheep),goat);

Then you will have to run CAT twice, once with --ref-genome=sheep and once with --ref-genome=goat. Since you are working with a tiny fraction of genome, this should take comparable time to the test dataset.

YPGG1234 commented 4 years ago

Yes, a few days later after I got CAT, the new version was released, I will try it.

YPGG1234 commented 4 years ago

Dear ifiddes,

Thanks for your great work for CAT, without anyother extrinsic hints the CAT is work well. Now I want to use RNA-seq and Iso-seq to annotate our genome (one sheep). The Iso-seq data and one paired-end polyA-selected RNA-seq data are come from testis of our genome, other RNA-seq data of sheeps are downloaded from NCBI.

I have three questions. First is which genome should the RNA-seq data and Iso-seq data mapping to, our genome or reference genome? Another is cause our compute resources are limited, there are only 2 cores with 48 threads can be use, and there are other users also use this resource, could you please tell me which parameters should I select to annotate our genome in a more efficient and economical way? The last question is the current sheep reference genome annotation is well enough, I wonder how much the genome annotation can be imporved after annotate with RNA-seq and Iso-seq.

Looking forword to your reply.

ifiddes commented 4 years ago

I am glad you got it to work.

  1. The IsoSeq should be mapped to your genome. Then pass that in as a ISO_SEQ_BAM to CAT, and use the flag --augustus-pb. This will have AUGUSTUS do single-genome prediction of novel loci and alternative isoforms on your genome. These predictions will be integrated with the mapped over genes.
  2. The above flag, --augustus-pb, is heavier on compute than the default CAT mode. It will spawn probably 2000-3000 jobs that will take 5-10 minutes each, depending on how deep your IsoSeq dataset is.

Incorporating RNA-seq can be helpful in two ways: 1. AugustusTMR, and 2. AugustusCGP (Comparative Gene Prediction).

The question you need to ask is what your want to use this annotation set for? AugustusPB will provide you with species-specific isoforms and loci, if they are present in the IsoSeq data. AugustusTMR (using RNA-seq) will clean up your transMap projections, fixing splice shifts and such, but will not add any new transcripts not present in the source genome. AugustusCGP (which is a significantly larger amount of compute) leverages the alignment itself in addition to RNA-seq to predict genes comparatively. This is most useful for larger trees with longer branch lengths.

Since it seems like you are annotating two closely related assemblies, I would suggest only using the IsoSeq data in the AugustusPB mode.

There is one other new feature of CAT to consider here as well. If you run the Cupcake pipeline on your IsoSeq data, you can then convert the output GFF to be suitable to pass to CAT as an external annotation. Doing this has two advantages: 1) It is a lot less compute, and 2) it will not break non-coding transcripts. It has one disadvantage: AugustusPB enforces a coding model, and so IsoSeq reads that are incomplete or that represent things like retained introns will be discarded. Each prediction has a realistic ORF.

To do this, after you run Cupcake https://github.com/Magdoll/cDNA_Cupcake/wiki you then convert the output GFF to a CAT-compatible GFF3 with this script https://github.com/ComparativeGenomicsToolkit/Comparative-Annotation-Toolkit/blob/master/programs/convert_cupcake_gff then you place that annotation file under the ANNOTATION section of the config file. So you would have 2 annotation files, 1 on the reference genome and 1 on your new genome.

ifiddes commented 4 years ago

One thing I forgot to mention -- you could combine both by running --augustus-pb AND using Cupcake

YPGG1234 commented 4 years ago

When I used STAR to build the genome index of our genome which is used for mapping RNA-seq data to, it complained that "EXITING because of FATAL INPUT PARAMETER ERROR: when generating genome without annotations", in my case does this mean I should provide a annotation file generated from CAT without anyother extrinsic hints?

ifiddes commented 4 years ago

I have not seen that error in STAR before; maybe newer versions require annotations? I haven't used STAR in a while. I suppose you could use the CAT annotation file, I am not sure if STAR supports GFF3 or not. If it doesn't, you might be able to use gff3ToGenePred followed by genePredToGtf to make the conversion.

ifiddes commented 4 years ago

You may have had an extra space https://www.biostars.org/p/344584/

diekhans commented 4 years ago

I can use start without annotations using this script:

https://github.com/diekhans/T2T_cenSat_hub/blob/master/bin/runEncodeAlign

stolen mostly from the EnCODE pipeline

Ian Fiddes notifications@github.com writes:

I have not seen that error in STAR before; maybe newer versions require annotations? I haven't used STAR in a while. I suppose you could use the CAT annotation file, I am not sure if STAR supports GFF3 or not. If it doesn't, you might be able to use gff3ToGenePred followed by genePredToGtf to make the conversion.

-- You are receiving this because you were mentioned. Reply to this email directly or view it on GitHub: https://github.com/ComparativeGenomicsToolkit/Comparative-Annotation-Toolkit/issues/163#issuecomment-625614172 I have not seen that error in STAR before; maybe newer versions require annotations? I haven't used STAR in a while. I suppose you could use the CAT annotation file, I am not sure if STAR supports GFF3 or not. If it doesn't, you might be able to use gff3ToGenePred followed by genePredToGtf to make the conversion.

YPGG1234 commented 4 years ago

@diekhans May I ask which version of STAR is you used? @ifiddes There are no spaces in my input fasta file headers. My STAR version is 2.7.3a

diekhans commented 4 years ago

I also have 2.7.3a.

YPGG1234 notifications@github.com writes:

I can use start without annotations using this script: https://github.com/diekhans/T2T_cenSat_hub/blob/master/bin/runEncodeAlign stolen mostly from the EnCODE pipeline Ian Fiddes notifications@github.com writes: I have not seen that error in STAR before; maybe newer versions require annotations? I haven't used STAR in a while. I suppose you could use the CAT annotation file, I am not sure if STAR supports GFF3 or not. If it doesn't, you might be able to use gff3ToGenePred followed by genePredToGtf to make the conversion. -- You are receiving this because you were mentioned. Reply to this email directly or view it on GitHub: #163 (comment) I have not seen that error in STAR before; maybe newer versions require annotations? I haven't used STAR in a while. I suppose you could use the CAT annotation file, I am not sure if STAR supports GFF3 or not. If it doesn't, you might be able to use gff3ToGenePred followed by genePredToGtf to make the conversion.

May I ask which version of STAR is you used? My STAR version is 2.7.3a

-- You are receiving this because you were mentioned. Reply to this email directly or view it on GitHub: https://github.com/ComparativeGenomicsToolkit/Comparative-Annotation-Toolkit/issues/163#issuecomment-625684138 I can use start without annotations using this script: https://github.com/ diekhans/T2T_cenSat_hub/blob/master/bin/runEncodeAlign stolen mostly from the EnCODE pipeline Ian Fiddes notifications@github.com writes: … I have not seen that error in STAR before; maybe newer versions require annotations? I haven't used STAR in a while. I suppose you could use the CAT annotation file, I am not sure if STAR supports GFF3 or not. If it doesn't, you might be able to use gff3ToGenePred followed by genePredToGtf to make the conversion. -- You are receiving this because you were mentioned. Reply to this email directly or view it on GitHub: #163 (comment) I have not seen that error in STAR before; maybe newer versions require annotations? I haven't used STAR in a while. I suppose you could use the CAT annotation file, I am not sure if STAR supports GFF3 or not. If it doesn't, you might be able to use gff3ToGenePred followed by genePredToGtf to make the conversion.

May I ask which version of STAR is you used? My STAR version is 2.7.3a

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or unsubscribe.*

ifiddes commented 4 years ago

I would look at your STAR command; that link above says that having extra spaces will lead to this error

YPGG1234 commented 4 years ago

Thanks for your help, after I remove the parameters 'sjdbGTFfile' and 'sjdbGTFfile', it works well, and the command line looks like:

STAR --runThreadN 20 --runMode genomeGenerate \
--genomeDir ./STAR_index \
--genomeFastaFiles inputfasta

and

STAR \
    --genomeDir ./STAR_index \
     --runThreadN 20 \
    --readFilesIn ${Fq1} ${Fq2} \
     --outFileNamePrefix ${Sample} \
    --outSAMtype BAM SortedByCoordinate \
    --outBAMsortingThreadN 10
YPGG1234 commented 4 years ago

Cause I want to report a genome with annotation which can be published within a paper, follow your recommand, I run the command: luigi --module cat RunCat --hal=../../input/01.C-A-T-Hal/Oar_DaiGoat_ARS.hal --target-genomes='("DairyGoat",)' --ref-genome=ARS --local-scheduler --workers=20 --config=ARS1.cfg --work-dir ARS_cat --out-dir ARS_cat --augustus-pb --augustus --binary-mode local

But it reported errors when it came to Augustus:

ERROR: [pid 39142] Worker Worker(salt=767751362, workers=20, host=subnet190-2.nwsuaf.edu.cn, username=liran, pid=41106) failed    ToilTask: AugustusPbDriverTask for DairyGoat using batchSys
Traceback (most recent call last):
  File "/home/liran/Software/miniconda2/envs/C-A-T/lib/python2.7/site-packages/luigi/worker.py", line 199, in run
    new_deps = self._run_get_new_deps()
  File "/home/liran/Software/miniconda2/envs/C-A-T/lib/python2.7/site-packages/luigi/worker.py", line 141, in _run_get_new_deps
    task_gen = self.task.run()
  File "/home/liran/Software/C-A-T/Comparative-Annotation-Toolkit/cat/__init__.py", line 1528, in run
    augustus_pb(augustus_pb_args, toil_options)
  File "/home/liran/Software/C-A-T/Comparative-Annotation-Toolkit/cat/augustus_pb.py", line 40, in augustus_pb
    input_file_ids.pb_cfg = FileID.forPath(t.importFile('file://' + args.pb_cfg), args.pb_cfg)
  File "/home/liran/Software/miniconda2/envs/C-A-T/lib/python2.7/site-packages/toil/common.py", line 900, in importFile
    return self._jobStore.importFile(srcUrl, sharedFileName=sharedFileName)
  File "/home/liran/Software/miniconda2/envs/C-A-T/lib/python2.7/site-packages/toil/jobStores/abstractJobStore.py", line 283, in importFile
    return self._importFile(otherCls, srcUrl, sharedFileName=sharedFileName)
  File "/home/liran/Software/miniconda2/envs/C-A-T/lib/python2.7/site-packages/toil/jobStores/fileJobStore.py", line 168, in _importFile
    shutil.copyfile(self._extractPathFromUrl(url), absPath)
  File "/home/liran/Software/miniconda2/envs/C-A-T/lib/python2.7/shutil.py", line 96, in copyfile
    with open(src, 'rb') as fsrc:
IOError: [Errno 2] No such file or directory: '/home/liran/GoatAnno20200502/output/TransCAT/augustus_cfgs/extrinsic.M.RM.PB.E.W.cfg'
subnet190-2.nwsuaf.edu.cn 2020-05-13 04:48:35,888 MainThread ERROR luigi-interface: [pid 39142] Worker Worker(salt=767751362, workers=20, host=subnet190-2.nwsuaf.edu.cn, username=liran, pid
Traceback (most recent call last):
  File "/home/liran/Software/miniconda2/envs/C-A-T/lib/python2.7/site-packages/luigi/worker.py", line 199, in run
    new_deps = self._run_get_new_deps()
  File "/home/liran/Software/miniconda2/envs/C-A-T/lib/python2.7/site-packages/luigi/worker.py", line 141, in _run_get_new_deps
    task_gen = self.task.run()
  File "/home/liran/Software/C-A-T/Comparative-Annotation-Toolkit/cat/__init__.py", line 1528, in run
    augustus_pb(augustus_pb_args, toil_options)
  File "/home/liran/Software/C-A-T/Comparative-Annotation-Toolkit/cat/augustus_pb.py", line 40, in augustus_pb
    input_file_ids.pb_cfg = FileID.forPath(t.importFile('file://' + args.pb_cfg), args.pb_cfg)
  File "/home/liran/Software/miniconda2/envs/C-A-T/lib/python2.7/site-packages/toil/common.py", line 900, in importFile
    return self._jobStore.importFile(srcUrl, sharedFileName=sharedFileName)
  File "/home/liran/Software/miniconda2/envs/C-A-T/lib/python2.7/site-packages/toil/jobStores/abstractJobStore.py", line 283, in importFile
    return self._importFile(otherCls, srcUrl, sharedFileName=sharedFileName)
  File "/home/liran/Software/miniconda2/envs/C-A-T/lib/python2.7/site-packages/toil/jobStores/fileJobStore.py", line 168, in _importFile
    shutil.copyfile(self._extractPathFromUrl(url), absPath)
  File "/home/liran/Software/miniconda2/envs/C-A-T/lib/python2.7/shutil.py", line 96, in copyfile
    with open(src, 'rb') as fsrc:
IOError: [Errno 2] No such file or directory: '/home/liran/GoatAnno20200502/output/TransCAT/augustus_cfgs/extrinsic.M.RM.PB.E.W.cfg'
INFO: Informed scheduler that task   AugustusPbDriverTask_False_False_True_682ae8a5a6   has status   FAILED

But I can work right with the test data buildin CAT, can you help me find what's wrong?

ifiddes commented 4 years ago

Are you running CAT from the installation directory? If you aren't running it from the directory, you need to use the flag --pb-cfg to point it to the path for that file ($catDir/augustus_cfgs/extrinsic.M.RM.PB.E.W.cfg). The same is true for CGP -- if you are running --augustus-cgp, you need to set --augustus-cgp-cfg-template to $catDir/augustus_cfgs/cgp_extrinsic_template.cfg.

YPGG1234 commented 4 years ago

Thanks for your reply, I had resubmitted the command.

------------------ 原始邮件 ------------------ 发件人: "Ian Fiddes"<notifications@github.com>; 发送时间: 2020年5月14日(星期四) 凌晨4:48 收件人: "ComparativeGenomicsToolkit/Comparative-Annotation-Toolkit"<Comparative-Annotation-Toolkit@noreply.github.com>; 抄送: "杨鹏"<872713998@qq.com>;"Author"<author@noreply.github.com>; 主题: Re: [ComparativeGenomicsToolkit/Comparative-Annotation-Toolkit] Error with luigi (#163)

Are you running CAT from the installation directory? If you aren't running it from the directory, you need to use the flag --pb-cfg to point it to the path for that file ($catDir/augustus_cfgs/extrinsic.M.RM.PB.E.W.cfg). The same is true for CGP -- if you are running --augustus-cgp, you need to set --augustus-cgp-cfg-template to $catDir/augustus_cfgs/cgp_extrinsic_template.cfg.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub, or unsubscribe.