I'm running hal2assemblyHub.py from my own jobTree script. I'm feeding it some --bedDirs, some of which contain only a single empty bed file. I suspect that this is confusing the field counting logic, resulting in the invalid Unix cut command lines that crash the script, as depicted below.
Reporting file: /cluster/home/anovak/hive/sgdev/mhc/tree/jobs/t3/t1/log.txt
log.txt: ---JOBTREE SLAVE OUTPUT LOG---
log.txt: Calling: hal2assemblyHub.py /cluster/home/anovak/hive/sgdev/mhc/tree/jobs/gTD0/tmp_CBamh4j0oW/tmp_p5wcMursyJhal out2/hubs/InexactCredit/mhc1-mhc2 --jobTree /scratch/tmp/tmp3qpeB4/localTempDir/tmp_Vuh6NdEV09 --bedDirs /cluster/home/anovak/hive/sgdev/mhc/tree/jobs/gTD0/tmp_CBamh4j0oW/tmp_cx0gnVCE0y/InexactCredit-ExactCredit-mhc1-mhc2,/cluster/home/anovak/hive/sgdev/mhc/tree/jobs/gTD0/tmp_CBamh4j0oW/tmp_cx0gnVCE0y/InexactCredit-Inexact-mhc1-mhc2,/cluster/home/anovak/hive/sgdev/mhc/tree/jobs/gTD0/tmp_CBamh4j0oW/tmp_cx0gnVCE0y/Exact-InexactCredit-mhc1-mhc2 --shortLabel InexactCredit --lod --cpHalFileToOut --noUcscNames
log.txt: The job seems to have left a log file, indicating failure: /scratch/tmp/tmp3qpeB4/localTempDir/tmp_Vuh6NdEV09/jobs/t0/t1/job
log.txt: Reporting file: /scratch/tmp/tmp3qpeB4/localTempDir/tmp_Vuh6NdEV09/jobs/t0/t1/log.txt
log.txt: log.txt: ---JOBTREE SLAVE OUTPUT LOG---
log.txt: log.txt: cut: invalid byte, character or field list
log.txt: log.txt: Try `cut --help' for more information.
log.txt: log.txt: Traceback (most recent call last):
log.txt: log.txt: File "/cluster/home/anovak/.local/lib/python2.7/site-packages/jobTree-1.0-py2.7.egg/jobTree/src/jobTreeSlave.py", line 271, in main
log.txt: log.txt: defaultMemory=defaultMemory, defaultCpu=defaultCpu, depth=depth)
log.txt: log.txt: File "/cluster/home/anovak/.local/lib/python2.7/site-packages/jobTree-1.0-py2.7.egg/jobTree/scriptTree/stack.py", line 153, in execute
log.txt: log.txt: self.target.run()
log.txt: log.txt: File "/hive/users/anovak/build/progressiveCactus/submodules/hal/assemblyHub/bedTrack.py", line 57, in run
log.txt: log.txt: system( "cat %s/*bed | cut -f-%d > %s" %(genomeindir, numfield, tempbed) )
log.txt: log.txt: File "/cluster/home/anovak/.local/lib/python2.7/site-packages/sonLib-1.0-py2.7.egg/sonLib/bioio.py", line 184, in system
log.txt: log.txt: raise RuntimeError("Command: %s exited with non-zero status %i" % (command, sts))
log.txt: log.txt: RuntimeError: Command: cat /cluster/home/anovak/hive/sgdev/mhc/tree/jobs/gTD0/tmp_CBamh4j0oW/tmp_cx0gnVCE0y/InexactCredit-Inexact-mhc1-mhc2/mhc1/*bed | cut -f-0 > out2/hubs/InexactCredit/mhc1-mhc2/liftoverbed/InexactCredit-Inexact-mhc1-mhc2/mhc1/mhc1-temp.bed exited with non-zero status 1
log.txt: log.txt: Exiting the slave because of a failed job on host kolossus
log.txt: log.txt: Due to failure we are reducing the remaining retry count of job /scratch/tmp/tmp3qpeB4/localTempDir/tmp_Vuh6NdEV09/jobs/t0/t1/job to 0
log.txt: log.txt: We have set the default memory of the failed job to 2147483648 bytes
log.txt: Job: /scratch/tmp/tmp3qpeB4/localTempDir/tmp_Vuh6NdEV09/jobs/t0/t1/job is completely failed
log.txt: The job seems to have left a log file, indicating failure: /scratch/tmp/tmp3qpeB4/localTempDir/tmp_Vuh6NdEV09/jobs/t0/t2/job
log.txt: Reporting file: /scratch/tmp/tmp3qpeB4/localTempDir/tmp_Vuh6NdEV09/jobs/t0/t2/log.txt
log.txt: log.txt: ---JOBTREE SLAVE OUTPUT LOG---
log.txt: log.txt: cut: invalid byte, character or field list
log.txt: log.txt: Try `cut --help' for more information.
log.txt: log.txt: Traceback (most recent call last):
log.txt: log.txt: File "/cluster/home/anovak/.local/lib/python2.7/site-packages/jobTree-1.0-py2.7.egg/jobTree/src/jobTreeSlave.py", line 271, in main
log.txt: log.txt: defaultMemory=defaultMemory, defaultCpu=defaultCpu, depth=depth)
log.txt: log.txt: File "/cluster/home/anovak/.local/lib/python2.7/site-packages/jobTree-1.0-py2.7.egg/jobTree/scriptTree/stack.py", line 153, in execute
log.txt: log.txt: self.target.run()
log.txt: log.txt: File "/hive/users/anovak/build/progressiveCactus/submodules/hal/assemblyHub/bedTrack.py", line 57, in run
log.txt: log.txt: system( "cat %s/*bed | cut -f-%d > %s" %(genomeindir, numfield, tempbed) )
log.txt: log.txt: File "/cluster/home/anovak/.local/lib/python2.7/site-packages/sonLib-1.0-py2.7.egg/sonLib/bioio.py", line 184, in system
log.txt: log.txt: raise RuntimeError("Command: %s exited with non-zero status %i" % (command, sts))
log.txt: log.txt: RuntimeError: Command: cat /cluster/home/anovak/hive/sgdev/mhc/tree/jobs/gTD0/tmp_CBamh4j0oW/tmp_cx0gnVCE0y/Exact-InexactCredit-mhc1-mhc2/mhc1/*bed | cut -f-0 > out2/hubs/InexactCredit/mhc1-mhc2/liftoverbed/Exact-InexactCredit-mhc1-mhc2/mhc1/mhc1-temp.bed exited with non-zero status 1
log.txt: log.txt: Exiting the slave because of a failed job on host kolossus
log.txt: log.txt: Due to failure we are reducing the remaining retry count of job /scratch/tmp/tmp3qpeB4/localTempDir/tmp_Vuh6NdEV09/jobs/t0/t2/job to 0
log.txt: log.txt: We have set the default memory of the failed job to 2147483648 bytes
log.txt: Job: /scratch/tmp/tmp3qpeB4/localTempDir/tmp_Vuh6NdEV09/jobs/t0/t2/job is completely failed
log.txt: The job seems to have left a log file, indicating failure: /scratch/tmp/tmp3qpeB4/localTempDir/tmp_Vuh6NdEV09/jobs/t0/t0/job
log.txt: Reporting file: /scratch/tmp/tmp3qpeB4/localTempDir/tmp_Vuh6NdEV09/jobs/t0/t0/log.txt
log.txt: log.txt: ---JOBTREE SLAVE OUTPUT LOG---
log.txt: log.txt: cut: invalid byte, character or field list
log.txt: log.txt: Try `cut --help' for more information.
log.txt: log.txt: Traceback (most recent call last):
log.txt: log.txt: File "/cluster/home/anovak/.local/lib/python2.7/site-packages/jobTree-1.0-py2.7.egg/jobTree/src/jobTreeSlave.py", line 271, in main
log.txt: log.txt: defaultMemory=defaultMemory, defaultCpu=defaultCpu, depth=depth)
log.txt: log.txt: File "/cluster/home/anovak/.local/lib/python2.7/site-packages/jobTree-1.0-py2.7.egg/jobTree/scriptTree/stack.py", line 153, in execute
log.txt: log.txt: self.target.run()
log.txt: log.txt: File "/hive/users/anovak/build/progressiveCactus/submodules/hal/assemblyHub/bedTrack.py", line 57, in run
log.txt: log.txt: system( "cat %s/*bed | cut -f-%d > %s" %(genomeindir, numfield, tempbed) )
log.txt: log.txt: File "/cluster/home/anovak/.local/lib/python2.7/site-packages/sonLib-1.0-py2.7.egg/sonLib/bioio.py", line 184, in system
log.txt: log.txt: raise RuntimeError("Command: %s exited with non-zero status %i" % (command, sts))
log.txt: log.txt: RuntimeError: Command: cat /cluster/home/anovak/hive/sgdev/mhc/tree/jobs/gTD0/tmp_CBamh4j0oW/tmp_cx0gnVCE0y/InexactCredit-ExactCredit-mhc1-mhc2/mhc1/*bed | cut -f-0 > out2/hubs/InexactCredit/mhc1-mhc2/liftoverbed/InexactCredit-ExactCredit-mhc1-mhc2/mhc1/mhc1-temp.bed exited with non-zero status 1
log.txt: log.txt: Exiting the slave because of a failed job on host kolossus
log.txt: log.txt: Due to failure we are reducing the remaining retry count of job /scratch/tmp/tmp3qpeB4/localTempDir/tmp_Vuh6NdEV09/jobs/t0/t0/job to 0
log.txt: log.txt: We have set the default memory of the failed job to 2147483648 bytes
joelarmstrong
commented
10 years ago
I'm running hal2assemblyHub.py from my own jobTree script. I'm feeding it some
--bedDirs, some of which contain only a single empty bed file. I suspect that this is confusing the field counting logic, resulting in the invalid Unixcutcommand lines that crash the script, as depicted below.