[ERROR] Task Contig01634:ErrorCounter Failed (__exit__)

[zzlyk]

2018-03-15 17:29:20,968 [INFO] Starting Contig01634:ErrorCounter 2018-03-15 17:29:20,984 [INFO] 124 reads to parse in Contig01634 2018-03-15 17:29:20,989 [INFO] 5% complete 2018-03-15 17:29:20,994 [INFO] 10% complete 2018-03-15 17:29:20,999 [INFO] 15% complete 2018-03-15 17:29:21,005 [INFO] 23% complete 2018-03-15 17:29:21,005 [INFO] 25% complete 2018-03-15 17:29:21,014 [INFO] 30% complete 2018-03-15 17:29:21,018 [INFO] 37% complete 2018-03-15 17:29:21,021 [INFO] 44% complete 2018-03-15 17:29:21,024 [INFO] 48% complete 2018-03-15 17:29:21,025 [INFO] 50% complete 2018-03-15 17:29:21,028 [INFO] 56% complete 2018-03-15 17:29:21,029 [INFO] 60% complete 2018-03-15 17:29:21,036 [INFO] 65% complete 2018-03-15 17:29:21,044 [INFO] 70% complete 2018-03-15 17:29:21,050 [INFO] 75% complete 2018-03-15 17:29:21,052 [INFO] 81% complete 2018-03-15 17:29:21,053 [INFO] 85% complete 2018-03-15 17:29:21,055 [INFO] 94% complete 2018-03-15 17:29:21,056 [INFO] 96% complete 2018-03-15 17:29:21,056 [ERROR] Exception raised in task Contig01634:ErrorCounter - exit 2018-03-15 17:29:21,056 [ERROR] Dumping Traceback: Traceback (most recent call last): File "/share/nas1/xiek/pipeline/svn/SVcalling/nextsv/aligners_and_callers/PBSuite_15.8.24/pbsuite/honey/HSpots.py", line 500, in run next_task(self.bam, self.reference, self.honH5) File "/share/nas1/xiek/pipeline/svn/SVcalling/nextsv/aligners_and_callers/PBSuite_15.8.24/pbsuite/honey/HSpots.py", line 597, in call with honH5.acquireH5('a') as h5dat: File "/share/nas2/genome/biosoft/Python/2.7.8/lib/python2.7/contextlib.py", line 17, in enter return self.gen.next() File "/share/nas1/xiek/pipeline/svn/SVcalling/nextsv/aligners_and_callers/PBSuite_15.8.24/pbsuite/honey/HSpots.py", line 439, in acquireH5 with self.lock: AttributeError: exit 2018-03-15 17:29:21,059 [ERROR] Task Contig01634:ErrorCounter Failed (exit__) 2018-03-15 17:29:21,059 [INFO] Finished 2018-03-15 17:29:21,059 [INFO] Thread Consumer-1: Exiting

please help me~

[fangli80]

Hi, Can you please upload the config file? It seems that this is an error of PBHoney, but I will try to figure it out.

Best, Li

[zzlyk]

## Input/Output settings ##

# sample name (required)
sample_name=mutant

#  path to a file that contains names of input files. Each line contains one input file. Examples of input file list can be found in example.fastq.fofn (required)
input_file_list=/workdir/AAA/Reseq_project/Building-Database-Sequencin/AAA170707-G016-S0101_Rosasp_Reseq/sandai/data.info

# full path to output directory (required)
out_dir=/workdir/AAA/Reseq_project/Building-Database-Sequencin/AAA170707-G016-S0101_Rosasp_Reseq/sandai/analysis

## cluster settings ##
maxproc=80
queue=middle.q
# number of threads (CPU cores) (default: 1)
n_thread=5

# running mode of NextSV. NextSV currently supports two modes: multiprocessing or sge (required)
mode=multiprocessing

# job submission command with parameters Please remember to specify the resources (e.g. memory and CPU cores) (required if mode=sge)
job_submission_command=[qsub -V -cwd -S /bin/bash -l h_vmem=4G -pe smp 1]

## PBHoney settings ##

# whether to enable SV calling using PBHoney-Spots (1 for enable, 0 for disable, required)
enable_PBHoney_Spots=1  

# threshold parameter of PBHoney-Spots (default: 2)
spots_threshold=2      

# minimal supporting reads for PBHoney-Spots (default: 2)
spots_minErrReads=2   

# method for polishing consensus for PBHoney-Spots(pbdagcon or None, default: None)
spots_consensus=None 

# whether to enable SV calling using PBHoney-Tails (1 for enable, 0 for disable, required)
enable_PBHoney_Tails=1 

# minimum supporting reads for PBHoney-Tails (default: 2)
tails_minBreads=2     

# minimum number of unique ZMWs for PBHoney-Tails (default: 2)
tails_minZMWs=2      

# buffer parameter of PBHoney-Tails(default: 600)
tails_buffer=600    

## Sniffles settings ##

# whether to enable SV calling using the combination of BWA and Sniffles (1 for enable, 0 for disable, required)
enable_bwa_Sniffles=1  

# whether to enable SV calling using the combination of NGMLR and Sniffles (1 for enable, 0 for disable, required)
enable_ngmlr_Sniffles=1 

# minimum supporting reads for Sniffles (default: 2)
sniffles_min_support=2 

# maximum distance to group SV together by Sniffles (default: 600)
sniffles_max_distance=600 

## bash5toos settings (settings for extracting fastq from hdf files) ##

# minimal subread length (default: 500)
minLength=500

# minReadScore (default: 0.75)
minReadScore=0.75 

## paths to software tools ##

# full path to bwa binary file (required)
bwa=/share/nas2/genome/bin/bwa 

# full path to samtools binary file (required)
samtools=/share/nas2/genome/biosoft/samtools/1.3.1/samtools 

# full path to ngmlr binary file (not required, default: path_to_nextsv/bin/ngmlr)
ngmlr=/workdir/AAA/pipeline/svn/SVcalling/nextsv/bin/ngmlr

# full path to sniffles binary file (not required if you have installed it by ran install_sniffles.sh)
sniffles=/workdir/AAA/pipeline/svn/SVcalling/nextsv/bin/sniffles

# full path to installed bash5tools.py on your system, required if your input files are in hdf5 format. This is for extracting fastq from hdf5 files.
bash5tools=/share/nas2/genome/biosoft/Python/2.7.8/bin/bash5tools.py 

## paths to reference fasta files ##

# full path to reference genome fasta file (for BWA and NGMLR aligners. If enable_bwa_Sniffles=1 is specified, the fasta file should be pre-indexed by BWA. If enable_bwa_Sniffles=0 is specified, the fasta file does not need to be pre-indexed) 
ref_fasta=/workdir/AAA/Reseq_project/Building-Database-Sequencin/AAA170707-G016-S0101_Rosasp_Reseq/sandai/database/bwa_ngmlr/final.genome.fasta 

# full path to reference genome fasta file (for blasr aligner)
ref_blasr=/workdir/AAA/Reseq_project/Building-Database-Sequencin/AAA170707-G016-S0101_Rosasp_Reseq/sandai/database/blasr/final.genome.fasta

# full path to precomputed suffix array of reference genome fasta file for blasr aligner (for information of generating the suffix array, please refer to https://github.com/PacificBiosciences/blasr/blob/master/README.MANUAL.md)
ref_sa_blasr=/workdir/AAA/Reseq_project/Building-Database-Sequencin/AAA170707-G016-S0101_Rosasp_Reseq/sandai/database/blasr/final.genome.fasta.sa

sorry for yout long time wait

[fangli80]

Sorry for the late reply. You added two new parameters in the config file (shown below). However, these parameters are not supported by NextSV. So please remove it.

maxproc=80
queue=middle.q

If you are running in "sge" mode and want to specify the queue, you can modify the job_submission_command parameter. For example,

job_submission_command=[qsub -V -cwd -S /bin/bash -l h_vmem=4G -pe smp 80 -q middle.q]

Since you were actually running in "multiprocessing" mode, the cluster settings could not affect your run and shouldn't be the reason for your error.

The other part of the config file is correct.

The error happens when running PBHoney-Spots to detect SVs from the blasr generated bam. Please check if the blasr generated bam is good. The bam file should in the /workdir/AAA/Reseq_project/Building-Database-Sequencin/AAA170707-G016-S0101_Rosasp_Reseq/sandai/analysis/blasr_pbhoney/2_merge_bam folder. You can use the following command: samtools quickcheck -vvv blasr.mutant.merge.bam. If the bam file is good, you will see something like this:

verbosity set to 3
checking blasr.mutant.merge.bam
opened blasr.mutant.merge.bam
blasr.mutant.merge.bam is sequence data
blasr.mutant.merge.bam has xxx targets in header.
blasr.mutant.merge.bam has good EOF block.

If the bam file is good, you can run the shell script manually to see if the error happens again. The shell script for PBHoney-Spots is located in the /workdir/AAA/Reseq_project/Building-Database-Sequencin/AAA170707-G016-S0101_Rosasp_Reseq/sandai/analysis/blasr_pbhoney/sh/ folder and the file name is probably blasr.mutant.spots.sh.