Issues and question NCLscan_v1.4

[vincenthahaut]

Hello!

I started to try to use NCLscan a few days ago but I encountered several issues. I hope you will be able to help me to solve them.

QUESTION:

1. Software requirements:

You said: "These executable files of bedtools, samtools, blat, bwa, novoalign, novoindex, and novoalign.lic should be copied to the file directory of NCLscan". By this do you mean that I have to only insert in the NCLscan_v1.4 folder the files bwa-0.7.12/bwa, bedtools2-master/bedtools,... or the full content of the bwa-0.7.12/* , bedtools2-master/*,...?

ISSUES: 3.3. BWA index preparation:

After downloading the data from genecodeServer inside the ./NCLscan_v1.4 folder I tried to generate the indexes using ./MakeBwaRef. Unfortunately I got this error:

..../NCLscan-master/NCLscan_v1.4$ ./MakeBwaRef -bash: ./MakeBwaRef: Permission denied

with this permission [-rw-r--r--] on the following files: ChangeLog, MakeBwaRef, NCL_Scan5.py, README, config.txt, novoalign.lic, utils.py, version.

I tried to solve it by using chmod 755 on MakeBwaRef then started again ./MakeBwaRef wich produced:

.../NCLscan_v1.4$ ./MakeBwaRef sh: 1: ./SeqOut: not found [bwa_index] Pack FASTA... 40.83 sec [bwa_index] Construct BWT for the packed sequence... [BWTIncCreate] textLength=6908410432, availableWord=498100904 [BWTIncConstructFromPacked] 10 iterations done. 100000000 characters processed. ... [BWTIncConstructFromPacked] 740 iterations done. 6867165504 characters processed. [BWTIncConstructFromPacked] 750 iterations done. 6890986976 characters processed. [bwt_gen] Finished constructing BWT in 759 iterations. [bwa_index] 4586.34 seconds elapse. [bwa_index] Update BWT... 26.99 sec [bwa_index] Pack forward-only FASTA... 23.93 sec [bwa_index] Construct SA from BWT and Occ... 1382.08 sec [main] Version: 0.7.12-r1039 [main] CMD: ./bwa index bwa_AllRef.fa [main] Real time: 6180.325 sec; CPU: 6060.187 sec

and a full index of the reference genome.

1. Execution of NCLscan:

Finally I tried to run the example as described.

./NCLscan.sh ./test_NCLscan/simu_5X_100PE_1.fastq ./test_NCLscan/simu_5X_100PE_2.fastq test_NCLscan 0 8 50 2>&1 | tee test_NCLscan/test_NCLscan.log

========== Start =========== Tue Oct 27 14:33:45 CET 2015 [M::bwa_idx_load_from_disk] read 0 ALT contigs [M::process] read 800000 sequences (80000000 bp)... [M::process] read 800000 sequences (80000000 bp)... [M::mem_pestat] # candidate unique pairs for (FF, FR, RF, RR): (0, 1166, 0, 0) [M::mem_pestat] skip orientation FF as there are not enough pairs [M::mem_pestat] analyzing insert size distribution for orientation FR... [M::mem_pestat](25, 50, 75) percentile: (320, 365, 414) [M::mem_pestat] low and high boundaries for computing mean and std.dev: (132, 602) [M::mem_pestat] mean and std.dev: (365.85, 67.80) [M::mem_pestat] low and high boundaries for proper pairs: (38, 696) [M::mem_pestat] skip orientation RF as there are not enough pairs [M::mem_pestat] skip orientation RR as there are not enough pairs [M::mem_process_seqs] Processed 800000 reads in 305.387 CPU sec, 38.199 real sec [M::process] read 800000 sequences (80000000 bp)... [M::mem_pestat] # candidate unique pairs for (FF, FR, RF, RR): (0, 1249, 0, 0) [M::mem_pestat] skip orientation FF as there are not enough pairs [M::mem_pestat] analyzing insert size distribution for orientation FR... [M::mem_pestat](25, 50, 75) percentile: (318, 366, 410) [M::mem_pestat] low and high boundaries for computing mean and std.dev: (134, 594) [M::mem_pestat] mean and std.dev: (364.18, 67.23) [M::mem_pestat] low and high boundaries for proper pairs: (42, 686) [M::mem_pestat] skip orientation RF as there are not enough pairs [M::mem_pestat] skip orientation RR as there are not enough pairs [M::mem_process_seqs] Processed 800000 reads in 302.903 CPU sec, 37.711 real sec [M::process] read 800000 sequences (80000000 bp)... [M::mem_pestat] # candidate unique pairs for (FF, FR, RF, RR): (0, 1223, 0, 0) [M::mem_pestat] skip orientation FF as there are not enough pairs [M::mem_pestat] analyzing insert size distribution for orientation FR... [M::mem_pestat](25, 50, 75) percentile: (320, 364, 410) [M::mem_pestat] low and high boundaries for computing mean and std.dev: (140, 590) [M::mem_pestat] mean and std.dev: (364.40, 66.86) [M::mem_pestat] low and high boundaries for proper pairs: (50, 680) [M::mem_pestat] skip orientation RF as there are not enough pairs [M::mem_pestat] skip orientation RR as there are not enough pairs [M::mem_process_seqs] Processed 800000 reads in 302.887 CPU sec, 37.693 real sec [M::process] read 800000 sequences (80000000 bp)... [M::mem_pestat] # candidate unique pairs for (FF, FR, RF, RR): (0, 1234, 0, 0) [M::mem_pestat] skip orientation FF as there are not enough pairs [M::mem_pestat] analyzing insert size distribution for orientation FR... [M::mem_pestat](25, 50, 75) percentile: (325, 367, 413) [M::mem_pestat] low and high boundaries for computing mean and std.dev: (149, 589) [M::mem_pestat] mean and std.dev: (367.27, 66.28) [M::mem_pestat] low and high boundaries for proper pairs: (61, 677) [M::mem_pestat] skip orientation RF as there are not enough pairs [M::mem_pestat] skip orientation RR as there are not enough pairs [M::mem_process_seqs] Processed 800000 reads in 302.251 CPU sec, 37.578 real sec [M::process] read 800000 sequences (80000000 bp)... [M::mem_pestat] # candidate unique pairs for (FF, FR, RF, RR): (0, 1225, 0, 0) [M::mem_pestat] skip orientation FF as there are not enough pairs [M::mem_pestat] analyzing insert size distribution for orientation FR... [M::mem_pestat](25, 50, 75) percentile: (318, 366, 410) [M::mem_pestat] low and high boundaries for computing mean and std.dev: (134, 594) [M::mem_pestat] mean and std.dev: (365.77, 66.80) [M::mem_pestat] low and high boundaries for proper pairs: (42, 686) [M::mem_pestat] skip orientation RF as there are not enough pairs [M::mem_pestat] skip orientation RR as there are not enough pairs [M::mem_process_seqs] Processed 800000 reads in 302.563 CPU sec, 37.662 real sec [M::process] read 800000 sequences (80000000 bp)... [M::mem_pestat] # candidate unique pairs for (FF, FR, RF, RR): (0, 1549, 0, 0) [M::mem_pestat] skip orientation FF as there are not enough pairs [M::mem_pestat] analyzing insert size distribution for orientation FR... [M::mem_pestat](25, 50, 75) percentile: (324, 369, 412) [M::mem_pestat] low and high boundaries for computing mean and std.dev: (148, 588) [M::mem_pestat] mean and std.dev: (367.77, 65.12) [M::mem_pestat] low and high boundaries for proper pairs: (60, 676) [M::mem_pestat] skip orientation RF as there are not enough pairs [M::mem_pestat] skip orientation RR as there are not enough pairs [M::mem_process_seqs] Processed 800000 reads in 302.791 CPU sec, 37.679 real sec [M::process] read 800000 sequences (80000000 bp)... [M::mem_pestat] # candidate unique pairs for (FF, FR, RF, RR): (0, 1849, 0, 0) [M::mem_pestat] skip orientation FF as there are not enough pairs [M::mem_pestat] analyzing insert size distribution for orientation FR... [M::mem_pestat](25, 50, 75) percentile: (322, 368, 414) [M::mem_pestat] low and high boundaries for computing mean and std.dev: (138, 598) [M::mem_pestat] mean and std.dev: (367.91, 67.05) [M::mem_pestat] low and high boundaries for proper pairs: (46, 690) [M::mem_pestat] skip orientation RF as there are not enough pairs [M::mem_pestat] skip orientation RR as there are not enough pairs [M::mem_process_seqs] Processed 800000 reads in 293.790 CPU sec, 36.529 real sec [M::process] read 800000 sequences (80000000 bp)... [M::mem_pestat] # candidate unique pairs for (FF, FR, RF, RR): (0, 1854, 0, 0) [M::mem_pestat] skip orientation FF as there are not enough pairs [M::mem_pestat] analyzing insert size distribution for orientation FR... [M::mem_pestat](25, 50, 75) percentile: (322, 370, 413) [M::mem_pestat] low and high boundaries for computing mean and std.dev: (140, 595) [M::mem_pestat] mean and std.dev: (367.74, 65.89) [M::mem_pestat] low and high boundaries for proper pairs: (49, 686) [M::mem_pestat] skip orientation RF as there are not enough pairs [M::mem_pestat] skip orientation RR as there are not enough pairs [M::mem_process_seqs] Processed 800000 reads in 294.418 CPU sec, 36.621 real sec [M::process] read 800000 sequences (80000000 bp)... [M::mem_pestat] # candidate unique pairs for (FF, FR, RF, RR): (0, 1932, 0, 0) [M::mem_pestat] skip orientation FF as there are not enough pairs [M::mem_pestat] analyzing insert size distribution for orientation FR... [M::mem_pestat](25, 50, 75) percentile: (324, 369, 412) [M::mem_pestat] low and high boundaries for computing mean and std.dev: (148, 588) [M::mem_pestat] mean and std.dev: (368.05, 67.10) [M::mem_pestat] low and high boundaries for proper pairs: (60, 676) [M::mem_pestat] skip orientation RF as there are not enough pairs [M::mem_pestat] skip orientation RR as there are not enough pairs [M::mem_process_seqs] Processed 800000 reads in 290.126 CPU sec, 36.130 real sec [M::process] read 800000 sequences (80000000 bp)... [M::mem_pestat] # candidate unique pairs for (FF, FR, RF, RR): (0, 680, 0, 0) [M::mem_pestat] skip orientation FF as there are not enough pairs [M::mem_pestat] analyzing insert size distribution for orientation FR... [M::mem_pestat](25, 50, 75) percentile: (313, 361, 407) [M::mem_pestat] low and high boundaries for computing mean and std.dev: (125, 595) [M::mem_pestat] mean and std.dev: (363.08, 68.53) [M::mem_pestat] low and high boundaries for proper pairs: (31, 689) [M::mem_pestat] skip orientation RF as there are not enough pairs [M::mem_pestat] skip orientation RR as there are not enough pairs [M::mem_process_seqs] Processed 800000 reads in 318.916 CPU sec, 39.598 real sec [M::process] read 800000 sequences (80000000 bp)... [M::mem_pestat] # candidate unique pairs for (FF, FR, RF, RR): (0, 492, 0, 0) [M::mem_pestat] skip orientation FF as there are not enough pairs [M::mem_pestat] analyzing insert size distribution for orientation FR... [M::mem_pestat](25, 50, 75) percentile: (309, 355, 397) [M::mem_pestat] low and high boundaries for computing mean and std.dev: (133, 573) [M::mem_pestat] mean and std.dev: (353.34, 65.91) [M::mem_pestat] low and high boundaries for proper pairs: (45, 661) [M::mem_pestat] skip orientation RF as there are not enough pairs [M::mem_pestat] skip orientation RR as there are not enough pairs [M::mem_process_seqs] Processed 800000 reads in 318.792 CPU sec, 39.687 real sec [M::process] read 75598 sequences (7559800 bp)... [M::mem_pestat] # candidate unique pairs for (FF, FR, RF, RR): (0, 401, 0, 0) [M::mem_pestat] skip orientation FF as there are not enough pairs [M::mem_pestat] analyzing insert size distribution for orientation FR... [M::mem_pestat](25, 50, 75) percentile: (297, 350, 400) [M::mem_pestat] low and high boundaries for computing mean and std.dev: (91, 606) [M::mem_pestat] mean and std.dev: (346.87, 70.52) [M::mem_pestat] low and high boundaries for proper pairs: (1, 709) [M::mem_pestat] skip orientation RF as there are not enough pairs [M::mem_pestat] skip orientation RR as there are not enough pairs [M::mem_process_seqs] Processed 800000 reads in 318.876 CPU sec, 39.793 real sec [M::mem_pestat] # candidate unique pairs for (FF, FR, RF, RR): (0, 34, 0, 0) [M::mem_pestat] skip orientation FF as there are not enough pairs [M::mem_pestat] analyzing insert size distribution for orientation FR... [M::mem_pestat](25, 50, 75) percentile: (301, 354, 391) [M::mem_pestat] low and high boundaries for computing mean and std.dev: (121, 571) [M::mem_pestat] mean and std.dev: (352.56, 77.28) [M::mem_pestat] low and high boundaries for proper pairs: (31, 661) [M::mem_pestat] skip orientation RF as there are not enough pairs [M::mem_pestat] skip orientation RR as there are not enough pairs [M::mem_process_seqs] Processed 75598 reads in 38.578 CPU sec, 4.600 real sec [main] Version: 0.7.12-r1039 [main] CMD: ./bwa mem -t 8 bwa_AllRef.fa ./test_NCLscan/simu_5X_100PE_1.fastq ./test_NCLscan/simu_5X_100PE_2.fastq [main] Real time: 534.858 sec; CPU: 3705.276 sec [samopen] SAM header is present: 119505 sequences. sh: 1: ./FastqOut: not found cat: write error: Broken pipe sh: 1: ./FastqOut: not found cat: write error: Broken pipe ./NCLscan.sh: line 14: ./NCL_Scan1: No such file or directory ./NCLscan.sh: line 15: ./NCL_Scan2: No such file or directory ./NCLscan.sh: line 16: ./NCL_Scan3: No such file or directory NCL_Scan4 v0.2.5 config.tmp can not be found! Please make sure that NCL_Scan2 has been finished properly. ./NCLscan.sh: line 18: ./NCL_Scan5.py: Permission denied =========== Stop ============ Tue Oct 27 14:43:08 CET 2015

Is it normal that the two files FastqOut and SeqOut that are reported as "not found" are binaries files in the NCLscan_v1.4 folder?

Thanks in advance for your answers.

Vincent

[chiangtw]

Hello!

I started to try to use NCLscan a few days ago but I encountered several issues. I hope you will be able to help me to solve them.

QUESTION:

1. Software requirements:

You said: "These executable files of bedtools, samtools, blat, bwa, novoalign, novoindex, and novoalign.lic should be copied to the file directory of NCLscan". By this do you mean that I have to only insert in the NCLscan_v1.4 folder the files bwa-0.7.12/bwa, bedtools2-master/bedtools,... or the full content of the bwa-0.7.12/* , bedtools2-master/*,...?

Only the executable files (eg. bwa-0.7.12/bwa, bedtools2-master/bedtools,...) have to be put into the NCLscan_v1.4 folder.

ISSUES: 3.3. BWA index preparation:

After downloading the data from genecodeServer inside the ./NCLscan_v1.4 folder I tried to generate the indexes using ./MakeBwaRef. Unfortunately I got this error:

..../NCLscan-master/NCLscan_v1.4$ ./MakeBwaRef -bash: ./MakeBwaRef: Permission denied

with this permission [-rw-r--r--] on the following files: ChangeLog, MakeBwaRef, NCL_Scan5.py, README, config.txt, novoalign.lic, utils.py, version.

I tried to solve it by using chmod 755 on MakeBwaRef then started again ./MakeBwaRef wich produced:

.../NCLscan_v1.4$ ./MakeBwaRef sh: 1: ./SeqOut: not found [bwa_index] Pack FASTA... 40.83 sec [bwa_index] Construct BWT for the packed sequence... [BWTIncCreate] textLength=6908410432, availableWord=498100904 [BWTIncConstructFromPacked] 10 iterations done. 100000000 characters processed. ... [BWTIncConstructFromPacked] 740 iterations done. 6867165504 characters processed. [BWTIncConstructFromPacked] 750 iterations done. 6890986976 characters processed. [bwt_gen] Finished constructing BWT in 759 iterations. [bwa_index] 4586.34 seconds elapse. [bwa_index] Update BWT... 26.99 sec [bwa_index] Pack forward-only FASTA... 23.93 sec [bwa_index] Construct SA from BWT and Occ... 1382.08 sec [main] Version: 0.7.12-r1039 [main] CMD: ./bwa index bwa_AllRef.fa [main] Real time: 6180.325 sec; CPU: 6060.187 sec

and a full index of the reference genome.

Please check the following first:

1. all the programs and reference files are in the same folder,
2. all the programs are executable.

1. Execution of NCLscan:

Finally I tried to run the example as described.

./NCLscan.sh ./test_NCLscan/simu_5X_100PE_1.fastq ./test_NCLscan/simu_5X_100PE_2.fastq test_NCLscan 0 8 50 2>&1 | tee test_NCLscan/test_NCLscan.log

========== Start =========== Tue Oct 27 14:33:45 CET 2015 [M::bwa_idx_load_from_disk] read 0 ALT contigs [M::process] read 800000 sequences (80000000 bp)... [M::process] read 800000 sequences (80000000 bp)... [M::mem_pestat] # candidate unique pairs for (FF, FR, RF, RR): (0, 1166, 0, 0) [M::mem_pestat] skip orientation FF as there are not enough pairs [M::mem_pestat] analyzing insert size distribution for orientation FR... M::mem_pestat percentile: (320, 365, 414) ... (ignored) ... [M::mem_pestat] low and high boundaries for computing mean and std.dev: (121, 571) [M::mem_pestat] mean and std.dev: (352.56, 77.28) [M::mem_pestat] low and high boundaries for proper pairs: (31, 661) [M::mem_pestat] skip orientation RF as there are not enough pairs [M::mem_pestat] skip orientation RR as there are not enough pairs [M::mem_process_seqs] Processed 75598 reads in 38.578 CPU sec, 4.600 real sec [main] Version: 0.7.12-r1039 [main] CMD: ./bwa mem -t 8 bwa_AllRef.fa ./test_NCLscan/simu_5X_100PE_1.fastq ./test_NCLscan/simu_5X_100PE_2.fastq [main] Real time: 534.858 sec; CPU: 3705.276 sec [samopen] SAM header is present: 119505 sequences. sh: 1: ./FastqOut: not found cat: write error: Broken pipe sh: 1: ./FastqOut: not found cat: write error: Broken pipe ./NCLscan.sh: line 14: ./NCL_Scan1: No such file or directory ./NCLscan.sh: line 15: ./NCL_Scan2: No such file or directory ./NCLscan.sh: line 16: ./NCL_Scan3: No such file or directory NCL_Scan4 v0.2.5 config.tmp can not be found! Please make sure that NCL_Scan2 has been finished properly. ./NCLscan.sh: line 18: ./NCL_Scan5.py: Permission denied =========== Stop ============ Tue Oct 27 14:43:08 CET 2015

Is it normal that the two files FastqOut and SeqOut that are reported as "not found" are binaries files in the NCLscan_v1.4 folder?

Thanks in advance for your answers.

Vincent

Do you mean that FastqOut and SeqOut are in the NCLscan_v1.4 folder, but reported as "not found"?

[vincenthahaut]

Thanks for your quick answer.

I checked and all the programs are executable. I put them all in the NCLscan folder. I did the same with the reference files. As described previously I am able to run the indexing and the first step of the NCLscan pipelin but got errors with FastqOut and SeqOut. There are both in the NCLscan_v1.4 folder but reported as not found when I run the different commands reported in the first message. They were not affected by the problem of permission as NCLscan_v1.4/MakeBwaRef. When I try to run only NCLscan_v1.4/SeqOut or NCLscan_v1.4/FastqOut it says: "No such file or directory" even if they are present in the folder.

:~/NCLscan-master/NCLscan_v1.4$ ./SeqOut -bash: ./SeqOut: No such file or directory

~$: ls -ll NCLscan_v1.4/ -rwxr-xr-x 1 vincent... 112569 Oct 12 2013 FastqOut -rwxr-xr-x 1 vincent... 94233 Oct 12 2013 SeqOut

[chiangtw]

Please provide more information about your system environment, such as your operating system and file system format.

[vincenthahaut]

Hope those are the informations you need:

Linux kosmos 3.2.0-4-amd64 #1 SMP Debian 3.2.46-1+deb7u1 x86_64

gcc (Debian 4.7.2-5) 4.7.2 g++ (Debian 4.7.2-5) 4.7.2

bwa-0.7.12 blat - Standalone BLAT v. 36x1 fast sequence search command line tool bedtools2 (last update 28/10/2015) Program: samtools (Tools for alignments in the SAM format) Version: 1.2 (using htslib 1.2.1) Novoalign V3.03.00 novoindex (3.3) - Universal k-mer index constructor.

• License from novocraft obtained one week ago.

I have not seen any error during the compiling of the code.

################################################################################# UPDATE: The error seems to come from the fact that I was running Debian on this machine. I migrated everything on another Linux machine and was able to start the first step of the pipeline without errors related to ./SeqOut or ./FastqOut.

Linux master2 2.6.32-279.22.1.el6.x86_64 #1 SMP Tue Feb 5 14:33:39 CST 2013 x86_64 x86_64 x86_64 GNU/Linux gcc (GCC) 4.4.7 20120313 (Red Hat 4.4.7-11) g++ (GCC) 4.4.7 20120313 (Red Hat 4.4.7-11) Python 2.6.6 (r266:84292, Jan 22 2014, 05:06:49) [GCC 4.4.7 20120313 (Red Hat 4.4.7-3)] on linux2 bwa-0.7.12 blat - Standalone BLAT v. 36x1 fast sequence search command line tool bedtools2 (last update 28/10/2015) Program: samtools (Tools for alignments in the SAM format) Version: 1.2 (using htslib 1.2.1) Novoalign V3.03.00 novoindex (3.3) - Universal k-mer index constructor.

• License from novocraft obtained one week ago.

Both ./MakeBwaRef and ./NCL_Scan5.py are still displaying a permission error.

I followed the instructions and was able to create the index without any error.

Then I started:

./NCLscan_v1.4 $ ./NCLscan.sh ../tumor_R1.fastq ../tumor_R1.fastq 20 4 50

It seems that you forget to mention that the argaprse library was required because I got this error:

File "./NCL_Scan0.py", line 3, in import argparse ImportError: No module named argparse

Solved by export PYTHONPATH=....../argparse-1.4.0/:$PYTHONPATH

I restarted the run which seemed to work well up to the first samtools command line:

NB: NCLscan_v1.4/ contains a ./samtools binary which is perfectly working.

vhahaut@nic4.master2 /scratch/ulg/genan/vhahaut/NCLscan_v1.4 $ ./NCLscan.sh ../samples_unizpped/tumor1-1_R1.fastq ../samples_unizpped/tumor1-1_R2.fastq 20 4 50 ############### Start ############### Thu Oct 29 15:10:52 CET 2015 ################################### [M::bwa_idx_load_from_disk] read 0 ALT contigs [M::process] read 4950496 sequences (500000096 bp)... [M::mem_pestat] # candidate unique pairs for (FF, FR, RF, RR): (31, 343859, 336, 27) [M::mem_pestat] analyzing insert size distribution for orientation FF... [M::mem_pestat](25, 50, 75) percentile: (43, 58, 146) [M::mem_pestat] low and high boundaries for computing mean and std.dev: (1, 352) [M::mem_pestat] mean and std.dev: (72.07, 47.81) [M::mem_pestat] low and high boundaries for proper pairs: (1, 455) [M::mem_pestat] analyzing insert size distribution for orientation FR... [M::mem_pestat](25, 50, 75) percentile: (137, 169, 214) [M::mem_pestat] low and high boundaries for computing mean and std.dev: (1, 368) [M::mem_pestat] mean and std.dev: (178.47, 57.06) [M::mem_pestat] low and high boundaries for proper pairs: (1, 445) [M::mem_pestat] analyzing insert size distribution for orientation RF... [M::mem_pestat](25, 50, 75) percentile: (154, 401, 986) [M::mem_pestat] low and high boundaries for computing mean and std.dev: (1, 2650) [M::mem_pestat] mean and std.dev: (533.51, 565.30) [M::mem_pestat] low and high boundaries for proper pairs: (1, 3482) [M::mem_pestat] analyzing insert size distribution for orientation RR... [M::mem_pestat](25, 50, 75) percentile: (64, 130, 142) [M::mem_pestat] low and high boundaries for computing mean and std.dev: (1, 298) [M::mem_pestat] mean and std.dev: (106.24, 56.55) [M::mem_pestat] low and high boundaries for proper pairs: (1, 376) [M::mem_pestat] skip orientation FF [M::mem_pestat] skip orientation RF [M::mem_pestat] skip orientation RR [M::mem_process_seqs] Processed 4950496 reads in 1831.882 CPU sec, 153.423 real sec [M::process] read 4950496 sequences (500000096 bp)... [M::process] read 4950496 sequences (500000096 bp)... [M::mem_pestat] # candidate unique pairs for (FF, FR, RF, RR): (46, 346038, 370, 23) [M::mem_pestat] analyzing insert size distribution for orientation FF... [M::mem_pestat](25, 50, 75) percentile: (36, 62, 118) [M::mem_pestat] low and high boundaries for computing mean and std.dev: (1, 282) [M::mem_pestat] mean and std.dev: (58.86, 40.90) [M::mem_pestat] low and high boundaries for proper pairs: (1, 364) [M::mem_pestat] analyzing insert size distribution for orientation FR... [M::mem_pestat](25, 50, 75) percentile: (136, 167, 211) [M::mem_pestat] low and high boundaries for computing mean and std.dev: (1, 361) [M::mem_pestat] mean and std.dev: (176.55, 55.78) [M::mem_pestat] low and high boundaries for proper pairs: (1, 436) [M::mem_pestat] analyzing insert size distribution for orientation RF... [M::mem_pestat](25, 50, 75) percentile: (128, 361, 1291) [M::mem_pestat] low and high boundaries for computing mean and std.dev: (1, 3617) [M::mem_pestat] mean and std.dev: (684.78, 831.44) [M::mem_pestat] low and high boundaries for proper pairs: (1, 4780) [M::mem_pestat] analyzing insert size distribution for orientation RR... [M::mem_pestat](25, 50, 75) percentile: (51, 110, 140) [M::mem_pestat] low and high boundaries for computing mean and std.dev: (1, 318) [M::mem_pestat] mean and std.dev: (86.68, 58.84) [M::mem_pestat] low and high boundaries for proper pairs: (1, 407) [M::mem_pestat] skip orientation FF [M::mem_pestat] skip orientation RF [M::mem_pestat] skip orientation RR [M::mem_process_seqs] Processed 4950496 reads in 1740.298 CPU sec, 123.885 real sec [M::process] read 4950496 sequences (500000096 bp)... [M::mem_pestat] # candidate unique pairs for (FF, FR, RF, RR): (39, 349147, 344, 19) [M::mem_pestat] analyzing insert size distribution for orientation FF... [M::mem_pestat](25, 50, 75) percentile: (33, 55, 100) [M::mem_pestat] low and high boundaries for computing mean and std.dev: (1, 234) [M::mem_pestat] mean and std.dev: (58.97, 41.43) [M::mem_pestat] low and high boundaries for proper pairs: (1, 301) [M::mem_pestat] analyzing insert size distribution for orientation FR... [M::mem_pestat](25, 50, 75) percentile: (136, 166, 209) [M::mem_pestat] low and high boundaries for computing mean and std.dev: (1, 355) [M::mem_pestat] mean and std.dev: (174.95, 54.27) [M::mem_pestat] low and high boundaries for proper pairs: (1, 428) [M::mem_pestat] analyzing insert size distribution for orientation RF... [M::mem_pestat](25, 50, 75) percentile: (113, 356, 890) [M::mem_pestat] low and high boundaries for computing mean and std.dev: (1, 2444) [M::mem_pestat] mean and std.dev: (484.73, 516.58) [M::mem_pestat] low and high boundaries for proper pairs: (1, 3221) [M::mem_pestat] analyzing insert size distribution for orientation RR... [M::mem_pestat](25, 50, 75) percentile: (61, 120, 175) [M::mem_pestat] low and high boundaries for computing mean and std.dev: (1, 403) [M::mem_pestat] mean and std.dev: (109.53, 69.88) [M::mem_pestat] low and high boundaries for proper pairs: (1, 517) [M::mem_pestat] skip orientation FF [M::mem_pestat] skip orientation RF [M::mem_pestat] skip orientation RR [M::mem_process_seqs] Processed 4950496 reads in 1733.054 CPU sec, 115.419 real sec [M::process] read 4950496 sequences (500000096 bp)... [M::mem_pestat] # candidate unique pairs for (FF, FR, RF, RR): (32, 341078, 350, 29) [M::mem_pestat] analyzing insert size distribution for orientation FF... [M::mem_pestat](25, 50, 75) percentile: (33, 60, 737) [M::mem_pestat] low and high boundaries for computing mean and std.dev: (1, 2145) [M::mem_pestat] mean and std.dev: (315.78, 494.48) [M::mem_pestat] low and high boundaries for proper pairs: (1, 2849) [M::mem_pestat] analyzing insert size distribution for orientation FR... [M::mem_pestat](25, 50, 75) percentile: (136, 168, 212) [M::mem_pestat] low and high boundaries for computing mean and std.dev: (1, 364) [M::mem_pestat] mean and std.dev: (177.10, 56.35) [M::mem_pestat] low and high boundaries for proper pairs: (1, 440) [M::mem_pestat] analyzing insert size distribution for orientation RF... [M::mem_pestat](25, 50, 75) percentile: (101, 332, 954) [M::mem_pestat] low and high boundaries for computing mean and std.dev: (1, 2660) [M::mem_pestat] mean and std.dev: (490.85, 566.85) [M::mem_pestat] low and high boundaries for proper pairs: (1, 3513) [M::mem_pestat] analyzing insert size distribution for orientation RR... [M::mem_pestat](25, 50, 75) percentile: (45, 118, 501) [M::mem_pestat] low and high boundaries for computing mean and std.dev: (1, 1413) [M::mem_pestat] mean and std.dev: (330.69, 443.70) [M::mem_pestat] low and high boundaries for proper pairs: (1, 1869) [M::mem_pestat] skip orientation FF [M::mem_pestat] skip orientation RF [M::mem_pestat] skip orientation RR [M::mem_process_seqs] Processed 4950496 reads in 1792.064 CPU sec, 119.611 real sec [M::process] read 4950496 sequences (500000096 bp)... [M::mem_pestat] # candidate unique pairs for (FF, FR, RF, RR): (35, 342973, 343, 24) [M::mem_pestat] analyzing insert size distribution for orientation FF... [M::mem_pestat](25, 50, 75) percentile: (27, 50, 90) [M::mem_pestat] low and high boundaries for computing mean and std.dev: (1, 216) [M::mem_pestat] mean and std.dev: (51.72, 34.83) [M::mem_pestat] low and high boundaries for proper pairs: (1, 279) [M::mem_pestat] analyzing insert size distribution for orientation FR... [M::mem_pestat](25, 50, 75) percentile: (136, 167, 211) [M::mem_pestat] low and high boundaries for computing mean and std.dev: (1, 361) [M::mem_pestat] mean and std.dev: (176.34, 55.92) [M::mem_pestat] low and high boundaries for proper pairs: (1, 436) [M::mem_pestat] analyzing insert size distribution for orientation RF... [M::mem_pestat](25, 50, 75) percentile: (119, 405, 924) [M::mem_pestat] low and high boundaries for computing mean and std.dev: (1, 2534) [M::mem_pestat] mean and std.dev: (525.11, 543.88) [M::mem_pestat] low and high boundaries for proper pairs: (1, 3339) [M::mem_pestat] analyzing insert size distribution for orientation RR... [M::mem_pestat](25, 50, 75) percentile: (106, 138, 494) [M::mem_pestat] low and high boundaries for computing mean and std.dev: (1, 1270) [M::mem_pestat] mean and std.dev: (168.95, 154.22) [M::mem_pestat] low and high boundaries for proper pairs: (1, 1658) [M::mem_pestat] skip orientation FF [M::mem_pestat] skip orientation RF [M::mem_pestat] skip orientation RR [M::mem_process_seqs] Processed 4950496 reads in 1742.037 CPU sec, 115.797 real sec [M::process] read 4950496 sequences (500000096 bp)... [M::mem_pestat] # candidate unique pairs for (FF, FR, RF, RR): (35, 345601, 356, 17) [M::mem_pestat] analyzing insert size distribution for orientation FF... [M::mem_pestat](25, 50, 75) percentile: (22, 48, 115) [M::mem_pestat] low and high boundaries for computing mean and std.dev: (1, 301) [M::mem_pestat] mean and std.dev: (69.82, 63.68) [M::mem_pestat] low and high boundaries for proper pairs: (1, 394) [M::mem_pestat] analyzing insert size distribution for orientation FR... [M::mem_pestat](25, 50, 75) percentile: (135, 166, 208) [M::mem_pestat] low and high boundaries for computing mean and std.dev: (1, 354) [M::mem_pestat] mean and std.dev: (174.34, 54.33) [M::mem_pestat] low and high boundaries for proper pairs: (1, 427) [M::mem_pestat] analyzing insert size distribution for orientation RF... [M::mem_pestat](25, 50, 75) percentile: (126, 299, 761) [M::mem_pestat] low and high boundaries for computing mean and std.dev: (1, 2031) [M::mem_pestat] mean and std.dev: (410.39, 419.84) [M::mem_pestat] low and high boundaries for proper pairs: (1, 2666) [M::mem_pestat] analyzing insert size distribution for orientation RR... [M::mem_pestat](25, 50, 75) percentile: (22, 118, 149) [M::mem_pestat] low and high boundaries for computing mean and std.dev: (1, 403) [M::mem_pestat] mean and std.dev: (100.38, 98.49) [M::mem_pestat] low and high boundaries for proper pairs: (1, 530) [M::mem_pestat] skip orientation FF [M::mem_pestat] skip orientation RF [M::mem_pestat] skip orientation RR [M::mem_process_seqs] Processed 4950496 reads in 1727.912 CPU sec, 114.590 real sec [M::process] read 4950496 sequences (500000096 bp)... [M::mem_pestat] # candidate unique pairs for (FF, FR, RF, RR): (30, 344377, 348, 16) [M::mem_pestat] analyzing insert size distribution for orientation FF... [M::mem_pestat](25, 50, 75) percentile: (25, 42, 121) [M::mem_pestat] low and high boundaries for computing mean and std.dev: (1, 313) [M::mem_pestat] mean and std.dev: (54.50, 48.15) [M::mem_pestat] low and high boundaries for proper pairs: (1, 409) [M::mem_pestat] analyzing insert size distribution for orientation FR... [M::mem_pestat](25, 50, 75) percentile: (136, 167, 211) [M::mem_pestat] low and high boundaries for computing mean and std.dev: (1, 361) [M::mem_pestat] mean and std.dev: (176.26, 55.84) [M::mem_pestat] low and high boundaries for proper pairs: (1, 436) [M::mem_pestat] analyzing insert size distribution for orientation RF... [M::mem_pestat](25, 50, 75) percentile: (158, 394, 887) [M::mem_pestat] low and high boundaries for computing mean and std.dev: (1, 2345) [M::mem_pestat] mean and std.dev: (502.27, 503.08) [M::mem_pestat] low and high boundaries for proper pairs: (1, 3074) [M::mem_pestat] analyzing insert size distribution for orientation RR... [M::mem_pestat](25, 50, 75) percentile: (85, 125, 867) [M::mem_pestat] low and high boundaries for computing mean and std.dev: (1, 2431) [M::mem_pestat] mean and std.dev: (422.56, 644.07) [M::mem_pestat] low and high boundaries for proper pairs: (1, 3213) [M::mem_pestat] skip orientation FF [M::mem_pestat] skip orientation RF [M::mem_pestat] skip orientation RR [M::mem_process_seqs] Processed 4950496 reads in 1752.193 CPU sec, 116.088 real sec [M::process] read 4950496 sequences (500000096 bp)... [M::mem_pestat] # candidate unique pairs for (FF, FR, RF, RR): (37, 344188, 298, 24) [M::mem_pestat] analyzing insert size distribution for orientation FF... [M::mem_pestat](25, 50, 75) percentile: (33, 53, 121) [M::mem_pestat] low and high boundaries for computing mean and std.dev: (1, 297) [M::mem_pestat] mean and std.dev: (77.53, 69.99) [M::mem_pestat] low and high boundaries for proper pairs: (1, 385) [M::mem_pestat] analyzing insert size distribution for orientation FR... [M::mem_pestat](25, 50, 75) percentile: (136, 167, 211) [M::mem_pestat] low and high boundaries for computing mean and std.dev: (1, 361) [M::mem_pestat] mean and std.dev: (175.98, 55.74) [M::mem_pestat] low and high boundaries for proper pairs: (1, 436) [M::mem_pestat] analyzing insert size distribution for orientation RF... [M::mem_pestat](25, 50, 75) percentile: (145, 396, 906) [M::mem_pestat] low and high boundaries for computing mean and std.dev: (1, 2428) [M::mem_pestat] mean and std.dev: (487.12, 501.37) [M::mem_pestat] low and high boundaries for proper pairs: (1, 3189) [M::mem_pestat] analyzing insert size distribution for orientation RR... [M::mem_pestat](25, 50, 75) percentile: (38, 89, 138) [M::mem_pestat] low and high boundaries for computing mean and std.dev: (1, 338) [M::mem_pestat] mean and std.dev: (68.50, 46.48) [M::mem_pestat] low and high boundaries for proper pairs: (1, 438) [M::mem_pestat] skip orientation FF [M::mem_pestat] skip orientation RF [M::mem_pestat] skip orientation RR [M::mem_process_seqs] Processed 4950496 reads in 1732.365 CPU sec, 115.426 real sec [M::process] read 4950496 sequences (500000096 bp)... [M::mem_pestat] # candidate unique pairs for (FF, FR, RF, RR): (55, 345866, 324, 21) [M::mem_pestat] analyzing insert size distribution for orientation FF... [M::mem_pestat](25, 50, 75) percentile: (30, 56, 107) [M::mem_pestat] low and high boundaries for computing mean and std.dev: (1, 261) [M::mem_pestat] mean and std.dev: (61.12, 41.69) [M::mem_pestat] low and high boundaries for proper pairs: (1, 338) [M::mem_pestat] analyzing insert size distribution for orientation FR... [M::mem_pestat](25, 50, 75) percentile: (135, 166, 208) [M::mem_pestat] low and high boundaries for computing mean and std.dev: (1, 354) [M::mem_pestat] mean and std.dev: (174.43, 54.46) [M::mem_pestat] low and high boundaries for proper pairs: (1, 427) [M::mem_pestat] analyzing insert size distribution for orientation RF... [M::mem_pestat](25, 50, 75) percentile: (132, 389, 1055) [M::mem_pestat] low and high boundaries for computing mean and std.dev: (1, 2901) [M::mem_pestat] mean and std.dev: (602.14, 651.93) [M::mem_pestat] low and high boundaries for proper pairs: (1, 3824) [M::mem_pestat] analyzing insert size distribution for orientation RR... [M::mem_pestat](25, 50, 75) percentile: (48, 120, 948) [M::mem_pestat] low and high boundaries for computing mean and std.dev: (1, 2748) [M::mem_pestat] mean and std.dev: (386.43, 544.74) [M::mem_pestat] low and high boundaries for proper pairs: (1, 3648) [M::mem_pestat] skip orientation FF [M::mem_pestat] skip orientation RF [M::mem_pestat] skip orientation RR [M::mem_process_seqs] Processed 4950496 reads in 1717.687 CPU sec, 113.671 real sec [M::process] read 4950496 sequences (500000096 bp)... [M::mem_pestat] # candidate unique pairs for (FF, FR, RF, RR): (22, 343335, 349, 28) [M::mem_pestat] analyzing insert size distribution for orientation FF... [M::mem_pestat](25, 50, 75) percentile: (20, 82, 129) [M::mem_pestat] low and high boundaries for computing mean and std.dev: (1, 347) [M::mem_pestat] mean and std.dev: (71.58, 63.32) [M::mem_pestat] low and high boundaries for proper pairs: (1, 456) [M::mem_pestat] analyzing insert size distribution for orientation FR... [M::mem_pestat](25, 50, 75) percentile: (136, 167, 211) [M::mem_pestat] low and high boundaries for computing mean and std.dev: (1, 361) [M::mem_pestat] mean and std.dev: (176.05, 55.60) [M::mem_pestat] low and high boundaries for proper pairs: (1, 436) [M::mem_pestat] analyzing insert size distribution for orientation RF... [M::mem_pestat](25, 50, 75) percentile: (127, 322, 803) [M::mem_pestat] low and high boundaries for computing mean and std.dev: (1, 2155) [M::mem_pestat] mean and std.dev: (466.25, 478.89) [M::mem_pestat] low and high boundaries for proper pairs: (1, 2831) [M::mem_pestat] analyzing insert size distribution for orientation RR... [M::mem_pestat](25, 50, 75) percentile: (54, 130, 283) [M::mem_pestat] low and high boundaries for computing mean and std.dev: (1, 741) [M::mem_pestat] mean and std.dev: (97.55, 72.07) [M::mem_pestat] low and high boundaries for proper pairs: (1, 970) [M::mem_pestat] skip orientation FF [M::mem_pestat] skip orientation RF [M::mem_pestat] skip orientation RR [M::mem_process_seqs] Processed 4950496 reads in 1790.845 CPU sec, 119.240 real sec [M::process] read 4950496 sequences (500000096 bp)... [M::mem_pestat] # candidate unique pairs for (FF, FR, RF, RR): (30, 342495, 380, 19) [M::mem_pestat] analyzing insert size distribution for orientation FF... [M::mem_pestat](25, 50, 75) percentile: (25, 52, 120) [M::mem_pestat] low and high boundaries for computing mean and std.dev: (1, 310) [M::mem_pestat] mean and std.dev: (54.28, 39.86) [M::mem_pestat] low and high boundaries for proper pairs: (1, 405) [M::mem_pestat] analyzing insert size distribution for orientation FR... [M::mem_pestat](25, 50, 75) percentile: (137, 168, 213) [M::mem_pestat] low and high boundaries for computing mean and std.dev: (1, 365) [M::mem_pestat] mean and std.dev: (177.63, 56.57) [M::mem_pestat] low and high boundaries for proper pairs: (1, 441) [M::mem_pestat] analyzing insert size distribution for orientation RF... [M::mem_pestat](25, 50, 75) percentile: (125, 390, 913) [M::mem_pestat] low and high boundaries for computing mean and std.dev: (1, 2489) [M::mem_pestat] mean and std.dev: (529.49, 556.27) [M::mem_pestat] low and high boundaries for proper pairs: (1, 3277) [M::mem_pestat] analyzing insert size distribution for orientation RR... [M::mem_pestat](25, 50, 75) percentile: (44, 119, 679) [M::mem_pestat] low and high boundaries for computing mean and std.dev: (1, 1949) [M::mem_pestat] mean and std.dev: (345.47, 486.41) [M::mem_pestat] low and high boundaries for proper pairs: (1, 2584) [M::mem_pestat] skip orientation FF [M::mem_pestat] skip orientation RF [M::mem_pestat] skip orientation RR [M::mem_process_seqs] Processed 4950496 reads in 1790.314 CPU sec, 119.051 real sec [M::process] read 4950496 sequences (500000096 bp)... [M::mem_pestat] # candidate unique pairs for (FF, FR, RF, RR): (40, 344455, 372, 19) [M::mem_pestat] analyzing insert size distribution for orientation FF... [M::mem_pestat](25, 50, 75) percentile: (26, 54, 107) [M::mem_pestat] low and high boundaries for computing mean and std.dev: (1, 269) [M::mem_pestat] mean and std.dev: (57.58, 41.21) [M::mem_pestat] low and high boundaries for proper pairs: (1, 350) [M::mem_pestat] analyzing insert size distribution for orientation FR... [M::mem_pestat](25, 50, 75) percentile: (136, 167, 211) [M::mem_pestat] low and high boundaries for computing mean and std.dev: (1, 361) [M::mem_pestat] mean and std.dev: (176.32, 55.60) [M::mem_pestat] low and high boundaries for proper pairs: (1, 436) [M::mem_pestat] analyzing insert size distribution for orientation RF... [M::mem_pestat](25, 50, 75) percentile: (124, 367, 978) [M::mem_pestat] low and high boundaries for computing mean and std.dev: (1, 2686) [M::mem_pestat] mean and std.dev: (550.47, 601.21) [M::mem_pestat] low and high boundaries for proper pairs: (1, 3540) [M::mem_pestat] analyzing insert size distribution for orientation RR... [M::mem_pestat](25, 50, 75) percentile: (50, 157, 1043) [M::mem_pestat] low and high boundaries for computing mean and std.dev: (1, 3029) [M::mem_pestat] mean and std.dev: (465.95, 636.23) [M::mem_pestat] low and high boundaries for proper pairs: (1, 4022) [M::mem_pestat] skip orientation FF [M::mem_pestat] skip orientation RF [M::mem_pestat] skip orientation RR [M::mem_process_seqs] Processed 4950496 reads in 1766.559 CPU sec, 116.162 real sec [M::process] read 4950496 sequences (500000096 bp)... [M::mem_pestat] # candidate unique pairs for (FF, FR, RF, RR): (44, 345437, 347, 20) [M::mem_pestat] analyzing insert size distribution for orientation FF... [M::mem_pestat](25, 50, 75) percentile: (33, 52, 77) [M::mem_pestat] low and high boundaries for computing mean and std.dev: (1, 165) [M::mem_pestat] mean and std.dev: (53.10, 37.05) [M::mem_pestat] low and high boundaries for proper pairs: (1, 209) [M::mem_pestat] analyzing insert size distribution for orientation FR... [M::mem_pestat](25, 50, 75) percentile: (136, 167, 210) [M::mem_pestat] low and high boundaries for computing mean and std.dev: (1, 358) [M::mem_pestat] mean and std.dev: (175.64, 54.75) [M::mem_pestat] low and high boundaries for proper pairs: (1, 432) [M::mem_pestat] analyzing insert size distribution for orientation RF... [M::mem_pestat](25, 50, 75) percentile: (91, 276, 810) [M::mem_pestat] low and high boundaries for computing mean and std.dev: (1, 2248) [M::mem_pestat] mean and std.dev: (427.60, 509.52) [M::mem_pestat] low and high boundaries for proper pairs: (1, 2967) [M::mem_pestat] analyzing insert size distribution for orientation RR... [M::mem_pestat](25, 50, 75) percentile: (63, 134, 272) [M::mem_pestat] low and high boundaries for computing mean and std.dev: (1, 690) [M::mem_pestat] mean and std.dev: (160.00, 155.19) [M::mem_pestat] low and high boundaries for proper pairs: (1, 899) [M::mem_pestat] skip orientation FF [M::mem_pestat] skip orientation RF [M::mem_pestat] skip orientation RR [M::mem_process_seqs] Processed 4950496 reads in 1774.469 CPU sec, 117.715 real sec [M::process] read 4950496 sequences (500000096 bp)... [M::mem_pestat] # candidate unique pairs for (FF, FR, RF, RR): (31, 340407, 366, 22) [M::mem_pestat] analyzing insert size distribution for orientation FF... [M::mem_pestat](25, 50, 75) percentile: (26, 49, 92) [M::mem_pestat] low and high boundaries for computing mean and std.dev: (1, 224) [M::mem_pestat] mean and std.dev: (55.61, 44.72) [M::mem_pestat] low and high boundaries for proper pairs: (1, 290) [M::mem_pestat] analyzing insert size distribution for orientation FR... [M::mem_pestat](25, 50, 75) percentile: (136, 168, 213) [M::mem_pestat] low and high boundaries for computing mean and std.dev: (1, 367) [M::mem_pestat] mean and std.dev: (177.57, 57.14) [M::mem_pestat] low and high boundaries for proper pairs: (1, 444) [M::mem_pestat] analyzing insert size distribution for orientation RF... [M::mem_pestat](25, 50, 75) percentile: (127, 407, 1025) [M::mem_pestat] low and high boundaries for computing mean and std.dev: (1, 2821) [M::mem_pestat] mean and std.dev: (566.25, 605.90) [M::mem_pestat] low and high boundaries for proper pairs: (1, 3719) [M::mem_pestat] analyzing insert size distribution for orientation RR... [M::mem_pestat](25, 50, 75) percentile: (29, 119, 425) [M::mem_pestat] low and high boundaries for computing mean and std.dev: (1, 1217) [M::mem_pestat] mean and std.dev: (93.24, 95.27) [M::mem_pestat] low and high boundaries for proper pairs: (1, 1613) [M::mem_pestat] skip orientation FF [M::mem_pestat] skip orientation RF [M::mem_pestat] skip orientation RR [M::mem_process_seqs] Processed 4950496 reads in 1800.448 CPU sec, 118.310 real sec [M::process] read 4950496 sequences (500000096 bp)... [M::mem_pestat] # candidate unique pairs for (FF, FR, RF, RR): (26, 342524, 324, 27) [M::mem_pestat] analyzing insert size distribution for orientation FF... [M::mem_pestat](25, 50, 75) percentile: (27, 57, 90) [M::mem_pestat] low and high boundaries for computing mean and std.dev: (1, 216) [M::mem_pestat] mean and std.dev: (49.64, 35.38) [M::mem_pestat] low and high boundaries for proper pairs: (1, 279) [M::mem_pestat] analyzing insert size distribution for orientation FR... [M::mem_pestat](25, 50, 75) percentile: (136, 167, 210) [M::mem_pestat] low and high boundaries for computing mean and std.dev: (1, 358) [M::mem_pestat] mean and std.dev: (175.74, 55.30) [M::mem_pestat] low and high boundaries for proper pairs: (1, 432) [M::mem_pestat] analyzing insert size distribution for orientation RF... [M::mem_pestat](25, 50, 75) percentile: (120, 342, 774) [M::mem_pestat] low and high boundaries for computing mean and std.dev: (1, 2082) [M::mem_pestat] mean and std.dev: (430.74, 442.38) [M::mem_pestat] low and high boundaries for proper pairs: (1, 2736) [M::mem_pestat] analyzing insert size distribution for orientation RR... [M::mem_pestat](25, 50, 75) percentile: (67, 115, 1253) [M::mem_pestat] low and high boundaries for computing mean and std.dev: (1, 3625) [M::mem_pestat] mean and std.dev: (521.81, 764.92) [M::mem_pestat] low and high boundaries for proper pairs: (1, 4811) [M::mem_pestat] skip orientation FF [M::mem_pestat] skip orientation RF [M::mem_pestat] skip orientation RR [M::mem_process_seqs] Processed 4950496 reads in 1772.556 CPU sec, 116.664 real sec [M::process] read 4950496 sequences (500000096 bp)... [M::mem_pestat] # candidate unique pairs for (FF, FR, RF, RR): (37, 345175, 359, 21) [M::mem_pestat] analyzing insert size distribution for orientation FF... [M::mem_pestat](25, 50, 75) percentile: (23, 66, 116) [M::mem_pestat] low and high boundaries for computing mean and std.dev: (1, 302) [M::mem_pestat] mean and std.dev: (69.86, 57.98) [M::mem_pestat] low and high boundaries for proper pairs: (1, 395) [M::mem_pestat] analyzing insert size distribution for orientation FR... [M::mem_pestat](25, 50, 75) percentile: (135, 166, 209) [M::mem_pestat] low and high boundaries for computing mean and std.dev: (1, 357) [M::mem_pestat] mean and std.dev: (174.69, 54.62) [M::mem_pestat] low and high boundaries for proper pairs: (1, 431) [M::mem_pestat] analyzing insert size distribution for orientation RF... [M::mem_pestat](25, 50, 75) percentile: (113, 353, 1025) [M::mem_pestat] low and high boundaries for computing mean and std.dev: (1, 2849) [M::mem_pestat] mean and std.dev: (551.16, 608.50) [M::mem_pestat] low and high boundaries for proper pairs: (1, 3761) [M::mem_pestat] analyzing insert size distribution for orientation RR... [M::mem_pestat](25, 50, 75) percentile: (48, 248, 1233) [M::mem_pestat] low and high boundaries for computing mean and std.dev: (1, 3603) [M::mem_pestat] mean and std.dev: (637.48, 704.66) [M::mem_pestat] low and high boundaries for proper pairs: (1, 4788) [M::mem_pestat] skip orientation FF [M::mem_pestat] skip orientation RF [M::mem_pestat] skip orientation RR [M::mem_process_seqs] Processed 4950496 reads in 1750.499 CPU sec, 115.922 real sec [M::process] read 4950496 sequences (500000096 bp)... [M::mem_pestat] # candidate unique pairs for (FF, FR, RF, RR): (29, 340688, 338, 27) [M::mem_pestat] analyzing insert size distribution for orientation FF... [M::mem_pestat](25, 50, 75) percentile: (40, 68, 772) [M::mem_pestat] low and high boundaries for computing mean and std.dev: (1, 2236) [M::mem_pestat] mean and std.dev: (315.48, 479.46) [M::mem_pestat] low and high boundaries for proper pairs: (1, 2968) [M::mem_pestat] analyzing insert size distribution for orientation FR... [M::mem_pestat](25, 50, 75) percentile: (136, 168, 214) [M::mem_pestat] low and high boundaries for computing mean and std.dev: (1, 370) [M::mem_pestat] mean and std.dev: (177.96, 57.47) [M::mem_pestat] low and high boundaries for proper pairs: (1, 448) [M::mem_pestat] analyzing insert size distribution for orientation RF... [M::mem_pestat](25, 50, 75) percentile: (119, 298, 918) [M::mem_pestat] low and high boundaries for computing mean and std.dev: (1, 2516) [M::mem_pestat] mean and std.dev: (490.85, 554.52) [M::mem_pestat] low and high boundaries for proper pairs: (1, 3315) [M::mem_pestat] analyzing insert size distribution for orientation RR... [M::mem_pestat](25, 50, 75) percentile: (35, 99, 265) [M::mem_pestat] low and high boundaries for computing mean and std.dev: (1, 725) [M::mem_pestat] mean and std.dev: (155.96, 183.44) [M::mem_pestat] low and high boundaries for proper pairs: (1, 955) [M::mem_pestat] skip orientation FF [M::mem_pestat] skip orientation RF [M::mem_pestat] skip orientation RR [M::mem_process_seqs] Processed 4950496 reads in 1794.254 CPU sec, 118.052 real sec [M::process] read 4950496 sequences (500000096 bp)... [M::mem_pestat] # candidate unique pairs for (FF, FR, RF, RR): (45, 344865, 310, 33) [M::mem_pestat] analyzing insert size distribution for orientation FF... [M::mem_pestat](25, 50, 75) percentile: (39, 74, 151) [M::mem_pestat] low and high boundaries for computing mean and std.dev: (1, 375) [M::mem_pestat] mean and std.dev: (67.64, 60.01) [M::mem_pestat] low and high boundaries for proper pairs: (1, 487) [M::mem_pestat] analyzing insert size distribution for orientation FR... [M::mem_pestat](25, 50, 75) percentile: (136, 167, 211) [M::mem_pestat] low and high boundaries for computing mean and std.dev: (1, 361) [M::mem_pestat] mean and std.dev: (176.11, 55.82) [M::mem_pestat] low and high boundaries for proper pairs: (1, 436) [M::mem_pestat] analyzing insert size distribution for orientation RF... [M::mem_pestat](25, 50, 75) percentile: (129, 409, 1078) [M::mem_pestat] low and high boundaries for computing mean and std.dev: (1, 2976) [M::mem_pestat] mean and std.dev: (553.95, 596.10) [M::mem_pestat] low and high boundaries for proper pairs: (1, 3925) [M::mem_pestat] analyzing insert size distribution for orientation RR... [M::mem_pestat](25, 50, 75) percentile: (94, 127, 288) [M::mem_pestat] low and high boundaries for computing mean and std.dev: (1, 676) [M::mem_pestat] mean and std.dev: (123.22, 87.19) [M::mem_pestat] low and high boundaries for proper pairs: (1, 870) [M::mem_pestat] skip orientation FF [M::mem_pestat] skip orientation RF [M::mem_pestat] skip orientation RR [M::mem_process_seqs] Processed 4950496 reads in 1751.091 CPU sec, 115.852 real sec [M::process] read 1158932 sequences (117052132 bp)... [M::mem_pestat] # candidate unique pairs for (FF, FR, RF, RR): (36, 346241, 375, 21) [M::mem_pestat] analyzing insert size distribution for orientation FF... [M::mem_pestat](25, 50, 75) percentile: (27, 58, 107) [M::mem_pestat] low and high boundaries for computing mean and std.dev: (1, 267) [M::mem_pestat] mean and std.dev: (52.00, 34.52) [M::mem_pestat] low and high boundaries for proper pairs: (1, 347) [M::mem_pestat] analyzing insert size distribution for orientation FR... [M::mem_pestat](25, 50, 75) percentile: (136, 166, 209) [M::mem_pestat] low and high boundaries for computing mean and std.dev: (1, 355) [M::mem_pestat] mean and std.dev: (174.81, 54.47) [M::mem_pestat] low and high boundaries for proper pairs: (1, 428) [M::mem_pestat] analyzing insert size distribution for orientation RF... [M::mem_pestat](25, 50, 75) percentile: (124, 368, 925) [M::mem_pestat] low and high boundaries for computing mean and std.dev: (1, 2527) [M::mem_pestat] mean and std.dev: (501.57, 546.96) [M::mem_pestat] low and high boundaries for proper pairs: (1, 3328) [M::mem_pestat] analyzing insert size distribution for orientation RR... [M::mem_pestat](25, 50, 75) percentile: (105, 125, 355) [M::mem_pestat] low and high boundaries for computing mean and std.dev: (1, 855) [M::mem_pestat] mean and std.dev: (119.35, 78.05) [M::mem_pestat] low and high boundaries for proper pairs: (1, 1105) [M::mem_pestat] skip orientation FF [M::mem_pestat] skip orientation RF [M::mem_pestat] skip orientation RR [M::mem_process_seqs] Processed 4950496 reads in 1728.418 CPU sec, 113.732 real sec [M::mem_pestat] # candidate unique pairs for (FF, FR, RF, RR): (8, 82540, 72, 4) [M::mem_pestat] skip orientation FF as there are not enough pairs [M::mem_pestat] analyzing insert size distribution for orientation FR... [M::mem_pestat](25, 50, 75) percentile: (135, 165, 207) [M::mem_pestat] low and high boundaries for computing mean and std.dev: (1, 351) [M::mem_pestat] mean and std.dev: (173.80, 53.43) [M::mem_pestat] low and high boundaries for proper pairs: (1, 423) [M::mem_pestat] analyzing insert size distribution for orientation RF... [M::mem_pestat](25, 50, 75) percentile: (78, 348, 1242) [M::mem_pestat] low and high boundaries for computing mean and std.dev: (1, 3570) [M::mem_pestat] mean and std.dev: (649.25, 806.09) [M::mem_pestat] low and high boundaries for proper pairs: (1, 4734) [M::mem_pestat] skip orientation RR as there are not enough pairs [M::mem_pestat] skip orientation RF [M::mem_process_seqs] Processed 1158932 reads in 412.306 CPU sec, 27.349 real sec [main] Version: 0.7.12-r1039 [main] CMD: ./bwa mem -t 50 bwa_AllRef.fa ../samples_unizpped/tumor1-1_R1.fastq ../samples_unizpped/tumor1-1_R2.fastq [main] Real time: 2303.324 sec; CPU: 33919.698 sec

sh: samtools: command not found

Traceback (most recent call last): File "./NCL_Scan0.py", line 43, in main() File "./NCL_Scan0.py", line 38, in main get_bwa_unmapped(args.Fastq_1, args.Fastq_2, args.Project_Name, args.bwa_index_prefix, args.bwa_thread) File "./NCL_Scan0.py", line 12, in get_bwa_unmapped os.mkdir("split_unmap.tmp") OSError: [Errno 17] File exists: 'split_unmap.tmp' novoindex (3.3) - Universal k-mer index constructor. (C) 2008 - 2011 NovoCraft Technologies Sdn Bhd novoindex AllRef.ndx AllRef.fa Creating 16 indexing threads. Building with 14-mer and step of 2 bp. tcmalloc: large alloc 1073750016 bytes == 0x10fa000 @ 0x503a3c 0x401841 0x40065a 0x491720 0x400c99 tcmalloc: large alloc 9237315584 bytes == 0x41104000 @ 0x504978 0x401a18 0x40065a 0x491720 0x400c99 novoindex construction dT = 96.8s Index memory size 8.603Gbyte. Done. Error opening sequence file: 20.bwa.unmapped_1.fastq Error: Cannot open input sequence files. 20.bwa.unmapped_1.fastq 20.bwa.unmapped_2.fastq cat: 20.bwa.unmapped_1.fastq: No such file or directory cat: 20.bwa.unmapped_2.fastq: No such file or directory End of file reading 4 bytes PslChimeraFilter v0.4

End of file reading 4 bytes End of file reading 4 bytes End of file reading 4 bytes JunctionSite2BED v0.3

Reading annotations on chr1. Reading annotations on chr2. Reading annotations on chr3. Reading annotations on chr4. Reading annotations on chr5. Reading annotations on chr6. Reading annotations on chr7. Reading annotations on chr8. Reading annotations on chr9. Reading annotations on chr10. Reading annotations on chr11. Reading annotations on chr12. Reading annotations on chr13. Reading annotations on chr14. Reading annotations on chr15. Reading annotations on chr16. Reading annotations on chr17. Reading annotations on chr18. Reading annotations on chr19. Reading annotations on chr20. Reading annotations on chr21. Reading annotations on chr22. Reading annotations on chrX. Reading annotations on chrY. Reading annotations on chrM. Read 57820 genes, 196520 transcripts and 1196293 exons from the gtf file. Format Error! Format Error! Format Error! Format Error! NCL_Scan3 v0.5 ./NCL_Scan3 (null) novoindex (3.3) - Universal k-mer index constructor. (C) 2008 - 2011 NovoCraft Technologies Sdn Bhd novoindex 20.JS.ndx 20.JS.fa Creating 16 indexing threads. Building with 6-mer and step of 1 bp. novoindex construction dT = 0.0s Index memory size 0.000Gbyte. Done.

Interrupted..11 Stack Dump ... ip = 0x 489c8c, sp = 0x 7ffffffc6520 ip = 0x 40533d, sp = 0x 7ffffffc6cf0 ip = 0x 55d150, sp = 0x 7ffffffc6d00 ip = 0x 40ed64, sp = 0x 7ffffffc72b0 ip = 0x 428f32, sp = 0x 7ffffffc72e0 ip = 0x 425e18, sp = 0x 7ffffffc7450 ip = 0x 400888, sp = 0x 7ffffffc7560 ip = 0x 55d7b0, sp = 0x 7fffffffdbc0 ip = 0x 405219, sp = 0x 7fffffffde70

Interrupted..11 Stack Dump ... ip = 0x 489c8c, sp = 0x 7ffffffc6520 ip = 0x 40533d, sp = 0x 7ffffffc6cf0 ip = 0x 55d150, sp = 0x 7ffffffc6d00 ip = 0x 40ed64, sp = 0x 7ffffffc72b0 ip = 0x 428f32, sp = 0x 7ffffffc72e0 ip = 0x 425e18, sp = 0x 7ffffffc7450 ip = 0x 400888, sp = 0x 7ffffffc7560 ip = 0x 55d7b0, sp = 0x 7fffffffdbc0 ip = 0x 405219, sp = 0x 7fffffffde70 End of file reading 4 bytes End of file reading 4 bytes PslChimeraFilter v0.4

JunctionSite2BED v0.3

Reading annotations on chr1. Reading annotations on chr2. Reading annotations on chr3. Reading annotations on chr4. Reading annotations on chr5. Reading annotations on chr6. Reading annotations on chr7. Reading annotations on chr8. Reading annotations on chr9. Reading annotations on chr10. Reading annotations on chr11. Reading annotations on chr12. Reading annotations on chr13. Reading annotations on chr14. Reading annotations on chr15. Reading annotations on chr16. Reading annotations on chr17. Reading annotations on chr18. Reading annotations on chr19. Reading annotations on chr20. Reading annotations on chr21. Reading annotations on chr22. Reading annotations on chrX. Reading annotations on chrY. Reading annotations on chrM. Read 57820 genes, 196520 transcripts and 1196293 exons from the gtf file. Format Error! Format Error! Format Error! Format Error! novoindex (3.3) - Universal k-mer index constructor. (C) 2008 - 2011 NovoCraft Technologies Sdn Bhd novoindex 20.JS2.ndx 20.JS2.fa Creating 16 indexing threads. Building with 6-mer and step of 1 bp. novoindex construction dT = 0.0s Index memory size 0.000Gbyte. Done. NCL_Scan4 v0.2.5

Interrupted..11 Stack Dump ... ip = 0x 489c8c, sp = 0x 7ffffffc6520 ip = 0x 40533d, sp = 0x 7ffffffc6cf0 ip = 0x 55d150, sp = 0x 7ffffffc6d00 ip = 0x 40ed64, sp = 0x 7ffffffc7290 ip = 0x 428f32, sp = 0x 7ffffffc72c0 ip = 0x 425e18, sp = 0x 7ffffffc7430 ip = 0x 400888, sp = 0x 7ffffffc7540 ip = 0x 55d7b0, sp = 0x 7fffffffdba0 ip = 0x 405219, sp = 0x 7fffffffde50 End of file reading 4 bytes End of file reading 4 bytes End of file reading 4 bytes End of file reading 4 bytes ./NCLscan.sh: line 18: ./NCL_Scan5.py: Permission denied ############### Stop############### Thu Oct 29 15:52:04 CET 2015 ##################################

[chiangtw]

Thanks for the information of your testing!

For the problem "samtools: command not found", it's a little bug in the python script 'NCL_Scan0.py'. To fix it, please add a "./" in front of "samtools", ie. fix it to "./samtools", in line 10 of 'NCL_Scan0.py'.

For the permission error, please "chmod 755" on them.

And the "argparse" package is in the standard library since Python2.7, we did forget to mention it.

The above problems will be fixed in the next release.

[vincenthahaut]

Thanks.

If you have time there is one thing that is missing and could be really helpfull for the next time:

When you run ./NCLscan.sh ../tumor-1_R1.fastq ../tumor-1_R2.fastq 20 4 50

it prints all the results to the directory where you are. You gave the advice to use "ln" to create a symbolic link in the output dir but for people like me that are not so comfortable with bioinformatic it would be really easier to have an -output directory parameter in the main command.

I am trying to run again the pipeline and see if everything is at its place now.

[vincenthahaut]

Hello!

New day, new feedback.

I find a new error (or at least something blocking the pipeline). Could please help me to solve it?

NB: The ./MakeBwaRef worked perfectly.

What I did:

==> run this sbatch script:

!/bin/sh

SBATCH --ntasks=1

SBATCH --cpus-per-task=6

SBATCH --mem-per-cpu=8G

SBATCH --partition=xxxxx

cd .../tmp #directory where I want the output ln -s /home/NCLscan_v1.4/* . pwd ls -ll echo NCLscan /home/NCLscan.sh /home/tumor_R1.fastq /home/tumor_R2.fastq tumor 20 6 50

The content of the directory after the link is the following (the binaries are working):

lrwxrwxrwx 1 94 Nov 3 14:41 Add_read_count.py -> /home/NCLscan_v1.4Add_read_count.py lrwxrwxrwx 1 90 Nov 3 14:41 AssembleExons -> /home/NCLscan_v1.4AssembleExons lrwxrwxrwx 1 90 Nov 3 14:41 AssembleFastq -> /home/NCLscan_v1.4AssembleFastq lrwxrwxrwx 1 89 Nov 3 14:41 AssembleJSeq -> /home/NCLscan_v1.4AssembleJSeq lrwxrwxrwx 1 86 Nov 3 14:41 ChangeLog -> /home/NCLscan_v1.4ChangeLog lrwxrwxrwx 1 85 Nov 3 14:41 FastqOut -> /home/NCLscan_v1.4FastqOut lrwxrwxrwx 1 102 Nov 3 14:41 GRCh37.p13.genome.fa -> /home/NCLscan_v1.4GRCh37.p13.genome.fa lrwxrwxrwx 1 84 Nov 3 14:41 GetInfo -> /home/NCLscan_v1.4GetInfo lrwxrwxrwx 1 83 Nov 3 14:41 GetKey -> /home/NCLscan_v1.4GetKey lrwxrwxrwx 1 89 Nov 3 14:41 GetNameB4Dot -> /home/NCLscan_v1.4GetNameB4Dot lrwxrwxrwx 1 89 Nov 3 14:41 InsertInList -> /home/NCLscan_v1.4InsertInList lrwxrwxrwx 1 85 Nov 3 14:41 JSFilter -> /home/NCLscan_v1.4JSFilter lrwxrwxrwx 1 85 Nov 3 14:41 JSParser -> /home/NCLscan_v1.4JSParser lrwxrwxrwx 1 93 Nov 3 14:41 JunctionSite2BED -> /home/NCLscan_v1.4JunctionSite2BED lrwxrwxrwx 1 87 Nov 3 14:41 MakeBwaRef -> /home/NCLscan_v1.4MakeBwaRef lrwxrwxrwx 1 89 Nov 3 14:41 NCL_Scan0.py -> /home/NCLscan_v1.4NCL_Scan0.py lrwxrwxrwx 1 86 Nov 3 14:41 NCL_Scan1 -> /home/NCLscan_v1.4NCL_Scan1 lrwxrwxrwx 1 86 Nov 3 14:41 NCL_Scan2 -> /home/NCLscan_v1.4NCL_Scan2 lrwxrwxrwx 1 86 Nov 3 14:41 NCL_Scan3 -> /home/NCLscan_v1.4NCL_Scan3 lrwxrwxrwx 1 86 Nov 3 14:41 NCL_Scan4 -> /home/NCLscan_v1.4NCL_Scan4 lrwxrwxrwx 1 89 Nov 3 14:41 NCL_Scan5.py -> /home/NCLscan_v1.4NCL_Scan5.py lrwxrwxrwx 1 87 Nov 3 14:41 NCLscan.sh -> /home/NCLscan_v1.4NCLscan.sh lrwxrwxrwx 1 93 Nov 3 14:41 PslChimeraFilter -> /home/NCLscan_v1.4PslChimeraFilter lrwxrwxrwx 1 83 Nov 3 14:41 README -> /home/NCLscan_v1.4README lrwxrwxrwx 1 89 Nov 3 14:41 RemoveInList -> /home/NCLscan_v1.4RemoveInList lrwxrwxrwx 1 87 Nov 3 14:41 RepChrM.fa -> /home/NCLscan_v1.4RepChrM.fa lrwxrwxrwx 1 89 Nov 3 14:41 RetainInList -> /home/NCLscan_v1.4RetainInList lrwxrwxrwx 1 89 Nov 3 14:41 RmBadMapping -> /home/NCLscan_v1.4RmBadMapping lrwxrwxrwx 1 96 Nov 3 14:41 RmColinearPairInSam -> /home/NCLscan_v1.4RmColinearPairInSam lrwxrwxrwx 1 89 Nov 3 14:41 RmRedundance -> /home/NCLscan_v1.4RmRedundance lrwxrwxrwx 1 83 Nov 3 14:41 SeqOut -> /home/NCLscan_v1.4SeqOut lrwxrwxrwx 1 85 Nov 3 14:41 bedtools -> /home/NCLscan_v1.4bedtools lrwxrwxrwx 1 81 Nov 3 14:41 blat -> /home/NCLscan_v1.4blat lrwxrwxrwx 1 80 Nov 3 14:41 bwa -> /home/NCLscan_v1.4bwa lrwxrwxrwx 1 90 Nov 3 14:41 bwa_AllRef.fa -> /home/NCLscan_v1.4bwa_AllRef.fa lrwxrwxrwx 1 94 Nov 3 14:41 bwa_AllRef.fa.amb -> /home/NCLscan_v1.4bwa_AllRef.fa.amb lrwxrwxrwx 1 94 Nov 3 14:41 bwa_AllRef.fa.ann -> /home/NCLscan_v1.4bwa_AllRef.fa.ann lrwxrwxrwx 1 94 Nov 3 14:41 bwa_AllRef.fa.bwt -> /home/NCLscan_v1.4bwa_AllRef.fa.bwt lrwxrwxrwx 1 94 Nov 3 14:41 bwa_AllRef.fa.pac -> /home/NCLscan_v1.4bwa_AllRef.fa.pac lrwxrwxrwx 1 93 Nov 3 14:41 bwa_AllRef.fa.sa -> /home/NCLscan_v1.4bwa_AllRef.fa.sa lrwxrwxrwx 1 87 Nov 3 14:41 config.txt -> /home/NCLscan_v1.4config.txt lrwxrwxrwx 1 103 Nov 3 14:41 gencode.v19.annotation.gtf -> /home/NCLscan_v1.4gencode.v19.annotation.gtf lrwxrwxrwx 1 110 Nov 3 14:41 gencode.v19.lncRNA_transcripts.fa -> /home/NCLscan_v1.4gencode.v19.lncRNA_transcripts.fa lrwxrwxrwx 1 106 Nov 3 14:41 gencode.v19.pc_transcripts.fa -> /home/NCLscan_v1.4gencode.v19.pc_transcripts.fa lrwxrwxrwx 1 93 Nov 3 14:41 get_gene_name.py -> /home/NCLscan_v1.4get_gene_name.py lrwxrwxrwx 1 86 Nov 3 14:41 novoalign -> /home/NCLscan_v1.4novoalign lrwxrwxrwx 1 90 Nov 3 14:41 novoalign.lic -> /home/NCLscan_v1.4novoalign.lic lrwxrwxrwx 1 86 Nov 3 14:41 novoindex -> /home/NCLscan_v1.4novoindex lrwxrwxrwx 1 85 Nov 3 14:41 samtools -> /home/NCLscan_v1.4samtools drwxr-sr-x 2 1024 Nov 3 14:42 split_unmap.tmp -rw-r--r-- 1 0 Nov 3 14:41 test_tumor_NCLscan.bwa.sam -rw-r--r-- 1 0 Nov 3 14:42 test_tumor_NCLscan.bwa.unmapped.sam lrwxrwxrwx 1 85 Nov 3 14:41 utils.py -> /home/NCLscan_v1.4utils.py lrwxrwxrwx 1 86 Nov 3 14:47 utils.pyc -> /home/NCLscan_v1.4utils.pyc lrwxrwxrwx 1 84 Nov 3 14:41 version -> /home/NCLscan_v1.4version

The programs are the same as described earlier.

Logfile: Due to its size I could not copy paste it directly so here it is the end with the error:

[M::mem_process_seqs] Processed 594060 reads in 225.666 CPU sec, 37.416 real sec [M::process] read 594060 sequences (60000060 bp)... [M::mem_pestat] # candidate unique pairs for (FF, FR, RF, RR): (4, 169046, 60, 5) [M::mem_pestat] skip orientation FF as there are not enough pairs [M::mem_pestat] analyzing insert size distribution for orientation FR... [M::mem_pestat](25, 50, 75) percentile: (134, 165, 211) [M::mem_pestat] low and high boundaries for computing mean and std.dev: (1, 365) [M::mem_pestat] mean and std.dev: (175.36, 57.04) [M::mem_pestat] low and high boundaries for proper pairs: (1, 442) [M::mem_pestat] analyzing insert size distribution for orientation RF... [M::mem_pestat](25, 50, 75) percentile: (107, 326, 824) [M::mem_pestat] low and high boundaries for computing mean and std.dev: (1, 2258) [M::mem_pestat] mean and std.dev: (451.93, 467.08) [M::mem_pestat] low and high boundaries for proper pairs: (1, 2975) [M::mem_pestat] skip orientation RR as there are not enough pairs [M::mem_pestat] skip orientation RF [M::mem_process_seqs] Processed 594060 reads in 222.926 CPU sec, 36.995 real sec [M::process] read 594060 sequences (60000060 bp)... [M::mem_pestat] # candidate unique pairs for (FF, FR, RF, RR): (4, 169372, 60, 2) [M::mem_pestat] skip orientation FF as there are not enough pairs [M::mem_pestat] analyzing insert size distribution for orientation FR... [M::mem_pestat](25, 50, 75) percentile: (134, 165, 211) [M::mem_pestat] low and high boundaries for computing mean and std.dev: (1, 365) [M::mem_pestat] mean and std.dev: (175.42, 56.91) [M::mem_pestat] low and high boundaries for proper pairs: (1, 442) [M::mem_pestat] analyzing insert size distribution for orientation RF... [M::mem_pestat](25, 50, 75) percentile: (111, 440, 853) [M::mem_pestat] low and high boundaries for computing mean and std.dev: (1, 2337) [M::mem_pestat] mean and std.dev: (505.74, 526.88) [M::mem_pestat] low and high boundaries for proper pairs: (1, 3079) [M::mem_pestat] skip orientation RR as there are not enough pairs [M::mem_pestat] skip orientation RF [M::mem_process_seqs] Processed 594060 reads in 235.751 CPU sec, 39.139 real sec [M::process] read 594060 sequences (60000060 bp)... [M::mem_pestat] # candidate unique pairs for (FF, FR, RF, RR): (2, 168596, 56, 3) [M::mem_pestat] skip orientation FF as there are not enough pairs [M::mem_pestat] analyzing insert size distribution for orientation FR... [M::mem_pestat](25, 50, 75) percentile: (133, 164, 210) [M::mem_pestat] low and high boundaries for computing mean and std.dev: (1, 364) [M::mem_pestat] mean and std.dev: (174.75, 56.64) [M::mem_pestat] low and high boundaries for proper pairs: (1, 441) [M::mem_pestat] analyzing insert size distribution for orientation RF... [M::mem_pestat](25, 50, 75) percentile: (125, 295, 710) [M::mem_pestat] low and high boundaries for computing mean and std.dev: (1, 1880) [M::mem_pestat] mean and std.dev: (368.36, 400.56) [M::mem_pestat] low and high boundaries for proper pairs: (1, 2465) [M::mem_pestat] skip orientation RR as there are not enough pairs [M::mem_pestat] skip orientation RF [M::mem_process_seqs] Processed 594060 reads in 235.503 CPU sec, 39.043 real sec [M::process] read 571646 sequences (57736246 bp)... [M::mem_pestat] # candidate unique pairs for (FF, FR, RF, RR): (3, 169231, 51, 4) [M::mem_pestat] skip orientation FF as there are not enough pairs [M::mem_pestat] analyzing insert size distribution for orientation FR... [M::mem_pestat](25, 50, 75) percentile: (133, 164, 210) [M::mem_pestat] low and high boundaries for computing mean and std.dev: (1, 364) [M::mem_pestat] mean and std.dev: (174.45, 56.51) [M::mem_pestat] low and high boundaries for proper pairs: (1, 441) [M::mem_pestat] analyzing insert size distribution for orientation RF... [M::mem_pestat](25, 50, 75) percentile: (153, 366, 1120) [M::mem_pestat] low and high boundaries for computing mean and std.dev: (1, 3054) [M::mem_pestat] mean and std.dev: (649.31, 756.49) [M::mem_pestat] low and high boundaries for proper pairs: (1, 4021) [M::mem_pestat] skip orientation RR as there are not enough pairs [M::mem_pestat] skip orientation RF [M::mem_process_seqs] Processed 594060 reads in 228.242 CPU sec, 37.852 real sec [M::mem_pestat] # candidate unique pairs for (FF, FR, RF, RR): (6, 162412, 51, 3) [M::mem_pestat] skip orientation FF as there are not enough pairs [M::mem_pestat] analyzing insert size distribution for orientation FR... [M::mem_pestat](25, 50, 75) percentile: (133, 164, 209) [M::mem_pestat] low and high boundaries for computing mean and std.dev: (1, 361) [M::mem_pestat] mean and std.dev: (174.09, 56.03) [M::mem_pestat] low and high boundaries for proper pairs: (1, 437) [M::mem_pestat] analyzing insert size distribution for orientation RF... [M::mem_pestat](25, 50, 75) percentile: (169, 395, 1064) [M::mem_pestat] low and high boundaries for computing mean and std.dev: (1, 2854) [M::mem_pestat] mean and std.dev: (496.34, 514.23) [M::mem_pestat] low and high boundaries for proper pairs: (1, 3749) [M::mem_pestat] skip orientation RR as there are not enough pairs [M::mem_pestat] skip orientation RF [M::mem_process_seqs] Processed 571646 reads in 220.566 CPU sec, 36.885 real sec [main] Version: 0.7.12-r1039 [main] CMD: ./bwa mem -t 6 bwa_AllRef.fa /home/tumor_R1.fastq /home/tumor_R2.fastq [main] Real time: 7875.404 sec; CPU: 47401.150 sec [W::sam_hdr_parse] duplicated sequence 'chrM' Traceback (most recent call last): File "./NCL_Scan0.py", line 43, in main() File "./NCL_Scan0.py", line 38, in main get_bwa_unmapped(args.Fastq_1, args.Fastq_2, args.Project_Name, args.bwa_index_prefix, args.bwa_thread) File "./NCL_Scan0.py", line 12, in get_bwa_unmapped os.mkdir("split_unmap.tmp") OSError: [Errno 17] File exists: 'split_unmap.tmp' /home/NCLscan_v1.4NCLscan.sh: line 14: ./NCL_Scan1: No such file or directory /home/NCLscan_v1.4NCLscan.sh: line 15: ./NCL_Scan2: No such file or directory /home/NCLscan_v1.4NCLscan.sh: line 16: ./NCL_Scan3: No such file or directory NCL_Scan4 v0.2.5 config.tmp can not be found! Please make sure that NCL_Scan2 has been finished properly. Traceback (most recent call last): File "./NCL_Scan5.py", line 28, in pj_name = get_parameter("config.tmp", 2) File "/home/NCLscan_v1.4utils.py", line 24, in get_parameter with open(config_file) as data_reader: IOError: [Errno 2] No such file or directory: 'config.tmp' =========== Stop ============

Tue Nov 3 17:06:22 CET 2015

[chiangtw]

Thanks for the feedback!

I think the problem is on the temp directory "split_unmap.tmp". Normally, it should be deleted automatically.

To solve this error, please remove this temp directory ("split_unmap.tmp") before running NCLscan.

[vincenthahaut]

Hello!

I think this time it's maybe the last question (at least I hope^^).

I ran a batch script (called NCLscan_script.sh) as followed using "sbatch NCLscan_script.sh"

!/bin/sh

SBATCH --ntasks=1

SBATCH --cpus-per-task=8

SBATCH --mem-per-cpu=6G

export PYTHONPATH=/.../argparse-1.2.1/:$PYTHONPATH

cd /.../test_NCLscan/ ln -s /.../NCLscan/NCLscan_v1.4/* . pwd ls -ll echo NCLscan ./NCLscan.sh ../test_NCLscan/simu_5X_100PE_1.fastq ../test_NCLscan/simu_5X_100PE_2.fastq simu_5X_100PE 20 8 50

Unfortunately got this error;

========== Start ===========

ven nov 6 12:22:53 CET 2015

[M::bwa_idx_load_from_disk] read 0 ALT contigs [M::process] read 800000 sequences (80000000 bp)... [M::process] read 800000 sequences (80000000 bp)... [M::mem_pestat] * candidate unique pairs for (FF, FR, RF, RR): (0, 1166, 0, 0) [M::mem_pestat] skip orientation FF as there are not enough pairs [M::mem_pestat] analyzing insert size distribution for orientation FR... [M::mem_pestat](25, 50, 75) percentile: (320, 365, 414) [M::mem_pestat] low and high boundaries for computing mean and std.dev: (132, 602) [M::mem_pestat] mean and std.dev: (365.85, 67.80) [M::mem_pestat] low and high boundaries for proper pairs: (38, 696) [M::mem_pestat] skip orientation RF as there are not enough pairs [M::mem_pestat] skip orientation RR as there are not enough pairs [M::mem_process_seqs] Processed 800000 reads in 291.816 CPU sec, 38.137 real sec [M::process] read 800000 sequences (80000000 bp)... [M::mem_pestat] * candidate unique pairs for (FF, FR, RF, RR): (0, 1249, 0, 0) [M::mem_pestat] skip orientation FF as there are not enough pairs [M::mem_pestat] analyzing insert size distribution for orientation FR... [M::mem_pestat](25, 50, 75) percentile: (318, 366, 410) [M::mem_pestat] low and high boundaries for computing mean and std.dev: (134, 594) [M::mem_pestat] mean and std.dev: (364.18, 67.23) [M::mem_pestat] low and high boundaries for proper pairs: (42, 686) [M::mem_pestat] skip orientation RF as there are not enough pairs [M::mem_pestat] skip orientation RR as there are not enough pairs [M::mem_process_seqs] Processed 800000 reads in 289.539 CPU sec, 36.537 real sec [M::process] read 800000 sequences (80000000 bp)... [M::mem_pestat] * candidate unique pairs for (FF, FR, RF, RR): (0, 1223, 0, 0) [M::mem_pestat] skip orientation FF as there are not enough pairs [M::mem_pestat] analyzing insert size distribution for orientation FR... [M::mem_pestat](25, 50, 75) percentile: (320, 364, 410) [M::mem_pestat] low and high boundaries for computing mean and std.dev: (140, 590) [M::mem_pestat] mean and std.dev: (364.40, 66.86) [M::mem_pestat] low and high boundaries for proper pairs: (50, 680) [M::mem_pestat] skip orientation RF as there are not enough pairs [M::mem_pestat] skip orientation RR as there are not enough pairs [M::mem_process_seqs] Processed 800000 reads in 290.963 CPU sec, 36.696 real sec [M::process] read 800000 sequences (80000000 bp)... [M::mem_pestat] * candidate unique pairs for (FF, FR, RF, RR): (0, 1234, 0, 0) [M::mem_pestat] skip orientation FF as there are not enough pairs [M::mem_pestat] analyzing insert size distribution for orientation FR... [M::mem_pestat](25, 50, 75) percentile: (325, 367, 413) [M::mem_pestat] low and high boundaries for computing mean and std.dev: (149, 589) [M::mem_pestat] mean and std.dev: (367.27, 66.28) [M::mem_pestat] low and high boundaries for proper pairs: (61, 677) [M::mem_pestat] skip orientation RF as there are not enough pairs [M::mem_pestat] skip orientation RR as there are not enough pairs [M::mem_process_seqs] Processed 800000 reads in 294.736 CPU sec, 37.184 real sec [M::process] read 800000 sequences (80000000 bp)... [M::mem_pestat] * candidate unique pairs for (FF, FR, RF, RR): (0, 1225, 0, 0) [M::mem_pestat] skip orientation FF as there are not enough pairs [M::mem_pestat] analyzing insert size distribution for orientation FR... [M::mem_pestat](25, 50, 75) percentile: (318, 366, 410) [M::mem_pestat] low and high boundaries for computing mean and std.dev: (134, 594) [M::mem_pestat] mean and std.dev: (365.77, 66.80) [M::mem_pestat] low and high boundaries for proper pairs: (42, 686) [M::mem_pestat] skip orientation RF as there are not enough pairs [M::mem_pestat] skip orientation RR as there are not enough pairs [M::mem_process_seqs] Processed 800000 reads in 292.950 CPU sec, 38.226 real sec [M::process] read 800000 sequences (80000000 bp)... [M::mem_pestat] * candidate unique pairs for (FF, FR, RF, RR): (0, 1549, 0, 0) [M::mem_pestat] skip orientation FF as there are not enough pairs [M::mem_pestat] analyzing insert size distribution for orientation FR... [M::mem_pestat](25, 50, 75) percentile: (324, 369, 412) [M::mem_pestat] low and high boundaries for computing mean and std.dev: (148, 588) [M::mem_pestat] mean and std.dev: (367.77, 65.12) [M::mem_pestat] low and high boundaries for proper pairs: (60, 676) [M::mem_pestat] skip orientation RF as there are not enough pairs [M::mem_pestat] skip orientation RR as there are not enough pairs [M::mem_process_seqs] Processed 800000 reads in 285.449 CPU sec, 35.861 real sec [M::process] read 800000 sequences (80000000 bp)... [M::mem_pestat] * candidate unique pairs for (FF, FR, RF, RR): (0, 1849, 0, 0) [M::mem_pestat] skip orientation FF as there are not enough pairs [M::mem_pestat] analyzing insert size distribution for orientation FR... [M::mem_pestat](25, 50, 75) percentile: (322, 368, 414) [M::mem_pestat] low and high boundaries for computing mean and std.dev: (138, 598) [M::mem_pestat] mean and std.dev: (367.91, 67.05) [M::mem_pestat] low and high boundaries for proper pairs: (46, 690) [M::mem_pestat] skip orientation RF as there are not enough pairs [M::mem_pestat] skip orientation RR as there are not enough pairs [M::mem_process_seqs] Processed 800000 reads in 280.145 CPU sec, 35.143 real sec [M::process] read 800000 sequences (80000000 bp)... [M::mem_pestat] * candidate unique pairs for (FF, FR, RF, RR): (0, 1854, 0, 0) [M::mem_pestat] skip orientation FF as there are not enough pairs [M::mem_pestat] analyzing insert size distribution for orientation FR... [M::mem_pestat](25, 50, 75) percentile: (322, 370, 413) [M::mem_pestat] low and high boundaries for computing mean and std.dev: (140, 595) [M::mem_pestat] mean and std.dev: (367.74, 65.89) [M::mem_pestat] low and high boundaries for proper pairs: (49, 686) [M::mem_pestat] skip orientation RF as there are not enough pairs [M::mem_pestat] skip orientation RR as there are not enough pairs [M::mem_process_seqs] Processed 800000 reads in 280.615 CPU sec, 35.245 real sec [M::mem_pestat] * candidate unique pairs for (FF, FR, RF, RR): (0, 1932, 0, 0) [M::mem_pestat] skip orientation FF as there are not enough pairs [M::mem_pestat] analyzing insert size distribution for orientation FR... [M::process] read 800000 sequences (80000000 bp)... [M::mem_pestat](25, 50, 75) percentile: (324, 369, 412) [M::mem_pestat] low and high boundaries for computing mean and std.dev: (148, 588) [M::mem_pestat] mean and std.dev: (368.05, 67.10) [M::mem_pestat] low and high boundaries for proper pairs: (60, 676) [M::mem_pestat] skip orientation RF as there are not enough pairs [M::mem_pestat] skip orientation RR as there are not enough pairs [M::mem_process_seqs] Processed 800000 reads in 277.329 CPU sec, 67.321 real sec [M::mem_pestat] * candidate unique pairs for (FF, FR, RF, RR): (0, 680, 0, 0) [M::process] read 800000 sequences (80000000 bp)... [M::mem_pestat] skip orientation FF as there are not enough pairs [M::mem_pestat] analyzing insert size distribution for orientation FR... [M::mem_pestat](25, 50, 75) percentile: (313, 361, 407) [M::mem_pestat] low and high boundaries for computing mean and std.dev: (125, 595) [M::mem_pestat] mean and std.dev: (363.08, 68.53) [M::mem_pestat] low and high boundaries for proper pairs: (31, 689) [M::mem_pestat] skip orientation RF as there are not enough pairs [M::mem_pestat] skip orientation RR as there are not enough pairs [M::mem_process_seqs] Processed 800000 reads in 304.266 CPU sec, 89.473 real sec [M::mem_pestat] * candidate unique pairs for (FF, FR, RF, RR): (0, 492, 0, 0) [M::mem_pestat] skip orientation FF as there are not enough pairs [M::mem_pestat] analyzing insert size distribution for orientation FR... [M::process] read 800000 sequences (80000000 bp)... [M::mem_pestat](25, 50, 75) percentile: (309, 355, 397) [M::mem_pestat] low and high boundaries for computing mean and std.dev: (133, 573) [M::mem_pestat] mean and std.dev: (353.34, 65.91) [M::mem_pestat] low and high boundaries for proper pairs: (45, 661) [M::mem_pestat] skip orientation RF as there are not enough pairs [M::mem_pestat] skip orientation RR as there are not enough pairs [M::mem_process_seqs] Processed 800000 reads in 308.702 CPU sec, 56.223 real sec [M::process] read 75598 sequences (7559800 bp)... [M::mem_pestat] * candidate unique pairs for (FF, FR, RF, RR): (0, 401, 0, 0) [M::mem_pestat] skip orientation FF as there are not enough pairs [M::mem_pestat] analyzing insert size distribution for orientation FR... [M::mem_pestat](25, 50, 75) percentile: (297, 350, 400) [M::mem_pestat] low and high boundaries for computing mean and std.dev: (91, 606) [M::mem_pestat] mean and std.dev: (346.87, 70.52) [M::mem_pestat] low and high boundaries for proper pairs: (1, 709) [M::mem_pestat] skip orientation RF as there are not enough pairs [M::mem_pestat] skip orientation RR as there are not enough pairs [M::mem_process_seqs] Processed 800000 reads in 317.367 CPU sec, 40.032 real sec [M::mem_pestat] * candidate unique pairs for (FF, FR, RF, RR): (0, 34, 0, 0) [M::mem_pestat] skip orientation FF as there are not enough pairs [M::mem_pestat] analyzing insert size distribution for orientation FR... [M::mem_pestat](25, 50, 75) percentile: (301, 354, 391) [M::mem_pestat] low and high boundaries for computing mean and std.dev: (121, 571) [M::mem_pestat] mean and std.dev: (352.56, 77.28) [M::mem_pestat] low and high boundaries for proper pairs: (31, 661) [M::mem_pestat] skip orientation RF as there are not enough pairs [M::mem_pestat] skip orientation RR as there are not enough pairs [M::mem_process_seqs] Processed 75598 reads in 32.149 CPU sec, 4.091 real sec [main] Version: 0.7.12-r1039 [main] CMD: ./bwa mem -t 8 bwa_AllRef.fa ../test_NCLscan/simu_5X_100PE_1.fastq ../test_NCLscan/simu_5X_100PE_2.fastq [main] Real time: 2152.924 sec; CPU: 3551.600 sec [W::sam_hdr_parse] duplicated sequence 'chrM' ./FastqOut: error while loading shared libraries: libstdc++.so.6: cannot open shared object file: No such file or directory cat: erreur d'écriture: Relais brisé (pipe) ./FastqOut: error while loading shared libraries: libstdc++.so.6: cannot open shared object file: No such file or directory cat: erreur d'écriture: Relais brisé (pipe) ./NCL_Scan1: error while loading shared libraries: libstdc++.so.6: cannot open shared object file: No such file or directory ./NCL_Scan2: error while loading shared libraries: libstdc++.so.6: cannot open shared object file: No such file or directory ./NCL_Scan3: error while loading shared libraries: libstdc++.so.6: cannot open shared object file: No such file or directory NCL_Scan4 v0.2.5 config.tmp can not be found! Please make sure that NCL_Scan2 has been finished properly. Traceback (most recent call last): File "./NCL_Scan5.py", line 5, in from utils import * File "......../NCLscan_v1.4/utils.py", line 32 return {key: origin_data[key] for key in wanted_list if key in origin_data} ^ SyntaxError: invalid syntax =========== Stop ============

ven nov 6 13:03:01 CET 2015

On the other hand running directly from the command line:

export PYTHONPATH=/.../argparse-1.2.1/:$PYTHONPATH ./NCLscan_v1.4/NCLscan.sh ../test_NCLscan/simu_5X_100PE_1.fastq ../test_NCLscan/simu_5X_100PE_2.fastq simu_5X_100PE 20 8 50

did not create this error but failed at the last step.

I suppose it is due to the way sbatch and the cluster I'm working on is handling the original script but would you have any idea to solve it? (I need to analyse 150 tumors samples and doing them one by one would be an issue).

Thanks in advance,

Vincent

Linux rfe2.... 2.6.32-431.11.2.el6.x86_64 *1 SMP Tue Mar 25 19:59:55 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Python 2.6.6 (r266:84292, Jan 22 2014, 09:42:36) [GCC 4.4.7 20120313 (Red Hat 4.4.7-4)] on linux2 samtools-1.2 bedtools2 novoalign V3.03.01 gcc version 4.4.7 20120313 (Red Hat 4.4.7-4) (GCC) echo $LD_LIBRARY_PATH ==> /opt/openmpi/lib:/opt/python/lib

EDIT: here is the log when I run it from the command line:

========== Start ===========

ven nov 6 14:04:17 CET 2015

[M::bwa_idx_load_from_disk] read 0 ALT contigs [M::process] read 5000000 sequences (500000000 bp)... [M::mem_pestat] * candidate unique pairs for (FF, FR, RF, RR): (0, 8114, 0, 0) [M::mem_pestat] skip orientation FF as there are not enough pairs [M::mem_pestat] analyzing insert size distribution for orientation FR... [M::mem_pestat](25, 50, 75) percentile: (321, 366, 412) [M::mem_pestat] low and high boundaries for computing mean and std.dev: (139, 594) [M::mem_pestat] mean and std.dev: (365.99, 66.73) [M::mem_pestat] low and high boundaries for proper pairs: (48, 685) [M::mem_pestat] skip orientation RF as there are not enough pairs [M::mem_pestat] skip orientation RR as there are not enough pairs [M::mem_process_seqs] Processed 5000000 reads in 2189.549 CPU sec, 842.674 real sec [M::process] read 4675598 sequences (467559800 bp)... [M::mem_pestat] * candidate unique pairs for (FF, FR, RF, RR): (0, 6774, 0, 0) [M::mem_pestat] skip orientation FF as there are not enough pairs [M::mem_pestat] analyzing insert size distribution for orientation FR... [M::mem_pestat](25, 50, 75) percentile: (319, 365, 410) [M::mem_pestat] low and high boundaries for computing mean and std.dev: (137, 592) [M::mem_pestat] mean and std.dev: (365.23, 67.20) [M::mem_pestat] low and high boundaries for proper pairs: (46, 683) [M::mem_pestat] skip orientation RF as there are not enough pairs [M::mem_pestat] skip orientation RR as there are not enough pairs [M::mem_process_seqs] Processed 4675598 reads in 2077.856 CPU sec, 363.064 real sec [main] Version: 0.7.12-r1039 [main] CMD: ./bwa mem -t 50 bwa_AllRef.fa ./simu_5X_100PE_1.fastq ./simu_5X_100PE_2.fastq [main] Real time: 1594.091 sec; CPU: 4374.047 sec [W::sam_hdr_parse] duplicated sequence 'chrM'

• novoindex (3.3) - Universal k-mer index constructor.
• (C) 2008 - 2011 NovoCraft Technologies Sdn Bhd
• novoindex AllRef.ndx AllRef.fa
• Creating 8 indexing threads.
• Building with 14-mer and step of 2 bp. tcmalloc: large alloc 1073750016 bytes == 0x28ec000 @ 0x503a3c 0x401841 0x40065a 0x491720 0x400c99 tcmalloc: large alloc 9237315584 bytes == 0x428f6000 @ 0x504978 0x401a18 0x40065a 0x491720 0x400c99
• novoindex construction dT = 922.5s
• Index memory size 8.603Gbyte.
• Done.
• novoalign (V3.03.01 - Build Oct 28 2015 @ 17:09:26) - A short read aligner with qualities.
• (C) 2008-2015 Novocraft Technologies Sdn Bhd.
• License file: novoalign.lic
• Licensed to
• novoalign -r A 1 -n 30 -d AllRef.ndx -f 20.bwa.unmapped_1.fastq 20.bwa.unmapped_2.fastq -o SAM
• Starting at Fri Nov 6 14:57:39 2015
• Interpreting input files as Sanger FASTQ.
• Index Build Version: 3.3
• Hash length: 14
• Step size: 2
• Paired Reads: 559
• Proper Pairs: 7 ( 1.3%)
• Read Sequences: 1118
• Unique Alignment: 2 ( 0.2%)
• Multi Mapped: 1112 (99.5%)
• No Mapping Found: 4 ( 0.4%)
• Elapsed Time: 10.640 (secs.)
• CPU Time: 17.54 (secs.)
• Fragment Length Distribution
• From To Count
• No pairs found
• Done at Fri Nov 6 15:02:17 2015 Loaded 3234843780 letters in 298 sequences Searched 101727 bases in 524 sequences PslChimeraFilter v0.4

Loaded 196542248 letters in 95309 sequences Searched 81922 bases in 422 sequences Loaded 22819188 letters in 23898 sequences Searched 81922 bases in 422 sequences Loaded 33138 letters in 1 sequences Searched 81922 bases in 422 sequences JunctionSite2BED v0.3

Reading annotations on chr1. Reading annotations on chr2. Reading annotations on chr3. Reading annotations on chr4. Reading annotations on chr5. Reading annotations on chr6. Reading annotations on chr7. Reading annotations on chr8. Reading annotations on chr9. Reading annotations on chr10. Reading annotations on chr11. Reading annotations on chr12. Reading annotations on chr13. Reading annotations on chr14. Reading annotations on chr15. Reading annotations on chr16. Reading annotations on chr17. Reading annotations on chr18. Reading annotations on chr19. Reading annotations on chr20. Reading annotations on chr21. Reading annotations on chr22. Reading annotations on chrX. Reading annotations on chrY. Reading annotations on chrM. Read 57820 genes, 196520 transcripts and 1196293 exons from the gtf file. index file GRCh37.p13.genome_HTLV.fa.fai not found, generating... NCL_Scan3 v0.5 ./NCL_Scan3 (null)

• novoindex (3.3) - Universal k-mer index constructor.
• (C) 2008 - 2011 NovoCraft Technologies Sdn Bhd
• novoindex 20.JS.ndx 20.JS.fa
• Creating 8 indexing threads.
• Building with 7-mer and step of 1 bp.
• novoindex construction dT = 0.1s
• Index memory size 0.000Gbyte.
• Done. rm: cannot remove « *.JS.sam »: no such file or directory
• novoalign (V3.03.01 - Build Oct 28 2015 @ 17:09:26) - A short read aligner with qualities.
• (C) 2008-2015 Novocraft Technologies Sdn Bhd.
• License file: novoalign.lic
• Licensed to
• novoalign -r A 1 -g 99 -x 99 -d 20.JS.ndx -f 20.unmapped_1.fastq -o SAM
• Starting at Fri Nov 6 15:11:19 2015
• Interpreting input files as Sanger FASTQ.
• Index Build Version: 3.3
• Hash length: 7
• Step size: 1
• Read Sequences: 524
• Unique Alignment: 82 (15.6%)
• Multi Mapped: 57 (10.9%)
• No Mapping Found: 385 (73.5%)
• Elapsed Time: 0.158 (secs.)
• CPU Time: 0.34 (secs.)
• Done at Fri Nov 6 15:11:19 2015
• novoalign (V3.03.01 - Build Oct 28 2015 @ 17:09:26) - A short read aligner with qualities.
• (C) 2008-2015 Novocraft Technologies Sdn Bhd.
• License file: novoalign.lic
• Licensed to
• novoalign -r A 1 -g 99 -x 99 -d 20.JS.ndx -f 20.unmapped_2.fastq -o SAM
• Starting at Fri Nov 6 15:11:19 2015
• Interpreting input files as Sanger FASTQ.
• Index Build Version: 3.3
• Hash length: 7
• Step size: 1
• Read Sequences: 524
• Unique Alignment: 87 (16.6%)
• Multi Mapped: 51 ( 9.7%)
• No Mapping Found: 386 (73.7%)
• Elapsed Time: 0.164 (secs.)
• CPU Time: 0.34 (secs.)
• Done at Fri Nov 6 15:11:20 2015 Loaded 3234843780 letters in 298 sequences Searched 12382 bases in 41 sequences Loaded 3234843780 letters in 298 sequences Searched 8076 bases in 41 sequences PslChimeraFilter v0.4

JunctionSite2BED v0.3

Reading annotations on chr1. Reading annotations on chr2. Reading annotations on chr3. Reading annotations on chr4. Reading annotations on chr5. Reading annotations on chr6. Reading annotations on chr7. Reading annotations on chr8. Reading annotations on chr9. Reading annotations on chr10. Reading annotations on chr11. Reading annotations on chr12. Reading annotations on chr13. Reading annotations on chr14. Reading annotations on chr15. Reading annotations on chr16. Reading annotations on chr17. Reading annotations on chr18. Reading annotations on chr19. Reading annotations on chr20. Reading annotations on chr21. Reading annotations on chr22. Reading annotations on chrX. Reading annotations on chrY. Reading annotations on chrM. Read 57820 genes, 196520 transcripts and 1196293 exons from the gtf file.

• novoindex (3.3) - Universal k-mer index constructor.
• (C) 2008 - 2011 NovoCraft Technologies Sdn Bhd
• novoindex 20.JS2.ndx 20.JS2.fa
• Creating 8 indexing threads.
• Building with 8-mer and step of 1 bp.
• novoindex construction dT = 0.1s
• Index memory size 0.002Gbyte.
• Done. NCL_Scan4 v0.2.5 rm: cannot remove « *.JS2.sam »: no such file or directory
• novoalign (V3.03.01 - Build Oct 28 2015 @ 17:09:26) - A short read aligner with qualities.
• (C) 2008-2015 Novocraft Technologies Sdn Bhd.
• License file: novoalign.lic
• Licensed to
• novoalign -r A 1 -t 0,1 -d 20.JS2.ndx -f 20.unmapped_1.fastq 20.unmapped_2.fastq --3Prime -o SAM
• Starting at Fri Nov 6 15:23:59 2015
• Interpreting input files as Sanger FASTQ.
• Index Build Version: 3.3
• Hash length: 8
• Step size: 1
• Paired Reads: 524
• Proper Pairs: 208 (39.7%)
• Read Sequences: 1048
• Unique Alignment: 96 ( 9.2%)
• Multi Mapped: 334 (31.9%)
• No Mapping Found: 618 (59.0%)
• Elapsed Time: 0.427 (secs.)
• CPU Time: 1.23 (secs.)
• Fragment Length Distribution
• From To Count
• 195 209 1
• 210 224 0
• 225 239 1
• 240 254 1
• 255 269 1
• 270 284 1
• 285 299 1
• 300 314 3
• 315 329 2
• 330 344 5
• 345 359 5
• 360 374 3
• 375 389 2
• 390 404 9
• 405 419 4
• 420 434 0
• 435 449 3
• 450 464 2
• 465 479 2
• 480 494 1
• Mean 368, Std Dev 63.8
• Done at Fri Nov 6 15:23:59 2015 rm: cannot remove « .um3.fastq »: no such file or directory rm: cannot remove « .um3.fa »: no such file or directory rm: cannot remove « .um3.psl »: no such file or directory Loaded 3234843780 letters in 298 sequences Searched 36599 bases in 188 sequences Loaded 196542248 letters in 95309 sequences Searched 36599 bases in 188 sequences Loaded 22819188 letters in 23898 sequences Searched 36599 bases in 188 sequences Loaded 33138 letters in 1 sequences Searched 36599 bases in 188 sequences rm: cannot remove « .JS2b.sam »: no such file or directory Traceback (most recent call last): File "./NCL_Scan5.py", line 5, in from utils import * File « …/NCLscan/NCLscan_v1.4/utils.py", line 32 return {key: origin_data[key] for key in wanted_list if key in origin_data} ^ SyntaxError: invalid syntax

[vincenthahaut]

Just for the record I think I found another issue in the NCL_Scan0.py file.

I installed everything on macOs (Yosemite 10.10.5) and used the test dataset to run: ./NCLscan.sh simu_5X_100PE_1.fastq simu_5X_100PE_2.fastq 20 8 50 ========== Start ===========

Ven 6 nov 2015 16:28:53 CET

[M::bwa_idx_load_from_disk] read 0 ALT contigs [M::process] read 5000000 sequences (500000000 bp)... [M::process] read 4675598 sequences (467559800 bp)... [M::mem_pestat] # candidate unique pairs for (FF, FR, RF, RR): (0, 361639, 16, 0) [M::mem_pestat] skip orientation FF as there are not enough pairs [M::mem_pestat] analyzing insert size distribution for orientation FR... [M::mem_pestat](25, 50, 75) percentile: (326, 370, 415) [M::mem_pestat] low and high boundaries for computing mean and std.dev: (148, 593) [M::mem_pestat] mean and std.dev: (370.34, 65.98) [M::mem_pestat] low and high boundaries for proper pairs: (59, 682) [M::mem_pestat] analyzing insert size distribution for orientation RF... [M::mem_pestat](25, 50, 75) percentile: (251, 759, 810) [M::mem_pestat] low and high boundaries for computing mean and std.dev: (1, 1928) [M::mem_pestat] mean and std.dev: (613.69, 367.70) [M::mem_pestat] low and high boundaries for proper pairs: (1, 2487) [M::mem_pestat] skip orientation RR as there are not enough pairs [M::mem_pestat] skip orientation RF [M::mem_process_seqs] Processed 5000000 reads in 1191.719 CPU sec, 359.031 real sec [M::mem_pestat] # candidate unique pairs for (FF, FR, RF, RR): (0, 311669, 0, 0) [M::mem_pestat] skip orientation FF as there are not enough pairs [M::mem_pestat] analyzing insert size distribution for orientation FR... [M::mem_pestat](25, 50, 75) percentile: (325, 370, 415) [M::mem_pestat] low and high boundaries for computing mean and std.dev: (145, 595) [M::mem_pestat] mean and std.dev: (370.02, 66.08) [M::mem_pestat] low and high boundaries for proper pairs: (55, 685) [M::mem_pestat] skip orientation RF as there are not enough pairs [M::mem_pestat] skip orientation RR as there are not enough pairs [M::mem_process_seqs] Processed 4675598 reads in 1159.401 CPU sec, 337.443 real sec [main] Version: 0.7.12-r1039 [main] CMD: ./bwa mem -t 50 bwa_AllRef.fa simu_5X_100PE_1.fastq simu_5X_100PE_2.fastq [main] Real time: 717.651 sec; CPU: 2370.396 sec split: illegal option -- C usage: split [-a sufflen] [-b byte_count] [-l line_count] [-p pattern] [file [prefix]]

Maybe something should be fixed here too.

[chiangtw]

Please try the latest version(v1.6) of NCLscan.