Issue running KNIFE

[Kevinzjy]

I am trying to use the test data to verify the installation, but can't get any result under circReads/glmReports. The results in OUTPUT_DIRECTORY/sampleStats seems ok

Here's my pipeline: cd $HOME/local/software/KNIFE/circularRNApipeline_StandAlone sh completeRun.sh $HOME/data/testData complete $HOME/data/ testData 8 phred64 circReads 40 2>&1 | tee out.log

Thanks

[lindaszabo]

If you attached a log file, I'm not able to see it.

[Kevinzjy]

sorry, here's my log file

[Kevinzjy]

sorry, something went wrong and i can't just upload it

./findCircularRNA.sh "/home/zhangjy/data/testData complete /home/zhangjy/data/ testData 8 phred64 circReads 40" MODE is phred64 Launched 3 align.sh's into the background Sat Sep 26 21:29:17 CST 2015 MODE is phred64 Launched 3 align.sh's into the background Sat Sep 26 21:29:17 CST 2015 23788484 reads; of these: 23788484 (100.00%) were unpaired; of these: 11248931 (47.29%) aligned 0 times 12538775 (52.71%) aligned exactly 1 time 778 (0.00%) aligned >1 times 52.71% overall alignment rate 37293080 reads; of these: 37293080 (100.00%) were unpaired; of these: 17870944 (47.92%) aligned 0 times 19421269 (52.08%) aligned exactly 1 time 867 (0.00%) aligned >1 times 52.08% overall alignment rate 23788484 reads; of these: 23788484 (100.00%) were unpaired; of these: 4972142 (20.90%) aligned 0 times 14526439 (61.07%) aligned exactly 1 time 4289903 (18.03%) aligned >1 times 79.10% overall alignment rate Done looping over bowtie, awaiting the last few align.sh to finish Sat Sep 26 21:53:46 CST 2015 23788484 reads; of these: 23788484 (100.00%) were unpaired; of these: 9978283 (41.95%) aligned 0 times 9416827 (39.59%) aligned exactly 1 time 4393374 (18.47%) aligned >1 times 58.05% overall alignment rate 37293080 reads; of these: 37293080 (100.00%) were unpaired; of these: 8447839 (22.65%) aligned 0 times 22436356 (60.16%) aligned exactly 1 time 6408885 (17.19%) aligned >1 times 77.35% overall alignment rate 37293080 reads; of these: 37293080 (100.00%) were unpaired; of these: 16440930 (44.09%) aligned 0 times 14217007 (38.12%) aligned exactly 1 time 6635143 (17.79%) aligned >1 times 55.91% overall alignment rate All non-junction align.sh are doneSat Sep 26 22:19:27 CST 2015 starting junction align Launched 2 junction aligns into the background Sat Sep 26 22:19:27 CST 2015 23788484 reads; of these: 23788484 (100.00%) were unpaired; of these: 21151627 (88.92%) aligned 0 times 626296 (2.63%) aligned exactly 1 time 2010561 (8.45%) aligned >1 times 11.08% overall alignment rate 23788484 reads; of these: 23788484 (100.00%) were unpaired; of these: 21764359 (91.49%) aligned 0 times 41640 (0.18%) aligned exactly 1 time 1982485 (8.33%) aligned >1 times 8.51% overall alignment rate Launched 2 junction aligns into the background Sat Sep 26 22:26:45 CST 2015 37293080 reads; of these: 37293080 (100.00%) were unpaired; of these: 33264909 (89.20%) aligned 0 times 865535 (2.32%) aligned exactly 1 time 3162636 (8.48%) aligned >1 times 10.80% overall alignment rate 37293080 reads; of these: 37293080 (100.00%) were unpaired; of these: 34120244 (91.49%) aligned 0 times 50765 (0.14%) aligned exactly 1 time 3122071 (8.37%) aligned >1 times 8.51% overall alignment rate All junction align.sh are doneSat Sep 26 22:36:31 CST 2015 ready to preprocess Launched preprocessAlignedReads.sh into the background Sat Sep 26 22:36:31 CST 2015 Launched preprocessAlignedReads.sh into the background Sat Sep 26 22:36:31 CST 2015 Done looping over preprocessing, awaiting the last few preprocessAlignedReads.sh to finish Sat Sep 26 22:36:31 CST 2015 Launched filterFDR.sh into the background Sat Sep 26 22:47:46 CST 2015 Launched filterFDR.sh into the background Sat Sep 26 22:47:46 CST 2015 SAMPLE ID: SRR1027187_1 SAMPLE_MATE_ID: SRR1027187_2 READ_NUM: 1 ALIGNED_SAMPLE_ID: SRR1027187_1 ALIGNED_SAMPLE_MATE_ID: SRR1027187_2 Done looping over filterFDR, awaiting the last few filterFDR.sh to finish Sat Sep 26 22:47:46 CST 2015 parentDir: /home/zhangjy/data//testData sampleId: SRR1027187_1 outDirName: circReads style: complete a1: None a2: None overhang: 8 id dir suffix: unaligned: False ignoreIds (aligned to genome): 30354792 lienarIds: 852460 scrambledIds: 3501 samFile:/home/zhangjy/data//testData/orig/reg/SRR1027187_1_reg_output.sam readType:r1rj samFile:/home/zhangjy/data//testData/orig/junction/SRR1027187_1_junction_output.sam readType:r1j samFile:/home/zhangjy/data//testData/orig/genome/SRR1027187_2_genome_output.sam readType:gMate samFile:/home/zhangjy/data//testData/orig/reg/SRR1027187_2_reg_output.sam readType:rjMate samFile:/home/zhangjy/data//testData/orig/junction/SRR1027187_2_junction_output.sam readType:jMate sample id: SRR1027187_1 single end? False num junctions with aligned reads: 123797 num reads: 855961 number of reads kept in each category after update (multimapped & unknown should be 0, anomaly & decoy should be 0 for SE data): circ: 1267 , decoy: 187 , linear: 562052 , anomaly: 18534 , unmapped: 273509 , multimapped: 0 , unknown: 0 855549 using decoy rate: 0.0123924890027 Completed 1 call to findCircularRNA.sh Sat Sep 26 22:57:46 CST 2015 ./parseForAnalysis.sh /home/zhangjy/data/ testData phred64 READ_NUM 2 READ_NUM 1 JUNC_SAM /home/zhangjy/data//testData/orig/junction/SRR1027187_1_junction_output.sam REG_SAM /home/zhangjy/data//testData/orig/reg/SRR1027187_1_reg_output.sam ./predictJunctions.sh /home/zhangjy/data/ testData phred64 circReads ./predictJunctions_tableData.r /home/zhangjy/data//testData/orig/temp/SRR1027187_1_reg_output.txt /home/zhangjy/data//testData/orig/temp/SRR1027187_1_junction_output.txt /home/zhangjy/data//testData/circReads/ids/SRR1027187_1output.txt /home/zhangjy/data//testData/circReads/glmModels/SRR1027187_1__glm.RData /home/zhangjy/data//testData/circReads/glmReports/SRR1027187_1linearJuncProbs.txt /home/zhangjy/data//testData/circReads/glmReports/SRR1027187_1circJuncProbs.txt ./predictJunctions.sh: line 36: ./predictJunctions_tableData.r: Permission denied ./qualityStatsAll.sh /home/zhangjy/data/ testData circReads 40 Launched qualityStatsSingleSample.sh into the background Sat Sep 26 23:03:21 CST 2015 Launched qualityStatsSingleSample.sh into the background Sat Sep 26 23:03:21 CST 2015 Done looping over qualityStatsSingleSample, awaiting the last few qualityStatsSingleSample.sh to finish Sat Sep 26 23:03:21 CST 2015 find: `/home/zhangjy/data//testData/circReads/ids/SRR1027187_2output.txt': No such file or directory perl process_directory_unaligned.pl /home/zhangjy/data/ testData phred64 40 1 mkdir -p /home/zhangjy/data//testData/denovo_script_out/FILES are unaligned_SRR1027187_1.fq start is and name is SRR1027187_1.fqperl align_unaligned_in_pieces_step2.pl SRR1027187_1.fq /home/zhangjy/data//testData/orig/unaligned/forDenovoIndex/unaligned_SRR1027187_1.fq index/hg19_genome 40 /home/zhangjy/data//testData/denovo_script_out/OPENING input file is /home/zhangjy/data//testData/orig/unaligned/forDenovoIndex/unaligned_SRR1027187_1.fq sample is and COMPLETED ALIGNMENTS files are /home/zhangjy/data//testData/denovo_script_out/3prime_40_SRR1027187_1.fq.out and /home/zhangjy/data//testData/denovo_script_out/5prime_40_SRR1027187_1.fq.out and threeprimeinputfile is /home/zhangjy/data//testData/denovo_script_out/3prime_40_SRR1027187_1.fq.out perl stat_read_matchup_3a.pl /home/zhangjy/data//testData/denovo_script_out/5prime_40_SRR1027187_1.fq.out /home/zhangjy/data//testData/denovo_script_out/3prime_40_SRR1027187_1.fq.out 1 /home/zhangjy/data//testData/orig/unaligned/forDenovoIndex/unaligned_SRR1027187_1.fq testData /home/zhangjy/data//testData/denovo_script_out/ > /home/zhangjy/data//testData/denovo_script_out/testtestData perl test_stats_step2_3c.pl /home/zhangjy/data//testData/denovo_script_out/ testData > /home/zhangjy/data//testData/denovo_script_out/testnewtestData perl retreive_represent.pl /home/zhangjy/data//testData/orig/unaligned/forDenovoIndex/unaligned_SRR1027187_1.fq /home/zhangjy/data//testData/denovo_script_out/testnewSRR1027187_1.fq /home/zhangjy/data//testData/denovo_script_out/output_SRR1027187_1.fq STARTING TO RETRIEVE STARTING TO RETRIEVE STARTING TO RETRIEVE Settings: Output files: "denovo_testData1..bt2" Line rate: 6 (line is 64 bytes) Lines per side: 1 (side is 64 bytes) Offset rate: 4 (one in 16) FTable chars: 10 Strings: unpacked Max bucket size: default Max bucket size, sqrt multiplier: default Max bucket size, len divisor: 4 Difference-cover sample period: 1024 Endianness: little Actual local endianness: little Sanity checking: disabled Assertions: disabled Random seed: 0 Sizeofs: void_:8, int:4, long:8, size_t:8 Input files DNA, FASTA: denovo_testData_onlycircles1.fa Building a SMALL index Reading reference sizes Time reading reference sizes: 00:00:00 Calculating joined length Writing header Reserving space for joined string Joining reference sequences Time to join reference sequences: 00:00:00 bmax according to bmaxDivN setting: 30 Using parameters --bmax 23 --dcv 1024 Doing ahead-of-time memory usage test Passed! Constructing with these parameters: --bmax 23 --dcv 1024 Constructing suffix-array element generator Building DifferenceCoverSample Building sPrime Building sPrimeOrder V-Sorting samples V-Sorting samples time: 00:00:00 Allocating rank array Ranking v-sort output Ranking v-sort output time: 00:00:00 Invoking Larsson-Sadakane on ranks Invoking Larsson-Sadakane on ranks time: 00:00:00 Sanity-checking and returning Building samples Reserving space for 12 sample suffixes Generating random suffixes QSorting 12 sample offsets, eliminating duplicates QSorting sample offsets, eliminating duplicates time: 00:00:00 Multikey QSorting 11 samples (Using difference cover) Multikey QSorting samples time: 00:00:00 Calculating bucket sizes Binary sorting into buckets 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% Binary sorting into buckets time: 00:00:00 Splitting and merging Splitting and merging time: 00:00:00 Split 1, merged 4; iterating... Binary sorting into buckets 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% Binary sorting into buckets time: 00:00:00 Splitting and merging Splitting and merging time: 00:00:00 Avg bucket size: 14.125 (target: 22) Converting suffix-array elements to index image Allocating ftab, absorbFtab Entering Ebwt loop Getting block 1 of 8 Reserving size (23) for bucket Calculating Z arrays Calculating Z arrays time: 00:00:00 Entering block accumulator loop: 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% Block accumulator loop time: 00:00:00 Sorting block of length 3 (Using difference cover) Sorting block time: 00:00:00 Returning block of 4 Getting block 2 of 8 Reserving size (23) for bucket Calculating Z arrays Calculating Z arrays time: 00:00:00 Entering block accumulator loop: 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% Block accumulator loop time: 00:00:00 Sorting block of length 22 (Using difference cover) Sorting block time: 00:00:00 Returning block of 23 Getting block 3 of 8 Reserving size (23) for bucket Calculating Z arrays Calculating Z arrays time: 00:00:00 Entering block accumulator loop: 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% Block accumulator loop time: 00:00:00 Sorting block of length 22 (Using difference cover) Sorting block time: 00:00:00 Returning block of 23 Getting block 4 of 8 Reserving size (23) for bucket Calculating Z arrays Calculating Z arrays time: 00:00:00 Entering block accumulator loop: 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% Block accumulator loop time: 00:00:00 Sorting block of length 12 (Using difference cover) Sorting block time: 00:00:00 Returning block of 13 Getting block 5 of 8 Reserving size (23) for bucket Calculating Z arrays Calculating Z arrays time: 00:00:00 Entering block accumulator loop: 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% Block accumulator loop time: 00:00:00 Sorting block of length 19 (Using difference cover) Sorting block time: 00:00:00 Returning block of 20 Getting block 6 of 8 Reserving size (23) for bucket Calculating Z arrays Calculating Z arrays time: 00:00:00 Entering block accumulator loop: 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% Block accumulator loop time: 00:00:00 Sorting block of length 13 (Using difference cover) Sorting block time: 00:00:00 Returning block of 14 Getting block 7 of 8 Reserving size (23) for bucket Calculating Z arrays Calculating Z arrays time: 00:00:00 Entering block accumulator loop: 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% Block accumulator loop time: 00:00:00 Sorting block of length 21 (Using difference cover) Sorting block time: 00:00:00 Returning block of 22 Getting block 8 of 8 Reserving size (23) for bucket Calculating Z arrays Calculating Z arrays time: 00:00:00 Entering block accumulator loop: 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% Block accumulator loop time: 00:00:00 Sorting block of length 1 (Using difference cover) Sorting block time: 00:00:00 Returning block of 2 Exited Ebwt loop fchr[A]: 0 fchr[C]: 15 fchr[G]: 62 fchr[T]: 94 fchr[$]: 120 Exiting Ebwt::buildToDisk() Returning from initFromVector Wrote 4194716 bytes to primary EBWT file: denovo_testData_1.1.bt2 Wrote 36 bytes to secondary EBWT file: denovo_testData_1.2.bt2 Re-opening _in1 and _in2 as input streams Returning from Ebwt constructor Headers: len: 120 bwtLen: 121 sz: 30 bwtSz: 31 lineRate: 6 offRate: 4 offMask: 0xfffffff0 ftabChars: 10 eftabLen: 20 eftabSz: 80 ftabLen: 1048577 ftabSz: 4194308 offsLen: 8 offsSz: 32 lineSz: 64 sideSz: 64 sideBwtSz: 48 sideBwtLen: 192 numSides: 1 numLines: 1 ebwtTotLen: 64 ebwtTotSz: 64 color: 0 reverse: 0 Total time for call to driver() for forward index: 00:00:01 Reading reference sizes Time reading reference sizes: 00:00:00 Calculating joined length Writing header Reserving space for joined string Joining reference sequences Time to join reference sequences: 00:00:00 Time to reverse reference sequence: 00:00:00 bmax according to bmaxDivN setting: 30 Using parameters --bmax 23 --dcv 1024 Doing ahead-of-time memory usage test Passed! Constructing with these parameters: --bmax 23 --dcv 1024 Constructing suffix-array element generator Building DifferenceCoverSample Building sPrime Building sPrimeOrder V-Sorting samples V-Sorting samples time: 00:00:00 Allocating rank array Ranking v-sort output Ranking v-sort output time: 00:00:00 Invoking Larsson-Sadakane on ranks Invoking Larsson-Sadakane on ranks time: 00:00:00 Sanity-checking and returning Building samples Reserving space for 12 sample suffixes Generating random suffixes QSorting 12 sample offsets, eliminating duplicates QSorting sample offsets, eliminating duplicates time: 00:00:00 Multikey QSorting 11 samples (Using difference cover) Multikey QSorting samples time: 00:00:00 Calculating bucket sizes Binary sorting into buckets 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% Binary sorting into buckets time: 00:00:00 Splitting and merging Splitting and merging time: 00:00:00 Avg bucket size: 19.1667 (target: 22) Converting suffix-array elements to index image Allocating ftab, absorbFtab Entering Ebwt loop Getting block 1 of 6 Reserving size (23) for bucket Calculating Z arrays Calculating Z arrays time: 00:00:00 Entering block accumulator loop: 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% Block accumulator loop time: 00:00:00 Sorting block of length 19 (Using difference cover) Sorting block time: 00:00:00 Returning block of 20 Getting block 2 of 6 Reserving size (23) for bucket Calculating Z arrays Calculating Z arrays time: 00:00:00 Entering block accumulator loop: 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% Block accumulator loop time: 00:00:00 Sorting block of length 21 (Using difference cover) Sorting block time: 00:00:00 Returning block of 22 Getting block 3 of 6 Reserving size (23) for bucket Calculating Z arrays Calculating Z arrays time: 00:00:00 Entering block accumulator loop: 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% Block accumulator loop time: 00:00:00 Sorting block of length 20 (Using difference cover) Sorting block time: 00:00:00 Returning block of 21 Getting block 4 of 6 Reserving size (23) for bucket Calculating Z arrays Calculating Z arrays time: 00:00:00 Entering block accumulator loop: 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% Block accumulator loop time: 00:00:00 Sorting block of length 12 (Using difference cover) Sorting block time: 00:00:00 Returning block of 13 Getting block 5 of 6 Reserving size (23) for bucket Calculating Z arrays Calculating Z arrays time: 00:00:00 Entering block accumulator loop: 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% Block accumulator loop time: 00:00:00 Sorting block of length 21 (Using difference cover) Sorting block time: 00:00:00 Returning block of 22 Getting block 6 of 6 Reserving size (23) for bucket Calculating Z arrays Calculating Z arrays time: 00:00:00 Entering block accumulator loop: 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% Block accumulator loop time: 00:00:00 Sorting block of length 22 (Using difference cover) Sorting block time: 00:00:00 Returning block of 23 Exited Ebwt loop fchr[A]: 0 fchr[C]: 15 fchr[G]: 62 fchr[T]: 94 fchr[$]: 120 Exiting Ebwt::buildToDisk() Returning from initFromVector Wrote 4194716 bytes to primary EBWT file: denovo_testData_1.rev.1.bt2 Wrote 36 bytes to secondary EBWT file: denovo_testData_1.rev.2.bt2 Re-opening _in1 and _in2 as input streams Returning from Ebwt constructor Headers: len: 120 bwtLen: 121 sz: 30 bwtSz: 31 lineRate: 6 offRate: 4 offMask: 0xfffffff0 ftabChars: 10 eftabLen: 20 eftabSz: 80 ftabLen: 1048577 ftabSz: 4194308 offsLen: 8 offsSz: 32 lineSz: 64 sideSz: 64 sideBwtSz: 48 sideBwtLen: 192 numSides: 1 numLines: 1 ebwtTotLen: 64 ebwtTotSz: 64 color: 0 reverse: 1 Total time for backward call to driver() for mirror index: 00:00:00 ./findCircularRNA.sh /home/zhangjy/data/testData complete /home/zhangjy/data/ testData 8 phred64_unaligned circReads 40 1
MODE is phred64_unaligned Launched align into the background Sat Sep 26 23:21:07 CST 2015 MODE is phred64_unaligned Launched align into the background Sat Sep 26 23:21:07 CST 2015 Done looping over bowtie, awaiting the last few align.sh to finish Sat Sep 26 23:21:08 CST 2015 3072689 reads; of these: 3072689 (100.00%) were unpaired; of these: 3072680 (100.00%) aligned 0 times 9 (0.00%) aligned exactly 1 time 0 (0.00%) aligned >1 times 0.00% overall alignment rate 5826686 reads; of these: 5826686 (100.00%) were unpaired; of these: 5826613 (100.00%) aligned 0 times 73 (0.00%) aligned exactly 1 time 0 (0.00%) aligned >1 times 0.00% overall alignment rate All non-junction align.sh are doneSat Sep 26 23:23:22 CST 2015 ready to preprocess Launched preprocessAlignedReads.sh into the background Sat Sep 26 23:23:22 CST 2015 Launched preprocessAlignedReads.sh into the background Sat Sep 26 23:23:22 CST 2015 Done looping over preprocessing, awaiting the last few preprocessAlignedReads.sh to finish Sat Sep 26 23:23:22 CST 2015 Launched filterFDR.sh into the background Sat Sep 26 23:28:22 CST 2015 Launched filterFDR.sh into the background Sat Sep 26 23:28:22 CST 2015 SAMPLE ID: unaligned_SRR1027187_1 SAMPLE_MATE_ID: unaligned_SRR1027187_2 READ_NUM: 1 ALIGNED_SAMPLE_ID: SRR1027187_1 ALIGNED_SAMPLE_MATE_ID: SRR1027187_2 Done looping over filterFDR, awaiting the last few filterFDR.sh to finish Sat Sep 26 23:28:23 CST 2015 parentDir: /home/zhangjy/data//testData sampleId: unaligned_SRR1027187_1 outDirName: circReads style: complete a1: None a2: None overhang: 8 id dir suffix: unaligned: True ignoreIds (aligned to genome): 0 lienarIds: 0 scrambledIds: 32 samFile:/home/zhangjy/data//testData/orig/denovo/unaligned_SRR1027187_1_denovo_output.sam readType:r1j samFile:/home/zhangjy/data//testData/orig/genome/SRR1027187_2_genome_output.sam readType:gMate samFile:/home/zhangjy/data//testData/orig/reg/SRR1027187_2_reg_output.sam readType:rjMate samFile:/home/zhangjy/data//testData/orig/junction/SRR1027187_2_junction_output.sam readType:jMate samFile:/home/zhangjy/data//testData/orig/denovo/unaligned_SRR1027187_2_denovo_output.sam readType:dMate sample id: unaligned_SRR1027187_1 single end? False num junctions with aligned reads: 2 num reads: 32 number of reads kept in each category after update (multimapped & unknown should be 0, anomaly & decoy should be 0 for SE data): circ: 3 , decoy: 20 , linear: 0 , anomaly: 0 , unmapped: 9 , multimapped: 0 , unknown: 0 32 using decoy rate: 0.00381119906186 Completed 1 call to findCircularRNA.sh Sat Sep 26 23:33:24 CST 2015

[lindaszabo]

I see this in your log/error file: ./predictJunctions.sh: line 36: ./predictJunctions_tableData.r: Permission denied

Looks like you need to add executable permissions to that file

[Kevinzjy]

thank you very much, i will see if this work

[Kevinzjy]

It works fine, thanks a lot

[lindaszabo]

great, thanks for confirming.

On Tue, Oct 13, 2015 at 10:21 PM, Kevinzjy notifications@github.com wrote:

It works fine, thanks a lot

— Reply to this email directly or view it on GitHub https://github.com/lindaszabo/KNIFE/issues/4#issuecomment-147937675.