Closed mmelendrez closed 8 years ago
just found another log file...
std.log = empty .log file:
2016-03-28 13:28:02,892 -- INFO -- runsample --- Starting Day1_10672 ---
2016-03-28 13:28:02,892 -- INFO -- runsample --- Using custom config from config_quasi.yaml ---
2016-03-28 13:28:02,893 -- DEBUG -- runsample Copying reference file KDH0053A_Thailand_NC_2010.fasta to Day1_10672/Day1_10672wHyL59runsample/KDH0053A_Thailand_NC_2010.fasta
2016-03-28 13:28:02,909 -- DEBUG -- runsample {'minth': 0.95, 'flagstats': 'Day1_10672/Day1_10672wHyL59runsample/flagstats.txt', 'readsdir': '/media/VD_Research/NGSData/ReadsBySample/10672/', 'vcf': 'Day1_10672/Day1_10672wHyL59runsample/Day1_10672.bam.vcf', 'reference': 'Day1_10672/Day1_10672wHyL59runsample/KDH0053A_Thailand_NC_2010.fasta', 'primer_info': (None, 2, 30, 20), 'samplename': 'Day1_10672', 'drop_ns': True, 'bamfile': 'Day1_10672/Day1_10672wHyL59runsample/Day1_10672.bam', 'filtered_dir': 'Day1_10672/Day1_10672wHyL59runsample/filtered', 'consensus': 'Day1_10672/Day1_10672wHyL59runsample/Day1_10672.bam.consensus.fasta', 'platforms': ['MiSeq', 'Sanger', 'Roche454', 'IonTorrent'], 'tdir': 'Day1_10672/Day1_10672wHyL59runsample', 'head_crop': 0, 'trim_qual': 20, 'index_min': 25, 'config': 'config_quasi.yaml', 'trim_outdir': 'Day1_10672/Day1_10672wHyL59runsample/trimmed_reads', 'CN': None}
2016-03-28 13:41:45,839 -- ERROR -- runsample
Within the filtered file - was the following stats:
ngs_filter found 3809635 reads in file /media/VD_Research/NGSData/ReadsBySample/10672/10672_S6_L001_R1_001_2015_11_20.fastq, and filtered out 71259 reads.
In file /media/VD_Research/NGSData/ReadsBySample/10672/10672_S6_L001_R1_001_2015_11_20.fastq, 70783 reads were filtered for poor quality index below 25. 476 reads had Ns and were filtered.
A couple things:
So did the error occur due to pairing problems? Does the pipeline try to pair and by filtering out all those reads, it now cannot pair? Just thinking out loud.
The top line is the total number of reads that were filtered out (71259). The second line breaks it down into which had Ns and which had poor quality. So 70783 (quality) + 476 (Ns) = 71259 total.
It's difficult to tell what went wrong here. (This is a good demonstration that the error reporting needs to be consolidated, which we are working on in #206. This is worsened by the fact that ngs_filter
doesn't log in consistent way, which is fixed in #215) Once those get resolved it will be easier to fix these kinds of issues. I will work on re-creating the problem. It's possible that one of the read files was filtered out completely, but it might be something else.
Ok - well you have my command and config file - let me know if you cannot access the reference.
I don't know that it would completely filter out one of the R files because total it only filtered about 70K and there is over 3 million reads between both files.
[melanie.melendrez@amedwrair015380 SubjCHD10_015_025]$ ls -lh /media/VD_Research/NGSData/ReadData/MiSeq/151120_M02261_0033_000000000-AHKJP/10672_S6_L001_*
-rw-r--r--. 1 607 607 267M Nov 23 16:30 /media/VD_Research/NGSData/ReadData/MiSeq/151120_M02261_0033_000000000-AHKJP/10672_S6_L001_I1_001_2015_11_20.fastq
-rw-r--r--. 1 607 607 267M Nov 23 16:30 /media/VD_Research/NGSData/ReadData/MiSeq/151120_M02261_0033_000000000-AHKJP/10672_S6_L001_I2_001_2015_11_20.fastq
-rw-r--r--. 1 607 607 2.2G Nov 23 16:31 /media/VD_Research/NGSData/ReadData/MiSeq/151120_M02261_0033_000000000-AHKJP/10672_S6_L001_R1_001_2015_11_20.fastq
-rw-r--r--. 1 607 607 2.3G Nov 23 16:33 /media/VD_Research/NGSData/ReadData/MiSeq/151120_M02261_0033_000000000-AHKJP/10672_S6_L001_R2_001_2015_11_20.fast
They both have equal amounts of data so with 3 million total - it's unlikely 2.2 or 2.3G represents only 70K reads which is what would have to be if one of the files was completely filtered out.
I am also running this a couple different ways. I have it running on a different sample and it's still going and I have it running on the same sample I just did but without the index filtering by Q25 (just pulling Ns out).
Version updated - ngs_filter fixed - reloading and running
Hmmm...run results:
[melanie.melendrez@amedwrair015380 SubjCHD10_015_025]$ RUNSAMPLEOPTIONS="-c config_quasi.yaml" runsamplesheet.sh /media/VD_Research/NGSData/ReadsBySample/ samplesheet_SubjCHD10_015_025
Using 12 cpus
2016-03-29 14:04:44,877 -- INFO -- runsample --- Starting 10669 ---
2016-03-29 14:04:44,877 -- INFO -- runsample --- Using custom config from config_quasi.yaml ---
2016-03-29 14:04:44,879 -- INFO -- runsample --- Starting 10668 ---
2016-03-29 14:04:44,879 -- INFO -- runsample --- Using custom config from config_quasi.yaml ---
2016-03-29 14:04:44,880 -- INFO -- runsample --- Starting 10670 ---
2016-03-29 14:04:44,880 -- INFO -- runsample --- Using custom config from config_quasi.yaml ---
2016-03-29 14:04:44,880 -- INFO -- runsample --- Starting 10671 ---
2016-03-29 14:04:44,880 -- INFO -- runsample --- Using custom config from config_quasi.yaml ---
2016-03-29 14:04:44,881 -- INFO -- runsample --- Starting 10667 ---
2016-03-29 14:04:44,881 -- INFO -- runsample --- Using custom config from config_quasi.yaml ---
2016-03-29 14:04:47,435 -- ERROR -- runsample Traceback (most recent call last):
File "/media/VD_Research/Admin/PBS/Software/ngs_mapper/miniconda/bin/ngs_filter", line 9, in <module>
load_entry_point('ngs-mapper==1.4.2', 'console_scripts', 'ngs_filter')()
File "/media/VD_Research/Admin/PBS/Software/ngs_mapper/miniconda/lib/python2.7/site-packages/ngs_mapper-1.4.2-py2.7.egg/ngs_mapper/nfilter.py", line 236, in main
run_from_config(args['<readdir>'], args['--outdir'], args['--config'])
File "/media/VD_Research/Admin/PBS/Software/ngs_mapper/miniconda/lib/python2.7/site-packages/ngs_mapper-1.4.2-py2.7.egg/ngs_mapper/nfilter.py", line 218, in run_from_config
outdir, defaults['threads']['default'])
KeyError: 'threads'
2016-03-29 14:04:47,435 -- ERROR -- runsample Traceback (most recent call last):
File "/media/VD_Research/Admin/PBS/Software/ngs_mapper/miniconda/bin/ngs_filter", line 9, in <module>
load_entry_point('ngs-mapper==1.4.2', 'console_scripts', 'ngs_filter')()
File "/media/VD_Research/Admin/PBS/Software/ngs_mapper/miniconda/lib/python2.7/site-packages/ngs_mapper-1.4.2-py2.7.egg/ngs_mapper/nfilter.py", line 236, in main
run_from_config(args['<readdir>'], args['--outdir'], args['--config'])
File "/media/VD_Research/Admin/PBS/Software/ngs_mapper/miniconda/lib/python2.7/site-packages/ngs_mapper-1.4.2-py2.7.egg/ngs_mapper/nfilter.py", line 218, in run_from_config
outdir, defaults['threads']['default'])
KeyError: 'threads'
2016-03-29 14:04:47,436 -- ERROR -- runsample Traceback (most recent call last):
File "/media/VD_Research/Admin/PBS/Software/ngs_mapper/miniconda/bin/ngs_filter", line 9, in <module>
load_entry_point('ngs-mapper==1.4.2', 'console_scripts', 'ngs_filter')()
File "/media/VD_Research/Admin/PBS/Software/ngs_mapper/miniconda/lib/python2.7/site-packages/ngs_mapper-1.4.2-py2.7.egg/ngs_mapper/nfilter.py", line 236, in main
run_from_config(args['<readdir>'], args['--outdir'], args['--config'])
File "/media/VD_Research/Admin/PBS/Software/ngs_mapper/miniconda/lib/python2.7/site-packages/ngs_mapper-1.4.2-py2.7.egg/ngs_mapper/nfilter.py", line 218, in run_from_config
outdir, defaults['threads']['default'])
KeyError: 'threads'
2016-03-29 14:04:47,437 -- ERROR -- runsample Traceback (most recent call last):
File "/media/VD_Research/Admin/PBS/Software/ngs_mapper/miniconda/bin/ngs_filter", line 9, in <module>
load_entry_point('ngs-mapper==1.4.2', 'console_scripts', 'ngs_filter')()
File "/media/VD_Research/Admin/PBS/Software/ngs_mapper/miniconda/lib/python2.7/site-packages/ngs_mapper-1.4.2-py2.7.egg/ngs_mapper/nfilter.py", line 236, in main
run_from_config(args['<readdir>'], args['--outdir'], args['--config'])
File "/media/VD_Research/Admin/PBS/Software/ngs_mapper/miniconda/lib/python2.7/site-packages/ngs_mapper-1.4.2-py2.7.egg/ngs_mapper/nfilter.py", line 218, in run_from_config
outdir, defaults['threads']['default'])
KeyError: 'threads'
2016-03-29 14:04:47,438 -- ERROR -- runsample Traceback (most recent call last):
File "/media/VD_Research/Admin/PBS/Software/ngs_mapper/miniconda/bin/ngs_filter", line 9, in <module>
load_entry_point('ngs-mapper==1.4.2', 'console_scripts', 'ngs_filter')()
File "/media/VD_Research/Admin/PBS/Software/ngs_mapper/miniconda/lib/python2.7/site-packages/ngs_mapper-1.4.2-py2.7.egg/ngs_mapper/nfilter.py", line 236, in main
run_from_config(args['<readdir>'], args['--outdir'], args['--config'])
File "/media/VD_Research/Admin/PBS/Software/ngs_mapper/miniconda/lib/python2.7/site-packages/ngs_mapper-1.4.2-py2.7.egg/ngs_mapper/nfilter.py", line 218, in run_from_config
outdir, defaults['threads']['default'])
KeyError: 'threads'
Using 12 cpus
Traceback (most recent call last):
File "/media/VD_Research/Admin/PBS/Software/ngs_mapper/miniconda/bin/graphsample", line 9, in <module>
load_entry_point('ngs-mapper==1.4.2', 'console_scripts', 'graphsample')()
File "/media/VD_Research/Admin/PBS/Software/ngs_mapper/miniconda/lib/python2.7/site-packages/ngs_mapper-1.4.2-py2.7.egg/ngs_mapper/graphsample.py", line 23, in main
Traceback (most recent call last):
File "/media/VD_Research/Admin/PBS/Software/ngs_mapper/miniconda/bin/graphsample", line 9, in <module>
Traceback (most recent call last):
File "/media/VD_Research/Admin/PBS/Software/ngs_mapper/miniconda/bin/graphsample", line 9, in <module>
load_entry_point('ngs-mapper==1.4.2', 'console_scripts', 'graphsample')()
Traceback (most recent call last):
File "/media/VD_Research/Admin/PBS/Software/ngs_mapper/miniconda/bin/graphsample", line 9, in <module>
File "/media/VD_Research/Admin/PBS/Software/ngs_mapper/miniconda/lib/python2.7/site-packages/ngs_mapper-1.4.2-py2.7.egg/ngs_mapper/graphsample.py", line 23, in main
load_entry_point('ngs-mapper==1.4.2', 'console_scripts', 'graphsample')()
File "/media/VD_Research/Admin/PBS/Software/ngs_mapper/miniconda/lib/python2.7/site-packages/ngs_mapper-1.4.2-py2.7.egg/ngs_mapper/graphsample.py", line 23, in main
load_entry_point('ngs-mapper==1.4.2', 'console_scripts', 'graphsample')()
File "/media/VD_Research/Admin/PBS/Software/ngs_mapper/miniconda/lib/python2.7/site-packages/ngs_mapper-1.4.2-py2.7.egg/ngs_mapper/graphsample.py", line 23, in main
Traceback (most recent call last):
File "/media/VD_Research/Admin/PBS/Software/ngs_mapper/miniconda/bin/graphsample", line 9, in <module>
load_entry_point('ngs-mapper==1.4.2', 'console_scripts', 'graphsample')()
File "/media/VD_Research/Admin/PBS/Software/ngs_mapper/miniconda/lib/python2.7/site-packages/ngs_mapper-1.4.2-py2.7.egg/ngs_mapper/graphsample.py", line 23, in main
pngfile = make_image( jfile, args.outpath )
pngfile = make_image( jfile, args.outpath )
File "/media/VD_Research/Admin/PBS/Software/ngs_mapper/miniconda/lib/python2.7/site-packages/ngs_mapper-1.4.2-py2.7.egg/ngs_mapper/graphsample.py", line 41, in make_image
File "/media/VD_Research/Admin/PBS/Software/ngs_mapper/miniconda/lib/python2.7/site-packages/ngs_mapper-1.4.2-py2.7.egg/ngs_mapper/graphsample.py", line 41, in make_image
pngfile = make_image( jfile, args.outpath )
File "/media/VD_Research/Admin/PBS/Software/ngs_mapper/miniconda/lib/python2.7/site-packages/ngs_mapper-1.4.2-py2.7.egg/ngs_mapper/graphsample.py", line 41, in make_image
pngfile = make_image( jfile, args.outpath )
File "/media/VD_Research/Admin/PBS/Software/ngs_mapper/miniconda/lib/python2.7/site-packages/ngs_mapper-1.4.2-py2.7.egg/ngs_mapper/graphsample.py", line 41, in make_image
pngfile = make_image( jfile, args.outpath )
File "/media/VD_Research/Admin/PBS/Software/ngs_mapper/miniconda/lib/python2.7/site-packages/ngs_mapper-1.4.2-py2.7.egg/ngs_mapper/graphsample.py", line 41, in make_image
j = json.load( open(jfile) )
j = json.load( open(jfile) )
IOError: [Errno 2] No such file or directory: 'Projects/10670/10670.bam.qualdepth.json'
j = json.load( open(jfile) )
j = json.load( open(jfile) )
IOError: IOError[Errno 2] No such file or directory: 'Projects/10667/10667.bam.qualdepth.json'IOError
: : [Errno 2] No such file or directory: 'Projects/10668/10668.bam.qualdepth.json' [Errno 2] No such file or directory: 'Projects/10671/10671.bam.qualdepth.json'
j = json.load( open(jfile) )
IOError: [Errno 2] No such file or directory: 'Projects/10669/10669.bam.qualdepth.json'
real 0m6.854s
user 0m0.250s
sys 0m0.146s
real 0m6.842s
user 0m0.249s
sys 0m0.151s
real 0m6.860s
real 0m6.851s
user 0m0.269s
user 0m0.255s
sys 0m0.138s
sys 0m0.135s
real 0m6.847s
user 0m0.269s
sys 0m0.140s
making a new config file
now it seems to be running...
ngs_filter
now requires a threads
option, rather than a parallel
option.So, that was fixed and that worked properly.
I am using a custom config file - config_quasi.yaml.txt
Used runsample and this was the output:
Only 1 log file generated - pipeline.log: This log isn't too informative as to what might have happened. No other logs were generated.
I think it might have something to do with the usage of filtering by index using ngs_filter (see config file).