Closed a113n closed 8 years ago
Hi!
thanks so much for looking into this.
I can add it today if that works with you.
Do you plan to run seqcluster inside bcbio? or you are running alone?
if you only need the gtf file you can look inside the scripts folder in the repository where there is one for mm10 if you can not wait.
thanks!
sent not from my computer
On Oct 8, 2015, at 03:30, a113n notifications@github.com wrote:
I am looking for a way to download data files for mouse mm10 genome. It seems that small RNA data for mm10 is not available from cloudbiolinux yet: fab -f cloudbiolinux/data_fabfile.py -H localhost -c fabric.txt install_data_ggd:srnaseq,mm10
May I know if there is any way to run seqcluster for other model organisms such as mouse? Thank you very much and looking forward to your reply.
— Reply to this email directly or view it on GitHub.
Dear Lorena,
Great to hear from you. I am running the pipeline from bcbio, and it also complains about the missing mirbase file. I will try to run the standalone version using the GTF file and see if it works. Thanks!
Allen
On Thu, 8 Oct 2015 at 7:51 PM, Lorena notifications@github.com wrote:
Hi!
thanks so much for looking into this.
I can add it today if that works with you.
Do you plan to run seqcluster inside bcbio? or you are running alone?
if you only need the gtf file you can look inside the scripts folder in the repository where there is one for mm10 if you can not wait.
thanks!
sent not from my computer
On Oct 8, 2015, at 03:30, a113n notifications@github.com wrote:
I am looking for a way to download data files for mouse mm10 genome. It seems that small RNA data for mm10 is not available from cloudbiolinux yet: fab -f cloudbiolinux/data_fabfile.py -H localhost -c fabric.txt install_data_ggd:srnaseq,mm10
May I know if there is any way to run seqcluster for other model organisms such as mouse? Thank you very much and looking forward to your reply.
— Reply to this email directly or view it on GitHub.
— Reply to this email directly or view it on GitHub https://github.com/lpantano/seqcluster/issues/12#issuecomment-146514881.
No problem. It was just to know how to fix it.
Will fix inside bcbio and you will be good to go!
thanks!
Hi,
If you do bcbio_nextgen.py upgrade -u development --genomes mm10
should add the missing files for the pipeline.
Let me know if you find any problem.
thanks
Thanks to the fix, the files can now be downloaded successfully.
However I ran into another issue regarding the trimming steps, it looks like bcbio assumes trimming must be done in small RNA seq pipeline, even if I explicitly specified trim_reads: False
.
Here is the traceback message:
Traceback (most recent call last):
File "/usr/local/bin/bcbio_nextgen.py", line 226, in <module>
main(**kwargs)
File "/usr/local/bin/bcbio_nextgen.py", line 43, in main
run_main(**kwargs)
File "/usr/local/share/bcbio-nextgen/anaconda/lib/python2.7/site-packages/bcbio/pipeline/main.py", line 37, in run_main
fc_dir, run_info_yaml)
File "/usr/local/share/bcbio-nextgen/anaconda/lib/python2.7/site-packages/bcbio/pipeline/main.py", line 80, in _run_toplevel
for xs in pipeline.run(config, run_info_yaml, parallel, dirs, samples):
File "/usr/local/share/bcbio-nextgen/anaconda/lib/python2.7/site-packages/bcbio/pipeline/main.py", line 353, in run
samples = run_parallel("trim_srna_sample", samples)
File "/usr/local/share/bcbio-nextgen/anaconda/lib/python2.7/site-packages/bcbio/distributed/multi.py", line 28, in run_parallel
return run_multicore(fn, items, config, parallel=parallel)
File "/usr/local/share/bcbio-nextgen/anaconda/lib/python2.7/site-packages/bcbio/distributed/multi.py", line 86, in run_multicore
for data in joblib.Parallel(parallel["num_jobs"])(joblib.delayed(fn)(x) for x in items):
File "/usr/local/share/bcbio-nextgen/anaconda/lib/python2.7/site-packages/joblib/parallel.py", line 657, in __call__
self.dispatch(function, args, kwargs)
File "/usr/local/share/bcbio-nextgen/anaconda/lib/python2.7/site-packages/joblib/parallel.py", line 404, in dispatch
job = ImmediateApply(func, args, kwargs)
File "/usr/local/share/bcbio-nextgen/anaconda/lib/python2.7/site-packages/joblib/parallel.py", line 142, in __init__
self.results = func(*args, **kwargs)
File "/usr/local/share/bcbio-nextgen/anaconda/lib/python2.7/site-packages/bcbio/utils.py", line 50, in wrapper
return apply(f, *args, **kwargs)
File "/usr/local/share/bcbio-nextgen/anaconda/lib/python2.7/site-packages/bcbio/distributed/multitasks.py", line 34, in trim_srna_sample
return srna.trim_srna_sample(*args)
File "/usr/local/share/bcbio-nextgen/anaconda/lib/python2.7/site-packages/bcbio/srna/sample.py", line 25, in trim_srna_sample
adapter = dd.get_adapters(data)[0]
IndexError: list index out of range
So apparantly bcbio disregards the trim_reads: False
setting and look for adapters anyway. Perhaps this issue should be reported in bcbio instead of here?
oh! no problem will fix that. since u have the issue open here :)
how do you have your input files? are only fastq where u removed the 3' adapter already or has some other step done? i am think of potential other problems.
if u have them wo adapter can i ask what tool u used to do it? thanks!
sent not from my computer
On Oct 9, 2015, at 04:52, a113n notifications@github.com wrote:
Thanks to the fix, the files can now be downloaded successfully. However I ran into another issue regarding the trimming steps, it looks like bcbio assumes trimming must be done in small RNA seq pipeline, even if I explicitly specified trim_reads: False.
Here is the traceback message:
Traceback (most recent call last): File "/usr/local/bin/bcbio_nextgen.py", line 226, in
main(kwargs) File "/usr/local/bin/bcbio_nextgen.py", line 43, in main run_main(kwargs) File "/usr/local/share/bcbio-nextgen/anaconda/lib/python2.7/site-packages/bcbio/pipeline/main.py", line 37, in run_main fc_dir, run_info_yaml) File "/usr/local/share/bcbio-nextgen/anaconda/lib/python2.7/site-packages/bcbio/pipeline/main.py", line 80, in _run_toplevel for xs in pipeline.run(config, run_info_yaml, parallel, dirs, samples): File "/usr/local/share/bcbio-nextgen/anaconda/lib/python2.7/site-packages/bcbio/pipeline/main.py", line 353, in run samples = run_parallel("trim_srna_sample", samples) File "/usr/local/share/bcbio-nextgen/anaconda/lib/python2.7/site-packages/bcbio/distributed/multi.py", line 28, in run_parallel return run_multicore(fn, items, config, parallel=parallel) File "/usr/local/share/bcbio-nextgen/anaconda/lib/python2.7/site-packages/bcbio/distributed/multi.py", line 86, in run_multicore for data in joblib.Parallel(parallel["num_jobs"])(joblib.delayed(fn)(x) for x in items): File "/usr/local/share/bcbio-nextgen/anaconda/lib/python2.7/site-packages/joblib/parallel.py", line 657, in call self.dispatch(function, args, kwargs) File "/usr/local/share/bcbio-nextgen/anaconda/lib/python2.7/site-packages/joblib/parallel.py", line 404, in dispatch job = ImmediateApply(func, args, kwargs) File "/usr/local/share/bcbio-nextgen/anaconda/lib/python2.7/site-packages/joblib/parallel.py", line 142, in init self.results = func(_args, _kwargs) File "/usr/local/share/bcbio-nextgen/anaconda/lib/python2.7/site-packages/bcbio/utils.py", line 50, in wrapper return apply(f, _args, _kwargs) File "/usr/local/share/bcbio-nextgen/anaconda/lib/python2.7/site-packages/bcbio/distributed/multitasks.py", line 34, in trim_srna_sample return srna.trim_srna_sample(*args) File "/usr/local/share/bcbio-nextgen/anaconda/lib/python2.7/site-packages/bcbio/srna/sample.py", line 25, in trim_srna_sample adapter = dd.get_adapters(data)[0] IndexError: list index out of range So apparantly bcbio disregards the trim_reads: False setting and look for adapters anyway. Perhaps this issue should be reported in bcbio instead of here? — Reply to this email directly or view it on GitHub.
I am trying to use multiple small RNA analysis tools on my files, and thus they were pre-trimmed using cutadapt for consistency, hope it makes sense. On Fri, 9 Oct 2015 at 8:38 PM, Lorena notifications@github.com wrote:
oh! no problem will fix that. since u have the issue open here :)
how do you have your input files? are only fastq where u removed the 3' adapter already or has some other step done? i am think of potential other problems.
if u have them wo adapter can i ask what tool u used to do it? thanks!
sent not from my computer
On Oct 9, 2015, at 04:52, a113n notifications@github.com wrote:
Thanks to the fix, the files can now be downloaded successfully. However I ran into another issue regarding the trimming steps, it looks like bcbio assumes trimming must be done in small RNA seq pipeline, even if I explicitly specified trim_reads: False.
Here is the traceback message:
Traceback (most recent call last): File "/usr/local/bin/bcbio_nextgen.py", line 226, in
main(kwargs) File "/usr/local/bin/bcbio_nextgen.py", line 43, in main run_main(kwargs) File "/usr/local/share/bcbio-nextgen/anaconda/lib/python2.7/site-packages/bcbio/pipeline/main.py", line 37, in run_main fc_dir, run_info_yaml) File "/usr/local/share/bcbio-nextgen/anaconda/lib/python2.7/site-packages/bcbio/pipeline/main.py", line 80, in _run_toplevel for xs in pipeline.run(config, run_info_yaml, parallel, dirs, samples): File "/usr/local/share/bcbio-nextgen/anaconda/lib/python2.7/site-packages/bcbio/pipeline/main.py", line 353, in run samples = run_parallel("trim_srna_sample", samples) File "/usr/local/share/bcbio-nextgen/anaconda/lib/python2.7/site-packages/bcbio/distributed/multi.py", line 28, in run_parallel return run_multicore(fn, items, config, parallel=parallel) File "/usr/local/share/bcbio-nextgen/anaconda/lib/python2.7/site-packages/bcbio/distributed/multi.py", line 86, in run_multicore for data in joblib.Parallel(parallel["num_jobs"])(joblib.delayed(fn)(x) for x in items): File "/usr/local/share/bcbio-nextgen/anaconda/lib/python2.7/site-packages/joblib/parallel.py", line 657, in call self.dispatch(function, args, kwargs) File "/usr/local/share/bcbio-nextgen/anaconda/lib/python2.7/site-packages/joblib/parallel.py", line 404, in dispatch job = ImmediateApply(func, args, kwargs) File "/usr/local/share/bcbio-nextgen/anaconda/lib/python2.7/site-packages/joblib/parallel.py", line 142, in init self.results = func(_args, _kwargs) File "/usr/local/share/bcbio-nextgen/anaconda/lib/python2.7/site-packages/bcbio/utils.py", line 50, in wrapper return apply(f, _args, _kwargs) File "/usr/local/share/bcbio-nextgen/anaconda/lib/python2.7/site-packages/bcbio/distributed/multitasks.py", line 34, in trim_srna_sample return srna.trim_srna_sample(*args) File "/usr/local/share/bcbio-nextgen/anaconda/lib/python2.7/site-packages/bcbio/srna/sample.py", line 25, in trim_srna_sample adapter = dd.get_adapters(data)[0] IndexError: list index out of range So apparantly bcbio disregards the trim_reads: False setting and look for adapters anyway. Perhaps this issue should be reported in bcbio instead of here? — Reply to this email directly or view it on GitHub.
— Reply to this email directly or view it on GitHub https://github.com/lpantano/seqcluster/issues/12#issuecomment-146855540.
That's cool. I wanted to know if there was any other step done like counting unique sequences, and in that case it would be more tricky to fix it. Working on that now. thanks!
Hi,
If you do bcbio_nextgen.py upgrade -u development
should get the last fix and work this time.
thanks for your patient.
The previous issues were fixed, but now I have encountered another issue:
INFO 638 clusters found
INFO counts after: 157563713
INFO # sequences after: 242430
INFO Solving multi-mapping events in the network of clusters
INFO Number of loci: 24083
0% | |^M['cluster', '-o', '/media/seqcluster/cluster', '-m', '/media/seqcluster/prepare/seqs.ma', '-a', '/media/align/seqs.bam', '-r', '/usr/local/share/bcbio-nextgen/genomes/Mmusculus/mm10/seq/mm10.fa', '-g', '/usr/local/share/bcbio-nextgen/genomes/Mmusculus/mm10/srnaseq/srna-transcripts.gtf']
Traceback (most recent call last):
File "/usr/local/share/bcbio-nextgen/anaconda/bin/seqcluster", line 6, in <module>
sys.exit(main())
File "/usr/local/share/bcbio-nextgen/anaconda/lib/python2.7/site-packages/seqcluster/command_line.py", line 27, in main
cluster(kwargs["args"])
File "/usr/local/share/bcbio-nextgen/anaconda/lib/python2.7/site-packages/seqcluster/make_clusters.py", line 68, in cluster
clusLred = _cleaning(clusL, args.dir_out)
File "/usr/local/share/bcbio-nextgen/anaconda/lib/python2.7/site-packages/seqcluster/make_clusters.py", line 286, in _cleaning
clus_obj = reduceloci(clusL, path)
File "/usr/local/share/bcbio-nextgen/anaconda/lib/python2.7/site-packages/seqcluster/detect/metacluster.py", line 148, in reduceloci
filtered, n_cluster = _iter_loci(c, clus_obj.clus, (clus_obj.loci, clus_obj.seq), filtered, n_cluster)
File "/usr/local/share/bcbio-nextgen/anaconda/lib/python2.7/site-packages/seqcluster/detect/metacluster.py", line 225, in _iter_loci
loci_similarity = _calculate_similarity(loci)
File "/usr/local/share/bcbio-nextgen/anaconda/lib/python2.7/site-packages/seqcluster/detect/metacluster.py", line 283, in _calculate_similarity
[ma.update({(idc, idc2): _common(set1, _get_seqs(c[idc2]), idc, idc2)}) for idc2 in c if idc != idc2 and (idc2, idc) not in ma]
File "/usr/local/share/bcbio-nextgen/anaconda/lib/python2.7/site-packages/seqcluster/detect/metacluster.py", line 303, in _common
is_gt = up_threshold(pct, t * 1.0, parameters.similar)
NameError: global name 'parameters' is not defined
' returned non-zero exit status 1
I looked at line 303 of seqcluster/detect/metacluster.py
and indeed the variable parameters
is not defined within the function. Could you please help look into the issue as well? Thank you very much!
Hi,
sorry about that. I was cleaning code. If you update bcbio:
bcbio_nextgen.py upgrade -u deps
it should update the last version of seqcluster.
Let me know if this fixes the problem.
Thanks a lot for all your help.
Many thanks for all your kind help along the way. The previous issue was fixed, but unfortunately there is another problem:
[2015-10-13T15:57Z] INFO Clusters too long to be analized: 0
[2015-10-13T15:57Z] INFO Number of clusters removed because low number of reads: 141
[2015-10-13T15:58Z] INFO counts after: 155925530
[2015-10-13T15:58Z] INFO # sequences after: 224826
[2015-10-13T15:58Z] INFO Clusters up to 842
[2015-10-13T15:58Z] INFO Creating bed file
[2015-10-13T15:58Z] INFO Annotating clusters
[2015-10-13T15:58Z] INFO Using /usr/local/share/bcbio-nextgen/genomes/Mmusculus/mm10/srnaseq/srna-transcripts.gtf
[2015-10-13T15:58Z] ['cluster', '-o', '/media/seqcluster/cluster', '-m', '/media/seqcluster/prepare/seqs.ma', '-a', '/media/align/seqs.bam', '-r', '/usr/local/share/bcbio-nextgen/genomes/Mmusculus/mm10/seq/mm10.fa', '-g', '/usr/local/share/bcbio-nextgen/genomes/Mmusculus/mm10/srnaseq/srna-transcripts.gtf']
[2015-10-13T15:58Z] Traceback (most recent call last):
[2015-10-13T15:58Z] File "/usr/local/share/bcbio-nextgen/anaconda/bin/seqcluster", line 6, in <module>
[2015-10-13T15:58Z] sys.exit(main())
[2015-10-13T15:58Z] File "/usr/local/share/bcbio-nextgen/anaconda/lib/python2.7/site-packages/seqcluster/command_line.py", line 27, in main
[2015-10-13T15:58Z] cluster(kwargs["args"])
[2015-10-13T15:58Z] File "/usr/local/share/bcbio-nextgen/anaconda/lib/python2.7/site-packages/seqcluster/make_clusters.py", line 83, in cluster
[2015-10-13T15:58Z] clusLred = _annotate(args, clusLred)
[2015-10-13T15:58Z] File "/usr/local/share/bcbio-nextgen/anaconda/lib/python2.7/site-packages/seqcluster/make_clusters.py", line 234, in _annotate
[2015-10-13T15:58Z] c = a.intersect(b, wo=True)
[2015-10-13T15:58Z] File "/usr/local/share/bcbio-nextgen/anaconda/lib/python2.7/site-packages/pybedtools/bedtool.py", line 775, in decorated
[2015-10-13T15:58Z] result = method(self, *args, **kwargs)
[2015-10-13T15:58Z] File "/usr/local/share/bcbio-nextgen/anaconda/lib/python2.7/site-packages/pybedtools/bedtool.py", line 336, in wrapped
[2015-10-13T15:58Z] decode_output=decode_output,
[2015-10-13T15:58Z] File "/usr/local/share/bcbio-nextgen/anaconda/lib/python2.7/site-packages/pybedtools/helpers.py", line 389, in call_bedtools
[2015-10-13T15:58Z] raise BEDToolsError(subprocess.list2cmdline(cmds), stderr)
[2015-10-13T15:58Z] pybedtools.helpers.BEDToolsError:
[2015-10-13T15:58Z] Command was:
[2015-10-13T15:58Z]
[2015-10-13T15:58Z] bedtools intersect -wo -b /usr/local/share/bcbio-nextgen/genomes/Mmusculus/mm10/srnaseq/srna-transcripts.gtf -a /tmp/pybedtools.ITX3o4.tmp
[2015-10-13T15:58Z]
[2015-10-13T15:58Z] Error message was:
[2015-10-13T15:58Z] Error: Invalid record in file /usr/local/share/bcbio-nextgen/genomes/Mmusculus/mm10/srnaseq/srna-transcripts.gtf. Record is
[2015-10-13T15:58Z] chrM . encode 0 68 . + . name mt-Tf;
[2015-10-13T15:58Z]
[2015-10-13T15:58Z] Uncaught exception occurred
Traceback (most recent call last):
File "/usr/local/share/bcbio-nextgen/anaconda/lib/python2.7/site-packages/bcbio/provenance/do.py", line 21, in run
_do_run(cmd, checks, log_stdout)
File "/usr/local/share/bcbio-nextgen/anaconda/lib/python2.7/site-packages/bcbio/provenance/do.py", line 95, in _do_run
raise subprocess.CalledProcessError(exitcode, error_msg)
CalledProcessError: Command 'set -o pipefail; /usr/local/share/bcbio-nextgen/anaconda/bin/seqcluster cluster -o /media/seqcluster/cluster -m /media/seqcluster/prepare/seqs.ma -a /media/align/seqs.bam -r /usr/local/share/bcbio-nextgen/genomes/Mmusculus/mm10/seq/mm10.fa -g /usr/local/share/bcbio-nextgen/genomes/Mmusculus/mm10/srnaseq/srna-transcripts.gtf
If I removed all chrM and chrUn records in srna-transcripts.gtf, the pipeline can finally complete successfully. I am not sure if this is a correct solution, but it works anyway.
Finally, there is another bug in bcbio qc summary generation:
Traceback (most recent call last):
File "/usr/local/bin/bcbio_nextgen.py", line 226, in <module>
main(**kwargs)
File "/usr/local/bin/bcbio_nextgen.py", line 43, in main
run_main(**kwargs)
File "/usr/local/share/bcbio-nextgen/anaconda/lib/python2.7/site-packages/bcbio/pipeline/main.py", line 37, in run_main
fc_dir, run_info_yaml)
File "/usr/local/share/bcbio-nextgen/anaconda/lib/python2.7/site-packages/bcbio/pipeline/main.py", line 80, in _run_toplevel
for xs in pipeline.run(config, run_info_yaml, parallel, dirs, samples):
File "/usr/local/share/bcbio-nextgen/anaconda/lib/python2.7/site-packages/bcbio/pipeline/main.py", line 373, in run
samples = qcsummary.generate_parallel(samples, run_parallel)
File "/usr/local/share/bcbio-nextgen/anaconda/lib/python2.7/site-packages/bcbio/pipeline/qcsummary.py", line 52, in generate_parallel
sum_samples = run_parallel("pipeline_summary", samples)
File "/usr/local/share/bcbio-nextgen/anaconda/lib/python2.7/site-packages/bcbio/distributed/multi.py", line 28, in run_parallel
return run_multicore(fn, items, config, parallel=parallel)
File "/usr/local/share/bcbio-nextgen/anaconda/lib/python2.7/site-packages/bcbio/distributed/multi.py", line 86, in run_multicore
for data in joblib.Parallel(parallel["num_jobs"])(joblib.delayed(fn)(x) for x in items):
File "/usr/local/share/bcbio-nextgen/anaconda/lib/python2.7/site-packages/joblib/parallel.py", line 657, in __call__
self.dispatch(function, args, kwargs)
File "/usr/local/share/bcbio-nextgen/anaconda/lib/python2.7/site-packages/joblib/parallel.py", line 404, in dispatch
job = ImmediateApply(func, args, kwargs)
File "/usr/local/share/bcbio-nextgen/anaconda/lib/python2.7/site-packages/joblib/parallel.py", line 142, in __init__
self.results = func(*args, **kwargs)
File "/usr/local/share/bcbio-nextgen/anaconda/lib/python2.7/site-packages/bcbio/utils.py", line 50, in wrapper
return apply(f, *args, **kwargs)
File "/usr/local/share/bcbio-nextgen/anaconda/lib/python2.7/site-packages/bcbio/distributed/multitasks.py", line 94, in pipeline_summary
return qcsummary.pipeline_summary(*args)
File "/usr/local/share/bcbio-nextgen/anaconda/lib/python2.7/site-packages/bcbio/pipeline/qcsummary.py", line 75, in pipeline_summary
data["summary"] = _run_qc_tools(work_bam, data)
File "/usr/local/share/bcbio-nextgen/anaconda/lib/python2.7/site-packages/bcbio/pipeline/qcsummary.py", line 133, in _run_qc_tools
ratio = bam.get_aligned_reads(bam_file, data)
File "/usr/local/share/bcbio-nextgen/anaconda/lib/python2.7/site-packages/bcbio/bam/__init__.py", line 97, in get_aligned_reads
index(in_bam, data["config"])
File "/usr/local/share/bcbio-nextgen/anaconda/lib/python2.7/site-packages/bcbio/bam/__init__.py", line 38, in index
assert is_bam(in_bam), "%s in not a BAM file" % in_bam
AssertionError: /media/trimmed/A020/A020_1.fastq.cutadapt.clean.fastq.gz in not a BAM file
so sorry about that. we changed QC a lot and some of that get in the middle of the srnaseq pipeline. I will fix the annotation problem anyway and fix the qc now. will let you know in a moment.
one question, do you have kraken on in the yaml file?
Hi @a113n,
I fixed the annotation, you can upgrade the data again. The only way I am seeing that error it is if I add kraken to the YAML file. For now that is not supported, I added an error there if that is the case. Could you try without that option in the file.
thanks!
Indeed, once I removed the kraken option, the error disappear. Thanks!
glad it worked! thanks for the patient. It is very helpful to get people testing the new pipelines!
On 10/15/2015 12:00 PM, a113n wrote:
Closed #12 https://github.com/lpantano/seqcluster/issues/12.
— Reply to this email directly or view it on GitHub https://github.com/lpantano/seqcluster/issues/12#event-436558809.
I am looking for a way to download data files for mouse mm10 genome. It seems that small RNA data for mm10 is not available from cloudbiolinux yet: fab -f cloudbiolinux/data_fabfile.py -H localhost -c fabric.txt install_data_ggd:srnaseq,mm10
May I know if there is any way to run seqcluster for other model organisms such as mouse? Thank you very much and looking forward to your reply.