Closed xbing88 closed 6 years ago
Start here - The index file is older than the data file: S13_bowtie2_sorted.bam.bai
. Re-index this bam file
Thanks for the comment but this is not the reason it is not working.
Hi @xbing88,
did I get this right that you did the mapping with bowtie2?
That's not how you are supposed to use slamdunk, you'll not make any use of the nt-conversion aware alignment and will be missing essential bam-tags (as the said XI).
@t-neumann Hey yes thank you this is definitely the problem. Can I install just ngm and remap the reads with that and plug into slamdunk starting at filter? I am having trouble installing slamdunk on our campus cluster.
Can you tell me what is causing you troubles? I could be quite tedious to run it from a plain NGM installation.
This happens after a few hundred lines into the installation, looks like it thinks Cython is not installed but it is:
copying pybedtools/_Window.c -> build/lib.linux-x86_64-2.7/pybedtools running build_ext building 'pybedtools.cbedtools' extension creating build/temp.linux-x86_64-2.7 creating build/temp.linux-x86_64-2.7/src creating build/temp.linux-x86_64-2.7/pybedtools gcc -pthread -B /home/xb37w/miniconda2/compiler_compat -Wl,--sysroot=/ -fno-strict-aliasing -g -O2 -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fPIC -Isrc/ -I/home/xb37w/miniconda2/include/python2.7 -c src/fileType.cpp -o build/temp.linux-x86_64-2.7/src/fileType.o gcc: error trying to exec 'cc1plus': execvp: No such file or directory error: command 'gcc' failed with exit status 1
Failed building wheel for pybedtools
Running setup.py clean for pybedtools
Failed to build slamdunk pybedtools
Installing collected packages: pybedtools, sortedcontainers, intervaltree, pytz, numpy, python-dateutil, pandas, biopython, Cython, slamdunk
Running setup.py install for pybedtools ... error
Complete output from command /home/xb37w/miniconda2/bin/python -u -c "import setuptools, tokenize;file='/tmp/pip-build-4aFtCh/pybedtools/setup.py';f=getattr(tokenize, 'open', open)(file);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, file, 'exec'))" install --record /tmp/pip-D2O3Mi-record/install-record.txt --single-version-externally-managed --compile --user --prefix=:
running install
running build
running build_py
creating build
creating build/lib.linux-x86_64-2.7
creating build/lib.linux-x86_64-2.7/pybedtools
copying pybedtools/genome_registry.py -> build/lib.linux-x86_64-2.7/pybedtools
copying pybedtools/parallel.py -> build/lib.linux-x86_64-2.7/pybedtools
copying pybedtools/version.py -> build/lib.linux-x86_64-2.7/pybedtools
copying pybedtools/main.py -> build/lib.linux-x86_64-2.7/pybedtools
copying pybedtools/logger.py -> build/lib.linux-x86_64-2.7/pybedtools
copying pybedtools/stats.py -> build/lib.linux-x86_64-2.7/pybedtools
copying pybedtools/init.py -> build/lib.linux-x86_64-2.7/pybedtools
copying pybedtools/helpers.py -> build/lib.linux-x86_64-2.7/pybedtools
copying pybedtools/filenames.py -> build/lib.linux-x86_64-2.7/pybedtools
copying pybedtools/bedtool.py -> build/lib.linux-x86_64-2.7/pybedtools
copying pybedtools/settings.py -> build/lib.linux-x86_64-2.7/pybedtools
creating build/lib.linux-x86_64-2.7/pybedtools/test
copying pybedtools/test/regression_tests.py -> build/lib.linux-x86_64-2.7/pybedtools/test
copying pybedtools/test/test_contrib.py -> build/lib.linux-x86_64-2.7/pybedtools/test
copying pybedtools/test/tfuncs.py -> build/lib.linux-x86_64-2.7/pybedtools/test
copying pybedtools/test/test_helpers.py -> build/lib.linux-x86_64-2.7/pybedtools/test
copying pybedtools/test/test_gzip_support.py -> build/lib.linux-x86_64-2.7/pybedtools/test
copying pybedtools/test/test1.py -> build/lib.linux-x86_64-2.7/pybedtools/test
copying pybedtools/test/test_cbedtools.py -> build/lib.linux-x86_64-2.7/pybedtools/test
copying pybedtools/test/test_iter.py -> build/lib.linux-x86_64-2.7/pybedtools/test
copying pybedtools/test/init.py -> build/lib.linux-x86_64-2.7/pybedtools/test
copying pybedtools/test/test_len_leak.py -> build/lib.linux-x86_64-2.7/pybedtools/test
copying pybedtools/test/test_scripts.py -> build/lib.linux-x86_64-2.7/pybedtools/test
creating build/lib.linux-x86_64-2.7/pybedtools/contrib
copying pybedtools/contrib/bigbed.py -> build/lib.linux-x86_64-2.7/pybedtools/contrib
copying pybedtools/contrib/plotting.py -> build/lib.linux-x86_64-2.7/pybedtools/contrib
copying pybedtools/contrib/init.py -> build/lib.linux-x86_64-2.7/pybedtools/contrib
copying pybedtools/contrib/classifier.py -> build/lib.linux-x86_64-2.7/pybedtools/contrib
copying pybedtools/contrib/long_range_interaction.py -> build/lib.linux-x86_64-2.7/pybedtools/contrib
copying pybedtools/contrib/venn_maker.py -> build/lib.linux-x86_64-2.7/pybedtools/contrib
copying pybedtools/contrib/bigwig.py -> build/lib.linux-x86_64-2.7/pybedtools/contrib
copying pybedtools/contrib/intersection_matrix.py -> build/lib.linux-x86_64-2.7/pybedtools/contrib
creating build/lib.linux-x86_64-2.7/pybedtools/scripts
copying pybedtools/scripts/peak_pie.py -> build/lib.linux-x86_64-2.7/pybedtools/scripts
copying pybedtools/scripts/venn_gchart.py -> build/lib.linux-x86_64-2.7/pybedtools/scripts
copying pybedtools/scripts/venn_mpl.py -> build/lib.linux-x86_64-2.7/pybedtools/scripts
copying pybedtools/scripts/init.py -> build/lib.linux-x86_64-2.7/pybedtools/scripts
copying pybedtools/scripts/annotate.py -> build/lib.linux-x86_64-2.7/pybedtools/scripts
copying pybedtools/scripts/py_ms_example.py -> build/lib.linux-x86_64-2.7/pybedtools/scripts
copying pybedtools/scripts/intron_exon_reads.py -> build/lib.linux-x86_64-2.7/pybedtools/scripts
copying pybedtools/scripts/intersection_matrix.py -> build/lib.linux-x86_64-2.7/pybedtools/scripts
creating build/lib.linux-x86_64-2.7/pybedtools/test/data
copying pybedtools/test/data/init.py -> build/lib.linux-x86_64-2.7/pybedtools/test/data
copying pybedtools/test/data/snps.bed.gz -> build/lib.linux-x86_64-2.7/pybedtools/test/data
copying pybedtools/test/data/vcf-stderr-test.vcf -> build/lib.linux-x86_64-2.7/pybedtools/test/data
copying pybedtools/test/data/b.bed -> build/lib.linux-x86_64-2.7/pybedtools/test/data
copying pybedtools/test/data/bedpe.bed -> build/lib.linux-x86_64-2.7/pybedtools/test/data
copying pybedtools/test/data/rmsk.hg18.chr21.small.bed.gz -> build/lib.linux-x86_64-2.7/pybedtools/test/data
copying pybedtools/test/data/test_bedpe.bed -> build/lib.linux-x86_64-2.7/pybedtools/test/data
copying pybedtools/test/data/test.fa.fai -> build/lib.linux-x86_64-2.7/pybedtools/test/data
copying pybedtools/test/data/gdc.gff.gz -> build/lib.linux-x86_64-2.7/pybedtools/test/data
copying pybedtools/test/data/expand_test.bed -> build/lib.linux-x86_64-2.7/pybedtools/test/data
copying pybedtools/test/data/test_peaks.bed -> build/lib.linux-x86_64-2.7/pybedtools/test/data
copying pybedtools/test/data/Cp190_Mbn2_Bushey_2009.bed -> build/lib.linux-x86_64-2.7/pybedtools/test/data
copying pybedtools/test/data/multibamcov_test.bed -> build/lib.linux-x86_64-2.7/pybedtools/test/data
copying pybedtools/test/data/x.bed -> build/lib.linux-x86_64-2.7/pybedtools/test/data
copying pybedtools/test/data/x.bam -> build/lib.linux-x86_64-2.7/pybedtools/test/data
copying pybedtools/test/data/a.bed -> build/lib.linux-x86_64-2.7/pybedtools/test/data
copying pybedtools/test/data/a.igv_script -> build/lib.linux-x86_64-2.7/pybedtools/test/data
copying pybedtools/test/data/gdc.50.200.bam -> build/lib.linux-x86_64-2.7/pybedtools/test/data
copying pybedtools/test/data/bedpe2.bed -> build/lib.linux-x86_64-2.7/pybedtools/test/data
copying pybedtools/test/data/m1.bed -> build/lib.linux-x86_64-2.7/pybedtools/test/data
copying pybedtools/test/data/venn.b.bed -> build/lib.linux-x86_64-2.7/pybedtools/test/data
copying pybedtools/test/data/SuHw_Kc_Bushey_2009.bed -> build/lib.linux-x86_64-2.7/pybedtools/test/data
copying pybedtools/test/data/dm3-chr2L-5M.gff.gz -> build/lib.linux-x86_64-2.7/pybedtools/test/data
copying pybedtools/test/data/gdc.1.100.bam -> build/lib.linux-x86_64-2.7/pybedtools/test/data
copying pybedtools/test/data/y.bam -> build/lib.linux-x86_64-2.7/pybedtools/test/data
copying pybedtools/test/data/a.links.html -> build/lib.linux-x86_64-2.7/pybedtools/test/data
copying pybedtools/test/data/gdc.sorted.bam -> build/lib.linux-x86_64-2.7/pybedtools/test/data
copying pybedtools/test/data/rmsk.hg18.chr21.small.bed -> build/lib.linux-x86_64-2.7/pybedtools/test/data
copying pybedtools/test/data/gdc.bam -> build/lib.linux-x86_64-2.7/pybedtools/test/data
copying pybedtools/test/data/gdc.sorted.bam.bai -> build/lib.linux-x86_64-2.7/pybedtools/test/data
copying pybedtools/test/data/hg19.gff -> build/lib.linux-x86_64-2.7/pybedtools/test/data
copying pybedtools/test/data/Cp190_Kc_Bushey_2009.bed -> build/lib.linux-x86_64-2.7/pybedtools/test/data
copying pybedtools/test/data/d.gff -> build/lib.linux-x86_64-2.7/pybedtools/test/data
copying pybedtools/test/data/mm9.bed12 -> build/lib.linux-x86_64-2.7/pybedtools/test/data
copying pybedtools/test/data/BEAF_Kc_Bushey_2009.bed -> build/lib.linux-x86_64-2.7/pybedtools/test/data
copying pybedtools/test/data/democonfig.yaml -> build/lib.linux-x86_64-2.7/pybedtools/test/data
copying pybedtools/test/data/dm3-chr2L-5M-invalid.gff.gz -> build/lib.linux-x86_64-2.7/pybedtools/test/data
copying pybedtools/test/data/gdc.50.200.bam.bai -> build/lib.linux-x86_64-2.7/pybedtools/test/data
copying pybedtools/test/data/hg38-base.bed -> build/lib.linux-x86_64-2.7/pybedtools/test/data
copying pybedtools/test/data/vcf-stderr-test.bed -> build/lib.linux-x86_64-2.7/pybedtools/test/data
copying pybedtools/test/data/tag_test2.bed -> build/lib.linux-x86_64-2.7/pybedtools/test/data
copying pybedtools/test/data/CTCF_Kc_Bushey_2009.bed -> build/lib.linux-x86_64-2.7/pybedtools/test/data
copying pybedtools/test/data/BEAF_Mbn2_Bushey_2009.bed -> build/lib.linux-x86_64-2.7/pybedtools/test/data
copying pybedtools/test/data/small.bam -> build/lib.linux-x86_64-2.7/pybedtools/test/data
copying pybedtools/test/data/CTCF_Mbn2_Bushey_2009.bed -> build/lib.linux-x86_64-2.7/pybedtools/test/data
copying pybedtools/test/data/SuHw_Mbn2_Bushey_2009.bed -> build/lib.linux-x86_64-2.7/pybedtools/test/data
copying pybedtools/test/data/c.gff -> build/lib.linux-x86_64-2.7/pybedtools/test/data
copying pybedtools/test/data/hg38-problem.bed -> build/lib.linux-x86_64-2.7/pybedtools/test/data
copying pybedtools/test/data/gdc.othersort.bam -> build/lib.linux-x86_64-2.7/pybedtools/test/data
copying pybedtools/test/data/tag_test1.bed -> build/lib.linux-x86_64-2.7/pybedtools/test/data
copying pybedtools/test/data/venn.c.bed -> build/lib.linux-x86_64-2.7/pybedtools/test/data
copying pybedtools/test/data/reads.bam -> build/lib.linux-x86_64-2.7/pybedtools/test/data
copying pybedtools/test/data/164.gtf -> build/lib.linux-x86_64-2.7/pybedtools/test/data
copying pybedtools/test/data/issue_121.bam -> build/lib.linux-x86_64-2.7/pybedtools/test/data
copying pybedtools/test/data/gdc.bed -> build/lib.linux-x86_64-2.7/pybedtools/test/data
copying pybedtools/test/data/small.fastq -> build/lib.linux-x86_64-2.7/pybedtools/test/data
copying pybedtools/test/data/1000genomes-example.vcf -> build/lib.linux-x86_64-2.7/pybedtools/test/data
copying pybedtools/test/data/test_tsses.bed -> build/lib.linux-x86_64-2.7/pybedtools/test/data
copying pybedtools/test/data/gdc.1.100.bam.bai -> build/lib.linux-x86_64-2.7/pybedtools/test/data
copying pybedtools/test/data/v.vcf -> build/lib.linux-x86_64-2.7/pybedtools/test/data
copying pybedtools/test/data/gdc.gff -> build/lib.linux-x86_64-2.7/pybedtools/test/data
copying pybedtools/test/data/exons.gff -> build/lib.linux-x86_64-2.7/pybedtools/test/data
copying pybedtools/test/data/test.fa -> build/lib.linux-x86_64-2.7/pybedtools/test/data
copying pybedtools/_Window.pyx -> build/lib.linux-x86_64-2.7/pybedtools
copying pybedtools/featurefuncs.pyx -> build/lib.linux-x86_64-2.7/pybedtools
copying pybedtools/cbedtools.pyx -> build/lib.linux-x86_64-2.7/pybedtools
copying pybedtools/cbedtools.pxd -> build/lib.linux-x86_64-2.7/pybedtools
running build_ext
Traceback (most recent call last):
File "
----------------------------------------
Command "/home/xb37w/miniconda2/bin/python -u -c "import setuptools, tokenize;file='/tmp/pip-build-4aFtCh/pybedtools/setup.py';f=getattr(tokenize, 'open', open)(file);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, file, 'exec'))" install --record /tmp/pip-D2O3Mi-record/install-record.txt --single-version-externally-managed --compile --user --prefix=" failed with error code 1 in /tmp/pip-build-4aFtCh/pybedtools/
What does this give you?
pip freeze | grep Cython
Is Docker/Singularity an alternative you could use on the cluster? This would be the ideal workaround otherwise.
@t-neumann Nothing happens when I do that. Same error msg from installer. I will try to see if I can use Docker. Thank you so much for the help!
Well but then this means Cython is not installed.
We use Singularity (which bootstraps Docker images) on our Cluster and it's a supernice workaround. Should deal with any of the above issues.
While I'm trying to get Docker to work, I ran ngm for my fastq file and ran slamdunk locally from filter, worked like a charm until "count", where the error for tag came up again. Is there a fix for this:
xinyangs-mbp:Slam-seq xinyangbing$ slamdunk count -o ./data/ -s ./data/ -r miRNA.fa -b mm10_miRNA.bed ./data/S41_filtered.bam
Running slamDunk tcount for 1 files (1 threads)
Traceback (most recent call last):
File "/Users/xinyangbing/miniconda2/bin/slamdunk", line 11, in
Hi,
yeah as I said it's quite tedious:
Did you check whether you checked out the correct ngm version? (0.5.2)
And then you also need to make sure to run it with the same parameter set, which should be something like this (assuming you used QuantSeq and use multimapper recovery):
'ngm -b -r reference.fasta -q fastq-file -t threads --slam-seq 2 -5 12 -l --rg-id readgroup --rg-sm samplename:sampletype:sampletime -n 100 --strata -o out.bam'
I really recommend to run it out of the box to not run into mistakes.
Hi, this may not be what's causing the error and please correct me if I'm wrong since I'm not familiar with C++, but looking through the source code for NextGenMap, it seems like --slam-seq 2
would evaluate to False
based on line 57 in NextGenMap/src/CS.cpp:
static bool const slamSeq = Config.GetInt(SLAM_SEQ) & 0x4;
If that's the case, perhaps there are SLAM-seq specific tags that NGM is supposed to be (and is assumed to be) adding but actually is not due to incorrect invocation?
I am not trying to run slamdunk on singularity on our cluster and I get this error, please help (loading updated numpy installed on the cluster does not help):
[xb37w@ghpcc06 ~]$ singularity shell docker://tobneu/slamdunk Docker image path: index.docker.io/tobneu/slamdunk:latest Cache folder set to /home/xb37w/.singularity/docker Creating container runtime... Exploding layer: sha256:50aff78429b146489e8a6cb9334d93a6d81d5de2edc4fbf5e2d4d9253625753e.tar.gz Exploding layer: sha256:f6d82e297bce031a3de1fa8c1587535e34579abce09a61e37f5a225a8667422f.tar.gz Exploding layer: sha256:275abb2c8a6f1ce8e67a388a11f3cc014e98b36ff993a6ed1cc7cd6ecb4dd61b.tar.gz Exploding layer: sha256:9f15a39356d6fc1df0a77012bf1aa2150b683e46be39d1c51bc7a320f913e322.tar.gz Exploding layer: sha256:fc0342a94c89e477c821328ccb542e6fb86ce4ef4ebbf1098e85669e051ef0dd.tar.gz Exploding layer: sha256:96006874cf3fecdc2a5294f0069888f750ed15de87a449acd0b7d6a910d20568.tar.gz Exploding layer: sha256:c6a9ef4b9995d615851d7786fbc2fe72f72321bee1a87d66919b881a0336525a.tar.gz WARNING: Non existent bind point (directory) in container: '/project' WARNING: Non existent bind point (directory) in container: '/lsf' WARNING: Non existent bind point (directory) in container: '/share' WARNING: Non existent bind point (directory) in container: '/nl' Singularity: Invoking an interactive shell within container...
Singularity slamdunk:~> slamdunk map -r miRNA.fa -o ./ -5 5 -n 1 S13_trim.fastq
Traceback (most recent call last):
File "/usr/local/bin/slamdunk", line 5, in
@xbing88
Looks like there's a problem in the current build. Try the official release for now
tobneu/slamdunk:v0.2.4
But I will have a look at this soonish.
Build fixed on docker hub
Where exactly should we keep the fasta, bed and fastq files. I keep getting the following error when running it from py2_env virtual enviromnent:
Traceback (most recent call last):
File "/home/aparajita/py2_env/bin/slamdunk", line 8, in
Since you created a new issue, I would let this one closed.
Please help:
-mbp:Slam-seq xbing88$ slamdunk filter -o output -b mm10_miRNA.bed S13_bowtie2_sorted.bam Running slamDunk filter for 1 files (1 threads) [W::hts_idx_load2] The index file is older than the data file: S13_bowtie2_sorted.bam.bai Traceback (most recent call last): File "/Users/xbing88/miniconda2/bin/slamdunk", line 11, in
sys.exit(run())
File "/Users/xbing88/miniconda2/lib/python2.7/site-packages/slamdunk/slamdunk.py", line 450, in run
results = Parallel(n_jobs=n, verbose=verbose)(delayed(runFilter)(tid, args.bam[tid], args.bed, args.mq, args.identity, args.nm, outputDirectory) for tid in range(0, len(args.bam)))
File "/Users/xbing88/miniconda2/lib/python2.7/site-packages/joblib/parallel.py", line 779, in call
while self.dispatch_one_batch(iterator):
File "/Users/xbing88/miniconda2/lib/python2.7/site-packages/joblib/parallel.py", line 625, in dispatch_one_batch
self._dispatch(tasks)
File "/Users/xbing88/miniconda2/lib/python2.7/site-packages/joblib/parallel.py", line 588, in _dispatch
job = self._backend.apply_async(batch, callback=cb)
File "/Users/xbing88/miniconda2/lib/python2.7/site-packages/joblib/_parallel_backends.py", line 111, in apply_async
result = ImmediateResult(func)
File "/Users/xinyangbing/miniconda2/lib/python2.7/site-packages/joblib/_parallel_backends.py", line 332, in init
self.results = batch()
File "/Users/xbing88/miniconda2/lib/python2.7/site-packages/joblib/parallel.py", line 131, in call
return [func(*args, **kwargs) for func, args, kwargs in self.items]
File "/Users/xbing88/miniconda2/lib/python2.7/site-packages/slamdunk/slamdunk.py", line 170, in runFilter
filter.Filter(bam, outputBAM, getLogFile(outputLOG), bed, mq, minIdentity, maxNM, printOnly, verbose)
File "/Users/xbing88/miniconda2/lib/python2.7/site-packages/slamdunk/dunks/filter.py", line 268, in Filter
mappedReads, unmappedReads, filteredReads, mqFiltered, idFiltered, nmFiltered, multimapper = multimapUTRRetainment (infile, outfile, bed, minIdentity, NM, log)
File "/Users/xbing88/miniconda2/lib/python2.7/site-packages/slamdunk/dunks/filter.py", line 105, in multimapUTRRetainment
if(float(read.get_tag("XI")) < minIdentity):
File "pysam/libcalignedsegment.pyx", line 2383, in pysam.libcalignedsegment.AlignedSegment.get_tag (pysam/libcalignedsegment.c:27819)
File "pysam/libcalignedsegment.pyx", line 2425, in pysam.libcalignedsegment.AlignedSegment.get_tag (pysam/libcalignedsegment.c:27229)
KeyError: "tag 'XI' not present"