snaptools alignment error

r3fang / SnapATAC

Analysis Pipeline for Single Cell ATAC-seq

GNU General Public License v3.0

301 stars 125 forks source link

snaptools alignment error #64

Closed cuperusj closed 4 years ago

cuperusj commented 5 years ago

Using 10x data, post fastq-dex, when i run the bwa aligner I continually receive this error: [M::mem_pestat] skip orientation RR as there are not enough pairs

I'm guessing this might have to do with the modification of the header to include the barcodes?

cuperusj commented 5 years ago

here's my command: snaptools align-paired-end --input-reference=TAIR10.fa --input-fastq1=RootR1.fq --input-fastq2=RootR3.fq --output-bam=Root_SNAP.bam --aligner=bwa --path-to-aligner=./bwa --min-cov=0 --num-threads=5 --if-sort=True --tmp-folder=./ --overwrite=TRUE

freshly installed bwa

r3fang commented 5 years ago

Did you change the read name for both RootR1.fq and RootR3.fq?

-- Rongxin Fang, Ren Lab Ludwig Cancer Research Bioinformatics Ph.D. Student University of California, San Diego

On Aug 4, 2019, at 11:48 AM, cuperusj notifications@github.com wrote:

here's my command: snaptools align-paired-end --input-reference=TAIR10.fa --input-fastq1=RootR1.fq --input-fastq2=RootR3.fq --output-bam=Root_SNAP.bam --aligner=bwa --path-to-aligner=./bwa --min-cov=0 --num-threads=5 --if-sort=True --tmp-folder=./ --overwrite=TRUE

freshly installed bwa

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/r3fang/SnapATAC/issues/64?email_source=notifications&email_token=ABT6GG4U4XEBPC3OOBHSZZTQC4P6PA5CNFSM4IJF7OY2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD3QHMFI#issuecomment-518026773, or mute the thread https://github.com/notifications/unsubscribe-auth/ABT6GG3DC2QHJHD7YMAPLJLQC4P6PANCNFSM4IJF7OYQ.

cuperusj commented 5 years ago

the read names have the barcode inserted into them, like the 10x part of tutorial says to do

@AATGGGTCTGCCACGT:NS500272:681:HN5NLAFXY:1:11101:5889:1051 1:N:0:ACGAGTAG GCCTANAAGAAGGAAAAAAAAAAAAAATCTCTTTTTGTTTGACCTTTTTT + AAAAA#AEEEE6AEEEEEEEEEEE6E//E<E/EAEEEEAAE/E/A/AEEE @AGTACAATAAAACTGA:NS500272:681:HN5NLAFXY:1:11101:3322:1051 1:N:0:ACGAGTAG GTGCANTAGTATGATCTTTCATTTCAAAATTAATAAATTATTATCACATG

r3fang commented 5 years ago

If I understand correctly, this is R1.fastq file, my question is did you add the barcode to R3.fastq? Can you show me this:

head RootR1.fq

And

head RootR3.fq

Best,

Rongxin Fang, Ren Lab Ludwig Cancer Research Bioinformatics Ph.D. Student University of California, San Diego

On Aug 4, 2019, at 11:52 AM, cuperusj notifications@github.com wrote:

the read names have the barcode inserted into them, like the 10x part of tutorial says to do

@AATGGGTCTGCCACGT:NS500272:681:HN5NLAFXY:1:11101:5889:1051 1:N:0:ACGAGTAG GCCTANAAGAAGGAAAAAAAAAAAAAATCTCTTTTTGTTTGACCTTTTTT + AAAAA#AEEEE6AEEEEEEEEEEE6E//E<E/EAEEEEAAE/E/A/AEEE @AGTACAATAAAACTGA:NS500272:681:HN5NLAFXY:1:11101:3322:1051 1:N:0:ACGAGTAG GTGCANTAGTATGATCTTTCATTTCAAAATTAATAAATTATTATCACATG

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/r3fang/SnapATAC/issues/64?email_source=notifications&email_token=ABT6GG565VU4NEY3VN2D34DQC4QPHA5CNFSM4IJF7OY2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD3QHOOI#issuecomment-518027065, or mute the thread https://github.com/notifications/unsubscribe-auth/ABT6GG4R7OYM54U43A4AHALQC4QPHANCNFSM4IJF7OYQ.

cuperusj commented 5 years ago

head RootR3.fq @NCAGTCCTGCAGTCTG:NS500272:681:HN5NLAFXY:1:11101:6969:1042 3:N:0:ACGAGTAG GGGGAATGGAATCCATTGAACGTGGCTTTGATTTTGTCTTGTCTGTAGTA + AAAAAEEEEEEEEE/EA//EAEEEEE</EAEEEEEEEEEEEEEEEEEEEE @NCTTGGTACGTCTTGA:NS500272:681:HN5NLAFXY:1:11101:10764:1042 3:N:0:ACGAGTAG GCGGTAAACTGTTTAAAAAGTTGCTATAACCCCTTCTCTTGTATTTTTTT + AAA/AE/EEEEE/EEE/EAEE/E/EEE6/E<<AEEAAEA/A/E/6EEEEA @NAGGGACGATTGTTTC:NS500272:681:HN5NLAFXY:1:11101:10556:1043 3:N:0:ACGAGTAG TTATGACAGAGAAATCACAAAACCCGAATGTGTGAAAATTCAATTGCAAG (base) D-10-19-52-121:Downloads cuperusj$ head RootR1.fq @NCAGTCCTGCAGTCTG:NS500272:681:HN5NLAFXY:1:11101:6969:1042 1:N:0:ACGAGTAG AGATGNGGATGACGTGTTCTTTTCTTATTATTACTACAGACAAGACAAAA + AAAAA#EEAAEEEEEEEEEEEEEE<EEAEAEEEEEEEAEEEEEEAEEEEE @NCTTGGTACGTCTTGA:NS500272:681:HN5NLAFXY:1:11101:10764:1042 1:N:0:ACGAGTAG GGCCANTCCTATGGCGTTCTTGTGAGTCGCCAACTTCAAAAAAAGATAAA + AAAA6#EEEEEA/EEEEEEEEEEEEEEEEEEEAEEEEEEEEEEEEE/EAE @NAGGGACGATTGTTTC:NS500272:681:HN5NLAFXY:1:11101:10556:1043 1:N:0:ACGAGTAG CTACGNCGTCTACGCCTCGCATGCCTATCATATTTAACCGTCAATAATGG

cuperusj commented 5 years ago

they seem to match up, so not sure what im missing.... even if i look ~10k into the files they seem identical

cuperusj commented 5 years ago

head -999997 RootR3.fq | tail -1 @CTTAGTCGAATCATGC:NS500272:681:HN5NLAFXY:1:11108:5497:3154 3:N:0:ACGAGTAG head -999997 RootR1.fq | tail -1 @CTTAGTCGAATCATGC:NS500272:681:HN5NLAFXY:1:11108:5497:3154 1:N:0:ACGAGTAG

cuperusj commented 5 years ago

I thought at first it was some issue with the concatenation, but i get the same error if i run it on just a pre-cat subset

r3fang commented 5 years ago

How about the fastq file before you ran demultiplexing? The error is because “3:N:0:ACGAGTAG” and “1:N:0:ACGAGTAG” does not match, this should be already there before you performed the demultiplexing.

-- Rongxin Fang, Ren Lab Ludwig Cancer Research Bioinformatics Ph.D. Student University of California, San Diego

On Aug 4, 2019, at 12:00 PM, cuperusj notifications@github.com wrote:

head -999997 RootR3.fq | tail -1 @CTTAGTCGAATCATGC:NS500272:681:HN5NLAFXY:1:11108:5497:3154 3:N:0:ACGAGTAG head -999997 RootR1.fq | tail -1 @CTTAGTCGAATCATGC:NS500272:681:HN5NLAFXY:1:11108:5497:3154 1:N:0:ACGAGTAG

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/r3fang/SnapATAC/issues/64?email_source=notifications&email_token=ABT6GG37I4HGGL526KSCNU3QC4RNZA5CNFSM4IJF7OY2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD3QHTTI#issuecomment-518027725, or mute the thread https://github.com/notifications/unsubscribe-auth/ABT6GG4PLZFF3PXBLZPEHKLQC4RNZANCNFSM4IJF7OYQ.

cuperusj commented 5 years ago

thats the output from a standard nextseq demultiplexing, I can try stripping that out, but seems like BWA should be fine with that.

r3fang commented 5 years ago

I think so. The reason that you haver received that error is because BWA requires read name from R1 and R2 to be exactly the same. I think the error will be gone if you script it out from the read name.

Best,

Rongxin Fang, Ren Lab Ludwig Cancer Research Bioinformatics Ph.D. Student University of California, San Diego

On Aug 4, 2019, at 12:08 PM, cuperusj notifications@github.com wrote:

thats the output from a standard nextseq demultiplexing, I can try stripping that out, but seems like BWA should be fine with that.

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/r3fang/SnapATAC/issues/64?email_source=notifications&email_token=ABT6GG3RUNA4TJSEPNJGR2TQC4SLPA5CNFSM4IJF7OY2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD3QHYHY#issuecomment-518028319, or mute the thread https://github.com/notifications/unsubscribe-auth/ABT6GG5MI3GQDECKQBED5FDQC4SLPANCNFSM4IJF7OYQ.

cuperusj commented 5 years ago

error maintains, when i stripped off that part

head -9999997 RootR1_stripped.fq | tail -1 @TCGCCTCCTACGTAGT:NS500272:681:HN5NLAFXY:1:21306:2984:16319 head -9999997 RootR3_stripped.fq | tail -1 @TCGCCTCCTACGTAGT:NS500272:681:HN5NLAFXY:1:21306:2984:16319

On Aug 4, 2019, at 12:10 PM, Rongxin Fang notifications@github.com wrote:

I think so. The reason that you haver received that error is because BWA requires read name from R1 and R2 to be exactly the same. I think the error will be gone if you script it out from the read name.

Best,

Rongxin Fang, Ren Lab Ludwig Cancer Research Bioinformatics Ph.D. Student University of California, San Diego

On Aug 4, 2019, at 12:08 PM, cuperusj notifications@github.com wrote:

thats the output from a standard nextseq demultiplexing, I can try stripping that out, but seems like BWA should be fine with that.

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/r3fang/SnapATAC/issues/64?email_source=notifications&email_token=ABT6GG3RUNA4TJSEPNJGR2TQC4SLPA5CNFSM4IJF7OY2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD3QHYHY#issuecomment-518028319, or mute the thread https://github.com/notifications/unsubscribe-auth/ABT6GG5MI3GQDECKQBED5FDQC4SLPANCNFSM4IJF7OYQ.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/r3fang/SnapATAC/issues/64?email_source=notifications&email_token=ABR4NOS47LM2EKRUZEWJ573QC4STXA5CNFSM4IJF7OY2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD3QHZKY#issuecomment-518028459, or mute the thread https://github.com/notifications/unsubscribe-auth/ABR4NOW4F5LCNDGDIBEMBZDQC4STXANCNFSM4IJF7OYQ.

r3fang commented 5 years ago

what if you just run BWA MEM on your demultiplexed fastq file?

cuperusj commented 5 years ago

nope, same error, Im just doing exactly what the tutorial said to do for the reads the dex-fastq

cuperusj commented 5 years ago

I would have assumed bwa uses the @NS handle, which dex-fastq removes

r3fang commented 5 years ago

This is wired, because I have ran BWA on demultiplexed fastq files all the time, never experienced this error. What’s the BWA version? Also, you can try bowtie as aligner.

Sent from my iPhone

On Aug 4, 2019, at 7:24 PM, cuperusj notifications@github.com wrote:

I would have assumed bwa uses the @ns handle, which dex-fastq removes

— You are receiving this because you commented. Reply to this email directly, view it on GitHub, or mute the thread.

cuperusj commented 5 years ago

Ok I'll try bowtie tomorrow, thanks for being so responsive!

On Sun, Aug 4, 2019, 8:39 PM Rongxin Fang notifications@github.com wrote:

This is wired, because I have ran BWA on demultiplexed fastq files all the time, never experienced this error. What’s the BWA version? Also, you can try bowtie as aligner.

Sent from my iPhone

On Aug 4, 2019, at 7:24 PM, cuperusj notifications@github.com wrote:

I would have assumed bwa uses the @ns handle, which dex-fastq removes

— You are receiving this because you commented. Reply to this email directly, view it on GitHub, or mute the thread.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/r3fang/SnapATAC/issues/64?email_source=notifications&email_token=ABR4NOXKY5MHGVAWNUJ6ECLQC6OIDA5CNFSM4IJF7OY2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD3QS72Y#issuecomment-518074347, or mute the thread https://github.com/notifications/unsubscribe-auth/ABR4NOUPVKFN6EP5XM6D2ALQC6OIDANCNFSM4IJF7OYQ .

cuperusj commented 5 years ago

I decided to just grab the bam from 10x instead, now getting this error from pre-snap:

snaptools snap-pre --input-file=sort.snap.bam --output-snap=RootCombined.snap --genome-name=../Arabidopsis_genome/fasta/genome --genome-size=chrom.length --min-mapq=30 --min-flen=0 --max-flen=1000 --keep-chrm=TRUE --keep-single=TRUE --keep-secondary=False --overwrite=True --max-num=1000000 --min-cov=100 --verbose=True error: --genome-name unrecoginized genome identifier ../Arabidopsis_genome/fasta/genome Do you mean: dasNov3,aptMan1,bosTau8,bosTau7,bosTau6,bosTau4,bosTau3,bosTau2,canFam3,canFam2,canFam1,triMan1,geoFor1,anoGam1,apiMel3,apiMel2,apiMel1,droSim1 ?

I dont see where/how/what this genome-name refers to exactly We are using Arabidopsis TAIR10

On Aug 4, 2019, at 10:11 PM, josh cuperus cuperusj@gmail.com wrote:

Ok I'll try bowtie tomorrow, thanks for being so responsive!

On Sun, Aug 4, 2019, 8:39 PM Rongxin Fang <notifications@github.com mailto:notifications@github.com> wrote: This is wired, because I have ran BWA on demultiplexed fastq files all the time, never experienced this error. What’s the BWA version? Also, you can try bowtie as aligner.

Sent from my iPhone

On Aug 4, 2019, at 7:24 PM, cuperusj <notifications@github.com mailto:notifications@github.com> wrote:

I would have assumed bwa uses the @ns handle, which dex-fastq removes

— You are receiving this because you commented. Reply to this email directly, view it on GitHub, or mute the thread. — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/r3fang/SnapATAC/issues/64?email_source=notifications&email_token=ABR4NOXKY5MHGVAWNUJ6ECLQC6OIDA5CNFSM4IJF7OY2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD3QS72Y#issuecomment-518074347, or mute the thread https://github.com/notifications/unsubscribe-auth/ABR4NOUPVKFN6EP5XM6D2ALQC6OIDANCNFSM4IJF7OYQ.

r3fang commented 5 years ago

Genome name is wrong. You can set it as “hg19”. This tag is currently not used

Sent from my iPhone

On Aug 5, 2019, at 4:49 PM, cuperusj notifications@github.com wrote:

I decided to just grab the bam from 10x instead, now getting this error from pre-snap:

snaptools snap-pre --input-file=sort.snap.bam --output-snap=RootCombined.snap --genome-name=../Arabidopsis_genome/fasta/genome --genome-size=chrom.length --min-mapq=30 --min-flen=0 --max-flen=1000 --keep-chrm=TRUE --keep-single=TRUE --keep-secondary=False --overwrite=True --max-num=1000000 --min-cov=100 --verbose=True error: --genome-name unrecoginized genome identifier ../Arabidopsis_genome/fasta/genome Do you mean: dasNov3,aptMan1,bosTau8,bosTau7,bosTau6,bosTau4,bosTau3,bosTau2,canFam3,canFam2,canFam1,triMan1,geoFor1,anoGam1,apiMel3,apiMel2,apiMel1,droSim1 ?

I dont see where/how/what this genome-name refers to exactly We are using Arabidopsis TAIR10

On Aug 4, 2019, at 10:11 PM, josh cuperus cuperusj@gmail.com wrote:

Ok I'll try bowtie tomorrow, thanks for being so responsive!

On Sun, Aug 4, 2019, 8:39 PM Rongxin Fang <notifications@github.com mailto:notifications@github.com> wrote: This is wired, because I have ran BWA on demultiplexed fastq files all the time, never experienced this error. What’s the BWA version? Also, you can try bowtie as aligner.

Sent from my iPhone

On Aug 4, 2019, at 7:24 PM, cuperusj <notifications@github.com mailto:notifications@github.com> wrote:

I would have assumed bwa uses the @ns handle, which dex-fastq removes

— You are receiving this because you commented. Reply to this email directly, view it on GitHub, or mute the thread. — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/r3fang/SnapATAC/issues/64?email_source=notifications&email_token=ABR4NOXKY5MHGVAWNUJ6ECLQC6OIDA5CNFSM4IJF7OY2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD3QS72Y#issuecomment-518074347, or mute the thread https://github.com/notifications/unsubscribe-auth/ABR4NOUPVKFN6EP5XM6D2ALQC6OIDANCNFSM4IJF7OYQ.

— You are receiving this because you commented. Reply to this email directly, view it on GitHub, or mute the thread.

cuperusj commented 5 years ago

snap file created successfully, now this:

snaptools snap-add-bmat --snap-file=RootCombined.snap --bin-size-lis 1000 5000 --verbose=True Traceback (most recent call last): File "/net/gs/vol3/software/modules-sw-python/2.7.3/snaptools/1.4.7/Linux/RHEL6/x86_64/bin/snaptools", line 4, in import('pkg_resources').run_script('snaptools==1.4.7', 'snaptools') File "/net/gs/vol3/software/modules-sw/python/2.7.3/Linux/RHEL6/x86_64/lib/python2.7/site-packages/pkg_resources/init.py", line 739, in run_script self.require(requires)[0].run_script(script_name, ns) File "/net/gs/vol3/software/modules-sw/python/2.7.3/Linux/RHEL6/x86_64/lib/python2.7/site-packages/pkg_resources/init.py", line 1494, in run_script exec(code, namespace, namespace) File "/net/gs/vol3/software/modules-sw-python/2.7.3/snaptools/1.4.7/Linux/RHEL6/x86_64/lib/python2.7/site-packages/snaptools-1.4.7-py2.7.egg/EGG-INFO/scripts/snaptools", line 38, in parse_args() File "/net/gs/vol3/software/modules-sw-python/2.7.3/snaptools/1.4.7/Linux/RHEL6/x86_64/lib/python2.7/site-packages/snaptools-1.4.7-py2.7.egg/snaptools/parser.py", line 168, in parse_args verbose=args.verbose) File "/net/gs/vol3/software/modules-sw-python/2.7.3/snaptools/1.4.7/Linux/RHEL6/x86_64/lib/python2.7/site-packages/snaptools-1.4.7-py2.7.egg/snaptools/add_bmat.py", line 112, in snap_bmat f = h5py.File(snap_file, "a", libver='earliest'); File "/net/gs/vol3/software/modules-sw-python/2.7.3/h5py/2.7.1/Linux/RHEL6/x86_64/lib/python2.7/site-packages/h5py-2.7.1-py2.7-linux-x86_64.egg/h5py/_hl/files.py", line 269, in init fid = make_fid(name, mode, userblock_size, fapl, swmr=swmr) File "/net/gs/vol3/software/modules-sw-python/2.7.3/h5py/2.7.1/Linux/RHEL6/x86_64/lib/python2.7/site-packages/h5py-2.7.1-py2.7-linux-x86_64.egg/h5py/_hl/files.py", line 113, in make_fid fid = h5f.create(name, h5f.ACC_EXCL, fapl=fapl, fcpl=fcpl) File "h5py/_objects.pyx", line 54, in h5py._objects.with_phil.wrapper (/data/nobackup/h5py-2.7.1/h5py/_objects.c:2846) File "h5py/_objects.pyx", line 55, in h5py._objects.with_phil.wrapper (/data/nobackup/h5py-2.7.1/h5py/_objects.c:2804) File "h5py/h5f.pyx", line 98, in h5py.h5f.create (/data/nobackup/h5py-2.7.1/h5py/h5f.c:2287) IOError: Unable to create file (unable to open file: name = 'RootCombined.snap', errno = 17, error message = 'File exists', flags = 15, o_flags = c2)

On Aug 5, 2019, at 4:52 PM, Rongxin Fang notifications@github.com wrote:

Genome name is wrong. You can set it as “hg19”. This tag is currently not used

Sent from my iPhone

On Aug 5, 2019, at 4:49 PM, cuperusj notifications@github.com wrote:

I decided to just grab the bam from 10x instead, now getting this error from pre-snap:

snaptools snap-pre --input-file=sort.snap.bam --output-snap=RootCombined.snap --genome-name=../Arabidopsis_genome/fasta/genome --genome-size=chrom.length --min-mapq=30 --min-flen=0 --max-flen=1000 --keep-chrm=TRUE --keep-single=TRUE --keep-secondary=False --overwrite=True --max-num=1000000 --min-cov=100 --verbose=True error: --genome-name unrecoginized genome identifier ../Arabidopsis_genome/fasta/genome Do you mean: dasNov3,aptMan1,bosTau8,bosTau7,bosTau6,bosTau4,bosTau3,bosTau2,canFam3,canFam2,canFam1,triMan1,geoFor1,anoGam1,apiMel3,apiMel2,apiMel1,droSim1 ?

I dont see where/how/what this genome-name refers to exactly We are using Arabidopsis TAIR10

On Aug 4, 2019, at 10:11 PM, josh cuperus cuperusj@gmail.com wrote:

Ok I'll try bowtie tomorrow, thanks for being so responsive!

On Sun, Aug 4, 2019, 8:39 PM Rongxin Fang <notifications@github.com mailto:notifications@github.com> wrote: This is wired, because I have ran BWA on demultiplexed fastq files all the time, never experienced this error. What’s the BWA version? Also, you can try bowtie as aligner.

Sent from my iPhone

On Aug 4, 2019, at 7:24 PM, cuperusj <notifications@github.com mailto:notifications@github.com> wrote:

I would have assumed bwa uses the @ns handle, which dex-fastq removes

— You are receiving this because you commented. Reply to this email directly, view it on GitHub, or mute the thread. — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/r3fang/SnapATAC/issues/64?email_source=notifications&email_token=ABR4NOXKY5MHGVAWNUJ6ECLQC6OIDA5CNFSM4IJF7OY2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD3QS72Y#issuecomment-518074347, or mute the thread https://github.com/notifications/unsubscribe-auth/ABR4NOUPVKFN6EP5XM6D2ALQC6OIDANCNFSM4IJF7OYQ.

— You are receiving this because you commented. Reply to this email directly, view it on GitHub, or mute the thread. — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/r3fang/SnapATAC/issues/64?email_source=notifications&email_token=ABR4NOXGRZJ4BJYXCHD4IHLQDC4MZA5CNFSM4IJF7OY2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD3TM42A#issuecomment-518442600, or mute the thread https://github.com/notifications/unsubscribe-auth/ABR4NOV2FK3MLPTV5Z6D2KLQDC4MZANCNFSM4IJF7OYQ.

cuperusj commented 5 years ago

nevermind, I think its fine

On Aug 5, 2019, at 8:42 PM, josh cuperus cuperusj@gmail.com wrote:

snap file created successfully, now this:

snaptools snap-add-bmat --snap-file=RootCombined.snap --bin-size-lis 1000 5000 --verbose=True Traceback (most recent call last): File "/net/gs/vol3/software/modules-sw-python/2.7.3/snaptools/1.4.7/Linux/RHEL6/x86_64/bin/snaptools", line 4, in import('pkg_resources').run_script('snaptools==1.4.7', 'snaptools') File "/net/gs/vol3/software/modules-sw/python/2.7.3/Linux/RHEL6/x86_64/lib/python2.7/site-packages/pkg_resources/init.py", line 739, in run_script self.require(requires)[0].run_script(script_name, ns) File "/net/gs/vol3/software/modules-sw/python/2.7.3/Linux/RHEL6/x86_64/lib/python2.7/site-packages/pkg_resources/init.py", line 1494, in run_script exec(code, namespace, namespace) File "/net/gs/vol3/software/modules-sw-python/2.7.3/snaptools/1.4.7/Linux/RHEL6/x86_64/lib/python2.7/site-packages/snaptools-1.4.7-py2.7.egg/EGG-INFO/scripts/snaptools", line 38, in parse_args() File "/net/gs/vol3/software/modules-sw-python/2.7.3/snaptools/1.4.7/Linux/RHEL6/x86_64/lib/python2.7/site-packages/snaptools-1.4.7-py2.7.egg/snaptools/parser.py", line 168, in parse_args verbose=args.verbose) File "/net/gs/vol3/software/modules-sw-python/2.7.3/snaptools/1.4.7/Linux/RHEL6/x86_64/lib/python2.7/site-packages/snaptools-1.4.7-py2.7.egg/snaptools/add_bmat.py", line 112, in snap_bmat f = h5py.File(snap_file, "a", libver='earliest'); File "/net/gs/vol3/software/modules-sw-python/2.7.3/h5py/2.7.1/Linux/RHEL6/x86_64/lib/python2.7/site-packages/h5py-2.7.1-py2.7-linux-x86_64.egg/h5py/_hl/files.py", line 269, in init fid = make_fid(name, mode, userblock_size, fapl, swmr=swmr) File "/net/gs/vol3/software/modules-sw-python/2.7.3/h5py/2.7.1/Linux/RHEL6/x86_64/lib/python2.7/site-packages/h5py-2.7.1-py2.7-linux-x86_64.egg/h5py/_hl/files.py", line 113, in make_fid fid = h5f.create(name, h5f.ACC_EXCL, fapl=fapl, fcpl=fcpl) File "h5py/_objects.pyx", line 54, in h5py._objects.with_phil.wrapper (/data/nobackup/h5py-2.7.1/h5py/_objects.c:2846) File "h5py/_objects.pyx", line 55, in h5py._objects.with_phil.wrapper (/data/nobackup/h5py-2.7.1/h5py/_objects.c:2804) File "h5py/h5f.pyx", line 98, in h5py.h5f.create (/data/nobackup/h5py-2.7.1/h5py/h5f.c:2287) IOError: Unable to create file (unable to open file: name = 'RootCombined.snap', errno = 17, error message = 'File exists', flags = 15, o_flags = c2)

On Aug 5, 2019, at 4:52 PM, Rongxin Fang <notifications@github.com mailto:notifications@github.com> wrote:

Genome name is wrong. You can set it as “hg19”. This tag is currently not used

Sent from my iPhone

On Aug 5, 2019, at 4:49 PM, cuperusj <notifications@github.com mailto:notifications@github.com> wrote:

I decided to just grab the bam from 10x instead, now getting this error from pre-snap:

snaptools snap-pre --input-file=sort.snap.bam --output-snap=RootCombined.snap --genome-name=../Arabidopsis_genome/fasta/genome --genome-size=chrom.length --min-mapq=30 --min-flen=0 --max-flen=1000 --keep-chrm=TRUE --keep-single=TRUE --keep-secondary=False --overwrite=True --max-num=1000000 --min-cov=100 --verbose=True error: --genome-name unrecoginized genome identifier ../Arabidopsis_genome/fasta/genome Do you mean: dasNov3,aptMan1,bosTau8,bosTau7,bosTau6,bosTau4,bosTau3,bosTau2,canFam3,canFam2,canFam1,triMan1,geoFor1,anoGam1,apiMel3,apiMel2,apiMel1,droSim1 ?

I dont see where/how/what this genome-name refers to exactly We are using Arabidopsis TAIR10

On Aug 4, 2019, at 10:11 PM, josh cuperus <cuperusj@gmail.com mailto:cuperusj@gmail.com> wrote:

Ok I'll try bowtie tomorrow, thanks for being so responsive!

On Sun, Aug 4, 2019, 8:39 PM Rongxin Fang <notifications@github.com mailto:notifications@github.com <mailto:notifications@github.com mailto:notifications@github.com>> wrote: This is wired, because I have ran BWA on demultiplexed fastq files all the time, never experienced this error. What’s the BWA version? Also, you can try bowtie as aligner.

Sent from my iPhone

On Aug 4, 2019, at 7:24 PM, cuperusj <notifications@github.com mailto:notifications@github.com <mailto:notifications@github.com mailto:notifications@github.com>> wrote:

I would have assumed bwa uses the @ns handle, which dex-fastq removes

— You are receiving this because you commented. Reply to this email directly, view it on GitHub, or mute the thread. — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <https://github.com/r3fang/SnapATAC/issues/64?email_source=notifications&email_token=ABR4NOXKY5MHGVAWNUJ6ECLQC6OIDA5CNFSM4IJF7OY2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD3QS72Y#issuecomment-518074347 https://github.com/r3fang/SnapATAC/issues/64?email_source=notifications&email_token=ABR4NOXKY5MHGVAWNUJ6ECLQC6OIDA5CNFSM4IJF7OY2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD3QS72Y#issuecomment-518074347>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ABR4NOUPVKFN6EP5XM6D2ALQC6OIDANCNFSM4IJF7OYQ https://github.com/notifications/unsubscribe-auth/ABR4NOUPVKFN6EP5XM6D2ALQC6OIDANCNFSM4IJF7OYQ>.

— You are receiving this because you commented. Reply to this email directly, view it on GitHub, or mute the thread. — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/r3fang/SnapATAC/issues/64?email_source=notifications&email_token=ABR4NOXGRZJ4BJYXCHD4IHLQDC4MZA5CNFSM4IJF7OY2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD3TM42A#issuecomment-518442600, or mute the thread https://github.com/notifications/unsubscribe-auth/ABR4NOV2FK3MLPTV5Z6D2KLQDC4MZANCNFSM4IJF7OYQ.

r3fang commented 5 years ago

Are you running this on windows?

Sent from my iPhone

On Aug 5, 2019, at 8:48 PM, cuperusj notifications@github.com wrote:

nevermind, I think its fine

On Aug 5, 2019, at 8:42 PM, josh cuperus cuperusj@gmail.com wrote:

snap file created successfully, now this:

snaptools snap-add-bmat --snap-file=RootCombined.snap --bin-size-lis 1000 5000 --verbose=True Traceback (most recent call last): File "/net/gs/vol3/software/modules-sw-python/2.7.3/snaptools/1.4.7/Linux/RHEL6/x86_64/bin/snaptools", line 4, in import('pkg_resources').run_script('snaptools==1.4.7', 'snaptools') File "/net/gs/vol3/software/modules-sw/python/2.7.3/Linux/RHEL6/x86_64/lib/python2.7/site-packages/pkg_resources/init.py", line 739, in run_script self.require(requires)[0].run_script(script_name, ns) File "/net/gs/vol3/software/modules-sw/python/2.7.3/Linux/RHEL6/x86_64/lib/python2.7/site-packages/pkg_resources/init.py", line 1494, in run_script exec(code, namespace, namespace) File "/net/gs/vol3/software/modules-sw-python/2.7.3/snaptools/1.4.7/Linux/RHEL6/x86_64/lib/python2.7/site-packages/snaptools-1.4.7-py2.7.egg/EGG-INFO/scripts/snaptools", line 38, in parse_args() File "/net/gs/vol3/software/modules-sw-python/2.7.3/snaptools/1.4.7/Linux/RHEL6/x86_64/lib/python2.7/site-packages/snaptools-1.4.7-py2.7.egg/snaptools/parser.py", line 168, in parse_args verbose=args.verbose) File "/net/gs/vol3/software/modules-sw-python/2.7.3/snaptools/1.4.7/Linux/RHEL6/x86_64/lib/python2.7/site-packages/snaptools-1.4.7-py2.7.egg/snaptools/add_bmat.py", line 112, in snap_bmat f = h5py.File(snap_file, "a", libver='earliest'); File "/net/gs/vol3/software/modules-sw-python/2.7.3/h5py/2.7.1/Linux/RHEL6/x86_64/lib/python2.7/site-packages/h5py-2.7.1-py2.7-linux-x86_64.egg/h5py/_hl/files.py", line 269, in init fid = make_fid(name, mode, userblock_size, fapl, swmr=swmr) File "/net/gs/vol3/software/modules-sw-python/2.7.3/h5py/2.7.1/Linux/RHEL6/x86_64/lib/python2.7/site-packages/h5py-2.7.1-py2.7-linux-x86_64.egg/h5py/_hl/files.py", line 113, in make_fid fid = h5f.create(name, h5f.ACC_EXCL, fapl=fapl, fcpl=fcpl) File "h5py/_objects.pyx", line 54, in h5py._objects.with_phil.wrapper (/data/nobackup/h5py-2.7.1/h5py/_objects.c:2846) File "h5py/_objects.pyx", line 55, in h5py._objects.with_phil.wrapper (/data/nobackup/h5py-2.7.1/h5py/_objects.c:2804) File "h5py/h5f.pyx", line 98, in h5py.h5f.create (/data/nobackup/h5py-2.7.1/h5py/h5f.c:2287) IOError: Unable to create file (unable to open file: name = 'RootCombined.snap', errno = 17, error message = 'File exists', flags = 15, o_flags = c2)

On Aug 5, 2019, at 4:52 PM, Rongxin Fang <notifications@github.com mailto:notifications@github.com> wrote:

Genome name is wrong. You can set it as “hg19”. This tag is currently not used

Sent from my iPhone

On Aug 5, 2019, at 4:49 PM, cuperusj <notifications@github.com mailto:notifications@github.com> wrote:

I decided to just grab the bam from 10x instead, now getting this error from pre-snap:

snaptools snap-pre --input-file=sort.snap.bam --output-snap=RootCombined.snap --genome-name=../Arabidopsis_genome/fasta/genome --genome-size=chrom.length --min-mapq=30 --min-flen=0 --max-flen=1000 --keep-chrm=TRUE --keep-single=TRUE --keep-secondary=False --overwrite=True --max-num=1000000 --min-cov=100 --verbose=True error: --genome-name unrecoginized genome identifier ../Arabidopsis_genome/fasta/genome Do you mean: dasNov3,aptMan1,bosTau8,bosTau7,bosTau6,bosTau4,bosTau3,bosTau2,canFam3,canFam2,canFam1,triMan1,geoFor1,anoGam1,apiMel3,apiMel2,apiMel1,droSim1 ?

I dont see where/how/what this genome-name refers to exactly We are using Arabidopsis TAIR10

On Aug 4, 2019, at 10:11 PM, josh cuperus <cuperusj@gmail.com mailto:cuperusj@gmail.com> wrote:

Ok I'll try bowtie tomorrow, thanks for being so responsive!

On Sun, Aug 4, 2019, 8:39 PM Rongxin Fang <notifications@github.com mailto:notifications@github.com <mailto:notifications@github.com mailto:notifications@github.com>> wrote: This is wired, because I have ran BWA on demultiplexed fastq files all the time, never experienced this error. What’s the BWA version? Also, you can try bowtie as aligner.

Sent from my iPhone

On Aug 4, 2019, at 7:24 PM, cuperusj <notifications@github.com mailto:notifications@github.com <mailto:notifications@github.com mailto:notifications@github.com>> wrote:

I would have assumed bwa uses the @ns handle, which dex-fastq removes

— You are receiving this because you commented. Reply to this email directly, view it on GitHub, or mute the thread. — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <https://github.com/r3fang/SnapATAC/issues/64?email_source=notifications&email_token=ABR4NOXKY5MHGVAWNUJ6ECLQC6OIDA5CNFSM4IJF7OY2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD3QS72Y#issuecomment-518074347 https://github.com/r3fang/SnapATAC/issues/64?email_source=notifications&email_token=ABR4NOXKY5MHGVAWNUJ6ECLQC6OIDA5CNFSM4IJF7OY2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD3QS72Y#issuecomment-518074347>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ABR4NOUPVKFN6EP5XM6D2ALQC6OIDANCNFSM4IJF7OYQ https://github.com/notifications/unsubscribe-auth/ABR4NOUPVKFN6EP5XM6D2ALQC6OIDANCNFSM4IJF7OYQ>.

— You are receiving this because you commented. Reply to this email directly, view it on GitHub, or mute the thread. — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/r3fang/SnapATAC/issues/64?email_source=notifications&email_token=ABR4NOXGRZJ4BJYXCHD4IHLQDC4MZA5CNFSM4IJF7OY2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD3TM42A#issuecomment-518442600, or mute the thread https://github.com/notifications/unsubscribe-auth/ABR4NOV2FK3MLPTV5Z6D2KLQDC4MZANCNFSM4IJF7OYQ.

— You are receiving this because you commented. Reply to this email directly, view it on GitHub, or mute the thread.

cuperusj commented 5 years ago

linux cluster, i think i accidentally submitted the job twice, thus the error, it seems to be adding bmat to the existing file

On Aug 5, 2019, at 8:59 PM, Rongxin Fang notifications@github.com wrote:

Are you running this on windows?

Sent from my iPhone

On Aug 5, 2019, at 8:48 PM, cuperusj notifications@github.com wrote:

nevermind, I think its fine

On Aug 5, 2019, at 8:42 PM, josh cuperus cuperusj@gmail.com wrote:

snap file created successfully, now this:

snaptools snap-add-bmat --snap-file=RootCombined.snap --bin-size-lis 1000 5000 --verbose=True Traceback (most recent call last): File "/net/gs/vol3/software/modules-sw-python/2.7.3/snaptools/1.4.7/Linux/RHEL6/x86_64/bin/snaptools", line 4, in import('pkg_resources').run_script('snaptools==1.4.7', 'snaptools') File "/net/gs/vol3/software/modules-sw/python/2.7.3/Linux/RHEL6/x86_64/lib/python2.7/site-packages/pkg_resources/init.py", line 739, in run_script self.require(requires)[0].run_script(script_name, ns) File "/net/gs/vol3/software/modules-sw/python/2.7.3/Linux/RHEL6/x86_64/lib/python2.7/site-packages/pkg_resources/init.py", line 1494, in run_script exec(code, namespace, namespace) File "/net/gs/vol3/software/modules-sw-python/2.7.3/snaptools/1.4.7/Linux/RHEL6/x86_64/lib/python2.7/site-packages/snaptools-1.4.7-py2.7.egg/EGG-INFO/scripts/snaptools", line 38, in parse_args() File "/net/gs/vol3/software/modules-sw-python/2.7.3/snaptools/1.4.7/Linux/RHEL6/x86_64/lib/python2.7/site-packages/snaptools-1.4.7-py2.7.egg/snaptools/parser.py", line 168, in parse_args verbose=args.verbose) File "/net/gs/vol3/software/modules-sw-python/2.7.3/snaptools/1.4.7/Linux/RHEL6/x86_64/lib/python2.7/site-packages/snaptools-1.4.7-py2.7.egg/snaptools/add_bmat.py", line 112, in snap_bmat f = h5py.File(snap_file, "a", libver='earliest'); File "/net/gs/vol3/software/modules-sw-python/2.7.3/h5py/2.7.1/Linux/RHEL6/x86_64/lib/python2.7/site-packages/h5py-2.7.1-py2.7-linux-x86_64.egg/h5py/_hl/files.py", line 269, in init fid = make_fid(name, mode, userblock_size, fapl, swmr=swmr) File "/net/gs/vol3/software/modules-sw-python/2.7.3/h5py/2.7.1/Linux/RHEL6/x86_64/lib/python2.7/site-packages/h5py-2.7.1-py2.7-linux-x86_64.egg/h5py/_hl/files.py", line 113, in make_fid fid = h5f.create(name, h5f.ACC_EXCL, fapl=fapl, fcpl=fcpl) File "h5py/_objects.pyx", line 54, in h5py._objects.with_phil.wrapper (/data/nobackup/h5py-2.7.1/h5py/_objects.c:2846) File "h5py/_objects.pyx", line 55, in h5py._objects.with_phil.wrapper (/data/nobackup/h5py-2.7.1/h5py/_objects.c:2804) File "h5py/h5f.pyx", line 98, in h5py.h5f.create (/data/nobackup/h5py-2.7.1/h5py/h5f.c:2287) IOError: Unable to create file (unable to open file: name = 'RootCombined.snap', errno = 17, error message = 'File exists', flags = 15, o_flags = c2)

On Aug 5, 2019, at 4:52 PM, Rongxin Fang <notifications@github.com mailto:notifications@github.com> wrote:

Genome name is wrong. You can set it as “hg19”. This tag is currently not used

Sent from my iPhone

On Aug 5, 2019, at 4:49 PM, cuperusj <notifications@github.com mailto:notifications@github.com> wrote:

I decided to just grab the bam from 10x instead, now getting this error from pre-snap:

snaptools snap-pre --input-file=sort.snap.bam --output-snap=RootCombined.snap --genome-name=../Arabidopsis_genome/fasta/genome --genome-size=chrom.length --min-mapq=30 --min-flen=0 --max-flen=1000 --keep-chrm=TRUE --keep-single=TRUE --keep-secondary=False --overwrite=True --max-num=1000000 --min-cov=100 --verbose=True error: --genome-name unrecoginized genome identifier ../Arabidopsis_genome/fasta/genome Do you mean: dasNov3,aptMan1,bosTau8,bosTau7,bosTau6,bosTau4,bosTau3,bosTau2,canFam3,canFam2,canFam1,triMan1,geoFor1,anoGam1,apiMel3,apiMel2,apiMel1,droSim1 ?

I dont see where/how/what this genome-name refers to exactly We are using Arabidopsis TAIR10

On Aug 4, 2019, at 10:11 PM, josh cuperus <cuperusj@gmail.com mailto:cuperusj@gmail.com> wrote:

Ok I'll try bowtie tomorrow, thanks for being so responsive!

On Sun, Aug 4, 2019, 8:39 PM Rongxin Fang <notifications@github.com mailto:notifications@github.com <mailto:notifications@github.com mailto:notifications@github.com>> wrote: This is wired, because I have ran BWA on demultiplexed fastq files all the time, never experienced this error. What’s the BWA version? Also, you can try bowtie as aligner.

Sent from my iPhone

On Aug 4, 2019, at 7:24 PM, cuperusj <notifications@github.com mailto:notifications@github.com <mailto:notifications@github.com mailto:notifications@github.com>> wrote:

I would have assumed bwa uses the @ns handle, which dex-fastq removes

— You are receiving this because you commented. Reply to this email directly, view it on GitHub, or mute the thread. — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <https://github.com/r3fang/SnapATAC/issues/64?email_source=notifications&email_token=ABR4NOXKY5MHGVAWNUJ6ECLQC6OIDA5CNFSM4IJF7OY2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD3QS72Y#issuecomment-518074347 https://github.com/r3fang/SnapATAC/issues/64?email_source=notifications&email_token=ABR4NOXKY5MHGVAWNUJ6ECLQC6OIDA5CNFSM4IJF7OY2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD3QS72Y#issuecomment-518074347>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ABR4NOUPVKFN6EP5XM6D2ALQC6OIDANCNFSM4IJF7OYQ https://github.com/notifications/unsubscribe-auth/ABR4NOUPVKFN6EP5XM6D2ALQC6OIDANCNFSM4IJF7OYQ>.

— You are receiving this because you commented. Reply to this email directly, view it on GitHub, or mute the thread. — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/r3fang/SnapATAC/issues/64?email_source=notifications&email_token=ABR4NOXGRZJ4BJYXCHD4IHLQDC4MZA5CNFSM4IJF7OY2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD3TM42A#issuecomment-518442600, or mute the thread https://github.com/notifications/unsubscribe-auth/ABR4NOV2FK3MLPTV5Z6D2KLQDC4MZANCNFSM4IJF7OYQ.

— You are receiving this because you commented. Reply to this email directly, view it on GitHub, or mute the thread. — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/r3fang/SnapATAC/issues/64?email_source=notifications&email_token=ABR4NOVJZVGC3JJ65YZCOYDQDDZJHA5CNFSM4IJF7OY2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD3TYAQQ#issuecomment-518488130, or mute the thread https://github.com/notifications/unsubscribe-auth/ABR4NOVLFLYQEVTVF7Z4JXLQDDZJHANCNFSM4IJF7OYQ.

r3fang commented 4 years ago

i will close this issue for now but feel free to let me know if you have more questions