Open terrycojones opened 10 years ago
OK, I've sorted a few of these out (mainly by installing things). My errors are down to
$ PYTHONPATH=. python -m discover
.....................EEE.............�BC����J�0�3�AE�|���Y�n��W��)�:���J��-Әd��|��[3��A
���w�_��ko5NO��4���]$i
6B7�q'�A7�i���~FIfѼnqB*�HB�CXi����>���|TR��kAh�O��C�g��x!g!���rVq;�lUd\Z�%(F��h��,�m��$��,����
��\�%*�8���Dd=�R�
�*pk$�5J��~�FT��
e��FL��}��34Lnyɠ(�T�P��15�,'-��}P�+�!�5�r!&\ҝ�:`�핀���B�t�˺y����fi��m;���f����y��8v��BCEAn unexpected error happened.
The seq_crumbs developers would appreciate your feedback.
Please send them the error log: filter_by_bowtie2.error
file header is empty (mode='r') - is it SAM/BAM format?Traceback (most recent call last):
File "/Users/terry/s/net/seq_crumbs/bin/filter_by_bowtie2", line 87, in <module>
sys.exit(main(filter_by_bowtie2))
File "/Users/terry/s/net/seq_crumbs/crumbs/utils/bin_utils.py", line 61, in main
return(funct())
File "/Users/terry/s/net/seq_crumbs/bin/filter_by_bowtie2", line 79, in filter_by_bowtie2
args['out_format'], workers=workers)
File "/Users/terry/s/net/seq_crumbs/crumbs/seqio.py", line 136, in write_filter_packets
seqs_diverted=SEQS_FILTERED_OUT)
File "/Users/terry/s/net/seq_crumbs/crumbs/seqio.py", line 112, in _write_filter_trim_packets
for packet in packets:
File "/Users/terry/s/net/seq_crumbs/crumbs/utils/seq_utils.py", line 94, in __call__
processed_packet = map_function(processed_packet)
File "/Users/terry/s/net/seq_crumbs/crumbs/filters.py", line 67, in __call__
self._setup_checks(filterpacket)
File "/Users/terry/s/net/seq_crumbs/crumbs/filters.py", line 310, in _setup_checks
self.mapped_reads = _get_mapped_reads(bam_fhand.name, self.min_mapq)
File "/Users/terry/s/net/seq_crumbs/crumbs/filters.py", line 184, in _get_mapped_reads
bam = pysam.Samfile(bam_fpath)
File "csamtools.pyx", line 736, in pysam.csamtools.Samfile.__cinit__ (pysam/csamtools.c:8722)
File "csamtools.pyx", line 781, in pysam.csamtools.Samfile._open (pysam/csamtools.c:9359)
File "csamtools.pyx", line 891, in pysam.csamtools.Samfile._open (pysam/csamtools.c:10460)
ValueError: file header is empty (mode='r') - is it SAM/BAM format?
�o�P����tk[]�#�{?�e���b��#;p`����/5�bm>�tω��BC..�BC�sr�e�������g�b�njjdf�g���gefi��1u7��03���q!i24�4�JY�˘Be,�M`2��)�9�ɉ%�)�E���y Ff���������Ԣ���DAjbI1�2s.6��y���
[�`bpg3�!�A��`ƌ"r)�2�P
�x��r�\ǐ4`vz_�BCE....................................................................................................
======================================================================
ERROR: test_blast_search (test.test_blast.BlastTest)
It does a blast search
----------------------------------------------------------------------
Traceback (most recent call last):
File "test/test_blast.py", line 67, in test_blast_search
dbtype='nucl')
File "crumbs/blast.py", line 109, in get_or_create_blastdb
_makeblastdb_plus(dbpath, dbtype)
File "crumbs/blast.py", line 58, in _makeblastdb_plus
check_process_finishes(process, binary=cmd[0])
File "crumbs/utils/bin_utils.py", line 162, in check_process_finishes
raise ExternalBinaryError(msg)
ExternalBinaryError: makeblastdb had a problem running
stdout:
Building a new DB, current time: 03/07/2014 12:20:09
New DB name: /var/folders/cv/v6h2zl0s56z5xnvnvw6lp1rw0000gn/T/blast_dbs_D1EkdN/arabidopsis_genes
New DB title: /var/folders/cv/v6h2zl0s56z5xnvnvw6lp1rw0000gn/T/blast_dbs_D1EkdN/arabidopsis_genes
Sequence type: Nucleotide
Keep Linkouts: T
Keep MBits: T
Maximum file size: 1000000000B
stderr:
BLAST options error: File /var/folders/cv/v6h2zl0s56z5xnvnvw6lp1rw0000gn/T/blast_dbs_D1EkdN/arabidopsis_genes does not exist
======================================================================
ERROR: test_blastdb (test.test_blast.BlastTest)
It creates a blast database.
----------------------------------------------------------------------
Traceback (most recent call last):
File "test/test_blast.py", line 50, in test_blastdb
dbtype='nucl')
File "crumbs/blast.py", line 109, in get_or_create_blastdb
_makeblastdb_plus(dbpath, dbtype)
File "crumbs/blast.py", line 58, in _makeblastdb_plus
check_process_finishes(process, binary=cmd[0])
File "crumbs/utils/bin_utils.py", line 162, in check_process_finishes
raise ExternalBinaryError(msg)
ExternalBinaryError: makeblastdb had a problem running
stdout:
Building a new DB, current time: 03/07/2014 12:20:09
New DB name: /var/folders/cv/v6h2zl0s56z5xnvnvw6lp1rw0000gn/T/blast_dbs_zfQ5Y_/arabidopsis_genes
New DB title: /var/folders/cv/v6h2zl0s56z5xnvnvw6lp1rw0000gn/T/blast_dbs_zfQ5Y_/arabidopsis_genes
Sequence type: Nucleotide
Keep Linkouts: T
Keep MBits: T
Maximum file size: 1000000000B
stderr:
BLAST options error: File /var/folders/cv/v6h2zl0s56z5xnvnvw6lp1rw0000gn/T/blast_dbs_zfQ5Y_/arabidopsis_genes does not exist
======================================================================
ERROR: test_get_or_create_blastdb (test.test_blast.BlastTest)
----------------------------------------------------------------------
Traceback (most recent call last):
File "test/test_blast.py", line 105, in test_get_or_create_blastdb
get_or_create_blastdb(blastdb, NUCL, directory.name)
File "crumbs/blast.py", line 109, in get_or_create_blastdb
_makeblastdb_plus(dbpath, dbtype)
File "crumbs/blast.py", line 58, in _makeblastdb_plus
check_process_finishes(process, binary=cmd[0])
File "crumbs/utils/bin_utils.py", line 162, in check_process_finishes
raise ExternalBinaryError(msg)
ExternalBinaryError: makeblastdb had a problem running
stdout:
Building a new DB, current time: 03/07/2014 12:20:09
New DB name: /var/folders/cv/v6h2zl0s56z5xnvnvw6lp1rw0000gn/T/QIo4FE/arabidopsis_genes
New DB title: /var/folders/cv/v6h2zl0s56z5xnvnvw6lp1rw0000gn/T/QIo4FE/arabidopsis_genes
Sequence type: Nucleotide
Keep Linkouts: T
Keep MBits: T
Maximum file size: 1000000000B
stderr:
BLAST options error: File /var/folders/cv/v6h2zl0s56z5xnvnvw6lp1rw0000gn/T/QIo4FE/arabidopsis_genes does not exist
======================================================================
ERROR: test_filter_by_bowtie2 (test.test_filters.FilterBowtie2Test)
----------------------------------------------------------------------
Traceback (most recent call last):
File "test/test_filters.py", line 522, in test_filter_by_bowtie2
filter_packets = filter_(filter_packet)
File "crumbs/filters.py", line 67, in __call__
self._setup_checks(filterpacket)
File "crumbs/filters.py", line 310, in _setup_checks
self.mapped_reads = _get_mapped_reads(bam_fhand.name, self.min_mapq)
File "crumbs/filters.py", line 184, in _get_mapped_reads
bam = pysam.Samfile(bam_fpath)
File "csamtools.pyx", line 736, in pysam.csamtools.Samfile.__cinit__ (pysam/csamtools.c:8722)
File "csamtools.pyx", line 781, in pysam.csamtools.Samfile._open (pysam/csamtools.c:9359)
File "csamtools.pyx", line 891, in pysam.csamtools.Samfile._open (pysam/csamtools.c:10460)
ValueError: file header is empty (mode='r') - is it SAM/BAM format?
======================================================================
ERROR: test_filter_by_bowtie2_bin (test.test_filters.FilterBowtie2Test)
----------------------------------------------------------------------
Traceback (most recent call last):
File "test/test_filters.py", line 544, in test_filter_by_bowtie2_bin
check_output(cmd)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/subprocess.py", line 544, in check_output
raise CalledProcessError(retcode, cmd, output=output)
CalledProcessError: Command '['/Users/terry/s/net/seq_crumbs/bin/filter_by_bowtie2', '-i', '/var/folders/cv/v6h2zl0s56z5xnvnvw6lp1rw0000gn/T/ieQCGK/arabidopsis_genes', '-o', '/var/folders/cv/v6h2zl0s56z5xnvnvw6lp1rw0000gn/T/tmpcD7Tth.seqs', '-e', '/var/folders/cv/v6h2zl0s56z5xnvnvw6lp1rw0000gn/T/tmpND27Ej.seqs', '/Users/terry/s/net/seq_crumbs/crumbs/test_data/arabidopsis_reads.fastq']' returned non-zero exit status 1
======================================================================
ERROR: test_map_with_bwa (test.test_mapping.Bwa2Test)
----------------------------------------------------------------------
Traceback (most recent call last):
File "test/test_mapping.py", line 92, in test_map_with_bwa
map_with_bwasw(index_fpath, bam_fhand.name, unpaired_fpath=reads_fpath)
File "crumbs/mapping.py", line 121, in map_with_bwasw
raise RuntimeError(open(stderr.name).read())
RuntimeError: [bsw2_aln] read 4 sequences/pairs (728 bp) ...
[main] Version: 0.7.7-r441
[main] CMD: bwa bwasw -t 1 /var/folders/cv/v6h2zl0s56z5xnvnvw6lp1rw0000gn/T/YPSJbQ/arabidopsis_genes /Users/terry/s/net/seq_crumbs/crumbs/test_data/arabidopsis_reads.fastq
[main] Real time: 0.003 sec; CPU: 0.005 sec
[samopen] SAM header is present: 6 sequences.
[main_samview] random alignment retrieval only works for indexed BAM files.
----------------------------------------------------------------------
Ran 171 tests in 30.582s
FAILED (errors=6)
Yes, the test suite is spewing binary output on my tty :-)
I'll try to fix more later. I'm doing my work in https://github.com/terrycojones/seq_crumbs/tree/fix-tests-9 (just edited the README and added a requirements.txt file).
Here are the contents of the filter_by_bowtie2.error file
<type 'exceptions.ValueError'>
Python 2.7.2: /Users/terry/.virtualenvs/seq_crumbs/bin/python
Fri Mar 7 12:20:16 2014
A problem occurred in a Python script. Here is the sequence of
function calls leading up to the error, in the order they occurred.
/Users/terry/s/net/seq_crumbs/crumbs/utils/bin_utils.py in main(funct=<function filter_by_bowtie2>)
59 if fail:
60 raise RuntimeError('Generating a test error')
61 return(funct())
62 except FileNotFoundError, error:
63 stderr.write(str(error) + '\n')
funct = <function filter_by_bowtie2>
/Users/terry/s/net/seq_crumbs/bin/filter_by_bowtie2 in filter_by_bowtie2()
77
78 write_filter_packets(passed_fhand, filtered_fhand, filter_packets,
79 args['out_format'], workers=workers)
80 flush_fhand(passed_fhand)
81 if filtered_fhand is not None:
args = {'fail_drags_pair': None, 'filtered_fhand': <open file '/var/folders/cv/v6h2zl0s56z5xnvnvw6lp1rw0000gn/T/tmpND27Ej.seqs', mode 'wt'>, 'in_fhands': [<open file '/Users/terry/s/net/seq_crumbs/crumbs/test_data/arabidopsis_reads.fastq', mode 'rt'>], 'index': '/var/folders/cv/v6h2zl0s56z5xnvnvw6lp1rw0000gn/T/ieQCGK/arabidopsis_genes', 'min_mapq': 0, 'original_in_fhands': [<open file '/Users/terry/s/net/seq_crumbs/crumbs/test_data/arabidopsis_reads.fastq', mode 'rt'>], 'out_fhand': <open file '/var/folders/cv/v6h2zl0s56z5xnvnvw6lp1rw0000gn/T/tmpcD7Tth.seqs', mode 'wt'>, 'out_format': 'fastq-illumina', 'paired_reads': False, 'processes': 1, ...}
workers = None
/Users/terry/s/net/seq_crumbs/crumbs/seqio.py in write_filter_packets(passed_fhand=<open file '/var/folders/cv/v6h2zl0s56z5xnvnvw6lp1rw0000gn/T/tmpcD7Tth.seqs', mode 'wt'>, filtered_fhand=<open file '/var/folders/cv/v6h2zl0s56z5xnvnvw6lp1rw0000gn/T/tmpND27Ej.seqs', mode 'wt'>, filter_packets=<itertools.imap object>, file_format='fastq-illumina', workers=None)
134 _write_filter_trim_packets(passed_fhand, filtered_fhand, filter_packets,
135 file_format=file_format, workers=workers,
136 seqs_diverted=SEQS_FILTERED_OUT)
137
138
seqs_diverted undefined
global SEQS_FILTERED_OUT = 'seqs_filtered_out'
/Users/terry/s/net/seq_crumbs/crumbs/seqio.py in _write_filter_trim_packets(passed_fhand=<open file '/var/folders/cv/v6h2zl0s56z5xnvnvw6lp1rw0000gn/T/tmpcD7Tth.seqs', mode 'wt'>, diverted_fhand=<open file '/var/folders/cv/v6h2zl0s56z5xnvnvw6lp1rw0000gn/T/tmpND27Ej.seqs', mode 'wt'>, packets=<itertools.imap object>, file_format='fastq-illumina', workers=None, seqs_diverted='seqs_filtered_out')
110
111 flatten_pairs = lambda pairs: (seq for pair in pairs for seq in pair)
112 for packet in packets:
113 try:
114 write_seqs(flatten_pairs(packet[SEQS_PASSED]), fhand=passed_fhand,
packet undefined
packets = <itertools.imap object>
/Users/terry/s/net/seq_crumbs/crumbs/utils/seq_utils.py in __call__(self=<crumbs.utils.seq_utils._FunctionRunner object>, seq_packet={'seqs_filtered_out': [], 'seqs_passed': [(SeqWrapper(kind='seqitem', object=SeqItem(name='...), annotations={}), file_format='fastq-illumina'),), (SeqWrapper(kind='seqitem', object=SeqItem(name='...), annotations={}), file_format='fastq-illumina'),), (SeqWrapper(kind='seqitem', object=SeqItem(name='...), annotations={}), file_format='fastq-illumina'),), (SeqWrapper(kind='seqitem', object=SeqItem(name='...), annotations={}), file_format='fastq-illumina'),)]})
92 processed_packet = seq_packet
93 for map_function in self.map_functions:
94 processed_packet = map_function(processed_packet)
95 return processed_packet
96
processed_packet = {'seqs_filtered_out': [], 'seqs_passed': [(SeqWrapper(kind='seqitem', object=SeqItem(name='...), annotations={}), file_format='fastq-illumina'),), (SeqWrapper(kind='seqitem', object=SeqItem(name='...), annotations={}), file_format='fastq-illumina'),), (SeqWrapper(kind='seqitem', object=SeqItem(name='...), annotations={}), file_format='fastq-illumina'),), (SeqWrapper(kind='seqitem', object=SeqItem(name='...), annotations={}), file_format='fastq-illumina'),)]}
map_function = <crumbs.filters.FilterBowtie2Match object>
/Users/terry/s/net/seq_crumbs/crumbs/filters.py in __call__(self=<crumbs.filters.FilterBowtie2Match object>, filterpacket={'seqs_filtered_out': [], 'seqs_passed': [(SeqWrapper(kind='seqitem', object=SeqItem(name='...), annotations={}), file_format='fastq-illumina'),), (SeqWrapper(kind='seqitem', object=SeqItem(name='...), annotations={}), file_format='fastq-illumina'),), (SeqWrapper(kind='seqitem', object=SeqItem(name='...), annotations={}), file_format='fastq-illumina'),), (SeqWrapper(kind='seqitem', object=SeqItem(name='...), annotations={}), file_format='fastq-illumina'),)]})
65
66 def __call__(self, filterpacket):
67 self._setup_checks(filterpacket)
68 reverse = self.reverse
69 failed_drags_pair = self.failed_drags_pair
self = <crumbs.filters.FilterBowtie2Match object>
self._setup_checks = <bound method FilterBowtie2Match._setup_checks of <crumbs.filters.FilterBowtie2Match object>>
filterpacket = {'seqs_filtered_out': [], 'seqs_passed': [(SeqWrapper(kind='seqitem', object=SeqItem(name='...), annotations={}), file_format='fastq-illumina'),), (SeqWrapper(kind='seqitem', object=SeqItem(name='...), annotations={}), file_format='fastq-illumina'),), (SeqWrapper(kind='seqitem', object=SeqItem(name='...), annotations={}), file_format='fastq-illumina'),), (SeqWrapper(kind='seqitem', object=SeqItem(name='...), annotations={}), file_format='fastq-illumina'),)]}
/Users/terry/s/net/seq_crumbs/crumbs/filters.py in _setup_checks(self=<crumbs.filters.FilterBowtie2Match object>, filterpacket={'seqs_filtered_out': [], 'seqs_passed': [(SeqWrapper(kind='seqitem', object=SeqItem(name='...), annotations={}), file_format='fastq-illumina'),), (SeqWrapper(kind='seqitem', object=SeqItem(name='...), annotations={}), file_format='fastq-illumina'),), (SeqWrapper(kind='seqitem', object=SeqItem(name='...), annotations={}), file_format='fastq-illumina'),), (SeqWrapper(kind='seqitem', object=SeqItem(name='...), annotations={}), file_format='fastq-illumina'),)]})
308 extra_params=extra_params)
309
310 self.mapped_reads = _get_mapped_reads(bam_fhand.name, self.min_mapq)
311
312 def _do_check(self, seq):
self = <crumbs.filters.FilterBowtie2Match object>
self.mapped_reads undefined
global _get_mapped_reads = <function _get_mapped_reads>
bam_fhand = <open file '<fdopen>', mode 'w+b'>
bam_fhand.name = '/var/folders/cv/v6h2zl0s56z5xnvnvw6lp1rw0000gn/T/tmpQp7Ybz.bam'
self.min_mapq = 0
/Users/terry/s/net/seq_crumbs/crumbs/filters.py in _get_mapped_reads(bam_fpath='/var/folders/cv/v6h2zl0s56z5xnvnvw6lp1rw0000gn/T/tmpQp7Ybz.bam', min_mapq=0)
182
183 def _get_mapped_reads(bam_fpath, min_mapq=0):
184 bam = pysam.Samfile(bam_fpath)
185 return [read.qname for read in bam if not read.is_unmapped and (not min_mapq or read.mapq > min_mapq)]
186
bam undefined
global pysam = <module 'pysam' from '/Users/terry/.virtualenvs/.../lib/python2.7/site-packages/pysam/__init__.pyc'>
pysam.Samfile = <type 'pysam.csamtools.Samfile'>
bam_fpath = '/var/folders/cv/v6h2zl0s56z5xnvnvw6lp1rw0000gn/T/tmpQp7Ybz.bam'
/Users/terry/s/net/seq_crumbs/csamtools.pyx in pysam.csamtools.Samfile.__cinit__ (pysam/csamtools.c:8722)()
/Users/terry/s/net/seq_crumbs/csamtools.pyx in pysam.csamtools.Samfile._open (pysam/csamtools.c:9359)()
/Users/terry/s/net/seq_crumbs/csamtools.pyx in pysam.csamtools.Samfile._open (pysam/csamtools.c:10460)()
<type 'exceptions.ValueError'>: file header is empty (mode='r') - is it SAM/BAM format?
__class__ = <type 'exceptions.ValueError'>
__delattr__ = <method-wrapper '__delattr__' of exceptions.ValueError object>
__dict__ = {}
__doc__ = 'Inappropriate argument value (of correct type).'
__format__ = <built-in method __format__ of exceptions.ValueError object>
__getattribute__ = <method-wrapper '__getattribute__' of exceptions.ValueError object>
__getitem__ = <method-wrapper '__getitem__' of exceptions.ValueError object>
__getslice__ = <method-wrapper '__getslice__' of exceptions.ValueError object>
__hash__ = <method-wrapper '__hash__' of exceptions.ValueError object>
__init__ = <method-wrapper '__init__' of exceptions.ValueError object>
__new__ = <built-in method __new__ of type object>
__reduce__ = <built-in method __reduce__ of exceptions.ValueError object>
__reduce_ex__ = <built-in method __reduce_ex__ of exceptions.ValueError object>
__repr__ = <method-wrapper '__repr__' of exceptions.ValueError object>
__setattr__ = <method-wrapper '__setattr__' of exceptions.ValueError object>
__setstate__ = <built-in method __setstate__ of exceptions.ValueError object>
__sizeof__ = <built-in method __sizeof__ of exceptions.ValueError object>
__str__ = <method-wrapper '__str__' of exceptions.ValueError object>
__subclasshook__ = <built-in method __subclasshook__ of type object>
__unicode__ = <built-in method __unicode__ of exceptions.ValueError object>
args = ("file header is empty (mode='r') - is it SAM/BAM format?",)
message = "file header is empty (mode='r') - is it SAM/BAM format?"
The above is a description of an error in a Python program. Here is
the original traceback:
Traceback (most recent call last):
File "/Users/terry/s/net/seq_crumbs/crumbs/utils/bin_utils.py", line 61, in main
return(funct())
File "/Users/terry/s/net/seq_crumbs/bin/filter_by_bowtie2", line 79, in filter_by_bowtie2
args['out_format'], workers=workers)
File "/Users/terry/s/net/seq_crumbs/crumbs/seqio.py", line 136, in write_filter_packets
seqs_diverted=SEQS_FILTERED_OUT)
File "/Users/terry/s/net/seq_crumbs/crumbs/seqio.py", line 112, in _write_filter_trim_packets
for packet in packets:
File "/Users/terry/s/net/seq_crumbs/crumbs/utils/seq_utils.py", line 94, in __call__
processed_packet = map_function(processed_packet)
File "/Users/terry/s/net/seq_crumbs/crumbs/filters.py", line 67, in __call__
self._setup_checks(filterpacket)
File "/Users/terry/s/net/seq_crumbs/crumbs/filters.py", line 310, in _setup_checks
self.mapped_reads = _get_mapped_reads(bam_fhand.name, self.min_mapq)
File "/Users/terry/s/net/seq_crumbs/crumbs/filters.py", line 184, in _get_mapped_reads
bam = pysam.Samfile(bam_fpath)
File "csamtools.pyx", line 736, in pysam.csamtools.Samfile.__cinit__ (pysam/csamtools.c:8722)
File "csamtools.pyx", line 781, in pysam.csamtools.Samfile._open (pysam/csamtools.c:9359)
File "csamtools.pyx", line 891, in pysam.csamtools.Samfile._open (pysam/csamtools.c:10460)
ValueError: file header is empty (mode='r') - is it SAM/BAM format?
The command was:
/Users/terry/s/net/seq_crumbs/bin/filter_by_bowtie2 -i /var/folders/cv/v6h2zl0s56z5xnvnvw6lp1rw0000gn/T/ieQCGK/arabidopsis_genes -o /var/folders/cv/v6h2zl0s56z5xnvnvw6lp1rw0000gn/T/tmpcD7Tth.seqs -e /var/folders/cv/v6h2zl0s56z5xnvnvw6lp1rw0000gn/T/tmpND27Ej.seqs /Users/terry/s/net/seq_crumbs/crumbs/test_data/arabidopsis_reads.fastq
There are quite a few errors hit when running the tests on OS X: