search

qiime2 / q2-deblur

BSD 3-Clause "New" or "Revised" License
2 stars 22 forks source link

numpy 1.14.3 and scipy 1.1.0 break denoise-* #63

Closed thermokarst closed 6 years ago

thermokarst commented 6 years ago

Not sure yet if this is an issue with the plugin, an issue with deblur, or an issue with both packages.

Failing tests on busywork.

cc @wasade

thermokarst commented 6 years ago

The integration tests as part of the doc building also are failing. Linking here for the record.

thermokarst commented 6 years ago

Just ran the deblur unit tests with numpy 1.14.3 and scipy 1.1.0 to see if these failures are limited to q2-deblur: the tests failed - 

nosetests --with-doctest                                                                                                                                                                                                                                                                                   
......./Users/matthew/.conda/envs/q2dev/lib/python3.5/site-packages/skbio/io/registry.py:548: FormatIdentificationWarning: <_io.BufferedReader name='/Users/matthew/src/biocore/deblur/deblur/test/data/seqs_empty.fasta'> does not look like a fasta file
  % (file, fmt), FormatIdentificationWarning)
......  E...............FF../Users/matthew/.conda/envs/q2dev/lib/python3.5/site-packages/skbio/io/registry.py:548: FormatIdentificationWarning: <_io.BufferedReader name='/Users/matthew/src/biocore/deblur/deblur/test/data/seqs_bad.fq'> does not look like a fastq file
  % (file, fmt), FormatIdentificationWarning)
....F...............
======================================================================
ERROR: Test parallel deblur using 3 simulated sequence files.
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Users/matthew/src/biocore/deblur/deblur/test/test_parallel_deblur.py", line 114, in test_parallel_deblur
    self.compare_result(deb1res, self.orig_s1_fp, trim_length=trim_length)
  File "/Users/matthew/src/biocore/deblur/deblur/test/test_parallel_deblur.py", line 77, in compare_result
    out_seqs = [item[1] for item in sequence_generator(simfilename)]
  File "/Users/matthew/src/biocore/deblur/deblur/test/test_parallel_deblur.py", line 77, in <listcomp>
    out_seqs = [item[1] for item in sequence_generator(simfilename)]
  File "/Users/matthew/src/biocore/deblur/deblur/workflow.py", line 81, in sequence_generator
    if sniff_fasta(input_fp)[0]:
  File "/Users/matthew/.conda/envs/q2dev/lib/python3.5/site-packages/skbio/io/registry.py", line 908, in wrapped_sniffer
    newline=newline, errors=errors, **kwargs) as fh:
  File "/Users/matthew/.conda/envs/q2dev/lib/python3.5/contextlib.py", line 59, in __enter__
    return next(self.gen)
  File "/Users/matthew/.conda/envs/q2dev/lib/python3.5/site-packages/skbio/io/util.py", line 246, in open_file
    with _resolve_file(file, **kwargs) as (file, source, is_binary_file):
  File "/Users/matthew/.conda/envs/q2dev/lib/python3.5/contextlib.py", line 59, in __enter__
    return next(self.gen)
  File "/Users/matthew/.conda/envs/q2dev/lib/python3.5/site-packages/skbio/io/util.py", line 203, in _resolve_file
    file, source, is_binary_file = _resolve(file, **kwargs)
  File "/Users/matthew/.conda/envs/q2dev/lib/python3.5/site-packages/skbio/io/util.py", line 57, in _resolve
    newfile = source.get_reader()
  File "/Users/matthew/.conda/envs/q2dev/lib/python3.5/site-packages/skbio/io/_iosources.py", line 94, in get_reader
    return io.open(self.file, mode='rb')
FileNotFoundError: [Errno 2] No such file or directory: '/var/folders/mc/hz3j4v4d6gg88w88llzpjxnc0000gn/T/tmpcrpiyhl2/deblur_working_dir/seqs_s1.fasta.trim.derep.no_artifacts.msa.deblur.no_chimeras'
-------------------- >> begin captured logging << --------------------
deblur.workflow: INFO: *************************
deblur.workflow: INFO: deblurring started
deblur.workflow: INFO: build_index_sortmerna files ('/Users/matthew/src/biocore/deblur/deblur/test/data/70_otus.fasta',) to dir /var/folders/mc/hz3j4v4d6gg88w88llzpjxnc0000gn/T/tmpcrpiyhl2
deblur.workflow: DEBUG: processing file /Users/matthew/src/biocore/deblur/deblur/test/data/70_otus.fasta into location /var/folders/mc/hz3j4v4d6gg88w88llzpjxnc0000gn/T/tmpcrpiyhl2/70_otus
deblur.workflow: DEBUG: system call: ['indexdb_rna', '--ref', '/Users/matthew/src/biocore/deblur/deblur/test/data/70_otus.fasta,/var/folders/mc/hz3j4v4d6gg88w88llzpjxnc0000gn/T/tmpcrpiyhl2/70_otus', '--tmpdir', '/var/folders/mc/hz3j4v4d6gg88w88llzpjxnc0000gn/T/tmpcrpiyhl2']
deblur.workflow: DEBUG: file /Users/matthew/src/biocore/deblur/deblur/test/data/70_otus.fasta indexed
deblur.parallel_deblur: INFO: parallel deblur started for 3 inputs
deblur.parallel_deblur: DEBUG: ready for functor ['--output-dir', '/var/folders/mc/hz3j4v4d6gg88w88llzpjxnc0000gn/T/tmpcrpiyhl2', '--pos-ref-fp', '/Users/matthew/src/biocore/deblur/deblur/test/data/70_otus.fasta', '-d', '1,0.06,0.02,0.02,0.01,0.005,0.005,0.005,0.001,0.001,0.001,0.0005', '-t', '100', '--pos-ref-db-fp', '/var/folders/mc/hz3j4v4d6gg88w88llzpjxnc0000gn/T/tmpcrpiyhl2/70_otus']
deblur.parallel_deblur: DEBUG: ready for pool 2 jobs
deblur.parallel_deblur: DEBUG: almost running...
--------------------- >> end captured logging << ---------------------

======================================================================
FAIL: Test dereplicate_seqs() method functionality,
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Users/matthew/src/biocore/deblur/deblur/test/test_workflow.py", line 250, in test_dereplicate_seqs
    self.assertEqual(act, exp)
AssertionError: Lists differ: [('seq1;size=3', 'TACCGGCAGCTCAAGTGATGACCGCTATTATTGGGCC[216 chars]CG')] != [('seq1;size=3;', 'TACCGGCAGCTCAAGTGATGACCGCTATTATTGGGC[220 chars]CG')]

First differing element 0:
('seq1;size=3', 'TACCGGCAGCTCAAGTGATGACCGCTATTATTGGGCCTAAAGCGTCCG')
('seq1;size=3;', 'TACCGGCAGCTCAAGTGATGACCGCTATTATTGGGCCTAAAGCGTCCG')

Diff is 656 characters long. Set self.maxDiff to None to see it.
-------------------- >> begin captured logging << --------------------
deblur.workflow: INFO: *************************
deblur.workflow: INFO: deblurring started
deblur.workflow: INFO: dereplicate seqs file /var/folders/mc/hz3j4v4d6gg88w88llzpjxnc0000gn/T/tmpn_jwqhuy/seqs.fasta
deblur.workflow: DEBUG: system call: ['vsearch', '--derep_fulllength', '/var/folders/mc/hz3j4v4d6gg88w88llzpjxnc0000gn/T/tmpn_jwqhuy/seqs.fasta', '--output', '/var/folders/mc/hz3j4v4d6gg88w88llzpjxnc0000gn/T/tmpn_jwqhuy/seqs_derep.fasta', '--sizeout', '--fasta_width', '0', '--minuniquesize', '1', '--quiet', '--threads', '1']
--------------------- >> end captured logging << ---------------------

======================================================================
FAIL: Test dereplicate_seqs() method functionality with
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Users/matthew/src/biocore/deblur/deblur/test/test_workflow.py", line 214, in test_dereplicate_seqs_remove_singletons
    self.assertEqual(act, exp)
AssertionError: Lists differ: [('seq1;size=3', 'TACCGGCAGCTCAAGTGATGACCGCTATTATTGGGCC[78 chars]AT')] != [('seq1;size=3;', 'TACCGGCAGCTCAAGTGATGACCGCTATTATTGGGC[80 chars]AT')]

First differing element 0:
('seq1;size=3', 'TACCGGCAGCTCAAGTGATGACCGCTATTATTGGGCCTAAAGCGTCCG')
('seq1;size=3;', 'TACCGGCAGCTCAAGTGATGACCGCTATTATTGGGCCTAAAGCGTCCG')

- [('seq1;size=3', 'TACCGGCAGCTCAAGTGATGACCGCTATTATTGGGCCTAAAGCGTCCG'),
+ [('seq1;size=3;', 'TACCGGCAGCTCAAGTGATGACCGCTATTATTGGGCCTAAAGCGTCCG'),
?               +

-  ('seq6;size=2', 'CTGCAAGGCTAGGGGGCGGGAGAGGCGGGTGGTACTTGAGGGGAGAAT')]
+  ('seq6;size=2;', 'CTGCAAGGCTAGGGGGCGGGAGAGGCGGGTGGTACTTGAGGGGAGAAT')]
?               +

-------------------- >> begin captured logging << --------------------
deblur.workflow: INFO: *************************
deblur.workflow: INFO: deblurring started
deblur.workflow: INFO: dereplicate seqs file /var/folders/mc/hz3j4v4d6gg88w88llzpjxnc0000gn/T/tmpq9lf54o3/seqs.fasta
deblur.workflow: DEBUG: system call: ['vsearch', '--derep_fulllength', '/var/folders/mc/hz3j4v4d6gg88w88llzpjxnc0000gn/T/tmpq9lf54o3/seqs.fasta', '--output', '/var/folders/mc/hz3j4v4d6gg88w88llzpjxnc0000gn/T/tmpq9lf54o3/seqs_derep.fasta', '--sizeout', '--fasta_width', '0', '--minuniquesize', '2', '--quiet', '--threads', '1']
--------------------- >> end captured logging << ---------------------

======================================================================
FAIL: test_launch_workflow_skip_trim (test_workflow.workflowTests)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Users/matthew/src/biocore/deblur/deblur/test/test_workflow.py", line 831, in test_launch_workflow_skip_trim
    self.assertEqual(exp, res)
AssertionError: Seque[133 chars]ze=10;'
Stats:
    length: 150
---------------[226 chars]CCACG != Seque[133 chars]ze=10'
Stats:
    length: 150
----------------[225 chars]CCACG
-------------------- >> begin captured logging << --------------------
deblur.workflow: INFO: *************************
deblur.workflow: INFO: deblurring started
deblur.workflow: INFO: build_index_sortmerna files ('/Users/matthew/src/biocore/deblur/deblur/test/data/70_otus.fasta',) to dir /var/folders/mc/hz3j4v4d6gg88w88llzpjxnc0000gn/T/tmp6_airx7k
deblur.workflow: DEBUG: processing file /Users/matthew/src/biocore/deblur/deblur/test/data/70_otus.fasta into location /var/folders/mc/hz3j4v4d6gg88w88llzpjxnc0000gn/T/tmp6_airx7k/70_otus
deblur.workflow: DEBUG: system call: ['indexdb_rna', '--ref', '/Users/matthew/src/biocore/deblur/deblur/test/data/70_otus.fasta,/var/folders/mc/hz3j4v4d6gg88w88llzpjxnc0000gn/T/tmp6_airx7k/70_otus', '--tmpdir', '/var/folders/mc/hz3j4v4d6gg88w88llzpjxnc0000gn/T/tmp6_airx7k']
deblur.workflow: DEBUG: file /Users/matthew/src/biocore/deblur/deblur/test/data/70_otus.fasta indexed
deblur.workflow: INFO: --------------------------------------------------------
deblur.workflow: INFO: launch_workflow for file /Users/matthew/src/biocore/deblur/deblur/test/data/seqs_s1.fasta
deblur.workflow: DEBUG: trimmed to length -1 (135 / 135 remaining)
deblur.workflow: INFO: dereplicate seqs file /var/folders/mc/hz3j4v4d6gg88w88llzpjxnc0000gn/T/tmp6_airx7k/seqs_s1.fasta.trim
deblur.workflow: DEBUG: system call: ['vsearch', '--derep_fulllength', '/var/folders/mc/hz3j4v4d6gg88w88llzpjxnc0000gn/T/tmp6_airx7k/seqs_s1.fasta.trim', '--output', '/var/folders/mc/hz3j4v4d6gg88w88llzpjxnc0000gn/T/tmp6_airx7k/seqs_s1.fasta.trim.derep', '--sizeout', '--fasta_width', '0', '--minuniquesize', '2', '--quiet', '--threads', '1']
deblur.workflow: INFO: remove_artifacts_seqs file /var/folders/mc/hz3j4v4d6gg88w88llzpjxnc0000gn/T/tmp6_airx7k/seqs_s1.fasta.trim.derep
deblur.workflow: DEBUG: running on ref_fp /Users/matthew/src/biocore/deblur/deblur/test/data/70_otus.fasta working dir /var/folders/mc/hz3j4v4d6gg88w88llzpjxnc0000gn/T/tmp6_airx7k refdb_fp /var/folders/mc/hz3j4v4d6gg88w88llzpjxnc0000gn/T/tmp6_airx7k/70_otus seqs /var/folders/mc/hz3j4v4d6gg88w88llzpjxnc0000gn/T/tmp6_airx7k/seqs_s1.fasta.trim.derep
deblur.workflow: DEBUG: system call: ['sortmerna', '--reads', '/var/folders/mc/hz3j4v4d6gg88w88llzpjxnc0000gn/T/tmp6_airx7k/seqs_s1.fasta.trim.derep', '--ref', '/Users/matthew/src/biocore/deblur/deblur/test/data/70_otus.fasta,/var/folders/mc/hz3j4v4d6gg88w88llzpjxnc0000gn/T/tmp6_airx7k/70_otus', '--aligned', '/var/folders/mc/hz3j4v4d6gg88w88llzpjxnc0000gn/T/tmp6_airx7k/seqs_s1.fasta.trim.derep.sortmerna', '--blast', '3', '--best', '1', '--print_all_reads', '-v', '-e', '100']
deblur.workflow: INFO: total sequences 10, passing sequences 10, failing sequences 0
deblur.workflow: INFO: multiple_sequence_alignment seqs file /var/folders/mc/hz3j4v4d6gg88w88llzpjxnc0000gn/T/tmp6_airx7k/seqs_s1.fasta.trim.derep.no_artifacts
deblur.workflow: DEBUG: system call: ['mafft', '--quiet', '--preservecase', '--parttree', '--auto', '--thread', '1', '/var/folders/mc/hz3j4v4d6gg88w88llzpjxnc0000gn/T/tmp6_airx7k/seqs_s1.fasta.trim.derep.no_artifacts']
deblur.deblurring: DEBUG: Using error profile [1, 0.06, 0.02, 0.02, 0.01, 0.005, 0.005, 0.005, 0.001, 0.001, 0.001, 0.0005]
deblur.deblurring: INFO: deblurring 10 sequences
deblur.deblurring: INFO: 10 unique sequences left following deblurring
deblur.workflow: INFO: remove_chimeras_denovo_from_seqs seqs file /var/folders/mc/hz3j4v4d6gg88w88llzpjxnc0000gn/T/tmp6_airx7k/seqs_s1.fasta.trim.derep.no_artifacts.msa.deblurto working dir /var/folders/mc/hz3j4v4d6gg88w88llzpjxnc0000gn/T/tmp6_airx7k
deblur.workflow: DEBUG: system call: ['vsearch', '--uchime_denovo', '/var/folders/mc/hz3j4v4d6gg88w88llzpjxnc0000gn/T/tmp6_airx7k/seqs_s1.fasta.trim.derep.no_artifacts.msa.deblur', '--nonchimeras', '/var/folders/mc/hz3j4v4d6gg88w88llzpjxnc0000gn/T/tmp6_airx7k/seqs_s1.fasta.trim.derep.no_artifacts.msa.deblur.no_chimeras', '-dn', '0.000001', '-xn', '1000', '-minh', '10000000', '--mindiffs', '5', '--fasta_width', '0', '--threads', '1']
deblur.workflow: INFO: finished processing file
--------------------- >> end captured logging << ---------------------

----------------------------------------------------------------------
Ran 53 tests in 574.656s

FAILED (errors=1, failures=3)
thermokarst commented 6 years ago

Fixed by pinning versions of numpy and scipy.