search

tanghaibao / jcvi

Python library to facilitate genome assembly, annotation, and comparative genomics
BSD 2-Clause "Simplified" License
741 stars 187 forks source link

Test sample data fail: RuntimeError: unexpected format. [Solved and Docker file attached] #101

Closed xinshuaiqi closed 6 years ago

xinshuaiqi commented 6 years ago

I got an error when running the test sample data. Any idea how to fix this?

python -m jcvi.assembly.allmaps path JM-2.bed scaffolds.fasta -w weights.txt

16:27:05 [base] Load file JM-2.bed 16:27:05 [allmaps] Map contains 120 markers in 2 linkage groups. 16:27:05 [allmaps] Retained 120 of 120 (100.0%) clean markers. 16:27:05 [base] Load file weights.txt 16:27:05 [base] Imported 2 records from weights.txt. 16:27:05 [allmaps] Map weights: [('JMMale', 1), ('JMFemale', 1)] 16:27:05 [allmaps] Linkage function: double-linkage 16:27:05 [allmaps] Partition LGs based on JMFemale 16:27:05 [base] faSize -detailed /home/biodocker/ALLMAPS-testdata/scaffolds.fasta >scaffolds.fasta.sizes 16:27:05 [base] Load file scaffolds.fasta.sizes 16:27:05 [allmaps] Working on JMMale-23|JMFemale-23 ... Traceback (most recent call last): File "/usr/lib/python2.7/runpy.py", line 174, in _run_module_as_main "main", fname, loader, pkg_name) File "/usr/lib/python2.7/runpy.py", line 72, in _run_code exec code in run_globals File "/home/biodocker/code/jcvi/assembly/allmaps.py", line 1871, in main() File "/home/biodocker/code/jcvi/assembly/allmaps.py", line 827, in main p.dispatch(globals()) File "/home/biodocker/code/jcvi/apps/base.py", line 96, in dispatch globalsaction File "/home/biodocker/code/jcvi/assembly/allmaps.py", line 1484, in path ngen=ngen, npop=npop, cpus=cpus, seed=seed) File "/home/biodocker/code/jcvi/assembly/allmaps.py", line 142, in init self.prepare_linkage_groups() # populate all data File "/home/biodocker/code/jcvi/assembly/allmaps.py", line 249, in prepare_linkage_groups function=self.function, linkage=self.linkage) File "/home/biodocker/code/jcvi/assembly/allmaps.py", line 92, in init self.oo[k] = get_rho(physical_to_cm) File "/home/biodocker/code/jcvi/assembly/allmaps.py", line 791, in get_rho rho = spearmanr(x, y) File "/home/biodocker/code/jcvi/algorithms/formula.py", line 112, in spearmanr return 1 - distancematrix((x, y), dist="s")[1][0] File "/usr/local/lib/python2.7/dist-packages/Bio/Cluster/init.py", line 500, in distancematrix _cluster.distancematrix(data, mask, weight, transpose, dist, matrix) RuntimeError: unexpected format.

ls

drwxr-xr-x 2 root root 4096 Jul 9 16:27 ./ drwxr-xr-x 12 biodocker biodocker 4096 Jul 9 16:26 ../ -rw-r--r-- 1 root root 8086 Jul 9 16:26 JM-2.bed -rw-r--r-- 1 root root 0 Jul 9 16:27 JM-2.chr.agp -rw-r--r-- 1 root root 0 Jul 9 16:27 JM-2.tour -rw-r--r-- 1 root root 1911 May 7 2014 JMFemale.csv -rw-r--r-- 1 root root 1847 May 7 2014 JMMale.csv -rw-r--r-- 1 root root 218 Sep 19 2014 run.sh -rw-r--r-- 1 root root 15525370 May 7 2014 scaffolds.fasta -rw-r--r-- 1 root root 444 Jul 9 16:27 scaffolds.fasta.sizes -rw-r--r-- 1 root root 20 May 22 2014 weights.txt

tanghaibao commented 6 years ago

@xinshuaiqi This seems to be a bug inside biopython version 1.72. A temporary solution may be to downgrade biopython to 1.70. Something like:

$ pip install --user biopython==1.70
xinshuaiqi commented 6 years ago

Thanks for your fast reply, Haibao. After installing biopython1.70, the run.sh made some progress, now I get a new error message, see below. Is LaTeX required for Allmaps?

...
Reading liftover chains
Mapping coordinates
18:42:54 [base] sort -k1,1 -k2,2n -k3,3n -k4,4 JM-2.lifted.bed -o JM-2.lifted.bed
18:42:54 [base] Load file `JM-2.bed`
18:42:54 [allmaps] Map contains 120 markers in 2 linkage groups.
18:42:54 [base] Load file `scaffolds.fasta.sizes`
18:42:54 [base] Load file `JM-2.chr.agp`
18:42:54 [base] Load file `JM-2.chr.agp`
18:42:54 [base] Load file `JM-2.lifted.bed`
18:42:54 [allmaps] Map contains 120 markers in 2 linkage groups.
18:42:54 [base] Load file `weights.txt`
18:42:54 [base] Imported 2 records from `weights.txt`.
18:42:54 [allmaps] Map weights: [('JMMale', 1), ('JMFemale', 1)]
18:42:54 [base] Load file `JM-2.agp`
18:42:55 [texmanager] serif font is not compatible with usetex.
18:42:55 [texmanager] serif font is not compatible with usetex.
18:42:55 [texmanager] family: serif, font: Computer Modern Roman, info: ('cmr', '')
18:42:55 [texmanager] family: sans-serif, font: Helvetica, info: ('phv', '\\usepackage{helvet}')
18:42:55 [texmanager] cursive font is not compatible with usetex.
18:42:55 [texmanager] cursive font is not compatible with usetex.
18:42:55 [texmanager] family: cursive, font: Zapf Chancery, info: ('pzc', '\\usepackage{chancery}')
18:42:55 [texmanager] monospace font is not compatible with usetex.
18:42:55 [texmanager] monospace font is not compatible with usetex.
18:42:55 [texmanager] family: monospace, font: Computer Modern Typewriter, info: ('cmtt', '')
18:42:55 [texmanager] following keys changed: ['text.latex.preamble', 'font.family', 'font.serif', 'font.sans-serif', 'font.c                                 ursive', 'font.monospace']
18:42:55 [texmanager] text.latex.preamble : None       -> []
18:42:55 [texmanager] font.family         : None       -> ['sans-serif']
18:42:55 [texmanager] font.serif          : None       -> ['DejaVu Serif', 'Bitstream Vera Serif', 'Computer Modern Roman', '                                 New Century Schoolbook', 'Century Schoolbook L', 'Utopia', 'ITC Bookman', 'Bookman', 'Nimbus Roman No9 L', 'Times New Roman',                                  'Times', 'Palatino', 'Charter', 'serif']
18:42:55 [texmanager] font.sans-serif     : None       -> ['Helvetica']
18:42:55 [texmanager] font.cursive        : None       -> ['Apple Chancery', 'Textile', 'Zapf Chancery', 'Sand', 'Script MT',                                  'Felipa', 'cursive']
18:42:55 [texmanager] font.monospace      : None       -> ['DejaVu Sans Mono', 'Bitstream Vera Sans Mono', 'Computer Modern T                                 ypewriter', 'Andale Mono', 'Nimbus Mono L', 'Courier New', 'Courier', 'Fixed', 'Terminal', 'monospace']
18:42:55 [texmanager] RE-INIT
old fontconfig: sans-serifcmrphvpzccmttd41d8cd98f00b204e9800998ecf8427e
18:42:55 [texmanager] serif font is not compatible with usetex.
18:42:55 [texmanager] serif font is not compatible with usetex.
18:42:55 [texmanager] family: serif, font: Computer Modern Roman, info: ('cmr', '')
18:42:55 [texmanager] family: sans-serif, font: Helvetica, info: ('phv', '\\usepackage{helvet}')
18:42:55 [texmanager] cursive font is not compatible with usetex.
18:42:55 [texmanager] cursive font is not compatible with usetex.
18:42:55 [texmanager] family: cursive, font: Zapf Chancery, info: ('pzc', '\\usepackage{chancery}')
18:42:55 [texmanager] monospace font is not compatible with usetex.
18:42:55 [texmanager] monospace font is not compatible with usetex.
18:42:55 [texmanager] family: monospace, font: Computer Modern Typewriter, info: ('cmtt', '')
18:42:55 [texmanager] fontconfig: sans-serifcmrphvpzccmttd41d8cd98f00b204e9800998ecf8427e
18:42:55 [texmanager] fontconfig: sans-serifcmrphvpzccmttd41d8cd98f00b204e9800998ecf8427e
18:42:55 [texmanager] ['latex', '-interaction=nonstopmode', '--halt-on-error', '/root/.cache/matplotlib/tex.cache/d2acc2706db                                 635f7f7afe7dc80ac47f1.tex']
18:42:55 [base] savefig failed. Reset usetex to False.
[Errno 2] No such file or directory: 'latex'
/usr/local/lib/python2.7/dist-packages/matplotlib/font_manager.py:1328: UserWarning: findfont: Font family [u'sans-serif'] no                                 t found. Falling back to DejaVu Sans
  (prop.get_family(), self.defaultFamily[fontext]))
18:42:55 [font_manager] findfont: Matching :family=DejaVu Sans:style=normal:variant=normal:weight=normal:stretch=normal:size=                                 10.0 to DejaVu Sans (u'/usr/local/lib/python2.7/dist-packages/matplotlib/mpl-data/fonts/ttf/DejaVuSans.ttf') with score of 0.                                 050000
18:42:55 [font_manager] findfont: Matching :family=STIXGeneral:style=normal:variant=normal:weight=normal:stretch=normal:size=                                 10.0 to STIXGeneral (u'/usr/local/lib/python2.7/dist-packages/matplotlib/mpl-data/fonts/ttf/STIXGeneral.ttf') with score of 0                                 .050000
18:42:55 [font_manager] findfont: Matching :family=STIXSizeOneSym:style=normal:variant=normal:weight=normal:stretch=normal:si                                 ze=10.0 to STIXSizeOneSym (u'/usr/local/lib/python2.7/dist-packages/matplotlib/mpl-data/fonts/ttf/STIXSizOneSymReg.ttf') with                                  score of 0.050000
18:42:55 [font_manager] findfont: Matching :family=STIXGeneral:style=normal:variant=normal:weight=bold:stretch=normal:size=10                                 .0 to STIXGeneral (u'/usr/local/lib/python2.7/dist-packages/matplotlib/mpl-data/fonts/ttf/STIXGeneralBol.ttf') with score of                                  0.000000
18:42:55 [font_manager] findfont: Matching :family=STIXSizeThreeSym:style=normal:variant=normal:weight=normal:stretch=normal:                                 size=10.0 to STIXSizeThreeSym (u'/usr/local/lib/python2.7/dist-packages/matplotlib/mpl-data/fonts/ttf/STIXSizThreeSymReg.ttf'                                 ) with score of 0.050000
18:42:55 [font_manager] findfont: Matching :family=STIXSizeFourSym:style=normal:variant=normal:weight=normal:stretch=normal:s                                 ize=10.0 to STIXSizeFourSym (u'/usr/local/lib/python2.7/dist-packages/matplotlib/mpl-data/fonts/ttf/STIXSizFourSymReg.ttf') w                                 ith score of 0.050000
18:42:55 [font_manager] findfont: Matching :family=STIXSizeFiveSym:style=normal:variant=normal:weight=normal:stretch=normal:s                                 ize=10.0 to STIXSizeFiveSym (u'/usr/local/lib/python2.7/dist-packages/matplotlib/mpl-data/fonts/ttf/STIXSizFiveSymReg.ttf') w                                 ith score of 0.050000
18:42:55 [font_manager] findfont: Matching :family=STIXSizeTwoSym:style=normal:variant=normal:weight=normal:stretch=normal:si                                 ze=10.0 to STIXSizeTwoSym (u'/usr/local/lib/python2.7/dist-packages/matplotlib/mpl-data/fonts/ttf/STIXSizTwoSymReg.ttf') with                                  score of 0.050000
18:42:55 [font_manager] findfont: Matching :family=STIXGeneral:style=italic:variant=normal:weight=normal:stretch=normal:size=                                 10.0 to STIXGeneral (u'/usr/local/lib/python2.7/dist-packages/matplotlib/mpl-data/fonts/ttf/STIXGeneralItalic.ttf') with scor                                 e of 0.050000
18:42:55 [font_manager] findfont: Matching :family=STIXNonUnicode:style=italic:variant=normal:weight=normal:stretch=normal:si                                 ze=10.0 to STIXNonUnicode (u'/usr/local/lib/python2.7/dist-packages/matplotlib/mpl-data/fonts/ttf/STIXNonUniIta.ttf') with sc                                 ore of 0.050000
18:42:55 [font_manager] findfont: Matching :family=STIXNonUnicode:style=normal:variant=normal:weight=normal:stretch=normal:si                                 ze=10.0 to STIXNonUnicode (u'/usr/local/lib/python2.7/dist-packages/matplotlib/mpl-data/fonts/ttf/STIXNonUni.ttf') with score                                  of 0.050000
18:42:55 [font_manager] findfont: Matching :family=STIXNonUnicode:style=normal:variant=normal:weight=bold:stretch=normal:size                                 =10.0 to STIXNonUnicode (u'/usr/local/lib/python2.7/dist-packages/matplotlib/mpl-data/fonts/ttf/STIXNonUniBol.ttf') with scor                                 e of 0.000000
18:42:55 [font_manager] findfont: Matching :family=cmb10:style=normal:variant=normal:weight=normal:stretch=normal:size=10.0 t                                 o cmb10 (u'/usr/local/lib/python2.7/dist-packages/matplotlib/mpl-data/fonts/ttf/cmb10.ttf') with score of 0.050000
18:42:55 [font_manager] findfont: Matching :family=cmtt10:style=normal:variant=normal:weight=normal:stretch=normal:size=10.0                                  to cmtt10 (u'/usr/local/lib/python2.7/dist-packages/matplotlib/mpl-data/fonts/ttf/cmtt10.ttf') with score of 0.050000
18:42:55 [font_manager] findfont: Matching :family=cmmi10:style=normal:variant=normal:weight=normal:stretch=normal:size=10.0                                  to cmmi10 (u'/usr/local/lib/python2.7/dist-packages/matplotlib/mpl-data/fonts/ttf/cmmi10.ttf') with score of 0.050000
18:42:55 [font_manager] findfont: Matching :family=cmex10:style=normal:variant=normal:weight=normal:stretch=normal:size=10.0                                  to cmex10 (u'/usr/local/lib/python2.7/dist-packages/matplotlib/mpl-data/fonts/ttf/cmex10.ttf') with score of 0.050000
18:42:55 [font_manager] findfont: Matching :family=cmsy10:style=normal:variant=normal:weight=normal:stretch=normal:size=10.0                                  to cmsy10 (u'/usr/local/lib/python2.7/dist-packages/matplotlib/mpl-data/fonts/ttf/cmsy10.ttf') with score of 0.050000
18:42:55 [font_manager] findfont: Matching :family=cmr10:style=normal:variant=normal:weight=normal:stretch=normal:size=10.0 t                                 o cmr10 (u'/usr/local/lib/python2.7/dist-packages/matplotlib/mpl-data/fonts/ttf/cmr10.ttf') with score of 0.050000
18:42:55 [font_manager] findfont: Matching :family=cmss10:style=normal:variant=normal:weight=normal:stretch=normal:size=10.0                                  to cmss10 (u'/usr/local/lib/python2.7/dist-packages/matplotlib/mpl-data/fonts/ttf/cmss10.ttf') with score of 0.050000
18:42:55 [font_manager] findfont: Matching :family=DejaVu Sans:style=normal:variant=normal:weight=bold:stretch=normal:size=10                                 .0 to DejaVu Sans (u'/usr/local/lib/python2.7/dist-packages/matplotlib/mpl-data/fonts/ttf/DejaVuSans-Bold.ttf') with score of                                  0.000000
18:42:55 [font_manager] findfont: Matching :family=DejaVu Sans Mono:style=normal:variant=normal:weight=normal:stretch=normal:                                 size=10.0 to DejaVu Sans Mono (u'/usr/local/lib/python2.7/dist-packages/matplotlib/mpl-data/fonts/ttf/DejaVuSansMono.ttf') wi                                 th score of 0.050000
18:42:55 [font_manager] findfont: Matching :family=DejaVu Sans:style=italic:variant=normal:weight=normal:stretch=normal:size=                                 10.0 to DejaVu Sans (u'/usr/local/lib/python2.7/dist-packages/matplotlib/mpl-data/fonts/ttf/DejaVuSans-Oblique.ttf') with sco                                 re of 0.150000
18:42:55 [font_manager] findfont: Matching :family=DejaVu Sans Display:style=normal:variant=normal:weight=normal:stretch=norm                                 al:size=10.0 to DejaVu Sans Display (u'/usr/local/lib/python2.7/dist-packages/matplotlib/mpl-data/fonts/ttf/DejaVuSansDisplay                                 .ttf') with score of 0.050000
18:42:55 [texmanager] serif font is not compatible with usetex.
18:42:55 [texmanager] serif font is not compatible with usetex.
18:42:55 [texmanager] family: serif, font: Computer Modern Roman, info: ('cmr', '')
18:42:55 [texmanager] family: sans-serif, font: Helvetica, info: ('phv', '\\usepackage{helvet}')
18:42:55 [texmanager] cursive font is not compatible with usetex.
18:42:55 [texmanager] cursive font is not compatible with usetex.
18:42:55 [texmanager] family: cursive, font: Zapf Chancery, info: ('pzc', '\\usepackage{chancery}')
18:42:55 [texmanager] monospace font is not compatible with usetex.
18:42:55 [texmanager] monospace font is not compatible with usetex.
18:42:55 [texmanager] family: monospace, font: Computer Modern Typewriter, info: ('cmtt', '')
18:42:55 [texmanager] following keys changed: ['text.latex.preamble', 'font.family', 'font.serif', 'font.sans-serif', 'font.c                                 ursive', 'font.monospace']
18:42:55 [texmanager] text.latex.preamble : None       -> []
18:42:55 [texmanager] font.family         : None       -> ['sans-serif']
18:42:55 [texmanager] font.serif          : None       -> ['DejaVu Serif', 'Bitstream Vera Serif', 'Computer Modern Roman', '                                 New Century Schoolbook', 'Century Schoolbook L', 'Utopia', 'ITC Bookman', 'Bookman', 'Nimbus Roman No9 L', 'Times New Roman',                                  'Times', 'Palatino', 'Charter', 'serif']
18:42:55 [texmanager] font.sans-serif     : None       -> ['Helvetica']
18:42:55 [texmanager] font.cursive        : None       -> ['Apple Chancery', 'Textile', 'Zapf Chancery', 'Sand', 'Script MT',                                  'Felipa', 'cursive']
  File "/home/biodocker/code/jcvi/assembly/allmaps.py", line 1867, in plotall
    plot(xargs + [seqid])
  File "/home/biodocker/code/jcvi/assembly/allmaps.py", line 1842, in plot
    savefig(image_name, dpi=iopts.dpi, iopts=iopts)
  File "/home/biodocker/code/jcvi/graphics/base.py", line 187, in savefig
    plt.savefig(figname, dpi=dpi)
  File "/usr/local/lib/python2.7/dist-packages/matplotlib/pyplot.py", line 710, in savefig
    res = fig.savefig(*args, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/matplotlib/figure.py", line 2035, in savefig
    self.canvas.print_figure(fname, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/matplotlib/backend_bases.py", line 2263, in print_figure
    **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/matplotlib/backends/backend_pdf.py", line 2586, in print_pdf
    self.figure.draw(renderer)
  File "/usr/local/lib/python2.7/dist-packages/matplotlib/artist.py", line 55, in draw_wrapper
    return draw(artist, renderer, *args, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/matplotlib/figure.py", line 1475, in draw
    renderer, self, artists, self.suppressComposite)
  File "/usr/local/lib/python2.7/dist-packages/matplotlib/image.py", line 141, in _draw_list_compositing_images
    a.draw(renderer)
  File "/usr/local/lib/python2.7/dist-packages/matplotlib/artist.py", line 55, in draw_wrapper
    return draw(artist, renderer, *args, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/matplotlib/axes/_base.py", line 2607, in draw
    mimage._draw_list_compositing_images(renderer, self, artists)
  File "/usr/local/lib/python2.7/dist-packages/matplotlib/image.py", line 141, in _draw_list_compositing_images
    a.draw(renderer)
  File "/usr/local/lib/python2.7/dist-packages/matplotlib/artist.py", line 55, in draw_wrapper
    return draw(artist, renderer, *args, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/matplotlib/text.py", line 751, in draw
    mtext=mtext)
  File "/usr/local/lib/python2.7/dist-packages/matplotlib/backends/backend_pdf.py", line 1919, in draw_tex
    dvifile = texmanager.make_dvi(s, fontsize)
  File "/usr/local/lib/python2.7/dist-packages/matplotlib/texmanager.py", line 365, in make_dvi
    texfile], tex)
  File "/usr/local/lib/python2.7/dist-packages/matplotlib/texmanager.py", line 335, in _run_checked_subprocess
    stderr=subprocess.STDOUT)
  File "/usr/local/lib/python2.7/dist-packages/subprocess32.py", line 334, in check_output
    process = Popen(stdout=PIPE, *popenargs, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/subprocess32.py", line 614, in __init__
    restore_signals, start_new_session)
  File "/usr/local/lib/python2.7/dist-packages/subprocess32.py", line 1393, in _execute_child
    raise child_exception_type(errno_num, err_msg)
OSError: [Errno 2] No such file or directory: 'latex'

root@4f7e77b83217:/home/biodocker/ALLMAPS-testdata# ll
total 45912
drwxr-xr-x  2 root      root          4096 Jul 10 18:42 ./
drwxr-xr-x 12 biodocker biodocker     4096 Jul 10 18:42 ../
-rw-r--r--  1 root      root          2300 Jul 10 18:42 JM-2.agp
-rw-r--r--  1 root      root          8086 Jul 10 18:42 JM-2.bed
-rw-r--r--  1 root      root          1912 Jul 10 18:42 JM-2.chain
-rw-r--r--  1 root      root          2300 Jul 10 18:42 JM-2.chr.agp
-rw-r--r--  1 root      root      15527195 Jul 10 18:42 JM-2.chr.fasta
-rw-r--r--  1 root      root      15527195 Jul 10 18:42 JM-2.fasta
-rw-r--r--  1 root      root            15 Jul 10 18:42 JM-2.fasta.sizes
-rw-r--r--  1 root      root          7590 Jul 10 18:42 JM-2.lifted.bed
-rw-r--r--  1 root      root          2160 Jul 10 18:42 JM-2.summary.txt
-rw-r--r--  1 root      root        359579 Jul 10 18:42 JM-2.tour
-rw-r--r--  1 root      root             0 Jul 10 18:42 JM-2.unplaced.agp
-rw-r--r--  1 root      root             0 Jul 10 18:42 JM-2.unplaced.fasta
-rw-r--r--  1 root      root          1911 May  7  2014 JMFemale.csv
-rw-r--r--  1 root      root          1847 May  7  2014 JMMale.csv
-rw-r--r--  1 root      root           537 Jul 10 18:42 chr23.pdf
-rw-r--r--  1 root      root           218 Sep 19  2014 run.sh
-rw-r--r--  1 root      root      15525370 May  7  2014 scaffolds.fasta
-rw-r--r--  1 root      root           444 Jul 10 18:42 scaffolds.fasta.sizes
-rw-r--r--  1 root      root             0 Jul 10 18:42 unmapped
-rw-r--r--  1 root      root            20 May 22  2014 weights.txt```
tanghaibao commented 6 years ago

@xinshuaiqi

Try google on how to install Latex (it was only an optional requirement before when ALLMAPS was developed a few years ago, however, matplotlib has changed its way to interact with latex in recent versions, so it's emitting this error). On Debian-like Linux OS you can do:

apt-get install -y texlive texlive-latex-extra texlive-latex-recommended

Haibao

xinshuaiqi commented 6 years ago

Now it works! Thanks, Haibao!