search

WGS-TB / MentaLiST

The MLST pipeline developed by the PathOGiST research group
MIT License
36 stars 11 forks source link

Error message download database #83

Open ELPhD opened 5 years ago

ELPhD commented 5 years ago

Hi, I used the following command: mentalist download_cgmlst -o lm_cgmlst_fasta -s 690488 -k 31 --db lm_cgmlst.db to try and download the L. mono cgmlst and but got (in addition to a load of warnings which i have ignored) an error message: ERROR: LoadError: LoadError: LoadError: MethodError: no method matching convert(::Type{AssertionError}, ::String) Closest candidates are: convert(!Matched::Type{Any}, ::ANY) at essentials.jl:28 convert(::Type{T}, !Matched::T) where T at essentials.jl:29 AssertionError(::String) at ./coreimg.jl:14 I was previously using mentalist 0.2.3 and got this error but updated to version 0.2.4 today and the problem hasnt gone away. We suspect it could be a problem with Julia which is running version 0.6.1 as downloaded with mentalist.

Thank you Erin

rpetit3 commented 5 years ago

I was about to post the same issue. I'm getting the same error as @ELPhD. I also received the same error message with v0.2.3. This error also occurs with download_pubmlst.

MentaLiST Version

Using the latest available from Bioconda (v0.2.4, h3c2d85e_0)

mentalist --version
WARNING: imported binding for VERSION overwritten in module Main
MentaLiST 0.3.

Commands

mentalist download_cgmlst -o cgmlst -s "Staphylococcus aureus"  -k 31 --db cgmlst/cgmlst-31.db

# or 

mentalist download_pubmlst -o mlst -s "Staphylococcus aureus"  -k 31 --db mlst/mlst-31.db

Error Message

ERROR: LoadError: LoadError: LoadError: MethodError: no method matching convert(::Type{AssertionError}, ::String)
Closest candidates are:
  convert(!Matched::Type{Any}, ::ANY) at essentials.jl:28
  convert(::Type{T}, !Matched::T) where T at essentials.jl:29
AssertionError(::String) at ./coreimg.jl:14
update_valid_age!(::UInt64, ::UInt64, ::Core.Inference.InferenceState) at ./inference.jl:2353
add_backedge!(::Core.MethodInstance, ::Core.Inference.InferenceState) at ./inference.jl:2366
abstract_call_gf_by_type(::Any, ::Any, ::Core.Inference.InferenceState) at ./inference.jl:1421
abstract_call(::Any, ::Array{Any,1}, ::Array{Any,1}, ::Array{Any,1}, ::Core.Inference.InferenceState) at ./inference.jl:1897
abstract_eval_call(::Expr, ::Array{Any,1}, ::Core.Inference.InferenceState) at ./inference.jl:1927
abstract_eval(::Any, ::Array{Any,1}, ::Core.Inference.InferenceState) at ./inference.jl:1950
abstract_interpret(::Any, ::Array{Any,1}, ::Core.Inference.InferenceState) at ./inference.jl:2084
typeinf_work(::Core.Inference.InferenceState) at ./inference.jl:2669
typeinf(::Core.Inference.InferenceState) at ./inference.jl:2787
typeinf_edge(::Method, ::Any, ::SimpleVector, ::Core.Inference.InferenceState) at ./inference.jl:2535
abstract_call_gf_by_type(::Any, ::Any, ::Core.Inference.InferenceState) at ./inference.jl:1420
abstract_call(::Any, ::Array{Any,1}, ::Array{Any,1}, ::Array{Any,1}, ::Core.Inference.InferenceState) at ./inference.jl:1897
abstract_eval_call(::Expr, ::Array{Any,1}, ::Core.Inference.InferenceState) at ./inference.jl:1927
abstract_eval(::Any, ::Array{Any,1}, ::Core.Inference.InferenceState) at ./inference.jl:1950
abstract_interpret(::Any, ::Array{Any,1}, ::Core.Inference.InferenceState) at ./inference.jl:2084
typeinf_work(::Core.Inference.InferenceState) at ./inference.jl:2669
typeinf(::Core.Inference.InferenceState) at ./inference.jl:2787
typeinf_edge(::Method, ::Any, ::SimpleVector, ::Core.Inference.InferenceState) at ./inference.jl:2535
abstract_call_gf_by_type(::Any, ::Any, ::Core.Inference.InferenceState) at ./inference.jl:1420
abstract_call(::Any, ::Array{Any,1}, ::Array{Any,1}, ::Array{Any,1}, ::Core.Inference.InferenceState) at ./inference.jl:1897
abstract_eval_call(::Expr, ::Array{Any,1}, ::Core.Inference.InferenceState) at ./inference.jl:1927
abstract_eval(::Any, ::Array{Any,1}, ::Core.Inference.InferenceState) at ./inference.jl:1950
typeinf_work(::Core.Inference.InferenceState) at ./inference.jl:2722
typeinf(::Core.Inference.InferenceState) at ./inference.jl:2787
typeinf_edge(::Method, ::Any, ::SimpleVector, ::Core.Inference.InferenceState) at ./inference.jl:2535
abstract_call_gf_by_type(::Any, ::Any, ::Core.Inference.InferenceState) at ./inference.jl:1420
abstract_call(::Any, ::Array{Any,1}, ::Array{Any,1}, ::Array{Any,1}, ::Core.Inference.InferenceState) at ./inference.jl:1897
abstract_eval_call(::Expr, ::Array{Any,1}, ::Core.Inference.InferenceState) at ./inference.jl:1927
abstract_eval(::Any, ::Array{Any,1}, ::Core.Inference.InferenceState) at ./inference.jl:1950
abstract_interpret(::Any, ::Array{Any,1}, ::Core.Inference.InferenceState) at ./inference.jl:2084
typeinf_work(::Core.Inference.InferenceState) at ./inference.jl:2669
typeinf(::Core.Inference.InferenceState) at ./inference.jl:2787
typeinf_edge(::Method, ::Any, ::SimpleVector, ::Core.Inference.InferenceState) at ./inference.jl:2535
abstract_call_gf_by_type(::Any, ::Any, ::Core.Inference.InferenceState) at ./inference.jl:1420
abstract_call(::Any, ::Tuple{}, ::Array{Any,1}, ::Array{Any,1}, ::Core.Inference.InferenceState) at ./inference.jl:1897
abstract_apply(::Any, ::Array{Any,1}, ::Array{Any,1}, ::Array{Any,1}, ::Core.Inference.InferenceState) at ./inference.jl:1561
abstract_call(::Any, ::Array{Any,1}, ::Array{Any,1}, ::Array{Any,1}, ::Core.Inference.InferenceState) at ./inference.jl:1689
abstract_eval_call(::Expr, ::Array{Any,1}, ::Core.Inference.InferenceState) at ./inference.jl:1927
abstract_eval(::Any, ::Array{Any,1}, ::Core.Inference.InferenceState) at ./inference.jl:1950
typeinf_work(::Core.Inference.InferenceState) at ./inference.jl:2722
typeinf(::Core.Inference.InferenceState) at ./inference.jl:2787
typeinf_edge(::Method, ::Any, ::SimpleVector, ::Core.Inference.InferenceState) at ./inference.jl:2535
abstract_call_gf_by_type(::Any, ::Any, ::Core.Inference.InferenceState) at ./inference.jl:1420
abstract_call(::Any, ::Tuple{}, ::Array{Any,1}, ::Array{Any,1}, ::Core.Inference.InferenceState) at ./inference.jl:1897
abstract_apply(::Any, ::Array{Any,1}, ::Array{Any,1}, ::Array{Any,1}, ::Core.Inference.InferenceState) at ./inference.jl:1561
abstract_call(::Any, ::Array{Any,1}, ::Array{Any,1}, ::Array{Any,1}, ::Core.Inference.InferenceState) at ./inference.jl:1689
abstract_eval_call(::Expr, ::Array{Any,1}, ::Core.Inference.InferenceState) at ./inference.jl:1927
abstract_eval(::Any, ::Array{Any,1}, ::Core.Inference.InferenceState) at ./inference.jl:1950
typeinf_work(::Core.Inference.InferenceState) at ./inference.jl:2722
typeinf(::Core.Inference.InferenceState) at ./inference.jl:2787
typeinf_edge(::Method, ::Any, ::SimpleVector, ::Core.Inference.InferenceState) at ./inference.jl:2535
abstract_call_gf_by_type(::Any, ::Any, ::Core.Inference.InferenceState) at ./inference.jl:1420
abstract_call(::Any, ::Tuple{}, ::Array{Any,1}, ::Array{Any,1}, ::Core.Inference.InferenceState) at ./inference.jl:1897
abstract_apply(::Any, ::Array{Any,1}, ::Array{Any,1}, ::Array{Any,1}, ::Core.Inference.InferenceState) at ./inference.jl:1561
abstract_call(::Any, ::Array{Any,1}, ::Array{Any,1}, ::Array{Any,1}, ::Core.Inference.InferenceState) at ./inference.jl:1689
abstract_eval_call(::Expr, ::Array{Any,1}, ::Core.Inference.InferenceState) at ./inference.jl:1927
abstract_eval(::Any, ::Array{Any,1}, ::Core.Inference.InferenceState) at ./inference.jl:1950
typeinf_work(::Core.Inference.InferenceState) at ./inference.jl:2722
typeinf(::Core.Inference.InferenceState) at ./inference.jl:2787
typeinf_frame(::Core.MethodInstance, ::Bool, ::Bool, ::Core.Inference.InferenceParams) at ./inference.jl:2504
typeinf_code(::Core.MethodInstance, ::Bool, ::Bool, ::Core.Inference.InferenceParams) at ./inference.jl:2583
typeinf_ext(::Core.MethodInstance, ::UInt64) at ./inference.jl:2622
macro expansion at ./loading.jl:180 [inlined]
(::Base.##691#693)() at ./task.jl:335

...and 1 more exception(s).

Stacktrace:
 [1] sync_end() at ./task.jl:287
 [2] macro expansion at ./task.jl:303 [inlined]
 [3] _require_from_serialized(::Int64, ::Symbol, ::String, ::Bool) at ./loading.jl:177
 [4] _require(::Symbol) at ./loading.jl:498
 [5] require(::Symbol) at ./loading.jl:405
 [6] include_from_node1(::String) at ./loading.jl:576
 [7] include(::String) at ./sysimg.jl:14
 [8] include_from_node1(::String) at ./loading.jl:576
 [9] include(::String) at ./sysimg.jl:14
 [10] include_from_node1(::String) at ./loading.jl:576
 [11] include(::String) at ./sysimg.jl:14
 [12] process_options(::Base.JLOptions) at ./client.jl:305
 [13] _start() at ./client.jl:371
while loading /home/rpetit/miniconda3/envs/mentalist/bin/db_graph.jl, in expression starting on line 2
while loading /home/rpetit/miniconda3/envs/mentalist/bin/build_db_functions.jl, in expression starting on line 6
while loading /home/rpetit/miniconda3/envs/mentalist/bin/mentalist, in expression starting on line 317
MethodError(Core.Inference.convert, (AssertionError, "invalid age range update"), 0x0000000000000ac6)

Conda Environment (conda list)

# packages in environment at /home/rpetit/miniconda3/envs/mentalist:
#
# Name                    Version                   Build  Channel
arpack                    3.5.0           blas_openblas_202  [blas_openblas]  conda-forge
blas                      1.1                    openblas    conda-forge
ca-certificates           2018.11.29           ha4d7672_0    conda-forge
curl                      7.63.0               h74213dd_0    conda-forge
fftw                      3.3.8                h470a237_1    conda-forge
gmp                       6.1.2                hfc679d8_0    conda-forge
hdf5                      1.10.3               hc401514_2    conda-forge
icu                       58.2                 hfc679d8_0    conda-forge
julia                     0.6.1           blas_openblashfe40789_203  [blas_openblas]  conda-forge
krb5                      1.16.2               hbb41f41_0    conda-forge
libcurl                   7.63.0               hbdb9355_0    conda-forge
libedit                   3.1.20170329         haf1bffa_1    conda-forge
libgcc                    7.2.0                h69d50b8_2    conda-forge
libgcc-ng                 7.2.0                hdf63c60_3    conda-forge
libgfortran               3.0.0                         1    conda-forge
libgit2                   0.27.7               h5ee2e84_0    conda-forge
libiconv                  1.15                 h470a237_3    conda-forge
libssh2                   1.8.0                h5b517e9_3    conda-forge
libstdcxx-ng              7.2.0                hdf63c60_3    conda-forge
libxml2                   2.9.8                h422b904_5    conda-forge
mentalist                 0.2.4                h3c2d85e_0    bioconda
metis                     5.1.0                hfc679d8_3    conda-forge
mpfr                      4.0.1                h16a7912_0    conda-forge
ncurses                   6.1                  hfc679d8_2    conda-forge
openblas                  0.2.20                        8    conda-forge
openlibm                  0.5.4                h470a237_0    conda-forge
openspecfun               0.5.3                h26a2512_0    conda-forge
openssl                   1.0.2p               h470a237_1    conda-forge
pcre2                     10.23                         2    conda-forge
suitesparse               4.5.6           blas_openblash55dcaef_201  [blas_openblas]  conda-forge
tbb                       2019.3               h2d50403_0    conda-forge
tk                        8.6.9                ha92aebf_0    conda-forge
unzip                     6.0                           0    conda-forge
xz                        5.2.4                h470a237_1    conda-forge
zlib                      1.2.11               h470a237_3    conda-forge
bastian-wur commented 5 years ago

Same here, I'm getting the same error as rpetit3 . Installed it today.

dfornika commented 5 years ago

Thanks for reporting @ELPhD @rpetit3 @bastian-wur. I was able to reproduce the error on my system too. I'm investigating now.

dfornika commented 5 years ago

Ok, I think I see what went wrong. When tagging the v0.2.4 release, I selected the wrong branch to tag against (I chose master instead of mentalist_v0.2).

I've deleted the old v0.2.4 release and created a new release against the correct branch. I'm not sure if that will propagate automatically to a new bioconda build, so I'll follow up with that in a day or two.

In the meantime, if the option is available to you I'd recommend grabbing an official julia-0.6 binary from:

https://julialang.org/downloads/oldreleases.html

...and add it to your $PATH. Then follow the instructions listed on the main README.md of the MentaLiST repo for running on macos to install the dependencies using Pkg.add():

https://github.com/WGS-TB/MentaLiST#macos-108

...Then grab the MentaLiST source tarball:

https://github.com/WGS-TB/MentaLiST/archive/v0.2.4.tar.gz

...and run from there (the src/mentalist file serves as the main entry point).

Please let us know if this work-around will work for you, and I'll try to get the bioconda build fixed as soon as possible if that is your only installation option.

dfornika commented 5 years ago

By the way @ELPhD, while testing the corrected v0.2.4 release, I ran into another error when building the exact database that you mention in your error report:

mentalist download_cgmlst -o lm_cgmlst_fasta -s 690488 -k 31 --db lm_cgmlst.db

I get an error as follows:

[...some lines omitted for brevity...]
Trying to download: https://www.cgmlst.org/ncs/schema/690488/alleles
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 15.1M    0 15.1M    0     0  2585k      0 --:--:--  0:00:06 --:--:-- 3307k
2018-12-12T11:12:03.879 - info: Unzipping cgMLST scheme into individual FASTA files for each locus ...
.........
2018-12-12T11:12:10.412 - info: 1701 loci found.
2018-12-12T11:12:10.546 - info: Building the k-mer database ...
2018-12-12T11:12:10.762 - info: Opening FASTA files ... 
    From worker 2:  CRITICAL ERROR: Error parsing file /home/dfornika/Downloads/lm_cgmlst_fasta/lmo0001.fasta, at record lmo0001_196, most likely some unkown characters. Please fix it and try again.
Worker 2 terminated.ERROR: 
LoadError:  From worker 3:  CRITICAL ERROR: Error parsing file /home/dfornika/Downloads/lm_cgmlst_fasta/lmo0002.fasta, at record lmo0002_242, most likely some unkown characters. Please fix it and try again.ProcessExitedException()
Worker 3 terminated.ERROR (unhandled task failure): EOFError: read end of file

Stacktrace:

ERROR (unhandled task failure): EOFError: read end of file
 [1] #573 at ./asyncmap.jl:178 [inlined]
 [2] foreach(::Base.##573#575, ::Array{Any,1}) at ./abstractarray.jl:1733
[...some lines omitted for brevity...]

It seems that the lmo0001.fasta file in that scheme may be corrupt. When I look at the file, I see some strange alleles around lmo0001_196 and lmo0001_197:

ATGAATTTGGTGGTCGAGATCGTACAACCGTCATCCATGCACATGAAAAAATATCGCAACTACTAAAAACCGATCAAGTGTTGAAAAATGACCTTGCCGAAATTGAAAAAAATTTAAGAAAAGCACAAAATATGTTTTAA
>lmo0001_196
lmo0001
>lmo0001_197
>lmo0001_198
GTGCAATCAATTGAAGACATCTGGCAGGAAACACTGCAAATTGTTAAAAAAAATATGAGTAAACCTAGTTACGATACATGGATGAAATCAACAACCGCTCATTCACTTGAAGGTAACACGTTTATTATTTCAGCGCCCAATAATTTTGTTCG
dfornika commented 5 years ago

@rpetit3 I've successfully built one of the databases you mentioned above, after following my proposed work-around using the official julia binary.

mentalist download_cgmlst -o cgmlst -s "Staphylococcus aureus"  -k 31 --db cgmlst/cgmlst-31.db

I still get a few warning messages but the database does build successfully.

rpetit3 commented 5 years ago

Thanks for the follow up Dan!

dfornika commented 5 years ago

I've submitted a pull request to re-generate the bioconda package for mentalist-0.2.4.

https://github.com/bioconda/bioconda-recipes/pull/12629

dfornika commented 5 years ago

Hmm... The new build of mentalist-0.2.4 is available now, but it still gives the same errors when building a cgMLST database.

The problem may be with the conda-forge julia-0.6.1 package. When I have a chance I'll try running MentaLiST against some older julia-0.6.1 builds and see if one of them works.

bastian-wur commented 5 years ago

That's too bad :/. Guess I'll try it without conda for now :). (EDIT: In case I can figure out if I can install Julia packages on the cluster here... )

EDIT: Okay, I managed to install everything. Apparently another dependency called "JuMP" is missing from the description, but that's not much of an issue. I still get an error though:

WARNING: Method definition (::Type{Bio.Align.SAMWriter{T} where T<:IO})(IO) in module Align at /home/bvhhornung/.julia/v0.6/Bio/src/align/hts/sam/writer.jl:14 overwritten at /home/bvhhornung/.julia/v0.6/Bio/src/align/hts/sam/writer.jl:22.
WARNING: Method definition (::Type{Bio.Align.SAMWriter{T} where T<:IO})(IO) in module Align at /home/bvhhornung/.julia/v0.6/Bio/src/align/hts/sam/writer.jl:14 overwritten at /home/bvhhornung/.julia/v0.6/Bio/src/align/hts/sam/writer.jl:22.
ERROR: LoadError: LoadError: LoadError: LoadError: MethodError: no method matching isless(::String, ::VersionNumber)
Closest candidates are:
  isless(!Matched::VersionNumber, ::VersionNumber) at version.jl:142
  isless(!Matched::Missings.Missing, ::Any) at /home/bvhhornung/.julia/v0.6/Missings/src/Missings.jl:74
  isless(::AbstractString, !Matched::AbstractString) at strings/basic.jl:130
  ...
Stacktrace:
 [1] <(::String, ::VersionNumber) at ./operators.jl:194
 [2] include_from_node1(::String) at ./loading.jl:576
while loading /home/bvhhornung/.julia/v0.6/JuMP/src/JuMP.jl, in expression starting on line 10
while loading /exports/mm-hpc/bacteriologie/bastian/tools/MentaLiST/src/db_graph.jl, in expression starting on line 8
while loading /exports/mm-hpc/bacteriologie/bastian/tools/MentaLiST/src/build_db_functions.jl, in expression starting on line 6
while loading /exports/mm-hpc/bacteriologie/bastian/tools/MentaLiST/src/mentalist, in expression starting on line 317
rpetit3 commented 5 years ago

I've had success with using build blas_openblas_202 of Julia 0.6.1 from conda-forge. Built a MLST (download_pubmlst) and a cgMLST (download_cgmlst) database, then successfully queried them (call)

MentaLiST Version

MentaLiST-0.2.4/src/mentalist --version
MentaLiST 0.2.4.

Conda list

Ignore the MentaLiST v0.2.2 below, the release for v0.2.4 was used to test. I only used v0.2.2 to install the other requirements associated with MentaLiST. 

conda install julia=0.6.1=blas_openblas_202 mentalist

conda list
# packages in environment at /home/rpetit/miniconda3/envs/julia:
#
# Name                    Version                   Build  Channel
arpack                    3.6.3           blas_openblash1f444ea_0  [blas_openblas]  conda-forge
blas                      1.1                    openblas    conda-forge
ca-certificates           2018.11.29           ha4d7672_0    conda-forge
curl                      7.63.0               h74213dd_0    conda-forge
fftw                      3.3.8                h470a237_1    conda-forge
gmp                       6.1.2                hfc679d8_0    conda-forge
hdf5                      1.10.4          nompi_h5598ddc_1003    conda-forge
icu                       58.2                 hfc679d8_0    conda-forge
julia                     0.6.1           blas_openblas_202  [blas_openblas]  conda-forge
krb5                      1.16.2               hbb41f41_0    conda-forge
libcurl                   7.63.0               hbdb9355_0    conda-forge
libedit                   3.1.20170329         haf1bffa_1    conda-forge
libgcc-ng                 7.2.0                hdf63c60_3    conda-forge
libgfortran               3.0.0                         1    conda-forge
libgit2                   0.27.7               h5ee2e84_0    conda-forge
libiconv                  1.15                 h470a237_3    conda-forge
libssh2                   1.8.0                h5b517e9_3    conda-forge
libstdcxx-ng              7.2.0                hdf63c60_3    conda-forge
libxml2                   2.9.8                h422b904_5    conda-forge
mentalist                 0.2.2                         1    bioconda
metis                     5.1.0                hfc679d8_3    conda-forge
mpfr                      3.1.5                         0    conda-forge
ncurses                   6.1                  hfc679d8_2    conda-forge
openblas                  0.2.20                        8    conda-forge
openlibm                  0.5.4                h470a237_0    conda-forge
openspecfun               0.5.3                h26a2512_0    conda-forge
openssl                   1.0.2p               h470a237_1    conda-forge
pcre2                     10.23                         2    conda-forge
suitesparse               4.5.6           blas_openblash55dcaef_201  [blas_openblas]  conda-forge
tbb                       2019.3               h2d50403_0    conda-forge
tk                        8.6.9                ha92aebf_0    conda-forge
unzip                     6.0                           0    conda-forge
xz                        5.2.4                h470a237_1    conda-forge
zlib                      1.2.11               h470a237_3    conda-forge
dfornika commented 5 years ago

Good to know @rpetit3 . Thanks for testing that out. We might be able to tag the MentaLiST bioconda recipe against that specific julia build.

In the long-run I think the solution will be to migrate to julia-1.0 but it may take some time to get there.

sean-la commented 5 years ago

I'm also having this same issue. @rpetit3, how were you able to make your version of mentalist v0.2.4, after using v0.2.2 to install dependencies? What I'm doing is running conda install julia=0.6.1=blas_openblas_202 mentalist, and then running conda install -c bioconda mentalist=0.2.4, but this is forcing me to update Julia.

rpetit3 commented 5 years ago

@realseanla Sorry for the delay. I was using the v0.2.4 release (https://github.com/WGS-TB/MentaLiST/releases/tag/v0.2.4) from this repo not BioConda. So,

conda install julia=0.6.1=blas_openblas_202 mentalist
wget https://github.com/WGS-TB/MentaLiST/archive/v0.2.4.tar.gz
tar xzvf v0.2.4.tar.gz
MentaLiST-0.2.4/src/mentalist --version

This was just to verify v0.2.4 executed using the same dependencies (see below) as BioConda's v0.2.2.

@dfornika Looking at the meta.yaml recipe for MentaLiST, I think you should be able to add the specific build version (Conda Docs). It might not be optimal, but its at least a patch until migration to Julia 1.0

requirements:
  host:
    - julia 0.6.1 blas_openblas_202
    - hdf5
    - mpfr
  run:
    - julia 0.6.1 blas_openblas_202
    - hdf5
    - mpfr
    - libxml2
    - unzip
jwcodee commented 5 years ago

Are there any updates on this issue?

rpetit3 commented 5 years ago

I could create a PR on bioconda-recipes with my proposed fix above. But I'd want approval for the fix before doing so.

dfornika commented 5 years ago

That would be great @rpetit3 . Looks good to me.

rpetit3 commented 5 years ago

Submitted a PR https://github.com/bioconda/bioconda-recipes/pull/13137

rpetit3 commented 5 years ago

Alright, merged into bioconda (https://github.com/bioconda/bioconda-recipes/commit/f27fe117e2c86e69167cd39ed6bb18690c9c6053). I probably won't have a chance to test till Monday. @ELPhD @bastian-wur @realseanla Want to give it a try?

rpetit3 commented 5 years ago

Sadly, I'm still getting the same error, even with julia 0.6.1 blas_openblas_202. Not sure why it works for v0.2.2 mentioned above, but here. The conda environments are nearly identical.

mentalist download_cgmlst -o cgmlst -s "Staphylococcus aureus"  -k 31 --db cgmlst/cgmlst-31.db
WARNING: imported binding for VERSION overwritten in module Main
WARNING: Method definition (::Type{Bio.Align.SAMWriter{T} where T<:IO})(IO) in module Align at /home/rpetit/miniconda3/envs/mentalist/share/julia/site/v0.6/Bio/src/align/hts/sam/writer.jl:14 overwritten at /home/rpetit/miniconda3/envs/mentalist/share/julia/site/v0.6/Bio/src/align/hts/sam/writer.jl:22.
ERROR: LoadError: LoadError: LoadError: MethodError: no method matching convert(::Type{AssertionError}, ::String)
Closest candidates are:
  convert(!Matched::Type{Any}, ::ANY) at essentials.jl:28
  convert(::Type{T}, !Matched::T) where T at essentials.jl:29
AssertionError(::String) at ./coreimg.jl:14
update_valid_age!(::UInt64, ::UInt64, ::Core.Inference.InferenceState) at ./inference.jl:2353
add_backedge!(::Core.MethodInstance, ::Core.Inference.InferenceState) at ./inference.jl:2366
abstract_call_gf_by_type(::Any, ::Any, ::Core.Inference.InferenceState) at ./inference.jl:1421
abstract_call(::Any, ::Array{Any,1}, ::Array{Any,1}, ::Array{Any,1}, ::Core.Inference.InferenceState) at ./inference.jl:1897
abstract_eval_call(::Expr, ::Array{Any,1}, ::Core.Inference.InferenceState) at ./inference.jl:1927
abstract_eval(::Any, ::Array{Any,1}, ::Core.Inference.InferenceState) at ./inference.jl:1950
abstract_interpret(::Any, ::Array{Any,1}, ::Core.Inference.InferenceState) at ./inference.jl:2084
typeinf_work(::Core.Inference.InferenceState) at ./inference.jl:2669
typeinf(::Core.Inference.InferenceState) at ./inference.jl:2787
typeinf_edge(::Method, ::Any, ::SimpleVector, ::Core.Inference.InferenceState) at ./inference.jl:2535
abstract_call_gf_by_type(::Any, ::Any, ::Core.Inference.InferenceState) at ./inference.jl:1420
abstract_call(::Any, ::Array{Any,1}, ::Array{Any,1}, ::Array{Any,1}, ::Core.Inference.InferenceState) at ./inference.jl:1897
abstract_eval_call(::Expr, ::Array{Any,1}, ::Core.Inference.InferenceState) at ./inference.jl:1927
abstract_eval(::Any, ::Array{Any,1}, ::Core.Inference.InferenceState) at ./inference.jl:1950
abstract_interpret(::Any, ::Array{Any,1}, ::Core.Inference.InferenceState) at ./inference.jl:2084
typeinf_work(::Core.Inference.InferenceState) at ./inference.jl:2669
typeinf(::Core.Inference.InferenceState) at ./inference.jl:2787
typeinf_edge(::Method, ::Any, ::SimpleVector, ::Core.Inference.InferenceState) at ./inference.jl:2535
abstract_call_gf_by_type(::Any, ::Any, ::Core.Inference.InferenceState) at ./inference.jl:1420
abstract_call(::Any, ::Array{Any,1}, ::Array{Any,1}, ::Array{Any,1}, ::Core.Inference.InferenceState) at ./inference.jl:1897
abstract_eval_call(::Expr, ::Array{Any,1}, ::Core.Inference.InferenceState) at ./inference.jl:1927
abstract_eval(::Any, ::Array{Any,1}, ::Core.Inference.InferenceState) at ./inference.jl:1950
typeinf_work(::Core.Inference.InferenceState) at ./inference.jl:2722
typeinf(::Core.Inference.InferenceState) at ./inference.jl:2787
typeinf_edge(::Method, ::Any, ::SimpleVector, ::Core.Inference.InferenceState) at ./inference.jl:2535
abstract_call_gf_by_type(::Any, ::Any, ::Core.Inference.InferenceState) at ./inference.jl:1420
abstract_call(::Any, ::Array{Any,1}, ::Array{Any,1}, ::Array{Any,1}, ::Core.Inference.InferenceState) at ./inference.jl:1897
abstract_eval_call(::Expr, ::Array{Any,1}, ::Core.Inference.InferenceState) at ./inference.jl:1927
abstract_eval(::Any, ::Array{Any,1}, ::Core.Inference.InferenceState) at ./inference.jl:1950
abstract_interpret(::Any, ::Array{Any,1}, ::Core.Inference.InferenceState) at ./inference.jl:2084
typeinf_work(::Core.Inference.InferenceState) at ./inference.jl:2669
typeinf(::Core.Inference.InferenceState) at ./inference.jl:2787
typeinf_edge(::Method, ::Any, ::SimpleVector, ::Core.Inference.InferenceState) at ./inference.jl:2535
abstract_call_gf_by_type(::Any, ::Any, ::Core.Inference.InferenceState) at ./inference.jl:1420
abstract_call(::Any, ::Tuple{}, ::Array{Any,1}, ::Array{Any,1}, ::Core.Inference.InferenceState) at ./inference.jl:1897
abstract_apply(::Any, ::Array{Any,1}, ::Array{Any,1}, ::Array{Any,1}, ::Core.Inference.InferenceState) at ./inference.jl:1561
abstract_call(::Any, ::Array{Any,1}, ::Array{Any,1}, ::Array{Any,1}, ::Core.Inference.InferenceState) at ./inference.jl:1689
abstract_eval_call(::Expr, ::Array{Any,1}, ::Core.Inference.InferenceState) at ./inference.jl:1927
abstract_eval(::Any, ::Array{Any,1}, ::Core.Inference.InferenceState) at ./inference.jl:1950
typeinf_work(::Core.Inference.InferenceState) at ./inference.jl:2722
typeinf(::Core.Inference.InferenceState) at ./inference.jl:2787
typeinf_edge(::Method, ::Any, ::SimpleVector, ::Core.Inference.InferenceState) at ./inference.jl:2535
abstract_call_gf_by_type(::Any, ::Any, ::Core.Inference.InferenceState) at ./inference.jl:1420
abstract_call(::Any, ::Tuple{}, ::Array{Any,1}, ::Array{Any,1}, ::Core.Inference.InferenceState) at ./inference.jl:1897
abstract_apply(::Any, ::Array{Any,1}, ::Array{Any,1}, ::Array{Any,1}, ::Core.Inference.InferenceState) at ./inference.jl:1561
abstract_call(::Any, ::Array{Any,1}, ::Array{Any,1}, ::Array{Any,1}, ::Core.Inference.InferenceState) at ./inference.jl:1689
abstract_eval_call(::Expr, ::Array{Any,1}, ::Core.Inference.InferenceState) at ./inference.jl:1927
abstract_eval(::Any, ::Array{Any,1}, ::Core.Inference.InferenceState) at ./inference.jl:1950
typeinf_work(::Core.Inference.InferenceState) at ./inference.jl:2722
typeinf(::Core.Inference.InferenceState) at ./inference.jl:2787
typeinf_edge(::Method, ::Any, ::SimpleVector, ::Core.Inference.InferenceState) at ./inference.jl:2535
abstract_call_gf_by_type(::Any, ::Any, ::Core.Inference.InferenceState) at ./inference.jl:1420
abstract_call(::Any, ::Tuple{}, ::Array{Any,1}, ::Array{Any,1}, ::Core.Inference.InferenceState) at ./inference.jl:1897
abstract_apply(::Any, ::Array{Any,1}, ::Array{Any,1}, ::Array{Any,1}, ::Core.Inference.InferenceState) at ./inference.jl:1561
abstract_call(::Any, ::Array{Any,1}, ::Array{Any,1}, ::Array{Any,1}, ::Core.Inference.InferenceState) at ./inference.jl:1689
abstract_eval_call(::Expr, ::Array{Any,1}, ::Core.Inference.InferenceState) at ./inference.jl:1927
abstract_eval(::Any, ::Array{Any,1}, ::Core.Inference.InferenceState) at ./inference.jl:1950
typeinf_work(::Core.Inference.InferenceState) at ./inference.jl:2722
typeinf(::Core.Inference.InferenceState) at ./inference.jl:2787
typeinf_frame(::Core.MethodInstance, ::Bool, ::Bool, ::Core.Inference.InferenceParams) at ./inference.jl:2504
typeinf_code(::Core.MethodInstance, ::Bool, ::Bool, ::Core.Inference.InferenceParams) at ./inference.jl:2583
typeinf_ext(::Core.MethodInstance, ::UInt64) at ./inference.jl:2622
macro expansion at ./loading.jl:180 [inlined]
(::Base.##691#693)() at ./task.jl:335

...and 1 more exception(s).

Stacktrace:
 [1] sync_end() at ./task.jl:287
 [2] macro expansion at ./task.jl:303 [inlined]
 [3] _require_from_serialized(::Int64, ::Symbol, ::String, ::Bool) at ./loading.jl:177
 [4] _require_search_from_serialized(::Int64, ::Symbol, ::String, ::Bool) at ./loading.jl:236
 [5] _require(::Symbol) at ./loading.jl:441
 [6] require(::Symbol) at ./loading.jl:405
 [7] include_from_node1(::String) at ./loading.jl:576
 [8] include(::String) at ./sysimg.jl:14
 [9] include_from_node1(::String) at ./loading.jl:576
 [10] include(::String) at ./sysimg.jl:14
 [11] include_from_node1(::String) at ./loading.jl:576
 [12] include(::String) at ./sysimg.jl:14
 [13] process_options(::Base.JLOptions) at ./client.jl:305
 [14] _start() at ./client.jl:371
while loading /home/rpetit/miniconda3/envs/mentalist/bin/db_graph.jl, in expression starting on line 2
while loading /home/rpetit/miniconda3/envs/mentalist/bin/build_db_functions.jl, in expression starting on line 6
while loading /home/rpetit/miniconda3/envs/mentalist/bin/mentalist, in expression starting on line 314
MethodError(Core.Inference.convert, (AssertionError, "invalid age range update"), 0x0000000000000ac6)

Conda list

conda list
# packages in environment at /home/rpetit/miniconda3/envs/mentalist:
#
# Name                    Version                   Build  Channel
arpack                    3.6.3           blas_openblash35bd731_1000  [blas_openblas]  conda-forge
blas                      1.1                    openblas    conda-forge
ca-certificates           2018.11.29           ha4d7672_0    conda-forge
curl                      7.63.0            h646f8bb_1000    conda-forge
fftw                      3.3.8             h14c3975_1001    conda-forge
gmp                       6.1.2             hf484d3e_1000    conda-forge
hdf5                      1.10.3            hba1933b_1001    conda-forge
icu                       58.2              hf484d3e_1000    conda-forge
julia                     0.6.1           blas_openblas_202  [blas_openblas]  conda-forge
krb5                      1.16.3            hc83ff2d_1000    conda-forge
libcurl                   7.63.0            h01ee5af_1000    conda-forge
libedit                   3.1.20170329      hf8c457e_1001    conda-forge
libgcc-ng                 7.3.0                hdf63c60_0    conda-forge
libgfortran               3.0.0                         1    conda-forge
libgfortran-ng            7.2.0                hdf63c60_3    conda-forge
libgit2                   0.27.7            hbd8068c_1000    conda-forge
libiconv                  1.15              h14c3975_1004    conda-forge
libssh2                   1.8.0             h1ad7b7a_1003    conda-forge
libstdcxx-ng              7.3.0                hdf63c60_0    conda-forge
libxml2                   2.9.8             h143f9aa_1005    conda-forge
mentalist                 0.2.4                h972260a_2    bioconda
metis                     5.1.0             hf484d3e_1003    conda-forge
mpfr                      3.1.5                         0    conda-forge
ncurses                   6.1               hf484d3e_1002    conda-forge
openblas                  0.2.20                        8    conda-forge
openlibm                  0.5.4             h14c3975_1000    conda-forge
openspecfun               0.5.3             h1c2f66e_1000    conda-forge
openssl                   1.0.2p            h14c3975_1002    conda-forge
pcre2                     10.23                         2    conda-forge
suitesparse               4.5.6           blas_openblash55dcaef_201  [blas_openblas]  conda-forge
tbb                       2019.3            h6bb024c_1000    conda-forge
tk                        8.6.9             h84994c4_1000    conda-forge
unzip                     6.0                           0    conda-forge
xz                        5.2.4             h14c3975_1001    conda-forge
zlib                      1.2.11            h14c3975_1004    conda-forge
moorembioinfo commented 5 years ago

Hi! Having the same issue at the moment. Tried to the suggested patches and Julia versions but I'm getting the same issue. Are there any updates? :)

dfornika commented 5 years ago

@moorembioinfo The best solution I can suggest at this point is to use one of the official julia-0.6 binaries from: https://julialang.org/downloads/oldreleases.html and to use the source code from this GitHub repo. Would that be a viable work-around for you?

pedrofeijao commented 5 years ago

Hi, I just updated MentaLiST to run on Julia 1, and I was hoping this issue would be fixed, but apparently I still got errors. But then I looked at the FASTA files for this scheme and they contain wrongly formatted entries. For instance:

grep lmo0001_196 -A3 lm_cgmlst_fasta/lmo0001.fasta
>lmo0001_196
lmo0001
>lmo0001_197
>lmo0001_198

and 

grep lmo0002_242 -A3 lm_cgmlst_fasta/lmo0002.fasta
>lmo0002_242
lmo0002
>lmo0002_243
>lmo0002_244

Could any of you check if you're also getting the same weird lines?

dfornika commented 5 years ago

@pedrofeijao Looks similar to what I reported earlier in this thread:

https://github.com/WGS-TB/MentaLiST/issues/83#issuecomment-446715335

pedrofeijao commented 5 years ago

@dfornika Oh, I don't know how I missed that, thanks for the heads up. With respect to the error

ERROR: LoadError: LoadError: LoadError: MethodError: no method matching convert(::Type{AssertionError}, ::String)

reported above, it seems that it is fixed on the current version, based on Julia 1.1, at least for all cgmlst schemes that I've tried. I'm currently working on better detection and message for wrongly formatted FASTA files.
@rpetit3 and @realseanla , I'd appreciate if you could install Julia1.1 and pull the current github version to test again.

rpetit3 commented 5 years ago

Apologies for the slow response, it took ~30 minutes (don't remember it taking that long) but I was able to successfully create a cgMLST database. I installed the latest Julia release (v1.0.3) from conda-forge and MentsLiST is a clone of the master branch.

time src/mentalist download_cgmlst -o cgmlst -s "Staphylococcus aureus"  -k 31 --db cgmlst/cgmlst-31.db
[ Info: Downloading cgMLST scheme ...
Trying to download: https://www.cgmlst.org/ncs/schema/141106/alleles
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 21.4M    0 21.4M    0     0  3160k      0 --:--:--  0:00:06 --:--:-- 3610k
[ Info: Unzipping cgMLST scheme into individual FASTA files for each locus ...
..........
[ Info: 1861 loci found.
[ Info: Building the k-mer database ...
[ Info: Opening FASTA files ...
[ Info: Combining results for each locus ...
[ Info: Saving DB ...
[ Info: Done!

real    32m24.520s
user    32m12.382s
sys 0m4.991s

Conda list

conda list
# packages in environment at /home/rpetit/miniconda3/envs/julia:
#
# Name                    Version                   Build  Channel
arpack                    3.6.3           blas_openblash6d6ea35_1002  [blas_openblas]  conda-forge
blas                      1.1                    openblas    conda-forge
ca-certificates           2018.11.29           ha4d7672_0    conda-forge
curl                      7.64.0               h646f8bb_0    conda-forge
fftw                      3.3.8             h14c3975_1002    conda-forge
gmp                       6.1.2             hf484d3e_1000    conda-forge
julia                     1.0.3           blas_openblash12d65f3_2  [blas_openblas]  conda-forge
krb5                      1.16.3            hc83ff2d_1000    conda-forge
libcurl                   7.64.0               h01ee5af_0    conda-forge
libedit                   3.1.20170329      hf8c457e_1001    conda-forge
libgcc-ng                 7.3.0                hdf63c60_0    conda-forge
libgfortran-ng            7.2.0                hdf63c60_3    conda-forge
libgit2                   0.28.1               hbd8068c_0    conda-forge
libssh2                   1.8.0             h1ad7b7a_1003    conda-forge
libstdcxx-ng              7.3.0                hdf63c60_0    conda-forge
metis                     5.1.0             hf484d3e_1003    conda-forge
mpfr                      4.0.1             ha14ba45_1000    conda-forge
ncurses                   6.1               hf484d3e_1002    conda-forge
openblas                  0.3.3             h9ac9557_1001    conda-forge
openlibm                  0.5.4             h14c3975_1000    conda-forge
openspecfun               0.5.3             h1c2f66e_1000    conda-forge
openssl                   1.0.2p            h14c3975_1002    conda-forge
pcre2                     10.23                         2    conda-forge
suitesparse               4.5.6           blas_openblash17e8c26_1201  [blas_openblas]  conda-forge
tbb                       2019.3            h6bb024c_1000    conda-forge
tk                        8.6.9             h84994c4_1000    conda-forge
zlib                      1.2.11            h14c3975_1004    conda-forge

I also created a MLST DB and that only took ~1 minute.

Are there plans to create a release of this version or is it still considered a work in progress? If a release is created I can submit to bioconda

jwcodee commented 5 years ago

@pedrofeijao

I installed julia 1.1 along with other packages and run the mentalist github code and it downloaded but didn't construct the database for both pubmlst and cgmlst and for enterobase, I am met with this error 

[ Info: Building the k-mer database ...
[ Info: Opening FASTA files ...
┌ Error: Error parsing file test_download_salmonella/AU28_RS22050.fa, around record AU28_RS22050_18, most likely some unkown characters are present; this allele will be skipped.
│ This might make the results on this locus unreliable; please fix this FASTA file and rerun.
└ @ Main ~/MentaLiST/src/db_graph.jl:94
┌ Error: Error parsing file test_download_salmonella/SLT-BT0051.fa, around record SLT-BT0051_58, most likely some unkown characters are present; this allele will be skipped.
│ This might make the results on this locus unreliable; please fix this FASTA file and rerun.
└ @ Main ~/MentaLiST/src/db_graph.jl:94
┌ Error: Error parsing file test_download_salmonella/SLT-BT0331.fa, around record SLT-BT0331_36, most likely some unkown characters are present; this allele will be skipped.
│ This might make the results on this locus unreliable; please fix this FASTA file and rerun.
└ @ Main ~/MentaLiST/src/db_graph.jl:94
┌ Error: Error parsing file test_download_salmonella/SLT-BT0331.fa, around record SLT-BT0331_36, most likely some unkown characters are present; this allele will be skipped.
│ This might make the results on this locus unreliable; please fix this FASTA file and rerun.
└ @ Main ~/MentaLiST/src/db_graph.jl:94
┌ Error: Error parsing file test_download_salmonella/SLT-BT0331.fa, around record SLT-BT0331_36, most likely some unkown characters are present; this allele will be skipped.
│ This might make the results on this locus unreliable; please fix this FASTA file and rerun.
└ @ Main ~/MentaLiST/src/db_graph.jl:94
┌ Error: Too many unrecoverable errors on this FASTA file; skipping this locus ...
└ @ Main ~/MentaLiST/src/db_graph.jl:97
Worker 2 terminated.
ERROR: LoadError: ProcessExitedException()
Stacktrace:
 [1] (::getfield(Base, Symbol("##696#698")))(::Task) at ./asyncmap.jl:178
 [2] foreach(::getfield(Base, Symbol("##696#698")), ::Array{Any,1}) at ./abstractarray.jl:1866
 [3] maptwice(::Function, ::Channel{Any}, ::Array{Any,1}, ::Array{String,1}) at ./asyncmap.jl:178
 [4] #async_usemap#681 at ./asyncmap.jl:154 [inlined]
 [5] #async_usemap at ./none:0 [inlined]
 [6] #asyncmap#680 at ./asyncmap.jl:81 [inlined]
 [7] #asyncmap at ./none:0 [inlined]
 [8] #pmap#215(::Bool, ::Int64, ::Nothing, ::Array{Any,1}, ::Nothing, ::Function, ::Function, ::WorkerPool, ::Array{String,1}) at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.1/Distributed/src/pmap.jl:126
 [9] pmap(::Function, ::WorkerPool, ::Array{String,1}) at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.1/Distributed/src/pmap.jl:101
 [10] #pmap#225(::Base.Iterators.Pairs{Union{},Union{},Tuple{},NamedTuple{(),Tuple{}}}, ::Function, ::Function, ::Array{String,1}) at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.1/Distributed/src/pmap.jl:156
 [11] pmap at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.1/Distributed/src/pmap.jl:156 [inlined]
 [12] kmer_class_for_each_locus(::Type{Kmer{DNA,32}}, ::Array{String,1}, ::Float64) at /home/klw17/MentaLiST/src/build_db_functions.jl:110
 [13] build_db(::Dict{String,Any}, ::String) at /home/klw17/MentaLiST/src/MentaLiST.jl:256
 [14] build_db(::Dict{String,Any}) at /home/klw17/MentaLiST/src/MentaLiST.jl:242
 [15] download_enterobase(::Dict{String,Any}) at /home/klw17/MentaLiST/src/MentaLiST.jl:237
 [16] top-level scope at /home/klw17/MentaLiST/src/MentaLiST.jl:366
 [17] include at ./boot.jl:326 [inlined]
 [18] include_relative(::Module, ::String) at ./loading.jl:1038
 [19] include(::Module, ::String) at ./sysimg.jl:29
 [20] exec_options(::Base.JLOptions) at ./client.jl:267
 [21] _start() at ./client.jl:436
in expression starting at /home/klw17/MentaLiST/src/MentaLiST.jl:323
pedrofeijao commented 5 years ago

@jowong4 Some cgMLST schemes have pretty bad FASTA files, and not always MentaLiST can parse them without errors. I'd suggest that you take a look at those FASTA files as seen in the error output, and try again. You don't have to redownload the FASTA files, as they should already be there on the folder you specified, so just run the 'build_db' command.

pedrofeijao commented 5 years ago

Thanks @rpetit3 ! I believe @dfornika is working on a conda release right now.

dfornika commented 5 years ago

@rpetit3 @pedrofeijao Yes, I've also been testing this out when I have time. I've just pushed an updated conda recipe to this repo:

https://github.com/WGS-TB/MentaLiST/tree/master/conda

It's not exactly ready for bioconda because it pulls the source code from the master branch of the repo rather than a stable tag. I also haven't really thought carefully if any of those dependencies in the meta.yaml file should change or be dropped, but there aren't really that many so it's probably fine.

I've also done a test build that's available on my personal conda channel:

https://anaconda.org/dfornika/mentalist

I've done some really quick testing with it, just building one cgMLST database and it worked.

Personally I'd like to see a bit more testing done using the conda-supplied julia before tagging v1.0.0 so we don't end up in a bugfix-release-bugfix-release cycle but I don't have much time to be doing the testing. So I won't object if you want to tag the v1.0.0 release soon, @pedrofeijao . It would be good if we could put together a brief summary of bugfixes & new features to include in the release as we've done with previous releases, but it is possible to edit that stuff after the release is tagged.

dfornika commented 5 years ago

@rpetit3 @pedrofeijao We did manage to sneak one more bugfix in. I've tagged a MentaLiST-v1.0.0 release now.

dfornika commented 5 years ago

Sorry, we've had another bug report. I'm trying to hold of on the 1.0.0 release until we've established that all of the basic functionality is working in the julia-1.x compatible version. I know we can't wait forever and there will probably always be little bugs here and there, but I'd like the 1.0.0 version to be as reliable as possible.