EpiDope failed on basic example

abrozzi commented 3 years ago

Dear all, I followed step by step your installation guide on my Mac BigSur 11.2.2 for conda but got this error just trying on a single sequence:

epidope -i P08191.fa -o ./results/

>P08191
MKRVITLFAVLLMGWSVNAWSFACKTANGTAIPIGGGSANVYVNLAPVVNVGQNLVVDLS
TQIFCHNDYPETITDYVTLQRGSAYGGVLSNFSGTVKYSGSSYPFPTTSETPRVVYNSRT
DKPWPVALYLTPVSSAGGVAIKAGSLIAVLILRQTNNYNSDDFQFVWNIYANNDVVVPTG
GCDVSARDVTVTLPDYPGSVPIPLTVYCAKSQNLGYYLSGTTADAGNSIFTNTASFSPAQ

Loading packages. Traceback (most recent call last): File "/Users/abrozzi/miniconda/envs/epidope/lib/python3.7/site-packages/numpy/core/init.py", line 17, in from . import multiarray File "/Users/abrozzi/miniconda/envs/epidope/lib/python3.7/site-packages/numpy/core/multiarray.py", line 14, in from . import overrides File "/Users/abrozzi/miniconda/envs/epidope/lib/python3.7/site-packages/numpy/core/overrides.py", line 7, in from numpy.core._multiarray_umath import ( ImportError: dlopen(/Users/abrozzi/miniconda/envs/epidope/lib/python3.7/site-packages/numpy/core/_multiarray_umath.cpython-37m-darwin.so, 2): Library not loaded: @rpath/libopenblas.dylib Referenced from: /Users/abrozzi/miniconda/envs/epidope/lib/python3.7/site-packages/numpy/core/_multiarray_umath.cpython-37m-darwin.so Reason: image not found

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "/Users/abrozzi/miniconda/envs/epidope/bin/epidope", line 33, in sys.exit(load_entry_point('EpiDope==0.2', 'console_scripts', 'epidope')()) File "/Users/abrozzi/miniconda/envs/epidope/lib/python3.7/site-packages/epidope/cli.py", line 49, in cli from epidope import epidope2 File "/Users/abrozzi/miniconda/envs/epidope/lib/python3.7/site-packages/epidope/epidope2.py", line 6, in from tensorflow.python.keras.backend import set_session File "/Users/abrozzi/miniconda/envs/epidope/lib/python3.7/site-packages/tensorflow/init.py", line 24, in from tensorflow.python import pywrap_tensorflow # pylint: disable=unused-import File "/Users/abrozzi/miniconda/envs/epidope/lib/python3.7/site-packages/tensorflow/python/init.py", line 47, in import numpy as np File "/Users/abrozzi/miniconda/envs/epidope/lib/python3.7/site-packages/numpy/init.py", line 142, in from . import core File "/Users/abrozzi/miniconda/envs/epidope/lib/python3.7/site-packages/numpy/core/init.py", line 47, in raise ImportError(msg) ImportError:

IMPORTANT: PLEASE READ THIS FOR ADVICE ON HOW TO SOLVE THIS ISSUE!

Importing the numpy c-extensions failed.

Try uninstalling and reinstalling numpy.
If you have already done that, then:
1. Check that you expected to use Python3.7 from "/Users/abrozzi/miniconda/envs/epidope/bin/python", and that you have no directories in your PATH or PYTHONPATH that can interfere with the Python and numpy version "1.17.4" you're trying to use.
2. If (1) looks fine, you can open a new issue at https://github.com/numpy/numpy/issues. Please include details on:
  - how you installed Python
  - how you installed numpy
  - your operating system
  - whether or not you have multiple versions of Python installed
  - if you built from source, your compiler versions and ideally a build log
If you're working with a numpy git repository, try git clean -xdf (removes all files not under version control) and rebuild numpy.

Note: this error has many possible causes, so please don't comment on an existing issue about this - open a new one instead.

Original error was: dlopen(/Users/abrozzi/miniconda/envs/epidope/lib/python3.7/site-packages/numpy/core/_multiarray_umath.cpython-37m-darwin.so, 2): Library not loaded: @rpath/libopenblas.dylib Referenced from: /Users/abrozzi/miniconda/envs/epidope/lib/python3.7/site-packages/numpy/core/_multiarray_umath.cpython-37m-darwin.so Reason: image not found

numpy is OK python 3.6 is OK

Thanks -A

abrozzi commented 3 years ago

Solved:

pip uninstall bumpy
pip install numpy
pip install mock
conda install mkl-service

flomock commented 3 years ago

@abrozzi Thx for your support by sending us the issues you have/had. For this issue, I'm wondering if we should adapt the conda build, as normally numpy, mock, and mkl are installed with our conda build. Were these packages missing in yours? Then please let me know. Was bumpy installed while installation of epidope? This wasn't the case in our tests.

As I suspect, you installed other software in the same environment which caused this issue?

We will have a look at the other issues you raised as well within the next few days. So far, we have not had the time to take care of it.

matdoering commented 3 years ago

EpiDope also didn't work for me with a basic example after following the conda installation instructions. I receive an error from allennlp, which seems to be a known problem of allennlp==0.9.0:

Traceback (most recent call last):
  File "/home/mdoering/.conda/envs/epidope/bin/epidope", line 33, in <module>
    sys.exit(load_entry_point('EpiDope==0.2', 'console_scripts', 'epidope')())
  File "/home/mdoering/.conda/envs/epidope/lib/python3.7/site-packages/epidope/cli.py", line 49, in cli
    from epidope import epidope2
  File "/home/mdoering/.conda/envs/epidope/lib/python3.7/site-packages/epidope/epidope2.py", line 18, in <module>
    from epidope.utils import embedder
  File "/home/mdoering/.conda/envs/epidope/lib/python3.7/site-packages/epidope/utils/embedder.py", line 3, in <module>
    from allennlp.commands.elmo import ElmoEmbedder
  File "/home/mdoering/.conda/envs/epidope/lib/python3.7/site-packages/allennlp/commands/__init__.py", line 8, in <module>
    from allennlp.commands.configure import Configure
  File "/home/mdoering/.conda/envs/epidope/lib/python3.7/site-packages/allennlp/commands/configure.py", line 26, in <module>
    from allennlp.service.config_explorer import make_app
  File "/home/mdoering/.conda/envs/epidope/lib/python3.7/site-packages/allennlp/service/config_explorer.py", line 24, in <module>
    from allennlp.common.configuration import configure, choices
  File "/home/mdoering/.conda/envs/epidope/lib/python3.7/site-packages/allennlp/common/configuration.py", line 17, in <module>
    from allennlp.data.dataset_readers import DatasetReader
  File "/home/mdoering/.conda/envs/epidope/lib/python3.7/site-packages/allennlp/data/__init__.py", line 1, in <module>
    from allennlp.data.dataset_readers.dataset_reader import DatasetReader
  File "/home/mdoering/.conda/envs/epidope/lib/python3.7/site-packages/allennlp/data/dataset_readers/__init__.py", line 10, in <module>
    from allennlp.data.dataset_readers.ccgbank import CcgBankDatasetReader
  File "/home/mdoering/.conda/envs/epidope/lib/python3.7/site-packages/allennlp/data/dataset_readers/ccgbank.py", line 9, in <module>
    from allennlp.data.dataset_readers.dataset_reader import DatasetReader
  File "/home/mdoering/.conda/envs/epidope/lib/python3.7/site-packages/allennlp/data/dataset_readers/dataset_reader.py", line 8, in <module>
    from allennlp.data.instance import Instance
  File "/home/mdoering/.conda/envs/epidope/lib/python3.7/site-packages/allennlp/data/instance.py", line 3, in <module>
    from allennlp.data.fields.field import DataArray, Field
  File "/home/mdoering/.conda/envs/epidope/lib/python3.7/site-packages/allennlp/data/fields/__init__.py", line 7, in <module>
    from allennlp.data.fields.array_field import ArrayField
  File "/home/mdoering/.conda/envs/epidope/lib/python3.7/site-packages/allennlp/data/fields/array_field.py", line 10, in <module>
    class ArrayField(Field[numpy.ndarray]):
  File "/home/mdoering/.conda/envs/epidope/lib/python3.7/site-packages/allennlp/data/fields/array_field.py", line 50, in ArrayField
    @overrides
  File "/home/mdoering/.conda/envs/epidope/lib/python3.7/site-packages/overrides/overrides.py", line 88, in overrides
    return _overrides(method, check_signature, check_at_runtime)
  File "/home/mdoering/.conda/envs/epidope/lib/python3.7/site-packages/overrides/overrides.py", line 114, in _overrides
    _validate_method(method, super_class, check_signature)
  File "/home/mdoering/.conda/envs/epidope/lib/python3.7/site-packages/overrides/overrides.py", line 135, in _validate_method
    ensure_signature_is_compatible(super_method, method, is_static)
  File "/home/mdoering/.conda/envs/epidope/lib/python3.7/site-packages/overrides/signature.py", line 93, in ensure_signature_is_compatible
    ensure_return_type_compatibility(super_type_hints, sub_type_hints, method_name)
  File "/home/mdoering/.conda/envs/epidope/lib/python3.7/site-packages/overrides/signature.py", line 288, in ensure_return_type_compatibility
    f"{method_name}: return type `{sub_return}` is not a `{super_return}`."
TypeError: ArrayField.empty_field: return type `None` is not a `<class 'allennlp.data.fields.field.Field'>`.

The same error has been described here: https://github.com/allenai/allennlp/issues/5203

Updating allennlp via pip install allennlp --upgrade and then downgrading via pip install allennlp==0.9.0 surprisingly fixes the issue but brings another problem to light when predicting:

Traceback (most recent call last):
  File "cli.py", line 59, in <module>
    cli(sys.argv[1:])
  File "cli.py", line 53, in cli
    epi_seqs=args.epitopes, non_epi_seqs=args.nonepitopes)
  File "/home/mdoering/.conda/envs/epidope/lib/python3.7/site-packages/epidope/epidope2.py", line 549, in start_pipeline
    protein_results_dict = predict_files(fasta, slicesize, epitope_threshold, threads)
  File "/home/mdoering/.conda/envs/epidope/lib/python3.7/site-packages/epidope/epidope2.py", line 332, in predict_files
    protein_result = predict_protein(model, X_test, protein, nb_samples, positions, epitope_threshold, threads)
  File "/home/mdoering/.conda/envs/epidope/lib/python3.7/site-packages/epidope/epidope2.py", line 294, in predict_protein
    inputs_test=X_test, suffix=suffix_weights)
  File "/home/mdoering/.conda/envs/epidope/lib/python3.7/site-packages/epidope/epidope2.py", line 192, in ensemble_prediction
    model.load_weights(fn, by_name=True)
  File "/home/mdoering/.local/lib/python3.7/site-packages/tensorflow/python/keras/engine/training.py", line 250, in load_weights
    return super(Model, self).load_weights(filepath, by_name, skip_mismatch)
  File "/home/mdoering/.local/lib/python3.7/site-packages/tensorflow/python/keras/engine/network.py", line 1264, in load_weights
    f, self.layers, skip_mismatch=skip_mismatch)
  File "/home/mdoering/.local/lib/python3.7/site-packages/tensorflow/python/keras/saving/hdf5_format.py", line 785, in load_weights_from_hdf5_group_by_name
    str(weight_values[i].shape) + '.')
ValueError: Layer #3 (named "bidirectional_1"), weight <tf.Variable 'bidirectional_1/forward_lstm_1/lstm_cell_4/kernel:0' shape=(10, 40) dtype=float32, numpy=
array([[-0.18077078,  0.14346477,  0.19063354,  0.12690404,  0.33600372,
         0.11456054, -0.21650246, -0.22245312,  0.06624106, -0.0107424 ,
        -0.02575135,  0.04133132,  0.23419356,  0.13926637, -0.0445554 ,
        -0.21326053,  0.12835333,  0.1801284 , -0.24888533,  0.18982345,
         0.3085792 , -0.2736091 , -0.25153345,  0.00952169, -0.12263319,
        -0.16180311,  0.05258289,  0.1869905 ,  0.20294511, -0.18624274,
        -0.07873866, -0.3055866 , -0.07104793,  0.18652767, -0.15077743,
         0.04231638, -0.17100528, -0.08641621,  0.16179311, -0.33430147],
       [ 0.03152376, -0.31361723,  0.10062197, -0.23628564,  0.2694025 ,
         0.16685975, -0.08385366,  0.13605368, -0.08914691, -0.1612215 ,
         0.24338943,  0.20479763,  0.18711352,  0.283571  , -0.09370887,
         0.01486591,  0.26742017, -0.12076177,  0.2797662 , -0.05494094,
        -0.3370727 , -0.32977608,  0.33581913, -0.07056302, -0.24453232,
        -0.00059077,  0.28159642, -0.16511722,  0.19494104,  0.24846387,
         0.18595994,  0.26320934, -0.23670727,  0.33723706,  0.18910962,
        -0.3021534 , -0.19798169, -0.29008535,  0.10612556,  0.06160459],
       [ 0.33258927, -0.0240964 ,  0.05907854,  0.24055797, -0.13456398,
         0.06963596,  0.30807996,  0.18918633, -0.12517954, -0.22476839,
        -0.25191137,  0.3089763 , -0.2519102 ,  0.05201003, -0.247968  ,
         0.22479248, -0.3237815 ,  0.20958138, -0.00163794,  0.12694508,
        -0.1123894 , -0.29016182, -0.27977598,  0.26310074, -0.00552648,
         0.19401956, -0.16391586,  0.31945753,  0.23013604,  0.04887879,
         0.32421243, -0.17677423, -0.09737566,  0.32851452,  0.25669277,
        -0.05708012,  0.01138577,  0.23496312, -0.26699284,  0.05320397],
       [-0.12933567,  0.13527551, -0.23005418,  0.22454202,  0.06425813,
         0.00788063,  0.21294606,  0.29877162, -0.14484395, -0.09566611,
         0.30550516,  0.25077707, -0.23339447, -0.1757103 ,  0.18988103,
         0.12457654,  0.32277626,  0.1531083 , -0.02706131, -0.0948042 ,
         0.03377429, -0.29054537,  0.22402829, -0.03336149,  0.00233683,
         0.10301271,  0.0016728 ,  0.19900137,  0.02971601, -0.08808604,
         0.14129636,  0.17179674,  0.01796058,  0.22662735,  0.2948658 ,
        -0.09538588, -0.2916028 ,  0.0228377 ,  0.19497615, -0.06846085],
       [ 0.04868534,  0.31603628, -0.17206681, -0.15246773,  0.1708529 ,
        -0.1747762 ,  0.32913023, -0.11295679,  0.02967721,  0.22818589,
         0.22092378, -0.2165469 ,  0.17546701, -0.18574473,  0.09494418,
         0.34239715,  0.06693208,  0.13843459,  0.21341687,  0.20835865,
        -0.16365546, -0.02979812,  0.07494596,  0.03943834,  0.29352534,
         0.07699278,  0.1281116 ,  0.09319615,  0.28116494,  0.27519077,
         0.33986735,  0.2507096 , -0.28731567, -0.09816894,  0.15142941,
        -0.2706724 , -0.17692132,  0.15866357,  0.16156518,  0.29728585],
       [-0.29347098,  0.13987836, -0.2614301 , -0.313763  ,  0.18009579,
         0.22907335,  0.05422965, -0.03147709,  0.21836776,  0.15709257,
        -0.19385381,  0.3324421 ,  0.13259238,  0.22795677, -0.23085663,
        -0.2133186 ,  0.14292273, -0.15498501, -0.16094045,  0.00792637,
         0.14969847,  0.27954197, -0.2280246 ,  0.19646722,  0.10577786,
        -0.2823738 , -0.17163205, -0.29763865, -0.18097445, -0.00794372,
        -0.3284062 , -0.16470832,  0.27418482, -0.04810029, -0.18458259,
        -0.11959939, -0.33404404,  0.05967346,  0.25239313, -0.33161712],
       [ 0.1260303 , -0.19883353,  0.30920827,  0.21870166, -0.0074904 ,
        -0.10536556,  0.2847033 , -0.26287255,  0.10818511,  0.20457286,
        -0.23119062,  0.14638147,  0.01927897, -0.21322964,  0.02822924,
         0.20207304,  0.32824785, -0.24022233, -0.16783999, -0.07042569,
         0.12515602, -0.18536316, -0.1440558 ,  0.06859514, -0.31227875,
        -0.09896363,  0.24968225,  0.02903166,  0.07155016,  0.08485904,
         0.3193888 , -0.30846325, -0.07016617, -0.07009786, -0.14531547,
         0.20821762,  0.18743163,  0.17487478,  0.0390254 ,  0.26228213],
       [ 0.18650699,  0.06260368,  0.01178998, -0.20705675, -0.31221747,
        -0.08739516,  0.29216862,  0.09280005, -0.23187712,  0.12848216,
        -0.2535522 , -0.0937283 , -0.18755643,  0.3080892 , -0.1517463 ,
        -0.10341939,  0.16361904,  0.15425038, -0.3180026 ,  0.21932203,
         0.27204484,  0.02436829, -0.24136141,  0.14788544,  0.26372987,
        -0.01766697, -0.29961497,  0.08276093,  0.00825608, -0.30847985,
        -0.26258647,  0.1786974 ,  0.151075  , -0.01194012,  0.28351206,
        -0.2676834 ,  0.3433842 ,  0.29664838,  0.22788155,  0.27787536],
       [ 0.0243246 ,  0.2901562 , -0.23252794,  0.25061798,  0.04829833,
         0.25552374, -0.04322216,  0.17674267,  0.23222852,  0.20371348,
        -0.2936256 ,  0.1263895 ,  0.28714395, -0.02634087, -0.02719593,
        -0.13430662,  0.27785242,  0.22151715, -0.08358616,  0.03552866,
        -0.05988613,  0.21257949, -0.24758957,  0.28713286, -0.18018234,
        -0.18799186,  0.14697883,  0.16529256, -0.24883488,  0.31141186,
         0.26593196, -0.02259216,  0.19848567, -0.23187737, -0.02050552,
        -0.0579364 ,  0.22357857,  0.03585324,  0.31875503, -0.05077085],
       [ 0.02373308,  0.16200554,  0.27509773,  0.31424814,  0.09089139,
         0.06600088, -0.18693586,  0.23417813, -0.08543611, -0.06840765,
         0.06788617,  0.17647946,  0.24433732, -0.18472077,  0.08080667,
        -0.2747816 ,  0.05349404, -0.2553951 , -0.20388304,  0.07736817,
         0.23323828,  0.12647325,  0.20032322,  0.04543716, -0.16783173,
        -0.17894389,  0.0428504 , -0.19310264,  0.27004445,  0.25456256,
        -0.21474932, -0.21083939,  0.0427978 , -0.04197082, -0.19106522,
         0.09802863, -0.14911322,  0.02378768,  0.14272904, -0.1892498 ]],
      dtype=float32)> has shape (10, 40), but the saved weight has shape (1024, 20).

So, the stored weights don't seem to match what the model expects. Since the docker image is working, I thought that the weights may be different but they seem to be the same. Looking into the pip environment of the docker container, however, showed that some requirements were different in my Conda env.

So, I downgraded the following two packages in my Conda env (tensorflow 2) to the versions in the Docker container:

tensorflow==1.13.1
Keras==2.3.1

This resulted in the theano backend being used, so I next issued

conda install mkl-service

This somehow enforces the tensorflow backend and allowed me to execute EpiDope.

In summary, I had to execute the following commands to make EpiDope work:

pip install allennlp --upgrade && pip install allenlp==0.9.0 && \
pip install tensorflow==1.13.1 Keras==2.3.1 && \
conda install mkl-service

Hope this helps someone else.

mcollatz commented 3 years ago

Thanks @matdoering for your detailed error description and how you were able to solve the problem.

flomock commented 3 years ago

This should be fixed in the newest version see https://github.com/flomock/EpiDope for the most recent version.

mcollatz / EpiDope

EpiDope failed on basic example #6