Closed abrozzi closed 3 years ago
Solved:
pip uninstall bumpy
pip install numpy
pip install mock
conda install mkl-service
@abrozzi Thx for your support by sending us the issues you have/had. For this issue, I'm wondering if we should adapt the conda build, as normally numpy, mock, and mkl are installed with our conda build. Were these packages missing in yours? Then please let me know. Was bumpy installed while installation of epidope? This wasn't the case in our tests.
As I suspect, you installed other software in the same environment which caused this issue?
We will have a look at the other issues you raised as well within the next few days. So far, we have not had the time to take care of it.
EpiDope also didn't work for me with a basic example after following the conda installation instructions. I receive an error from allennlp, which seems to be a known problem of allennlp==0.9.0:
Traceback (most recent call last):
File "/home/mdoering/.conda/envs/epidope/bin/epidope", line 33, in <module>
sys.exit(load_entry_point('EpiDope==0.2', 'console_scripts', 'epidope')())
File "/home/mdoering/.conda/envs/epidope/lib/python3.7/site-packages/epidope/cli.py", line 49, in cli
from epidope import epidope2
File "/home/mdoering/.conda/envs/epidope/lib/python3.7/site-packages/epidope/epidope2.py", line 18, in <module>
from epidope.utils import embedder
File "/home/mdoering/.conda/envs/epidope/lib/python3.7/site-packages/epidope/utils/embedder.py", line 3, in <module>
from allennlp.commands.elmo import ElmoEmbedder
File "/home/mdoering/.conda/envs/epidope/lib/python3.7/site-packages/allennlp/commands/__init__.py", line 8, in <module>
from allennlp.commands.configure import Configure
File "/home/mdoering/.conda/envs/epidope/lib/python3.7/site-packages/allennlp/commands/configure.py", line 26, in <module>
from allennlp.service.config_explorer import make_app
File "/home/mdoering/.conda/envs/epidope/lib/python3.7/site-packages/allennlp/service/config_explorer.py", line 24, in <module>
from allennlp.common.configuration import configure, choices
File "/home/mdoering/.conda/envs/epidope/lib/python3.7/site-packages/allennlp/common/configuration.py", line 17, in <module>
from allennlp.data.dataset_readers import DatasetReader
File "/home/mdoering/.conda/envs/epidope/lib/python3.7/site-packages/allennlp/data/__init__.py", line 1, in <module>
from allennlp.data.dataset_readers.dataset_reader import DatasetReader
File "/home/mdoering/.conda/envs/epidope/lib/python3.7/site-packages/allennlp/data/dataset_readers/__init__.py", line 10, in <module>
from allennlp.data.dataset_readers.ccgbank import CcgBankDatasetReader
File "/home/mdoering/.conda/envs/epidope/lib/python3.7/site-packages/allennlp/data/dataset_readers/ccgbank.py", line 9, in <module>
from allennlp.data.dataset_readers.dataset_reader import DatasetReader
File "/home/mdoering/.conda/envs/epidope/lib/python3.7/site-packages/allennlp/data/dataset_readers/dataset_reader.py", line 8, in <module>
from allennlp.data.instance import Instance
File "/home/mdoering/.conda/envs/epidope/lib/python3.7/site-packages/allennlp/data/instance.py", line 3, in <module>
from allennlp.data.fields.field import DataArray, Field
File "/home/mdoering/.conda/envs/epidope/lib/python3.7/site-packages/allennlp/data/fields/__init__.py", line 7, in <module>
from allennlp.data.fields.array_field import ArrayField
File "/home/mdoering/.conda/envs/epidope/lib/python3.7/site-packages/allennlp/data/fields/array_field.py", line 10, in <module>
class ArrayField(Field[numpy.ndarray]):
File "/home/mdoering/.conda/envs/epidope/lib/python3.7/site-packages/allennlp/data/fields/array_field.py", line 50, in ArrayField
@overrides
File "/home/mdoering/.conda/envs/epidope/lib/python3.7/site-packages/overrides/overrides.py", line 88, in overrides
return _overrides(method, check_signature, check_at_runtime)
File "/home/mdoering/.conda/envs/epidope/lib/python3.7/site-packages/overrides/overrides.py", line 114, in _overrides
_validate_method(method, super_class, check_signature)
File "/home/mdoering/.conda/envs/epidope/lib/python3.7/site-packages/overrides/overrides.py", line 135, in _validate_method
ensure_signature_is_compatible(super_method, method, is_static)
File "/home/mdoering/.conda/envs/epidope/lib/python3.7/site-packages/overrides/signature.py", line 93, in ensure_signature_is_compatible
ensure_return_type_compatibility(super_type_hints, sub_type_hints, method_name)
File "/home/mdoering/.conda/envs/epidope/lib/python3.7/site-packages/overrides/signature.py", line 288, in ensure_return_type_compatibility
f"{method_name}: return type `{sub_return}` is not a `{super_return}`."
TypeError: ArrayField.empty_field: return type `None` is not a `<class 'allennlp.data.fields.field.Field'>`.
The same error has been described here: https://github.com/allenai/allennlp/issues/5203
Updating allennlp via pip install allennlp --upgrade
and then downgrading via pip install allennlp==0.9.0
surprisingly fixes the issue but brings another problem to light when predicting:
Traceback (most recent call last):
File "cli.py", line 59, in <module>
cli(sys.argv[1:])
File "cli.py", line 53, in cli
epi_seqs=args.epitopes, non_epi_seqs=args.nonepitopes)
File "/home/mdoering/.conda/envs/epidope/lib/python3.7/site-packages/epidope/epidope2.py", line 549, in start_pipeline
protein_results_dict = predict_files(fasta, slicesize, epitope_threshold, threads)
File "/home/mdoering/.conda/envs/epidope/lib/python3.7/site-packages/epidope/epidope2.py", line 332, in predict_files
protein_result = predict_protein(model, X_test, protein, nb_samples, positions, epitope_threshold, threads)
File "/home/mdoering/.conda/envs/epidope/lib/python3.7/site-packages/epidope/epidope2.py", line 294, in predict_protein
inputs_test=X_test, suffix=suffix_weights)
File "/home/mdoering/.conda/envs/epidope/lib/python3.7/site-packages/epidope/epidope2.py", line 192, in ensemble_prediction
model.load_weights(fn, by_name=True)
File "/home/mdoering/.local/lib/python3.7/site-packages/tensorflow/python/keras/engine/training.py", line 250, in load_weights
return super(Model, self).load_weights(filepath, by_name, skip_mismatch)
File "/home/mdoering/.local/lib/python3.7/site-packages/tensorflow/python/keras/engine/network.py", line 1264, in load_weights
f, self.layers, skip_mismatch=skip_mismatch)
File "/home/mdoering/.local/lib/python3.7/site-packages/tensorflow/python/keras/saving/hdf5_format.py", line 785, in load_weights_from_hdf5_group_by_name
str(weight_values[i].shape) + '.')
ValueError: Layer #3 (named "bidirectional_1"), weight <tf.Variable 'bidirectional_1/forward_lstm_1/lstm_cell_4/kernel:0' shape=(10, 40) dtype=float32, numpy=
array([[-0.18077078, 0.14346477, 0.19063354, 0.12690404, 0.33600372,
0.11456054, -0.21650246, -0.22245312, 0.06624106, -0.0107424 ,
-0.02575135, 0.04133132, 0.23419356, 0.13926637, -0.0445554 ,
-0.21326053, 0.12835333, 0.1801284 , -0.24888533, 0.18982345,
0.3085792 , -0.2736091 , -0.25153345, 0.00952169, -0.12263319,
-0.16180311, 0.05258289, 0.1869905 , 0.20294511, -0.18624274,
-0.07873866, -0.3055866 , -0.07104793, 0.18652767, -0.15077743,
0.04231638, -0.17100528, -0.08641621, 0.16179311, -0.33430147],
[ 0.03152376, -0.31361723, 0.10062197, -0.23628564, 0.2694025 ,
0.16685975, -0.08385366, 0.13605368, -0.08914691, -0.1612215 ,
0.24338943, 0.20479763, 0.18711352, 0.283571 , -0.09370887,
0.01486591, 0.26742017, -0.12076177, 0.2797662 , -0.05494094,
-0.3370727 , -0.32977608, 0.33581913, -0.07056302, -0.24453232,
-0.00059077, 0.28159642, -0.16511722, 0.19494104, 0.24846387,
0.18595994, 0.26320934, -0.23670727, 0.33723706, 0.18910962,
-0.3021534 , -0.19798169, -0.29008535, 0.10612556, 0.06160459],
[ 0.33258927, -0.0240964 , 0.05907854, 0.24055797, -0.13456398,
0.06963596, 0.30807996, 0.18918633, -0.12517954, -0.22476839,
-0.25191137, 0.3089763 , -0.2519102 , 0.05201003, -0.247968 ,
0.22479248, -0.3237815 , 0.20958138, -0.00163794, 0.12694508,
-0.1123894 , -0.29016182, -0.27977598, 0.26310074, -0.00552648,
0.19401956, -0.16391586, 0.31945753, 0.23013604, 0.04887879,
0.32421243, -0.17677423, -0.09737566, 0.32851452, 0.25669277,
-0.05708012, 0.01138577, 0.23496312, -0.26699284, 0.05320397],
[-0.12933567, 0.13527551, -0.23005418, 0.22454202, 0.06425813,
0.00788063, 0.21294606, 0.29877162, -0.14484395, -0.09566611,
0.30550516, 0.25077707, -0.23339447, -0.1757103 , 0.18988103,
0.12457654, 0.32277626, 0.1531083 , -0.02706131, -0.0948042 ,
0.03377429, -0.29054537, 0.22402829, -0.03336149, 0.00233683,
0.10301271, 0.0016728 , 0.19900137, 0.02971601, -0.08808604,
0.14129636, 0.17179674, 0.01796058, 0.22662735, 0.2948658 ,
-0.09538588, -0.2916028 , 0.0228377 , 0.19497615, -0.06846085],
[ 0.04868534, 0.31603628, -0.17206681, -0.15246773, 0.1708529 ,
-0.1747762 , 0.32913023, -0.11295679, 0.02967721, 0.22818589,
0.22092378, -0.2165469 , 0.17546701, -0.18574473, 0.09494418,
0.34239715, 0.06693208, 0.13843459, 0.21341687, 0.20835865,
-0.16365546, -0.02979812, 0.07494596, 0.03943834, 0.29352534,
0.07699278, 0.1281116 , 0.09319615, 0.28116494, 0.27519077,
0.33986735, 0.2507096 , -0.28731567, -0.09816894, 0.15142941,
-0.2706724 , -0.17692132, 0.15866357, 0.16156518, 0.29728585],
[-0.29347098, 0.13987836, -0.2614301 , -0.313763 , 0.18009579,
0.22907335, 0.05422965, -0.03147709, 0.21836776, 0.15709257,
-0.19385381, 0.3324421 , 0.13259238, 0.22795677, -0.23085663,
-0.2133186 , 0.14292273, -0.15498501, -0.16094045, 0.00792637,
0.14969847, 0.27954197, -0.2280246 , 0.19646722, 0.10577786,
-0.2823738 , -0.17163205, -0.29763865, -0.18097445, -0.00794372,
-0.3284062 , -0.16470832, 0.27418482, -0.04810029, -0.18458259,
-0.11959939, -0.33404404, 0.05967346, 0.25239313, -0.33161712],
[ 0.1260303 , -0.19883353, 0.30920827, 0.21870166, -0.0074904 ,
-0.10536556, 0.2847033 , -0.26287255, 0.10818511, 0.20457286,
-0.23119062, 0.14638147, 0.01927897, -0.21322964, 0.02822924,
0.20207304, 0.32824785, -0.24022233, -0.16783999, -0.07042569,
0.12515602, -0.18536316, -0.1440558 , 0.06859514, -0.31227875,
-0.09896363, 0.24968225, 0.02903166, 0.07155016, 0.08485904,
0.3193888 , -0.30846325, -0.07016617, -0.07009786, -0.14531547,
0.20821762, 0.18743163, 0.17487478, 0.0390254 , 0.26228213],
[ 0.18650699, 0.06260368, 0.01178998, -0.20705675, -0.31221747,
-0.08739516, 0.29216862, 0.09280005, -0.23187712, 0.12848216,
-0.2535522 , -0.0937283 , -0.18755643, 0.3080892 , -0.1517463 ,
-0.10341939, 0.16361904, 0.15425038, -0.3180026 , 0.21932203,
0.27204484, 0.02436829, -0.24136141, 0.14788544, 0.26372987,
-0.01766697, -0.29961497, 0.08276093, 0.00825608, -0.30847985,
-0.26258647, 0.1786974 , 0.151075 , -0.01194012, 0.28351206,
-0.2676834 , 0.3433842 , 0.29664838, 0.22788155, 0.27787536],
[ 0.0243246 , 0.2901562 , -0.23252794, 0.25061798, 0.04829833,
0.25552374, -0.04322216, 0.17674267, 0.23222852, 0.20371348,
-0.2936256 , 0.1263895 , 0.28714395, -0.02634087, -0.02719593,
-0.13430662, 0.27785242, 0.22151715, -0.08358616, 0.03552866,
-0.05988613, 0.21257949, -0.24758957, 0.28713286, -0.18018234,
-0.18799186, 0.14697883, 0.16529256, -0.24883488, 0.31141186,
0.26593196, -0.02259216, 0.19848567, -0.23187737, -0.02050552,
-0.0579364 , 0.22357857, 0.03585324, 0.31875503, -0.05077085],
[ 0.02373308, 0.16200554, 0.27509773, 0.31424814, 0.09089139,
0.06600088, -0.18693586, 0.23417813, -0.08543611, -0.06840765,
0.06788617, 0.17647946, 0.24433732, -0.18472077, 0.08080667,
-0.2747816 , 0.05349404, -0.2553951 , -0.20388304, 0.07736817,
0.23323828, 0.12647325, 0.20032322, 0.04543716, -0.16783173,
-0.17894389, 0.0428504 , -0.19310264, 0.27004445, 0.25456256,
-0.21474932, -0.21083939, 0.0427978 , -0.04197082, -0.19106522,
0.09802863, -0.14911322, 0.02378768, 0.14272904, -0.1892498 ]],
dtype=float32)> has shape (10, 40), but the saved weight has shape (1024, 20).
So, the stored weights don't seem to match what the model expects. Since the docker image is working, I thought that the weights may be different but they seem to be the same. Looking into the pip environment of the docker container, however, showed that some requirements were different in my Conda env.
So, I downgraded the following two packages in my Conda env (tensorflow 2) to the versions in the Docker container:
tensorflow==1.13.1
Keras==2.3.1
This resulted in the theano backend being used, so I next issued
conda install mkl-service
This somehow enforces the tensorflow backend and allowed me to execute EpiDope.
In summary, I had to execute the following commands to make EpiDope work:
pip install allennlp --upgrade && pip install allenlp==0.9.0 && \
pip install tensorflow==1.13.1 Keras==2.3.1 && \
conda install mkl-service
Hope this helps someone else.
Thanks @matdoering for your detailed error description and how you were able to solve the problem.
This should be fixed in the newest version see https://github.com/flomock/EpiDope for the most recent version.
Dear all, I followed step by step your installation guide on my Mac BigSur 11.2.2 for conda but got this error just trying on a single sequence:
Loading packages. Traceback (most recent call last): File "/Users/abrozzi/miniconda/envs/epidope/lib/python3.7/site-packages/numpy/core/init.py", line 17, in
from . import multiarray
File "/Users/abrozzi/miniconda/envs/epidope/lib/python3.7/site-packages/numpy/core/multiarray.py", line 14, in
from . import overrides
File "/Users/abrozzi/miniconda/envs/epidope/lib/python3.7/site-packages/numpy/core/overrides.py", line 7, in
from numpy.core._multiarray_umath import (
ImportError: dlopen(/Users/abrozzi/miniconda/envs/epidope/lib/python3.7/site-packages/numpy/core/_multiarray_umath.cpython-37m-darwin.so, 2): Library not loaded: @rpath/libopenblas.dylib
Referenced from: /Users/abrozzi/miniconda/envs/epidope/lib/python3.7/site-packages/numpy/core/_multiarray_umath.cpython-37m-darwin.so
Reason: image not found
During handling of the above exception, another exception occurred:
Traceback (most recent call last): File "/Users/abrozzi/miniconda/envs/epidope/bin/epidope", line 33, in
sys.exit(load_entry_point('EpiDope==0.2', 'console_scripts', 'epidope')())
File "/Users/abrozzi/miniconda/envs/epidope/lib/python3.7/site-packages/epidope/cli.py", line 49, in cli
from epidope import epidope2
File "/Users/abrozzi/miniconda/envs/epidope/lib/python3.7/site-packages/epidope/epidope2.py", line 6, in
from tensorflow.python.keras.backend import set_session
File "/Users/abrozzi/miniconda/envs/epidope/lib/python3.7/site-packages/tensorflow/init.py", line 24, in
from tensorflow.python import pywrap_tensorflow # pylint: disable=unused-import
File "/Users/abrozzi/miniconda/envs/epidope/lib/python3.7/site-packages/tensorflow/python/init.py", line 47, in
import numpy as np
File "/Users/abrozzi/miniconda/envs/epidope/lib/python3.7/site-packages/numpy/init.py", line 142, in
from . import core
File "/Users/abrozzi/miniconda/envs/epidope/lib/python3.7/site-packages/numpy/core/init.py", line 47, in
raise ImportError(msg)
ImportError:
IMPORTANT: PLEASE READ THIS FOR ADVICE ON HOW TO SOLVE THIS ISSUE!
Importing the numpy c-extensions failed.
Try uninstalling and reinstalling numpy.
If you have already done that, then:
If you're working with a numpy git repository, try
git clean -xdf
(removes all files not under version control) and rebuild numpy.Note: this error has many possible causes, so please don't comment on an existing issue about this - open a new one instead.
Original error was: dlopen(/Users/abrozzi/miniconda/envs/epidope/lib/python3.7/site-packages/numpy/core/_multiarray_umath.cpython-37m-darwin.so, 2): Library not loaded: @rpath/libopenblas.dylib Referenced from: /Users/abrozzi/miniconda/envs/epidope/lib/python3.7/site-packages/numpy/core/_multiarray_umath.cpython-37m-darwin.so Reason: image not found
numpy is OK python 3.6 is OK
Thanks -A