search

LiLabAtVT / DeepTE

Neural network classification of TE
BSD 3-Clause "New" or "Revised" License
85 stars 7 forks source link

'tensorflow' has no attribute 'placeholder' ERROR #9

Closed francicco closed 3 years ago

francicco commented 3 years ago

Hi,

I installed DeepTE but the test doesn't seem to work

I executed DeepTE like this: (venv) [tk19812@bc4login2 DeepTE]$ DeepTE.py -d working_dir -o output_dir -i example_data/input_test.fasta -sp P -m_dir working_dir/download_P_model_dir/Plants_model/

And I got that:

Using TensorFlow backend.
2021-05-10 12:06:29.176528: W tensorflow/stream_executor/platform/default/dso_loader.cc:60] Could not load dynamic library 'libcudart.so.11.0'; dlerror: libcudart.so.11.0: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /mnt/storage/easybuild/software/CUDA/8.0.44-GCC-5.4.0-2.26/lib64:/mnt/storage/easybuild/software/binutils/2.26-GCCcore-5.4.0/lib:/mnt/storage/easybuild/software/GCCcore/5.4.0/lib/gcc/x86_64-unknown-linux-gnu/5.4.0:/mnt/storage/easybuild/software/GCCcore/5.4.0/lib64:/mnt/storage/easybuild/software/GCCcore/5.4.0/lib:/mnt/storage/software/languages/java/jdk1.8.0_141/lib:/mnt/storage/software/apps/ncbi-blast/gnu/2.7.1/lib:/mnt/storage/easybuild/software/Java/1.8.0_92/lib:/mnt/storage/easybuild/software/GSL/2.3-foss-2017a/lib:/mnt/storage/easybuild/software/ScaLAPACK/2.0.2-gompi-2017a-OpenBLAS-0.2.19-LAPACK-3.7.0/lib:/mnt/storage/easybuild/software/OpenBLAS/0.2.19-GCC-6.3.0-2.27-LAPACK-3.7.0/lib:/mnt/storage/easybuild/software/FFTW/3.3.6-gompi-2017a/lib:/mnt/storage/easybuild/software/OpenMPI/2.0.2-GCC-6.3.0-2.27/lib:/mnt/storage/easybuild/software/hwloc/1.11.5-GCC-6.3.0-2.27/lib:/mnt/storage/easybuild/software/numactl/2.0.11-GCC-6.3.0-2.27/lib:/mnt/storage/easybuild/software/HDF5/1.8.17-foss-2016b/lib:/mnt/storage/easybuild/software/Szip/2.1-foss-2016b/lib:/mnt/storage/easybuild/software/zlib/1.2.8-foss-2016b/lib:/mnt/storage/easybuild/software/SQLite/3.13.0-GCC-5.4.0-2.26/lib:/mnt/storage/easybuild/software/Tcl/8.6.5-GCC-5.4.0-2.26/lib:/mnt/storage/easybuild/software/libreadline/6.3-GCC-5.4.0-2.26/lib:/mnt/storage/easybuild/software/ncurses/6.0-GCC-5.4.0-2.26/lib:/mnt/storage/home/tk19812/scratch/software/lpsolve-5.5.0.10i/lib:/mnt/storage/home/tk19812/scratch/software/sparsehash/include:/mnt/storage/home/tk19812/scratch/software/bamtools/usr/local/lib64:/mnt/storage/home/tk19812/scratch/software/lzo-2.10/lib:/mnt/storage/home/tk19812/scratch/software/WiggleTools/lib:/mnt/storage/home/tk19812/scratch/software/cactus/lua-5.1.5/lib:/mnt/storage/home/tk19812/scratch/software/lzo-2.10/lib:/mnt/storage/home/tk19812/scratch/software/kyoto/lib:/mnt/storage/home/tk19812/scratch/software/swig-4.0.1/Lib:/mnt/storage/home/tk19812/scratch/software/openssl-1.1.1d/lib:/mnt/storage/home/tk19812/scratch/software/libBigWig/lib:/mnt/storage/home/tk19812/scratch/software/SuiteSparse/lib:/mnt/storage/home/tk19812/scratch/software/libuv-v1.37.0/.libs:/mnt/storage/home/tk19812/scratch/software/nghttp2-1.40.0/lib/.libs:/mnt/storage/home/tk19812/scratch/software/openldap/libraries/liblmdb:/mnt/storage/home/tk19812/scratch/CIRCOS/circos-0.69-9/libgd-2.3.0/lib:/mnt/storage/home/tk19812/scratch/software/jemalloc/lib:/mnt/storage/home/tk19812/.conda/envs/test3k/lib:/mnt/storage/home/tk19812/scratch/software/portcullis/boost_1_67_0/stage/lib:/mnt/storage/software/languages/anaconda/Anaconda3-2020.02-tflow-2.2.0/pkgs/python-3.7.6-h0371630_2/lib
2021-05-10 12:06:29.176604: I tensorflow/stream_executor/cuda/cudart_stub.cc:29] Ignore above cudart dlerror if you do not have a GPU set up on your machine.
start time is 1620644790.7771568
Step1: transfer fasta data to CNN input data
Step2: classify TEs
Step2: 2) domain information is not exist
Traceback (most recent call last):
  File "/mnt/storage/home/tk19812/scratch/software/DeepTE/DeepTE.py", line 274, in <module>
    main()
  File "/mnt/storage/home/tk19812/scratch/software/DeepTE/DeepTE.py", line 249, in main
    pipeline_no_m.classify_pipeline(model_dir, input_CNN_data_file, temp_store_opt_dir, sp_type,te_fam,prop_thr)
  File "/mnt/storage/scratch/tk19812/software/DeepTE/scripts/DeepTE_pipeline_no_modification.py", line 339, in classify_pipeline
    predict_te(model_file_dic[model_name], model_name, x_all_test_list, y_all_test_nm_list,input_spe_type,y_all_test_nm_list,prop_thr)
  File "/mnt/storage/scratch/tk19812/software/DeepTE/scripts/DeepTE_pipeline_no_modification.py", line 210, in predict_te
    model = load_model(model)
  File "/mnt/storage/scratch/tk19812/software/DeepTE/venv/lib/python3.7/site-packages/keras/engine/saving.py", line 419, in load_model
    model = _deserialize_model(f, custom_objects, compile)
  File "/mnt/storage/scratch/tk19812/software/DeepTE/venv/lib/python3.7/site-packages/keras/engine/saving.py", line 225, in _deserialize_model
    model = model_from_config(model_config, custom_objects=custom_objects)
  File "/mnt/storage/scratch/tk19812/software/DeepTE/venv/lib/python3.7/site-packages/keras/engine/saving.py", line 458, in model_from_config
    return deserialize(config, custom_objects=custom_objects)
  File "/mnt/storage/scratch/tk19812/software/DeepTE/venv/lib/python3.7/site-packages/keras/layers/__init__.py", line 55, in deserialize
    printable_module_name='layer')
  File "/mnt/storage/scratch/tk19812/software/DeepTE/venv/lib/python3.7/site-packages/keras/utils/generic_utils.py", line 145, in deserialize_keras_object
    list(custom_objects.items())))
  File "/mnt/storage/scratch/tk19812/software/DeepTE/venv/lib/python3.7/site-packages/keras/engine/sequential.py", line 301, in from_config
    model.add(layer)
  File "/mnt/storage/scratch/tk19812/software/DeepTE/venv/lib/python3.7/site-packages/keras/engine/sequential.py", line 161, in add
    name=layer.name + '_input')
  File "/mnt/storage/scratch/tk19812/software/DeepTE/venv/lib/python3.7/site-packages/keras/engine/input_layer.py", line 178, in Input
    input_tensor=tensor)
  File "/mnt/storage/scratch/tk19812/software/DeepTE/venv/lib/python3.7/site-packages/keras/legacy/interfaces.py", line 91, in wrapper
    return func(*args, **kwargs)
  File "/mnt/storage/scratch/tk19812/software/DeepTE/venv/lib/python3.7/site-packages/keras/engine/input_layer.py", line 87, in __init__
    name=self.name)
  File "/mnt/storage/scratch/tk19812/software/DeepTE/venv/lib/python3.7/site-packages/keras/backend/tensorflow_backend.py", line 517, in placeholder
    x = tf.placeholder(dtype, shape=shape, name=name)
AttributeError: module 'tensorflow' has no attribute 'placeholder'

Cheers F

yanhaidong1 commented 3 years ago

Hi francicco,

it looks like this issue is from version of tensorflow. Would you please check or change your tensorflow version to be 1.14.0? I just tried this version without problem using your command line.

Please let me know whether it works. Thanks.

Best wishes, Haidong

francicco commented 3 years ago

Hi,

pip install tensorflow==1.14.0
Collecting tensorflow==1.14.0
  Using cached tensorflow-1.14.0-cp37-cp37m-manylinux1_x86_64.whl (109.3 MB)
Collecting tensorflow-estimator<1.15.0rc0,>=1.14.0rc0
  Using cached tensorflow_estimator-1.14.0-py2.py3-none-any.whl (488 kB)
Requirement already satisfied: termcolor>=1.1.0 in ./venv/lib/python3.7/site-packages (from tensorflow==1.14.0) (1.1.0)
Requirement already satisfied: astor>=0.6.0 in ./venv/lib/python3.7/site-packages (from tensorflow==1.14.0) (0.8.1)
Requirement already satisfied: google-pasta>=0.1.6 in ./venv/lib/python3.7/site-packages (from tensorflow==1.14.0) (0.2.0)
Requirement already satisfied: protobuf>=3.6.1 in ./venv/lib/python3.7/site-packages (from tensorflow==1.14.0) (3.16.0)
Collecting tensorboard<1.15.0,>=1.14.0
  Using cached tensorboard-1.14.0-py3-none-any.whl (3.1 MB)
Requirement already satisfied: numpy<2.0,>=1.14.5 in ./venv/lib/python3.7/site-packages (from tensorflow==1.14.0) (1.19.5)
Requirement already satisfied: wrapt>=1.11.1 in ./venv/lib/python3.7/site-packages (from tensorflow==1.14.0) (1.12.1)
Requirement already satisfied: wheel>=0.26 in ./venv/lib/python3.7/site-packages (from tensorflow==1.14.0) (0.36.2)
Requirement already satisfied: keras-applications>=1.0.6 in ./venv/lib/python3.7/site-packages (from tensorflow==1.14.0) (1.0.8)
Requirement already satisfied: absl-py>=0.7.0 in ./venv/lib/python3.7/site-packages (from tensorflow==1.14.0) (0.12.0)
Requirement already satisfied: six>=1.10.0 in ./venv/lib/python3.7/site-packages (from tensorflow==1.14.0) (1.15.0)
Requirement already satisfied: grpcio>=1.8.6 in ./venv/lib/python3.7/site-packages (from tensorflow==1.14.0) (1.32.0)
Requirement already satisfied: gast>=0.2.0 in ./venv/lib/python3.7/site-packages (from tensorflow==1.14.0) (0.3.3)
Requirement already satisfied: keras-preprocessing>=1.0.5 in ./venv/lib/python3.7/site-packages (from tensorflow==1.14.0) (1.1.2)
Requirement already satisfied: h5py in ./venv/lib/python3.7/site-packages (from keras-applications>=1.0.6->tensorflow==1.14.0) (2.10.0)
Requirement already satisfied: setuptools>=41.0.0 in ./venv/lib/python3.7/site-packages (from tensorboard<1.15.0,>=1.14.0->tensorflow==1.14.0) (45.3.0)
Requirement already satisfied: werkzeug>=0.11.15 in ./venv/lib/python3.7/site-packages (from tensorboard<1.15.0,>=1.14.0->tensorflow==1.14.0) (1.0.1)
Requirement already satisfied: markdown>=2.6.8 in ./venv/lib/python3.7/site-packages (from tensorboard<1.15.0,>=1.14.0->tensorflow==1.14.0) (3.3.4)
Requirement already satisfied: importlib-metadata in ./venv/lib/python3.7/site-packages (from markdown>=2.6.8->tensorboard<1.15.0,>=1.14.0->tensorflow==1.14.0) (4.0.1)
Requirement already satisfied: typing-extensions>=3.6.4 in ./venv/lib/python3.7/site-packages (from importlib-metadata->markdown>=2.6.8->tensorboard<1.15.0,>=1.14.0->tensorflow==1.14.0) (3.7.4.3)
Requirement already satisfied: zipp>=0.5 in ./venv/lib/python3.7/site-packages (from importlib-metadata->markdown>=2.6.8->tensorboard<1.15.0,>=1.14.0->tensorflow==1.14.0) (3.4.1)
Installing collected packages: tensorflow-estimator, tensorboard, tensorflow
  Attempting uninstall: tensorflow-estimator
    Found existing installation: tensorflow-estimator 2.4.0
    Uninstalling tensorflow-estimator-2.4.0:
      Successfully uninstalled tensorflow-estimator-2.4.0
  Attempting uninstall: tensorboard
    Found existing installation: tensorboard 2.5.0
    Uninstalling tensorboard-2.5.0:
      Successfully uninstalled tensorboard-2.5.0
  Attempting uninstall: tensorflow
    Found existing installation: tensorflow 2.4.1
    Uninstalling tensorflow-2.4.1:
      Successfully uninstalled tensorflow-2.4.1
Successfully installed tensorboard-1.14.0 tensorflow-1.14.0 tensorflow-estimator-1.14.0

But then I got this:

(venv) [tk19812@bc4login2 DeepTE]$ DeepTE.py -d working_dir -o output_dir -i example_data/input_test.fasta -sp P -m_dir working_dir/download_P_model_dir/Plants_model/
Using TensorFlow backend.
Traceback (most recent call last):
  File "/mnt/storage/scratch/tk19812/software/DeepTE/venv/lib/python3.7/site-packages/tensorflow/python/pywrap_tensorflow.py", line 58, in <module>
    from tensorflow.python.pywrap_tensorflow_internal import *
  File "/mnt/storage/scratch/tk19812/software/DeepTE/venv/lib/python3.7/site-packages/tensorflow/python/pywrap_tensorflow_internal.py", line 28, in <module>
    _pywrap_tensorflow_internal = swig_import_helper()
  File "/mnt/storage/scratch/tk19812/software/DeepTE/venv/lib/python3.7/site-packages/tensorflow/python/pywrap_tensorflow_internal.py", line 24, in swig_import_helper
    _mod = imp.load_module('_pywrap_tensorflow_internal', fp, pathname, description)
  File "/mnt/storage/software/languages/anaconda/Anaconda3-2019-3.7/lib/python3.7/imp.py", line 242, in load_module
    return load_dynamic(name, filename, file)
  File "/mnt/storage/software/languages/anaconda/Anaconda3-2019-3.7/lib/python3.7/imp.py", line 342, in load_dynamic
    return _load(spec)
ImportError: /usr/lib64/libm.so.6: version `GLIBC_2.23' not found (required by /mnt/storage/scratch/tk19812/software/DeepTE/venv/lib/python3.7/site-packages/tensorflow/python/_pywrap_tensorflow_internal.so)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/mnt/storage/home/tk19812/scratch/software/DeepTE/DeepTE.py", line 17, in <module>
    from scripts import DeepTE_pipeline_no_modification as pipeline_no_m
  File "/mnt/storage/scratch/tk19812/software/DeepTE/scripts/DeepTE_pipeline_no_modification.py", line 14, in <module>
    from keras.models import load_model
  File "/mnt/storage/scratch/tk19812/software/DeepTE/venv/lib/python3.7/site-packages/keras/__init__.py", line 3, in <module>
    from . import utils
  File "/mnt/storage/scratch/tk19812/software/DeepTE/venv/lib/python3.7/site-packages/keras/utils/__init__.py", line 6, in <module>
    from . import conv_utils
  File "/mnt/storage/scratch/tk19812/software/DeepTE/venv/lib/python3.7/site-packages/keras/utils/conv_utils.py", line 9, in <module>
    from .. import backend as K
  File "/mnt/storage/scratch/tk19812/software/DeepTE/venv/lib/python3.7/site-packages/keras/backend/__init__.py", line 89, in <module>
    from .tensorflow_backend import *
  File "/mnt/storage/scratch/tk19812/software/DeepTE/venv/lib/python3.7/site-packages/keras/backend/tensorflow_backend.py", line 5, in <module>
    import tensorflow as tf
  File "/mnt/storage/scratch/tk19812/software/DeepTE/venv/lib/python3.7/site-packages/tensorflow/__init__.py", line 28, in <module>
    from tensorflow.python import pywrap_tensorflow  # pylint: disable=unused-import
  File "/mnt/storage/scratch/tk19812/software/DeepTE/venv/lib/python3.7/site-packages/tensorflow/python/__init__.py", line 49, in <module>
    from tensorflow.python import pywrap_tensorflow
  File "/mnt/storage/scratch/tk19812/software/DeepTE/venv/lib/python3.7/site-packages/tensorflow/python/pywrap_tensorflow.py", line 74, in <module>
    raise ImportError(msg)
ImportError: Traceback (most recent call last):
  File "/mnt/storage/scratch/tk19812/software/DeepTE/venv/lib/python3.7/site-packages/tensorflow/python/pywrap_tensorflow.py", line 58, in <module>
    from tensorflow.python.pywrap_tensorflow_internal import *
  File "/mnt/storage/scratch/tk19812/software/DeepTE/venv/lib/python3.7/site-packages/tensorflow/python/pywrap_tensorflow_internal.py", line 28, in <module>
    _pywrap_tensorflow_internal = swig_import_helper()
  File "/mnt/storage/scratch/tk19812/software/DeepTE/venv/lib/python3.7/site-packages/tensorflow/python/pywrap_tensorflow_internal.py", line 24, in swig_import_helper
    _mod = imp.load_module('_pywrap_tensorflow_internal', fp, pathname, description)
  File "/mnt/storage/software/languages/anaconda/Anaconda3-2019-3.7/lib/python3.7/imp.py", line 242, in load_module
    return load_dynamic(name, filename, file)
  File "/mnt/storage/software/languages/anaconda/Anaconda3-2019-3.7/lib/python3.7/imp.py", line 342, in load_dynamic
    return _load(spec)
ImportError: /usr/lib64/libm.so.6: version `GLIBC_2.23' not found (required by /mnt/storage/scratch/tk19812/software/DeepTE/venv/lib/python3.7/site-packages/tensorflow/python/_pywrap_tensorflow_internal.so)

Failed to load the native TensorFlow runtime.

See https://www.tensorflow.org/install/errors

for some common reasons and solutions.  Include the entire stack trace
above this error message when asking for help.

Cheers F

songliVT commented 3 years ago

Looks like you need to install GLIBC_2.23.

Song

On Mon, May 10, 2021 at 12:14 PM Francesco Cicconardi < @.***> wrote:

Hi,

pip install tensorflow==1.14.0 Collecting tensorflow==1.14.0 Using cached tensorflow-1.14.0-cp37-cp37m-manylinux1_x86_64.whl (109.3 MB) Collecting tensorflow-estimator<1.15.0rc0,>=1.14.0rc0 Using cached tensorflow_estimator-1.14.0-py2.py3-none-any.whl (488 kB) Requirement already satisfied: termcolor>=1.1.0 in ./venv/lib/python3.7/site-packages (from tensorflow==1.14.0) (1.1.0) Requirement already satisfied: astor>=0.6.0 in ./venv/lib/python3.7/site-packages (from tensorflow==1.14.0) (0.8.1) Requirement already satisfied: google-pasta>=0.1.6 in ./venv/lib/python3.7/site-packages (from tensorflow==1.14.0) (0.2.0) Requirement already satisfied: protobuf>=3.6.1 in ./venv/lib/python3.7/site-packages (from tensorflow==1.14.0) (3.16.0) Collecting tensorboard<1.15.0,>=1.14.0 Using cached tensorboard-1.14.0-py3-none-any.whl (3.1 MB) Requirement already satisfied: numpy<2.0,>=1.14.5 in ./venv/lib/python3.7/site-packages (from tensorflow==1.14.0) (1.19.5) Requirement already satisfied: wrapt>=1.11.1 in ./venv/lib/python3.7/site-packages (from tensorflow==1.14.0) (1.12.1) Requirement already satisfied: wheel>=0.26 in ./venv/lib/python3.7/site-packages (from tensorflow==1.14.0) (0.36.2) Requirement already satisfied: keras-applications>=1.0.6 in ./venv/lib/python3.7/site-packages (from tensorflow==1.14.0) (1.0.8) Requirement already satisfied: absl-py>=0.7.0 in ./venv/lib/python3.7/site-packages (from tensorflow==1.14.0) (0.12.0) Requirement already satisfied: six>=1.10.0 in ./venv/lib/python3.7/site-packages (from tensorflow==1.14.0) (1.15.0) Requirement already satisfied: grpcio>=1.8.6 in ./venv/lib/python3.7/site-packages (from tensorflow==1.14.0) (1.32.0) Requirement already satisfied: gast>=0.2.0 in ./venv/lib/python3.7/site-packages (from tensorflow==1.14.0) (0.3.3) Requirement already satisfied: keras-preprocessing>=1.0.5 in ./venv/lib/python3.7/site-packages (from tensorflow==1.14.0) (1.1.2) Requirement already satisfied: h5py in ./venv/lib/python3.7/site-packages (from keras-applications>=1.0.6->tensorflow==1.14.0) (2.10.0) Requirement already satisfied: setuptools>=41.0.0 in ./venv/lib/python3.7/site-packages (from tensorboard<1.15.0,>=1.14.0->tensorflow==1.14.0) (45.3.0) Requirement already satisfied: werkzeug>=0.11.15 in ./venv/lib/python3.7/site-packages (from tensorboard<1.15.0,>=1.14.0->tensorflow==1.14.0) (1.0.1) Requirement already satisfied: markdown>=2.6.8 in ./venv/lib/python3.7/site-packages (from tensorboard<1.15.0,>=1.14.0->tensorflow==1.14.0) (3.3.4) Requirement already satisfied: importlib-metadata in ./venv/lib/python3.7/site-packages (from markdown>=2.6.8->tensorboard<1.15.0,>=1.14.0->tensorflow==1.14.0) (4.0.1) Requirement already satisfied: typing-extensions>=3.6.4 in ./venv/lib/python3.7/site-packages (from importlib-metadata->markdown>=2.6.8->tensorboard<1.15.0,>=1.14.0->tensorflow==1.14.0) (3.7.4.3) Requirement already satisfied: zipp>=0.5 in ./venv/lib/python3.7/site-packages (from importlib-metadata->markdown>=2.6.8->tensorboard<1.15.0,>=1.14.0->tensorflow==1.14.0) (3.4.1) Installing collected packages: tensorflow-estimator, tensorboard, tensorflow Attempting uninstall: tensorflow-estimator Found existing installation: tensorflow-estimator 2.4.0 Uninstalling tensorflow-estimator-2.4.0: Successfully uninstalled tensorflow-estimator-2.4.0 Attempting uninstall: tensorboard Found existing installation: tensorboard 2.5.0 Uninstalling tensorboard-2.5.0: Successfully uninstalled tensorboard-2.5.0 Attempting uninstall: tensorflow Found existing installation: tensorflow 2.4.1 Uninstalling tensorflow-2.4.1: Successfully uninstalled tensorflow-2.4.1 Successfully installed tensorboard-1.14.0 tensorflow-1.14.0 tensorflow-estimator-1.14.0

But then I got this:

(venv) @.** DeepTE]$ DeepTE.py -d working_dir -o output_dir -i example_data/input_test.fasta -sp P -m_dir working_dir/download_P_model_dir/Plants_model/ Using TensorFlow backend. Traceback (most recent call last): File "/mnt/storage/scratch/tk19812/software/DeepTE/venv/lib/python3.7/site-packages/tensorflow/python/pywrap_tensorflow.py", line 58, in from tensorflow.python.pywrap_tensorflow_internal import File "/mnt/storage/scratch/tk19812/software/DeepTE/venv/lib/python3.7/site-packages/tensorflow/python/pywrap_tensorflow_internal.py", line 28, in _pywrap_tensorflow_internal = swig_import_helper() File "/mnt/storage/scratch/tk19812/software/DeepTE/venv/lib/python3.7/site-packages/tensorflow/python/pywrap_tensorflow_internal.py", line 24, in swig_import_helper _mod = imp.load_module('_pywrap_tensorflow_internal', fp, pathname, description) File "/mnt/storage/software/languages/anaconda/Anaconda3-2019-3.7/lib/python3.7/imp.py", line 242, in load_module return load_dynamic(name, filename, file) File "/mnt/storage/software/languages/anaconda/Anaconda3-2019-3.7/lib/python3.7/imp.py", line 342, in load_dynamic return _load(spec) ImportError: /usr/lib64/libm.so.6: version `GLIBC_2.23' not found (required by /mnt/storage/scratch/tk19812/software/DeepTE/venv/lib/python3.7/site-packages/tensorflow/python/_pywrap_tensorflow_internal.so)

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "/mnt/storage/home/tk19812/scratch/software/DeepTE/DeepTE.py", line 17, in from scripts import DeepTE_pipeline_no_modification as pipeline_no_m File "/mnt/storage/scratch/tk19812/software/DeepTE/scripts/DeepTE_pipeline_no_modification.py", line 14, in from keras.models import load_model File "/mnt/storage/scratch/tk19812/software/DeepTE/venv/lib/python3.7/site-packages/keras/init.py", line 3, in from . import utils File "/mnt/storage/scratch/tk19812/software/DeepTE/venv/lib/python3.7/site-packages/keras/utils/init.py", line 6, in from . import conv_utils File "/mnt/storage/scratch/tk19812/software/DeepTE/venv/lib/python3.7/site-packages/keras/utils/conv_utils.py", line 9, in from .. import backend as K File "/mnt/storage/scratch/tk19812/software/DeepTE/venv/lib/python3.7/site-packages/keras/backend/init.py", line 89, in from .tensorflow_backend import File "/mnt/storage/scratch/tk19812/software/DeepTE/venv/lib/python3.7/site-packages/keras/backend/tensorflow_backend.py", line 5, in import tensorflow as tf File "/mnt/storage/scratch/tk19812/software/DeepTE/venv/lib/python3.7/site-packages/tensorflow/init.py", line 28, in from tensorflow.python import pywrap_tensorflow # pylint: disable=unused-import File "/mnt/storage/scratch/tk19812/software/DeepTE/venv/lib/python3.7/site-packages/tensorflow/python/init.py", line 49, in from tensorflow.python import pywrap_tensorflow File "/mnt/storage/scratch/tk19812/software/DeepTE/venv/lib/python3.7/site-packages/tensorflow/python/pywrap_tensorflow.py", line 74, in raise ImportError(msg) ImportError: Traceback (most recent call last): File "/mnt/storage/scratch/tk19812/software/DeepTE/venv/lib/python3.7/site-packages/tensorflow/python/pywrap_tensorflow.py", line 58, in from tensorflow.python.pywrap_tensorflow_internal import File "/mnt/storage/scratch/tk19812/software/DeepTE/venv/lib/python3.7/site-packages/tensorflow/python/pywrap_tensorflow_internal.py", line 28, in _pywrap_tensorflow_internal = swig_import_helper() File "/mnt/storage/scratch/tk19812/software/DeepTE/venv/lib/python3.7/site-packages/tensorflow/python/pywrap_tensorflow_internal.py", line 24, in swig_import_helper _mod = imp.load_module('_pywrap_tensorflow_internal', fp, pathname, description) File "/mnt/storage/software/languages/anaconda/Anaconda3-2019-3.7/lib/python3.7/imp.py", line 242, in load_module return load_dynamic(name, filename, file) File "/mnt/storage/software/languages/anaconda/Anaconda3-2019-3.7/lib/python3.7/imp.py", line 342, in load_dynamic return _load(spec) ImportError: /usr/lib64/libm.so.6: version `GLIBC_2.23' not found (required by /mnt/storage/scratch/tk19812/software/DeepTE/venv/lib/python3.7/site-packages/tensorflow/python/_pywrap_tensorflow_internal.so)

Failed to load the native TensorFlow runtime.

See https://www.tensorflow.org/install/errors

for some common reasons and solutions. Include the entire stack trace above this error message when asking for help.

Cheers F

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/LiLabAtVT/DeepTE/issues/9#issuecomment-836903585, or unsubscribe https://github.com/notifications/unsubscribe-auth/ACEEENQZO7NRSKVSHC5TQWDTNAA6PANCNFSM44Q5GLMA .

-- Associate Professor in Plant Genomics and Bioinformatics School of Plant and Environmental Sciences Virginia Polytechnic Institute and State University

zoom https://virginiatech.zoom.us/j/8790572835

yanhaidong1 commented 3 years ago

Hi F, Alternative solution is to use conda to build an environment like the following:

Use conda to install required packages (Recommend) Install conda: https://www.anaconda.com/products/individual conda create -n py36 python=3.6 conda activate py36 conda install tensorflow-gpu=1.14.0 conda install biopython conda install keras=2.2.4 conda install numpy=1.16.0

this problem is alway occurred with the version of python.

Best wishes, Haidong

yanhaidong1 commented 3 years ago

Also, please check the tensorflow issue here: https://github.com/tensorflow/tensorflow/issues/28660

francicco commented 3 years ago

Reinstalled with python2.6:

(venv) [tk19812@bc4login1 DeepTE]$ pip install biopython==1.72 keras==2.2.4 tensorflow==1.14.0 numpy==1.16.0
Collecting biopython==1.72
  Using cached biopython-1.72-cp36-cp36m-manylinux1_x86_64.whl (2.1 MB)
Collecting keras==2.2.4
  Using cached Keras-2.2.4-py2.py3-none-any.whl (312 kB)
Requirement already satisfied: tensorflow==1.14.0 in ./venv/lib/python3.6/site-packages (1.14.0)
Collecting numpy==1.16.0
  Using cached numpy-1.16.0-cp36-cp36m-manylinux1_x86_64.whl (17.3 MB)
Requirement already satisfied: h5py in ./venv/lib/python3.6/site-packages (from keras==2.2.4) (3.1.0)
Requirement already satisfied: keras-preprocessing>=1.0.5 in ./venv/lib/python3.6/site-packages (from keras==2.2.4) (1.1.2)
Requirement already satisfied: pyyaml in ./venv/lib/python3.6/site-packages (from keras==2.2.4) (5.4.1)
Requirement already satisfied: scipy>=0.14 in ./venv/lib/python3.6/site-packages (from keras==2.2.4) (1.5.4)
Requirement already satisfied: six>=1.9.0 in ./venv/lib/python3.6/site-packages (from keras==2.2.4) (1.16.0)
Requirement already satisfied: keras-applications>=1.0.6 in ./venv/lib/python3.6/site-packages (from keras==2.2.4) (1.0.8)
Requirement already satisfied: tensorboard<1.15.0,>=1.14.0 in ./venv/lib/python3.6/site-packages (from tensorflow==1.14.0) (1.14.0)
Requirement already satisfied: termcolor>=1.1.0 in ./venv/lib/python3.6/site-packages (from tensorflow==1.14.0) (1.1.0)
Requirement already satisfied: gast>=0.2.0 in ./venv/lib/python3.6/site-packages (from tensorflow==1.14.0) (0.4.0)
Requirement already satisfied: wrapt>=1.11.1 in ./venv/lib/python3.6/site-packages (from tensorflow==1.14.0) (1.12.1)
Requirement already satisfied: absl-py>=0.7.0 in ./venv/lib/python3.6/site-packages (from tensorflow==1.14.0) (0.12.0)
Requirement already satisfied: wheel>=0.26 in ./venv/lib/python3.6/site-packages (from tensorflow==1.14.0) (0.34.2)
Requirement already satisfied: protobuf>=3.6.1 in ./venv/lib/python3.6/site-packages (from tensorflow==1.14.0) (3.16.0)
Requirement already satisfied: tensorflow-estimator<1.15.0rc0,>=1.14.0rc0 in ./venv/lib/python3.6/site-packages (from tensorflow==1.14.0) (1.14.0)
Requirement already satisfied: google-pasta>=0.1.6 in ./venv/lib/python3.6/site-packages (from tensorflow==1.14.0) (0.2.0)
Requirement already satisfied: astor>=0.6.0 in ./venv/lib/python3.6/site-packages (from tensorflow==1.14.0) (0.8.1)
Requirement already satisfied: grpcio>=1.8.6 in ./venv/lib/python3.6/site-packages (from tensorflow==1.14.0) (1.37.1)
Requirement already satisfied: markdown>=2.6.8 in ./venv/lib/python3.6/site-packages (from tensorboard<1.15.0,>=1.14.0->tensorflow==1.14.0) (3.3.4)
Requirement already satisfied: setuptools>=41.0.0 in ./venv/lib/python3.6/site-packages (from tensorboard<1.15.0,>=1.14.0->tensorflow==1.14.0) (45.3.0)
Requirement already satisfied: werkzeug>=0.11.15 in ./venv/lib/python3.6/site-packages (from tensorboard<1.15.0,>=1.14.0->tensorflow==1.14.0) (1.0.1)
Requirement already satisfied: importlib-metadata in ./venv/lib/python3.6/site-packages (from markdown>=2.6.8->tensorboard<1.15.0,>=1.14.0->tensorflow==1.14.0) (4.0.1)
Requirement already satisfied: cached-property in ./venv/lib/python3.6/site-packages (from h5py->keras==2.2.4) (1.5.2)
Requirement already satisfied: typing-extensions>=3.6.4 in ./venv/lib/python3.6/site-packages (from importlib-metadata->markdown>=2.6.8->tensorboard<1.15.0,>=1.14.0->tensorflow==1.14.0) (3.10.0.0)
Requirement already satisfied: zipp>=0.5 in ./venv/lib/python3.6/site-packages (from importlib-metadata->markdown>=2.6.8->tensorboard<1.15.0,>=1.14.0->tensorflow==1.14.0) (3.4.1)
Installing collected packages: numpy, keras, biopython
  Attempting uninstall: numpy
    Found existing installation: numpy 1.19.5
    Uninstalling numpy-1.19.5:
      Successfully uninstalled numpy-1.19.5
  Attempting uninstall: keras
    Found existing installation: Keras 2.4.3
    Uninstalling Keras-2.4.3:
      Successfully uninstalled Keras-2.4.3
  Attempting uninstall: biopython
    Found existing installation: biopython 1.78
    Uninstalling biopython-1.78:
      Successfully uninstalled biopython-1.78
Successfully installed biopython-1.72 keras-2.2.4 numpy-1.16.0

But still something wrong:

(venv) [tk19812@bc4login1 DeepTE]$ DeepTE.py -d working_dir -o output_dir -i example_data/input_test.fasta -sp P -m_dir working_dir/download_P_model_dir/Plants_model/
Using TensorFlow backend.
start time is 1620664997.4408085
Step1: transfer fasta data to CNN input data
Step2: classify TEs
Step2: 2) domain information is not exist
Traceback (most recent call last):
  File "/mnt/storage/home/tk19812/scratch/software/DeepTE/DeepTE.py", line 274, in <module>
    main()
  File "/mnt/storage/home/tk19812/scratch/software/DeepTE/DeepTE.py", line 249, in main
    pipeline_no_m.classify_pipeline(model_dir, input_CNN_data_file, temp_store_opt_dir, sp_type,te_fam,prop_thr)
  File "/mnt/storage/scratch/tk19812/software/DeepTE/scripts/DeepTE_pipeline_no_modification.py", line 339, in classify_pipeline
    predict_te(model_file_dic[model_name], model_name, x_all_test_list, y_all_test_nm_list,input_spe_type,y_all_test_nm_list,prop_thr)
  File "/mnt/storage/scratch/tk19812/software/DeepTE/scripts/DeepTE_pipeline_no_modification.py", line 210, in predict_te
    model = load_model(model)
  File "/mnt/storage/scratch/tk19812/software/DeepTE/venv/lib/python3.6/site-packages/keras/engine/saving.py", line 419, in load_model
    model = _deserialize_model(f, custom_objects, compile)
  File "/mnt/storage/scratch/tk19812/software/DeepTE/venv/lib/python3.6/site-packages/keras/engine/saving.py", line 224, in _deserialize_model
    model_config = json.loads(model_config.decode('utf-8'))
AttributeError: 'str' object has no attribute 'decode'
francicco commented 3 years ago

If I delete the working dir I get this instead:

Using TensorFlow backend.
start time is 1620665412.8375952
Step1: transfer fasta data to CNN input data
Step2: classify TEs
Step2: 2) domain information is not exist
Traceback (most recent call last):
  File "/mnt/storage/home/tk19812/scratch/software/DeepTE/DeepTE.py", line 274, in <module>
    main()
  File "/mnt/storage/home/tk19812/scratch/software/DeepTE/DeepTE.py", line 249, in main
    pipeline_no_m.classify_pipeline(model_dir, input_CNN_data_file, temp_store_opt_dir, sp_type,te_fam,prop_thr)
  File "/mnt/storage/scratch/tk19812/software/DeepTE/scripts/DeepTE_pipeline_no_modification.py", line 339, in classify_pipeline
    predict_te(model_file_dic[model_name], model_name, x_all_test_list, y_all_test_nm_list,input_spe_type,y_all_test_nm_list,prop_thr)
KeyError: 'All'

Nothing changes if I install it with conda F

yanhaidong1 commented 3 years ago

Hi F,

Use python3.6 instead of 2.6. Also, let's try this:

  1. 'mkdir test' under the DeepTE folder.
  2. clean your working_dir and output_dir
  3. cd test
  4. mkdir working_dir output_dir
  5. python ../DeepTE.py -o output_dir/ -i ../example_data/input_test.fasta -sp P -m_dir ../collect_model_P/Plants_model/

keep in mind, Plants_model dir contains model files (*.h5).

Best wishes, haidong

francicco commented 3 years ago 
[tk19812@bc4login1 DeepTE]$ virtualenv -p python3.6 venv
created virtual environment CPython3.6.5.final.0-64 in 331ms
  creator CPython3Posix(dest=/mnt/storage/scratch/tk19812/software/DeepTE/venv, clear=False, global=False)
  seeder FromAppData(download=False, pip=latest, setuptools=latest, wheel=latest, via=copy, app_data_dir=/mnt/storage/home/tk19812/.local/share/virtualenv/seed-app-data/v1)
  activators PythonActivator,FishActivator,XonshActivator,CShellActivator,PowerShellActivator,BashActivator
[tk19812@bc4login1 DeepTE]$ pip install pip --upgrade
WARNING: pip is being invoked by an old script wrapper. This will fail in a future version of pip.
Please see https://github.com/pypa/pip/issues/5599 for advice on fixing the underlying issue.
To avoid this problem you can invoke Python with '-m pip' instead of running pip directly.
Defaulting to user installation because normal site-packages is not writeable
Requirement already satisfied: pip in /mnt/storage/home/tk19812/.local/lib/python3.6/site-packages (21.1.1)
[tk19812@bc4login1 DeepTE]$ source venv/bin/activate
(venv) [tk19812@bc4login1 DeepTE]$ pip install biopython==1.72 keras==2.2.4 tensorflow==1.14.0 numpy==1.16.0
Collecting biopython==1.72
  Using cached biopython-1.72-cp36-cp36m-manylinux1_x86_64.whl (2.1 MB)
Collecting keras==2.2.4
  Using cached Keras-2.2.4-py2.py3-none-any.whl (312 kB)
Collecting tensorflow==1.14.0
  Using cached tensorflow-1.14.0-cp36-cp36m-manylinux1_x86_64.whl (109.2 MB)
Collecting numpy==1.16.0
  Using cached numpy-1.16.0-cp36-cp36m-manylinux1_x86_64.whl (17.3 MB)
Collecting h5py
  Using cached h5py-3.1.0-cp36-cp36m-manylinux1_x86_64.whl (4.0 MB)
Collecting scipy>=0.14
  Using cached scipy-1.5.4-cp36-cp36m-manylinux1_x86_64.whl (25.9 MB)
Collecting keras-applications>=1.0.6
  Using cached Keras_Applications-1.0.8-py3-none-any.whl (50 kB)
Collecting keras-preprocessing>=1.0.5
  Using cached Keras_Preprocessing-1.1.2-py2.py3-none-any.whl (42 kB)
Collecting pyyaml
  Using cached PyYAML-5.4.1-cp36-cp36m-manylinux1_x86_64.whl (640 kB)
Collecting six>=1.9.0
  Using cached six-1.16.0-py2.py3-none-any.whl (11 kB)
Processing /mnt/storage/home/tk19812/.cache/pip/wheels/32/42/7f/23cae9ff6ef66798d00dc5d659088e57dbba01566f6c60db63/wrapt-1.12.1-cp36-cp36m-linux_x86_64.whl
Collecting google-pasta>=0.1.6
  Using cached google_pasta-0.2.0-py3-none-any.whl (57 kB)
Collecting absl-py>=0.7.0
  Using cached absl_py-0.12.0-py3-none-any.whl (129 kB)
Collecting tensorflow-estimator<1.15.0rc0,>=1.14.0rc0
  Using cached tensorflow_estimator-1.14.0-py2.py3-none-any.whl (488 kB)
Collecting tensorboard<1.15.0,>=1.14.0
  Using cached tensorboard-1.14.0-py3-none-any.whl (3.1 MB)
Collecting astor>=0.6.0
  Using cached astor-0.8.1-py2.py3-none-any.whl (27 kB)
Processing /mnt/storage/home/tk19812/.cache/pip/wheels/93/2a/eb/e58dbcbc963549ee4f065ff80a59f274cc7210b6eab962acdc/termcolor-1.1.0-py3-none-any.whl
Collecting grpcio>=1.8.6
  Using cached grpcio-1.37.1-cp36-cp36m-manylinux2014_x86_64.whl (4.2 MB)
Collecting protobuf>=3.6.1
  Using cached protobuf-3.16.0-cp36-cp36m-manylinux_2_5_x86_64.manylinux1_x86_64.whl (1.0 MB)
Collecting gast>=0.2.0
  Using cached gast-0.4.0-py3-none-any.whl (9.8 kB)
Requirement already satisfied: wheel>=0.26 in ./venv/lib/python3.6/site-packages (from tensorflow==1.14.0) (0.34.2)
Collecting cached-property; python_version < "3.8"
  Using cached cached_property-1.5.2-py2.py3-none-any.whl (7.6 kB)
Collecting markdown>=2.6.8
  Using cached Markdown-3.3.4-py3-none-any.whl (97 kB)
Requirement already satisfied: setuptools>=41.0.0 in ./venv/lib/python3.6/site-packages (from tensorboard<1.15.0,>=1.14.0->tensorflow==1.14.0) (45.3.0)
Collecting werkzeug>=0.11.15
  Using cached Werkzeug-1.0.1-py2.py3-none-any.whl (298 kB)
Collecting importlib-metadata; python_version < "3.8"
  Using cached importlib_metadata-4.0.1-py3-none-any.whl (16 kB)
Collecting typing-extensions>=3.6.4; python_version < "3.8"
  Using cached typing_extensions-3.10.0.0-py3-none-any.whl (26 kB)
Collecting zipp>=0.5
  Using cached zipp-3.4.1-py3-none-any.whl (5.2 kB)
Installing collected packages: numpy, biopython, cached-property, h5py, scipy, keras-applications, six, keras-preprocessing, pyyaml, keras, wrapt, google-pasta, absl-py, tensorflow-estimator, protobuf, typing-extensions, zipp, importlib-metadata, markdown, werkzeug, grpcio, tensorboard, astor, termcolor, gast, tensorflow
Successfully installed absl-py-0.12.0 astor-0.8.1 biopython-1.72 cached-property-1.5.2 gast-0.4.0 google-pasta-0.2.0 grpcio-1.37.1 h5py-3.1.0 importlib-metadata-4.0.1 keras-2.2.4 keras-applications-1.0.8 keras-preprocessing-1.1.2 markdown-3.3.4 numpy-1.16.0 protobuf-3.16.0 pyyaml-5.4.1 scipy-1.5.4 six-1.16.0 tensorboard-1.14.0 tensorflow-1.14.0 tensorflow-estimator-1.14.0 termcolor-1.1.0 typing-extensions-3.10.0.0 werkzeug-1.0.1 wrapt-1.12.1 zipp-3.4.1

Still:

(venv) [tk19812@bc4login1 DeepTE]$ mkdir test
(venv) [tk19812@bc4login1 DeepTE]$ cd test
(venv) [tk19812@bc4login1 test]$ mkdir working_dir output_dir
(venv) [tk19812@bc4login1 test]$ python ../DeepTE.py -o output_dir/ -i ../example_data/input_test.fasta -sp P -m_dir P                               
Using TensorFlow backend.
start time is 1620666933.927086
Step1: transfer fasta data to CNN input data
Step2: classify TEs
Step2: 2) domain information is not exist
Traceback (most recent call last):
  File "../DeepTE.py", line 274, in <module>
    main()
  File "../DeepTE.py", line 249, in main
    pipeline_no_m.classify_pipeline(model_dir, input_CNN_data_file, temp_store_opt_dir, sp_type,te_fam,prop_thr)
  File "/mnt/storage/scratch/tk19812/software/DeepTE/scripts/DeepTE_pipeline_no_modification.py", line 339, in classify_pipeline
    predict_te(model_file_dic[model_name], model_name, x_all_test_list, y_all_test_nm_list,input_spe_type,y_all_test_nm_list,prop_thr)
KeyError: 'All'

F

yanhaidong1 commented 3 years ago

Hi F,

'-m_dir P': The P is a directory that should contain all the model files including All_model.h5, ClassII_model.h5, ClassI_model.h5, Domain_model.h5, LINE_model.h5, LTR_model.h5, SINE_model.h5, nLTR_model.h5. The KeyError: 'All' means there is something wrong with the -m_dir P. Please make sure the P directory contains these h5 format model files.

Best wishes, Haidong

francicco commented 3 years ago 
(venv) [tk19812@bc4login1 test]$ ls -l Plants_model
total 5539840
-rw-r--r-- 1 tk19812 EBaB_Lab 708916184 Dec  7  2019 All_model.h5
-rw-r--r-- 1 tk19812 EBaB_Lab 708920360 Mar 30  2020 ClassII_model.h5
-rw-r--r-- 1 tk19812 EBaB_Lab 708915352 Dec 13  2019 ClassI_model.h5
-rw-r--r-- 1 tk19812 EBaB_Lab 708915272 Dec 13  2019 Domain_model.h5
-rw-r--r-- 1 tk19812 EBaB_Lab 708915384 Dec 13  2019 LINE_model.h5
-rw-r--r-- 1 tk19812 EBaB_Lab 708915352 Dec  7  2019 LTR_model.h5
-rw-r--r-- 1 tk19812 EBaB_Lab 708915392 Dec  7  2019 SINE_model.h5
-rw-r--r-- 1 tk19812 EBaB_Lab 708917288 Dec  7  2019 nLTR_model.h5
(venv) [tk19812@bc4login1 test]$ python3.6 ../DeepTE.py -o output_dir/ -i ../example_data/input_test.fasta -sp P -m_dir Plants_model/
Using TensorFlow backend.
start time is 1620667866.232269
Step1: transfer fasta data to CNN input data
Step2: classify TEs
Step2: 2) domain information is not exist
Traceback (most recent call last):
  File "../DeepTE.py", line 274, in <module>
    main()
  File "../DeepTE.py", line 249, in main
    pipeline_no_m.classify_pipeline(model_dir, input_CNN_data_file, temp_store_opt_dir, sp_type,te_fam,prop_thr)
  File "/mnt/storage/scratch/tk19812/software/DeepTE/scripts/DeepTE_pipeline_no_modification.py", line 339, in classify_pipeline
    predict_te(model_file_dic[model_name], model_name, x_all_test_list, y_all_test_nm_list,input_spe_type,y_all_test_nm_list,prop_thr)
  File "/mnt/storage/scratch/tk19812/software/DeepTE/scripts/DeepTE_pipeline_no_modification.py", line 210, in predict_te
    model = load_model(model)
  File "/mnt/storage/scratch/tk19812/software/DeepTE/venv/lib/python3.6/site-packages/keras/engine/saving.py", line 419, in load_model
    model = _deserialize_model(f, custom_objects, compile)
  File "/mnt/storage/scratch/tk19812/software/DeepTE/venv/lib/python3.6/site-packages/keras/engine/saving.py", line 224, in _deserialize_model
    model_config = json.loads(model_config.decode('utf-8'))
AttributeError: 'str' object has no attribute 'decode'
yanhaidong1 commented 3 years ago

Hi F,

See this https://github.com/scikit-optimize/scikit-optimize/issues/981. 'scipy' version 1.5.2 or 1.5.3 could fix this issue.

Best wishes, Haidong

francicco commented 3 years ago

Nothing!

(venv) [tk19812@bc4login1 DeepTE]$ pip install scipy==1.5.2
Collecting scipy==1.5.2
  Downloading scipy-1.5.2-cp36-cp36m-manylinux1_x86_64.whl (25.9 MB)
     |################################| 25.9 MB 11.8 MB/s 
Requirement already satisfied: numpy>=1.14.5 in ./venv/lib/python3.6/site-packages (from scipy==1.5.2) (1.16.0)
Installing collected packages: scipy
  Attempting uninstall: scipy
    Found existing installation: scipy 1.5.3
    Uninstalling scipy-1.5.3:
      Successfully uninstalled scipy-1.5.3
Successfully installed scipy-1.5.2
(venv) [tk19812@bc4login1 test]$ python3.6 ../DeepTE.py -o output_dir/ -i ../example_data/input_test.fasta -sp P -m_dir Plants_model/
Using TensorFlow backend.
start time is 1620673576.4417727
Step1: transfer fasta data to CNN input data
Step2: classify TEs
Step2: 2) domain information is not exist
Traceback (most recent call last):
  File "../DeepTE.py", line 274, in <module>
    main()
  File "../DeepTE.py", line 249, in main
    pipeline_no_m.classify_pipeline(model_dir, input_CNN_data_file, temp_store_opt_dir, sp_type,te_fam,prop_thr)
  File "/mnt/storage/scratch/tk19812/software/DeepTE/scripts/DeepTE_pipeline_no_modification.py", line 339, in classify_pipeline
    predict_te(model_file_dic[model_name], model_name, x_all_test_list, y_all_test_nm_list,input_spe_type,y_all_test_nm_list,prop_thr)
  File "/mnt/storage/scratch/tk19812/software/DeepTE/scripts/DeepTE_pipeline_no_modification.py", line 210, in predict_te
    model = load_model(model)
  File "/mnt/storage/scratch/tk19812/software/DeepTE/venv/lib/python3.6/site-packages/keras/engine/saving.py", line 419, in load_model
    model = _deserialize_model(f, custom_objects, compile)
  File "/mnt/storage/scratch/tk19812/software/DeepTE/venv/lib/python3.6/site-packages/keras/engine/saving.py", line 224, in _deserialize_model
    model_config = json.loads(model_config.decode('utf-8'))
AttributeError: 'str' object has no attribute 'decode'
yanhaidong1 commented 3 years ago

Hi F,

I am wondering it is a problem from loading model step (' File "/mnt/storage/scratch/tk19812/software/DeepTE/scripts/DeepTE_pipeline_no_modification.py", line 210, in predict_te model = load_model(model)') using the keras.

Can you check more about your h5py package version? Mine is 2.9.0. Some people said h5py==2.1.0 would be okay (https://stackoverflow.com/questions/53740577/does-any-one-got-attributeerror-str-object-has-no-attribute-decode-whi).

Best wishes, Haidong

francicco commented 3 years ago

I installed the 2.9.0 and I got this:

(venv) [tk19812@bc4login1 test]$ DeepTE.py -d working_dir -o output_dir -i input_test.fasta -sp P -m_dir Plants_model/ Using TensorFlow backend. /mnt/storage/scratch/tk19812/software/DeepTE/venv/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:516: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'. _np_qint8 = np.dtype([("qint8", np.int8, 1)]) /mnt/storage/scratch/tk19812/software/DeepTE/venv/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:517: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'. _np_quint8 = np.dtype([("quint8", np.uint8, 1)]) /mnt/storage/scratch/tk19812/software/DeepTE/venv/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:518: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'. _np_qint16 = np.dtype([("qint16", np.int16, 1)]) /mnt/storage/scratch/tk19812/software/DeepTE/venv/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:519: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'. _np_quint16 = np.dtype([("quint16", np.uint16, 1)]) /mnt/storage/scratch/tk19812/software/DeepTE/venv/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:520: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'. _np_qint32 = np.dtype([("qint32", np.int32, 1)]) /mnt/storage/scratch/tk19812/software/DeepTE/venv/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:525: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'. np_resource = np.dtype([("resource", np.ubyte, 1)]) /mnt/storage/scratch/tk19812/software/DeepTE/venv/lib/python3.6/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:541: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'. _np_qint8 = np.dtype([("qint8", np.int8, 1)]) /mnt/storage/scratch/tk19812/software/DeepTE/venv/lib/python3.6/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:542: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'. _np_quint8 = np.dtype([("quint8", np.uint8, 1)]) /mnt/storage/scratch/tk19812/software/DeepTE/venv/lib/python3.6/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:543: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'. _np_qint16 = np.dtype([("qint16", np.int16, 1)]) /mnt/storage/scratch/tk19812/software/DeepTE/venv/lib/python3.6/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:544: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'. _np_quint16 = np.dtype([("quint16", np.uint16, 1)]) /mnt/storage/scratch/tk19812/software/DeepTE/venv/lib/python3.6/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:545: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'. _np_qint32 = np.dtype([("qint32", np.int32, 1)]) /mnt/storage/scratch/tk19812/software/DeepTE/venv/lib/python3.6/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:550: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'. np_resource = np.dtype([("resource", np.ubyte, 1)]) start time is 1620677616.1457555 There was an error opening the file! --- 0.0009019374847412109 seconds ---

francicco commented 3 years ago

I also tried to install it on another cluster, unfortunately there I don't have python3.6, there I get this:

Step1: transfer fasta data to CNN input data
Step2: classify TEs
Step2: 2) domain information is not exist
2021-05-10 21:09:08.717845: I tensorflow/compiler/jit/xla_cpu_device.cc:41] Not creating XLA devices, tf_xla_enable_xla_devices not set
2021-05-10 21:09:08.721516: W tensorflow/stream_executor/platform/default/dso_loader.cc:60] Could not load dynamic library 'libcuda.so.1'; dlerror: libcuda.so.1: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /sw/lang/cuda_11.2.2/lib:/sw/lang/cuda_11.2.2/lib64:/sw/lib/cudnn-11.2/cuda/lib64:/sw/lang/gcc-9.1.0/lib64:/sw/lang/gcc-9.1.0/lib:/home/tk19812/software/lua-5.3.5/lib:/home/tk19812/software/lzo-2.10/include:/home/tk19812/software/bedtools2/src/utils/BamTools/include:/home/tk19812/software/boost_1_72_0/stage/lib:/home/tk19812/software/boost_1_72_0:/home/tk19812/software/jemalloc/lib
2021-05-10 21:09:08.721558: W tensorflow/stream_executor/cuda/cuda_driver.cc:326] failed call to cuInit: UNKNOWN ERROR (303)
2021-05-10 21:09:08.721635: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:156] kernel driver does not appear to be running on this host (bp1-login01.data.bp.acrc.priv): /proc/driver/nvidia/version does not exist
2021-05-10 21:09:08.722075: I tensorflow/core/platform/cpu_feature_guard.cc:142] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 AVX512F FMA
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
2021-05-10 21:09:08.726089: I tensorflow/compiler/jit/xla_gpu_device.cc:99] Not creating XLA devices, tf_xla_enable_xla_devices not set
2021-05-10 21:09:19.703481: I tensorflow/compiler/mlir/mlir_graph_optimization_pass.cc:116] None of the MLIR optimization passes are enabled (registered 2)
2021-05-10 21:09:19.704120: I tensorflow/core/platform/profile_utils/cpu_utils.cc:112] CPU Frequency: 2600000000 Hz
WARNING:tensorflow:5 out of the last 30 calls to <function Model.make_predict_function.<locals>.predict_function at 0x7f78ac783c10> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has experimental_relax_shapes=True option that relaxes argument shapes that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for  more details.
Step3: generate final output
Traceback (most recent call last):
  File "./DeepTE.py", line 274, in <module>
    main()
  File "./DeepTE.py", line 254, in main
    combine_opt.extract_combine_infor(temp_store_opt_dir, output_dir,sp_type)
  File "/work/tk19812/software/DeepTE/scripts/DeepTE_combine_opt.py", line 251, in extract_combine_infor
    with open (input_opt_combine_DeepTE_dir + '/opt_DeepTE.txt','w+') as opt:
FileNotFoundError: [Errno 2] No such file or directory: 'output_dir//opt_DeepTE.txt'
yanhaidong1 commented 3 years ago

it looks like prediction is successful. the problem is output_dir may not be set? Since the final step is to write results to the opt_DeepTE.txt but cannot find the output_dir

francicco commented 3 years ago

How can I fix this? F

yanhaidong1 commented 3 years ago

would you check the -o output_dir. it looks like output_dir does not exist or is not on the right path.

francicco commented 3 years ago

I'm getting this:

(venv) [tk19812@bc4login1 test]$ DeepTE.py -d . -o output_dir -i ../example_data/input_test.fasta -sp P -m_dir Plants_model/
Using TensorFlow backend.
/mnt/storage/scratch/tk19812/software/DeepTE/venv/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:516: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_qint8 = np.dtype([("qint8", np.int8, 1)])
/mnt/storage/scratch/tk19812/software/DeepTE/venv/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:517: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_quint8 = np.dtype([("quint8", np.uint8, 1)])
/mnt/storage/scratch/tk19812/software/DeepTE/venv/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:518: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_qint16 = np.dtype([("qint16", np.int16, 1)])
/mnt/storage/scratch/tk19812/software/DeepTE/venv/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:519: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_quint16 = np.dtype([("quint16", np.uint16, 1)])
/mnt/storage/scratch/tk19812/software/DeepTE/venv/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:520: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_qint32 = np.dtype([("qint32", np.int32, 1)])
/mnt/storage/scratch/tk19812/software/DeepTE/venv/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:525: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  np_resource = np.dtype([("resource", np.ubyte, 1)])
/mnt/storage/scratch/tk19812/software/DeepTE/venv/lib/python3.6/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:541: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_qint8 = np.dtype([("qint8", np.int8, 1)])
/mnt/storage/scratch/tk19812/software/DeepTE/venv/lib/python3.6/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:542: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_quint8 = np.dtype([("quint8", np.uint8, 1)])
/mnt/storage/scratch/tk19812/software/DeepTE/venv/lib/python3.6/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:543: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_qint16 = np.dtype([("qint16", np.int16, 1)])
/mnt/storage/scratch/tk19812/software/DeepTE/venv/lib/python3.6/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:544: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_quint16 = np.dtype([("quint16", np.uint16, 1)])
/mnt/storage/scratch/tk19812/software/DeepTE/venv/lib/python3.6/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:545: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_qint32 = np.dtype([("qint32", np.int32, 1)])
/mnt/storage/scratch/tk19812/software/DeepTE/venv/lib/python3.6/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:550: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  np_resource = np.dtype([("resource", np.ubyte, 1)])
start time is 1620734913.0244358
Step1: transfer fasta data to CNN input data
Step2: classify TEs
Step2: 2) domain information is not exist
WARNING:tensorflow:From /mnt/storage/scratch/tk19812/software/DeepTE/venv/lib/python3.6/site-packages/keras/backend/tensorflow_backend.py:517: The name tf.placeholder is deprecated. Please use tf.compat.v1.placeholder instead.

WARNING:tensorflow:From /mnt/storage/scratch/tk19812/software/DeepTE/venv/lib/python3.6/site-packages/keras/backend/tensorflow_backend.py:4138: The name tf.random_uniform is deprecated. Please use tf.random.uniform instead.

WARNING:tensorflow:From /mnt/storage/scratch/tk19812/software/DeepTE/venv/lib/python3.6/site-packages/keras/backend/tensorflow_backend.py:3976: The name tf.nn.max_pool is deprecated. Please use tf.nn.max_pool2d instead.

WARNING:tensorflow:From /mnt/storage/scratch/tk19812/software/DeepTE/venv/lib/python3.6/site-packages/keras/backend/tensorflow_backend.py:131: The name tf.get_default_graph is deprecated. Please use tf.compat.v1.get_default_graph instead.

WARNING:tensorflow:From /mnt/storage/scratch/tk19812/software/DeepTE/venv/lib/python3.6/site-packages/keras/backend/tensorflow_backend.py:133: The name tf.placeholder_with_default is deprecated. Please use tf.compat.v1.placeholder_with_default instead.

WARNING:tensorflow:From /mnt/storage/scratch/tk19812/software/DeepTE/venv/lib/python3.6/site-packages/keras/backend/tensorflow_backend.py:3445: calling dropout (from tensorflow.python.ops.nn_ops) with keep_prob is deprecated and will be removed in a future version.
Instructions for updating:
Please use `rate` instead of `keep_prob`. Rate should be set to `rate = 1 - keep_prob`.
WARNING:tensorflow:From /mnt/storage/scratch/tk19812/software/DeepTE/venv/lib/python3.6/site-packages/keras/backend/tensorflow_backend.py:174: The name tf.get_default_session is deprecated. Please use tf.compat.v1.get_default_session instead.

2021-05-11 13:08:33.884334: I tensorflow/core/platform/cpu_feature_guard.cc:142] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA
2021-05-11 13:08:33.891525: I tensorflow/core/platform/profile_utils/cpu_utils.cc:94] CPU Frequency: 2399965000 Hz
2021-05-11 13:08:33.892565: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x2b5c3b4c5da0 executing computations on platform Host. Devices:
2021-05-11 13:08:33.892605: I tensorflow/compiler/xla/service/service.cc:175]   StreamExecutor device (0): <undefined>, <undefined>
2021-05-11 13:08:33.940600: W tensorflow/compiler/jit/mark_for_compilation_pass.cc:1412] (One-time warning): Not using XLA:CPU for cluster because envvar TF_XLA_FLAGS=--tf_xla_cpu_global_jit was not set.  If you want XLA:CPU, either set that envvar, or use experimental_jit_scope to enable XLA:CPU.  To confirm that XLA is active, pass --vmodule=xla_compilation_cache=1 (as a proper command-line flag, not via TF_XLA_FLAGS) or set the envvar XLA_FLAGS=--xla_hlo_profile.
WARNING:tensorflow:From /mnt/storage/scratch/tk19812/software/DeepTE/venv/lib/python3.6/site-packages/keras/optimizers.py:790: The name tf.train.Optimizer is deprecated. Please use tf.compat.v1.train.Optimizer instead.

WARNING:tensorflow:From /mnt/storage/scratch/tk19812/software/DeepTE/venv/lib/python3.6/site-packages/tensorflow/python/ops/math_grad.py:1250: add_dispatch_support.<locals>.wrapper (from tensorflow.python.ops.array_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use tf.where in 2.0, which has the same broadcast rule as np.where
Step3: generate final output
--- 79.57548213005066 seconds ---

and:

(venv) [tk19812@bc4login1 output_dir]$ ll
total 6.0M
-rw-r--r-- 1 tk19812 EBaB_Lab 5.8M May 11 13:09 opt_DeepTE.fasta
-rw-r--r-- 1 tk19812 EBaB_Lab  42K May 11 13:09 opt_DeepTE.txt

Is it working now? Cheers F

yanhaidong1 commented 3 years ago

Yes, you can check the output. also, the warning message shows some conflictions of package version. it is okay with them.

francicco commented 3 years ago

Great! Thanks a lot! F

yanhaidong1 commented 3 years ago

no problem. I am glad you get it. if you have other issues, please let us know.