search

LiLabAtVT / DeepTE

Neural network classification of TE
BSD 3-Clause "New" or "Revised" License
82 stars 7 forks source link

error running DeepTE.py #18

Open svedwards opened 2 years ago

svedwards commented 2 years ago

Hello -

I installed all dependencies without a problem but I get the following errow with the basic command:

(py36) [sedwards@holy7c24603 moa]$python DeepTE.py -d deepte_in -o deepte_out -i ratite_unknown_repeats.fasta -sp M -m M

Traceback (most recent call last): File "DeepTE.py", line 17, in from scripts import DeepTE_pipeline_no_modification as pipeline_no_m ImportError: cannot import name 'DeepTE_pipeline_no_modification' from 'scripts' (/n/sw/eb/apps/centos7/Anaconda3/2020.11/lib/python3.8/site-packages/scripts/init.py

I have attached the input file. Thanks for any help.

Scott

ratite_unknown_repeats.fasta.zip )

svedwards commented 2 years ago

Hello again -

It seems to be working! So no need to respond (yet). I rebooted and it started to work. I will let you know if I am successful.

Scott

svedwards commented 2 years ago

Well, now it seems there is a problem with tensor flow. Please see below. The Metazoan models loaded fine but then problems began. You should consider making a docker with all the correct versions in it, no?

Metazoans_model/All_model.h5 Metazoans_model/ClassII_model.h5 Metazoans_model/ClassI_model.h5 Metazoans_model/Domain_model.h5 Metazoans_model/LINE_model.h5 Metazoans_model/LTR_model.h5 Metazoans_model/nLTR_model.h5 Metazoans_model/SINE_model.h5 Step1: transfer fasta data to CNN input data Step2: classify TEs Step2: 2) domain information is not exist WARNING:tensorflow:From /n/home06/sedwards/.conda/envs/py36/lib/python3.6/site-packages/keras/backend/tensorflow_backend.py:517: The name tf.placeholder is deprecated. Please use tf.compat.v1.placeholder instead.

WARNING:tensorflow:From /n/home06/sedwards/.conda/envs/py36/lib/python3.6/site-packages/keras/backend/tensorflow_backend.py:4138: The name tf.random_uniform is deprecated. Please use tf.random.uniform instead.

WARNING:tensorflow:From /n/home06/sedwards/.conda/envs/py36/lib/python3.6/site-packages/keras/backend/tensorflow_backend.py:3976: The name tf.nn.max_pool is deprecated. Please use tf.nn.max_pool2d instead.

WARNING:tensorflow:From /n/home06/sedwards/.conda/envs/py36/lib/python3.6/site-packages/keras/backend/tensorflow_backend.py:131: The name tf.get_default_graph is deprecated. Please use tf.compat.v1.get_default_graph instead.

WARNING:tensorflow:From /n/home06/sedwards/.conda/envs/py36/lib/python3.6/site-packages/keras/backend/tensorflow_backend.py:133: The name tf.placeholder_with_default is deprecated. Please use tf.compat.v1.placeholder_with_default instead.

WARNING:tensorflow:From /n/home06/sedwards/.conda/envs/py36/lib/python3.6/site-packages/keras/backend/tensorflow_backend.py:3445: calling dropout (from tensorflow.python.ops.nn_ops) with keep_prob is deprecated and will be removed in a future version. Instructions for updating: Please use rate instead of keep_prob. Rate should be set to rate = 1 - keep_prob. WARNING:tensorflow:From /n/home06/sedwards/.conda/envs/py36/lib/python3.6/site-packages/keras/backend/tensorflow_backend.py:174: The name tf.get_default_session is deprecated. Please use tf.compat.v1.get_default_session instead.

2022-08-08 08:36:36.458979: I tensorflow/core/platform/cpu_feature_guard.cc:142] Your CPU supports instructions that this TensorFlow binary was not compiled to use: SSE4.1 SSE4.2 AVX AVX2 AVX512F FMA 2022-08-08 08:36:36.493025: I tensorflow/core/platform/profile_utils/cpu_utils.cc:94] CPU Frequency: 2900000000 Hz 2022-08-08 08:36:36.493258: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x5623a45abd50 executing computations on platform Host. Devices: 2022-08-08 08:36:36.493330: I tensorflow/compiler/xla/service/service.cc:175] StreamExecutor device (0): , 2022-08-08 08:36:36.631402: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Could not dlopen library 'libcuda.so.1'; dlerror: libcuda.so.1: cannot open shared object file: No such file or directory 2022-08-08 08:36:36.631571: E tensorflow/stream_executor/cuda/cuda_driver.cc:318] failed call to cuInit: UNKNOWN ERROR (303) 2022-08-08 08:36:36.631708: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:156] kernel driver does not appear to be running on this host (holy7c24603.rc.fas.harvard.edu): /proc/driver/nvidia/version does not exist 2022-08-08 08:36:36.952726: W tensorflow/compiler/jit/mark_for_compilation_pass.cc:1412] (One-time warning): Not using XLA:CPU for cluster because envvar TF_XLA_FLAGS=--tf_xla_cpu_global_jit was not set. If you want XLA:CPU, either set that envvar, or use experimental_jit_scope to enable XLA:CPU. To confirm that XLA is active, pass --vmodule=xla_compilation_cache=1 (as a proper command-line flag, not via TF_XLA_FLAGS) or set the envvar XLA_FLAGS=--xla_hlo_profile. WARNING:tensorflow:From /n/home06/sedwards/.conda/envs/py36/lib/python3.6/site-packages/keras/optimizers.py:790: The name tf.train.Optimizer is deprecated. Please use tf.compat.v1.train.Optimizer instead.

WARNING:tensorflow:From /n/home06/sedwards/.conda/envs/py36/lib/python3.6/site-packages/tensorflow/python/ops/math_grad.py:1250: add_dispatch_support..wrapper (from tensorflow.python.ops.array_ops) is deprecated and will be removed in a future version. Instructions for updating: Use tf.where in 2.0, which has the same broadcast rule as np.where Killed (py36) [sedwards@holy7c24603 DeepTE-master]$

yanhaidong1 commented 2 years ago

Hey Scott,

Thanks for providing the suggestion. We would consider to update it in the future. Have you got the results finally? I guess you could get the output if you installed the dependences with corrected version shown in the github. You may ignore the warnings if you could get results. These warnings are probably generated by using old package loadings during training.

Best, Haidong

svedwards commented 2 years ago

Hi Haidong,

Thanks for your reply. Unfortunately, I did not get any outputs. As you can see from my text above, the process was killed and aborted when it encountered the wrong version of tensorflow. I uploaded precisely the packages you indicated on Github to my py36 environment, but it still did not work. Let me know if you have any further suggestions, or I guess I will have to use another package. Thanks again for your response. Scott

yanhaidong1 commented 2 years ago

Hey Scott,

I just tried your test data you provided and can make a prediction and your file has no problem. Also, you may try to install regular tensorflow (1.14.0) other than tensorflow-gpu since it might have some issue of gpu settings. Would you please tell me the memory you used for the predicting? Sometimes, it may be killed by the limited memory.

Best, Haidong

svedwards commented 2 years ago

Dear Haidong,

Thanks for your help and sorry for the delay.

I was able to get DeepTE to run with some help from my local IT administrator. He got be set up on a GPU machine and installed a more recent version of Tensor Flow (version 2.5) in a new conda environment (py38). He also added the comment: "Recent TF include the Keras interface in the distribution starting with version 2.0."

To activate and use the new environment I ran these commands on an interactive session on a GPU machine:

module load python/3.8.5-fasrc01 cuda/11.1.0-fasrc01 cudnn/8.1.0.77_cuda11.2-fasrc01 source activate py38

From the DeepTE-master directory I ran this command:

python DeepTE.py -d ../deepte_in -o ../deepte_out -i ../bird_unknown_repeats.fasta -sp M -m M

And I think it worked! I attach my results. Let me know what you think. It looks like it classified all but 71 of 1167 TEs. Can you tell me what the designation "ClassI" by itself means?

I ran this with 8G memory and 1 core and it took only a few minutes.

Thanks!

Scott opt_DeepTE.txt

yanhaidong1 commented 2 years ago

Hey Scott,

I am glad you solved the issue.

In your case, the DeepTE tool cannot determine the sub families under the Class I family so it will label it as 'Class I', since the predicted probability of no sub family can reach to 0.6 that is a default setting. You may use '-prop_thr' to get a lower value like 0.5, which may allow you find the sub families under the Class I. The same way as 'unknown' class, and the DeepTE cannot determine the major sub classes because no subclass achieves probability of 0.6. The higher probability setting may give you higher confident results.

Best wishes, Haidong

jordana-olive commented 1 year ago

I had several issues with tensor. I've tried almost everything. The solution that worked for me was created a 37 python env and then install tensor=2.2.0 + keras=2.3.1

I found this solution here: https://stackoverflow.com/questions/62690377/tensorflow-compatibility-with-keras Maybe it will be useful for someone.

Cheers!

yanhaidong1 commented 1 year ago

Hey Jordana,

Thank you so much for trying. I will include a new section to provide suggestions to solve this issue.

Best wishes, haidong

On Wed, Oct 19, 2022 at 4:20 PM Jordana Oliveira @.***> wrote:

I had several issues with tensor. I've tried almost everything. The solution that worked for me was created a 37 python env and then install tensor=2.2.0 + keras=2.3.1

I found this solution here: https://stackoverflow.com/questions/62690377/tensorflow-compatibility-with-keras Maybe it will be useful for someone.

Cheers!

— Reply to this email directly, view it on GitHub https://github.com/LiLabAtVT/DeepTE/issues/18#issuecomment-1284527624, or unsubscribe https://github.com/notifications/unsubscribe-auth/AIO5OYAAY57ZP4FXLDRGVDDWEBJXDANCNFSM553VY5UA . You are receiving this because you commented.Message ID: @.***>