search

jessieren / DeepVirFinder

Identifying viruses from metagenomic data by deep learning
Other
116 stars 32 forks source link

Running time #21

Open rubenbucio opened 3 years ago

rubenbucio commented 3 years ago

Hi!

I'm trying to use DeepVirFinder with the test metagenome provided by authors (CRC_meta.fa), just as a way to see how much time does it takes to finish...

It has been running for over 27 hours now, is it normal? something is corrupted?

Used command: python dvf.py -i test/CRC_meta.fa -l 1000 -c 14

I have:

jessieren commented 3 years ago

Thanks for using DeepVirFinder. Do you see any log output?

From: rubenbucio @.***> Sent: Tuesday, April 6, 2021 10:36 AM To: jessieren/DeepVirFinder Cc: Subscribed Subject: [jessieren/DeepVirFinder] Running time (#21)

Hi!

I'm trying to use DeepVirFinder with the test metagenome provided by authors (CRC_meta.fa), just as a way to see how much time does it takes to finish...

It has been running for over 27 hours now, is it normal? something is corrupted?

Used command: python dvf.py -i test/CRC_meta.fa -l 1000 -c 14

I have:

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHubhttps://urldefense.com/v3/__https://github.com/jessieren/DeepVirFinder/issues/21__;!!LIr3w8kk_Xxm!8D69Ql2LOm1rCbKVSkKYqSIyyy0zJ4gIjWmhqfqARCD7bWDlIy-rT2KxS5dJ$, or unsubscribehttps://urldefense.com/v3/__https://github.com/notifications/unsubscribe-auth/AB4CTPBEP3PTSJWN672LQ6TTHNBALANCNFSM42PFT4MQ__;!!LIr3w8kk_Xxm!8D69Ql2LOm1rCbKVSkKYqSIyyy0zJ4gIjWmhqfqARCD7bWDlIy-rT5YMd1a2$.

rubenbucio commented 3 years ago

Hi! This is the kind of output I have

2021-04-05 15:39:34.824010: I tensorflow/compiler/jit/xla_cpu_device.cc:41] Not creating XLA devices, tf_xla_enable_xla_devices not set 2021-04-05 15:39:34.825857: I tensorflow/core/platform/cpu_feature_guard.cc:142] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:
SSE4.1 SSE4.2 AVX AVX2 AVX512F FMA To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags. 2021-04-05 15:39:34.836641: I tensorflow/core/common_runtime/process_util.cc:146] Creating new thread pool with default inter op setting: 2. Tune using inter_op_parallelism_threads for best performance. WARNING:tensorflow:Error in loading the saved optimizer state. As a result, your model is starting with a freshly initialized optimizer. WARNING:tensorflow:Error in loading the saved optimizer state. As a result, your model is starting with a freshly initialized optimizer. WARNING:tensorflow:Error in loading the saved optimizer state. As a result, your model is starting with a freshly initialized optimizer. WARNING:tensorflow:Error in loading the saved optimizer state. As a result, your model is starting with a freshly initialized optimizer. 2021-04-05 15:39:39.035725: I tensorflow/compiler/mlir/mlir_graph_optimization_pass.cc:116] None of the MLIR optimization passes are enabled (registered 2) 2021-04-05 15:39:39.035725: I tensorflow/compiler/mlir/mlir_graph_optimization_pass.cc:116] None of the MLIR optimization passes are enabled (registered 2) 2021-04-05 15:39:39.035725: I tensorflow/compiler/mlir/mlir_graph_optimization_pass.cc:116] None of the MLIR optimization passes are enabled (registered 2) 2021-04-05 15:39:39.035725: I tensorflow/compiler/mlir/mlir_graph_optimization_pass.cc:116] None of the MLIR optimization passes are enabled (registered 2) 2021-04-05 15:39:39.035725: I tensorflow/compiler/mlir/mlir_graph_optimization_pass.cc:116] None of the MLIR optimization passes are enabled (registered 2) 2021-04-05 15:39:39.036003: I tensorflow/compiler/mlir/mlir_graph_optimization_pass.cc:116] None of the MLIR optimization passes are enabled (registered 2) 2021-04-05 15:39:39.037460: I tensorflow/core/platform/profile_utils/cpu_utils.cc:112] CPU Frequency: 2300000000 Hz . . .

  1. Loading Models. model directory /bettik/virclust/software/DeepVirFinder/models
  2. Encoding and Predicting Sequences. processing line 1 processing line 519

On 06.04.2021 15:10, Jessie Jie Ren wrote:

Thanks for using DeepVirFinder. Do you see any log output?

From: rubenbucio @.***> Sent: Tuesday, April 6, 2021 10:36 AM To: jessieren/DeepVirFinder Cc: Subscribed Subject: [jessieren/DeepVirFinder] Running time (#21)

Hi!

I'm trying to use DeepVirFinder with the test metagenome provided by authors (CRC_meta.fa), just as a way to see how much time does it takes to finish...

It has been running for over 27 hours now, is it normal? something is corrupted?

Used command: python dvf.py -i test/CRC_meta.fa -l 1000 -c 14

I have:

  • h5py 2.10.0
  • keras 2.4.3
  • theano 1.0.5 (1.0.3 was not available for python 3.8)

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHubhttps://urldefense.com/v3/__https://github.com/jessieren/DeepVirFinder/issues/21__;!!LIr3w8kk_Xxm!8D69Ql2LOm1rCbKVSkKYqSIyyy0zJ4gIjWmhqfqARCD7bWDlIy-rT2KxS5dJ$, or unsubscribehttps://urldefense.com/v3/__https://github.com/notifications/unsubscribe-auth/AB4CTPBEP3PTSJWN672LQ6TTHNBALANCNFSM42PFT4MQ__;!!LIr3w8kk_Xxm!8D69Ql2LOm1rCbKVSkKYqSIyyy0zJ4gIjWmhqfqARCD7bWDlIy-rT5YMd1a2$.

You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub [1], or unsubscribe [2].

Links:

[1] https://github.com/jessieren/DeepVirFinder/issues/21#issuecomment-814407640 [2] https://github.com/notifications/unsubscribe-auth/ASUSBXFYOJZP4SJ7WA7FGODTHNTDZANCNFSM42PFT4MQ

jessieren commented 3 years ago

Hi there,

Are you using CPU for computation? It will speed up a lot if you use GPU. Thanks!

Jie

From: rubenbucio @.***> Sent: Tuesday, April 6, 2021 3:11 PM To: jessieren/DeepVirFinder Cc: Jie Ren; Comment Subject: Re: [jessieren/DeepVirFinder] Running time (#21)

Hi! This is the kind of output I have

2021-04-05 15:39:34.824010: I tensorflow/compiler/jit/xla_cpu_device.cc:41] Not creating XLA devices, tf_xla_enable_xla_devices not set 2021-04-05 15:39:34.825857: I tensorflow/core/platform/cpu_feature_guard.cc:142] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations: SSE4.1 SSE4.2 AVX AVX2 AVX512F FMA To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags. 2021-04-05 15:39:34.836641: I tensorflow/core/common_runtime/process_util.cc:146] Creating new thread pool with default inter op setting: 2. Tune using inter_op_parallelism_threads for best performance. WARNING:tensorflow:Error in loading the saved optimizer state. As a result, your model is starting with a freshly initialized optimizer. WARNING:tensorflow:Error in loading the saved optimizer state. As a result, your model is starting with a freshly initialized optimizer. WARNING:tensorflow:Error in loading the saved optimizer state. As a result, your model is starting with a freshly initialized optimizer. WARNING:tensorflow:Error in loading the saved optimizer state. As a result, your model is starting with a freshly initialized optimizer. 2021-04-05 15:39:39.035725: I tensorflow/compiler/mlir/mlir_graph_optimization_pass.cc:116] None of the MLIR optimization passes are enabled (registered 2) 2021-04-05 15:39:39.035725: I tensorflow/compiler/mlir/mlir_graph_optimization_pass.cc:116] None of the MLIR optimization passes are enabled (registered 2) 2021-04-05 15:39:39.035725: I tensorflow/compiler/mlir/mlir_graph_optimization_pass.cc:116] None of the MLIR optimization passes are enabled (registered 2) 2021-04-05 15:39:39.035725: I tensorflow/compiler/mlir/mlir_graph_optimization_pass.cc:116] None of the MLIR optimization passes are enabled (registered 2) 2021-04-05 15:39:39.035725: I tensorflow/compiler/mlir/mlir_graph_optimization_pass.cc:116] None of the MLIR optimization passes are enabled (registered 2) 2021-04-05 15:39:39.036003: I tensorflow/compiler/mlir/mlir_graph_optimization_pass.cc:116] None of the MLIR optimization passes are enabled (registered 2) 2021-04-05 15:39:39.037460: I tensorflow/core/platform/profile_utils/cpu_utils.cc:112] CPU Frequency: 2300000000 Hz . . .

  1. Loading Models. model directory /bettik/virclust/software/DeepVirFinder/models
  2. Encoding and Predicting Sequences. processing line 1 processing line 519

On 06.04.2021 15:10, Jessie Jie Ren wrote:

Thanks for using DeepVirFinder. Do you see any log output?

From: rubenbucio @.***> Sent: Tuesday, April 6, 2021 10:36 AM To: jessieren/DeepVirFinder Cc: Subscribed Subject: [jessieren/DeepVirFinder] Running time (#21)

Hi!

I'm trying to use DeepVirFinder with the test metagenome provided by authors (CRC_meta.fa), just as a way to see how much time does it takes to finish...

It has been running for over 27 hours now, is it normal? something is corrupted?

Used command: python dvf.py -i test/CRC_meta.fa -l 1000 -c 14

I have:

  • h5py 2.10.0
  • keras 2.4.3
  • theano 1.0.5 (1.0.3 was not available for python 3.8)

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub<https://urldefense.com/v3/__https://github.com/jessieren/DeepVirFinder/issues/21__;!!LIr3w8kk_Xxm!8D69Ql2LOm1rCbKVSkKYqSIyyy0zJ4gIjWmhqfqARCD7bWDlIy-rT2KxS5dJ$%3E, or unsubscribe<https://urldefense.com/v3/__https://github.com/notifications/unsubscribe-auth/AB4CTPBEP3PTSJWN672LQ6TTHNBALANCNFSM42PFT4MQ__;!!LIr3w8kk_Xxm!8D69Ql2LOm1rCbKVSkKYqSIyyy0zJ4gIjWmhqfqARCD7bWDlIy-rT5YMd1a2$%3E.

You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub [1], or unsubscribe [2].

Links:

[1] https://github.com/jessieren/DeepVirFinder/issues/21#issuecomment-814407640https://urldefense.com/v3/__https://github.com/jessieren/DeepVirFinder/issues/21*issuecomment-814407640__;Iw!!LIr3w8kk_Xxm!_EltxFG-R6ZITDjrMDRLDvIOIMemK-zcndPMPFshWKqMUv4anlIJ0nGJexXQ$ [2] https://github.com/notifications/unsubscribe-auth/ASUSBXFYOJZP4SJ7WA7FGODTHNTDZANCNFSM42PFT4MQhttps://urldefense.com/v3/__https://github.com/notifications/unsubscribe-auth/ASUSBXFYOJZP4SJ7WA7FGODTHNTDZANCNFSM42PFT4MQ__;!!LIr3w8kk_Xxm!_EltxFG-R6ZITDjrMDRLDvIOIMemK-zcndPMPFshWKqMUv4anlIJ0iCQMTYs$

— You are receiving this because you commented. Reply to this email directly, view it on GitHubhttps://urldefense.com/v3/__https://github.com/jessieren/DeepVirFinder/issues/21*issuecomment-814469710__;Iw!!LIr3w8kk_Xxm!_EltxFG-R6ZITDjrMDRLDvIOIMemK-zcndPMPFshWKqMUv4anlIJ0vfvCUDs$, or unsubscribehttps://urldefense.com/v3/__https://github.com/notifications/unsubscribe-auth/AB4CTPHB6JEOCE53QFQRJZLTHOBJNANCNFSM42PFT4MQ__;!!LIr3w8kk_Xxm!_EltxFG-R6ZITDjrMDRLDvIOIMemK-zcndPMPFshWKqMUv4anlIJ0kDVfNmG$.

merytouceda commented 1 year ago

Hi! I have the same issue as rubenbucio here. I am running deepvirfinder on my University's HPC, this is the code I am using to send the job: 

!/bin/bash

#SBATCH --job-name=deepvirfinder
#SBATCH --output=deepvirfinder.out
#SBATCH --account=gornish
#SBATCH --mail-type=ALL
#SBATCH --mail-user=mtoucedasuarez@hpc.arizona.edu
#SBATCH --partition=standard
#SBATCH --ntasks=1
#SBATCH --nodes=1
#SBATCH --mem=5gb
#SBATCH --time=24:00:00

module load anaconda
source ~/.bashrc && conda activate
conda activate deepvirfinder

for sample in `awk '{print $1}' /xdisk/barberan/mtoucedasuarez/landuse/s_regex.txt`
do
   if [ "$sample" == "sample" ]; then continue; fi

   python /groups/barberan/software/DeepVirFinder/dvf.py -i /xdisk/barberan/mtoucedasuarez/landuse/assembly/contigs/${sample}_contig.fa -o /xdisk/barberan/mtoucedasuarez/landuse/virus/deepvirfinder/ -l 1500 -c 8

done

This is the error I get: 

2022-12-02 15:07:22.159336: I tensorflow/core/platform/cpu_feature_guard.cc:193] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 FMA
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
2022-12-02 15:07:22.372586: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcudart.so.11.0'; dlerror: libcudart.so.11.0: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /opt/ohpc/pub/apps/anaconda/2022.05/lib:/opt/ohpc/pub/mpi/openmpi3-gnu8/3.1.4/lib:/opt/ohpc/pub/compiler/gcc/8.3.0/lib64
2022-12-02 15:07:22.372630: I tensorflow/compiler/xla/stream_executor/cuda/cudart_stub.cc:29] Ignore above cudart dlerror if you do not have a GPU set up on your machine.
2022-12-02 15:07:23.834156: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer.so.7'; dlerror: libnvinfer.so.7: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /opt/ohpc/pub/apps/anaconda/2022.05/lib:/opt/ohpc/pub/mpi/openmpi3-gnu8/3.1.4/lib:/opt/ohpc/pub/compiler/gcc/8.3.0/lib64
2022-12-02 15:07:23.834282: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer_plugin.so.7'; dlerror: libnvinfer_plugin.so.7: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /opt/ohpc/pub/apps/anaconda/2022.05/lib:/opt/ohpc/pub/mpi/openmpi3-gnu8/3.1.4/lib:/opt/ohpc/pub/compiler/gcc/8.3.0/lib64
2022-12-02 15:07:23.834297: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Cannot dlopen some TensorRT libraries. If you would like to use Nvidia GPU with TensorRT, please make sure the missing libraries mentioned above are installed properly.
2022-12-02 15:07:25.996780: E tensorflow/compiler/xla/stream_executor/cuda/cuda_driver.cc:267] failed call to cuInit: CUDA_ERROR_NO_DEVICE: no CUDA-capable device is detected
2022-12-02 15:07:25.997115: I tensorflow/compiler/xla/stream_executor/cuda/cuda_diagnostics.cc:169] retrieving CUDA diagnostic information for host: i16n0.ocelote.hpc.arizona.edu
2022-12-02 15:07:25.997129: I tensorflow/compiler/xla/stream_executor/cuda/cuda_diagnostics.cc:176] hostname: i16n0.ocelote.hpc.arizona.edu
2022-12-02 15:07:25.997215: I tensorflow/compiler/xla/stream_executor/cuda/cuda_diagnostics.cc:200] libcuda reported version is: 520.61.5
2022-12-02 15:07:25.997265: I tensorflow/compiler/xla/stream_executor/cuda/cuda_diagnostics.cc:204] kernel reported version is: 520.61.5
2022-12-02 15:07:25.997277: I tensorflow/compiler/xla/stream_executor/cuda/cuda_diagnostics.cc:310] kernel version seems to match DSO: 520.61.5
2022-12-02 15:07:25.997590: I tensorflow/core/platform/cpu_feature_guard.cc:193] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 FMA
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
WARNING:tensorflow:Error in loading the saved optimizer state. As a result, your model is starting with a freshly initialized optimizer.
WARNING:tensorflow:Error in loading the saved optimizer state. As a result, your model is starting with a freshly initialized optimizer.
WARNING:tensorflow:Error in loading the saved optimizer state. As a result, your model is starting with a freshly initialized optimizer.
WARNING:tensorflow:Error in loading the saved optimizer state. As a result, your model is starting with a freshly initialized optimizer.
1. Loading Models.
   model directory /groups/barberan/software/DeepVirFinder/models
2. Encoding and Predicting Sequences.
   processing line 1
   processing line 18044
Dec 03 15:07:19.999226 29969 slurmstepd   0x2b6d39ca6700: error: *** JOB 1468064 ON i16n0 CANCELLED AT 2022-12-03T15:07:19 DUE TO TIME LIMIT ***

Is there anything I can do to solve this?