Open rubenbucio opened 3 years ago
Thanks for using DeepVirFinder. Do you see any log output?
From: rubenbucio @.***> Sent: Tuesday, April 6, 2021 10:36 AM To: jessieren/DeepVirFinder Cc: Subscribed Subject: [jessieren/DeepVirFinder] Running time (#21)
Hi!
I'm trying to use DeepVirFinder with the test metagenome provided by authors (CRC_meta.fa), just as a way to see how much time does it takes to finish...
It has been running for over 27 hours now, is it normal? something is corrupted?
Used command: python dvf.py -i test/CRC_meta.fa -l 1000 -c 14
I have:
— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHubhttps://urldefense.com/v3/__https://github.com/jessieren/DeepVirFinder/issues/21__;!!LIr3w8kk_Xxm!8D69Ql2LOm1rCbKVSkKYqSIyyy0zJ4gIjWmhqfqARCD7bWDlIy-rT2KxS5dJ$, or unsubscribehttps://urldefense.com/v3/__https://github.com/notifications/unsubscribe-auth/AB4CTPBEP3PTSJWN672LQ6TTHNBALANCNFSM42PFT4MQ__;!!LIr3w8kk_Xxm!8D69Ql2LOm1rCbKVSkKYqSIyyy0zJ4gIjWmhqfqARCD7bWDlIy-rT5YMd1a2$.
Hi! This is the kind of output I have
2021-04-05 15:39:34.824010: I
tensorflow/compiler/jit/xla_cpu_device.cc:41] Not creating XLA devices,
tf_xla_enable_xla_devices not set
2021-04-05 15:39:34.825857: I
tensorflow/core/platform/cpu_feature_guard.cc:142] This TensorFlow
binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to
use the following CPU instructions in performance-critical operations:
SSE4.1 SSE4.2 AVX AVX2 AVX512F FMA
To enable them in other operations, rebuild TensorFlow with the
appropriate compiler flags.
2021-04-05 15:39:34.836641: I
tensorflow/core/common_runtime/process_util.cc:146] Creating new thread
pool with default inter op setting: 2. Tune using
inter_op_parallelism_threads for best performance.
WARNING:tensorflow:Error in loading the saved optimizer state. As a
result, your model is starting with a freshly initialized optimizer.
WARNING:tensorflow:Error in loading the saved optimizer state. As a
result, your model is starting with a freshly initialized optimizer.
WARNING:tensorflow:Error in loading the saved optimizer state. As a
result, your model is starting with a freshly initialized optimizer.
WARNING:tensorflow:Error in loading the saved optimizer state. As a
result, your model is starting with a freshly initialized optimizer.
2021-04-05 15:39:39.035725: I
tensorflow/compiler/mlir/mlir_graph_optimization_pass.cc:116] None of
the MLIR optimization passes are enabled (registered 2)
2021-04-05 15:39:39.035725: I
tensorflow/compiler/mlir/mlir_graph_optimization_pass.cc:116] None of
the MLIR optimization passes are enabled (registered 2)
2021-04-05 15:39:39.035725: I
tensorflow/compiler/mlir/mlir_graph_optimization_pass.cc:116] None of
the MLIR optimization passes are enabled (registered 2)
2021-04-05 15:39:39.035725: I
tensorflow/compiler/mlir/mlir_graph_optimization_pass.cc:116] None of
the MLIR optimization passes are enabled (registered 2)
2021-04-05 15:39:39.035725: I
tensorflow/compiler/mlir/mlir_graph_optimization_pass.cc:116] None of
the MLIR optimization passes are enabled (registered 2)
2021-04-05 15:39:39.036003: I
tensorflow/compiler/mlir/mlir_graph_optimization_pass.cc:116] None of
the MLIR optimization passes are enabled (registered 2)
2021-04-05 15:39:39.037460: I
tensorflow/core/platform/profile_utils/cpu_utils.cc:112] CPU Frequency:
2300000000 Hz
.
.
.
On 06.04.2021 15:10, Jessie Jie Ren wrote:
Thanks for using DeepVirFinder. Do you see any log output?
From: rubenbucio @.***> Sent: Tuesday, April 6, 2021 10:36 AM To: jessieren/DeepVirFinder Cc: Subscribed Subject: [jessieren/DeepVirFinder] Running time (#21)
Hi!
I'm trying to use DeepVirFinder with the test metagenome provided by authors (CRC_meta.fa), just as a way to see how much time does it takes to finish...
It has been running for over 27 hours now, is it normal? something is corrupted?
Used command: python dvf.py -i test/CRC_meta.fa -l 1000 -c 14
I have:
- h5py 2.10.0
- keras 2.4.3
- theano 1.0.5 (1.0.3 was not available for python 3.8)
— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHubhttps://urldefense.com/v3/__https://github.com/jessieren/DeepVirFinder/issues/21__;!!LIr3w8kk_Xxm!8D69Ql2LOm1rCbKVSkKYqSIyyy0zJ4gIjWmhqfqARCD7bWDlIy-rT2KxS5dJ$, or unsubscribehttps://urldefense.com/v3/__https://github.com/notifications/unsubscribe-auth/AB4CTPBEP3PTSJWN672LQ6TTHNBALANCNFSM42PFT4MQ__;!!LIr3w8kk_Xxm!8D69Ql2LOm1rCbKVSkKYqSIyyy0zJ4gIjWmhqfqARCD7bWDlIy-rT5YMd1a2$.
You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub [1], or unsubscribe [2].
Links:
[1] https://github.com/jessieren/DeepVirFinder/issues/21#issuecomment-814407640 [2] https://github.com/notifications/unsubscribe-auth/ASUSBXFYOJZP4SJ7WA7FGODTHNTDZANCNFSM42PFT4MQ
Hi there,
Are you using CPU for computation? It will speed up a lot if you use GPU. Thanks!
Jie
From: rubenbucio @.***> Sent: Tuesday, April 6, 2021 3:11 PM To: jessieren/DeepVirFinder Cc: Jie Ren; Comment Subject: Re: [jessieren/DeepVirFinder] Running time (#21)
Hi! This is the kind of output I have
2021-04-05 15:39:34.824010: I tensorflow/compiler/jit/xla_cpu_device.cc:41] Not creating XLA devices, tf_xla_enable_xla_devices not set 2021-04-05 15:39:34.825857: I tensorflow/core/platform/cpu_feature_guard.cc:142] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations: SSE4.1 SSE4.2 AVX AVX2 AVX512F FMA To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags. 2021-04-05 15:39:34.836641: I tensorflow/core/common_runtime/process_util.cc:146] Creating new thread pool with default inter op setting: 2. Tune using inter_op_parallelism_threads for best performance. WARNING:tensorflow:Error in loading the saved optimizer state. As a result, your model is starting with a freshly initialized optimizer. WARNING:tensorflow:Error in loading the saved optimizer state. As a result, your model is starting with a freshly initialized optimizer. WARNING:tensorflow:Error in loading the saved optimizer state. As a result, your model is starting with a freshly initialized optimizer. WARNING:tensorflow:Error in loading the saved optimizer state. As a result, your model is starting with a freshly initialized optimizer. 2021-04-05 15:39:39.035725: I tensorflow/compiler/mlir/mlir_graph_optimization_pass.cc:116] None of the MLIR optimization passes are enabled (registered 2) 2021-04-05 15:39:39.035725: I tensorflow/compiler/mlir/mlir_graph_optimization_pass.cc:116] None of the MLIR optimization passes are enabled (registered 2) 2021-04-05 15:39:39.035725: I tensorflow/compiler/mlir/mlir_graph_optimization_pass.cc:116] None of the MLIR optimization passes are enabled (registered 2) 2021-04-05 15:39:39.035725: I tensorflow/compiler/mlir/mlir_graph_optimization_pass.cc:116] None of the MLIR optimization passes are enabled (registered 2) 2021-04-05 15:39:39.035725: I tensorflow/compiler/mlir/mlir_graph_optimization_pass.cc:116] None of the MLIR optimization passes are enabled (registered 2) 2021-04-05 15:39:39.036003: I tensorflow/compiler/mlir/mlir_graph_optimization_pass.cc:116] None of the MLIR optimization passes are enabled (registered 2) 2021-04-05 15:39:39.037460: I tensorflow/core/platform/profile_utils/cpu_utils.cc:112] CPU Frequency: 2300000000 Hz . . .
On 06.04.2021 15:10, Jessie Jie Ren wrote:
Thanks for using DeepVirFinder. Do you see any log output?
From: rubenbucio @.***> Sent: Tuesday, April 6, 2021 10:36 AM To: jessieren/DeepVirFinder Cc: Subscribed Subject: [jessieren/DeepVirFinder] Running time (#21)
Hi!
I'm trying to use DeepVirFinder with the test metagenome provided by authors (CRC_meta.fa), just as a way to see how much time does it takes to finish...
It has been running for over 27 hours now, is it normal? something is corrupted?
Used command: python dvf.py -i test/CRC_meta.fa -l 1000 -c 14
I have:
- h5py 2.10.0
- keras 2.4.3
- theano 1.0.5 (1.0.3 was not available for python 3.8)
— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub<https://urldefense.com/v3/__https://github.com/jessieren/DeepVirFinder/issues/21__;!!LIr3w8kk_Xxm!8D69Ql2LOm1rCbKVSkKYqSIyyy0zJ4gIjWmhqfqARCD7bWDlIy-rT2KxS5dJ$%3E, or unsubscribe<https://urldefense.com/v3/__https://github.com/notifications/unsubscribe-auth/AB4CTPBEP3PTSJWN672LQ6TTHNBALANCNFSM42PFT4MQ__;!!LIr3w8kk_Xxm!8D69Ql2LOm1rCbKVSkKYqSIyyy0zJ4gIjWmhqfqARCD7bWDlIy-rT5YMd1a2$%3E.
You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub [1], or unsubscribe [2].
Links:
[1] https://github.com/jessieren/DeepVirFinder/issues/21#issuecomment-814407640https://urldefense.com/v3/__https://github.com/jessieren/DeepVirFinder/issues/21*issuecomment-814407640__;Iw!!LIr3w8kk_Xxm!_EltxFG-R6ZITDjrMDRLDvIOIMemK-zcndPMPFshWKqMUv4anlIJ0nGJexXQ$ [2] https://github.com/notifications/unsubscribe-auth/ASUSBXFYOJZP4SJ7WA7FGODTHNTDZANCNFSM42PFT4MQhttps://urldefense.com/v3/__https://github.com/notifications/unsubscribe-auth/ASUSBXFYOJZP4SJ7WA7FGODTHNTDZANCNFSM42PFT4MQ__;!!LIr3w8kk_Xxm!_EltxFG-R6ZITDjrMDRLDvIOIMemK-zcndPMPFshWKqMUv4anlIJ0iCQMTYs$
— You are receiving this because you commented. Reply to this email directly, view it on GitHubhttps://urldefense.com/v3/__https://github.com/jessieren/DeepVirFinder/issues/21*issuecomment-814469710__;Iw!!LIr3w8kk_Xxm!_EltxFG-R6ZITDjrMDRLDvIOIMemK-zcndPMPFshWKqMUv4anlIJ0vfvCUDs$, or unsubscribehttps://urldefense.com/v3/__https://github.com/notifications/unsubscribe-auth/AB4CTPHB6JEOCE53QFQRJZLTHOBJNANCNFSM42PFT4MQ__;!!LIr3w8kk_Xxm!_EltxFG-R6ZITDjrMDRLDvIOIMemK-zcndPMPFshWKqMUv4anlIJ0kDVfNmG$.
Hi! I have the same issue as rubenbucio here. I am running deepvirfinder on my University's HPC, this is the code I am using to send the job:
!/bin/bash
#SBATCH --job-name=deepvirfinder
#SBATCH --output=deepvirfinder.out
#SBATCH --account=gornish
#SBATCH --mail-type=ALL
#SBATCH --mail-user=mtoucedasuarez@hpc.arizona.edu
#SBATCH --partition=standard
#SBATCH --ntasks=1
#SBATCH --nodes=1
#SBATCH --mem=5gb
#SBATCH --time=24:00:00
module load anaconda
source ~/.bashrc && conda activate
conda activate deepvirfinder
for sample in `awk '{print $1}' /xdisk/barberan/mtoucedasuarez/landuse/s_regex.txt`
do
if [ "$sample" == "sample" ]; then continue; fi
python /groups/barberan/software/DeepVirFinder/dvf.py -i /xdisk/barberan/mtoucedasuarez/landuse/assembly/contigs/${sample}_contig.fa -o /xdisk/barberan/mtoucedasuarez/landuse/virus/deepvirfinder/ -l 1500 -c 8
done
This is the error I get:
2022-12-02 15:07:22.159336: I tensorflow/core/platform/cpu_feature_guard.cc:193] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations: AVX2 FMA
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
2022-12-02 15:07:22.372586: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcudart.so.11.0'; dlerror: libcudart.so.11.0: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /opt/ohpc/pub/apps/anaconda/2022.05/lib:/opt/ohpc/pub/mpi/openmpi3-gnu8/3.1.4/lib:/opt/ohpc/pub/compiler/gcc/8.3.0/lib64
2022-12-02 15:07:22.372630: I tensorflow/compiler/xla/stream_executor/cuda/cudart_stub.cc:29] Ignore above cudart dlerror if you do not have a GPU set up on your machine.
2022-12-02 15:07:23.834156: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer.so.7'; dlerror: libnvinfer.so.7: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /opt/ohpc/pub/apps/anaconda/2022.05/lib:/opt/ohpc/pub/mpi/openmpi3-gnu8/3.1.4/lib:/opt/ohpc/pub/compiler/gcc/8.3.0/lib64
2022-12-02 15:07:23.834282: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer_plugin.so.7'; dlerror: libnvinfer_plugin.so.7: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /opt/ohpc/pub/apps/anaconda/2022.05/lib:/opt/ohpc/pub/mpi/openmpi3-gnu8/3.1.4/lib:/opt/ohpc/pub/compiler/gcc/8.3.0/lib64
2022-12-02 15:07:23.834297: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Cannot dlopen some TensorRT libraries. If you would like to use Nvidia GPU with TensorRT, please make sure the missing libraries mentioned above are installed properly.
2022-12-02 15:07:25.996780: E tensorflow/compiler/xla/stream_executor/cuda/cuda_driver.cc:267] failed call to cuInit: CUDA_ERROR_NO_DEVICE: no CUDA-capable device is detected
2022-12-02 15:07:25.997115: I tensorflow/compiler/xla/stream_executor/cuda/cuda_diagnostics.cc:169] retrieving CUDA diagnostic information for host: i16n0.ocelote.hpc.arizona.edu
2022-12-02 15:07:25.997129: I tensorflow/compiler/xla/stream_executor/cuda/cuda_diagnostics.cc:176] hostname: i16n0.ocelote.hpc.arizona.edu
2022-12-02 15:07:25.997215: I tensorflow/compiler/xla/stream_executor/cuda/cuda_diagnostics.cc:200] libcuda reported version is: 520.61.5
2022-12-02 15:07:25.997265: I tensorflow/compiler/xla/stream_executor/cuda/cuda_diagnostics.cc:204] kernel reported version is: 520.61.5
2022-12-02 15:07:25.997277: I tensorflow/compiler/xla/stream_executor/cuda/cuda_diagnostics.cc:310] kernel version seems to match DSO: 520.61.5
2022-12-02 15:07:25.997590: I tensorflow/core/platform/cpu_feature_guard.cc:193] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations: AVX2 FMA
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
WARNING:tensorflow:Error in loading the saved optimizer state. As a result, your model is starting with a freshly initialized optimizer.
WARNING:tensorflow:Error in loading the saved optimizer state. As a result, your model is starting with a freshly initialized optimizer.
WARNING:tensorflow:Error in loading the saved optimizer state. As a result, your model is starting with a freshly initialized optimizer.
WARNING:tensorflow:Error in loading the saved optimizer state. As a result, your model is starting with a freshly initialized optimizer.
1. Loading Models.
model directory /groups/barberan/software/DeepVirFinder/models
2. Encoding and Predicting Sequences.
processing line 1
processing line 18044
Dec 03 15:07:19.999226 29969 slurmstepd 0x2b6d39ca6700: error: *** JOB 1468064 ON i16n0 CANCELLED AT 2022-12-03T15:07:19 DUE TO TIME LIMIT ***
Is there anything I can do to solve this?
Hi!
I'm trying to use DeepVirFinder with the test metagenome provided by authors (CRC_meta.fa), just as a way to see how much time does it takes to finish...
It has been running for over 27 hours now, is it normal? something is corrupted?
Used command: python dvf.py -i test/CRC_meta.fa -l 1000 -c 14
I have: