Open chklopp opened 1 year ago
alisandra
commented
1 year ago Hi, thanks for the message, it's a little crazy right now, so to not keep you waiting longer while we investigate, I would simply suggest rolling back to version 0.3.0 (there are small differences in speed and parameter naming, but you should be able to achieve the same results).
https://github.com/gglyptodon/helixer-docker/tree/helixer_v0.3.0
philippbayer
commented
1 year ago I've just had the same issue, and downgrading to 0.3.0 didn't solve it. I got the same warning.
However, that particular library comes from TensorRT, and Helixer doesn't seem to rely on TensorRT? If I run either 0.3.0 or 0.3.1 and check the GPU usage, python3 pops up in nvidia-smi so it seems to talk with the GPU correctly. I ran the first chromosome of GCF_902827115.1
One potential change in the manual: Instead of running docker run --runtime=nvidia I used docker run --gpus=all
Hi,
Following the install procedure I've run, on a GPU node, an annotation test. But I get warnings about missing libraries : libnvinfer.so.7, libnvinfer_plugin.so.7. And when I monitor the server, the job seems to run only on CPU.
What should I do to fix this?
singularity run --nv helixer-docker_helixer_v0.3.1_cuda_11.2.0-cudnn8.sif Helixer.py --fasta-path my.fasta --lineage land_plant --gff-output-path my.gff3 2023-03-13 16:06:09.959811: I tensorflow/core/platform/cpu_feature_guard.cc:193] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations: AVX2 AVX512F AVX512_VNNI FMA To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags. 2023-03-13 16:06:10.359479: I tensorflow/core/util/port.cc:104] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable
TF_ENABLE_ONEDNN_OPTS=0. 2023-03-13 16:06:12.183775: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer.so.7'; dlerror: libnvinfer.so.7: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /usr/local/nvidia/lib:/usr/local/nvidia/lib64:/.singularity.d/libs 2023-03-13 16:06:12.187243: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer_plugin.so.7'; dlerror: libnvinfer_plugin.so.7: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /usr/local/nvidia/lib:/usr/local/nvidia/lib64:/.singularity.d/libs 2023-03-13 16:06:12.187293: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Cannot dlopen some TensorRT libraries. If you would like to use Nvidia GPU with TensorRT, please make sure the missing libraries mentioned above are installed properly.