jinuhwang
commented
2 years ago Hi @Nier4Ryu, my best guess is that you have a different TensorRT version. Could you check the TensorRT version and your CUDA version on your host?
Closed Nier4Ryu closed 2 years ago
jinuhwang
commented
2 years ago Hi @Nier4Ryu, my best guess is that you have a different TensorRT version. Could you check the TensorRT version and your CUDA version on your host?
Nier4Ryu
commented
2 years ago On my local windows11 host I currently have cuda 11.7 installed and no TensorRT installed would I have to uninstall this?
Nier4Ryu
commented
2 years ago I just built the docker image with out installing TF, the current image shows
nvcc --version nvcc: NVIDIA (R) Cuda compiler driver Copyright (c) 2005-2021 NVIDIA Corporation Built on Sun_Aug_15_21:14:11_PDT_2021 Cuda compilation tools, release 11.4, V11.4.120 Build cuda_11.4.r11.4/compiler.30300941_0
dpkg -l | grep nvinfer ii libnvinfer-bin 8.4.2-1+cuda11.6 amd64 TensorRT binaries ii libnvinfer-dev 8.4.2-1+cuda11.6 amd64 TensorRT development libraries and headers ii libnvinfer-plugin-dev 8.4.2-1+cuda11.6 amd64 TensorRT plugin libraries ii libnvinfer-plugin8 8.4.2-1+cuda11.6 amd64 TensorRT plugin libraries ii libnvinfer-samples 8.4.2-1+cuda11.6 all TensorRT samples ii libnvinfer8 8.4.2-1+cuda11.6 amd64 TensorRT runtime libraries ii python3-libnvinfer 8.4.2-1+cuda11.6 amd64 Python 3 bindings for TensorRT ii python3-libnvinfer-dev 8.4.2-1+cuda11.6 amd64 Python 3 development package for TensorRT
I got the tensorrt package for TensorRT 8.2 GA Update 3 for x86_64 Architecture - TensorRT 8.2 GA Update 3 for Ubuntu 18.04 and CUDA 11.0, 11.1, 11.2, 11.3, 11.4 and 11.5 DEB local repo Package As this file had the correct name of the required tensorrt deb file
could there be a problem from here?
Nier4Ryu
commented
2 years ago I resolved the tensorrt installation by changing the install_trt.sh from
dpkg -i nv-tensorrt-repo-${os}-${tag}_1-1_amd64.deb
apt-key add /var/nv-tensorrt-repo-${os}-${tag}/7fa2af80.pub
apt-get update
apt-get install -y tensorrt
apt-get install -y python3-libnvinfer-dev uff-converter-tf onnx-graphsurgeonto
dpkg -i nv-tensorrt-repo-ubuntu1804-cuda11.4-trt8.2.4.2-ga-20220324_1-1_amd64.deb
apt-key add /var/nv-tensorrt-repo-ubuntu1804-cuda11.4-trt8.2.4.2-ga-20220324/*.pub
apt-get update
apt-get install libnvinfer8=8.2.4-1+cuda11.4
apt-get install libnvinfer-plugin8=8.2.4-1+cuda11.4
apt-get install libnvparsers8=8.2.4-1+cuda11.4
apt-get install libnvonnxparsers8=8.2.4-1+cuda11.4
apt-get install libnvinfer-bin=8.2.4-1+cuda11.4
apt-get install libnvinfer-dev=8.2.4-1+cuda11.4
apt-get install libnvinfer-plugin-dev=8.2.4-1+cuda11.4
apt-get install libnvparsers-dev=8.2.4-1+cuda11.4
apt-get install libnvonnxparsers-dev=8.2.4-1+cuda11.4
apt-get install libnvinfer-samples=8.2.4-1+cuda11.4
apt-get install libnvinfer-doc=8.2.4-1+cuda11.4
apt-get install tensorrt=8.2.4.2-1+cuda11.4
apt-get install python3-libnvinfer=8.2.4-1+cuda11.4
apt-get install python3-libnvinfer-dev=8.2.4-1+cuda11.4
apt-get install graphsurgeon-tf=8.2.4-1+cuda11.4
apt-get install uff-converter-tf=8.2.4-1+cuda11.4
apt-get install onnx-graphsurgeon=8.2.4-1+cuda11.4
so now
dpkg -l | grep nvinfer
ii libnvinfer-bin 8.2.4-1+cuda11.4 amd64 TensorRT binaries
ii libnvinfer-dev 8.2.4-1+cuda11.4 amd64 TensorRT development libraries and headers
ii libnvinfer-doc 8.2.4-1+cuda11.4 all TensorRT documentation
ii libnvinfer-plugin-dev 8.2.4-1+cuda11.4 amd64 TensorRT plugin libraries
ii libnvinfer-plugin8 8.2.4-1+cuda11.4 amd64 TensorRT plugin libraries
ii libnvinfer-samples 8.2.4-1+cuda11.4 all TensorRT samples
ii libnvinfer8 8.2.4-1+cuda11.4 amd64 TensorRT runtime libraries
ii python3-libnvinfer 8.2.4-1+cuda11.4 amd64 Python 3 bindings for TensorRT
ii python3-libnvinfer-dev 8.2.4-1+cuda11.4 amd64 Python 3 development package for TensorRTso I guess now I have properly installed tensorrt 8.2.4, but still I have this issue of
ERROR: /build/tf/tensorflow/tensorflow/compiler/tf2tensorrt/BUILD:43:11: Compiling tensorflow/compiler/tf2tensorrt/stub/nvinfer_stub.cc failed: (Exit 1): crosstool_wrapper_driver_is_not_gcc failed: error executing command external/local_config_cuda/crosstool/clang/bin/crosstool_wrapper_driver_is_not_gcc -MD -MF bazel-out/k8-opt/bin/tensorflow/compiler/tf2tensorrt/_objs/tensorrt_stub/nvinfer_stub.pic.d ... (remaining 150 argument(s) skipped)
tensorflow/compiler/tf2tensorrt/stub/nvinfer_stub.cc:66:2: error: #error This version of TensorRT is not supported.
error This version of TensorRT is not supported.what could be the problem?
jinuhwang
commented
2 years ago To me, it seems to be an issue with the Tensorflow and TensorRT version, if you weren't able to get the TensorRT version that I used as in the install_trt.sh, you probably have to change the Tensorflow to matching version as well.
Nier4Ryu
commented
2 years ago After deleting the current image and reinstalling from scratch did the job!
changes made are the following
in install_trt.sh
from
dpkg -i nv-tensorrt-repo-${os}-${tag}_1-1_amd64.deb
apt-key add /var/nv-tensorrt-repo-${os}-${tag}/7fa2af80.pub
apt-get update
apt-get install -y tensorrt
apt-get install -y python3-libnvinfer-dev uff-converter-tf onnx-graphsurgeonto
dpkg -i nv-tensorrt-repo-ubuntu1804-cuda11.4-trt8.2.4.2-ga-20220324_1-1_amd64.deb
apt-key add /var/nv-tensorrt-repo-ubuntu1804-cuda11.4-trt8.2.4.2-ga-20220324/*.pub
apt-get update
apt-get install -y libnvinfer8=8.2.4-1+cuda11.4 -f
apt-get install -y libnvinfer-plugin8=8.2.4-1+cuda11.4 -f
apt-get install -y libnvparsers8=8.2.4-1+cuda11.4 -f
apt-get install -y libnvonnxparsers8=8.2.4-1+cuda11.4 -f
apt-get install -y libnvinfer-bin=8.2.4-1+cuda11.4 -f
apt-get install -y libnvinfer-dev=8.2.4-1+cuda11.4 -f
apt-get install -y libnvinfer-plugin-dev=8.2.4-1+cuda11.4 -f
apt-get install -y libnvparsers-dev=8.2.4-1+cuda11.4 -f
apt-get install -y libnvonnxparsers-dev=8.2.4-1+cuda11.4 -f
apt-get install -y libnvinfer-samples=8.2.4-1+cuda11.4 -f
apt-get install -y libnvinfer-doc=8.2.4-1+cuda11.4 -f
apt-get install -y tensorrt=8.2.4.2-1+cuda11.4 -f
apt-get install -y python3-libnvinfer=8.2.4-1+cuda11.4 -f
apt-get install -y python3-libnvinfer-dev=8.2.4-1+cuda11.4 -f
apt-get install -y graphsurgeon-tf=8.2.4-1+cuda11.4 -f
apt-get install -y uff-converter-tf=8.2.4-1+cuda11.4 -f
apt-get install -y onnx-graphsurgeon=8.2.4-1+cuda11.4 -fadditionaly in tf_configure.bazelrc
build --action_env TF_CUDA_COMPUTE_CAPABILITIES="8.6"should change to
build --action_env TF_CUDA_COMPUTE_CAPABILITIES=<Compute Capability of Your GPU -> Can be found from CUDA Wikipedia>
Hello! I'm currently trying to reproduce your work and I'm having trouble in building your work from the part where tensorflow 2.8 should be built in the image. The current problem is this line "RUN /usr/local/lib/bazel/bin/bazel build --config=opt //tensorflow/tools/pip_package:build_pip_package"
It keeps giving me the following error :
35 5637.7 ERROR: /build/tf/tensorflow/tensorflow/compiler/tf2tensorrt/BUILD:43:11: Compiling tensorflow/compiler/tf2tensorrt/stub/nvinfer_plugin_stub.cc failed: (Exit 1): crosstool_wrapper_driver_is_not_gcc failed: error executing command external/local_config_cuda/crosstool/clang/bin/crosstool_wrapper_driver_is_not_gcc -MD -MF bazel-out/k8-opt/bin/tensorflow/compiler/tf2tensorrt/_objs/tensorrt_stub/nvinfer_plugin_stub.pic.d ... (remaining 150 argument(s) skipped)
35 5637.7 tensorflow/compiler/tf2tensorrt/stub/nvinfer_plugin_stub.cc:66:2: error: #error This version of TensorRT is not supported.
What I have tried is
Adding code to clean bazel cache following [https://github.com/tensorflow/tensorflow/issues/55662] -> "RUN /usr/local/lib/bazel/bin/bazel clean --expunge"
As the above didn't worked, I'm currently trying to build gcc-7.3.1 as from the above link, it says that setting the correct gcc-version is important and as the current gcc version is gcc-7.5.0, but as I wasn't able to find any distributions, I'm trying to build it from source using the source from [https://src.fedoraproject.org/lookaside/extras/gcc/gcc-7.3.1-20180303.tar.bz2/sha512/3c65092ea40f401c7bb5c220079f367323b389668763a93962c321a98caa4d72487748c034903f5a6381d9edadc1bc848831f3bc5404db283a234cf2d7bb82f1/] but I have little hopes for this to sceed, so any help would be greatful
Even though this is building an docker-image, this info would not be that usefull but
My current plat form is Ubuntu20.04LTS on WSL2 Docker Engine 20.10.16
The total error log is the following => ERROR [tf-builder 6/7] RUN /usr/local/lib/bazel/bin/bazel build --config=opt //tensorflow/tools/pip_package:build_pip_package 5642.0s