Closed NKCSRzChen closed 3 years ago
hello, I think the reason of this issue has already been figured out. The CUDA version of my system is 9.1, however tensorflow v1.15 requires at least CUDA 10. More details could be found in https://www.tensorflow.org/install/source#common_installation_problems
Now I wonder why I cannot build tensorflow 1.15 from source code in my system environment, yet I could use pip to install tensorflow 1.15 or 2.0 version?
Today I tried to build tensorflow 1.14 on ubuntu 16.04 with CUDA 9.1 and CUDNN 7 (using nvidia/cuda:9.1-cudnn7-devel-ubuntu16.04 docker image) and my build failed with same error:
ERROR: /opt/tensorflow/tensorflow/tensorflow/contrib/reduce_slice_ops/BUILD:50:1: Linking of rule '//tensorflow/contrib/reduce_slice_ops:gen_reduce_slice_ops_py_wrappers_cc' failed (Exit 1)
bazel-out/host/bin/_solib_local/_U_S_Stensorflow_Scontrib_Sreduce_Uslice_Uops_Cgen_Ureduce_Uslice_Uops_Upy_Uwrappers_Ucc___Utensorflow/libtensorflow_framework.so.1: undefined reference to `cublasGemmBatchedEx'
bazel-out/host/bin/_solib_local/_U_S_Stensorflow_Scontrib_Sreduce_Uslice_Uops_Cgen_Ureduce_Uslice_Uops_Upy_Uwrappers_Ucc___Utensorflow/libtensorflow_framework.so.1: undefined reference to `cublasGemmStridedBatchedEx'
collect2: error: ld returned 1 exit status
Target //tensorflow/tools/pip_package:build_pip_package failed to build
Use --verbose_failures to see the command lines of failed build steps.
INFO: Elapsed time: 813.405s, Critical Path: 96.34s
INFO: 4817 processes: 4817 local.
FAILED: Build did NOT complete successfully
FAILED: Build did NOT complete successfully
For anyone facing the same issue - I was using nvidia docker container with CUDA 9.1 and CUDNN 7. I had to add following step before building tensorflow:
echo "/usr/local/cuda-9.1/targets/x86_64-linux/lib" > /etc/ld.so.conf.d/cuda_targets.conf
ldconfig
For whatever reason /usr/local/cuda-9.1/targets/x86_64-linux/lib
was not there.
Hello,
I can confirm the same behavior while attempting to build 1.14 with CUDA coming from either Ubuntu 18.04 or Debian 9 with backports.
Any help would be greatly appreciated,
Regards, Adam.
Hi There,
We are checking to see if you still need help on this, as you are using an older version of tensorflow which is officially considered end of life . We recommend that you upgrade to the latest 2.x version and let us know if the issue still persists in newer versions. Please open a new issue for any help you need against 2.x, and we will get you the right help.
This issue will be closed automatically 7 days from now. If you still need help with this issue, please provide us with more information.
Please make sure that this is a build/installation issue. As per our GitHub Policy, we only address code/doc bugs, performance issues, feature requests and build/installation issues on GitHub. tag:build_template
System information
Describe the problem
Provide the exact sequence of commands / steps that you executed before running into the problem You have bazel 0.29.1- (@non-git) installed. Please specify the location of python. [Default is /usr/bin/python]: /usr/bin/python
Found possible Python library paths: /usr/lib/python3/dist-packages /usr/local/lib/python3.5/dist-packages Please input the desired Python library path to use. Default is [/usr/lib/python3/dist-packages] /usr/local/lib/python3.5/dist-packages Do you wish to build TensorFlow with XLA JIT support? [Y/n]: y XLA JIT support will be enabled for TensorFlow.
Do you wish to build TensorFlow with OpenCL SYCL support? [y/N]: n No OpenCL SYCL support will be enabled for TensorFlow.
Do you wish to build TensorFlow with ROCm support? [y/N]: n No ROCm support will be enabled for TensorFlow.
Do you wish to build TensorFlow with CUDA support? [y/N]: y CUDA support will be enabled for TensorFlow.
Do you wish to build TensorFlow with TensorRT support? [y/N]: n No TensorRT support will be enabled for TensorFlow.
Found CUDA 9.1 in: /usr/local/cuda/lib64 /usr/local/cuda/include Found cuDNN 7 in: /usr/local/cuda/lib64 /usr/local/cuda/include
Please specify a list of comma-separated CUDA compute capabilities you want to build with. You can find the compute capability of your device at: https://developer.nvidia.com/cuda-gpus. Please note that each additional compute capability significantly increases your build time and binary size, and that TensorFlow only supports compute capabilities >= 3.5 [Default is: 5.2]: 5.2
Do you want to use clang as CUDA compiler? [y/N]: n nvcc will be used as CUDA compiler.
Please specify which gcc should be used by nvcc as the host compiler. [Default is /usr/bin/gcc]:
Please specify optimization flags to use during compilation when bazel option "--config=opt" is specified [Default is -march=native -Wno-sign-compare]:
Would you like to interactively configure ./WORKSPACE for Android builds? [y/N]: n Not configuring the WORKSPACE for Android builds.
Preconfigured Bazel build configs. You can use any of the below by adding "--config=<>" to your build command. See .bazelrc for more details. --config=mkl # Build with MKL support. --config=monolithic # Config for mostly static monolithic build. --config=ngraph # Build with Intel nGraph support. --config=numa # Build with NUMA support. --config=dynamic_kernels # (Experimental) Build kernels into separate shared objects. --config=v2 # Build TensorFlow 2.x instead of 1.x. Preconfigured Bazel build configs to DISABLE default on features: --config=noaws # Disable AWS S3 filesystem support. --config=nogcp # Disable GCP support. --config=nohdfs # Disable HDFS support. --config=nonccl # Disable NVIDIA NCCL support. Configuration finished
$bazel build --verbose_failures --action_env=LD_LIBRARY_PATH=/usr/local/cuda/lib64 --config=cuda --config=opt //tensorflow/tools/pip_package:build_pip_package --cxxopt="-D_GLIBCXX_USE_CXX11_ABI=0" --config=nonccl --config=nohdfs --config=mkl --config=monolithic --config=noaws --config=nogcp --config=monolithic
Any other info / logs Include any logs or source code that would be helpful to diagnose the problem. If including tracebacks, please include the full traceback. Large logs and files should be attached.
ERROR: /home/tensorflow/tensorflow/tensorflow/lite/toco/BUILD:439:1: Linking of rule '//tensorflow/lite/toco:toco' failed (Exit 1) [19,054 / 21,282] 46 actions running Compiling tensorflow/core/kernels/slice_op_gpu.cu.cc [for host]; 181s local Compiling tensorflow/core/kernels/pad_op_gpu.cu.cc [for host]; 154s local Compiling tensorflow/core/kernels/tile_functor_cpu.cc [for host]; 123s local Compiling tensorflow/core/kernels/resource_variable_ops.cc [for host]; 106s local Compiling tensorflow/core/kernels/conv_ops_fused_float.cc [for host]; 94s local Compiling tensorflow/core/kernels/conv_ops_fused_double.cc [for host]; 91s local Compiling tensorflow/core/kernels/conv_ops.cc [for host]; 91s local Compiling tensorflow/core/kernels/conv_grad_ops_3d.cc [for host]; 90s local ... bazel-out/host/bin/tensorflow/stream_executor/cuda/libcublas_plugin.lo(cuda_blas.o): In function
tensorflow::Status stream_executor::gpu::CUDABlas::DoBlasGemmBatchedInternal<Eigen::half, float, cublasStatus_t (*)(cublasContext*, cublasOperation_t, cublasOperation_t, int, int, int, float const*, float const**, int, float const**, int, float const*, float**, int, int)>(cublasStatus_t (*)(cublasContext*, cublasOperation_t, cublasOperation_t, int, int, int, float const*, float const**, int, float const**, int, float const*, float**, int, int), stream_executor::Stream*, stream_executor::blas::Transpose, stream_executor::blas::Transpose, unsigned long long, unsigned long long, unsigned long long, float, absl::Span<stream_executor::DeviceMemory<Eigen::half>* const> const&, int, absl::Span<stream_executor::DeviceMemory<Eigen::half>* const> const&, int, float, absl::Span<stream_executor::DeviceMemory<Eigen::half>* const> const&, int, int, stream_executor::ScratchAllocator*)': cuda_blas.cc:(.text._ZN15stream_executor3gpu8CUDABlas25DoBlasGemmBatchedInternalIN5Eigen4halfEfPF14cublasStatus_tP13cublasContext17cublasOperation_tS8_iiiPKfPSA_iSB_iSA_PPfiiEEEN10tensorflow6StatusET1_PNS_6StreamENS_4blas9TransposeESM_yyyT0_RKN4absl4SpanIKPNS_12DeviceMemoryIT_EEEEiSX_iSN_SX_iiPNS_16ScratchAllocatorE[_ZN15stream_executor3gpu8CUDABlas25DoBlasGemmBatchedInternalIN5Eigen4halfEfPF14cublasStatus_tP13cublasContext17cublasOperation_tS8_iiiPKfPSA_iSB_iSA_PPfiiEEEN10tensorflow6StatusET1_PNS_6StreamENS_4blas9TransposeESM_yyyT0_RKN4absl4SpanIKPNS_12DeviceMemoryIT_EEEEiSX_iSN_SX_iiPNS_16ScratchAllocatorE]+0x8c2): undefined reference to
cublasGemmBatchedEx' bazel-out/host/bin/tensorflow/stream_executor/cuda/libcublas_plugin.lo(cuda_blas.o): In functiontensorflow::Status stream_executor::gpu::CUDABlas::DoBlasGemmBatchedInternal<float, float, cublasStatus_t (*)(cublasContext*, cublasOperation_t, cublasOperation_t, int, int, int, float const*, float const**, int, float const**, int, float const*, float**, int, int)>(cublasStatus_t (*)(cublasContext*, cublasOperation_t, cublasOperation_t, int, int, int, float const*, float const**, int, float const**, int, float const*, float**, int, int), stream_executor::Stream*, stream_executor::blas::Transpose, stream_executor::blas::Transpose, unsigned long long, unsigned long long, unsigned long long, float, absl::Span<stream_executor::DeviceMemory<float>* const> const&, int, absl::Span<stream_executor::DeviceMemory<float>* const> const&, int, float, absl::Span<stream_executor::DeviceMemory<float>* const> const&, int, int, stream_executor::ScratchAllocator*)': cuda_blas.cc:(.text._ZN15stream_executor3gpu8CUDABlas25DoBlasGemmBatchedInternalIffPF14cublasStatus_tP13cublasContext17cublasOperation_tS6_iiiPKfPS8_iS9_iS8_PPfiiEEEN10tensorflow6StatusET1_PNS_6StreamENS_4blas9TransposeESK_yyyT0_RKN4absl4SpanIKPNS_12DeviceMemoryIT_EEEEiSV_iSL_SV_iiPNS_16ScratchAllocatorE[_ZN15stream_executor3gpu8CUDABlas25DoBlasGemmBatchedInternalIffPF14cublasStatus_tP13cublasContext17cublasOperation_tS6_iiiPKfPS8_iS9_iS8_PPfiiEEEN10tensorflow6StatusET1_PNS_6StreamENS_4blas9TransposeESK_yyyT0_RKN4absl4SpanIKPNS_12DeviceMemoryIT_EEEEiSV_iSL_SV_iiPNS_16ScratchAllocatorE]+0x9c7): undefined reference to
cublasGemmBatchedEx' bazel-out/host/bin/tensorflow/stream_executor/cuda/libcublas_plugin.lo(cuda_blas.o): In functiontensorflow::Status stream_executor::gpu::CUDABlas::DoBlasGemmBatchedInternal<double, double, cublasStatus_t (*)(cublasContext*, cublasOperation_t, cublasOperation_t, int, int, int, double const*, double const**, int, double const**, int, double const*, double**, int, int)>(cublasStatus_t (*)(cublasContext*, cublasOperation_t, cublasOperation_t, int, int, int, double const*, double const**, int, double const**, int, double const*, double**, int, int), stream_executor::Stream*, stream_executor::blas::Transpose, stream_executor::blas::Transpose, unsigned long long, unsigned long long, unsigned long long, double, absl::Span<stream_executor::DeviceMemory<double>* const> const&, int, absl::Span<stream_executor::DeviceMemory<double>* const> const&, int, double, absl::Span<stream_executor::DeviceMemory<double>* const> const&, int, int, stream_executor::ScratchAllocator*)': cuda_blas.cc:(.text._ZN15stream_executor3gpu8CUDABlas25DoBlasGemmBatchedInternalIddPF14cublasStatus_tP13cublasContext17cublasOperation_tS6_iiiPKdPS8_iS9_iS8_PPdiiEEEN10tensorflow6StatusET1_PNS_6StreamENS_4blas9TransposeESK_yyyT0_RKN4absl4SpanIKPNS_12DeviceMemoryIT_EEEEiSV_iSL_SV_iiPNS_16ScratchAllocatorE[_ZN15stream_executor3gpu8CUDABlas25DoBlasGemmBatchedInternalIddPF14cublasStatus_tP13cublasContext17cublasOperation_tS6_iiiPKdPS8_iS9_iS8_PPdiiEEEN10tensorflow6StatusET1_PNS_6StreamENS_4blas9TransposeESK_yyyT0_RKN4absl4SpanIKPNS_12DeviceMemoryIT_EEEEiSV_iSL_SV_iiPNS_16ScratchAllocatorE]+0xceb): undefined reference to
cublasGemmBatchedEx' bazel-out/host/bin/tensorflow/stream_executor/cuda/libcublas_plugin.lo(cuda_blas.o): In functiontensorflow::Status stream_executor::gpu::CUDABlas::DoBlasGemmBatchedInternal<std::complex<float>, std::complex<float>, cublasStatus_t (*)(cublasContext*, cublasOperation_t, cublasOperation_t, int, int, int, float2 const*, float2 const**, int, float2 const**, int, float2 const*, float2**, int, int)>(cublasStatus_t (*)(cublasContext*, cublasOperation_t, cublasOperation_t, int, int, int, float2 const*, float2 const**, int, float2 const**, int, float2 const*, float2**, int, int), stream_executor::Stream*, stream_executor::blas::Transpose, stream_executor::blas::Transpose, unsigned long long, unsigned long long, unsigned long long, std::complex<float>, absl::Span<stream_executor::DeviceMemory<std::complex<float> >* const> const&, int, absl::Span<stream_executor::DeviceMemory<std::complex<float> >* const> const&, int, std::complex<float>, absl::Span<stream_executor::DeviceMemory<std::complex<float> >* const> const&, int, int, stream_executor::ScratchAllocator*)': cuda_blas.cc:(.text._ZN15stream_executor3gpu8CUDABlas25DoBlasGemmBatchedInternalISt7complexIfES4_PF14cublasStatus_tP13cublasContext17cublasOperation_tS8_iiiPK6float2PSB_iSC_iSB_PPS9_iiEEEN10tensorflow6StatusET1_PNS_6StreamENS_4blas9TransposeESN_yyyT0_RKN4absl4SpanIKPNS_12DeviceMemoryIT_EEEEiSY_iSO_SY_iiPNS_16ScratchAllocatorE[_ZN15stream_executor3gpu8CUDABlas25DoBlasGemmBatchedInternalISt7complexIfES4_PF14cublasStatus_tP13cublasContext17cublasOperation_tS8_iiiPK6float2PSB_iSC_iSB_PPS9_iiEEEN10tensorflow6StatusET1_PNS_6StreamENS_4blas9TransposeESN_yyyT0_RKN4absl4SpanIKPNS_12DeviceMemoryIT_EEEEiSY_iSO_SY_iiPNS_16ScratchAllocatorE]+0xd2b): undefined reference to
cublasGemmBatchedEx' bazel-out/host/bin/tensorflow/stream_executor/cuda/libcublas_plugin.lo(cuda_blas.o): In functiontensorflow::Status stream_executor::gpu::CUDABlas::DoBlasGemmBatchedInternal<std::complex<double>, std::complex<double>, cublasStatus_t (*)(cublasContext*, cublasOperation_t, cublasOperation_t, int, int, int, double2 const*, double2 const**, int, double2 const**, int, double2 const*, double2**, int, int)>(cublasStatus_t (*)(cublasContext*, cublasOperation_t, cublasOperation_t, int, int, int, double2 const*, double2 const**, int, double2 const**, int, double2 const*, double2**, int, int), stream_executor::Stream*, stream_executor::blas::Transpose, stream_executor::blas::Transpose, unsigned long long, unsigned long long, unsigned long long, std::complex<double>, absl::Span<stream_executor::DeviceMemory<std::complex<double> >* const> const&, int, absl::Span<stream_executor::DeviceMemory<std::complex<double> >* const> const&, int, std::complex<double>, absl::Span<stream_executor::DeviceMemory<std::complex<double> >* const> const&, int, int, stream_executor::ScratchAllocator*)': cuda_blas.cc:(.text._ZN15stream_executor3gpu8CUDABlas25DoBlasGemmBatchedInternalISt7complexIdES4_PF14cublasStatus_tP13cublasContext17cublasOperation_tS8_iiiPK7double2PSB_iSC_iSB_PPS9_iiEEEN10tensorflow6StatusET1_PNS_6StreamENS_4blas9TransposeESN_yyyT0_RKN4absl4SpanIKPNS_12DeviceMemoryIT_EEEEiSY_iSO_SY_iiPNS_16ScratchAllocatorE[_ZN15stream_executor3gpu8CUDABlas25DoBlasGemmBatchedInternalISt7complexIdES4_PF14cublasStatus_tP13cublasContext17cublasOperation_tS8_iiiPK7double2PSB_iSC_iSB_PPS9_iiEEEN10tensorflow6StatusET1_PNS_6StreamENS_4blas9TransposeESN_yyyT0_RKN4absl4SpanIKPNS_12DeviceMemoryIT_EEEEiSY_iSO_SY_iiPNS_16ScratchAllocatorE]+0xcfb): undefined reference to
cublasGemmBatchedEx' bazel-out/host/bin/tensorflow/stream_executor/cuda/libcublas_plugin.lo(cuda_blas.o): In functionstream_executor::gpu::CUDABlas::DoBlasGemmStridedBatched(stream_executor::Stream*, stream_executor::blas::Transpose, stream_executor::blas::Transpose, unsigned long long, unsigned long long, unsigned long long, float, stream_executor::DeviceMemory<Eigen::half> const&, int, long long, stream_executor::DeviceMemory<Eigen::half> const&, int, long long, float, stream_executor::DeviceMemory<Eigen::half>*, int, long long, int)': cuda_blas.cc:(.text._ZN15stream_executor3gpu8CUDABlas24DoBlasGemmStridedBatchedEPNS_6StreamENS_4blas9TransposeES5_yyyfRKNS_12DeviceMemoryIN5Eigen4halfEEEixSB_ixfPS9_ixi+0x2e3): undefined reference to
cublasGemmStridedBatchedEx' collect2: error: ld returned 1 exit status [19,054 / 21,282] 46 actions running Compiling tensorflow/core/kernels/slice_op_gpu.cu.cc [for host]; 181s local Compiling tensorflow/core/kernels/pad_op_gpu.cu.cc [for host]; 154s local Compiling tensorflow/core/kernels/tile_functor_cpu.cc [for host]; 123s local Compiling tensorflow/core/kernels/resource_variable_ops.cc [for host]; 106s local Compiling tensorflow/core/kernels/conv_ops_fused_float.cc [for host]; 94s local Compiling tensorflow/core/kernels/conv_ops_fused_double.cc [for host]; 91s local Compiling tensorflow/core/kernels/conv_ops.cc [for host]; 91s local Compiling tensorflow/core/kernels/conv_grad_ops_3d.cc [for host]; 90s local ... Target //tensorflow/tools/pip_package:build_pip_package failed to build [19,101 / 21,282] checking cached actions INFO: Elapsed time: 1270.016s, Critical Path: 336.18s [19,101 / 21,282] checking cached actions INFO: 10667 processes: 10667 local. [19,101 / 21,282] checking cached actions FAILED: Build did NOT complete successfully FAILED: Build did NOT complete successfullyI just search all the issues about "undefined reference to `cublasGemmStridedBatchedEx'", but I believe mine is different.