mcdonasd1212 commented 2 years ago

System information (Please provide as much relevant information as possible)

OS Platform and Distribution: Ubuntu 18.04
Compiler version: gcc / g++ 7.5
Programming Language and version: Python 3.7
Installed using virtualenv? pip? Conda? (if python): virualenv pip
MediaPipe version: cf101e6
Bazel version: 4.2.1
OpenCV version (if running on desktop): 4.5.3 custom build
Cuda: 11.3 / cudnn: 8.2.1

Describe the problem: Trying to build MediaPipe with TensorFlow GPU per instructions https://google.github.io/mediapipe/getting_started/gpu_support.html#tensorflow-cuda-support-and-setup-on-linux-desktop and get this error

`ERROR: /home/feedback/.cache/bazel/_bazel_feedback/2fe47e1d8cb429227698b0916e2803b6/external/org_tensorflow/tensorflow/compiler/mlir/tensorflow/BUILD:573:11: Compiling tensorflow/compiler/mlir/tensorflow/ir/tf_ops.cc failed: (Exit 4): crosstool_wrapper_driver_is_not_gcc failed: error executing command external/local_config_cuda/crosstool/clang/bin/crosstool_wrapper_driver_is_not_gcc -MD -MF ... (remaining 240 argument(s) skipped)
In file included from external/org_tensorflow/tensorflow/core/common_runtime/inline_function_utils.h:25:0,
                 from external/org_tensorflow/tensorflow/compiler/mlir/tensorflow/ir/tf_ops.cc:71:
external/org_tensorflow/tensorflow/core/common_runtime/lower_function_call_inline_policy.h:38:1: warning: multi-line comment [-Wcomment]
 // LINT.ThenChange(inline_function_utils.h,\
 ^
gcc: internal compiler error: Killed (program cc1plus)
Please submit a full bug report,
with preprocessed source if appropriate.
See <file:///usr/share/doc/gcc-7/README.Bugs> for instructions.
Target //mediapipe/examples/desktop/object_detection:object_detection_tensorflow failed to build
Use --verbose_failures to see the command lines of failed build steps.
INFO: Elapsed time: 1882.613s, Critical Path: 622.56s
INFO: 6166 processes: 247 internal, 5919 local.
FAILED: Build did NOT complete successfully`

tensorboard 2.8.0 tensorboard-data-server 0.6.1 tensorboard-plugin-wit 1.8.1 tensorflow 2.7.0 tensorflow-estimator 2.7.0 tensorflow-io-gcs-filesystem 0.23.1

mcdonasd1212 commented 2 years ago

I will uninstall Cuda 11.3 an go with 10.2 (that will be fun!) and see what happens. Will I need to build Tensorflow from source our can I use the version installed via Pip?

mcdonasd1212 commented 2 years ago

Hi @sgowroji

I started from a fresh install. So here is every step.

OS

Ubuntu 18.04

Hardware

Intel® Core™ i7-8700 CPU @ 3.20GHz × 12 GeForce GTX 1660 SUPER/PCIe/SSE2

PREREQUISITES

$ sudo apt update
$ sudo apt install build-essential
$ gcc --version
$ sudo apt install -y libfreeimage-dev

Intall Cuda and cuDnn

Install Cuda 11.2 (Highest version that works with Tensorflow per https://www.tensorflow.org/install/gpu)

https://developer.nvidia.com/cuda-11.2.2-download-archive?target_os=Linux&target_arch=x86_64&target_distro=Ubuntu&target_version=1804&target_type=runfilelocal

$ echo "export PATH=/usr/local/cuda-11.2/bin${PATH:+:${PATH}}" >> ~/.bashrc $ source ~/.bashrc

Install cuDNN 8.2.1

https://docs.nvidia.com/deeplearning/cudnn/archives/cudnn-821/install-guide/index.html -- make sure you verify installation per the Nvidia cuDNN Documentation -- Install packages will say Cuda 11.3 but that's ok compatibility is shown as 11.X

Verify Driver 460 and Cuda 11.2
```
$ nvcc --version
$ nvidia-smi
```

Make virutal enviroment

$ sudo apt update
$ sudo apt upgrade
$ sudo apt install python3-pip
$ sudo apt install software-properties-common
$ sudo add-apt-repository ppa:deadsnakes/ppa
$ sudo apt install python3.7
$ python3.7 --version
$ sudo apt install python3.7-dev
$ sudo pip3 install virtualenv virtualenvwrapper
$ sudo rm -rf ~/get-pip.py ~/.cache/pip
$ echo -e "\n# virtualenv and virtualenvwrapper" >> ~/.bashrc
$ echo "export WORKON_HOME=$HOME/.virtualenvs" >> ~/.bashrc
$ echo "export VIRTUALENVWRAPPER_PYTHON=/usr/bin/python3" >> ~/.bashrc 
$ echo "source /usr/local/bin/virtualenvwrapper.sh" >> ~/.bashrc
$ echo "export PYTHONPATH=/usr/lib/python3.7/dist-packages:$PYTHONPATH" >> ~/.bashrc
$ source ~/.bashrc
$ mkvirtualenv  mp_env -p python3.7
$ workon mp_env

Build OpenCV from source

$ sudo apt purge libreoffice*
$ sudo apt clean
$ sudo apt update && sudo apt-get upgrade
$ sudo apt install git
$ sudo apt install -y python-configparser
$ sudo apt install unixodbc-dev
$ sudo apt install python-cffi
$ sudo apt install -y libffi-dev
$ sudo apt install python3-matplotlib
$ sudo apt install libtbb-dev   
$ sudo apt install curl 
$ sudo apt install ffmpeg   
$ sudo apt install -y python-yaml
$ sudo apt install build-essential cmake unzip pkg-config
$ sudo apt install libjpeg-dev libpng-dev libtiff-dev
$ sudo apt install libavcodec-dev libavformat-dev libswscale-dev libv4l-dev
$ sudo apt install libxvidcore-dev libx264-dev
$ sudo apt install libgtk-3-dev
$ sudo apt install libatlas-base-dev gfortran
$ sudo apt install libgles2-mesa-dev

$ cd ~
$ wget -O opencv.zip https://github.com/opencv/opencv/archive/4.5.3.zip
$ wget -O opencv_contrib.zip https://github.com/opencv/opencv_contrib/archive/4.5.3.zip
$ unzip opencv.zip
$ unzip opencv_contrib.zip
$ mv opencv-4.5.3 opencv
$ mv opencv_contrib-4.5.3 opencv_contrib
$ workon mp_env #This is important for the build to have the correct paths
[mp_env]$ pip install numpy
[mp_env]$ cd ~/opencv
[mp_env]$ mkdir build
[mp_env]$ cd build

[mp_env]$ cmake -D CMAKE_BUILD_TYPE=RELEASE \
        -D CMAKE_INSTALL_PREFIX=/usr/local \
        -D INSTALL_PYTHON_EXAMPLES=ON \
        -D INSTALL_C_EXAMPLES=OFF \
        -D OPENCV_ENABLE_NONFREE=ON \
        -D OPENCV_EXTRA_MODULES_PATH=~/opencv_contrib/modules \
        -D PYTHON_EXECUTABLE=~/.virtualenvs/mp_env/bin/python \
        -D WITH_TBB=ON \
        -D WITH_CUDA=ON \
        -D BUILD_opencv_cudacodec=ON \
        -D ENABLE_FAST_MATH=1 \
        -D CUDA_FAST_MATH=1 \
        -D WITH_CUBLAS=1 \
        -D WITH_V4L=ON \
        -D WITH_GSTREAMER=ON \
        -D OPENCV_GENERATE_PKGCONFIG=ON \
        -D OPENCV_PC_FILE_NAME=opencv.pc \
        -D BUILD_EXAMPLES=ON ..

Build worked if Python 3 there is a path to the virtual env (python 3.7)
```
$ make -j12
```
Go for a walk
```
$ sudo make install
$ sudo ldconfig
```
move the file into the virual enviroment $ cd ~ $ cd /usr/local/lib/python3.7/site-packages/cv2/python-3.7/ $ sudo mv cv2.cpython-37m-x86_64-linux-gnu.so cv2.so $ cd ~/.virtualenvs/mp_env/lib/python3.7/site-packages/ $ ln -s /usr/local/lib/python3.7/site-packages/cv2/python-3.7/cv2.so cv2.so

Verify cv2 install

$ workon mp_env
[mp_env] $ python
>>> import cv2
>>> cv2.__version__
'4.5.3'
>>>print(cv2.cuda.getCudaEnabledDeviceCount())
'1'
>>>print (cv2.__file__)
>>>print (cv2.getBuildInformation())
>>>exit()

Build MediaPipe for GPU

Instructions to Build from Source

https://google.github.io/mediapipe/getting_started/install.html

Step 1: Install Bazelisk use this link https://docs.bazel.build/versions/main/install-ubuntu.html
```
sudo apt update && sudo apt install bazel
sudo apt install bazel-4.2.1
```
Step 2: Follow as is
Step 3: Use Option 3 (not OpenCV has already been built from source) -- In the the WORKSPACE file do a search for new_local_repository update /usr to /usr/local -- In opencv_linux uncomment lines required for OpenCV build from source DO NOT copy and past provided example. -- For ffmpeg no change required
Step 4 and 5 are not required

https://google.github.io/mediapipe/getting_started/python.html#building-mediapipe-python-package

Step 1: skip sudo apt install python3-dev and sudo apt install cmake
Step 2: skip

Step 3: Use

$ cd mediapipe
$ workon mp_env 
[mp_env] $ pip3 install -r requirements.txt
[mp_env] $ pip install tensorflow

Verify tensorflow install

$ workon mp_env
[mp_env] $ python
>>> import tensorflow as tf
>>> tf.__version__
'2.7.0'
>>>print(cv2.cuda.getCudaEnabledDeviceCount())
'1'
>>>exit()

Step 6: skip
Instructions to Build with GPU support (All TensorFlow requirements are meet) https://google.github.io/mediapipe/getting_started/gpu_support.html#tensorflow-cuda-support-and-setup-on-linux-desktop

Update ~/.bashrc

$ echo "export PATH=/usr/local/cuda-11.2/bin${PATH:+:${PATH}}" >> ~/.bashrc
$ echo "export LD_LIBRARY_PATH=/usr/local/cuda/extras/CUPTI/lib64,/usr/local/cuda-11.2/lib64${LD_LIBRARY_PATH:+:${LD_LIBRARY_PATH}}" >> ~/.bashrc
$ echo "export TF_CUDA_PATHS=/usr/local/cuda-11.2,/usr/lib/x86_64-linux-gnu,/usr/include" >> ~/.bashrc
$ source ~/.bashrc
$ sudo ldconfig

Make sure hidden files are visiabile and update the .bazelrc file with reference from tensorflow .bazelrc file

$ workon mp_env
[mp_env] $ cd mediapipe
[mp_env] $ bazel build -c opt --config=cuda --spawn_strategy=local \
        --define no_aws_support=true --copt -DMESA_EGL_NO_X11_HEADERS \
        mediapipe/examples/desktop/object_detection:object_detection_tensorflow

Same error as originally reported.

mcdonasd1212 commented 2 years ago

@mcclanahoochie , @jiuqiant - any recommendations of what to try next?

homuler commented 2 years ago

gcc: internal compiler error: Killed (program cc1plus)

Isn't the process killed by OOM killer?

mcdonasd1212 commented 2 years ago

@homuler yes it was a memory problem, Thank you.

I tried to compile again this time with the I added --jobs=1 to get around the memory issue.

bazel build --jobs=1 -c opt --config=cuda --spawn_strategy=local \
            --define no_aws_support=true --copt -DMESA_EGL_NO_X11_HEADERS \
            mediapipe/examples/desktop/object_detection:object_detection_tensorflow

Here is the error.

INFO: From Compiling tensorflow/core/kernels/histogram_op_gpu.cu.cc:
nvcc warning : The 'compute_35', 'compute_37', 'compute_50', 'sm_35', 'sm_37' and 'sm_50' architectures are deprecated, and may be removed in a future release (Use -Wno-deprecated-gpu-targets to suppress warning).
nvcc warning : The 'compute_35', 'compute_37', 'compute_50', 'sm_35', 'sm_37' and 'sm_50' architectures are deprecated, and may be removed in a future release (Use -Wno-deprecated-gpu-targets to suppress warning).
ERROR: /home/feedback/mediapipe/mediapipe/examples/desktop/object_detection/BUILD:19:10: Linking mediapipe/examples/desktop/object_detection/object_detection_tensorflow failed: (Exit 1): crosstool_wrapper_driver_is_not_gcc failed: error executing command external/local_config_cuda/crosstool/clang/bin/crosstool_wrapper_driver_is_not_gcc @bazel-out/k8-opt/bin/mediapipe/examples/desktop/object_detection/object_detection_tensorflow-2.params
bazel-out/k8-opt/bin/external/org_tensorflow/tensorflow/core/kernels/libunique_op_gpu.lo(unique_op_gpu_0.cu.o): In function `tensorflow::kernel_factory::OpKernelRegistrar::OpKernelRegistrar(tensorflow::KernelDef const*, absl::lts_20210324::string_view, tensorflow::OpKernel* (*)(tensorflow::OpKernelConstruction*))':
tmpxft_00006765_00000000-6_unique_op_gpu_0.cu.compute_52.cudafe1.cpp:(.text._ZN10tensorflow14kernel_factory17OpKernelRegistrarC2EPKNS_9KernelDefEN4absl12lts_2021032411string_viewEPFPNS_8OpKernelEPNS_20OpKernelConstructionEE[_ZN10tensorflow14kernel_factory17OpKernelRegistrarC5EPKNS_9KernelDefEN4absl12lts_2021032411string_viewEPFPNS_8OpKernelEPNS_20OpKernelConstructionEE]+0x60): undefined reference to `tensorflow::kernel_factory::OpKernelRegistrar::InitInternal(tensorflow::KernelDef const*, absl::lts_20210324::string_view, std::unique_ptr<tensorflow::kernel_factory::OpKernelFactory, std::default_delete<tensorflow::kernel_factory::OpKernelFactory> >)'
bazel-out/k8-opt/bin/external/org_tensorflow/tensorflow/core/kernels/libunique_op_gpu.lo(unique_op_gpu_0.cu.o): In function `tensorflow::Status tensorflow::errors::Internal<char const*, char const*>(char const*, char const*)':
tmpxft_00006765_00000000-6_unique_op_gpu_0.cu.compute_52.cudafe1.cpp:(.text._ZN10tensorflow6errors8InternalIJPKcS3_EEENS_6StatusEDpT_[_ZN10tensorflow6errors8InternalIJPKcS3_EEENS_6StatusEDpT_]+0xea): undefined reference to `tensorflow::Status::Status(tensorflow::error::Code, absl::lts_20210324::string_view, std::vector<tensorflow::StackFrame, std::allocator<tensorflow::StackFrame> >&&)'
bazel-out/k8-opt/bin/external/org_tensorflow/tensorflow/core/kernels/libunique_op_gpu.lo(unique_op_gpu_0.cu.o): In function `tensorflow::Status tensorflow::errors::Internal<char const*, unsigned long, char const*, char const*>(char const*, unsigned long, char const*, char const*)':
tmpxft_00006765_00000000-6_unique_op_gpu_0.cu.compute_52.cudafe1.cpp:(.text._ZN10tensorflow6errors8InternalIJPKcmS3_S3_EEENS_6StatusEDpT_[_ZN10tensorflow6errors8InternalIJPKcmS3_S3_EEENS_6StatusEDpT_]+0x55f): undefined reference to `tensorflow::Status::Status(tensorflow::error::Code, absl::lts_20210324::string_view, std::vector<tensorflow::StackFrame, std::allocator<tensorflow::StackFrame> >&&)'
bazel-out/k8-opt/bin/external/org_tensorflow/tensorflow/core/kernels/libdynamic_partition_op_gpu.lo(dynamic_partition_op_gpu.cu.o): In function `tensorflow::{lambda(tensorflow::KernelDef const*)#1}::operator()(tensorflow::KernelDef const*) const::{lambda(tensorflow::OpKernelConstruction*)#1}::_FUN(tensorflow::OpKernelConstruction)':
tmpxft_000023a9_00000000-6_dynamic_partition_op_gpu.cu.compute_52.cudafe1.cpp:(.text+0x42c): undefined reference to `tensorflow::GetNodeAttr(tensorflow::AttrSlice const&, absl::lts_20210324::string_view, int*)'
bazel-out/k8-opt/bin/external/org_tensorflow/tensorflow/core/kernels/libdynamic_partition_op_gpu.lo(dynamic_partition_op_gpu.cu.o): In function `tensorflow::{lambda(tensorflow::KernelDef const*)#2}::operator()(tensorflow::KernelDef const*) const::{lambda(tensorflow::OpKernelConstruction*)#1}::_FUN(tensorflow::OpKernelConstruction)':
tmpxft_000023a9_00000000-6_dynamic_partition_op_gpu.cu.compute_52.cudafe1.cpp:(.text+0x59c): undefined reference to `tensorflow::GetNodeAttr(tensorflow::AttrSlice const&, absl::lts_20210324::string_view, int*)'
bazel-out/k8-opt/bin/external/org_tensorflow/tensorflow/core/kernels/libdynamic_partition_op_gpu.lo(dynamic_partition_op_gpu.cu.o): In function `tensorflow::{lambda(tensorflow::KernelDef const*)#3}::operator()(tensorflow::KernelDef const*) const::{lambda(tensorflow::OpKernelConstruction*)#1}::_FUN(tensorflow::OpKernelConstruction)':
tmpxft_000023a9_00000000-6_dynamic_partition_op_gpu.cu.compute_52.cudafe1.cpp:(.text+0x70c): undefined reference to `tensorflow::GetNodeAttr(tensorflow::AttrSlice const&, absl::lts_20210324::string_view, int*)'
bazel-out/k8-opt/bin/external/org_tensorflow/tensorflow/core/kernels/libdynamic_partition_op_gpu.lo(dynamic_partition_op_gpu.cu.o): In function `tensorflow::{lambda(tensorflow::KernelDef const*)#4}::operator()(tensorflow::KernelDef const*) const::{lambda(tensorflow::OpKernelConstruction*)#1}::_FUN(tensorflow::OpKernelConstruction)':
tmpxft_000023a9_00000000-6_dynamic_partition_op_gpu.cu.compute_52.cudafe1.cpp:(.text+0x87c): undefined reference to `tensorflow::GetNodeAttr(tensorflow::AttrSlice const&, absl::lts_20210324::string_view, int*)'
bazel-out/k8-opt/bin/external/org_tensorflow/tensorflow/core/kernels/libdynamic_partition_op_gpu.lo(dynamic_partition_op_gpu.cu.o): In function `tensorflow::{lambda(tensorflow::KernelDef const*)#5}::operator()(tensorflow::KernelDef const*) const::{lambda(tensorflow::OpKernelConstruction*)#1}::_FUN(tensorflow::OpKernelConstruction)':
tmpxft_000023a9_00000000-6_dynamic_partition_op_gpu.cu.compute_52.cudafe1.cpp:(.text+0x9ec): undefined reference to `tensorflow::GetNodeAttr(tensorflow::AttrSlice const&, absl::lts_20210324::string_view, int*)'
bazel-out/k8-opt/bin/external/org_tensorflow/tensorflow/core/kernels/libdynamic_partition_op_gpu.lo(dynamic_partition_op_gpu.cu.o): In function `tensorflow::DynamicPartitionOpGPU<Eigen::half>::AllocateOutputs(tensorflow::OpKernelContext*, tensorflow::Tensor const*, tensorflow::Tensor const*, tensorflow::Tensor const*, tensorflow::OpOutputList*, std::function<void ()>)':
tmpxft_000023a9_00000000-6_dynamic_partition_op_gpu.cu.compute_52.cudafe1.cpp:(.text._ZN10tensorflow21DynamicPartitionOpGPUIN5Eigen4halfEE15AllocateOutputsEPNS_15OpKernelContextEPKNS_6TensorES8_S8_PNS_12OpOutputListESt8functionIFvvEE[_ZN10tensorflow21DynamicPartitionOpGPUIN5Eigen4halfEE15AllocateOutputsEPNS_15OpKernelContextEPKNS_6TensorES8_S8_PNS_12OpOutputListESt8functionIFvvEE]+0x83): undefined reference to `tensorflow::OpKernelContext::output_list(absl::lts_20210324::string_view, tensorflow::OpOutputList*)'
bazel-out/k8-opt/bin/external/org_tensorflow/tensorflow/core/kernels/libdynamic_partition_op_gpu.lo(dynamic_partition_op_gpu.cu.o): In function `tensorflow::DynamicPartitionOpGPU<float>::AllocateOutputs(tensorflow::OpKernelContext*, tensorflow::Tensor const*, tensorflow::Tensor const*, tensorflow::Tensor const*, tensorflow::OpOutputList*, std::function<void ()>)':
tmpxft_000023a9_00000000-6_dynamic_partition_op_gpu.cu.compute_52.cudafe1.cpp:(.text._ZN10tensorflow21DynamicPartitionOpGPUIfE15AllocateOutputsEPNS_15OpKernelContextEPKNS_6TensorES6_S6_PNS_12OpOutputListESt8functionIFvvEE[_ZN10tensorflow21DynamicPartitionOpGPUIfE15AllocateOutputsEPNS_15OpKernelContextEPKNS_6TensorES6_S6_PNS_12OpOutputListESt8functionIFvvEE]+0x83): undefined reference to `tensorflow::OpKernelContext::output_list(absl::lts_20210324::string_view, tensorflow::OpOutputList*)'
bazel-out/k8-opt/bin/external/org_tensorflow/tensorflow/core/kernels/libdynamic_partition_op_gpu.lo(dynamic_partition_op_gpu.cu.o): In function `tensorflow::DynamicPartitionOpGPU<double>::AllocateOutputs(tensorflow::OpKernelContext*, tensorflow::Tensor const*, tensorflow::Tensor const*, tensorflow::Tensor const*, tensorflow::OpOutputList*, std::function<void ()>)':
tmpxft_000023a9_00000000-6_dynamic_partition_op_gpu.cu.compute_52.cudafe1.cpp:(.text._ZN10tensorflow21DynamicPartitionOpGPUIdE15AllocateOutputsEPNS_15OpKernelContextEPKNS_6TensorES6_S6_PNS_12OpOutputListESt8functionIFvvEE[_ZN10tensorflow21DynamicPartitionOpGPUIdE15AllocateOutputsEPNS_15OpKernelContextEPKNS_6TensorES6_S6_PNS_12OpOutputListESt8functionIFvvEE]+0x83): undefined reference to `tensorflow::OpKernelContext::output_list(absl::lts_20210324::string_view, tensorflow::OpOutputList*)'
bazel-out/k8-opt/bin/external/org_tensorflow/tensorflow/core/kernels/libdynamic_partition_op_gpu.lo(dynamic_partition_op_gpu.cu.o): In function `tensorflow::DynamicPartitionOpGPU<std::complex<float> >::AllocateOutputs(tensorflow::OpKernelContext*, tensorflow::Tensor const*, tensorflow::Tensor const*, tensorflow::Tensor const*, tensorflow::OpOutputList*, std::function<void ()>)':
tmpxft_000023a9_00000000-6_dynamic_partition_op_gpu.cu.compute_52.cudafe1.cpp:(.text._ZN10tensorflow21DynamicPartitionOpGPUISt7complexIfEE15AllocateOutputsEPNS_15OpKernelContextEPKNS_6TensorES8_S8_PNS_12OpOutputListESt8functionIFvvEE[_ZN10tensorflow21DynamicPartitionOpGPUISt7complexIfEE15AllocateOutputsEPNS_15OpKernelContextEPKNS_6TensorES8_S8_PNS_12OpOutputListESt8functionIFvvEE]+0x83): undefined reference to `tensorflow::OpKernelContext::output_list(absl::lts_20210324::string_view, tensorflow::OpOutputList*)'
bazel-out/k8-opt/bin/external/org_tensorflow/tensorflow/core/kernels/libdynamic_partition_op_gpu.lo(dynamic_partition_op_gpu.cu.o): In function `tensorflow::DynamicPartitionOpGPU<std::complex<double> >::AllocateOutputs(tensorflow::OpKernelContext*, tensorflow::Tensor const*, tensorflow::Tensor const*, tensorflow::Tensor const*, tensorflow::OpOutputList*, std::function<void ()>)':
tmpxft_000023a9_00000000-6_dynamic_partition_op_gpu.cu.compute_52.cudafe1.cpp:(.text._ZN10tensorflow21DynamicPartitionOpGPUISt7complexIdEE15AllocateOutputsEPNS_15OpKernelContextEPKNS_6TensorES8_S8_PNS_12OpOutputListESt8functionIFvvEE[_ZN10tensorflow21DynamicPartitionOpGPUISt7complexIdEE15AllocateOutputsEPNS_15OpKernelContextEPKNS_6TensorES8_S8_PNS_12OpOutputListESt8functionIFvvEE]+0x83): undefined reference to `tensorflow::OpKernelContext::output_list(absl::lts_20210324::string_view, tensorflow::OpOutputList*)'
bazel-out/k8-opt/bin/external/org_tensorflow/tensorflow/core/kernels/libdynamic_partition_op_gpu.lo(dynamic_partition_op_gpu.cu.o):tmpxft_000023a9_00000000-6_dynamic_partition_op_gpu.cu.compute_52.cudafe1.cpp:(.text._ZNSt17_Function_handlerIFvvEZN10tensorflow21DynamicPartitionOpGPUIN5Eigen4halfEE12ComputeAsyncEPNS1_15OpKernelContextESt8functionIS0_EEUlvE_E9_M_invokeERKSt9_Any_data[_ZNSt17_Function_handlerIFvvEZN10tensorflow21DynamicPartitionOpGPUIN5Eigen4halfEE12ComputeAsyncEPNS1_15OpKernelContextESt8functionIS0_EEUlvE_E9_M_invokeERKSt9_Any_data]+0x130): more undefined references to `tensorflow::OpKernelContext::output_list(absl::lts_20210324::string_view, tensorflow::OpOutputList*)' follow
bazel-out/k8-opt/bin/external/org_tensorflow/tensorflow/core/kernels/libbroadcast_to_op_gpu.lo(broadcast_to_op_gpu.cu.o): In function `tensorflow::Status tensorflow::errors::Unimplemented<char const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, char const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, char const*>(char const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, char const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, char const*)':
tmpxft_000002a0_00000000-6_broadcast_to_op_gpu.cu.compute_52.cudafe1.cpp:(.text._ZN10tensorflow6errors13UnimplementedIJPKcNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEES3_S9_S3_EEENS_6StatusEDpT_[_ZN10tensorflow6errors13UnimplementedIJPKcNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEES3_S9_S3_EEENS_6StatusEDpT_]+0x193): undefined reference to `tensorflow::strings::internal::CatPieces[abi:cxx11](std::initializer_list<absl::lts_20210324::string_view>)'
tmpxft_000002a0_00000000-6_broadcast_to_op_gpu.cu.compute_52.cudafe1.cpp:(.text._ZN10tensorflow6errors13UnimplementedIJPKcNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEES3_S9_S3_EEENS_6StatusEDpT_[_ZN10tensorflow6errors13UnimplementedIJPKcNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEES3_S9_S3_EEENS_6StatusEDpT_]+0x1d2): undefined reference to `tensorflow::Status::Status(tensorflow::error::Code, absl::lts_20210324::string_view, std::vector<tensorflow::StackFrame, std::allocator<tensorflow::StackFrame> >&&)'
bazel-out/k8-opt/bin/external/org_tensorflow/tensorflow/core/kernels/libsparse_fill_empty_rows_op_gpu.lo(sparse_fill_empty_rows_op_gpu.cu.o): In function `tensorflow::functor::SparseFillEmptyRows<Eigen::GpuDevice, unsigned long, long>::AllocateOutputsExceptEmptyRowIndicator(tensorflow::OpKernelContext*, long, int, long, long**, unsigned long**, long**)':
tmpxft_00000eea_00000000-6_sparse_fill_empty_rows_op_gpu.cu.compute_52.cudafe1.cpp:(.text._ZN10tensorflow7functor19SparseFillEmptyRowsIN5Eigen9GpuDeviceEmlE38AllocateOutputsExceptEmptyRowIndicatorEPNS_15OpKernelContextElilPPlPPmS8_[_ZN10tensorflow7functor19SparseFillEmptyRowsIN5Eigen9GpuDeviceEmlE38AllocateOutputsExceptEmptyRowIndicatorEPNS_15OpKernelContextElilPPlPPmS8_]+0xa1): undefined reference to `tensorflow::OpKernelContext::allocate_output(absl::lts_20210324::string_view, tensorflow::TensorShape const&, tensorflow::Tensor**)'
tmpxft_00000eea_00000000-6_sparse_fill_empty_rows_op_gpu.cu.compute_52.cudafe1.cpp:(.text._ZN10tensorflow7functor19SparseFillEmptyRowsIN5Eigen9GpuDeviceEmlE38AllocateOutputsExceptEmptyRowIndicatorEPNS_15OpKernelContextElilPPlPPmS8_[_ZN10tensorflow7functor19SparseFillEmptyRowsIN5Eigen9GpuDeviceEmlE38AllocateOutputsExceptEmptyRowIndicatorEPNS_15OpKernelContextElilPPlPPmS8_]+0xfd): undefined reference to `tensorflow::OpKernelContext::allocate_output(absl::lts_20210324::string_view, tensorflow::TensorShape const&, tensorflow::Tensor**)'
bazel-out/k8-opt/bin/external/org_tensorflow/tensorflow/core/kernels/libsparse_fill_empty_rows_op_gpu.lo(sparse_fill_empty_rows_op_gpu.cu.o): In function `tensorflow::functor::SparseFillEmptyRows<Eigen::GpuDevice, long, long>::AllocateOutputsExceptEmptyRowIndicator(tensorflow::OpKernelContext*, long, int, long, long**, long**, long**)':
tmpxft_00000eea_00000000-6_sparse_fill_empty_rows_op_gpu.cu.compute_52.cudafe1.cpp:(.text._ZN10tensorflow7functor19SparseFillEmptyRowsIN5Eigen9GpuDeviceEllE38AllocateOutputsExceptEmptyRowIndicatorEPNS_15OpKernelContextElilPPlS8_S8_[_ZN10tensorflow7functor19SparseFillEmptyRowsIN5Eigen9GpuDeviceEllE38AllocateOutputsExceptEmptyRowIndicatorEPNS_15OpKernelContextElilPPlS8_S8_]+0xa0): undefined reference to `tensorflow::OpKernelContext::allocate_output(absl::lts_20210324::string_view, tensorflow::TensorShape const&, tensorflow::Tensor**)'
tmpxft_00000eea_00000000-6_sparse_fill_empty_rows_op_gpu.cu.compute_52.cudafe1.cpp:(.text._ZN10tensorflow7functor19SparseFillEmptyRowsIN5Eigen9GpuDeviceEllE38AllocateOutputsExceptEmptyRowIndicatorEPNS_15OpKernelContextElilPPlS8_S8_[_ZN10tensorflow7functor19SparseFillEmptyRowsIN5Eigen9GpuDeviceEllE38AllocateOutputsExceptEmptyRowIndicatorEPNS_15OpKernelContextElilPPlS8_S8_]+0x100): undefined reference to `tensorflow::OpKernelContext::allocate_output(absl::lts_20210324::string_view, tensorflow::TensorShape const&, tensorflow::Tensor**)'
bazel-out/k8-opt/bin/external/org_tensorflow/tensorflow/core/kernels/libsparse_fill_empty_rows_op_gpu.lo(sparse_fill_empty_rows_op_gpu.cu.o): In function `tensorflow::functor::SparseFillEmptyRows<Eigen::GpuDevice, unsigned int, long>::AllocateOutputsExceptEmptyRowIndicator(tensorflow::OpKernelContext*, long, int, long, long**, unsigned int**, long**)':
tmpxft_00000eea_00000000-6_sparse_fill_empty_rows_op_gpu.cu.compute_52.cudafe1.cpp:(.text._ZN10tensorflow7functor19SparseFillEmptyRowsIN5Eigen9GpuDeviceEjlE38AllocateOutputsExceptEmptyRowIndicatorEPNS_15OpKernelContextElilPPlPPjS8_[_ZN10tensorflow7functor19SparseFillEmptyRowsIN5Eigen9GpuDeviceEjlE38AllocateOutputsExceptEmptyRowIndicatorEPNS_15OpKernelContextElilPPlPPjS8_]+0xa1): undefined reference to `tensorflow::OpKernelContext::allocate_output(absl::lts_20210324::string_view, tensorflow::TensorShape const&, tensorflow::Tensor**)'
bazel-out/k8-opt/bin/external/org_tensorflow/tensorflow/core/kernels/libsparse_fill_empty_rows_op_gpu.lo(sparse_fill_empty_rows_op_gpu.cu.o):tmpxft_00000eea_00000000-6_sparse_fill_empty_rows_op_gpu.cu.compute_52.cudafe1.cpp:(.text._ZN10tensorflow7functor19SparseFillEmptyRowsIN5Eigen9GpuDeviceEjlE38AllocateOutputsExceptEmptyRowIndicatorEPNS_15OpKernelContextElilPPlPPjS8_[_ZN10tensorflow7functor19SparseFillEmptyRowsIN5Eigen9GpuDeviceEjlE38AllocateOutputsExceptEmptyRowIndicatorEPNS_15OpKernelContextElilPPlPPjS8_]+0xfd): more undefined references to `tensorflow::OpKernelContext::allocate_output(absl::lts_20210324::string_view, tensorflow::TensorShape const&, tensorflow::Tensor**)' follow
bazel-out/k8-opt/bin/external/org_tensorflow/tensorflow/core/kernels/linalg/libtridiagonal_matmul_op_gpu.lo(tridiagonal_matmul_op_gpu.cu.o): In function `tensorflow::Status tensorflow::errors::InvalidArgument<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, char const*, int, char const*, int>(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, char const*, int, char const*, int)':
tmpxft_000011ba_00000000-6_tridiagonal_matmul_op_gpu.cu.compute_52.cudafe1.cpp:(.text._ZN10tensorflow6errors15InvalidArgumentIJNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEEPKciS9_iEEENS_6StatusEDpT_[_ZN10tensorflow6errors15InvalidArgumentIJNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEEPKciS9_iEEENS_6StatusEDpT_]+0x1c3): undefined reference to `tensorflow::Status::Status(tensorflow::error::Code, absl::lts_20210324::string_view, std::vector<tensorflow::StackFrame, std::allocator<tensorflow::StackFrame> >&&)'
bazel-out/k8-opt/bin/external/org_tensorflow/tensorflow/core/kernels/linalg/libtridiagonal_matmul_op_gpu.lo(tridiagonal_matmul_op_gpu.cu.o): In function `tensorflow::Status tensorflow::errors::InvalidArgument<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, char const*, int, char const*, long, char const*, long>(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, char const*, int, char const*, long, char const*, long)':
tmpxft_000011ba_00000000-6_tridiagonal_matmul_op_gpu.cu.compute_52.cudafe1.cpp:(.text._ZN10tensorflow6errors15InvalidArgumentIJNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEEPKciS9_lS9_lEEENS_6StatusEDpT_[_ZN10tensorflow6errors15InvalidArgumentIJNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEEPKciS9_lS9_lEEENS_6StatusEDpT_]+0x21d): undefined reference to `tensorflow::strings::internal::CatPieces[abi:cxx11](std::initializer_list<absl::lts_20210324::string_view>)'
tmpxft_000011ba_00000000-6_tridiagonal_matmul_op_gpu.cu.compute_52.cudafe1.cpp:(.text._ZN10tensorflow6errors15InvalidArgumentIJNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEEPKciS9_lS9_lEEENS_6StatusEDpT_[_ZN10tensorflow6errors15InvalidArgumentIJNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEEPKciS9_lS9_lEEENS_6StatusEDpT_]+0x26a): undefined reference to `tensorflow::Status::Status(tensorflow::error::Code, absl::lts_20210324::string_view, std::vector<tensorflow::StackFrame, std::allocator<tensorflow::StackFrame> >&&)'
bazel-out/k8-opt/bin/external/org_tensorflow/tensorflow/core/kernels/linalg/libtridiagonal_matmul_op_gpu.lo(tridiagonal_matmul_op_gpu.cu.o): In function `tensorflow::Status tensorflow::errors::InvalidArgument<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, char const*, long>(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, char const*, long)':
tmpxft_000011ba_00000000-6_tridiagonal_matmul_op_gpu.cu.compute_52.cudafe1.cpp:(.text._ZN10tensorflow6errors15InvalidArgumentIJNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEEPKclEEENS_6StatusEDpT_[_ZN10tensorflow6errors15InvalidArgumentIJNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEEPKclEEENS_6StatusEDpT_]+0x12c): undefined reference to `tensorflow::Status::Status(tensorflow::error::Code, absl::lts_20210324::string_view, std::vector<tensorflow::StackFrame, std::allocator<tensorflow::StackFrame> >&&)'
bazel-out/k8-opt/bin/external/org_tensorflow/tensorflow/core/kernels/linalg/libtridiagonal_matmul_op_gpu.lo(tridiagonal_matmul_op_gpu.cu.o): In function `tensorflow::Status tensorflow::errors::InvalidArgument<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, char const*, long, char const*, long>(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, char const*, long, char const*, long)':
tmpxft_000011ba_00000000-6_tridiagonal_matmul_op_gpu.cu.compute_52.cudafe1.cpp:(.text._ZN10tensorflow6errors15InvalidArgumentIJNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEEPKclS9_lEEENS_6StatusEDpT_[_ZN10tensorflow6errors15InvalidArgumentIJNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEEPKclS9_lEEENS_6StatusEDpT_]+0x1c3): undefined reference to `tensorflow::Status::Status(tensorflow::error::Code, absl::lts_20210324::string_view, std::vector<tensorflow::StackFrame, std::allocator<tensorflow::StackFrame> >&&)'
bazel-out/k8-opt/bin/external/org_tensorflow/tensorflow/core/kernels/image/libgenerate_box_proposals_op_gpu.lo(generate_box_proposals_op.cu.o): In function `tensorflow::{lambda(tensorflow::KernelDef const*)#1}::operator()(tensorflow::KernelDef const*) const::{lambda(tensorflow::OpKernelConstruction*)#1}::_FUN(tensorflow::OpKernelConstruction)':
tmpxft_00003ca5_00000000-6_generate_box_proposals_op.cu.compute_52.cudafe1.cpp:(.text+0x8cc): undefined reference to `tensorflow::GetNodeAttr(tensorflow::AttrSlice const&, absl::lts_20210324::string_view, int*)'
bazel-out/k8-opt/bin/external/org_tensorflow/tensorflow/core/kernels/image/libgenerate_box_proposals_op_gpu.lo(generate_box_proposals_op.cu.o): In function `_GLOBAL__sub_I_tmpxft_00003ca5_00000000_6_generate_box_proposals_op.cu.compute_52.cudafe1.cpp':
tmpxft_00003ca5_00000000-6_generate_box_proposals_op.cu.compute_52.cudafe1.cpp:(.text.startup+0x58b): undefined reference to `tensorflow::kernel_factory::OpKernelRegistrar::InitInternal(tensorflow::KernelDef const*, absl::lts_20210324::string_view, std::unique_ptr<tensorflow::kernel_factory::OpKernelFactory, std::default_delete<tensorflow::kernel_factory::OpKernelFactory> >)'
bazel-out/k8-opt/bin/external/org_tensorflow/tensorflow/core/kernels/image/libnon_max_suppression_op_gpu.lo(non_max_suppression_op.cu.o): In function `tensorflow::{lambda(tensorflow::KernelDef const*)#3}::operator()(tensorflow::KernelDef const*) const::{lambda(tensorflow::OpKernelConstruction*)#1}::_FUN(tensorflow::OpKernelConstruction)':
tmpxft_00001ed1_00000000-6_non_max_suppression_op.cu.compute_52.cudafe1.cpp:(.text+0x5cc): undefined reference to `tensorflow::GetNodeAttr(tensorflow::AttrSlice const&, absl::lts_20210324::string_view, bool*)'
bazel-out/k8-opt/bin/external/org_tensorflow/tensorflow/core/kernels/libbincount_op_gpu.lo(bincount_op_gpu.cu.o): In function `tensorflow::Status tensorflow::errors::Internal<char const*, char const*, char const*>(char const*, char const*, char const*)':
tmpxft_0000039a_00000000-6_bincount_op_gpu.cu.compute_52.cudafe1.cpp:(.text._ZN10tensorflow6errors8InternalIJPKcS3_S3_EEENS_6StatusEDpT_[_ZN10tensorflow6errors8InternalIJPKcS3_S3_EEENS_6StatusEDpT_]+0x131): undefined reference to `tensorflow::Status::Status(tensorflow::error::Code, absl::lts_20210324::string_view, std::vector<tensorflow::StackFrame, std::allocator<tensorflow::StackFrame> >&&)'
bazel-out/k8-opt/bin/external/org_tensorflow/tensorflow/core/kernels/libsegment_reduction_ops_gpu.lo(segment_reduction_ops_gpu_0.cu.o): In function `tensorflow::UseDeterministicSegmentReductions()::{lambda()#1}::operator()() const [clone .isra.520]':
tmpxft_00001477_00000000-6_segment_reduction_ops_gpu_0.cu.compute_52.cudafe1.cpp:(.text+0x66f3): undefined reference to `tensorflow::ReadBoolFromEnvVar(absl::lts_20210324::string_view, bool, bool*)'
bazel-out/k8-opt/bin/external/org_tensorflow/tensorflow/core/kernels/libsegment_reduction_ops_gpu.lo(segment_reduction_ops_gpu_0.cu.o): In function `tensorflow::DisableSegmentReductionOpDeterminismExceptions()::{lambda()#1}::operator()() const [clone .isra.521]':
tmpxft_00001477_00000000-6_segment_reduction_ops_gpu_0.cu.compute_52.cudafe1.cpp:(.text+0x6983): undefined reference to `tensorflow::ReadBoolFromEnvVar(absl::lts_20210324::string_view, bool, bool*)'
bazel-out/k8-opt/bin/external/org_tensorflow/tensorflow/core/kernels/linalg/libsvd_op_gpu.lo(svd_op_gpu.cu.o): In function `tensorflow::{lambda(tensorflow::KernelDef const*)#1}::operator()(tensorflow::KernelDef const*) const::{lambda(tensorflow::OpKernelConstruction*)#1}::_FUN(tensorflow::OpKernelConstruction)':
tmpxft_00001231_00000000-6_svd_op_gpu.cu.compute_52.cudafe1.cpp:(.text+0x23a): undefined reference to `tensorflow::GetNodeAttr(tensorflow::AttrSlice const&, absl::lts_20210324::string_view, bool*)'
tmpxft_00001231_00000000-6_svd_op_gpu.cu.compute_52.cudafe1.cpp:(.text+0x272): undefined reference to `tensorflow::GetNodeAttr(tensorflow::AttrSlice const&, absl::lts_20210324::string_view, bool*)'
bazel-out/k8-opt/bin/external/org_tensorflow/tensorflow/core/kernels/linalg/libsvd_op_gpu.lo(svd_op_gpu.cu.o): In function `tensorflow::{lambda(tensorflow::KernelDef const*)#4}::operator()(tensorflow::KernelDef const*) const::{lambda(tensorflow::OpKernelConstruction*)#1}::_FUN(tensorflow::OpKernelConstruction)':
tmpxft_00001231_00000000-6_svd_op_gpu.cu.compute_52.cudafe1.cpp:(.text+0x5aa): undefined reference to `tensorflow::GetNodeAttr(tensorflow::AttrSlice const&, absl::lts_20210324::string_view, bool*)'
tmpxft_00001231_00000000-6_svd_op_gpu.cu.compute_52.cudafe1.cpp:(.text+0x5e2): undefined reference to `tensorflow::GetNodeAttr(tensorflow::AttrSlice const&, absl::lts_20210324::string_view, bool*)'
bazel-out/k8-opt/bin/external/org_tensorflow/tensorflow/core/kernels/linalg/libsvd_op_gpu.lo(svd_op_gpu.cu.o): In function `tensorflow::{lambda(tensorflow::KernelDef const*)#3}::operator()(tensorflow::KernelDef const*) const::{lambda(tensorflow::OpKernelConstruction*)#1}::_FUN(tensorflow::OpKernelConstruction)':
tmpxft_00001231_00000000-6_svd_op_gpu.cu.compute_52.cudafe1.cpp:(.text+0x91a): undefined reference to `tensorflow::GetNodeAttr(tensorflow::AttrSlice const&, absl::lts_20210324::string_view, bool*)'
bazel-out/k8-opt/bin/external/org_tensorflow/tensorflow/core/kernels/linalg/libsvd_op_gpu.lo(svd_op_gpu.cu.o):tmpxft_00001231_00000000-6_svd_op_gpu.cu.compute_52.cudafe1.cpp:(.text+0x952): more undefined references to `tensorflow::GetNodeAttr(tensorflow::AttrSlice const&, absl::lts_20210324::string_view, bool*)' follow
bazel-out/k8-opt/bin/external/org_tensorflow/tensorflow/core/kernels/linalg/libtridiagonal_solve_op_gpu.lo(tridiagonal_solve_op_gpu.cu.o): In function `tensorflow::{lambda(tensorflow::KernelDef const*)#1}::operator()(tensorflow::KernelDef const*) const::{lambda(tensorflow::OpKernelConstruction*)#1}::_FUN(tensorflow::OpKernelConstruction)':
tmpxft_00001143_00000000-6_tridiagonal_solve_op_gpu.cu.compute_52.cudafe1.cpp:(.text+0x349): undefined reference to `tensorflow::OpKernelConstruction::HasAttr(absl::lts_20210324::string_view) const'
tmpxft_00001143_00000000-6_tridiagonal_solve_op_gpu.cu.compute_52.cudafe1.cpp:(.text+0x38e): undefined reference to `tensorflow::GetNodeAttr(tensorflow::AttrSlice const&, absl::lts_20210324::string_view, bool*)'
tmpxft_00001143_00000000-6_tridiagonal_solve_op_gpu.cu.compute_52.cudafe1.cpp:(.text+0x3f5): undefined reference to `tensorflow::GetNodeAttr(tensorflow::AttrSlice const&, absl::lts_20210324::string_view, bool*)'
bazel-out/k8-opt/bin/external/org_tensorflow/tensorflow/core/kernels/linalg/libtridiagonal_solve_op_gpu.lo(tridiagonal_solve_op_gpu.cu.o): In function `tensorflow::{lambda(tensorflow::KernelDef const*)#2}::operator()(tensorflow::KernelDef const*) const::{lambda(tensorflow::OpKernelConstruction*)#1}::_FUN(tensorflow::OpKernelConstruction)':
tmpxft_00001143_00000000-6_tridiagonal_solve_op_gpu.cu.compute_52.cudafe1.cpp:(.text+0x5e2): undefined reference to `tensorflow::GetNodeAttr(tensorflow::AttrSlice const&, absl::lts_20210324::string_view, bool*)'
tmpxft_00001143_00000000-6_tridiagonal_solve_op_gpu.cu.compute_52.cudafe1.cpp:(.text+0x609): undefined reference to `tensorflow::OpKernelConstruction::HasAttr(absl::lts_20210324::string_view) const'
tmpxft_00001143_00000000-6_tridiagonal_solve_op_gpu.cu.compute_52.cudafe1.cpp:(.text+0x64e): undefined reference to `tensorflow::GetNodeAttr(tensorflow::AttrSlice const&, absl::lts_20210324::string_view, bool*)'
tmpxft_00001143_00000000-6_tridiagonal_solve_op_gpu.cu.compute_52.cudafe1.cpp:(.text+0x6b5): undefined reference to `tensorflow::GetNodeAttr(tensorflow::AttrSlice const&, absl::lts_20210324::string_view, bool*)'
bazel-out/k8-opt/bin/external/org_tensorflow/tensorflow/core/kernels/linalg/libtridiagonal_solve_op_gpu.lo(tridiagonal_solve_op_gpu.cu.o): In function `tensorflow::{lambda(tensorflow::KernelDef const*)#3}::operator()(tensorflow::KernelDef const*) const::{lambda(tensorflow::OpKernelConstruction*)#1}::_FUN(tensorflow::OpKernelConstruction)':
tmpxft_00001143_00000000-6_tridiagonal_solve_op_gpu.cu.compute_52.cudafe1.cpp:(.text+0x8a2): undefined reference to `tensorflow::GetNodeAttr(tensorflow::AttrSlice const&, absl::lts_20210324::string_view, bool*)'
tmpxft_00001143_00000000-6_tridiagonal_solve_op_gpu.cu.compute_52.cudafe1.cpp:(.text+0x8c9): undefined reference to `tensorflow::OpKernelConstruction::HasAttr(absl::lts_20210324::string_view) const'
tmpxft_00001143_00000000-6_tridiagonal_solve_op_gpu.cu.compute_52.cudafe1.cpp:(.text+0x90e): undefined reference to `tensorflow::GetNodeAttr(tensorflow::AttrSlice const&, absl::lts_20210324::string_view, bool*)'
tmpxft_00001143_00000000-6_tridiagonal_solve_op_gpu.cu.compute_52.cudafe1.cpp:(.text+0x975): undefined reference to `tensorflow::GetNodeAttr(tensorflow::AttrSlice const&, absl::lts_20210324::string_view, bool*)'
bazel-out/k8-opt/bin/external/org_tensorflow/tensorflow/core/kernels/linalg/libtridiagonal_solve_op_gpu.lo(tridiagonal_solve_op_gpu.cu.o): In function `tensorflow::{lambda(tensorflow::KernelDef const*)#4}::operator()(tensorflow::KernelDef const*) const::{lambda(tensorflow::OpKernelConstruction*)#1}::_FUN(tensorflow::OpKernelConstruction)':
tmpxft_00001143_00000000-6_tridiagonal_solve_op_gpu.cu.compute_52.cudafe1.cpp:(.text+0xb62): undefined reference to `tensorflow::GetNodeAttr(tensorflow::AttrSlice const&, absl::lts_20210324::string_view, bool*)'
tmpxft_00001143_00000000-6_tridiagonal_solve_op_gpu.cu.compute_52.cudafe1.cpp:(.text+0xb89): undefined reference to `tensorflow::OpKernelConstruction::HasAttr(absl::lts_20210324::string_view) const'
tmpxft_00001143_00000000-6_tridiagonal_solve_op_gpu.cu.compute_52.cudafe1.cpp:(.text+0xbce): undefined reference to `tensorflow::GetNodeAttr(tensorflow::AttrSlice const&, absl::lts_20210324::string_view, bool*)'
tmpxft_00001143_00000000-6_tridiagonal_solve_op_gpu.cu.compute_52.cudafe1.cpp:(.text+0xc35): undefined reference to `tensorflow::GetNodeAttr(tensorflow::AttrSlice const&, absl::lts_20210324::string_view, bool*)'
bazel-out/k8-opt/bin/external/org_tensorflow/tensorflow/core/kernels/liblist_kernels_gpu.lo(list_kernels.cu.o): In function `tensorflow::{lambda(tensorflow::KernelDef const*)#15}::operator()(tensorflow::KernelDef const*) const::{lambda(tensorflow::OpKernelConstruction*)#1}::_FUN(tensorflow::OpKernelConstruction)':
tmpxft_000074bd_00000000-6_list_kernels.cu.compute_52.cudafe1.cpp:(.text+0x39cc): undefined reference to `tensorflow::GetNodeAttr(tensorflow::AttrSlice const&, absl::lts_20210324::string_view, tensorflow::DataType*)'
bazel-out/k8-opt/bin/external/org_tensorflow/tensorflow/core/kernels/liblist_kernels_gpu.lo(list_kernels.cu.o): In function `tensorflow::{lambda(tensorflow::KernelDef const*)#16}::operator()(tensorflow::KernelDef const*) const::{lambda(tensorflow::OpKernelConstruction*)#1}::_FUN(tensorflow::OpKernelConstruction)':
tmpxft_000074bd_00000000-6_list_kernels.cu.compute_52.cudafe1.cpp:(.text+0x3aec): undefined reference to `tensorflow::GetNodeAttr(tensorflow::AttrSlice const&, absl::lts_20210324::string_view, tensorflow::DataType*)'
bazel-out/k8-opt/bin/external/org_tensorflow/tensorflow/core/kernels/liblist_kernels_gpu.lo(list_kernels.cu.o): In function `tensorflow::{lambda(tensorflow::KernelDef const*)#2}::operator()(tensorflow::KernelDef const*) const::{lambda(tensorflow::OpKernelConstruction*)#1}::_FUN(tensorflow::OpKernelConstruction)':
tmpxft_000074bd_00000000-6_list_kernels.cu.compute_52.cudafe1.cpp:(.text+0x3c0c): undefined reference to `tensorflow::GetNodeAttr(tensorflow::AttrSlice const&, absl::lts_20210324::string_view, tensorflow::DataType*)'
bazel-out/k8-opt/bin/external/org_tensorflow/tensorflow/core/kernels/liblist_kernels_gpu.lo(list_kernels.cu.o): In function `tensorflow::{lambda(tensorflow::KernelDef const*)#19}::operator()(tensorflow::KernelDef const*) const::{lambda(tensorflow::OpKernelConstruction*)#1}::_FUN(tensorflow::OpKernelConstruction)':
tmpxft_000074bd_00000000-6_list_kernels.cu.compute_52.cudafe1.cpp:(.text+0x3d2c): undefined reference to `tensorflow::GetNodeAttr(tensorflow::AttrSlice const&, absl::lts_20210324::string_view, tensorflow::DataType*)'
bazel-out/k8-opt/bin/external/org_tensorflow/tensorflow/core/kernels/liblist_kernels_gpu.lo(list_kernels.cu.o): In function `tensorflow::{lambda(tensorflow::KernelDef const*)#7}::operator()(tensorflow::KernelDef const*) const::{lambda(tensorflow::OpKernelConstruction*)#1}::_FUN(tensorflow::OpKernelConstruction)':
tmpxft_000074bd_00000000-6_list_kernels.cu.compute_52.cudafe1.cpp:(.text+0x3e4c): undefined reference to `tensorflow::GetNodeAttr(tensorflow::AttrSlice const&, absl::lts_20210324::string_view, tensorflow::DataType*)'
bazel-out/k8-opt/bin/external/org_tensorflow/tensorflow/core/kernels/liblist_kernels_gpu.lo(list_kernels.cu.o):tmpxft_000074bd_00000000-6_list_kernels.cu.compute_52.cudafe1.cpp:(.text+0x3f6c): more undefined references to `tensorflow::GetNodeAttr(tensorflow::AttrSlice const&, absl::lts_20210324::string_view, tensorflow::DataType*)' follow
bazel-out/k8-opt/bin/external/org_tensorflow/tensorflow/core/kernels/liblist_kernels_gpu.lo(list_kernels.cu.o): In function `tensorflow::{lambda(tensorflow::KernelDef const*)#25}::operator()(tensorflow::KernelDef const*) const::{lambda(tensorflow::OpKernelConstruction*)#1}::_FUN(tensorflow::OpKernelConstruction)':
tmpxft_000074bd_00000000-6_list_kernels.cu.compute_52.cudafe1.cpp:(.text+0x63a2): undefined reference to `tensorflow::GetNodeAttr(tensorflow::AttrSlice const&, absl::lts_20210324::string_view, int*)'
bazel-out/k8-opt/bin/external/org_tensorflow/tensorflow/core/kernels/liblist_kernels_gpu.lo(list_kernels.cu.o): In function `tensorflow::{lambda(tensorflow::KernelDef const*)#97}::operator()(tensorflow::KernelDef const*) const::{lambda(tensorflow::OpKernelConstruction*)#1}::_FUN(tensorflow::OpKernelConstruction)':
tmpxft_000074bd_00000000-6_list_kernels.cu.compute_52.cudafe1.cpp:(.text+0x64ea): undefined reference to `tensorflow::GetNodeAttr(tensorflow::AttrSlice const&, absl::lts_20210324::string_view, tensorflow::DataType*)'
tmpxft_000074bd_00000000-6_list_kernels.cu.compute_52.cudafe1.cpp:(.text+0x6522): undefined reference to `tensorflow::GetNodeAttr(tensorflow::AttrSlice const&, absl::lts_20210324::string_view, int*)'
bazel-out/k8-opt/bin/external/org_tensorflow/tensorflow/core/kernels/liblist_kernels_gpu.lo(list_kernels.cu.o): In function `tensorflow::{lambda(tensorflow::KernelDef const*)#61}::operator()(tensorflow::KernelDef const*) const::{lambda(tensorflow::OpKernelConstruction*)#1}::_FUN(tensorflow::OpKernelConstruction)':
tmpxft_000074bd_00000000-6_list_kernels.cu.compute_52.cudafe1.cpp:(.text+0x666a): undefined reference to `tensorflow::GetNodeAttr(tensorflow::AttrSlice const&, absl::lts_20210324::string_view, tensorflow::DataType*)'
tmpxft_000074bd_00000000-6_list_kernels.cu.compute_52.cudafe1.cpp:(.text+0x66a2): undefined reference to `tensorflow::GetNodeAttr(tensorflow::AttrSlice const&, absl::lts_20210324::string_view, int*)'
bazel-out/k8-opt/bin/external/org_tensorflow/tensorflow/core/kernels/liblist_kernels_gpu.lo(list_kernels.cu.o): In function `tensorflow::{lambda(tensorflow::KernelDef const*)#13}::operator()(tensorflow::KernelDef const*) const::{lambda(tensorflow::OpKernelConstruction*)#1}::_FUN(tensorflow::OpKernelConstruction)':
tmpxft_000074bd_00000000-6_list_kernels.cu.compute_52.cudafe1.cpp:(.text+0x67ea): undefined reference to `tensorflow::GetNodeAttr(tensorflow::AttrSlice const&, absl::lts_20210324::string_view, tensorflow::DataType*)'
tmpxft_000074bd_00000000-6_list_kernels.cu.compute_52.cudafe1.cpp:(.text+0x6822): undefined reference to `tensorflow::GetNodeAttr(tensorflow::AttrSlice const&, absl::lts_20210324::string_view, int*)'
bazel-out/k8-opt/bin/external/org_tensorflow/tensorflow/core/kernels/liblist_kernels_gpu.lo(list_kernels.cu.o): In function `tensorflow::{lambda(tensorflow::KernelDef const*)#49}::operator()(tensorflow::KernelDef const*) const::{lambda(tensorflow::OpKernelConstruction*)#1}::_FUN(tensorflow::OpKernelConstruction)':
tmpxft_000074bd_00000000-6_list_kernels.cu.compute_52.cudafe1.cpp:(.text+0x696a): undefined reference to `tensorflow::GetNodeAttr(tensorflow::AttrSlice const&, absl::lts_20210324::string_view, tensorflow::DataType*)'
tmpxft_000074bd_00000000-6_list_kernels.cu.compute_52.cudafe1.cpp:(.text+0x69a2): undefined reference to `tensorflow::GetNodeAttr(tensorflow::AttrSlice const&, absl::lts_20210324::string_view, int*)'
bazel-out/k8-opt/bin/external/org_tensorflow/tensorflow/core/kernels/liblist_kernels_gpu.lo(list_kernels.cu.o): In function `tensorflow::{lambda(tensorflow::KernelDef const*)#1}::operator()(tensorflow::KernelDef const*) const::{lambda(tensorflow::OpKernelConstruction*)#1}::_FUN(tensorflow::OpKernelConstruction)':
tmpxft_000074bd_00000000-6_list_kernels.cu.compute_52.cudafe1.cpp:(.text+0x6aea): undefined reference to `tensorflow::GetNodeAttr(tensorflow::AttrSlice const&, absl::lts_20210324::string_view, tensorflow::DataType*)'
tmpxft_000074bd_00000000-6_list_kernels.cu.compute_52.cudafe1.cpp:(.text+0x6b22): undefined reference to `tensorflow::GetNodeAttr(tensorflow::AttrSlice const&, absl::lts_20210324::string_view, int*)'
bazel-out/k8-opt/bin/external/org_tensorflow/tensorflow/core/kernels/liblist_kernels_gpu.lo(list_kernels.cu.o): In function `tensorflow::{lambda(tensorflow::KernelDef const*)#85}::operator()(tensorflow::KernelDef const*) const::{lambda(tensorflow::OpKernelConstruction*)#1}::_FUN(tensorflow::OpKernelConstruction)':
tmpxft_000074bd_00000000-6_list_kernels.cu.compute_52.cudafe1.cpp:(.text+0x6c6a): undefined reference to `tensorflow::GetNodeAttr(tensorflow::AttrSlice const&, absl::lts_20210324::string_view, tensorflow::DataType*)'
tmpxft_000074bd_00000000-6_list_kernels.cu.compute_52.cudafe1.cpp:(.text+0x6ca2): undefined reference to `tensorflow::GetNodeAttr(tensorflow::AttrSlice const&, absl::lts_20210324::string_view, int*)'
bazel-out/k8-opt/bin/external/org_tensorflow/tensorflow/core/kernels/liblist_kernels_gpu.lo(list_kernels.cu.o): In function `tensorflow::{lambda(tensorflow::KernelDef const*)#37}::operator()(tensorflow::KernelDef const*) const::{lambda(tensorflow::OpKernelConstruction*)#1}::_FUN(tensorflow::OpKernelConstruction)':
tmpxft_000074bd_00000000-6_list_kernels.cu.compute_52.cudafe1.cpp:(.text+0x6dea): undefined reference to `tensorflow::GetNodeAttr(tensorflow::AttrSlice const&, absl::lts_20210324::string_view, tensorflow::DataType*)'
tmpxft_000074bd_00000000-6_list_kernels.cu.compute_52.cudafe1.cpp:(.text+0x6e22): undefined reference to `tensorflow::GetNodeAttr(tensorflow::AttrSlice const&, absl::lts_20210324::string_view, int*)'
bazel-out/k8-opt/bin/external/org_tensorflow/tensorflow/core/kernels/liblist_kernels_gpu.lo(list_kernels.cu.o): In function `tensorflow::{lambda(tensorflow::KernelDef const*)#73}::operator()(tensorflow::KernelDef const*) const::{lambda(tensorflow::OpKernelConstruction*)#1}::_FUN(tensorflow::OpKernelConstruction)':
tmpxft_000074bd_00000000-6_list_kernels.cu.compute_52.cudafe1.cpp:(.text+0x6f6a): undefined reference to `tensorflow::GetNodeAttr(tensorflow::AttrSlice const&, absl::lts_20210324::string_view, tensorflow::DataType*)'
tmpxft_000074bd_00000000-6_list_kernels.cu.compute_52.cudafe1.cpp:(.text+0x6fa2): undefined reference to `tensorflow::GetNodeAttr(tensorflow::AttrSlice const&, absl::lts_20210324::string_view, int*)'
bazel-out/k8-opt/bin/external/org_tensorflow/tensorflow/core/kernels/liblist_kernels_gpu.lo(list_kernels.cu.o): In function `tensorflow::{lambda(tensorflow::KernelDef const*)#65}::operator()(tensorflow::KernelDef const*) const::{lambda(tensorflow::OpKernelConstruction*)#1}::_FUN(tensorflow::OpKernelConstruction)':
tmpxft_000074bd_00000000-6_list_kernels.cu.compute_52.cudafe1.cpp:(.text+0x70fb): undefined reference to `tensorflow::GetNodeAttr(tensorflow::AttrSlice const&, absl::lts_20210324::string_view, tensorflow::DataType*)'
tmpxft_000074bd_00000000-6_list_kernels.cu.compute_52.cudafe1.cpp:(.text+0x711a): undefined reference to `tensorflow::OpKernelConstruction::HasAttr(absl::lts_20210324::string_view) const'
tmpxft_000074bd_00000000-6_list_kernels.cu.compute_52.cudafe1.cpp:(.text+0x7176): undefined reference to `tensorflow::GetNodeAttr(tensorflow::AttrSlice const&, absl::lts_20210324::string_view, tensorflow::PartialTensorShape*)'
bazel-out/k8-opt/bin/external/org_tensorflow/tensorflow/core/kernels/liblist_kernels_gpu.lo(list_kernels.cu.o): In function `tensorflow::{lambda(tensorflow::KernelDef const*)#53}::operator()(tensorflow::KernelDef const*) const::{lambda(tensorflow::OpKernelConstruction*)#1}::_FUN(tensorflow::OpKernelConstruction)':
tmpxft_000074bd_00000000-6_list_kernels.cu.compute_52.cudafe1.cpp:(.text+0x72cb): undefined reference to `tensorflow::GetNodeAttr(tensorflow::AttrSlice const&, absl::lts_20210324::string_view, tensorflow::DataType*)'
tmpxft_000074bd_00000000-6_list_kernels.cu.compute_52.cudafe1.cpp:(.text+0x72ea): undefined reference to `tensorflow::OpKernelConstruction::HasAttr(absl::lts_20210324::string_view) const'
tmpxft_000074bd_00000000-6_list_kernels.cu.compute_52.cudafe1.cpp:(.text+0x7346): undefined reference to `tensorflow::GetNodeAttr(tensorflow::AttrSlice const&, absl::lts_20210324::string_view, tensorflow::PartialTensorShape*)'
bazel-out/k8-opt/bin/external/org_tensorflow/tensorflow/core/kernels/liblist_kernels_gpu.lo(list_kernels.cu.o): In function `tensorflow::{lambda(tensorflow::KernelDef const*)#6}::operator()(tensorflow::KernelDef const*) const::{lambda(tensorflow::OpKernelConstruction*)#1}::_FUN(tensorflow::OpKernelConstruction)':
tmpxft_000074bd_00000000-6_list_kernels.cu.compute_52.cudafe1.cpp:(.text+0x749b): undefined reference to `tensorflow::GetNodeAttr(tensorflow::AttrSlice const&, absl::lts_20210324::string_view, tensorflow::DataType*)'
tmpxft_000074bd_00000000-6_list_kernels.cu.compute_52.cudafe1.cpp:(.text+0x74ba): undefined reference to `tensorflow::OpKernelConstruction::HasAttr(absl::lts_20210324::string_view) const'
tmpxft_000074bd_00000000-6_list_kernels.cu.compute_52.cudafe1.cpp:(.text+0x7516): undefined reference to `tensorflow::GetNodeAttr(tensorflow::AttrSlice const&, absl::lts_20210324::string_view, tensorflow::PartialTensorShape*)'
bazel-out/k8-opt/bin/external/org_tensorflow/tensorflow/core/kernels/liblist_kernels_gpu.lo(list_kernels.cu.o): In function `tensorflow::{lambda(tensorflow::KernelDef const*)#18}::operator()(tensorflow::KernelDef const*) const::{lambda(tensorflow::OpKernelConstruction*)#1}::_FUN(tensorflow::OpKernelConstruction)':
tmpxft_000074bd_00000000-6_list_kernels.cu.compute_52.cudafe1.cpp:(.text+0x766b): undefined reference to `tensorflow::GetNodeAttr(tensorflow::AttrSlice const&, absl::lts_20210324::string_view, tensorflow::DataType*)'
tmpxft_000074bd_00000000-6_list_kernels.cu.compute_52.cudafe1.cpp:(.text+0x768a): undefined reference to `tensorflow::OpKernelConstruction::HasAttr(absl::lts_20210324::string_view) const'
tmpxft_000074bd_00000000-6_list_kernels.cu.compute_52.cudafe1.cpp:(.text+0x76e6): undefined reference to `tensorflow::GetNodeAttr(tensorflow::AttrSlice const&, absl::lts_20210324::string_view, tensorflow::PartialTensorShape*)'
bazel-out/k8-opt/bin/external/org_tensorflow/tensorflow/core/kernels/liblist_kernels_gpu.lo(list_kernels.cu.o): In function `tensorflow::{lambda(tensorflow::KernelDef const*)#29}::operator()(tensorflow::KernelDef const*) const::{lambda(tensorflow::OpKernelConstruction*)#1}::_FUN(tensorflow::OpKernelConstruction)':
tmpxft_000074bd_00000000-6_list_kernels.cu.compute_52.cudafe1.cpp:(.text+0x783b): undefined reference to `tensorflow::GetNodeAttr(tensorflow::AttrSlice const&, absl::lts_20210324::string_view, tensorflow::DataType*)'
tmpxft_000074bd_00000000-6_list_kernels.cu.compute_52.cudafe1.cpp:(.text+0x785a): undefined reference to `tensorflow::OpKernelConstruction::HasAttr(absl::lts_20210324::string_view) const'
tmpxft_000074bd_00000000-6_list_kernels.cu.compute_52.cudafe1.cpp:(.text+0x78b6): undefined reference to `tensorflow::GetNodeAttr(tensorflow::AttrSlice const&, absl::lts_20210324::string_view, tensorflow::PartialTensorShape*)'
bazel-out/k8-opt/bin/external/org_tensorflow/tensorflow/core/kernels/liblist_kernels_gpu.lo(list_kernels.cu.o): In function `tensorflow::{lambda(tensorflow::KernelDef const*)#77}::operator()(tensorflow::KernelDef const*) const::{lambda(tensorflow::OpKernelConstruction*)#1}::_FUN(tensorflow::OpKernelConstruction)':
tmpxft_000074bd_00000000-6_list_kernels.cu.compute_52.cudafe1.cpp:(.text+0x7a0b): undefined reference to `tensorflow::GetNodeAttr(tensorflow::AttrSlice const&, absl::lts_20210324::string_view, tensorflow::DataType*)'
tmpxft_000074bd_00000000-6_list_kernels.cu.compute_52.cudafe1.cpp:(.text+0x7a2a): undefined reference to `tensorflow::OpKernelConstruction::HasAttr(absl::lts_20210324::string_view) const'
tmpxft_000074bd_00000000-6_list_kernels.cu.compute_52.cudafe1.cpp:(.text+0x7a86): undefined reference to `tensorflow::GetNodeAttr(tensorflow::AttrSlice const&, absl::lts_20210324::string_view, tensorflow::PartialTensorShape*)'
bazel-out/k8-opt/bin/external/org_tensorflow/tensorflow/core/kernels/liblist_kernels_gpu.lo(list_kernels.cu.o): In function `tensorflow::{lambda(tensorflow::KernelDef const*)#30}::operator()(tensorflow::KernelDef const*) const::{lambda(tensorflow::OpKernelConstruction*)#1}::_FUN(tensorflow::OpKernelConstruction)':
tmpxft_000074bd_00000000-6_list_kernels.cu.compute_52.cudafe1.cpp:(.text+0x7bdb): undefined reference to `tensorflow::GetNodeAttr(tensorflow::AttrSlice const&, absl::lts_20210324::string_view, tensorflow::DataType*)'
tmpxft_000074bd_00000000-6_list_kernels.cu.compute_52.cudafe1.cpp:(.text+0x7bfa): undefined reference to `tensorflow::OpKernelConstruction::HasAttr(absl::lts_20210324::string_view) const'
tmpxft_000074bd_00000000-6_list_kernels.cu.compute_52.cudafe1.cpp:(.text+0x7c56): undefined reference to `tensorflow::GetNodeAttr(tensorflow::AttrSlice const&, absl::lts_20210324::string_view, tensorflow::PartialTensorShape*)'
bazel-out/k8-opt/bin/external/org_tensorflow/tensorflow/core/kernels/liblist_kernels_gpu.lo(list_kernels.cu.o): In function `tensorflow::{lambda(tensorflow::KernelDef const*)#101}::operator()(tensorflow::KernelDef const*) const::{lambda(tensorflow::OpKernelConstruction*)#1}::_FUN(tensorflow::OpKernelConstruction)':
tmpxft_000074bd_00000000-6_list_kernels.cu.compute_52.cudafe1.cpp:(.text+0x7dab): undefined reference to `tensorflow::GetNodeAttr(tensorflow::AttrSlice const&, absl::lts_20210324::string_view, tensorflow::DataType*)'
tmpxft_000074bd_00000000-6_list_kernels.cu.compute_52.cudafe1.cpp:(.text+0x7dca): undefined reference to `tensorflow::OpKernelConstruction::HasAttr(absl::lts_20210324::string_view) const'
tmpxft_000074bd_00000000-6_list_kernels.cu.compute_52.cudafe1.cpp:(.text+0x7e26): undefined reference to `tensorflow::GetNodeAttr(tensorflow::AttrSlice const&, absl::lts_20210324::string_view, tensorflow::PartialTensorShape*)'
bazel-out/k8-opt/bin/external/org_tensorflow/tensorflow/core/kernels/liblist_kernels_gpu.lo(list_kernels.cu.o): In function `tensorflow::{lambda(tensorflow::KernelDef const*)#17}::operator()(tensorflow::KernelDef const*) const::{lambda(tensorflow::OpKernelConstruction*)#1}::_FUN(tensorflow::OpKernelConstruction)':
tmpxft_000074bd_00000000-6_list_kernels.cu.compute_52.cudafe1.cpp:(.text+0x7f7b): undefined reference to `tensorflow::GetNodeAttr(tensorflow::AttrSlice const&, absl::lts_20210324::string_view, tensorflow::DataType*)'
tmpxft_000074bd_00000000-6_list_kernels.cu.compute_52.cudafe1.cpp:(.text+0x7f9a): undefined reference to `tensorflow::OpKernelConstruction::HasAttr(absl::lts_20210324::string_view) const'
tmpxft_000074bd_00000000-6_list_kernels.cu.compute_52.cudafe1.cpp:(.text+0x7ff6): undefined reference to `tensorflow::GetNodeAttr(tensorflow::AttrSlice const&, absl::lts_20210324::string_view, tensorflow::PartialTensorShape*)'
bazel-out/k8-opt/bin/external/org_tensorflow/tensorflow/core/kernels/liblist_kernels_gpu.lo(list_kernels.cu.o): In function `tensorflow::{lambda(tensorflow::KernelDef const*)#5}::operator()(tensorflow::KernelDef const*) const::{lambda(tensorflow::OpKernelConstruction*)#1}::_FUN(tensorflow::OpKernelConstruction)':
tmpxft_000074bd_00000000-6_list_kernels.cu.compute_52.cudafe1.cpp:(.text+0x814b): undefined reference to `tensorflow::GetNodeAttr(tensorflow::AttrSlice const&, absl::lts_20210324::string_view, tensorflow::DataType*)'
tmpxft_000074bd_00000000-6_list_kernels.cu.compute_52.cudafe1.cpp:(.text+0x816a): undefined reference to `tensorflow::OpKernelConstruction::HasAttr(absl::lts_20210324::string_view) const'
tmpxft_000074bd_00000000-6_list_kernels.cu.compute_52.cudafe1.cpp:(.text+0x81c6): undefined reference to `tensorflow::GetNodeAttr(tensorflow::AttrSlice const&, absl::lts_20210324::string_view, tensorflow::PartialTensorShape*)'
bazel-out/k8-opt/bin/external/org_tensorflow/tensorflow/core/kernels/liblist_kernels_gpu.lo(list_kernels.cu.o): In function `tensorflow::{lambda(tensorflow::KernelDef const*)#89}::operator()(tensorflow::KernelDef const*) const::{lambda(tensorflow::OpKernelConstruction*)#1}::_FUN(tensorflow::OpKernelConstruction)':
tmpxft_000074bd_00000000-6_list_kernels.cu.compute_52.cudafe1.cpp:(.text+0x831b): undefined reference to `tensorflow::GetNodeAttr(tensorflow::AttrSlice const&, absl::lts_20210324::string_view, tensorflow::DataType*)'
tmpxft_000074bd_00000000-6_list_kernels.cu.compute_52.cudafe1.cpp:(.text+0x833a): undefined reference to `tensorflow::OpKernelConstruction::HasAttr(absl::lts_20210324::string_view) const'
tmpxft_000074bd_00000000-6_list_kernels.cu.compute_52.cudafe1.cpp:(.text+0x8396): undefined reference to `tensorflow::GetNodeAttr(tensorflow::AttrSlice const&, absl::lts_20210324::string_view, tensorflow::PartialTensorShape*)'
bazel-out/k8-opt/bin/external/org_tensorflow/tensorflow/core/kernels/liblist_kernels_gpu.lo(list_kernels.cu.o): In function `tensorflow::{lambda(tensorflow::KernelDef const*)#54}::operator()(tensorflow::KernelDef const*) const::{lambda(tensorflow::OpKernelConstruction*)#1}::_FUN(tensorflow::OpKernelConstruction)':
tmpxft_000074bd_00000000-6_list_kernels.cu.compute_52.cudafe1.cpp:(.text+0x84eb): undefined reference to `tensorflow::GetNodeAttr(tensorflow::AttrSlice const&, absl::lts_20210324::string_view, tensorflow::DataType*)'
tmpxft_000074bd_00000000-6_list_kernels.cu.compute_52.cudafe1.cpp:(.text+0x850a): undefined reference to `tensorflow::OpKernelConstruction::HasAttr(absl::lts_20210324::string_view) const'
tmpxft_000074bd_00000000-6_list_kernels.cu.compute_52.cudafe1.cpp:(.text+0x8566): undefined reference to `tensorflow::GetNodeAttr(tensorflow::AttrSlice const&, absl::lts_20210324::string_view, tensorflow::PartialTensorShape*)'
bazel-out/k8-opt/bin/external/org_tensorflow/tensorflow/core/kernels/liblist_kernels_gpu.lo(list_kernels.cu.o): In function `tensorflow::{lambda(tensorflow::KernelDef const*)#41}::operator()(tensorflow::KernelDef const*) const::{lambda(tensorflow::OpKernelConstruction*)#1}::_FUN(tensorflow::OpKernelConstruction)':
tmpxft_000074bd_00000000-6_list_kernels.cu.compute_52.cudafe1.cpp:(.text+0x86bb): undefined reference to `tensorflow::GetNodeAttr(tensorflow::AttrSlice const&, absl::lts_20210324::string_view, tensorflow::DataType*)'
tmpxft_000074bd_00000000-6_list_kernels.cu.compute_52.cudafe1.cpp:(.text+0x86da): undefined reference to `tensorflow::OpKernelConstruction::HasAttr(absl::lts_20210324::string_view) const'
tmpxft_000074bd_00000000-6_list_kernels.cu.compute_52.cudafe1.cpp:(.text+0x8736): undefined reference to `tensorflow::GetNodeAttr(tensorflow::AttrSlice const&, absl::lts_20210324::string_view, tensorflow::PartialTensorShape*)'
bazel-out/k8-opt/bin/external/org_tensorflow/tensorflow/core/kernels/liblist_kernels_gpu.lo(list_kernels.cu.o): In function `tensorflow::{lambda(tensorflow::KernelDef const*)#66}::operator()(tensorflow::KernelDef const*) const::{lambda(tensorflow::OpKernelConstruction*)#1}::_FUN(tensorflow::OpKernelConstruction)':
tmpxft_000074bd_00000000-6_list_kernels.cu.compute_52.cudafe1.cpp:(.text+0x888b): undefined reference to `tensorflow::GetNodeAttr(tensorflow::AttrSlice const&, absl::lts_20210324::string_view, tensorflow::DataType*)'
tmpxft_000074bd_00000000-6_list_kernels.cu.compute_52.cudafe1.cpp:(.text+0x88aa): undefined reference to `tensorflow::OpKernelConstruction::HasAttr(absl::lts_20210324::string_view) const'
tmpxft_000074bd_00000000-6_list_kernels.cu.compute_52.cudafe1.cpp:(.text+0x8906): undefined reference to `tensorflow::GetNodeAttr(tensorflow::AttrSlice const&, absl::lts_20210324::string_view, tensorflow::PartialTensorShape*)'
bazel-out/k8-opt/bin/external/org_tensorflow/tensorflow/core/kernels/liblist_kernels_gpu.lo(list_kernels.cu.o): In function `tensorflow::{lambda(tensorflow::KernelDef const*)#42}::operator()(tensorflow::KernelDef const*) const::{lambda(tensorflow::OpKernelConstruction*)#1}::_FUN(tensorflow::OpKernelConstruction)':
tmpxft_000074bd_00000000-6_list_kernels.cu.compute_52.cudafe1.cpp:(.text+0x8a5b): undefined reference to `tensorflow::GetNodeAttr(tensorflow::AttrSlice const&, absl::lts_20210324::string_view, tensorflow::DataType*)'
tmpxft_000074bd_00000000-6_list_kernels.cu.compute_52.cudafe1.cpp:(.text+0x8a7a): undefined reference to `tensorflow::OpKernelConstruction::HasAttr(absl::lts_20210324::string_view) const'
tmpxft_000074bd_00000000-6_list_kernels.cu.compute_52.cudafe1.cpp:(.text+0x8ad6): undefined reference to `tensorflow::GetNodeAttr(tensorflow::AttrSlice const&, absl::lts_20210324::string_view, tensorflow::PartialTensorShape*)'
bazel-out/k8-opt/bin/external/org_tensorflow/tensorflow/core/kernels/liblist_kernels_gpu.lo(list_kernels.cu.o): In function `tensorflow::{lambda(tensorflow::KernelDef const*)#78}::operator()(tensorflow::KernelDef const*) const::{lambda(tensorflow::OpKernelConstruction*)#1}::_FUN(tensorflow::OpKernelConstruction)':
tmpxft_000074bd_00000000-6_list_kernels.cu.compute_52.cudafe1.cpp:(.text+0x8c2b): undefined reference to `tensorflow::GetNodeAttr(tensorflow::AttrSlice const&, absl::lts_20210324::string_view, tensorflow::DataType*)'
tmpxft_000074bd_00000000-6_list_kernels.cu.compute_52.cudafe1.cpp:(.text+0x8c4a): undefined reference to `tensorflow::OpKernelConstruction::HasAttr(absl::lts_20210324::string_view) const'
tmpxft_000074bd_00000000-6_list_kernels.cu.compute_52.cudafe1.cpp:(.text+0x8ca6): undefined reference to `tensorflow::GetNodeAttr(tensorflow::AttrSlice const&, absl::lts_20210324::string_view, tensorflow::PartialTensorShape*)'
bazel-out/k8-opt/bin/external/org_tensorflow/tensorflow/core/kernels/liblist_kernels_gpu.lo(list_kernels.cu.o): In function `tensorflow::{lambda(tensorflow::KernelDef const*)#90}::operator()(tensorflow::KernelDef const*) const::{lambda(tensorflow::OpKernelConstruction*)#1}::_FUN(tensorflow::OpKernelConstruction)':
tmpxft_000074bd_00000000-6_list_kernels.cu.compute_52.cudafe1.cpp:(.text+0x8dfb): undefined reference to `tensorflow::GetNodeAttr(tensorflow::AttrSlice const&, absl::lts_20210324::string_view, tensorflow::DataType*)'
tmpxft_000074bd_00000000-6_list_kernels.cu.compute_52.cudafe1.cpp:(.text+0x8e1a): undefined reference to `tensorflow::OpKernelConstruction::HasAttr(absl::lts_20210324::string_view) const'
tmpxft_000074bd_00000000-6_list_kernels.cu.compute_52.cudafe1.cpp:(.text+0x8e76): undefined reference to `tensorflow::GetNodeAttr(tensorflow::AttrSlice const&, absl::lts_20210324::string_view, tensorflow::PartialTensorShape*)'
bazel-out/k8-opt/bin/external/org_tensorflow/tensorflow/core/kernels/liblist_kernels_gpu.lo(list_kernels.cu.o): In function `tensorflow::{lambda(tensorflow::KernelDef const*)#102}::operator()(tensorflow::KernelDef const*) const::{lambda(tensorflow::OpKernelConstruction*)#1}::_FUN(tensorflow::OpKernelConstruction)':
tmpxft_000074bd_00000000-6_list_kernels.cu.compute_52.cudafe1.cpp:(.text+0x8fcb): undefined reference to `tensorflow::GetNodeAttr(tensorflow::AttrSlice const&, absl::lts_20210324::string_view, tensorflow::DataType*)'
tmpxft_000074bd_00000000-6_list_kernels.cu.compute_52.cudafe1.cpp:(.text+0x8fea): undefined reference to `tensorflow::OpKernelConstruction::HasAttr(absl::lts_20210324::string_view) const'
tmpxft_000074bd_00000000-6_list_kernels.cu.compute_52.cudafe1.cpp:(.text+0x9046): undefined reference to `tensorflow::GetNodeAttr(tensorflow::AttrSlice const&, absl::lts_20210324::string_view, tensorflow::PartialTensorShape*)'
bazel-out/k8-opt/bin/external/org_tensorflow/tensorflow/core/kernels/liblist_kernels_gpu.lo(list_kernels.cu.o): In function `std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > tensorflow::strings::StrCat<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >(tensorflow::strings::AlphaNum const&, tensorflow::strings::AlphaNum const&, tensorflow::strings::AlphaNum const&, tensorflow::strings::AlphaNum const&, tensorflow::strings::AlphaNum const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)':
tmpxft_000074bd_00000000-6_list_kernels.cu.compute_52.cudafe1.cpp:(.text._ZN10tensorflow7strings6StrCatIJNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEEEEES7_RKNS0_8AlphaNumESA_SA_SA_SA_DpRKT_[_ZN10tensorflow7strings6StrCatIJNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEEEEES7_RKNS0_8AlphaNumESA_SA_SA_SA_DpRKT_]+0x91): undefined reference to `tensorflow::strings::internal::CatPieces[abi:cxx11](std::initializer_list<absl::lts_20210324::string_view>)'
collect2: error: ld returned 1 exit status
Target //mediapipe/examples/desktop/object_detection:object_detection_tensorflow failed to build
Use --verbose_failures to see the command lines of failed build steps.
INFO: Elapsed time: 31322.474s, Critical Path: 185.47s
INFO: 24459 processes: 7727 internal, 16732 local.
FAILED: Build did NOT complete successfully

@mcclanahoochie, @jiuqiant @sgowroji any ideas what to try next?

KulkarniKaustubh commented 2 years ago

Are there any updates on this issue? I am also getting an error while building Mediapipe for GPU.

I am running a container with: CUDA 11.2 Tensorflow 2.7.0

Here is the output of the bazel command given in the mediapipe docs:

root@pop-os:/workspace/repos/mediapipe# bazel build -c opt --config=cuda --spawn_strategy=local     --define no_aws_support=true --copt -DMESA_EGL_NO_X11_HEADERS     mediapipe/examples/desktop/object_detection:object_detection_tensorflow
DEBUG: Rule 'rules_foreign_cc' indicated that a canonical reproducible form can be obtained by modifying arguments sha256 = "c2cdcf55ffaf49366725639e45dedd449b8c3fe22b54e31625eb80ce3a240f1e"
DEBUG: Repository rules_foreign_cc instantiated at:
  /workspace/repos/mediapipe/WORKSPACE:42:13: in <toplevel>
Repository rule http_archive defined at:
  /root/.cache/bazel/_bazel_root/f232ff2f81a27f8c8fd1521d116cbf0f/external/bazel_tools/tools/build_defs/repo/http.bzl:336:31: in <toplevel>
WARNING: Download from https://storage.googleapis.com/mirror.tensorflow.org/github.com/llvm/llvm-project/archive/a87782c34d667d1c1a18fe82a9a7abfc72fd345b.tar.gz failed: class com.google.devtools.build.lib.bazel.repository.downloader.UnrecoverableHttpException GET returned 404 Not Found
WARNING: Download from http://mirror.tensorflow.org/github.com/bazelbuild/rules_closure/archive/cf1e44edb908e9616030cc83d085989b8e6cd6df.tar.gz failed: class com.google.devtools.build.lib.bazel.repository.downloader.UnrecoverableHttpException GET returned 404 Not Found
WARNING: Download from https://storage.googleapis.com/mirror.tensorflow.org/github.com/tensorflow/runtime/archive/fc70a32ecd248dd7612d75d2177125ed14541367.tar.gz failed: class com.google.devtools.build.lib.bazel.repository.downloader.UnrecoverableHttpException GET returned 404 Not Found
DEBUG: /root/.cache/bazel/_bazel_root/f232ff2f81a27f8c8fd1521d116cbf0f/external/org_tensorflow/third_party/repo.bzl:122:14: 
Warning: skipping import of repository 'com_google_absl' because it already exists.
DEBUG: /root/.cache/bazel/_bazel_root/f232ff2f81a27f8c8fd1521d116cbf0f/external/org_tensorflow/third_party/repo.bzl:122:14: 
Warning: skipping import of repository 'com_google_benchmark' because it already exists.
DEBUG: /root/.cache/bazel/_bazel_root/f232ff2f81a27f8c8fd1521d116cbf0f/external/org_tensorflow/third_party/repo.bzl:122:14: 
Warning: skipping import of repository 'pybind11_bazel' because it already exists.
DEBUG: /root/.cache/bazel/_bazel_root/f232ff2f81a27f8c8fd1521d116cbf0f/external/org_tensorflow/third_party/repo.bzl:122:14: 
Warning: skipping import of repository 'com_google_protobuf' because it already exists.
DEBUG: /root/.cache/bazel/_bazel_root/f232ff2f81a27f8c8fd1521d116cbf0f/external/org_tensorflow/third_party/repo.bzl:122:14: 
Warning: skipping import of repository 'com_google_googletest' because it already exists.
DEBUG: /root/.cache/bazel/_bazel_root/f232ff2f81a27f8c8fd1521d116cbf0f/external/org_tensorflow/third_party/repo.bzl:122:14: 
Warning: skipping import of repository 'com_github_gflags_gflags' because it already exists.
DEBUG: /root/.cache/bazel/_bazel_root/f232ff2f81a27f8c8fd1521d116cbf0f/external/org_tensorflow/third_party/repo.bzl:122:14: 
Warning: skipping import of repository 'build_bazel_rules_apple' because it already exists.
DEBUG: /root/.cache/bazel/_bazel_root/f232ff2f81a27f8c8fd1521d116cbf0f/external/org_tensorflow/third_party/repo.bzl:122:14: 
Warning: skipping import of repository 'build_bazel_rules_swift' because it already exists.
DEBUG: /root/.cache/bazel/_bazel_root/f232ff2f81a27f8c8fd1521d116cbf0f/external/org_tensorflow/third_party/repo.bzl:122:14: 
Warning: skipping import of repository 'build_bazel_apple_support' because it already exists.
DEBUG: /root/.cache/bazel/_bazel_root/f232ff2f81a27f8c8fd1521d116cbf0f/external/org_tensorflow/third_party/repo.bzl:122:14: 
Warning: skipping import of repository 'xctestrunner' because it already exists.
DEBUG: /root/.cache/bazel/_bazel_root/f232ff2f81a27f8c8fd1521d116cbf0f/external/org_tensorflow/third_party/repo.bzl:122:14: 
Warning: skipping import of repository 'pybind11' because it already exists.
INFO: Repository local_config_cuda instantiated at:
  /workspace/repos/mediapipe/WORKSPACE:400:14: in <toplevel>
  /root/.cache/bazel/_bazel_root/f232ff2f81a27f8c8fd1521d116cbf0f/external/org_tensorflow/tensorflow/workspace2.bzl:876:19: in workspace
  /root/.cache/bazel/_bazel_root/f232ff2f81a27f8c8fd1521d116cbf0f/external/org_tensorflow/tensorflow/workspace2.bzl:94:19: in _tf_toolchains
Repository rule cuda_configure defined at:
  /root/.cache/bazel/_bazel_root/f232ff2f81a27f8c8fd1521d116cbf0f/external/org_tensorflow/third_party/gpus/cuda_configure.bzl:1448:33: in <toplevel>
ERROR: An error occurred during the fetch of repository 'local_config_cuda':
   Traceback (most recent call last):
    File "/root/.cache/bazel/_bazel_root/f232ff2f81a27f8c8fd1521d116cbf0f/external/org_tensorflow/third_party/gpus/cuda_configure.bzl", line 1401, column 38, in _cuda_autoconf_impl
        _create_local_cuda_repository(repository_ctx)
    File "/root/.cache/bazel/_bazel_root/f232ff2f81a27f8c8fd1521d116cbf0f/external/org_tensorflow/third_party/gpus/cuda_configure.bzl", line 978, column 35, in _create_local_cuda_repository
        cuda_config = _get_cuda_config(repository_ctx, find_cuda_config_script)
    File "/root/.cache/bazel/_bazel_root/f232ff2f81a27f8c8fd1521d116cbf0f/external/org_tensorflow/third_party/gpus/cuda_configure.bzl", line 666, column 30, in _get_cuda_config
        config = find_cuda_config(repository_ctx, find_cuda_config_script, ["cuda", "cudnn"])
    File "/root/.cache/bazel/_bazel_root/f232ff2f81a27f8c8fd1521d116cbf0f/external/org_tensorflow/third_party/gpus/cuda_configure.bzl", line 643, column 41, in find_cuda_config
        exec_result = _exec_find_cuda_config(repository_ctx, script_path, cuda_libraries)
    File "/root/.cache/bazel/_bazel_root/f232ff2f81a27f8c8fd1521d116cbf0f/external/org_tensorflow/third_party/gpus/cuda_configure.bzl", line 637, column 19, in _exec_find_cuda_config
        return execute(repository_ctx, [python_bin, "-c", decompress_and_execute_cmd])
    File "/root/.cache/bazel/_bazel_root/f232ff2f81a27f8c8fd1521d116cbf0f/external/org_tensorflow/third_party/remote_config/common.bzl", line 230, column 13, in execute
        fail(
Error in fail: Repository command failed
Could not find any cublas_api.h matching version '' in any subdirectory:
        ''
        'include'
        'include/cuda'
        'include/*-linux-gnu'
        'extras/CUPTI/include'
        'include/cuda/CUPTI'
of:
        '/lib/x86_64-linux-gnu'
        '/usr'
        '/usr/local/cuda'
        '/usr/local/cuda/lib64/stubs'
        '/usr/local/cuda/targets/x86_64-linux/lib'
ERROR: Error fetching repository: Traceback (most recent call last):
    File "/root/.cache/bazel/_bazel_root/f232ff2f81a27f8c8fd1521d116cbf0f/external/org_tensorflow/third_party/gpus/cuda_configure.bzl", line 1401, column 38, in _cuda_autoconf_impl
        _create_local_cuda_repository(repository_ctx)
    File "/root/.cache/bazel/_bazel_root/f232ff2f81a27f8c8fd1521d116cbf0f/external/org_tensorflow/third_party/gpus/cuda_configure.bzl", line 978, column 35, in _create_local_cuda_repository
        cuda_config = _get_cuda_config(repository_ctx, find_cuda_config_script)
    File "/root/.cache/bazel/_bazel_root/f232ff2f81a27f8c8fd1521d116cbf0f/external/org_tensorflow/third_party/gpus/cuda_configure.bzl", line 666, column 30, in _get_cuda_config
        config = find_cuda_config(repository_ctx, find_cuda_config_script, ["cuda", "cudnn"])
    File "/root/.cache/bazel/_bazel_root/f232ff2f81a27f8c8fd1521d116cbf0f/external/org_tensorflow/third_party/gpus/cuda_configure.bzl", line 643, column 41, in find_cuda_config
        exec_result = _exec_find_cuda_config(repository_ctx, script_path, cuda_libraries)
    File "/root/.cache/bazel/_bazel_root/f232ff2f81a27f8c8fd1521d116cbf0f/external/org_tensorflow/third_party/gpus/cuda_configure.bzl", line 637, column 19, in _exec_find_cuda_config
        return execute(repository_ctx, [python_bin, "-c", decompress_and_execute_cmd])
    File "/root/.cache/bazel/_bazel_root/f232ff2f81a27f8c8fd1521d116cbf0f/external/org_tensorflow/third_party/remote_config/common.bzl", line 230, column 13, in execute
        fail(
Error in fail: Repository command failed
Could not find any cublas_api.h matching version '' in any subdirectory:
        ''
        'include'
        'include/cuda'
        'include/*-linux-gnu'
        'extras/CUPTI/include'
        'include/cuda/CUPTI'
of:
        '/lib/x86_64-linux-gnu'
        '/usr'
        '/usr/local/cuda'
        '/usr/local/cuda/lib64/stubs'
        '/usr/local/cuda/targets/x86_64-linux/lib'
INFO: Repository com_google_protobuf instantiated at:
  /workspace/repos/mediapipe/WORKSPACE:130:13: in <toplevel>
Repository rule http_archive defined at:
  /root/.cache/bazel/_bazel_root/f232ff2f81a27f8c8fd1521d116cbf0f/external/bazel_tools/tools/build_defs/repo/http.bzl:336:31: in <toplevel>
DEBUG: Rule 'rules_cc' indicated that a canonical reproducible form can be obtained by modifying arguments sha256 = "3839996049629e6377abdfd04681ddeeb0cc3db13b9d2ff81bf46700cb4529f7"
DEBUG: Repository rules_cc instantiated at:
  /workspace/repos/mediapipe/WORKSPACE:36:13: in <toplevel>
Repository rule http_archive defined at:
  /root/.cache/bazel/_bazel_root/f232ff2f81a27f8c8fd1521d116cbf0f/external/bazel_tools/tools/build_defs/repo/http.bzl:336:31: in <toplevel>
ERROR: /workspace/repos/mediapipe/mediapipe/examples/desktop/object_detection/BUILD:19:10: //mediapipe/examples/desktop/object_detection:object_detection_tensorflow depends on @local_config_cuda//crosstool:toolchain in repository @local_config_cuda which failed to fetch. no such package '@local_config_cuda//crosstool': Repository command failed
Could not find any cublas_api.h matching version '' in any subdirectory:
        ''
        'include'
        'include/cuda'
        'include/*-linux-gnu'
        'extras/CUPTI/include'
        'include/cuda/CUPTI'
of:
        '/lib/x86_64-linux-gnu'
        '/usr'
        '/usr/local/cuda'
        '/usr/local/cuda/lib64/stubs'
        '/usr/local/cuda/targets/x86_64-linux/lib'
ERROR: Analysis of target '//mediapipe/examples/desktop/object_detection:object_detection_tensorflow' failed; build aborted: Analysis failed
INFO: Elapsed time: 0.834s
INFO: 0 processes.
FAILED: Build did NOT complete successfully (0 packages loaded, 0 targets configured)
    currently loading: @org_tensorflow//tensorflow/core
    Fetching @local_config_cc; fetching

mcdonasd1212 commented 2 years ago

@KulkarniKaustubh have you been able to build Mediapipe for GPU? I am about your solution.

KulkarniKaustubh commented 2 years ago

@mcdonasd1212 No, I haven't been able to build it for CUDA. Still waiting for a mod to reply 😅

mcdonasd1212 commented 2 years ago

I have mediapipe pose running on a machine with an i5 cpu and GTX 1660 gpu. I followed the Open GL instructions since I could not get a successful build with tensorflow. With the light model, gpu usage is around 23% and fps is 25-27 fps.

For our implementation we need 30+ fps. Any recommendations?

RohitSingh1226 commented 2 years ago

I have mediapipe pose running on a machine with an i5 cpu and GTX 1660 gpu. I followed the Open GL instructions since I could not get a successful build with tensorflow. With the light model, gpu usage is around 23% and fps is 25-27 fps.

For our implementation we need 30+ fps. Any recommendations?

@mcdonasd1212 But for the Open GL part, we would still require an opencv build correct? I was running mediapipe hands on the gpu, There was some error with the opencv installation.

Do I need to build opencv with cudnn,cuda?

mcdonasd1212 commented 2 years ago

I have never had good luck with the pip install of OpenCV. So yes, O would do an openCV build. My notes show what worked.

kuaashish commented 1 year ago

Hello @mcdonasd1212, We are upgrading the MediaPipe Legacy Solutions to new MediaPipe solutions However, the libraries, documentation, and source code for all the MediaPipe Legacy Solutions will continue to be available in our GitHub repository and through library distribution services, such as Maven and NPM.

You can continue to use those legacy solutions in your applications if you choose. Though, we would request you to check new MediaPipe solutions which can help you more easily build and customize ML solutions for your applications. These new solutions will provide a superset of capabilities available in the legacy solutions.

github-actions[bot] commented 1 year ago

This issue has been marked stale because it has no recent activity since 7 days. It will be closed if no further activity occurs. Thank you.

github-actions[bot] commented 1 year ago

This issue was closed due to lack of activity after being marked stale for past 7 days.

google-ml-butler[bot] commented 1 year ago

Are you satisfied with the resolution of your issue? Yes No

google-ai-edge / mediapipe

Mediapipe Build for GPU ERROR #3020

OS

Hardware

PREREQUISITES

Intall Cuda and cuDnn

Install Cuda 11.2 (Highest version that works with Tensorflow per https://www.tensorflow.org/install/gpu)

Install cuDNN 8.2.1

Make virutal enviroment

Build OpenCV from source

Build MediaPipe for GPU

Instructions to Build from Source