search

ROCm / tensorflow-upstream

TensorFlow ROCm port
https://tensorflow.org
Apache License 2.0
688 stars 94 forks source link

//tensorflow/python:bfloat16_lib' failed (Exit 1),crosstool_wrapper_driver_is_not_gcc failed #1036

Closed ccywxhn closed 4 years ago

ccywxhn commented 4 years ago

ERROR: /disk/ccy/ROCm/TF/tensorflow-upstream/tensorflow/python/BUILD:455:1: C++ compilation of rule '//tensorflow/python:bfloat16_lib' failed (Exit 1): crosstool_wrapper_driver_is_not_gcc failed: error executing command (cd /home/ccy/.cache/bazel/_bazel_ccy/3d2a9c8d528118f0815a98d1432af665/execroot/org_tensorflow && \ exec env - \ PATH=/home/ccy/.local/bin:/home/ccy/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/opt/rocm/bin:/opt/rocm/profiler/bin:/opt/rocm/opencl/bin/x86_64:/opt/rocm/bin:/opt/rocm/profiler/bin:/opt/rocm/opencl/bin/x86_64:/home/ccy/bin \ PWD=/proc/self/cwd \ PYTHON_BIN_PATH=/usr/bin/python3 \ PYTHON_LIB_PATH=/usr/lib/python3/dist-packages \ TF2_BEHAVIOR=1 \ TF_CONFIGURE_IOS=0 \ TF_ENABLE_XLA=1 \ TF_NEED_ROCM=1 \ external/local_config_rocm/crosstool/clang/bin/crosstool_wrapper_driver_is_not_gcc -U_FORTIFY_SOURCE -fstack-protector -Wall -Wunused-but-set-parameter -Wno-free-nonheap-object -fno-omit-frame-pointer -g0 -O2 '-D_FORTIFY_SOURCE=1' -DNDEBUG -ffunction-sections -fdata-sections '-std=c++11' -MD -MF bazel-out/k8-opt/bin/tensorflow/python/_objs/bfloat16_lib/bfloat16.pic.d '-frandom-seed=bazel-out/k8-opt/bin/tensorflow/python/_objs/bfloat16_lib/bfloat16.pic.o' -fPIC -DSQLITE_OMIT_DEPRECATED -DLLVM_ENABLE_STATS -DSTDC_LIMIT_MACROS -DSTDC_CONSTANT_MACROS -DSTDC_FORMAT_MACROS -DLLVM_BUILD_GLOBAL_ISEL -DTENSORFLOW_USE_CUSTOM_CONTRACTION_KERNEL -DTENSORFLOW_USE_MKLDNN_CONTRACTION_KERNEL '-DGRPC_ARES=0' -DHAVE_SYS_UIO_H -DTF_USE_SNAPPY -DCURL_STATICLIB -DPLATFORM_LINUX -DENABLE_CURL_CLIENT -DOPENSSL_IS_BORINGSSL -DEIGEN_MPL2_ONLY '-DEIGEN_MAX_ALIGN_BYTES=64' '-DEIGEN_HAS_TYPE_TRAITS=0' -D__CLANG_SUPPORT_DYN_ANNOTATION -iquote . -iquote bazel-out/k8-opt/bin -iquote external/local_config_python -iquote bazel-out/k8-opt/bin/external/local_config_python -iquote external/com_google_absl -iquote bazel-out/k8-opt/bin/external/com_google_absl -iquote external/eigen_archive -iquote bazel-out/k8-opt/bin/external/eigen_archive -iquote external/local_config_sycl -iquote bazel-out/k8-opt/bin/external/local_config_sycl -iquote external/nsync -iquote bazel-out/k8-opt/bin/external/nsync -iquote external/gif -iquote bazel-out/k8-opt/bin/external/gif -iquote external/libjpeg_turbo -iquote bazel-out/k8-opt/bin/external/libjpeg_turbo -iquote external/com_google_protobuf -iquote bazel-out/k8-opt/bin/external/com_google_protobuf -iquote external/com_googlesource_code_re2 -iquote bazel-out/k8-opt/bin/external/com_googlesource_code_re2 -iquote external/farmhash_archive -iquote bazel-out/k8-opt/bin/external/farmhash_archive -iquote external/fft2d -iquote bazel-out/k8-opt/bin/external/fft2d -iquote external/highwayhash -iquote bazel-out/k8-opt/bin/external/highwayhash -iquote external/zlib -iquote bazel-out/k8-opt/bin/external/zlib -iquote external/local_config_rocm -iquote bazel-out/k8-opt/bin/external/local_config_rocm -iquote external/local_config_cuda -iquote bazel-out/k8-opt/bin/external/local_config_cuda -iquote external/double_conversion -iquote bazel-out/k8-opt/bin/external/double_conversion -iquote external/snappy -iquote bazel-out/k8-opt/bin/external/snappy -iquote external/local_config_tensorrt -iquote bazel-out/k8-opt/bin/external/local_config_tensorrt -iquote external/curl -iquote bazel-out/k8-opt/bin/external/curl -iquote external/boringssl -iquote bazel-out/k8-opt/bin/external/boringssl -iquote external/jsoncpp_git -iquote bazel-out/k8-opt/bin/external/jsoncpp_git -iquote external/aws -iquote bazel-out/k8-opt/bin/external/aws -iquote external/aws-c-common -iquote bazel-out/k8-opt/bin/external/aws-c-common -iquote external/aws-c-event-stream -iquote bazel-out/k8-opt/bin/external/aws-c-event-stream -iquote external/aws-checksums -iquote bazel-out/k8-opt/bin/external/aws-checksums -iquote external/mkl_dnn -iquote bazel-out/k8-opt/bin/external/mkl_dnn -iquote external/llvm-project -iquote bazel-out/k8-opt/bin/external/llvm-project -iquote external/png -iquote bazel-out/k8-opt/bin/external/png -iquote external/lmdb -iquote bazel-out/k8-opt/bin/external/lmdb -iquote external/gemmlowp -iquote bazel-out/k8-opt/bin/external/gemmlowp -iquote external/icu -iquote bazel-out/k8-opt/bin/external/icu -iquote external/org_sqlite -iquote bazel-out/k8-opt/bin/external/org_sqlite -iquote external/com_github_grpc_grpc -iquote bazel-out/k8-opt/bin/external/com_github_grpc_grpc -iquote external/upb -iquote bazel-out/k8-opt/bin/external/upb -Ibazel-out/k8-opt/bin/external/local_config_cuda/cuda/_virtual_includes/cudnn_header -Ibazel-out/k8-opt/bin/external/local_config_cuda/cuda/_virtual_includes/cuda_headers_virtual -Ibazel-out/k8-opt/bin/external/local_config_tensorrt/_virtual_includes/tensorrt_headers -Ibazel-out/k8-opt/bin/external/llvm-project/mlir/_virtual_includes/CallOpInterfacesIncGen -Ibazel-out/k8-opt/bin/external/llvm-project/mlir/_virtual_includes/DialectSymbolRegistry -Ibazel-out/k8-opt/bin/external/llvm-project/mlir/_virtual_includes/InferTypeOpInterfaceIncGen -Ibazel-out/k8-opt/bin/external/llvm-project/mlir/_virtual_includes/OpAsmInterfacesIncGen -Ibazel-out/k8-opt/bin/external/llvm-project/mlir/_virtual_includes/SideEffectInterfacesIncGen -Ibazel-out/k8-opt/bin/external/llvm-project/mlir/_virtual_includes/AffineOpsIncGen -Ibazel-out/k8-opt/bin/external/llvm-project/mlir/_virtual_includes/LoopLikeInterfaceIncGen -Ibazel-out/k8-opt/bin/external/llvm-project/mlir/_virtual_includes/ControlFlowInterfacesIncGen -Ibazel-out/k8-opt/bin/external/llvm-project/mlir/_virtual_includes/StandardOpsIncGen -Ibazel-out/k8-opt/bin/external/llvm-project/mlir/_virtual_includes/ViewLikeInterfaceIncGen -Ibazel-out/k8-opt/bin/external/llvm-project/mlir/_virtual_includes/LoopOpsIncGen -Ibazel-out/k8-opt/bin/external/llvm-project/mlir/_virtual_includes/DerivedAttributeOpInterfaceIncGen -Ibazel-out/k8-opt/bin/external/llvm-project/mlir/_virtual_includes/ParserTokenKinds -Ibazel-out/k8-opt/bin/external/llvm-project/mlir/_virtual_includes/TransformsPassIncGen -Ibazel-out/k8-opt/bin/external/llvm-project/mlir/_virtual_includes/LoopPassIncGen -Ibazel-out/k8-opt/bin/external/llvm-project/mlir/_virtual_includes/ShapeOpsIncGen -isystem external/local_config_python/numpy_include -isystem bazel-out/k8-opt/bin/external/local_config_python/numpy_include -isystem external/local_config_python/python_include -isystem bazel-out/k8-opt/bin/external/local_config_python/python_include -isystem external/eigen_archive -isystem bazel-out/k8-opt/bin/external/eigen_archive -isystem external/nsync/public -isystem bazel-out/k8-opt/bin/external/nsync/public -isystem external/gif -isystem bazel-out/k8-opt/bin/external/gif -isystem external/com_google_protobuf/src -isystem bazel-out/k8-opt/bin/external/com_google_protobuf/src -isystem external/farmhash_archive/src -isystem bazel-out/k8-opt/bin/external/farmhash_archive/src -isystem external/zlib -isystem bazel-out/k8-opt/bin/external/zlib -isystem external/local_config_rocm/rocm -isystem bazel-out/k8-opt/bin/external/local_config_rocm/rocm -isystem external/local_config_rocm/rocm/rocm/include -isystem bazel-out/k8-opt/bin/external/local_config_rocm/rocm/rocm/include -isystem external/local_config_rocm/rocm/rocm/include/rocrand -isystem bazel-out/k8-opt/bin/external/local_config_rocm/rocm/rocm/include/rocrand -isystem external/local_config_cuda/cuda -isystem bazel-out/k8-opt/bin/external/local_config_cuda/cuda -isystem external/local_config_cuda/cuda/cuda/include -isystem bazel-out/k8-opt/bin/external/local_config_cuda/cuda/cuda/include -isystem external/double_conversion -isystem bazel-out/k8-opt/bin/external/double_conversion -isystem external/curl/include -isystem bazel-out/k8-opt/bin/external/curl/include -isystem external/boringssl/src/include -isystem bazel-out/k8-opt/bin/external/boringssl/src/include -isystem external/jsoncpp_git/include -isystem bazel-out/k8-opt/bin/external/jsoncpp_git/include -isystem external/aws/aws-cpp-sdk-core/include -isystem bazel-out/k8-opt/bin/external/aws/aws-cpp-sdk-core/include -isystem external/aws/aws-cpp-sdk-s3/include -isystem bazel-out/k8-opt/bin/external/aws/aws-cpp-sdk-s3/include -isystem external/aws/aws-cpp-sdk-transfer/include -isystem bazel-out/k8-opt/bin/external/aws/aws-cpp-sdk-transfer/include -isystem external/aws-c-common/include -isystem bazel-out/k8-opt/bin/external/aws-c-common/include -isystem external/aws-c-event-stream/include -isystem bazel-out/k8-opt/bin/external/aws-c-event-stream/include -isystem external/aws-checksums/include -isystem bazel-out/k8-opt/bin/external/aws-checksums/include -isystem external/mkl_dnn/include -isystem bazel-out/k8-opt/bin/external/mkl_dnn/include -isystem external/mkl_dnn/src -isystem bazel-out/k8-opt/bin/external/mkl_dnn/src -isystem external/mkl_dnn/src/common -isystem bazel-out/k8-opt/bin/external/mkl_dnn/src/common -isystem external/mkl_dnn/src/cpu -isystem bazel-out/k8-opt/bin/external/mkl_dnn/src/cpu -isystem external/mkl_dnn/src/cpu/gemm -isystem bazel-out/k8-opt/bin/external/mkl_dnn/src/cpu/gemm -isystem external/mkl_dnn/src/cpu/xbyak -isystem bazel-out/k8-opt/bin/external/mkl_dnn/src/cpu/xbyak -isystem external/llvm-project/llvm/include -isystem bazel-out/k8-opt/bin/external/llvm-project/llvm/include -isystem external/local_config_rocm/rocm/rocm/include/hipcub -isystem bazel-out/k8-opt/bin/external/local_config_rocm/rocm/rocm/include/hipcub -isystem external/local_config_rocm/rocm/rocm/include/rocprim -isystem bazel-out/k8-opt/bin/external/local_config_rocm/rocm/rocm/include/rocprim -isystem external/png -isystem bazel-out/k8-opt/bin/external/png -isystem external/icu/icu4c/source/common -isystem bazel-out/k8-opt/bin/external/icu/icu4c/source/common -isystem external/llvm-project/mlir/include -isystem bazel-out/k8-opt/bin/external/llvm-project/mlir/include -isystem tensorflow/compiler/mlir/tensorflow/include -isystem bazel-out/k8-opt/bin/tensorflow/compiler/mlir/tensorflow/include -isystem tensorflow/compiler/mlir/xla/include -isystem bazel-out/k8-opt/bin/tensorflow/compiler/mlir/xla/include -isystem external/llvm-project/llvm/include/llvm/IR -isystem bazel-out/k8-opt/bin/external/llvm-project/llvm/include/llvm/IR -isystem external/llvm-project/llvm/lib/Transforms/InstCombine -isystem bazel-out/k8-opt/bin/external/llvm-project/llvm/lib/Transforms/InstCombine -isystem external/llvm-project/llvm/lib/Target/X86 -isystem bazel-out/k8-opt/bin/external/llvm-project/llvm/lib/Target/X86 -isystem external/llvm-project/llvm/lib/Target/AMDGPU -isystem bazel-out/k8-opt/bin/external/llvm-project/llvm/lib/Target/AMDGPU -isystem external/llvm-project/llvm/lib/Target/NVPTX -isystem bazel-out/k8-opt/bin/external/llvm-project/llvm/lib/Target/NVPTX -isystem external/com_github_grpc_grpc/include -isystem bazel-out/k8-opt/bin/external/com_github_grpc_grpc/include -isystem external/com_github_grpc_grpc/src/core/ext/upb-generated -isystem bazel-out/k8-opt/bin/external/com_github_grpc_grpc/src/core/ext/upb-generated -isystem external/com_github_grpc_grpc/third_party/address_sorting/include -isystem bazel-out/k8-opt/bin/external/com_github_grpc_grpc/third_party/address_sorting/include -w -DAUTOLOAD_DYNAMIC_KERNELS '-march=haswell' -Wno-sign-compare '-std=c++14' -fno-canonical-system-headers -Wno-builtin-macro-redefined '-DDATE="redacted"' '-DTIMESTAMP="redacted"' '-DTIME="redacted"' '-DTENSORFLOW_USE_ROCM=1' -DHIP_PLATFORM_HCC -DEIGEN_USE_HIP -no-canonical-prefixes -fno-canonical-system-headers -c tensorflow/python/lib/core/bfloat16.cc -o bazel-out/k8-opt/bin/tensorflow/python/_objs/bfloat16_lib/bfloat16.pic.o) Execution platform: @local_config_platform//:host tensorflow/python/lib/core/bfloat16.cc: In function 'bool tensorflow::{anonymous}::Initialize()': tensorflow/python/lib/core/bfloat16.cc:636:36: error: no match for call to '(tensorflow::{anonymous}::Initialize()::<lambda(const char, PyUFuncGenericFunction, const std::array<int, 3>&)>) (const char [6], , const std::array<int, 3>&)' compare_types)) { ^ tensorflow/python/lib/core/bfloat16.cc:610:60: note: candidate: tensorflow::{anonymous}::Initialize()::<lambda(const char, PyUFuncGenericFunction, const std::array<int, 3>&)> const std::array<int, 3>& types) { ^ tensorflow/python/lib/core/bfloat16.cc:610:60: note: no known conversion for argument 2 from '' to 'PyUFuncGenericFunction {aka void (*)(char, const long int, const long int, void)}' tensorflow/python/lib/core/bfloat16.cc:640:36: error: no match for call to '(tensorflow::{anonymous}::Initialize()::<lambda(const char, PyUFuncGenericFunction, const std::array<int, 3>&)>) (const char [10], , const std::array<int, 3>&)' compare_types)) { ^ tensorflow/python/lib/core/bfloat16.cc:610:60: note: candidate: tensorflow::{anonymous}::Initialize()::<lambda(const char, PyUFuncGenericFunction, const std::array<int, 3>&)> const std::array<int, 3>& types) { ^ tensorflow/python/lib/core/bfloat16.cc:610:60: note: no known conversion for argument 2 from '' to 'PyUFuncGenericFunction {aka void ()(char, const long int, const long int, void)}' tensorflow/python/lib/core/bfloat16.cc:643:77: error: no match for call to '(tensorflow::{anonymous}::Initialize()::<lambda(const char, PyUFuncGenericFunction, const std::array<int, 3>&)>) (const char [5], , const std::array<int, 3>&)' if (!register_ufunc("less", CompareUFunc, compare_types)) { ^ tensorflow/python/lib/core/bfloat16.cc:610:60: note: candidate: tensorflow::{anonymous}::Initialize()::<lambda(const char, PyUFuncGenericFunction, const std::array<int, 3>&)> const std::array<int, 3>& types) { ^ tensorflow/python/lib/core/bfloat16.cc:610:60: note: no known conversion for argument 2 from '' to 'PyUFuncGenericFunction {aka void ()(char, const long int, const long int, void)}' tensorflow/python/lib/core/bfloat16.cc:647:36: error: no match for call to '(tensorflow::{anonymous}::Initialize()::<lambda(const char, PyUFuncGenericFunction, const std::array<int, 3>&)>) (const char [8], , const std::array<int, 3>&)' compare_types)) { ^ tensorflow/python/lib/core/bfloat16.cc:610:60: note: candidate: tensorflow::{anonymous}::Initialize()::<lambda(const char, PyUFuncGenericFunction, const std::array<int, 3>&)> const std::array<int, 3>& types) { ^ tensorflow/python/lib/core/bfloat16.cc:610:60: note: no known conversion for argument 2 from '' to 'PyUFuncGenericFunction {aka void ()(char, const long int, const long int, void)}' tensorflow/python/lib/core/bfloat16.cc:651:36: error: no match for call to '(tensorflow::{anonymous}::Initialize()::<lambda(const char, PyUFuncGenericFunction, const std::array<int, 3>&)>) (const char [11], , const std::array<int, 3>&)' compare_types)) { ^ tensorflow/python/lib/core/bfloat16.cc:610:60: note: candidate: tensorflow::{anonymous}::Initialize()::<lambda(const char, PyUFuncGenericFunction, const std::array<int, 3>&)> const std::array<int, 3>& types) { ^ tensorflow/python/lib/core/bfloat16.cc:610:60: note: no known conversion for argument 2 from '' to 'PyUFuncGenericFunction {aka void ()(char, const long int, const long int, void)}' tensorflow/python/lib/core/bfloat16.cc:655:36: error: no match for call to '(tensorflow::{anonymous}::Initialize()::<lambda(const char, PyUFuncGenericFunction, const std::array<int, 3>&)>) (const char [14], , const std::array<int, 3>&)' compare_types)) { ^ tensorflow/python/lib/core/bfloat16.cc:610:60: note: candidate: tensorflow::{anonymous}::Initialize()::<lambda(const char, PyUFuncGenericFunction, const std::array<int, 3>&)> const std::array<int, 3>& types) { ^ tensorflow/python/lib/core/bfloat16.cc:610:60: note: no known conversion for argument 2 from '' to 'PyUFuncGenericFunction {aka void ()(char, const long int, const long int, void*)}' Target //tensorflow/tools/pip_package:build_pip_package failed to build INFO: Elapsed time: 6.582s, Critical Path: 6.21s INFO: 7 processes: 7 local. FAILED: Build did NOT complete successfully

jerryyin commented 4 years ago

Please provide more details in regards on how to reproduce:

ccywxhn commented 4 years ago
  1. Ubuntu18.04 , gcc 7.5
  2. ROCm3.3 platform
  3. vega10 card
  4. Install bazel-3.0.0
  5. git clone https://github.com/ROCmSoftwarePlatform/tensorflow-upstream.git
  6. git checkout release-doc-rocm3.3-tf2.2-rc
  7. ./configure, and set ROCm support
  8. bazel build --local_ram_resources=2048 --config=opt --config=rocm //tensorflow/tools/pip_package:build_pip_package --verbose_failures

  1. bazel build --local_ram_resources=2048 --spawn_strategy=standalone --config=opt --config=rocm //tensorflow/tools/pip_package:build_pip_package --verbose_failures

  2. bazel build -c opt --local_ram_resources=2048 --config=opt --config=rocm //tensorflow/tools/pip_package:build_pip_package

I want to build tensorflow2.xx based on ROCm, should choose stable branch or tag?

jerryyin commented 4 years ago

You have two options as workaround:

yiakwy-xpu-ml-framework-team commented 5 months ago

@ccywxhn @jerryyin This complaints due to upstream change from numpy where

https://github.com/numpy/numpy/pull/15251

the const-ness of PyUFuncGenericFunction changed. The quick fix is to revert numpy before 1.19.0.

Note if you are python 3.10 + tensorflow 1.15 user, you have to apply the diff#1 from tensorflow 1.15 side to reflect change of Numpy ABI since numpy <1.19.0 cannot compile directly without apply diff#2 from numpy side: