Building ngraph-tf fails

[pjattke]

When trying to build he-transformer, I received the following error:

...
ERROR: /home/he-transformer/build/ext_ngraph_tf/src/ext_ngraph_tf/build_cmake/tensorflow/tensorflow/python/BUILD:336:1: C++ compilation of rule '//tensorflow/python:bfloat16_lib' failed (Exit 1)
tensorflow/python/lib/core/bfloat16.cc: In lambda function:
tensorflow/python/lib/core/bfloat16.cc:615:32: warning: comparison between signed and unsigned integer expressions [-Wsign-compare]
     if (types.size() != ufunc->nargs) {
                                ^
tensorflow/python/lib/core/bfloat16.cc: In function 'bool tensorflow::{anonymous}::Initialize()':
tensorflow/python/lib/core/bfloat16.cc:634:36: error: no match for call to '(tensorflow::{anonymous}::Initialize()::__lambda0) (const char [6], <unresolved overloaded function type>, const std::array<int, 3ul>&)'
                       compare_types)) {
                                    ^
tensorflow/python/lib/core/bfloat16.cc:607:27: note: candidate is:
   auto register_ufunc = [&](const char* name, PyUFuncGenericFunction fn,
                           ^
tensorflow/python/lib/core/bfloat16.cc:608:60: note: tensorflow::{anonymous}::Initialize()::__lambda0
                             const std::array<int, 3>& types) {
                                                            ^
tensorflow/python/lib/core/bfloat16.cc:608:60: note:   no known conversion for argument 2 from '<unresolved overloaded function type>' to 'PyUFuncGenericFunction {aka void (*)(char**, const long int*, const long int*, void*)}'
tensorflow/python/lib/core/bfloat16.cc:638:36: error: no match for call to '(tensorflow::{anonymous}::Initialize()::__lambda0) (const char [10], <unresolved overloaded function type>, const std::array<int, 3ul>&)'
                       compare_types)) {
                                    ^
tensorflow/python/lib/core/bfloat16.cc:607:27: note: candidate is:
   auto register_ufunc = [&](const char* name, PyUFuncGenericFunction fn,
                           ^
tensorflow/python/lib/core/bfloat16.cc:608:60: note: tensorflow::{anonymous}::Initialize()::__lambda0
                             const std::array<int, 3>& types) {
                                                            ^
tensorflow/python/lib/core/bfloat16.cc:608:60: note:   no known conversion for argument 2 from '<unresolved overloaded function type>' to 'PyUFuncGenericFunction {aka void (*)(char**, const long int*, const long int*, void*)}'
tensorflow/python/lib/core/bfloat16.cc:641:77: error: no match for call to '(tensorflow::{anonymous}::Initialize()::__lambda0) (const char [5], <unresolved overloaded function type>, const std::array<int, 3ul>&)'
   if (!register_ufunc("less", CompareUFunc<Bfloat16LtFunctor>, compare_types)) {
                                                                             ^
tensorflow/python/lib/core/bfloat16.cc:607:27: note: candidate is:
   auto register_ufunc = [&](const char* name, PyUFuncGenericFunction fn,
                           ^
tensorflow/python/lib/core/bfloat16.cc:608:60: note: tensorflow::{anonymous}::Initialize()::__lambda0
                             const std::array<int, 3>& types) {
                                                            ^
tensorflow/python/lib/core/bfloat16.cc:608:60: note:   no known conversion for argument 2 from '<unresolved overloaded function type>' to 'PyUFuncGenericFunction {aka void (*)(char**, const long int*, const long int*, void*)}'
tensorflow/python/lib/core/bfloat16.cc:645:36: error: no match for call to '(tensorflow::{anonymous}::Initialize()::__lambda0) (const char [8], <unresolved overloaded function type>, const std::array<int, 3ul>&)'
                       compare_types)) {
                                    ^
tensorflow/python/lib/core/bfloat16.cc:607:27: note: candidate is:
   auto register_ufunc = [&](const char* name, PyUFuncGenericFunction fn,
                           ^
tensorflow/python/lib/core/bfloat16.cc:608:60: note: tensorflow::{anonymous}::Initialize()::__lambda0
                             const std::array<int, 3>& types) {
                                                            ^
tensorflow/python/lib/core/bfloat16.cc:608:60: note:   no known conversion for argument 2 from '<unresolved overloaded function type>' to 'PyUFuncGenericFunction {aka void (*)(char**, const long int*, const long int*, void*)}'
tensorflow/python/lib/core/bfloat16.cc:649:36: error: no match for call to '(tensorflow::{anonymous}::Initialize()::__lambda0) (const char [11], <unresolved overloaded function type>, const std::array<int, 3ul>&)'
                       compare_types)) {
                                    ^
tensorflow/python/lib/core/bfloat16.cc:607:27: note: candidate is:
   auto register_ufunc = [&](const char* name, PyUFuncGenericFunction fn,
                           ^
tensorflow/python/lib/core/bfloat16.cc:608:60: note: tensorflow::{anonymous}::Initialize()::__lambda0
                             const std::array<int, 3>& types) {
                                                            ^
tensorflow/python/lib/core/bfloat16.cc:608:60: note:   no known conversion for argument 2 from '<unresolved overloaded function type>' to 'PyUFuncGenericFunction {aka void (*)(char**, const long int*, const long int*, void*)}'
tensorflow/python/lib/core/bfloat16.cc:653:36: error: no match for call to '(tensorflow::{anonymous}::Initialize()::__lambda0) (const char [14], <unresolved overloaded function type>, const std::array<int, 3ul>&)'
                       compare_types)) {
                                    ^
tensorflow/python/lib/core/bfloat16.cc:607:27: note: candidate is:
   auto register_ufunc = [&](const char* name, PyUFuncGenericFunction fn,
                           ^
tensorflow/python/lib/core/bfloat16.cc:608:60: note: tensorflow::{anonymous}::Initialize()::__lambda0
                             const std::array<int, 3>& types) {
                                                            ^
tensorflow/python/lib/core/bfloat16.cc:608:60: note:   no known conversion for argument 2 from '<unresolved overloaded function type>' to 'PyUFuncGenericFunction {aka void (*)(char**, const long int*, const long int*, void*)}'
Target //tensorflow/tools/pip_package:build_pip_package failed to build
Use --verbose_failures to see the command lines of failed build steps.
INFO: Elapsed time: 400.244s, Critical Path: 123.78s
INFO: 6738 processes: 6738 local.
FAILED: Build did NOT complete successfully
FAILED: Build did NOT complete successfully

As the provided Dockerfiles (see contrib/docker) also failed building he-transformer, I built my own Dockerfile:

# ******************************************************************************
# Copyright 2018-2020 Intel Corporation
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
#     http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
# ******************************************************************************

# Source: https://github.com/IntelAI/he-transformer/blob/master/contrib/docker/Dockerfile.he_transformer.ubuntu1804

# Environment to build and unit-test he-transformer
# with g++ 7.4.0
# with clang++ 9.0.1
# with python 3.6.8
# with cmake 3.14.4

FROM ubuntu:18.04

RUN apt-get update && apt-get install -y \
    python3-pip virtualenv python3-dev \
    git \
    unzip wget \
    sudo \
    bash-completion \
    build-essential cmake \
    software-properties-common \
    git \
    wget patch diffutils libtinfo-dev \
    autoconf libtool \
    doxygen graphviz \
    yapf3 python3-yapf \
    python python-dev python3 python3-dev \
    libomp-dev

RUN python3.6 -m pip install pip --upgrade && \ 
    pip3 install -U --user pip six 'numpy<1.19.0' wheel setuptools mock 'future>=0.17.1' && \ 
    pip3 install -U --user keras_applications --no-deps && \ 
    pip3 install -U --user keras_preprocessing --no-deps && \ 
    rm -rf /usr/bin/python && \
    ln -s /usr/bin/python3.6 /usr/bin/python

# Install clang-9
RUN wget -O - https://apt.llvm.org/llvm-snapshot.gpg.key | sudo apt-key add -
RUN apt-add-repository "deb http://apt.llvm.org/bionic/ llvm-toolchain-bionic-9 main"
RUN apt-get update && apt install -y clang-9 clang-tidy-9 clang-format-9

RUN apt-get update && apt-get install -y gcc-4.8 g++-4.8
RUN update-alternatives --install /usr/bin/gcc gcc /usr/bin/gcc-4.8 60 --slave /usr/bin/g++ g++ /usr/bin/g++-4.8 && \
  update-alternatives --install /usr/bin/gcc gcc /usr/bin/gcc-7 60 --slave /usr/bin/g++ g++ /usr/bin/g++-7

RUN apt-get clean autoclean && apt-get autoremove -y

# For ngraph-tf integration testing
RUN pip3 install --upgrade pip setuptools virtualenv==16.1.0

# SEAL requires newer version of CMake
RUN pip3 install cmake --upgrade

# Get bazel for ng-tf
RUN wget https://github.com/bazelbuild/bazel/releases/download/0.25.2/bazel-0.25.2-installer-linux-x86_64.sh
RUN chmod +x ./bazel-0.25.2-installer-linux-x86_64.sh
RUN bash ./bazel-0.25.2-installer-linux-x86_64.sh
WORKDIR /home

# *** end of Dockerfile from IntelAI/he-transformer repository ****************

ENV HE_TRANSFORMER /home/he-transformer

# Build HE-Transformer
# https://github.com/IntelAI/he-transformer#1-build-he-transformer
WORKDIR /home
RUN git clone https://github.com/IntelAI/he-transformer.git he-transformer

WORKDIR $HE_TRANSFORMER
# this is the same as the original cmake file but adds --verbose_build to the bazel build comand
COPY ngraph-tf.cmake /home/he-transformer/cmake/ngraph-tf.cmake

RUN mkdir build && \
    cd build && \ 
    cmake .. -DCMAKE_BUILD_TYPE=Release -DCMAKE_CXX_COMPILER=clang++-9 -DCMAKE_C_COMPILER=clang-9 -Werror
WORKDIR $HE_TRANSFORMER/build
RUN env VERBOSE=1 make -j 144 install

# Build the Python bindings for client
# https://github.com/IntelAI/he-transformer#1c-python-bindings-for-client
# RUN cd $HE_TRANSFORMER/build && \
#     source external/venv-tf-py3/bin/activate && \
#     make install python_client && \
#     pip install python/dist/pyhe_client-*.whl && \
#     python3 -c "import pyhe_client"

CMD ["/bin/bash"]

As you can see in the Dockerfile, it uses the master's latest commit to build he-transformer.

@fboemer Do you have any suggestions what could be the reason? Many thanks for your help!

[pjattke]

The indicated error C++ compilation of rule '//tensorflow/python:bfloat16_lib' failed (Exit 1) seems to be related to these Tensorflow issues:

• C++ compilation of rule '//tensorflow/python:bfloat16_lib' failed #40688

  • I am already using numpy<1.19.0 but it does not help.

• C++ compilation of rule '//tensorflow/python:bfloat16_lib' failed (Exit 1) tensorflow/python/lib/core/bfloat16.cc #41061

  • I used the exact same dependencies + versions as indicated here but it did not help.

I would highly appreciate any help and tips to solve this issue!

What are the tested and known to be working versions for he-transformer and ngraph-bridge?

[pjattke]

As suggested in this issue, I am modifying the tensorflow file bfloat16.cc by using the given command:

perl -pi.bak -e 's%, CompareUFunc%, (PyUFuncGenericFunction) CompareUFunc%g' tensorflow/python/lib/core/bfloat16.cc \

I have created a patch that is added as PATCH_COMMAND in ngraph-tf.cmake and modifies the file build_utils.py to make the required changes on the tensorflow files. The patch is given following:

diff --git a/tools/build_utils.py b/tools/build_utils.py
index 36c70ee7..42d93573 100755
--- a/tools/build_utils.py
+++ b/tools/build_utils.py
@@ -211,6 +211,7 @@ def build_tensorflow(venv_dir, src_dir, artifacts_dir, target_arch, verbosity):
     os.environ["TF_SET_ANDROID_WORKSPACE"] = "0"
     os.environ["CC_OPT_FLAGS"] = "-march=" + target_arch

+    command_executor("perl -pi.bak -e 's%, CompareUFunc%, (PyUFuncGenericFunction) CompareUFunc%g' tensorflow/python/lib/core/bfloat16.cc")
     command_executor("./configure")

     # Build the python package

Now, however, I am getting the following error:

adding 'ngraph_tensorflow_bridge-0.22.0rc3.data/purelib/ngraph_bridge/include/ngraph/frontend/onnx_import/op/div.hpp'
adding 'ngraph_tensorflow_bridge-0.22.0rc3.data/purelib/ngraph_bridge/include/ngraph/frontend/onnx_import/op/dropout.hpp'
adding 'ngraph_tensorflow_bridge-0.22.0rc3.data/purelib/ngraph_bridge/include/ngraph/frontend/onnx_import/op/elu.hpp'
...
adding 'ngraph_tensorflow_bridge-0.22.0rc3.data/purelib/ngraph_bridge/licenses/tensorflow-license.txt'
adding 'ngraph_tensorflow_bridge-0.22.0rc3.data/purelib/ngraph_bridge/licenses/tensorflow-models-license.txt'
adding 'ngraph_tensorflow_bridge-0.22.0rc3.dist-info/METADATA'
adding 'ngraph_tensorflow_bridge-0.22.0rc3.dist-info/WHEEL'
adding 'ngraph_tensorflow_bridge-0.22.0rc3.dist-info/top_level.txt'
adding 'ngraph_tensorflow_bridge-0.22.0rc3.dist-info/RECORD'
removing build/bdist.linux-x86_64/wheel
-- Installing: /home/he-transformer/build/ext_ngraph_tf/src/ext_ngraph_tf/build_cmake/artifacts/tools/tf2ngraph.py
-- Installing: /home/he-transformer/build/ext_ngraph_tf/src/ext_ngraph_tf/build_cmake/artifacts/tools/build_utils.py
-- Installing: /home/he-transformer/build/ext_ngraph_tf/src/ext_ngraph_tf/build_cmake/artifacts/tools/log_parser.py
-- Installing: /home/he-transformer/build/ext_ngraph_tf/src/ext_ngraph_tf/build_cmake/artifacts/lib/libngraph_bridge_device.so
-- Set runtime path of "/home/he-transformer/build/ext_ngraph_tf/src/ext_ngraph_tf/build_cmake/artifacts/lib/libngraph_bridge_device.so" to "$ORIGIN:$ORIGIN/../tensorflow"
-- Installing: /home/he-transformer/build/ext_ngraph_tf/src/ext_ngraph_tf/build_cmake/artifacts/test/gtest_ngtf
-- Set runtime path of "/home/he-transformer/build/ext_ngraph_tf/src/ext_ngraph_tf/build_cmake/artifacts/test/gtest_ngtf" to "$ORIGIN:$ORIGIN/../tensorflow"
-- Installing: /home/he-transformer/build/ext_ngraph_tf/src/ext_ngraph_tf/build_cmake/artifacts/test/test_axpy.pbtxt
-- Installing: /home/he-transformer/build/ext_ngraph_tf/src/ext_ngraph_tf/build_cmake/artifacts/test/test_axpy_launchop.pbtxt
-- Installing: /home/he-transformer/build/ext_ngraph_tf/src/ext_ngraph_tf/build_cmake/artifacts/test/test_axpy_8bit.pbtxt
-- Installing: /home/he-transformer/build/ext_ngraph_tf/src/ext_ngraph_tf/build_cmake/artifacts/test/test_axpy_int8_launchop.pbtxt
-- Installing: /home/he-transformer/build/ext_ngraph_tf/src/ext_ngraph_tf/build_cmake/artifacts/test/test_capture_prefetch.pbtxt
-- Installing: /home/he-transformer/build/ext_ngraph_tf/src/ext_ngraph_tf/build_cmake/artifacts/test/test_capture_prefetch_1.pbtxt
-- Installing: /home/he-transformer/build/ext_ngraph_tf/src/ext_ngraph_tf/build_cmake/artifacts/test/test_general_graph.pbtxt
-- Installing: /home/he-transformer/build/ext_ngraph_tf/src/ext_ngraph_tf/build_cmake/artifacts/test/test_catalog_for_prefetch.pbtxt
-- Installing: /home/he-transformer/build/ext_ngraph_tf/src/ext_ngraph_tf/build_cmake/artifacts/test/test_catalog_for_prefetch_1.pbtxt
-- Installing: /home/he-transformer/build/ext_ngraph_tf/src/ext_ngraph_tf/build_cmake/artifacts/test/python
-- Installing: /home/he-transformer/build/ext_ngraph_tf/src/ext_ngraph_tf/build_cmake/artifacts/test/python/test_tf2ngraph_shape_hints.py
-- Installing: /home/he-transformer/build/ext_ngraph_tf/src/ext_ngraph_tf/build_cmake/artifacts/test/python/test_set_backend.py
...
-- Installing: /home/he-transformer/build/ext_ngraph_tf/src/ext_ngraph_tf/build_cmake/artifacts/test/model_level_tests/models/bert/test1/core_run.sh
-- Installing: /home/he-transformer/build/ext_ngraph_tf/src/ext_ngraph_tf/build_cmake/artifacts/test/model_level_tests/models/bert/enable_ngraph.patch
make[3]: Leaving directory '/home/he-transformer/build/ext_ngraph_tf/src/ext_ngraph_tf/build_cmake'
Processing ./artifacts/ngraph_tensorflow_bridge-0.22.0rc3-py2.py3-none-manylinux1_x86_64.whl
Requirement already satisfied, skipping upgrade: tensorflow==1.14.0 in ./venv-tf-py3/lib/python3.6/site-packages (from ngraph-tensorflow-bridge==0.22.0rc3) (1.14.0)
Requirement already satisfied, skipping upgrade: tensorboard<1.15.0,>=1.14.0 in ./venv-tf-py3/lib/python3.6/site-packages (from tensorflow==1.14.0->ngraph-tensorflow-bridge==0.22.0rc3) (1.14.0)
Requirement already satisfied, skipping upgrade: tensorflow-estimator<1.15.0rc0,>=1.14.0rc0 in ./venv-tf-py3/lib/python3.6/site-packages (from tensorflow==1.14.0->ngraph-tensorflow-bridge==0.22.0rc3) (1.14.0)
Requirement already satisfied, skipping upgrade: wheel>=0.26 in ./venv-tf-py3/lib/python3.6/site-packages (from tensorflow==1.14.0->ngraph-tensorflow-bridge==0.22.0rc3) (0.34.2)
Requirement already satisfied, skipping upgrade: six>=1.10.0 in ./venv-tf-py3/lib/python3.6/site-packages (from tensorflow==1.14.0->ngraph-tensorflow-bridge==0.22.0rc3) (1.15.0)
Requirement already satisfied, skipping upgrade: wrapt>=1.11.1 in ./venv-tf-py3/lib/python3.6/site-packages (from tensorflow==1.14.0->ngraph-tensorflow-bridge==0.22.0rc3) (1.12.1)
Requirement already satisfied, skipping upgrade: absl-py>=0.7.0 in ./venv-tf-py3/lib/python3.6/site-packages (from tensorflow==1.14.0->ngraph-tensorflow-bridge==0.22.0rc3) (0.9.0)
Requirement already satisfied, skipping upgrade: keras-preprocessing>=1.0.5 in ./venv-tf-py3/lib/python3.6/site-packages (from tensorflow==1.14.0->ngraph-tensorflow-bridge==0.22.0rc3) (1.0.5)
Requirement already satisfied, skipping upgrade: keras-applications>=1.0.6 in ./venv-tf-py3/lib/python3.6/site-packages (from tensorflow==1.14.0->ngraph-tensorflow-bridge==0.22.0rc3) (1.0.8)
Requirement already satisfied, skipping upgrade: numpy<2.0,>=1.14.5 in ./venv-tf-py3/lib/python3.6/site-packages (from tensorflow==1.14.0->ngraph-tensorflow-bridge==0.22.0rc3) (1.19.1)
Requirement already satisfied, skipping upgrade: termcolor>=1.1.0 in ./venv-tf-py3/lib/python3.6/site-packages (from tensorflow==1.14.0->ngraph-tensorflow-bridge==0.22.0rc3) (1.1.0)
Requirement already satisfied, skipping upgrade: astor>=0.6.0 in ./venv-tf-py3/lib/python3.6/site-packages (from tensorflow==1.14.0->ngraph-tensorflow-bridge==0.22.0rc3) (0.8.1)
Requirement already satisfied, skipping upgrade: grpcio>=1.8.6 in ./venv-tf-py3/lib/python3.6/site-packages (from tensorflow==1.14.0->ngraph-tensorflow-bridge==0.22.0rc3) (1.30.0)
Requirement already satisfied, skipping upgrade: google-pasta>=0.1.6 in ./venv-tf-py3/lib/python3.6/site-packages (from tensorflow==1.14.0->ngraph-tensorflow-bridge==0.22.0rc3) (0.2.0)
Requirement already satisfied, skipping upgrade: gast>=0.2.0 in ./venv-tf-py3/lib/python3.6/site-packages (from tensorflow==1.14.0->ngraph-tensorflow-bridge==0.22.0rc3) (0.3.3)
Requirement already satisfied, skipping upgrade: protobuf>=3.6.1 in ./venv-tf-py3/lib/python3.6/site-packages (from tensorflow==1.14.0->ngraph-tensorflow-bridge==0.22.0rc3) (3.12.2)
Requirement already satisfied, skipping upgrade: setuptools>=41.0.0 in ./venv-tf-py3/lib/python3.6/site-packages (from tensorboard<1.15.0,>=1.14.0->tensorflow==1.14.0->ngraph-tensorflow-bridge==0.22.0rc3) (49.2.0)
Requirement already satisfied, skipping upgrade: markdown>=2.6.8 in ./venv-tf-py3/lib/python3.6/site-packages (from tensorboard<1.15.0,>=1.14.0->tensorflow==1.14.0->ngraph-tensorflow-bridge==0.22.0rc3) (3.2.2)
Requirement already satisfied, skipping upgrade: werkzeug>=0.11.15 in ./venv-tf-py3/lib/python3.6/site-packages (from tensorboard<1.15.0,>=1.14.0->tensorflow==1.14.0->ngraph-tensorflow-bridge==0.22.0rc3) (1.0.1)
Requirement already satisfied, skipping upgrade: h5py in ./venv-tf-py3/lib/python3.6/site-packages (from keras-applications>=1.0.6->tensorflow==1.14.0->ngraph-tensorflow-bridge==0.22.0rc3) (2.10.0)
Requirement already satisfied, skipping upgrade: importlib-metadata; python_version < "3.8" in ./venv-tf-py3/lib/python3.6/site-packages (from markdown>=2.6.8->tensorboard<1.15.0,>=1.14.0->tensorflow==1.14.0->ngraph-tensorflow-bridge==0.22.0rc3) (1.7.0)
Requirement already satisfied, skipping upgrade: zipp>=0.5 in ./venv-tf-py3/lib/python3.6/site-packages (from importlib-metadata; python_version < "3.8"->markdown>=2.6.8->tensorboard<1.15.0,>=1.14.0->tensorflow==1.14.0->ngraph-tensorflow-bridge==0.22.0rc3) (3.1.0)
Installing collected packages: ngraph-tensorflow-bridge
Successfully installed ngraph-tensorflow-bridge-0.22.0rc3
/home/he-transformer/build/ext_ngraph_tf/src/ext_ngraph_tf/build_cmake/venv-tf-py3/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:516: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_qint8 = np.dtype([("qint8", np.int8, 1)])
/home/he-transformer/build/ext_ngraph_tf/src/ext_ngraph_tf/build_cmake/venv-tf-py3/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:517: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_quint8 = np.dtype([("quint8", np.uint8, 1)])
/home/he-transformer/build/ext_ngraph_tf/src/ext_ngraph_tf/build_cmake/venv-tf-py3/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:518: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_qint16 = np.dtype([("qint16", np.int16, 1)])
/home/he-transformer/build/ext_ngraph_tf/src/ext_ngraph_tf/build_cmake/venv-tf-py3/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:519: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_quint16 = np.dtype([("quint16", np.uint16, 1)])
/home/he-transformer/build/ext_ngraph_tf/src/ext_ngraph_tf/build_cmake/venv-tf-py3/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:520: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_qint32 = np.dtype([("qint32", np.int32, 1)])
/home/he-transformer/build/ext_ngraph_tf/src/ext_ngraph_tf/build_cmake/venv-tf-py3/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:525: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  np_resource = np.dtype([("resource", np.ubyte, 1)])
/home/he-transformer/build/ext_ngraph_tf/src/ext_ngraph_tf/build_cmake/venv-tf-py3/lib/python3.6/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:541: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_qint8 = np.dtype([("qint8", np.int8, 1)])
/home/he-transformer/build/ext_ngraph_tf/src/ext_ngraph_tf/build_cmake/venv-tf-py3/lib/python3.6/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:542: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_quint8 = np.dtype([("quint8", np.uint8, 1)])
/home/he-transformer/build/ext_ngraph_tf/src/ext_ngraph_tf/build_cmake/venv-tf-py3/lib/python3.6/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:543: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_qint16 = np.dtype([("qint16", np.int16, 1)])
/home/he-transformer/build/ext_ngraph_tf/src/ext_ngraph_tf/build_cmake/venv-tf-py3/lib/python3.6/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:544: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_quint16 = np.dtype([("quint16", np.uint16, 1)])
/home/he-transformer/build/ext_ngraph_tf/src/ext_ngraph_tf/build_cmake/venv-tf-py3/lib/python3.6/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:545: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_qint32 = np.dtype([("qint32", np.int32, 1)])
/home/he-transformer/build/ext_ngraph_tf/src/ext_ngraph_tf/build_cmake/venv-tf-py3/lib/python3.6/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:550: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  np_resource = np.dtype([("resource", np.ubyte, 1)])
Building in with VERBOSE output messages

ARTIFACTS location: /home/he-transformer/build/ext_ngraph_tf/src/ext_ngraph_tf/build_cmake/artifacts
Loading virtual environment from: /home/he-transformer/build/ext_ngraph_tf/src/ext_ngraph_tf/build_cmake/venv-tf-py3
Loading virtual environment from: /home/he-transformer/build/ext_ngraph_tf/src/ext_ngraph_tf/build_cmake/venv-tf-py3
PIP location
Target Arch: native
Building TensorFlow from source
PYTHON_BIN_PATH: /home/he-transformer/build/ext_ngraph_tf/src/ext_ngraph_tf/build_cmake/venv-tf-py3/bin/python
SOURCE DIR: /home/he-transformer/build/ext_ngraph_tf/src/ext_ngraph_tf/build_cmake/tensorflow
ARTIFACTS DIR: /home/he-transformer/build/ext_ngraph_tf/src/ext_ngraph_tf/build_cmake/artifacts/tensorflow
TF Wheel: /home/he-transformer/build/ext_ngraph_tf/src/ext_ngraph_tf/build_cmake/artifacts/tensorflow/tensorflow-1.14.0-cp36-cp36m-linux_x86_64.whl
PYTHON_BIN_PATH: /home/he-transformer/build/ext_ngraph_tf/src/ext_ngraph_tf/build_cmake/venv-tf-py3/bin/python
SOURCE DIR: /home/he-transformer/build/ext_ngraph_tf/src/ext_ngraph_tf/build_cmake/tensorflow
ARTIFACTS DIR: /home/he-transformer/build/ext_ngraph_tf/src/ext_ngraph_tf/build_cmake/artifacts/tensorflow
Cannot remove: /home/he-transformer/build/ext_ngraph_tf/src/ext_ngraph_tf/build_cmake/artifacts/tensorflow/libtensorflow_cc.so.1
Copying bazel-bin/tensorflow/libtensorflow_cc.so.1 to /home/he-transformer/build/ext_ngraph_tf/src/ext_ngraph_tf/build_cmake/artifacts/tensorflow
Loading virtual environment from: /home/he-transformer/build/ext_ngraph_tf/src/ext_ngraph_tf/build_cmake/venv-tf-py3
LIB: /home/he-transformer/build/ext_ngraph_tf/src/ext_ngraph_tf/build_cmake/venv-tf-py3/lib/python3.6/site-packages/tensorflow
CXX_ABI: 0
nGraph Version:  v0.28.0-rc.1
Source location: /home/he-transformer/build/ext_ngraph_tf/src/ext_ngraph_tf/build_cmake/ngraph
Running COMMAND: cmake -DNGRAPH_INSTALL_PREFIX=/home/he-transformer/build/ext_ngraph_tf/src/ext_ngraph_tf/build_cmake/artifacts -DNGRAPH_USE_CXX_ABI=0 -DNGRAPH_DEX_ONLY=TRUE -DNGRAPH_DEBUG_ENABLE=NO -DNGRAPH_UNIT_TEST_ENABLE=NO -DNGRAPH_TARGET_ARCH=native -DNGRAPH_TUNE_ARCH=native -DNGRAPH_TBB_ENABLE=FALSE -DNGRAPH_DISTRIBUTED_ENABLE=OFF -DNGRAPH_TOOLS_ENABLE=YES -DNGRAPH_GPU_ENABLE=NO -DNGRAPH_PLAIDML_ENABLE=NO -DNGRAPH_INTELGPU_ENABLE=NO /home/he-transformer/build/ext_ngraph_tf/src/ext_ngraph_tf/build_cmake/ngraph
Running COMMAND: make -j72 VERBOSE=1
Running COMMAND: make install
TF_SRC_DIR:  /home/he-transformer/build/ext_ngraph_tf/src/ext_ngraph_tf/build_cmake/tensorflow
Loading virtual environment from: /home/he-transformer/build/ext_ngraph_tf/src/ext_ngraph_tf/build_cmake/venv-tf-py3
Source location: /home/he-transformer/build/ext_ngraph_tf/src/ext_ngraph_tf
OUTPUT WHL FILE: ngraph_tensorflow_bridge-0.22.0rc3-py2.py3-none-manylinux1_x86_64.whl
OUTPUT WHL DST: /home/he-transformer/build/ext_ngraph_tf/src/ext_ngraph_tf/build_cmake/artifacts/ngraph_tensorflow_bridge-0.22.0rc3-py2.py3-none-manylinux1_x86_64.whl
SUCCESSFULLY generated wheel: ngraph_tensorflow_bridge-0.22.0rc3-py2.py3-none-manylinux1_x86_64.whl
PWD: /home/he-transformer/build/ext_ngraph_tf/src/ext_ngraph_tf/build_cmake
Running COMMAND: cp -r /home/he-transformer/build/ext_ngraph_tf/src/ext_ngraph_tf/build_cmake/tensorflow/tensorflow/python /home/he-transformer/build/ext_ngraph_tf/src/ext_ngraph_tf/build_cmake/artifacts/tensorflow
Loading virtual environment from: /home/he-transformer/build/ext_ngraph_tf/src/ext_ngraph_tf/build_cmake/venv-tf-py3
Version information
TensorFlow version:  1.14.0
C Compiler version used in building TensorFlow:  4.8.5
nGraph bridge version: b'0.22.0-rc3'
nGraph version used for this build: b'0.28.0-rc.1+d2cd873'
TensorFlow version used for this build: v1.14.0-0-g87989f6959
CXX11_ABI flag used for this build: 0
nGraph bridge built with Grappler: True
nGraph bridge built with Variables and Optimizers Enablement: False
nGraph bridge built with Distributed Build: 0
Build successful
cd /home/he-transformer/build/ext_ngraph_tf/src/ext_ngraph_tf && /usr/local/lib/python3.6/dist-packages/cmake/data/bin/cmake -E touch /home/he-transformer/build/ext_ngraph_tf/src/ext_ngraph_tf-stamp/ext_ngraph_tf-build
[ 68%] Performing install step for 'ext_ngraph_tf'
cd /home/he-transformer/build/ext_ngraph_tf/src/ext_ngraph_tf && ln -fs /home/he-transformer/build/ext_ngraph_tf/src/ext_ngraph_tf/build_cmake/venv-tf-py3 /home/he-transformer/build/external
cd /home/he-transformer/build/ext_ngraph_tf/src/ext_ngraph_tf && /usr/local/lib/python3.6/dist-packages/cmake/data/bin/cmake -E touch /home/he-transformer/build/ext_ngraph_tf/src/ext_ngraph_tf-stamp/ext_ngraph_tf-install
[ 68%] Completed 'ext_ngraph_tf'
/usr/local/lib/python3.6/dist-packages/cmake/data/bin/cmake -E make_directory /home/he-transformer/build/CMakeFiles
/usr/local/lib/python3.6/dist-packages/cmake/data/bin/cmake -E touch /home/he-transformer/build/CMakeFiles/ext_ngraph_tf-complete
/usr/local/lib/python3.6/dist-packages/cmake/data/bin/cmake -E touch /home/he-transformer/build/ext_ngraph_tf/src/ext_ngraph_tf-stamp/ext_ngraph_tf-done
make[2]: Leaving directory '/home/he-transformer/build'
[ 68%] Built target ext_ngraph_tf
make[1]: Leaving directory '/home/he-transformer/build'
make: *** [all] Error 2
Makefile:148: recipe for target 'all' failed
The command '/bin/sh -c mkdir build &&     cd build &&     cmake .. -DCMAKE_BUILD_TYPE=Release -DCMAKE_CXX_COMPILER=g++-7 -DCMAKE_C_COMPILER=gcc-7 -Werror &&     make VERBOSE=1 -j 172 install' returned a non-zero code: 2

Although it states "Build successful" it somehow does not finish successful ("Error 2." and "Makefile:148: recipe for target 'all' failed"). Verbosity is already enabled but the error message doesn't help. Can I find anywhere more detailed information? I couldn't find anything in the build directory.

What is the last tested commit of this repository that was built and is known to be working?

[pjattke]

Meanwhile, the issue could be solved thanks to the help of @fboemer. The issue was that my Dockerfile installed gcc-4.8 (inspired by the official Dockerfile by ngraph-bridge). Although it was configured as alternative to gcc-7 with lower priority, ngraph-bridge used it and the build failed due to a CXX ABI incompatibility.

You can find the working Dockerfile including the patch for the bfloat16.cc issue in my repository pjattke/docker-he-transformer. I will also submit a PR for that.