search

bazelbuild / bazel

a fast, scalable, multi-language and extensible build system
https://bazel.build
Apache License 2.0
23.26k stars 4.08k forks source link

Using bazel to cross-compile tensorflow for other targets. #1353

Closed sachinpro closed 7 years ago

sachinpro commented 8 years ago

I compiled the tensorflow source using https://github.com/tensorflow/tensorflow/tree/master/tensorflow/examples/label_image example for C++ on Ubuntu 14.04. Now I want to learn how to cross-compile for some other target architecture. I started with ARM. I installed the arm-linux-gnueabi-gcc tools and edited the bazel/tools/cpp/CROSSTOOL.tpl file and its android toolchain for the armeabi-v7a cpu, so that, my new toolchain looks like this :

toolchain { abi_version: "armeabi-v7a" abi_libc_version: "armeabi-v7a" builtin_sysroot: "" compiler: "compiler" host_system_name: "armeabi-v7a" needsPic: true supports_gold_linker: false supports_incremental_linker: false supports_fission: false supports_interface_shared_objects: false supports_normalizing_ar: false supports_start_end_lib: false supports_thin_archives: false target_libc: "armeabi-v7a" target_cpu: "armeabi-v7a" target_system_name: "armeabi-v7a" toolchain_identifier: "stub_armeabi-v7a"

tool_path { name: "ar" path: "/usr/bin/arm-linux-gnueabi-ar" } tool_path { name: "compat-ld" path: "/bin/false" } tool_path { name: "cpp" path: "/usr/bin/arm-linux-gnueabi-cpp" } tool_path { name: "dwp" path: "/bin/false" } tool_path { name: "gcc" path: "/usr/bin/arm-linux-gnueabi-gcc" } tool_path { name: "gcov" path: "/bin/false" } tool_path { name: "ld" path: "/usr/bin/arm-linux-gnueabi-ld" }

tool_path { name: "nm" path: "/bin/false" } tool_path { name: "objcopy" path: "/bin/false" } tool_path { name: "objdump" path: "/bin/false" } tool_path { name: "strip" path: "/bin/false" } linking_mode_flags { mode: DYNAMIC } }

But, when compiling tensorflow label_image using the command $bazel build --cpu=armeabi-v7a -s tensorflow/examples/label_image/... , I get an error:

INFO: Found 2 targets... ERROR: /home/sachin/.cache/bazel/_bazel_sachin/dc84bb70b01d0f4a792c48d02549c4f8/external/highwayhash/BUILD:17:1: C++ compilation of rule '@highwayhash//:sip_hash' failed: arm-linux-gnueabi-gcc failed: error executing command /usr/bin/arm-linux-gnueabi-gcc '-frandom-seed=bazel-out/stub_armeabi-v7a-fastbuild/bin/external/highwayhash/_objs/sip_hash/external/highwayhash/highwayhash/sip_hash.pic.o' -fPIC -iquote ... (remaining 20 argument(s) skipped): com.google.devtools.build.lib.shell.BadExitStatusException: Process exited with status 1. In file included from external/highwayhash/highwayhash/sip_hash.h:23:0, from external/highwayhash/highwayhash/sip_hash.cc:15: external/highwayhash/highwayhash/statehelpers.h: In function 'void highwayhash::UpdateState(const char, highwayhash::uint64, State_)': external/highwayhash/highwayhash/state_helpers.h:36:76: error: there are no arguments to 'static_assert' that depend on a template parameter, so a declaration of 'static_assert' must be available [-fpermissive] external/highwayhash/highwayhash/state_helpers.h:36:76: note: (if you use '-fpermissive', G++ will accept your code, but allowing the use of an undeclared name is deprecated) In file included from external/highwayhash/highwayhash/sip_hash.cc:15:0: external/highwayhash/highwayhash/sip_hash.h: At global scope: external/highwayhash/highwayhash/sip_hash.h:31:9: error: expected nested-name-specifier before 'Key' external/highwayhash/highwayhash/sip_hash.h:31:9: error: using-declaration for non-member at class scope external/highwayhash/highwayhash/sip_hash.h:31:13: error: expected ';' before '=' token external/highwayhash/highwayhash/sip_hash.h:31:13: error: expected unqualified-id before '=' token

I am pretty clueless about these errors and I am new to bazel. Is there anything more I have to do?If yes, can someone tell me what steps I have to follow to have my build for different architectures? I am using bazel 0.2.3.

Thanks in advance.

meteorcloudy commented 8 years ago

Here is the wiki page about how to use a custom toolchain, and the example might be very close to what you need. :) https://github.com/bazelbuild/bazel/wiki/Building-with-a-custom-toolchain

sachinpro commented 8 years ago

Thanks. I will try this . :)

sachinpro commented 8 years ago

The link you gave me helped to a great extent. I have setup everything exactly according to the tutorial, I am just confused about the heirarchy of the tree . My current structure for the tools directory is in the attachment . structure.txt

In the same folder, there is my WORKSPACE file and another folder linaro_linux_gcc_4.9_repo/ which contains all the binaries.

There is a file ./compilers/linaro_linux_gcc_4.9.BUILD .

In my WORSPACE file, I added:

new_http_archive( name = "linaro_linux_gcc_4.9_repo", build_file = "compilers/linaro_linux_gcc_4.9.BUILD", sha256 = '25e97bcb0af4fd7cd626d5bb1b303c7d2cb13acf2474e335e3d431d1a53fbb52', url = 'http://localhost:8000/gcc-linaro-4.9-2015.05-x86_64_arm-linux-gnueabihf.tar.gz', )

But when compile using command "$bazel build --crosstool_top=//tools/arm_compiler:toolchain --cpu=armeabi-v7a tensorflow/examples/label_image/..." , I get the following error:

ERROR: /home/sachin/Downloads/tensorflow/WORKSPACE:420:1: new_http_archive rule //external:linaro_linux_gcc_4.9_repo's name field must be a legal workspace name. ERROR: Error evaluating WORKSPACE file. ERROR: package contains errors: tensorflow/examples/label_image. ERROR: error loading package 'external': Package 'external' contains errors. INFO: Elapsed time: 0.135s

There must be something wrong with my directory structure , but I don't know what.

sachinpro commented 8 years ago

Where should I put my linaro_linux_gcc_4.9_repo folder? I think the problem is with the location of this folder only.

sachinpro commented 8 years ago

I removed the error I mentioned above and after making a few changes in my WORKSPACE and CROSSTOOL file, I am getting the following error:

ERROR: The toolchain rule '//tools/arm_compiler:cc-compiler-local' does not exist.

Please help me with this.

meteorcloudy commented 8 years ago

The error is actually as the error messages said linaro_linux_gcc_4.9_repo is not a valid workspace name, linaro_linux_gcc_repo works for me. We should update the wiki page, sorry for that.

But using new_http_archive still seems to be a little bit trouble, because there is no usable url for downloading. However, you can decompress gcc-linaro-4.9-2015.05-x86_64_arm-linux-gnueabihf.tar.gz into a folder, and then use new_local_repository. Like this:

pcloudy@pcloudy0 ~/workspace/bazel_local_repository
$ tree -L 2 .
.
├── BUILD
├── compilers
│   ├── gcc-linaro-4.9-2015.05-x86_64_arm-linux-gnueabihf
│   └── linaro_linux_gcc_4.9.BUILD
└── WORKSPACE

2 directories, 3 files
pcloudy@pcloudy0 ~/workspace/bazel_local_repository
$ cat WORKSPACE 
new_local_repository(
    name = 'linaro_linux_gcc_repo',
    build_file = 'compilers/linaro_linux_gcc_4.9.BUILD',
    path = 'compilers/gcc-linaro-4.9-2015.05-x86_64_arm-linux-gnueabihf',
)
pcloudy@pcloudy0 ~/workspace/bazel_local_repository
$ cat BUILD 

filegroup(
    name = "test",
    srcs = ["@linaro_linux_gcc_repo//:gcc"],
)

cc_binary(
    name = "hello",
    srcs = ["hello.cc"],
    data = [":test"],
)
pcloudy@pcloudy0 ~/workspace/bazel_local_repository
$ bazel build :hello
INFO: Found 1 target...
Target //:hello up-to-date:
  bazel-bin/hello
INFO: Elapsed time: 0.782s, Critical Path: 0.53s

Hope this can help you. :)

meteorcloudy commented 8 years ago

OK.. Can you show me your CROSSTOOL file?

sachinpro commented 8 years ago

I also removed the error using linaro_linux_gcc_repo instead. CROSSTOOL file is attached. CROSSTOOL.txt

sachinpro commented 8 years ago

I added the k8 toolchain because it was giving an error "cpu k8 is not supported" . And I am currently using my git repository to download the binaries by adding this in my WORKSPACE file:

new_git_repository( name = "linaro_linux_gcc_repo", build_file = "compilers/linaro_linux_gcc_4.9.BUILD", remote = "https://github.com/sachinpro/sachinpro.github.io.git",

tag = "",

commit="4e1c212901bf72e8b3def5bb3bcb84fdffb75592" )

But I will try what you are telling too.

meteorcloudy commented 8 years ago

And what is the command line giving this error?

sachinpro commented 8 years ago

sachin@sachin-pc:~/Downloads/tensorflow$ bazel build --crosstool_top=//tools/arm_compiler:toolchain --verbose_failures --sandbox_debug --cpu=armeabi-v7a tensorflow/examples/label_image/... ERROR: No toolchain found for cpu 'k8'. Valid cpus are: [ armeabi-v7a, ]. INFO: Elapsed time: 1.240s

This is the error if I get when follow the original tutorial without adding the k8 toolchain. You are asking about this error only , right?

meteorcloudy commented 8 years ago

So after you add the k8 toolchain, the same command gave this error ? ERROR: The toolchain rule '//tools/arm_compiler:cc-compiler-local' does not exist.

sachinpro commented 8 years ago

Yes, exactly.

meteorcloudy commented 8 years ago

Oh, yeah, I was just trying to show how new_local_repository works.

And how about the BUILD file, I think you should also modify it. Maybe it's missing this:

cc_toolchain(
    name = "cc-compiler-local",
    all_files = ":empty",
    compiler_files = ":empty",
    cpu = "local",
    dwp_files = ":empty",
    dynamic_runtime_libs = [":empty"],
    linker_files = ":empty",
    objcopy_files = ":empty",
    static_runtime_libs = [":empty"],
    strip_files = ":empty",
    supports_param_files = 1,
)
sachinpro commented 8 years ago

Here is my current WORKSPACE file . WORKSPACE.txt

meteorcloudy commented 8 years ago

I mean the BUILD file next to the CROSSTOOL. Like this one: https://github.com/bazelbuild/bazel/wiki/Building-with-a-custom-toolchain#writing-the-build-file

Did you accidentally close this issue?

sachinpro commented 8 years ago

Which BUILD file are you talking about? I have BUILD files present in ./tools/arm_compiler , ./tools/arm_compiler/linaro_linux_gcc and another for my label_image example.

meteorcloudy commented 8 years ago

Should be tools/arm_compiler/BUILD

sachinpro commented 8 years ago

Sorry for closing, it was by mistake. I just added the cc_toolchain() you told me to and now there is a new error:

sachin@sachin-pc:~/Downloads/tensorflow$ bazel build --crosstool_top=//tools/arm_compiler:toolchain --verbose_failures --sandbox_debug --cpu=armeabi-v7a tensorflow/examples/label_image/... ERROR: /home/sachin/.cache/bazel/_bazel_sachin/dc84bb70b01d0f4a792c48d02549c4f8/external/bazel_tools/tools/cpp/BUILD:7:1: Target '//tools/arm_compiler:cc-compiler-armeabi-v7a' is not visible from target '@bazel_tools//tools/cpp:stl'. Check the visibility declaration of the former target if you think the dependency is legitimate. ERROR: Analysis of target '//tensorflow/examples/label_image:label_image' failed; build aborted.

meteorcloudy commented 8 years ago

Add package(default_visibility = ["//visibility:public"]) to the same BUILD file

sachinpro commented 8 years ago

Now i get the following error:

ERROR: /home/sachin/Downloads/tensorflow/tensorflow/cc/BUILD:28:1: //tensorflow/cc:cc_ops: missing input file '//tools/arm_compiler/linaro_linux_gcc:clang_bin/as'. ERROR: /home/sachin/Downloads/tensorflow/tensorflow/cc/BUILD:28:1 1 input file(s) do not exist.

meteorcloudy commented 8 years ago

Follow the error message, check if tools/arm_compiler/linaro_linux_gcc/clang_bin/as exists?

changyun79 commented 8 years ago

I am new to bazel too but trying to do the same thing so just want to give you some head up that tensor flow is using protocol buffer 3.0.0 beta2 which you may need to cross compile first. It can be done by pretty much following this with some minor updates: https://github.com/eurotech/edc-examples/wiki/Cross-compiling-protobuf-for-ARM-architecture

I am going to be on tensor flow this weekend.

sachinpro commented 8 years ago

There is just a 'clang' file, in clang_bin, but no tools/arm_compiler/linaro_linux_gcc/clang_bin/as or ld exists. Can you please tell me how to create those symlinks as mentioned in https://github.com/bazelbuild/bazel/wiki/Building-with-a-custom-toolchain#writing-the-wrapper-scripts ? Where should they point to?

meteorcloudy commented 8 years ago

You should create a wrapper script at tools/arm_compiler/linaro_linux_gcc/arm-linux-gnueabihf-as for as. Should be similar to this:

#!/bin/bash --norc

exec -a arm-linux-gnueabihf-nm \
    external/linaro_linux_gcc_4.9_repo/bin/arm-linux-gnueabihf-nm \
    "$@"

And create a symlink tools/arm_compiler/linaro_linux_gcc/clang_bin/as pointing to the wrapper script.

sachinpro commented 8 years ago

Done that. Now error again:

ERROR: missing input file '//tools/arm_compiler:empty'. ERROR: /home/sachin/Downloads/tensorflow/tensorflow/cc/BUILD:28:1: //tensorflow/cc:ops/training_ops_gen_cc: missing input file '//tools/arm_compiler:empty'. ERROR: /home/sachin/Downloads/tensorflow/tensorflow/cc/BUILD:28:1 1 input file(s) do not exist. INFO: Elapsed time: 70.388s, Critical Path: 5.72s

meteorcloudy commented 8 years ago

I think you can just create the missing file, it might doesn't do anything, but it's required by Bazel.

sachinpro commented 8 years ago

Thanks , next ERROR : config.status: executing depfiles commands config.status: executing libtool commands /home/sachin/.cache/bazel/_bazel_sachin/dc84bb70b01d0f4a792c48d02549c4f8/execroot/tensorflow/external/jpeg_archive/jpeg-9a /home/sachin/.cache/bazel/_bazel_sachin/dc84bb70b01d0f4a792c48d02549c4f8/execroot/tensorflow /home/sachin/.cache/bazel/_bazel_sachin/dc84bb70b01d0f4a792c48d02549c4f8/execroot/tensorflow ERROR: /home/sachin/.cache/bazel/_bazel_sachin/dc84bb70b01d0f4a792c48d02549c4f8/external/protobuf/BUILD:111:1: C++ compilation of rule '@protobuf//:protobuf' failed: clang failed: error executing command (cd /home/sachin/.cache/bazel/_bazel_sachin/dc84bb70b01d0f4a792c48d02549c4f8/execroot/tensorflow && \ exec env - \ PATH=/home/sachin/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/usr/local/java/jdk1.8.0_74/bin:/usr/local/java/jdk1.8.0_74/jre/bin:/home/sachin/bazel/bazel-bin/src/:/home/sachin/bazel/output/ \ tools/arm_compiler/linaro_linux_gcc/clang_bin/clang -target armv7a-arm-linux-gnueabif '--sysroot=external/linaro_linux_gcc_repo/arm-linux-gnueabihf/libc' '-mfloat-abi=hard' -nostdinc -isystem /usr/lib/clang/3.6/include -isystem external/linaro_linux_gcc_repo/lib/gcc/arm-linux-gnueabihf/4.9.3/include -isystem external/linaro_linux_gcc_repo/arm-linux-gnueabihf/libc/usr/include -isystem external/linaro_linux_gcc_repo/lib/gcc/arm-linux-gnueabihf/4.9.3/include-fixed -isystem external/linaro_linux_gcc_repo/arm-linux-gnueabihf/libc/usr/include -U_FORTIFY_SOURCE -fstack-protector -fPIE '-fdiagnostics-color=always' -Wall -Wunused-but-set-parameter -Wno-free-nonheap-object -fno-omit-frame-pointer -isystem external/linaro_linux_gcc_repo/arm-linux-gnueabihf/include/c++/4.9.3/arm-linux-gnueabihf -isystem external/linaro_linux_gcc_repo/arm-linux-gnueabihf/include/c++/4.9.3 -isystem external/linaro_linux_gcc_repo/include/c++/4.9.3/arm-linux-gnueabihf -isystem external/linaro_linux_gcc_repo/include/c++/4.9.3 '-frandom-seed=bazel-out/clang_linux_armhf-fastbuild/bin/external/protobuf/_objs/protobuf/external/protobuf/src/google/protobuf/any.pic.o' -fPIC -iquote external/protobuf -iquote bazel-out/clang_linux_armhf-fastbuild/genfiles/external/protobuf -iquote external/bazel_tools -iquote bazel-out/clang_linux_armhf-fastbuild/genfiles/external/bazel_tools -isystem external/protobuf/src -isystem bazel-out/clang_linux_armhf-fastbuild/genfiles/external/protobuf/src -isystem external/bazel_tools/tools/cpp/gcc3 -DHAVE_PTHREAD -Wall -Wwrite-strings -Woverloaded-virtual -Wno-sign-compare '-Wno-error=unused-function' -no-canonical-prefixes -Wno-builtin-macro-redefined '-DDATE="redacted"' '-DTIMESTAMP="redacted"' '-DTIME="redacted"' -MD -MF bazel-out/clang_linux_armhf-fastbuild/bin/external/protobuf/_objs/protobuf/external/protobuf/src/google/protobuf/any.pic.d -c external/protobuf/src/google/protobuf/any.cc -o bazel-out/clang_linux_armhf-fastbuild/bin/external/protobuf/_objs/protobuf/external/protobuf/src/google/protobuf/any.pic.o): com.google.devtools.build.lib.shell.BadExitStatusException: Process exited with status 1. process-wrapper: execvp("tools/arm_compiler/linaro_linux_gcc/clang_bin/clang", ...): Permission denied INFO: Elapsed time: 135.534s, Critical Path: 68.99s

meteorcloudy commented 8 years ago

Did you give execute permission to clang? Try chmod +x tools/arm_compiler/linaro_linux_gcc/clang_bin/clang ?

sachinpro commented 8 years ago

Oh yeah, sorry about that. I should have done that. But error again: ERROR: /home/sachin/.cache/bazel/_bazel_sachin/dc84bb70b01d0f4a792c48d02549c4f8/external/png_archive/BUILD:33:1: C++ compilation of rule '@png_archive//:png' failed: clang failed: error executing command (cd /home/sachin/.cache/bazel/_bazel_sachin/dc84bb70b01d0f4a792c48d02549c4f8/execroot/tensorflow && \ exec env - \ PATH=/home/sachin/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/usr/local/java/jdk1.8.0_74/bin:/usr/local/java/jdk1.8.0_74/jre/bin:/home/sachin/bazel/bazel-bin/src/:/home/sachin/bazel/output/ \ tools/arm_compiler/linaro_linux_gcc/clang_bin/clang -target armv7a-arm-linux-gnueabif '--sysroot=external/linaro_linux_gcc_repo/arm-linux-gnueabihf/libc' '-mfloat-abi=hard' -nostdinc -isystem /usr/lib/clang/3.6/include -isystem external/linaro_linux_gcc_repo/lib/gcc/arm-linux-gnueabihf/4.9.3/include -isystem external/linaro_linux_gcc_repo/arm-linux-gnueabihf/libc/usr/include -isystem external/linaro_linux_gcc_repo/lib/gcc/arm-linux-gnueabihf/4.9.3/include-fixed -isystem external/linaro_linux_gcc_repo/arm-linux-gnueabihf/libc/usr/include -U_FORTIFY_SOURCE -fstack-protector -fPIE '-fdiagnostics-color=always' -Wall -Wunused-but-set-parameter -Wno-free-nonheap-object -fno-omit-frame-pointer -fPIC -iquote external/png_archive -iquote bazel-out/clang_linux_armhf-fastbuild/genfiles/external/png_archive -iquote external/bazel_tools -iquote bazel-out/clang_linux_armhf-fastbuild/genfiles/external/bazel_tools -isystem external/png_archive/libpng-1.2.53 -isystem bazel-out/clang_linux_armhf-fastbuild/genfiles/external/png_archive/libpng-1.2.53 -isystem external/bazel_tools/tools/cpp/gcc3 -no-canonical-prefixes -Wno-builtin-macro-redefined '-DDATE="redacted"' '-DTIMESTAMP="redacted"' '-DTIME="redacted"' -MD -MF bazel-out/clang_linux_armhf-fastbuild/bin/external/png_archive/_objs/png/external/png_archive/libpng-1.2.53/pngmem.pic.d -c external/png_archive/libpng-1.2.53/pngmem.c -o bazel-out/clang_linux_armhf-fastbuild/bin/external/png_archive/_objs/png/external/png_archive/libpng-1.2.53/pngmem.pic.o): com.google.devtools.build.lib.shell.BadExitStatusException: Process exited with status 1. warning: unknown warning option '-Wunused-but-set-parameter'; did you mean '-Wunused-parameter'? [-Wunknown-warning-option] warning: unknown warning option '-Wno-free-nonheap-object'; did you mean '-Wno-sequence-point'? [-Wunknown-warning-option] In file included from external/png_archive/libpng-1.2.53/pngmem.c:22: external/png_archive/libpng-1.2.53/png.h:548:10: fatal error: 'zlib.h' file not found

include "zlib.h"

     ^

2 warnings and 1 error generated.

By the way, I have zlib1g-dev installed.

sachinpro commented 8 years ago

zlib.h is present in /home/sachin/bazel/third_party/zlib .And /home/sachin/bazel/ is the location where my bazel is installed.

sachinpro commented 8 years ago

I tried to rebuild and this time , a different error came:

external/highwayhash/highwayhash/state_helpers.h:36:3: error: use of undeclared identifier 'static_assert'; did you mean 'static_cast'? static_assert((kPacketSize & (kPacketSize - 1)) == 0, "Size must be 2^i."); ^ In file included from external/highwayhash/highwayhash/sip_hash.cc:15: external/highwayhash/highwayhash/sip_hash.h:31:15: warning: alias declarations are a C++11 extension [-Wc++11-extensions] using Key = uint64[2]; ^ 3 warnings and 1 error generated.

sachinpro commented 8 years ago

Hey @meteorcloudy , can you please help me with the errors mentioned in previous comments or could you please add someone else who might help me with this??

meteorcloudy commented 8 years ago

For the zlib one, try sudo apt-get install zlib1g-dev And the second one, maybe you should add following to CROSSTOOL

cxx_flag: "-std=c++11"
sachinpro commented 8 years ago

I already tried both of the above. But the "'zlib.h' file not found" error still persists..

sachinpro commented 8 years ago

UPDATE: Using cxx_flag: "-std=c++11" removes the corresponding error , but I still cannot find a way to rectify the "'zlib.h' file not found" error.

meteorcloudy commented 8 years ago

I found this bug from TensorFlow's repository. https://github.com/tensorflow/tensorflow/issues/2536 Can you try the solution mentioned there?

sachinpro commented 8 years ago

Thanks,I tried the solution but still cannot resolve the 'zlib.h' problem. But, I have successfully built for ARM by removing all dependencies related to png image format. I had to do some more changes in source code because apparently some parts of code have problems with compilation using clang but not gcc .

uvaidya commented 8 years ago

Hi Sachinpro, can you please help in letting me know , how to remove all dependencies related to png image format. I did all that is discussed above and finally landed on 'zlib.h' problem. Tried to remove dependencies by removing png.BUILD call from workspace, but still some errors kept on coming up. Please let me know what all need to be removed / or to be commented so that i can also build arm binary. Thanks in advance .

sachinpro commented 8 years ago

Deleting png.BUILD will not do. I don't know the most efficient way to do this but here's what I did: 1)Check all the BUILD files for any dependency and remove it by commenting the corresponding statements. 2) Edit summary_image_op.cc and your label_image/main.cc to do the same. Just comment the headers related to png and the parts of code which are trying to read through png files.

Note: You may still have build problems after this . You can either find the root of the errors manually or ask here. If you are doing exactly what I did, I can help . I will share my code and a doc for building it next week.

uvaidya commented 8 years ago

thanks Sachinpro , I will carry out procedure you have mentioned and will let you know status after that .

Yes, i did exactly what you did . Your code will definitely help me . What i want is to run arm binary on android device with arm processor, so aim was to build this label image binary for arm and then run it on device .

uvaidya commented 8 years ago

Hi Sachinpro , I tried removing png dependencies, but error kept on popping up . Mentioning here details : 1) here is the structure of tools directory :- structure_uvaidya.txt 2) made 2 directories in root of tensor flow directory :- a) compilers :- (i have taken this from the wiki link mentioned by meteorcloudy) , attaching its structure if it can help . compiler_dir_struc.txt b) linaro_linux_gcc_repo :- this folder contains all binaries, attaching its structure, repo_structure.txt 3)From your earlier comments and discussion that you had with meteorclody, following are the build, crosstool and workspace file that i used :- a)build file is at tools/ arm_compiler/ . build.txt b)crosstool file is at : tools/arm_compiler crosstool.txt c)workspace file at tensorflow root is : workspace.txt

Please let me know if you find any thing undesired in above files,

Now, with above changes i tried to build arm binary for label image project and got same zlib error , then i tried to remove png dependencies as you told , but always landed on error , mentioning errors observed and workaround which i used to resolve it (though it didnt worked out :-) ) 1)commented out png related arguments from build files .Following is the first error which i observed :-

_"ERROR: /home/dl-audio/tensorflow/tensorflow/core/BUILD:801:1: C++ compilation of rule '//tensorflow/core:lib_internal' failed: clang failed: error executing command (cd /home/dl-audio/.cache/bazel/_bazel_dl-audio/41010b521a4c389f530e4c8517656b86/execroot/tensorflow && \ exec env - \ PATH=/usr/local/cuda/bin:/home/dl-audio/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games \ tools/arm_compiler/linaro_linux_gcc/clang_bin/clang -target armv7a-arm-linux-gnueabif '--sysroot=external/linaro_linux_gcc_repo/arm-linux-gnueabihf/libc' '-mfloat-abi=hard' -nostdinc -isystem /usr/lib/clang/3.6/include -isystem external/linaro_linux_gcc_repo/lib/gcc/arm-linux-gnueabihf/4.9.3/include -isystem external/linaro_linux_gcc_repo/arm-linux-gnueabihf/libc/usr/include -isystem external/linaro_linux_gcc_repo/lib/gcc/arm-linux-gnueabihf/4.9.3/include-fixed -isystem external/linaro_linux_gcc_repo/arm-linux-gnueabihf/libc/usr/include -U_FORTIFY_SOURCE -fstack-protector -fPIE '-fdiagnostics-color=always' -Wall -Wunused-but-set-parameter -Wno-free-nonheap-object -fno-omit-frame-pointer -isystem external/linaro_linux_gcc_repo/arm-linux-gnueabihf/include/c++/4.9.3/arm-linux-gnueabihf -isystem external/linaro_linux_gcc_repo/arm-linux-gnueabihf/include/c++/4.9.3 -isystem external/linaro_linux_gcc_repo/include/c++/4.9.3/arm-linux-gnueabihf -isystem external/linaro_linux_gcc_repo/include/c++/4.9.3 '-std=c++11' '-frandom-seed=bazel-out/clang_linux_armhf-fastbuild/bin/tensorflow/core/_objs/lib_internal/tensorflow/core/lib/jpeg/jpeg_handle.pic.o' -fPIC -iquote . -iquote bazel-out/clang_linux_armhf-fastbuild/genfiles -iquote external/protobuf -iquote bazel-out/clang_linux_armhf-fastbuild/genfiles/external/protobuf -iquote external/bazel_tools -iquote bazel-out/clang_linux_armhf-fastbuild/genfiles/external/bazel_tools -iquote external/farmhash_archive -iquote bazel-out/clang_linux_armhf-fastbuild/genfiles/external/farmhash_archive -iquote external/jpeg_archive -iquote bazel-out/clang_linux_armhf-fastbuild/genfiles/external/jpeg_archive -iquote external/highwayhash -iquote bazel-out/clang_linux_armhf-fastbuild/genfiles/external/highwayhash -iquote external/re2 -iquote bazel-out/clang_linux_armhf-fastbuild/genfiles/external/re2 -iquote external/eigen_archive -iquote bazel-out/clang_linux_armhf-fastbuild/genfiles/external/eigen_archive -iquote external/zlib_archive -iquote bazel-out/clang_linux_armhf-fastbuild/genfiles/external/zlib_archive -isystem external/protobuf/src -isystem bazel-out/clang_linux_armhf-fastbuild/genfiles/external/protobuf/src -isystem external/bazel_tools/tools/cpp/gcc3 -isystem external/farmhash_archive/farmhash-34c13ddfab0e35422f4c3979f360635a8c050260 -isystem bazel-out/clang_linux_armhf-fastbuild/genfiles/external/farmhash_archive/farmhash-34c13ddfab0e35422f4c3979f360635a8c050260 -isystem external/jpeg_archive/jpeg-9a -isystem bazel-out/clang_linux_armhf-fastbuild/genfiles/external/jpeg_archive/jpeg-9a -isystem external/highwayhash -isystem bazel-out/clang_linux_armhf-fastbuild/genfiles/external/highwayhash -isystem external/re2 -isystem bazel-out/clang_linux_armhf-fastbuild/genfiles/external/re2 -isystem third_party/eigen3 -isystem bazel-out/clang_linux_armhf-fastbuild/genfiles/third_party/eigen3 -isystem external/eigen_archive/eigen-eigen-5f86b31739cd -isystem bazel-out/clang_linux_armhf-fastbuild/genfiles/external/eigen_archive/eigen-eigen-5f86b31739cd -isystem external/zlib_archive/zlib-1.2.8 -isystem bazel-out/clang_linux_armhf-fastbuild/genfiles/external/zlib_archive/zlib-1.2.8 -fno-exceptions -DEIGEN_AVOID_STL_ARRAY -pthread -no-canonical-prefixes -Wno-builtin-macro-redefined '-DDATE="redacted"' '-DTIMESTAMP="redacted"' '-DTIME="redacted"' -MD -MF bazel-out/clang_linux_armhf-fastbuild/bin/tensorflow/core/_objs/lib_internal/tensorflow/core/lib/jpeg/jpeg_handle.pic.d -c tensorflow/core/lib/jpeg/jpeg_handle.cc -o bazel-out/clang_linux_armhf-fastbuild/bin/tensorflow/core/_objs/lib_internal/tensorflow/core/lib/jpeg/jpeg_handle.pic.o): com.google.devtools.build.lib.shell.BadExitStatusException: Process exited with status 1. warning: unknown warning option '-Wunused-but-set-parameter'; did you mean '-Wunused-parameter'? [-Wunknown-warning-option] warning: unknown warning option '-Wno-free-nonheap-object'; did you mean '-Wno-sequence-point'? [-Wunknown-warning-option] In file included from tensorflow/core/lib/jpeg/jpeg_handle.cc:22: In file included from ./tensorflow/core/lib/jpeg/jpeghandle.h:22: ./tensorflow/core/platform/jpeg.h:31:2: error: Define the appropriate PLATFORM macro for this platform

error Define the appropriate PLATFORM_ macro for this platform

^ In file included from tensorflow/core/lib/jpeg/jpeg_handle.cc:22: ./tensorflow/core/lib/jpeg/jpeg_handle.h:29:17: error: unknown type name 'j_common_ptr' void CatchError(j_common_ptr cinfo); ^ ./tensorflow/core/lib/jpeg/jpeg_handle.h:32:31: error: field has incomplete type 'struct jpeg_destination_mgr' struct jpeg_destination_mgr pub; ^ ./tensorflow/core/lib/jpeg/jpeg_handle.h:32:10: note: forward declaration of 'tensorflow::jpeg::jpeg_destination_mgr' struct jpeg_destination_mgr pub; ^ ./tensorflow/core/lib/jpeg/jpeg_handle.h:33:3: error: unknown type name 'JOCTET' JOCTET _buffer; ^ ./tensorflow/core/lib/jpeg/jpeg_handle.h:40:26: error: field has incomplete type 'struct jpeg_source_mgr' struct jpeg_source_mgr pub; ^ ./tensorflow/core/lib/jpeg/jpeg_handle.h:40:10: note: forward declaration of 'tensorflow::jpeg::jpeg_source_mgr' struct jpeg_source_mgr pub; ^ ./tensorflow/core/lib/jpeg/jpeg_handle.h:46:13: error: unknown type name 'j_decompress_ptr' void SetSrc(j_decompress_ptr cinfo, const void data, ^ ./tensorflow/core/lib/jpeg/jpeg_handle.h:51:14: error: unknown type name 'j_compress_ptr' void SetDest(j_compress_ptr cinfo, void buffer, int bufsize); ^ ./tensorflow/core/lib/jpeg/jpeg_handle.h:54:14: error: unknown type name 'j_compress_ptr' void SetDest(j_compress_ptr cinfo, void buffer, int bufsize, ^ tensorflow/core/lib/jpeg/jpeg_handle.cc:32:17: error: unknown type name 'j_common_ptr' void CatchError(j_common _ptr cinfo) { ^ tensorflow/core/lib/jpeg/jpeg_handle.cc:45:25: error: unknown type name 'j_compress_ptr' void MemInitDestination(j_compress_ptr cinfo) { ^ tensorflow/core/lib/jpeg/jpeg_handle.cc:57:1: error: unknown type name 'boolean' boolean MemEmptyOutputBuffer(j_compress_ptr cinfo) { ^ tensorflow/core/lib/jpeg/jpeg_handle.cc:57:30: error: unknown type name 'j_compress_ptr' boolean MemEmptyOutputBuffer(j_compress_ptr cinfo) { ^ tensorflow/core/lib/jpeg/jpeg_handle.cc:65:10: error: use of undeclared identifier 'TRUE' return TRUE; ^ tensorflow/core/lib/jpeg/jpeg_handle.cc:69:25: error: unknown type name 'j_compress_ptr' void MemTermDestination(j_compress_ptr cinfo) { ^ tensorflow/core/lib/jpeg/jpeg_handle.cc:81:14: error: unknown type name 'j_compress_ptr' void SetDest(j_compress_ptr cinfo, void buffer, int bufsize) { ^ tensorflow/core/lib/jpeg/jpeg_handle.cc:86:14: error: unknown type name 'j_compress_ptr' void SetDest(j_compress_ptr cinfo, void buffer, int bufsize, ^ tensorflow/core/lib/jpeg/jpeg_handle.cc:91:53: error: unknown type name 'j_common_ptr' (_cinfo->mem->alloc_small)(reinterpret_cast(cinfo), ^ tensorflow/core/lib/jpeg/jpeg_handle.cc:92:36: error: use of undeclared identifier 'JPOOL_PERMANENT' JPOOL_PERMANENT, sizeof(MemDestMgr))); ^ tensorflow/core/lib/jpeg/jpeg_handle.cc:97:30: error: unknown type name 'JOCTET' dest->buffer = static_cast<JOCTET >(buffer); ^ fatal error: too many errors emitted, stopping now [-ferror-limit=] 2 warnings and 20 errors generated. "_

I think above error is because platform is not defined, that resulted in" #error Define the appropriate PLATFORM macro for this platform_ " in ./tensorflow/core/platform/jpeg.h

to remove this , in ./tensorflow/core/platform/jpeg.h i added same include files as they were for platforms above .

But then came other sets of erros . I will mention those other errors also here, meanwhile want to ask you that why platform definition is not been picked up , do we need to do it somewhere ? Have you faced such an error ?

Thanks

uvaidya commented 8 years ago

got the same error , now for png,h ("./tensorflow/core/platform/png.h:26:2: error: Define the appropriate PLATFORM macro for this platform

error Define the appropriate PLATFORM_ macro for this platform

^")_ . This error also pointed to platform not getting defined .

on another run of build command didn't saw this error (didn't made any change in code base though)

Next error which came is : _"ERROR: /home/dl-audio/tensorflow/tensorflow/core/BUILD:801:1: C++ compilation of rule '//tensorflow/core:lib_internal' failed: gcc failed: error executing command (cd /home/dl-audio/.cache/bazel/_bazel_dl-audio/41010b521a4c389f530e4c8517656b86/execroot/tensorflow && \ exec env - \ PATH=/usr/local/cuda/bin:/home/dl-audio/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games \ /usr/bin/gcc -U_FORTIFY_SOURCE '-D_FORTIFY_SOURCE=1' -fstack-protector -Wall -Wunused-but-set-parameter -Wno-free-nonheap-object -fno-omit-frame-pointer -g0 -O2 -DNDEBUG -ffunction-sections -fdata-sections -g0 '-std=c++0x' '-frandom-seed=bazel-out/host/bin/tensorflow/core/_objs/lib_internal/tensorflow/core/lib/png/png_io.o' -iquote . -iquote bazel-out/host/genfiles -iquote external/protobuf -iquote bazel-out/host/genfiles/external/protobuf -iquote external/bazel_tools -iquote bazel-out/host/genfiles/external/bazel_tools -iquote external/farmhash_archive -iquote bazel-out/host/genfiles/external/farmhash_archive -iquote external/jpeg_archive -iquote bazel-out/host/genfiles/external/jpeg_archive -iquote external/highwayhash -iquote bazel-out/host/genfiles/external/highwayhash -iquote external/re2 -iquote bazel-out/host/genfiles/external/re2 -iquote external/eigen_archive -iquote bazel-out/host/genfiles/external/eigen_archive -iquote external/zlib_archive -iquote bazel-out/host/genfiles/external/zlib_archive -isystem external/protobuf/src -isystem bazel-out/host/genfiles/external/protobuf/src -isystem external/bazel_tools/tools/cpp/gcc3 -isystem external/farmhash_archive/farmhash-34c13ddfab0e35422f4c3979f360635a8c050260 -isystem bazel-out/host/genfiles/external/farmhash_archive/farmhash-34c13ddfab0e35422f4c3979f360635a8c050260 -isystem external/jpeg_archive/jpeg-9a -isystem bazel-out/host/genfiles/external/jpeg_archive/jpeg-9a -isystem external/highwayhash -isystem bazel-out/host/genfiles/external/highwayhash -isystem external/re2 -isystem bazel-out/host/genfiles/external/re2 -isystem third_party/eigen3 -isystem bazel-out/host/genfiles/third_party/eigen3 -isystem external/eigen_archive/eigen-eigen-5f86b31739cd -isystem bazel-out/host/genfiles/external/eigen_archive/eigen-eigen-5f86b31739cd -isystem external/zlib_archive/zlib-1.2.8 -isystem bazel-out/host/genfiles/external/zlib_archive/zlib-1.2.8 -fno-exceptions -DEIGEN_AVOID_STL_ARRAY -pthread -no-canonical-prefixes -fno-canonical-system-headers -Wno-builtin-macro-redefined '-DDATE="redacted"' '-DTIMESTAMP="redacted"' '-DTIME="redacted"' -MD -MF bazel-out/host/bin/tensorflow/core/_objs/lib_internal/tensorflow/core/lib/png/png_io.d -c tensorflow/core/lib/png/png_io.cc -o bazel-out/host/bin/tensorflow/core/_objs/lib_internal/tensorflow/core/lib/png/png_io.o): com.google.devtools.build.lib.shell.BadExitStatusException: Process exited with status 1. In file included from ./tensorflow/core/lib/png/png_io.h:39:0, from tensorflow/core/lib/png/png_io.cc:27: ./tensorflow/core/platform/png.h:24:31: fatal error: libpng-1.2.53/png.h: No such file or directory

include "libpng-1.2.53/png.h""_

Commented above "libpng-1.2.53/png.h"" file then also error kept on popping up. Have you also seen this error ? Please suggest what can be done to overcome these ?

Thanks in advance :-)

sachinpro commented 8 years ago

Use this https://github.com/sachinpro/sachinpro.github.io.git

uvaidya commented 8 years ago

Thanks Sachinpro will use link you provided , then will update the status .

uvaidya commented 8 years ago

Hi Sachinpro, I tried with your code , got the following error : _"dl-audio@powerbox:~/TF_01/sachinpro.github.io$ bazel build --crosstool_top=//tools/arm_compiler:toolchain --verbose_failures --sandbox_debug --cpu=armeabi-v7a tensorflow/examples/label_image/... WARNING: /home/dl-audio/.cache/bazel/_bazel_dl-audio/e1cbd95e2c3289a79fe4b6e4c2b8efab/external/protobuf/WORKSPACE:1: Workspace name in /home/dl-audio/.cache/bazel/_bazel_dl-audio/e1cbd95e2c3289a79fe4b6e4c2b8efab/external/protobuf/WORKSPACE (@main) does not match the name given in the repository's definition (@protobuf); this will cause a build error in future versions. WARNING: /home/dl-audio/.cache/bazel/_bazel_dl-audio/e1cbd95e2c3289a79fe4b6e4c2b8efab/external/highwayhash/WORKSPACE:1: Workspace name in /home/dl-audio/.cache/bazel/_bazel_dl-audio/e1cbd95e2c3289a79fe4b6e4c2b8efab/external/highwayhash/WORKSPACE (@main) does not match the name given in the repository's definition (@highwayhash); this will cause a build error in future versions. WARNING: /home/dl-audio/.cache/bazel/_bazel_dl-audio/e1cbd95e2c3289a79fe4b6e4c2b8efab/external/re2/WORKSPACE:1: Workspace name in /home/dl-audio/.cache/bazel/_bazel_dl-audio/e1cbd95e2c3289a79fe4b6e4c2b8efab/external/re2/WORKSPACE (@main) does not match the name given in the repository's definition (@re2); this will cause a build error in future versions. INFO: Found 2 targets... ERROR: missing input file '//tools/arm_compiler/linaro_linux_gcc:clang_bin/as'. ERROR: /home/dl-audio/TF_01/sachinpro.github.io/tensorflow/cc/BUILD:28:1: //tensorflow/cc:cc_ops: missing input file '//tools/arm_compiler/linaro_linux_gcc:clang_bin/as'. ERROR: /home/dl-audio/TF_01/sachinpro.github.io/tensorflow/cc/BUILD:28:1 1 input file(s) do not exist. INFO: Elapsed time: 5.863s, Critical Path: 3.06s dl-audio@powerbox:~/TF01/sachinpro.github.io$ " In one of your comments you also discussed about this error and you also resolved this error right ? But i still saw this error , i will refer to your earlier comment to resolve it .Wanted to confirm if this was fixed before in your code then why it again occurred, am i still missing something ?

Thanks

uvaidya commented 8 years ago

For error mentioned above , do we need to have a change like filegroup( name = 'clang-as', srcs = [ 'clang_bin/as', ':as', ], ) In tools/arm_compiler/linaro_linux_gcc/BUILD ?

sachinpro commented 8 years ago

Did you copy your arm toolchain to ./compilers ??

uvaidya commented 8 years ago

I copied gcc-linaro-4.9-2015.05-x86_64_arm-linux-gnueabihf , which was mentioned in https://github.com/bazelbuild/bazel/wiki/Building-with-a-custom-toolchain in compilers .

command used is : _bazel build --crosstool_top=//tools/arm_compiler:toolchain --verbose_failures --sandbox_debug --cpu=armeabi-v7a tensorflow/examples/labelimage/...

sachinpro commented 8 years ago

Make the symlinks /tools/arm_compiler/linaro_linux_gcc/clang_bin/as and /tools/arm_compiler/linaro_linux_gcc/clang_bin/ld point to /tools/arm_compiler/linaro_linux_gcc/clang_bin/arm-linux-gnueabihf-as and /tools/arm_compiler/linaro_linux_gcc/clang_bin/arm-linux-gnueabihf-ld respectively . I forgot to keep this in mind. This should work.