During TF build with bazel, protoc seems not built fresh ?

beyret commented 7 years ago

Hi,

Thank you for comprehensive instructions. I am newbie on this :) and I may be doing something wrong but I could not yet compile TF due to the issue below.

Thank you in advance for any suggestions !

Cheers Ersin

Describe the Issue

When following step-by-step instructions, I managed to build bazel. However, during TF build step at some point I get following error (copy/pasted from ~/.cache/bazel/_bazel_pi/369d1e70a26639ff91587139b0224d09/command.log)

=======

ERROR: /home/pi/.cache/bazel/_bazel_pi/369d1e70a26639ff91587139b0224d09/external/io_bazel_rules_closure/java/io/bazel/rules/closure/webfiles/s erver/BUILD:48:1: error executing shell command: 'JAR='external/local_jdk/bin/jar' OUTPUT='bazel-out/host/bin/external/io_bazel_rules_closure/java/io/bazel/r ules/closure/webfiles/server/libbuild_info_java_proto_srcjar.srcjar' PROTO_COMPILER='exter...' failed: bash failed: error executing command (cd /home/pi/.cache/bazel/_bazel_pi/369d1e70a26639ff91587139b0224d09/execroot/tensorflow && \ exec env - \ PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/local/games:/usr/games \ /bin/bash -c 'JAR='\''external/local_jdk/bin/jar'\'' OUTPUT='\''bazel-out/host/bin/external/io_bazel_rules_closure/java/io/bazel/rules/closure/webfiles/ser ver/libbuild_info_java_proto_srcjar.srcjar'\'' PROTO_COMPILER='\''external/com_google_protobuf_protoc/bin/protoc'\'' SOURCE='\''external/io_bazel_rules_closu re/java/io/bazel/rules/closure/webfiles/server/build_info.proto'\'' INCLUDES='\''-I. -Iexternal/io_bazel_rules_closure'\'' bazel-out/host/bin/external/io_baz el_rules_closure/closure/private/gensrcjar'): com.google.devtools.build.lib.shell.BadExitStatusException: Process exited with status 1. external/com_google_protobuf_protoc/bin/protoc: 1: external/com_google_protobuf_protoc/bin/protoc: cannot create �/@@��: Directory nonexistent external/com_google_protobuf_protoc/bin/protoc: 1: external/com_google_protobuf_protoc/bin/protoc: ELF: not found external/com_google_protobuf_protoc/bin/protoc: 2: external/com_google_protobuf_protoc/bin/protoc: Syntax error: ")" unexpected gensrcjar: proto_compiler failed Target //tensorflow/tools/pip_package:build_pip_package failed to build

=======

After noticing strange characters, I checked protoc binary (at ~/.cache/bazel/_bazel_pi/369d1e70a26639ff91587139b0224d09/external/com_google_protobuf_protoc/bin) and found that it somehow has been copied from somewhere rather than getting compiled from its source :

-rwxr-x--- 1 pi pi 4114152 Sep 25 2016 protoc

and for wrong architecture (64bit and not for arm):

file protoc protoc: ELF 64-bit LSB executable, x86-64, version 1 (GNU/Linux), dynamically linked, interpreter /lib64/ld-linux-x86-64.so.2, for GNU/Linux 2.6.24, BuildID[sha1]=afe1ae3a88d6b51b14fb21830cc133d9107e6ce6, stripped

I am guessing that is the reason. Maybe in some config file an adjustment is needed to tell bazel to compile protoc rather than download its executable form.

Steps to Reproduce

At step-by-step instructions, issue occurs deterministically at this step:

bazel build -c opt --copt="-mfpu=neon-vfpv4" --copt="-funsafe-math-optimizations" --copt="-ftree-vectorize" --copt="-fomit-frame-pointer" --local_resources 1024,1.0,1.0 --verbose_failures tensorflow/tools/pip_package:build_pip_package

Hardware/Software Info

Please provide the following information about your Raspberry Pi setup:

Raspberry Pi model: 3
Operating System used: Raspbian GNU/Linux 8 (jessie)
Version of Python used: 3.4
SD card memory size: 32
Size of USB/other device used as swap (if building from source): 16
TensorFlow git commit hash (if building from source): Cloned from master on 6/18/2017

Relevant Console Output/Logs

samjabrahams commented 7 years ago

Thanks for opening this up @beyret - unfortunately, I'm not in a position to be able to walk through the steps on a Pi of my own for the next few weeks, but I'll try to come up with some suggestions that might help.

The first is to try using a different version of Bazel- If you used 0.4.5, try 0.5.1, the newest version. If you did 0.5.1, try the reverse. If those don't work, let me know and we'll try more things from there.

beyret commented 7 years ago

Thank you @samjabrahams for quick follow-up :) Just an update after some more progress:

I compiled Bazel 0.5.1. Encountered same issue (the protoc executable is obtained from somewhere instead of being freshly built).
I went ahead and compiled protoc executable by doing this:

cd ~/.cache/bazel/_bazel_pi/369d1e70a26639ff91587139b0224d09/external/protobuf bazel build protoc

And confirmed that it is built for ARM 32bit:

file bazel-bin/protoc

bazel-bin/protoc: ELF 32-bit LSB executable, ARM, EABI5 version 1 (SYSV), dynamically linked, interpreter /lib/ld-linux-armhf.so.3, for GNU/Linux 2.6.32, BuildID[sha1]=5d9674ac8c36cc8637c6bb6dc37bb60d2c7ad7c7, not stripped

Then copied it over to where bazel for TF uses it from (hoping that it won't overwrite)

cp bazel-bin/protoc ~/.cache/bazel/_bazel_pi/369d1e70a26639ff91587139b0224d09/external/com_google_protobuf_protoc/bin

And invoked the TF build again

bazel build ... tensorflow/tools/pip_package:build_pip_package

I think it went passed the error stage but later down the road it failed with seemingly another issue:

ERROR: /home/pi/.cache/bazel/_bazel_pi/369d1e70a26639ff91587139b0224d09/external/io_bazel_rules_closure/java/io/bazel/rules/closure/BUILD:62:1: Building external/io_bazel_rules_closure/java/io/bazel/rules/closure/libbuild_info_java_proto.jar (1 source jar) failed: Worker process sent response with exit code: 1. /io/bazel/rules/closure/BuildInfo.java:767: error: no suitable method found for parseFrom(ByteBuffer) return PARSER.parseFrom(data); ^ method Parser.parseFrom(CodedInputStream) is not applicable (argument mismatch; ByteBuffer cannot be converted to CodedInputStream) method Parser.parseFrom(ByteString) is not applicable (argument mismatch; ByteBuffer cannot be converted to ByteString) method Parser.parseFrom(byte[]) is not applicable (argument mismatch; ByteBuffer cannot be converted to byte[]) method Parser.parseFrom(InputStream) is not applicable (argument mismatch; ByteBuffer cannot be converted to InputStream) /io/bazel/rules/closure/BuildInfo.java:773: error: no suitable method found for parseFrom(ByteBuffer,ExtensionRegistryLite) return PARSER.parseFrom(data, extensionRegistry); ^ method Parser.parseFrom(CodedInputStream,ExtensionRegistryLite) is not applicable (argument mismatch; ByteBuffer cannot be converted to CodedInputStream) method Parser.parseFrom(ByteString,ExtensionRegistryLite) is not applicable (argument mismatch; ByteBuffer cannot be converted to ByteString) method Parser.parseFrom(byte[],ExtensionRegistryLite) is not applicable (argument mismatch; ByteBuffer cannot be converted to byte[]) method Parser.parseFrom(InputStream,ExtensionRegistryLite) is not applicable (argument mismatch; ByteBuffer cannot be converted to InputStream)

Is there a particular stable branch/date of tensorflow codestream you would recommend ? Maybe me trying to use the master code stream is not a good idea. By nature, it must have a lot of flux and things may be yet to be discovered/settled.

beyret commented 7 years ago

Just an update: I've been able to compile TF :)

I've used r1.2 branch instead of master (by doing "git checkout r1.2" after "git clone" of tensorflow repo). I anyway do not need very latest version at this point.
At 1.2, there is apparently a known (and recently fixed) issue there w.r.t Eigen (issue #101 thx @SteveBetter !). I've applied the patch from its raw diff as such:

cd ~/.cache/bazel/_bazel_pi/369d1e70a26639ff91587139b0224d09/external/eigen_archive git apply

Then TF build completed :)

bazel build ... tensorflow/tools/pip_package:build_pip_package

After creating the wheel per instructions, I used pip3 to install it:

sudo pip3 install /tmp/tensorflow_pkg/tensorflow-1.2.0-cp34-cp34m-linux_armv7l.whl

samjabrahams / tensorflow-on-raspberry-pi