aws-cpp-sdk Issue when building TensorFlow Serving for Jetson TX2 (aarch64 chipset)

[booglerz]

Hello, I hope this is as good a place as any to try and understand my installation issue. I've gone to both the TF Serving and Nvidia Jetson forums for help but apparently no one has tried this before. I'm attempting to install TensorFlow Serving on a Jetson TX2 (our TF models cannot be converted to UFFs easily at this time). However, when I bazel build tensorflow serving I get the following error, which seems to be related to AWS (see error below).

Any ideas what would cause this?

Is AWS cpp SDK compatible with aarch64 (arm) (the chipset the Jetson TX2 uses (https://www.nvidia.com/en-us/autonomous-machines/embedded-systems-dev-kits-modules/)?

Thanks for the help, really stuck on this one atm.

Andy

Distro: Ubuntu 16.04 with JetPack 3.2 on Jetson TX2 board.

bazel 0.11 (TF Serving requires 0.5.4 or higher, so I just went with the latest, I also get the same issue on 0.5.4)

Not sure cmake arguments (hidden somewere in tensorflow serving and bazel but I will try and find) gcc version: gcc (Ubuntu/Linaro 5.4.0-6ubuntu1~16.04.9) 5.4.0 20160609

Link to question on Nvidia devtalk: https://devtalk.nvidia.com/default/topic/1031642/tensorflow-serving-on-jetson-tx2/#5248552

Link to question on TF Serving github: https://github.com/tensorflow/serving/issues/832

Log: /home/nvidia/serving/tensorflow_serving/model_servers/BUILD:205:1: Linking of rule '//tensorflow_serving/model_servers:tensorflow_model_server' failed (Exit 1): gcc failed: error executing command (cd /home/nvidia/.cache/bazel/_bazel_root/e07dd11400dd0f5e80daa6d5086c0965/execroot/tf_serving && \ exec env - \ PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/snap/bin \ PWD=/proc/self/cwd \ PYTHON_BIN_PATH=/usr/bin/python \ /usr/bin/gcc -o bazel-out/arm-opt/bin/tensorflow_serving/model_servers/tensorflow_model_server -pthread -pthread -pthread -pthread -pthread -pthread -pthread -pthread -pthread -pthread -pthread -pthread -pthread -pthread -pthread -pthread -pthread -pthread -pthread -pthread -pthread -pthread -pthread -pthread -pthread -pthread -pthread -pthread -pthread -pthread -pthread -pthread -pthread -pthread -pthread -pthread -pthread -pthread -pthread -pthread '-fuse-ld=gold' -Wl,-no-as-needed -Wl,-z,relro,-z,now -B/usr/bin -B/usr/bin -pass-exit-codes -Wl,--gc-sections -Wl,@bazel-out/arm-opt/bin/tensorflow_serving/model_servers/tensorflow_model_server-2.params) bazel-out/arm-opt/bin/external/aws/_objs/aws/external/aws/aws-cpp-sdk-core/source/client/ClientConfiguration.o:ClientConfiguration.cpp:function Aws::Client::ComputeUserAgentString(): error: undefined reference to 'Aws::OSVersionInfo::ComputeOSVersionString[abi:cxx11]()' bazel-out/arm-opt/bin/external/aws/_objs/aws/external/aws/aws-cpp-sdk-core/source/utils/DateTimeCommon.o:DateTimeCommon.cpp:function Aws::Utils::DateTime::ToLocalTimeString[abi:cxx11](char const) const: error: undefined reference to 'Aws::Time::LocalTime(tm, long)' bazel-out/arm-opt/bin/external/aws/_objs/aws/external/aws/aws-cpp-sdk-core/source/utils/DateTimeCommon.o:DateTimeCommon.cpp:function Aws::Utils::DateTime::ToGmtString[abi:cxx11](char const) const: error: undefined reference to 'Aws::Time::GMTime(tm, long)' bazel-out/arm-opt/bin/external/aws/_objs/aws/external/aws/aws-cpp-sdk-core/source/utils/DateTimeCommon.o:DateTimeCommon.cpp:function Aws::Utils::DateTime::CalculateCurrentHour(): error: undefined reference to 'Aws::Time::LocalTime(tm, long)' bazel-out/arm-opt/bin/external/aws/_objs/aws/external/aws/aws-cpp-sdk-core/source/utils/DateTimeCommon.o:DateTimeCommon.cpp:function Aws::Utils::DateTime::ConvertTimestampStringToTimePoint(char const, Aws::Utils::DateFormat): error: undefined reference to 'Aws::Time::TimeGM(tm)' bazel-out/arm-opt/bin/external/aws/_objs/aws/external/aws/aws-cpp-sdk-core/source/utils/TempFile.o:TempFile.cpp:function Aws::Utils::TempFile::~TempFile(): error: undefined reference to 'Aws::FileSystem::RemoveFileIfExists(char const)' bazel-out/arm-opt/bin/external/aws/_objs/aws/external/aws/aws-cpp-sdk-core/source/utils/TempFile.o:TempFile.cpp:function Aws::Utils::ComputeTempFileName(char const, char const): error: undefined reference to 'Aws::FileSystem::CreateTempFilePath[abi:cxx11]()' bazel-out/arm-opt/bin/external/aws/_objs/aws/external/aws/aws-cpp-sdk-core/source/auth/AWSCredentialsProvider.o:AWSCredentialsProvider.cpp:function Aws::Auth::EnvironmentAWSCredentialsProvider::GetAWSCredentials(): error: undefined reference to 'Aws::Environment::GetEnv[abi:cxx11](char const)' bazel-out/arm-opt/bin/external/aws/_objs/aws/external/aws/aws-cpp-sdk-core/source/auth/AWSCredentialsProvider.o:AWSCredentialsProvider.cpp:function Aws::Auth::EnvironmentAWSCredentialsProvider::GetAWSCredentials(): error: undefined reference to 'Aws::Environment::GetEnv[abi:cxx11](char const)' bazel-out/arm-opt/bin/external/aws/_objs/aws/external/aws/aws-cpp-sdk-core/source/auth/AWSCredentialsProvider.o:AWSCredentialsProvider.cpp:function Aws::Auth::EnvironmentAWSCredentialsProvider::GetAWSCredentials(): error: undefined reference to 'Aws::Environment::GetEnv[abi:cxx11](char const)' bazel-out/arm-opt/bin/external/aws/_objs/aws/external/aws/aws-cpp-sdk-core/source/auth/AWSCredentialsProvider.o:AWSCredentialsProvider.cpp:function Aws::Auth::ProfileConfigFileAWSCredentialsProvider::GetConfigProfileFilename[abi:cxx11](): error: undefined reference to 'Aws::FileSystem::GetHomeDirectory[abi:cxx11]()' bazel-out/arm-opt/bin/external/aws/_objs/aws/external/aws/aws-cpp-sdk-core/source/auth/AWSCredentialsProvider.o:AWSCredentialsProvider.cpp:function Aws::Auth::ProfileConfigFileAWSCredentialsProvider::GetCredentialsProfileFilename[abi:cxx11](): error: undefined reference to 'Aws::Environment::GetEnv[abi:cxx11](char const)' bazel-out/arm-opt/bin/external/aws/_objs/aws/external/aws/aws-cpp-sdk-core/source/auth/AWSCredentialsProvider.o:AWSCredentialsProvider.cpp:function Aws::Auth::ProfileConfigFileAWSCredentialsProvider::GetCredentialsProfileFilename[abi:cxx11](): error: undefined reference to 'Aws::FileSystem::GetHomeDirectory[abi:cxx11]()' collect2: error: ld returned 1 exit status

[bretambrose]

All of the functions that it's complaining about are platform-specific and their implementations are pulled in from a directory underneath "aws-cpp-sdk-core/source/platform" based on the platform discovered at cmake setup time. From that, it seems likely that cmake is not recognizing your platform at all, and so no implementation source is pulled in for those functions.

Can you post the output from your setup run of cmake before building the AWS SDK?

[jesen8]

yes!,it's the error with aws,and i solve it by this issue

only add option: --config=noaws

[jesen8]

@bretambrose ,thanks so much!

[marianopeck]

I am currently facing the same issue on the Pi 3. I need to compile TensorFlow 1.13.1 and I am doing:

env JAVA_HOME="/usr/lib/jvm/java-1.8.0-openjdk-armhf" \
bazel --host_jvm_args=-Xmx1024m --host_jvm_args=-Xms384m build \
--config opt --verbose_failures --jobs=10 --local_resources 1024,1.0,1.0 \
--copt=-mfpu=neon-vfpv4 \
--copt=-ftree-vectorize \
--copt=-funsafe-math-optimizations \
--copt=-ftree-loop-vectorize \
--copt=-fomit-frame-pointer \
--copt=-DRASPBERRY_PI \
--host_copt=-mfpu=neon-vfpv4 \
--host_copt=-ftree-vectorize \
--host_copt=-funsafe-math-optimizations \
--host_copt=-ftree-loop-vectorize \
--host_copt=-fomit-frame-pointer \
--host_copt=-DRASPBERRY_PI \
//tensorflow/tools/lib_package:libtensorflow

How could I adapt that with the --config=noaws ? is that avaialble in 1.13.1? if not, what' the patch? I am not on aarch64 but arm 32 bits.

thanks

[marianopeck]

Well, this seems to have worked:

env JAVA_HOME="/usr/lib/jvm/java-1.8.0-openjdk-armhf" \
bazel --host_jvm_args=-Xmx1024m --host_jvm_args=-Xms384m build \
--config=noaws \
--config opt --verbose_failures --local_resources 1024,1.0,1.0 \
--copt=-mfpu=neon-vfpv4 \
--copt=-ftree-vectorize \
--copt=-funsafe-math-optimizations \
--copt=-ftree-loop-vectorize \
--copt=-fomit-frame-pointer \
--copt=-DRASPBERRY_PI \
--host_copt=-mfpu=neon-vfpv4 \
--host_copt=-ftree-vectorize \
--host_copt=-funsafe-math-optimizations \
--host_copt=-ftree-loop-vectorize \
--host_copt=-fomit-frame-pointer \
--host_copt=-DRASPBERRY_PI \
//tensorflow/tools/lib_package:libtensorflow