search

grpc / grpc-java

The Java gRPC implementation. HTTP/2 based RPC
https://grpc.io/docs/languages/java/
Apache License 2.0
11.44k stars 3.85k forks source link

linker fails when building under Alpine #11475

Open strophy opened 2 months ago

strophy commented 2 months ago

I'm trying to package grpc-java 1.66.0 for Alpine Linux. I have modified compiler/build.gradle as follows:

cppCompiler.executable = 'aarch64-alpine-linux-musl-c++'
linker.executable = 'aarch64-alpine-linux-musl-c++'

I'm using the following steps to run the build:

apk add openjdk21-jre-headless protobuf-dev nss
wget https://github.com/grpc/grpc-java/archive/v1.66.0.tar.gz
tar xzf v1.66.0.tar.gz 
cd grpc-java-1.66.0/compiler
export GRADLE_USER_HOME="$srcdir"/.gradle
export CXXFLAGS="--std=c++17"
export LDFLAGS="-L/usr/lib"
../gradlew --no-daemon --parallel --info java_pluginExecutable -PskipAndroid=true

The build seems to succeed, but linking fails with:

> Task :grpc-compiler:compileJava_pluginExecutableJava_pluginCpp UP-TO-DATE
Found all include files for ':grpc-compiler:compileJava_pluginExecutableJava_pluginCpp'
Caching disabled for task ':grpc-compiler:compileJava_pluginExecutableJava_pluginCpp' because:
  Build cache is disabled
Skipping task ':grpc-compiler:compileJava_pluginExecutableJava_pluginCpp' as it is up-to-date.
Resolve mutations for :grpc-compiler:linkJava_pluginExecutable (Thread[#74,Execution worker Thread 5,5,main]) started.
:grpc-compiler:linkJava_pluginExecutable (Thread[#74,Execution worker Thread 5,5,main]) started.

> Task :grpc-compiler:linkJava_pluginExecutable FAILED
Caching disabled for task ':grpc-compiler:linkJava_pluginExecutable' because:
  Build cache is disabled
  Not made cacheable, yet
Task ':grpc-compiler:linkJava_pluginExecutable' is not up-to-date because:
  Task has failed previously.
See file:///home/builder/package/grpc-java-1.66.0/compiler/build/tmp/linkJava_pluginExecutable/output.txt for all output for linkJava_pluginExecutable.
Starting process 'command '/usr/bin/g++''. Working directory: /home/builder/package/grpc-java-1.66.0/compiler Command: /usr/bin/g++ @/home/builder/package/grpc-java-1.66.0/compiler/build/tmp/linkJava_pluginExecutable/options.txt -m64
Successfully started process 'command '/usr/bin/g++''
/usr/lib/gcc/x86_64-alpine-linux-musl/13.2.1/../../../../x86_64-alpine-linux-musl/bin/ld: cannot find -lprotoc: No such file or directory
/usr/lib/gcc/x86_64-alpine-linux-musl/13.2.1/../../../../x86_64-alpine-linux-musl/bin/ld: cannot find -lprotobuf: No such file or directory
collect2: error: ld returned 1 exit status

Finished linkJava_pluginExecutable, see full log file:///home/builder/package/grpc-java-1.66.0/compiler/build/tmp/linkJava_pluginExecutable/output.txt.

FAILURE: Build failed with an exception.

But the libraries installed by protobuf-dev exist:

/home/builder/package/grpc-java-1.66.0/compiler # ls -lha /usr/lib/libproto*
lrwxrwxrwx 1 root root   22 Aug 13 21:18 /usr/lib/libprotobuf-lite.so -> libprotobuf-lite.so.24
lrwxrwxrwx 1 root root   26 Aug 13 21:18 /usr/lib/libprotobuf-lite.so.24 -> libprotobuf-lite.so.24.4.0
-rwxr-xr-x 1 root root 487K Jul 22 17:26 /usr/lib/libprotobuf-lite.so.24.4.0
lrwxrwxrwx 1 root root   17 Aug 13 21:18 /usr/lib/libprotobuf.so -> libprotobuf.so.24
lrwxrwxrwx 1 root root   21 Aug 13 21:18 /usr/lib/libprotobuf.so.24 -> libprotobuf.so.24.4.0
-rwxr-xr-x 1 root root 2.7M Jul 22 17:26 /usr/lib/libprotobuf.so.24.4.0
lrwxrwxrwx 1 root root   15 Aug 13 21:18 /usr/lib/libprotoc.so -> libprotoc.so.24
lrwxrwxrwx 1 root root   19 Aug 13 21:18 /usr/lib/libprotoc.so.24 -> libprotoc.so.24.4.0
-rwxr-xr-x 1 root root 2.4M Jul 22 17:26 /usr/lib/libprotoc.so.24.4.0

Can anyone help me resolve this linking issue?

strophy commented 2 months ago

After a bit more head scratching, I came across this previous attempt: https://gitlab.alpinelinux.org/alpine/aports/-/merge_requests/54120

According to https://github.com/grpc/grpc-java/blob/v1.52.1/compiler/build.gradle#L112 it seems grpc-java wants to be linked with static versions of libprotoc and libprotobuf.

Since Alpine now builds protobuf with cmake instead of make as shown in the compile environment setup steps described here, static linking is not an option any longer. Alpine also won't accept a PR that vendors static protobuf together with grpc-java, so my question is now how can I modify build.gradle to dynamically link against the `/usr/lib/libproto*.so files?

I tried modifying the linker.args a few times like this:

linker.args "-lprotoc", "-lprotobuf", "-labsl_raw_hash_set", "-labsl_hash", "-labsl_base", "-labsl_raw_logging_internal", "-lpthread", "-s",
// linker.args "-lprotoc", "-lprotobuf", "-labsl_raw_hash_set", "-labsl_hash", "-labsl_base", "-lpthread", "-s",
// linker.args "-lprotoc", "-lprotobuf", "-labsl_raw_hash_set", "-labsl_base", "-lpthread", "-s"
// linker.args "-lprotoc", "-lprotobuf", "-labsl_raw_hash_set", "-lpthread", "-s"
// linker.args "-lprotoc", "-lprotobuf", "-lpthread", "-s",
// linker.args "-Wl,-Bstatic", "-lprotoc", "-lprotobuf", "-static-libgcc",
// "-Wl,-Bdynamic", "-pthread", "-s"

Which ultimately results in the error:

> Task :grpc-compiler:linkJava_pluginExecutable FAILED
/usr/lib/gcc/x86_64-alpine-linux-musl/13.2.1/../../../../x86_64-alpine-linux-musl/bin/ld: /home/builder/package/grpc-java-1.66.0/compiler/build/objs/java_plugin/java_pluginCpp/cmkx8cb9wgxcy6i9dx7v2pqn9/java_generator.o: in function `java_grpc_generator::MessageFullJavaName(google::protobuf::Descriptor const*)':
java_generator.cpp:(.text+0x5d0): undefined reference to `google::protobuf::compiler::java::ClassName[abi:cxx11](google::protobuf::Descriptor const*)'
/usr/lib/gcc/x86_64-alpine-linux-musl/13.2.1/../../../../x86_64-alpine-linux-musl/bin/ld: /home/builder/package/grpc-java-1.66.0/compiler/build/objs/java_plugin/java_pluginCpp/cmkx8cb9wgxcy6i9dx7v2pqn9/java_generator.o: in function `java_grpc_generator::PrintGetServiceDescriptorMethod(google::protobuf::ServiceDescriptor const*, std::map<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::less<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > >*, google::protobuf::io::Printer*, java_grpc_generator::ProtoFlavor)':
java_generator.cpp:(.text+0x5a15): undefined reference to `google::protobuf::compiler::java::ClassName[abi:cxx11](google::protobuf::FileDescriptor const*)'
/usr/lib/gcc/x86_64-alpine-linux-musl/13.2.1/../../../../x86_64-alpine-linux-musl/bin/ld: /home/builder/package/grpc-java-1.66.0/compiler/build/objs/java_plugin/java_pluginCpp/cmkx8cb9wgxcy6i9dx7v2pqn9/java_generator.o: in function `java_grpc_generator::ServiceJavaPackage[abi:cxx11](google::protobuf::FileDescriptor const*)':
java_generator.cpp:(.text+0x8b00): undefined reference to `google::protobuf::compiler::java::ClassName[abi:cxx11](google::protobuf::FileDescriptor const*)'
collect2: error: ld returned 1 exit status

FAILURE: Build failed with an exception

Is this an issue with different compiler versions or CXXFLAGS specifying C++ standards being applied at the time Cmake is building protobuf? It is trying to link against protobuf 24.4 which would seem to satisfy the requirement for protobuf 21.7+. The change requiring static linking was made 9 years ago when Gradle 2.3 was in use, is dynamic linking still impossible today?

ejona86 commented 2 months ago

You might want to glance at https://github.com/grpc/grpc-java/pull/10048 . We haven't been able to update to any absl-based protobuf release for our official binaries. Our Bazel build is working with the newer protobuf.

Since Alpine now builds protobuf with cmake instead of make as shown in the compile environment setup steps described here, static linking is not an option any longer.

Static linking is definitely still an option. But overall absl is the issue and makes our build a PITA. I got Linux building locally by hard-coding the huge list of -l (as protobuf was broken at the time). Later I tried a bit to use pkg-config, but Windows was in a bad situation then; pkgconf makes things better but we haven't gotten back to it. We are seriously considering swapping the protoc-gen-grpc-java build exclusively to Bazel as the Gradle build just doesn't seem like it will work well any more. And we could then follow protobuf and provide fully-static binaries built on modern distros.