Closed j256 closed 6 years ago
1) for me I could solve similar issue using jetty-alpn-agent
jar - see my reproducable codebase here - https://github.com/upd-machine/nlu-pipeline/issues/1 (I am using against google-dialogflow though)
wget http://central.maven.org/maven2/org/mortbay/jetty/alpn/jetty-alpn-agent/2.0.7/jetty-alpn-agent-2.0.7.jar
java -jar -javaagent:jetty-alpn-agent-2.0.7.jar nlu-intent-toolkit.jar agent1 "where is porcupine tree playing?" credentials.json
But that is not a prod ready code so i m not sure if that's the best way i will fix it. This might be helpful - https://github.com/grpc/grpc-java/blob/master/SECURITY.md#tls-with-jdk-jetty-alpnnpn
2) ALSO, their google-cloud-compat-checker
might be useful. Its bit painful to use it because they use parent versions. I don't understand why dont make it a jar that can be used anywhere.
the compat-checker tells if you have ALPN supported in your environment, example,
$ mvn exec:java
[INFO] Scanning for projects...
[INFO]
[INFO] ------------------------------------------------------------------------
[INFO] Building Google Cloud Java Compatibility Checker 0.39.0-alpha
[INFO] ------------------------------------------------------------------------
[INFO]
[INFO] --- exec-maven-plugin:1.6.0:java (default-cli) @ google-cloud-compat-checker ---
OS details:
os.detected.name: osx
os.detected.arch: x86_64
os.detected.classifier: osx-x86_64
os.detected.release: null
os.detected.release.version: null
JVM details:
Java version: 1.8.0_151
Java specification version: 1.8
JVM bit mode: 64
OpenSSL details:
open ssl is available: false
ALPN is supported: false
Checking compatibility...
[PASS] This OS + architecture is supported.
[PASS] 64-bit JVM is supported.
[FAIL] Open SSL is NOT available
Open SSL Unavailability cause:
java.lang.IllegalArgumentException: Failed to load any of the given libraries: [netty-tcnative-osx-x86_64, netty-tcnative]
at io.netty.util.internal.NativeLibraryLoader.loadFirstAvailable(NativeLibraryLoader.java:178)
at io.netty.handler.ssl.OpenSsl.loadTcNative(OpenSsl.java:403)
at io.netty.handler.ssl.OpenSsl.<clinit>(OpenSsl.java:85)
at com.google.cloud.compatchecker.GoogleCloudCompatChecker.check(GoogleCloudCompatChecker.java:58)
at com.google.cloud.compatchecker.GoogleCloudCompatChecker.main(GoogleCloudCompatChecker.java:47)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.codehaus.mojo.exec.ExecJavaMojo$1.run(ExecJavaMojo.java:282)
at java.lang.Thread.run(Thread.java:748)
[FAIL] Open SSL ALPN is NOT supported
Result: FAIL
Your environment is not supported by Forked Tomcat Native.
See http://netty.io/wiki/forked-tomcat-native.html for details.
This means that you won't be able to use grpc-based APIs, but
http1-based APIs should still work.
That didn't seem to work for me. Neither jetty-alpn-agent
nor jetty-alpn-client
. The problem is in io.grpc.netty.JettyTlsUtil
which is doing a Class.forName("org.eclipse.jetty.alpn.ALPN", true, null);
in the isJettyAlpnConfigured()
method.
@j256 I'm glad you figured this out, but this is rather surprising. We should get to the bottom of this.
ALPN is a red herring of sorts. The ssl layer that's supposed to be used is netty-tcnative-boringssl-static, which should be built into grpc-netty-shaded. If you are facing issues with ssl not being available, it's more likely a platform issue. Could you run google-cloud-compat-checker
like @prayagupd suggested?
Hi @j256 I took your Test Subscriber code and tested in my MacOS with java 7/java 8. Its working fine - Here's my testcode for both emitting and consuming events from google pubsub -
https://github.com/duwamish-os/eventstream-skipper-ii.java (run mvn exec:java
. It will show you google-cloud logs as well). Please let me know if you have different settings than how I tested.
Dependency jars same as yours;
mvn dependency:tree
[INFO] Scanning for projects...
[INFO]
[INFO] ------------------------------------------------------------------------
[INFO] Building eventstream-skipper-java 1.0-SNAPSHOT
[INFO] ------------------------------------------------------------------------
[INFO]
[INFO] --- maven-dependency-plugin:2.8:tree (default-cli) @ eventstream-skipper-java ---
[INFO] com.eventstream.googlepubsub:eventstream-skipper-java:jar:1.0-SNAPSHOT
[INFO] +- com.google.cloud:google-cloud-pubsub:jar:0.39.0-beta:compile
[INFO] | +- com.google.cloud:google-cloud-core:jar:1.21.0:compile
[INFO] | | +- com.google.guava:guava:jar:20.0:compile
[INFO] | | +- joda-time:joda-time:jar:2.9.2:compile
[INFO] | | +- com.google.http-client:google-http-client:jar:1.23.0:compile
[INFO] | | | \- org.apache.httpcomponents:httpclient:jar:4.0.1:compile
[INFO] | | | +- org.apache.httpcomponents:httpcore:jar:4.0.1:compile
[INFO] | | | +- commons-logging:commons-logging:jar:1.1.1:compile
[INFO] | | | \- commons-codec:commons-codec:jar:1.3:compile
[INFO] | | +- com.google.code.findbugs:jsr305:jar:3.0.1:compile
[INFO] | | +- com.google.api:api-common:jar:1.4.0:compile
[INFO] | | +- com.google.api:gax:jar:1.19.0:compile
[INFO] | | | +- org.threeten:threetenbp:jar:1.3.3:compile
[INFO] | | | \- com.google.auth:google-auth-library-oauth2-http:jar:0.9.0:compile
[INFO] | | | \- com.google.http-client:google-http-client-jackson2:jar:1.19.0:compile
[INFO] | | | \- com.fasterxml.jackson.core:jackson-core:jar:2.1.3:compile
[INFO] | | +- com.google.protobuf:protobuf-java-util:jar:3.5.1:compile
[INFO] | | | \- com.google.code.gson:gson:jar:2.7:compile
[INFO] | | +- com.google.api.grpc:proto-google-common-protos:jar:1.3.0:compile
[INFO] | | \- com.google.api.grpc:proto-google-iam-v1:jar:0.4.0:compile
[INFO] | +- com.google.cloud:google-cloud-core-grpc:jar:1.21.0:compile
[INFO] | | +- com.google.auth:google-auth-library-credentials:jar:0.9.0:compile
[INFO] | | +- com.google.protobuf:protobuf-java:jar:3.5.1:compile
[INFO] | | +- io.grpc:grpc-protobuf:jar:1.9.0:compile
[INFO] | | | \- io.grpc:grpc-protobuf-lite:jar:1.9.0:compile
[INFO] | | +- io.grpc:grpc-context:jar:1.9.0:compile
[INFO] | | \- com.google.api:gax-grpc:jar:1.19.0:compile
[INFO] | +- com.google.api.grpc:proto-google-cloud-pubsub-v1:jar:0.4.0:compile
[INFO] | +- com.google.api.grpc:grpc-google-cloud-pubsub-v1:jar:0.4.0:compile
[INFO] | +- io.grpc:grpc-netty-shaded:jar:1.9.0:compile
[INFO] | | \- io.grpc:grpc-core:jar:1.9.0:compile (version selected from constraint [1.9.0,1.9.0])
[INFO] | | +- com.google.errorprone:error_prone_annotations:jar:2.1.2:compile
[INFO] | | +- com.google.instrumentation:instrumentation-api:jar:0.4.3:compile
[INFO] | | +- io.opencensus:opencensus-api:jar:0.10.0:compile
[INFO] | | \- io.opencensus:opencensus-contrib-grpc-metrics:jar:0.10.0:compile
[INFO] | +- io.grpc:grpc-stub:jar:1.9.0:compile
[INFO] | +- io.grpc:grpc-auth:jar:1.9.0:compile
[INFO] | \- com.google.auto.value:auto-value:jar:1.4:compile
[INFO] \- com.google.cloud:google-cloud-logging-logback:jar:0.39.0-alpha:compile
[INFO] +- ch.qos.logback:logback-classic:jar:1.2.3:compile
[INFO] | +- ch.qos.logback:logback-core:jar:1.2.3:compile
[INFO] | \- org.slf4j:slf4j-api:jar:1.7.25:compile
[INFO] \- com.google.cloud:google-cloud-logging:jar:1.21.0:compile
[INFO] \- com.google.api.grpc:proto-google-cloud-logging-v2:jar:0.4.0:compile
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 1.335 s
[INFO] Finished at: 2018-03-20T23:13:01-07:00
[INFO] Final Memory: 15M/309M
[INFO] ------------------------------------------------------------------------
platform config is as below
$ mvn exec:java
[INFO] Scanning for projects...
[INFO]
[INFO] ------------------------------------------------------------------------
[INFO] Building Google Cloud Java Compatibility Checker 0.39.0-alpha
[INFO] ------------------------------------------------------------------------
[INFO]
[INFO] --- exec-maven-plugin:1.6.0:java (default-cli) @ google-cloud-compat-checker ---
OS details:
os.detected.name: osx
os.detected.arch: x86_64
os.detected.classifier: osx-x86_64
os.detected.release: null
os.detected.release.version: null
JVM details:
Java version: 1.7.0_80
Java specification version: 1.7
JVM bit mode: 64
OpenSSL details:
open ssl is available: false
ALPN is supported: false
Checking compatibility...
[PASS] This OS + architecture is supported.
[PASS] 64-bit JVM is supported.
[FAIL] Open SSL is NOT available
Open SSL Unavailability cause:
java.lang.IllegalArgumentException: Failed to load any of the given libraries: [netty-tcnative-osx-x86_64, netty-tcnative]
at io.netty.util.internal.NativeLibraryLoader.loadFirstAvailable(NativeLibraryLoader.java:178)
at io.netty.handler.ssl.OpenSsl.loadTcNative(OpenSsl.java:403)
at io.netty.handler.ssl.OpenSsl.<clinit>(OpenSsl.java:85)
at com.google.cloud.compatchecker.GoogleCloudCompatChecker.check(GoogleCloudCompatChecker.java:58)
at com.google.cloud.compatchecker.GoogleCloudCompatChecker.main(GoogleCloudCompatChecker.java:47)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.codehaus.mojo.exec.ExecJavaMojo$1.run(ExecJavaMojo.java:282)
at java.lang.Thread.run(Thread.java:745)
[FAIL] Open SSL ALPN is NOT supported
[WARN] gRPC doesn't work on Google App Engine Standard under Java 1.7
Result: FAIL
Your environment is not supported by Forked Tomcat Native.
See http://netty.io/wiki/forked-tomcat-native.html for details.
This means that you won't be able to use grpc-based APIs, but
http1-based APIs should still work.
Ok. Swapping this back in. Thanks for the help @prayagupd . I'm getting the following from the compatibility checker which seems to match your's.
google-cloud-compat-checker {localhost} mvn exec:java
[INFO] Scanning for projects...
[INFO] Inspecting build with total of 1 modules...
[INFO] Installing Nexus Staging features:
[INFO] ... total of 1 executions of maven-deploy-plugin replaced with nexus-staging-maven-plugin
[INFO]
[INFO] Using the builder org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder with a thread count of 1
[INFO]
[INFO] ------------------------------------------------------------------------
[INFO] Building Google Cloud Java Compatibility Checker 0.40.1-alpha-SNAPSHOT
[INFO] ------------------------------------------------------------------------
```[INFO]
[INFO] --- exec-maven-plugin:1.6.0:java (default-cli) @ google-cloud-compat-checker ---
OS details:
os.detected.name: linux
os.detected.arch: x86_64
os.detected.classifier: linux-x86_64
os.detected.release: "rhel"
os.detected.release.version: 7.3
JVM details:
Java version: 1.7.0_79
Java specification version: 1.7
JVM bit mode: 64
OpenSSL details:
open ssl is available: false
ALPN is supported: false
Checking compatibility...
[PASS] This OS + architecture is supported.
[PASS] 64-bit JVM is supported.
[FAIL] Open SSL is NOT available
Open SSL Unavailability cause:
java.lang.IllegalArgumentException: Failed to load any of the given libraries: [netty-tcnative-linux-x86_64, netty-tcnative-linux-x86_64-fedora, netty-tcnative]
at io.netty.util.internal.NativeLibraryLoader.loadFirstAvailable(NativeLibraryLoader.java:178)
at io.netty.handler.ssl.OpenSsl.loadTcNative(OpenSsl.java:403)
at io.netty.handler.ssl.OpenSsl.<clinit>(OpenSsl.java:85)
at com.google.cloud.compatchecker.GoogleCloudCompatChecker.check(GoogleCloudCompatChecker.java:58)
at com.google.cloud.compatchecker.GoogleCloudCompatChecker.main(GoogleCloudCompatChecker.java:47)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.codehaus.mojo.exec.ExecJavaMojo$1.run(ExecJavaMojo.java:282)
at java.lang.Thread.run(Thread.java:745)
[FAIL] Open SSL ALPN is NOT supported
[WARN] gRPC doesn't work on Google App Engine Standard under Java 1.7
Result: FAIL
Your environment is not supported by Forked Tomcat Native.
See http://netty.io/wiki/forked-tomcat-native.html for details.
This means that you won't be able to use grpc-based APIs, but
http1-based APIs should still work.
However, when I go to run your program I get similar errors to mine and I'm seeing the following in my logs.
17:18:03.502 [main] DEBUG i.g.n.s.io.netty.handler.ssl.OpenSsl - Failed to load netty-tcnative; OpenSslEngine will be unavailable, unless the application has already loaded the symbols by some other means. See http://netty.io/wiki/forked-tomcat-native.html for more information.
java.lang.IllegalArgumentException: Failed to load any of the given libraries: [netty_tcnative_linux_x86_32, netty_tcnative_linux_x86_32_fedora, netty_tcnative_x86_32, netty_tcnative]
Ultimately I get the followign with the -javaagent:jetty-alpn-agent-2.0.7.jar argument on the commandline.
Caused by: javax.net.ssl.SSLHandshakeException: No appropriate protocol (protocol is disabled or cipher suites are inappropriate)
I'm running on a:
> uname -a
Linux gwatson-l 3.10.0-693.5.2.el7.x86_64 #1 SMP Fri Oct 13 10:46:25 EDT 2017 x86_64 x86_64 x86_64 GNU/Linux
> cat /etc/redhat-release
Red Hat Enterprise Linux Server release 7.3 (Maipo)
Looks like none of the shared libraries load for me. I see a bunch of log messages like:
17:18:03.492 [main] DEBUG i.g.n.s.i.n.u.i.NativeLibraryLoader - Unable to load the library 'io_grpc_netty_shaded_netty_tcnative_linux_x86_32', trying other loading mechanism.
java.lang.UnsatisfiedLinkError: no io_grpc_netty_shaded_netty_tcnative_linux_x86_32 in java.library.path
17:18:03.493 [main] DEBUG i.g.n.s.i.n.u.i.NativeLibraryLoader - io_grpc_netty_shaded_netty_tcnative_linux_x86_32 cannot be loaded from java.libary.path, now trying export to -Dio.netty.native.workdir: /tmp
java.lang.UnsatisfiedLinkError: no io_grpc_netty_shaded_netty_tcnative_linux_x86_32 in java.library.path
17:18:03.494 [main] DEBUG i.g.n.s.i.n.u.i.NativeLibraryLoader - Unable to load the library 'netty_tcnative_linux_x86_32', trying next name...
17:18:03.494 [main] DEBUG i.g.n.s.i.n.u.i.NativeLibraryLoader - Unable to load the library 'io_grpc_netty_shaded_netty_tcnative_linux_x86_32_fedora', trying other loading mechanism.
17:18:03.494 [main] DEBUG i.g.n.s.i.n.u.i.NativeLibraryLoader - io_grpc_netty_shaded_netty_tcnative_linux_x86_32_fedora cannot be loaded from java.libary.path, now trying export to -Dio.netty.native.workdir: /tmp
17:18:03.495 [main] DEBUG i.g.n.s.i.n.u.i.NativeLibraryLoader - Unable to load the library 'netty_tcnative_linux_x86_32_fedora', trying next name...
17:18:03.495 [main] DEBUG i.g.n.s.i.n.u.i.NativeLibraryLoader - Unable to load the library 'io_grpc_netty_shaded_netty_tcnative_x86_32', trying other loading mechanism.
17:18:03.498 [main] DEBUG i.g.n.s.i.n.u.i.NativeLibraryLoader - io_grpc_netty_shaded_netty_tcnative_x86_32 cannot be loaded from java.libary.path, now trying export to -Dio.netty.native.workdir: /tmp
17:18:03.498 [main] DEBUG i.g.n.s.i.n.u.i.NativeLibraryLoader - Unable to load the library 'netty_tcnative_x86_32', trying next name...
17:18:03.498 [main] DEBUG i.g.n.s.i.n.u.i.NativeLibraryLoader - Unable to load the library 'io_grpc_netty_shaded_netty_tcnative', trying other loading mechanism.
17:18:03.499 [main] DEBUG i.g.n.s.i.n.u.i.NativeLibraryLoader - io_grpc_netty_shaded_netty_tcnative cannot be loaded from java.libary.path, now trying export to -Dio.netty.native.workdir: /tmp
17:18:03.499 [main] DEBUG i.g.n.s.i.n.u.i.NativeLibraryLoader - Unable to load the library 'netty_tcnative', trying next name...
17:18:03.502 [main] DEBUG i.g.n.s.io.netty.handler.ssl.OpenSsl - Failed to load netty-tcnative; OpenSslEngine will be unavailable, unless the application has already loaded the symbols by some other means. See http://netty.io/wiki/forked-tomcat-native.html for more information.
17:18:03.503 [main] DEBUG i.g.n.s.io.netty.handler.ssl.OpenSsl - Failed to initialize netty-tcnative; OpenSslEngine will be unavailable. See http://netty.io/wiki/forked-tomcat-native.html for more information.
That's interesting because I'm on a non-fedora 64-bit machine but the messages seem to be trying to load a 32-bit version. Let me see if I can force that selection. I've tried the static linked stuff and forcing the native classifier <classifier>linux-x86_64</classifier>
.
Wow, this just came up again on another developer's box. Related to https://github.com/grpc/grpc-java/issues/3025
@j256 , the real error that is relevant is "gRPC doesn't work on Google App Engine Standard under Java 1.7". The easiest way to fix this is to upgrade to Java 1.8 on App Engine, which should work fine.
This applies to @prayagupd 's case too.
Moving to Java 8 just isn't possible given my corporate software requirements.
That is very unfortunate - the Java 7 runtime of App Engine will be entirely shut down in January next year (2019): https://cloud.google.com/appengine/docs/deprecations/java7
Ok. I think I've finally figured this out. We run with a lot of different JREs on our servers supporting various Java variants and I believe this is a 32bit/64bit issue and not a Java7 versus 8 issue. I'm going to mark this closed but I'll add some details here as I get them. Thanks for all of the help.
For the record, I continue to have problems with this on older build boxes running older variants of LibC, mismatched 32-bit and 64-bit JREs, and mismatched JRE versions. It's tricky. Older servers are especially susceptible to this.
Also, older servers with glibc < version 2.7 will not work because of netty requirements. Yes, sigh, in 2018 my company has to support our code on these older servers.
Although I've seen similar issues, they all seem to be closed, I'm still seeing the following error with recent versions of this API. Related to: https://github.com/GoogleCloudPlatform/google-cloud-java/issues/2740 https://github.com/GoogleCloudPlatform/google-cloud-java/issues/3036
My test program is doing the following:
Here's my maven pom.xml dependency. Dependency tree attached:
Any idea what I'm doing wrong? Can I even run this outside of the Google network? Please excuse the obvious ignorance of this stuff.
dependencies.txt