search

GoogleCloudPlatform / DataflowJavaSDK

Google Cloud Dataflow provides a simple, powerful model for building both batch and streaming parallel data processing pipelines.
http://cloud.google.com/dataflow
855 stars 324 forks source link

tcnative-boringssl initialization errors with grpc-java 1.5.0 #601

Closed ankurcha closed 2 years ago

ankurcha commented 7 years ago

In our latest version of a dataflow pipeline we introduced a dependency on grpc-java 1.5.0 and tcnative-boringssl 2.0.3.Final. Running this locally with directrunner works file but on the dataflow service I get this error

com.google.common.util.concurrent.UncheckedExecutionException: java.lang.IllegalArgumentException: Jetty ALPN/NPN has not been properly configured.
    at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2217)
    at com.google.common.cache.LocalCache.get(LocalCache.java:4154)
    at com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:4158)
    at com.google.common.cache.LocalCache$LocalLoadingCache.get(LocalCache.java:5147)
    at com.brightcove.rna.transforms.ContraService.get(ContraService.java:45)
    at com.brightcove.rna.transforms.functions.ViewMaterializerDoFn.startBundle(ViewMaterializerDoFn.java:63)
Caused by: java.lang.IllegalArgumentException: Jetty ALPN/NPN has not been properly configured.
    at io.grpc.netty.GrpcSslContexts.selectApplicationProtocolConfig(GrpcSslContexts.java:159)
    at io.grpc.netty.GrpcSslContexts.configure(GrpcSslContexts.java:136)
    at io.grpc.netty.GrpcSslContexts.configure(GrpcSslContexts.java:124)
    at io.grpc.netty.GrpcSslContexts.forClient(GrpcSslContexts.java:94)
    at com.brightcove.rna.model.Views.viewRegistryChannel(Views.java:21)
    at com.brightcove.rna.tools.ViewRegistryCacheLoader.createNewViewRegistryConnection(ViewRegistryCacheLoader.java:88)
    at com.brightcove.rna.tools.ViewRegistryCacheLoader.getClient(ViewRegistryCacheLoader.java:81)
    at com.brightcove.rna.tools.ViewRegistryCacheLoader.getViewsFromRegistry(ViewRegistryCacheLoader.java:97)
    at com.brightcove.rna.tools.ViewRegistryCacheLoader.load(ViewRegistryCacheLoader.java:65)
    at com.brightcove.rna.tools.ViewRegistryCacheLoader.load(ViewRegistryCacheLoader.java:32)
    at com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3716)
    at com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2424)
    at com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2298)
    at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2211)
    at com.google.common.cache.LocalCache.get(LocalCache.java:4154)
    at com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:4158)
    at com.google.common.cache.LocalCache$LocalLoadingCache.get(LocalCache.java:5147)
    at com.brightcove.rna.transforms.ContraService.get(ContraService.java:45)
    at com.brightcove.rna.transforms.functions.ViewMaterializerDoFn.startBundle(ViewMaterializerDoFn.java:63)
    at com.brightcove.rna.transforms.functions.ViewMaterializerDoFn$DoFnInvoker.invokeStartBundle(Unknown Source)
    at org.apache.beam.runners.core.SimpleDoFnRunner.startBundle(SimpleDoFnRunner.java:127)
    at com.google.cloud.dataflow.worker.SimpleParDoFn.reallyStartBundle(SimpleParDoFn.java:212)
    at com.google.cloud.dataflow.worker.SimpleParDoFn.processElement(SimpleParDoFn.java:222)
    at com.google.cloud.dataflow.worker.util.common.worker.ParDoOperation.process(ParDoOperation.java:48)
    at com.google.cloud.dataflow.worker.util.common.worker.OutputReceiver.process(OutputReceiver.java:52)
    at com.google.cloud.dataflow.worker.AssignWindowsParDoFnFactory$AssignWindowsParDoFn.processElement(AssignWindowsParDoFnFactory.java:116)
    at com.google.cloud.dataflow.worker.util.common.worker.ParDoOperation.process(ParDoOperation.java:48)
    at com.google.cloud.dataflow.worker.util.common.worker.OutputReceiver.process(OutputReceiver.java:52)
    at com.google.cloud.dataflow.worker.SimpleParDoFn$1.output(SimpleParDoFn.java:183)
    at org.apache.beam.runners.core.SimpleDoFnRunner.outputWindowedValue(SimpleDoFnRunner.java:211)
    at org.apache.beam.runners.core.SimpleDoFnRunner.access$700(SimpleDoFnRunner.java:66)
    at org.apache.beam.runners.core.SimpleDoFnRunner$OnTimerArgumentProvider.output(SimpleDoFnRunner.java:664)
    at org.apache.beam.runners.core.SimpleDoFnRunner$OnTimerArgumentProvider.output(SimpleDoFnRunner.java:654)
    at java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:184)
    at java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193)
    at java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193)
    at java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193)
    at java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193)
    at java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193)
    at java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193)
    at java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193)
    at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1376)
    at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:481)
    at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:471)
    at java.util.stream.ForEachOps$ForEachOp.evaluateSequential(ForEachOps.java:151)
    at java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateSequential(ForEachOps.java:174)
    at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
    at java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:418)
    at com.brightcove.rna.transforms.session.MassageSessionFn.onFinallyTimer(MassageSessionFn.java:242)
    at com.brightcove.rna.transforms.session.MassageSessionFn$OnTimerInvoker$finallyTimer$ZmluYWxseVRpbWVy.invokeOnTimer(Unknown Source)
    at org.apache.beam.sdk.transforms.reflect.ByteBuddyDoFnInvokerFactory$DoFnInvokerBase.invokeOnTimer(ByteBuddyDoFnInvokerFactory.java:172)
    at org.apache.beam.runners.core.SimpleDoFnRunner.onTimer(SimpleDoFnRunner.java:171)
    at com.google.cloud.dataflow.worker.SimpleParDoFn.processUserTimer(SimpleParDoFn.java:257)
    at com.google.cloud.dataflow.worker.SimpleParDoFn.access$500(SimpleParDoFn.java:65)
    at com.google.cloud.dataflow.worker.SimpleParDoFn$TimerType$1.processTimer(SimpleParDoFn.java:336)
    at com.google.cloud.dataflow.worker.SimpleParDoFn.processTimers(SimpleParDoFn.java:365)
    at com.google.cloud.dataflow.worker.SimpleParDoFn.processTimers(SimpleParDoFn.java:250)
    at com.google.cloud.dataflow.worker.util.common.worker.ParDoOperation.finish(ParDoOperation.java:56)
    at com.google.cloud.dataflow.worker.util.common.worker.MapTaskExecutor.execute(MapTaskExecutor.java:76)
    at com.google.cloud.dataflow.worker.StreamingDataflowWorker.process(StreamingDataflowWorker.java:998)
    at com.google.cloud.dataflow.worker.StreamingDataflowWorker.access$800(StreamingDataflowWorker.java:133)
    at com.google.cloud.dataflow.worker.StreamingDataflowWorker$7.run(StreamingDataflowWorker.java:771)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:745)

This seems to suggest that for some reason the native library is not being loaded correctly. Turning up the logging some more revealed:

java.lang.UnsatisfiedLinkError: io.netty.internal.tcnative.Library.aprMajorVersion()I
    at io.netty.internal.tcnative.Library.aprMajorVersion(Native Method)
    at io.netty.internal.tcnative.Library.initialize(Library.java:123)
    at io.netty.internal.tcnative.Library.initialize(Library.java:108)
    at io.netty.handler.ssl.OpenSsl.initializeTcNative(OpenSsl.java:408)
    at io.netty.handler.ssl.OpenSsl.<clinit>(OpenSsl.java:96)
    at io.grpc.netty.GrpcSslContexts.defaultSslProvider(GrpcSslContexts.java:143)
    at io.grpc.netty.GrpcSslContexts.configure(GrpcSslContexts.java:124)
    at io.grpc.netty.GrpcSslContexts.forClient(GrpcSslContexts.java:94)
    at com.brightcove.rna.model.Views.viewRegistryChannel(Views.java:21)
    at com.brightcove.rna.tools.ViewRegistryCacheLoader.createNewViewRegistryConnection(ViewRegistryCacheLoader.java:88)
    at com.brightcove.rna.tools.ViewRegistryCacheLoader.getClient(ViewRegistryCacheLoader.java:81)
    at com.brightcove.rna.tools.ViewRegistryCacheLoader.getViewsFromRegistry(ViewRegistryCacheLoader.java:97)
    at com.brightcove.rna.tools.ViewRegistryCacheLoader.load(ViewRegistryCacheLoader.java:65)
    at com.brightcove.rna.tools.ViewRegistryCacheLoader.load(ViewRegistryCacheLoader.java:32)
    at com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3716)
    at com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2424)
    at com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2298)
    at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2211)
    at com.google.common.cache.LocalCache.get(LocalCache.java:4154)
    at com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:4158)
    at com.google.common.cache.LocalCache$LocalLoadingCache.get(LocalCache.java:5147)
    at com.brightcove.rna.transforms.ContraService.get(ContraService.java:45)
    at com.brightcove.rna.transforms.functions.ViewMaterializerDoFn.startBundle(ViewMaterializerDoFn.java:63)
    at com.brightcove.rna.transforms.functions.ViewMaterializerDoFn$DoFnInvoker.invokeStartBundle(Unknown Source)
    at org.apache.beam.runners.core.SimpleDoFnRunner.startBundle(SimpleDoFnRunner.java:127)
    at com.google.cloud.dataflow.worker.SimpleParDoFn.reallyStartBundle(SimpleParDoFn.java:212)
    at com.google.cloud.dataflow.worker.SimpleParDoFn.processElement(SimpleParDoFn.java:222)
    at com.google.cloud.dataflow.worker.util.common.worker.ParDoOperation.process(ParDoOperation.java:48)
    at com.google.cloud.dataflow.worker.util.common.worker.OutputReceiver.process(OutputReceiver.java:52)
    at com.google.cloud.dataflow.worker.AssignWindowsParDoFnFactory$AssignWindowsParDoFn.processElement(AssignWindowsParDoFnFactory.java:116)
    at com.google.cloud.dataflow.worker.util.common.worker.ParDoOperation.process(ParDoOperation.java:48)
    at com.google.cloud.dataflow.worker.util.common.worker.OutputReceiver.process(OutputReceiver.java:52)
    at com.google.cloud.dataflow.worker.SimpleParDoFn$1.output(SimpleParDoFn.java:183)
    at org.apache.beam.runners.core.SimpleDoFnRunner.outputWindowedValue(SimpleDoFnRunner.java:211)
    at org.apache.beam.runners.core.SimpleDoFnRunner.access$700(SimpleDoFnRunner.java:66)
    at org.apache.beam.runners.core.SimpleDoFnRunner$DoFnProcessContext.output(SimpleDoFnRunner.java:436)
    at org.apache.beam.runners.core.SimpleDoFnRunner$DoFnProcessContext.output(SimpleDoFnRunner.java:424)
    at com.brightcove.rna.transforms.streaming.StreamingAdditions$MeasurementToRow.processElement(StreamingAdditions.java:38)
    at com.brightcove.rna.transforms.streaming.StreamingAdditions$MeasurementToRow$DoFnInvoker.invokeProcessElement(Unknown Source)
    at org.apache.beam.runners.core.SimpleDoFnRunner.invokeProcessElement(SimpleDoFnRunner.java:177)
    at org.apache.beam.runners.core.SimpleDoFnRunner.processElement(SimpleDoFnRunner.java:141)
    at com.google.cloud.dataflow.worker.SimpleParDoFn.processElement(SimpleParDoFn.java:233)
    at com.google.cloud.dataflow.worker.util.common.worker.ParDoOperation.process(ParDoOperation.java:48)
    at com.google.cloud.dataflow.worker.util.common.worker.OutputReceiver.process(OutputReceiver.java:52)
    at com.google.cloud.dataflow.worker.SimpleParDoFn$1.output(SimpleParDoFn.java:183)
    at org.apache.beam.runners.core.SimpleDoFnRunner.outputWindowedValue(SimpleDoFnRunner.java:211)
    at org.apache.beam.runners.core.SimpleDoFnRunner.access$700(SimpleDoFnRunner.java:66)
    at org.apache.beam.runners.core.SimpleDoFnRunner$DoFnProcessContext.output(SimpleDoFnRunner.java:436)
    at org.apache.beam.runners.core.SimpleDoFnRunner$DoFnProcessContext.output(SimpleDoFnRunner.java:424)
    at com.brightcove.rna.transforms.functions.StripHighCardinalityDimensionsFn.processElement(StripHighCardinalityDimensionsFn.java:47)
    at com.brightcove.rna.transforms.functions.StripHighCardinalityDimensionsFn$DoFnInvoker.invokeProcessElement(Unknown Source)
    at org.apache.beam.runners.core.SimpleDoFnRunner.invokeProcessElement(SimpleDoFnRunner.java:177)
    at org.apache.beam.runners.core.SimpleDoFnRunner.processElement(SimpleDoFnRunner.java:141)
    at com.google.cloud.dataflow.worker.SimpleParDoFn.processElement(SimpleParDoFn.java:233)
    at com.google.cloud.dataflow.worker.util.common.worker.ParDoOperation.process(ParDoOperation.java:48)
    at com.google.cloud.dataflow.worker.util.common.worker.OutputReceiver.process(OutputReceiver.java:52)
    at com.google.cloud.dataflow.worker.SimpleParDoFn$1.output(SimpleParDoFn.java:183)
    at org.apache.beam.runners.core.SimpleDoFnRunner.outputWindowedValue(SimpleDoFnRunner.java:211)
    at org.apache.beam.runners.core.SimpleDoFnRunner.access$700(SimpleDoFnRunner.java:66)
    at org.apache.beam.runners.core.SimpleDoFnRunner$DoFnProcessContext.outputWithTimestamp(SimpleDoFnRunner.java:443)
    at org.apache.beam.runners.core.SimpleDoFnRunner$DoFnProcessContext.outputWithTimestamp(SimpleDoFnRunner.java:430)
    at com.brightcove.rna.transforms.functions.ReadCollectorMeasurements.processElement(ReadCollectorMeasurements.java:41)
    at com.brightcove.rna.transforms.functions.ReadCollectorMeasurements$DoFnInvoker.invokeProcessElement(Unknown Source)
    at org.apache.beam.runners.core.SimpleDoFnRunner.invokeProcessElement(SimpleDoFnRunner.java:177)
    at org.apache.beam.runners.core.SimpleDoFnRunner.processElement(SimpleDoFnRunner.java:141)
    at com.google.cloud.dataflow.worker.SimpleParDoFn.processElement(SimpleParDoFn.java:233)
    at com.google.cloud.dataflow.worker.util.common.worker.ParDoOperation.process(ParDoOperation.java:48)
    at com.google.cloud.dataflow.worker.util.common.worker.OutputReceiver.process(OutputReceiver.java:52)
    at com.google.cloud.dataflow.worker.SimpleParDoFn$1.output(SimpleParDoFn.java:183)
    at org.apache.beam.runners.core.SimpleDoFnRunner.outputWindowedValue(SimpleDoFnRunner.java:211)
    at org.apache.beam.runners.core.SimpleDoFnRunner.access$700(SimpleDoFnRunner.java:66)
    at org.apache.beam.runners.core.SimpleDoFnRunner$DoFnProcessContext.output(SimpleDoFnRunner.java:436)
    at org.apache.beam.runners.core.SimpleDoFnRunner$DoFnProcessContext.output(SimpleDoFnRunner.java:424)
    at com.brightcove.rna.pipeline.AnalyticsPipeline$1.processElement(AnalyticsPipeline.java:103)
    at com.brightcove.rna.pipeline.AnalyticsPipeline$1$DoFnInvoker.invokeProcessElement(Unknown Source)
    at org.apache.beam.runners.core.SimpleDoFnRunner.invokeProcessElement(SimpleDoFnRunner.java:177)
    at org.apache.beam.runners.core.SimpleDoFnRunner.processElement(SimpleDoFnRunner.java:141)
    at com.google.cloud.dataflow.worker.SimpleParDoFn.processElement(SimpleParDoFn.java:233)
    at com.google.cloud.dataflow.worker.util.common.worker.ParDoOperation.process(ParDoOperation.java:48)
    at com.google.cloud.dataflow.worker.util.common.worker.OutputReceiver.process(OutputReceiver.java:52)
    at com.google.cloud.dataflow.worker.SimpleParDoFn$1.output(SimpleParDoFn.java:183)
    at org.apache.beam.runners.core.SimpleDoFnRunner.outputWindowedValue(SimpleDoFnRunner.java:211)
    at org.apache.beam.runners.core.SimpleDoFnRunner.access$700(SimpleDoFnRunner.java:66)
    at org.apache.beam.runners.core.SimpleDoFnRunner$DoFnProcessContext.output(SimpleDoFnRunner.java:436)
    at org.apache.beam.runners.core.SimpleDoFnRunner$DoFnProcessContext.output(SimpleDoFnRunner.java:424)
    at org.apache.beam.sdk.transforms.MapElements$1.processElement(MapElements.java:122)
    at org.apache.beam.sdk.transforms.MapElements$1$DoFnInvoker.invokeProcessElement(Unknown Source)
    at org.apache.beam.runners.core.SimpleDoFnRunner.invokeProcessElement(SimpleDoFnRunner.java:177)
    at org.apache.beam.runners.core.SimpleDoFnRunner.processElement(SimpleDoFnRunner.java:141)
    at com.google.cloud.dataflow.worker.SimpleParDoFn.processElement(SimpleParDoFn.java:233)
    at com.google.cloud.dataflow.worker.util.common.worker.ParDoOperation.process(ParDoOperation.java:48)
    at com.google.cloud.dataflow.worker.util.common.worker.OutputReceiver.process(OutputReceiver.java:52)
    at com.google.cloud.dataflow.worker.util.common.worker.ReadOperation.runReadLoop(ReadOperation.java:187)
    at com.google.cloud.dataflow.worker.util.common.worker.ReadOperation.start(ReadOperation.java:148)
    at com.google.cloud.dataflow.worker.util.common.worker.MapTaskExecutor.execute(MapTaskExecutor.java:68)
    at com.google.cloud.dataflow.worker.StreamingDataflowWorker.process(StreamingDataflowWorker.java:998)
    at com.google.cloud.dataflow.worker.StreamingDataflowWorker.access$800(StreamingDataflowWorker.java:133)
    at com.google.cloud.dataflow.worker.StreamingDataflowWorker$7.run(StreamingDataflowWorker.java:771)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:745)

Which seems to suggest some problem outside the scope of the sdk. Any suggestions on how to work around this?

lukecwik commented 7 years ago

Use the same version of gRPC as the version of Dataflow that you are using has defined.

ankurcha commented 7 years ago

I am trying to use bigtable client 1.0.0-pre3 (in addition to some internal grpc services) which is compiled against grpc-java 1.5.0 is there a suggested way i can achieve using that with the dataflow sdk?

lukecwik commented 7 years ago

Unfortunately there are gRPC/protobuf version incompatibilities between the version you want to use and the version that Dataflow is using like this ALPN/NPN issue your running into. You might have luck trying other versions of BoringSSL or using one of the other alternative but unsupported ALPN/NPN providers for Java as well.

ankurcha commented 7 years ago

What would it take to change the version for grpc in the beam sdk dependency? Are we talking about incompatibilities in just the compile code level or something more on the logical side?

lukecwik commented 7 years ago

The Dataflow worker also uses the same version of protobuf/gRPC as Apache Beam, so even if it changed externally there would need to be the same corresponding change internally.

Unfortunately it is very hard to say whether it will work for everyone because what we have seen in the past is that generated code is version locked to the same version the protoc plugin expects so updating it breaks with runtime exceptions of UnsatisfiedLinkError or MethodNotFoundError or similar like errors. This is a better question for gRPC / protobuf folks as to the risk of using a newer gRPC/protobuf library with code that used an older version or protoc for code generation.

ankurcha commented 6 years ago

Seems like shading the tcnative, grpc and protobuf dependencies + setting the shaded.io.netty.packagePrefix property is the way to get things to work.