search

GoogleCloudPlatform / DataflowJavaSDK

Google Cloud Dataflow provides a simple, powerful model for building both batch and streaming parallel data processing pipelines.
http://cloud.google.com/dataflow
855 stars 324 forks source link

Unable to use google dataflow JAVA SDK 1.8 version with google-cloud-pubsub version 0.20.0-beta #584

Closed seehans closed 7 years ago

seehans commented 7 years ago

I am trying to initialize com.google.cloud.pubsub.v1.Publisher within the dataflow job.

I am getting following exception: java.lang.RuntimeException: java.lang.NoClassDefFoundError: Could not initialize class com.google.cloud.pubsub.v1.TopicAdminSettings

Stacktrace: exception: "java.lang.RuntimeException: java.lang.NoClassDefFoundError: Could not initialize class com.google.cloud.pubsub.v1.TopicAdminSettings at com.google.cloud.dataflow.sdk.runners.worker.SimpleParDoFn$1.output(SimpleParDoFn.java:162) at com.google.cloud.dataflow.sdk.util.DoFnRunnerBase$DoFnContext.outputWindowedValue(DoFnRunnerBase.java:288) at com.google.cloud.dataflow.sdk.util.DoFnRunnerBase$DoFnProcessContext.output(DoFnRunnerBase.java:450) at com.FileNameParser$BucketFileNameParser.processElement(FileNameParser.java:93) Caused by: java.lang.NoClassDefFoundError: Could not initialize class com.google.cloud.pubsub.v1.TopicAdminSettings at com.google.cloud.pubsub.v1.Publisher$Builder.<init>(Publisher.java:579) at com.google.cloud.pubsub.v1.Publisher$Builder.<init>(Publisher.java:528) at com.google.cloud.pubsub.v1.Publisher.defaultBuilder(Publisher.java:524) at com.ReadFile$ReadBucketFile.createPublisherWithCustomCredentials(ReadFile.java:176) at com.ReadFile$ReadBucketFile.publishMessage(ReadFile.java:144) at com.ReadFile$ReadBucketFile.processElement(ReadFile.java:118) at com.google.cloud.dataflow.sdk.util.SimpleDoFnRunner.invokeProcessElement(SimpleDoFnRunner.java:49) at com.google.cloud.dataflow.sdk.util.DoFnRunnerBase.processElement(DoFnRunnerBase.java:139) at com.google.cloud.dataflow.sdk.runners.worker.SimpleParDoFn.processElement(SimpleParDoFn.java:190) at com.google.cloud.dataflow.sdk.runners.worker.ForwardingParDoFn.processElement(ForwardingParDoFn.java:42) at com.google.cloud.dataflow.sdk.runners.worker.DataflowWorkerLoggingParDoFn.processElement(DataflowWorkerLoggingParDoFn.java:47) at com.google.cloud.dataflow.sdk.util.common.worker.ParDoOperation.process(ParDoOperation.java:55) at com.google.cloud.dataflow.sdk.util.common.worker.OutputReceiver.process(OutputReceiver.java:52) at com.google.cloud.dataflow.sdk.runners.worker.SimpleParDoFn$1.output(SimpleParDoFn.java:160) at com.google.cloud.dataflow.sdk.util.DoFnRunnerBase$DoFnContext.outputWindowedValue(DoFnRunnerBase.java:288) at com.google.cloud.dataflow.sdk.util.DoFnRunnerBase$DoFnProcessContext.output(DoFnRunnerBase.java:450) at com.FileNameParser$BucketFileNameParser.processElement(FileNameParser.java:93) at com.google.cloud.dataflow.sdk.util.SimpleDoFnRunner.invokeProcessElement(SimpleDoFnRunner.java:49) at com.google.cloud.dataflow.sdk.util.DoFnRunnerBase.processElement(DoFnRunnerBase.java:139) at com.google.cloud.dataflow.sdk.runners.worker.SimpleParDoFn.processElement(SimpleParDoFn.java:190) at com.google.cloud.dataflow.sdk.runners.worker.ForwardingParDoFn.processElement(ForwardingParDoFn.java:42) at com.google.cloud.dataflow.sdk.runners.worker.DataflowWorkerLoggingParDoFn.processElement(DataflowWorkerLoggingParDoFn.java:47) at com.google.cloud.dataflow.sdk.util.common.worker.ParDoOperation.process(ParDoOperation.java:55) at com.google.cloud.dataflow.sdk.util.common.worker.OutputReceiver.process(OutputReceiver.java:52) at com.google.cloud.dataflow.sdk.util.common.worker.ReadOperation.runReadLoop(ReadOperation.java:224) at com.google.cloud.dataflow.sdk.util.common.worker.ReadOperation.start(ReadOperation.java:185) at com.google.cloud.dataflow.sdk.util.common.worker.MapTaskExecutor.execute(MapTaskExecutor.java:72) at com.google.cloud.dataflow.sdk.runners.worker.StreamingDataflowWorker.process(StreamingDataflowWorker.java:710) at com.google.cloud.dataflow.sdk.runners.worker.StreamingDataflowWorker.access$500(StreamingDataflowWorker.java:99) at com.google.cloud.dataflow.sdk.runners.worker.StreamingDataflowWorker$8.run(StreamingDataflowWorker.java:799) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745)

Is any DataflowJavaSDK compatiable with Google Cloud Java Client for Pub/Sub (gRPC )?

lukecwik commented 7 years ago

1.6.0 is the first version that used the gRPC version of Pubsub: com.google.api.grpc » grpc-pubsub-v1 0.0.2

2.0.0 is the latest version and it uses: com.google.api.grpc » grpc-google-pubsub-v1 0.1.0

There is currently no version which uses 0.20.0-beta, you should use the same version of pubsub and same versions of transitive dependencies as the version of Dataflow that you want to use to prevent dependency mismatch issues (e.g. MethodNotFound/ClassNotFound).