search

spring-attic / spring-cloud-gcp

Integration for Google Cloud Platform APIs with Spring
Apache License 2.0
704 stars 693 forks source link

ClassNotFoundException on CloudResourceManager$Builder #442

Closed yunus89 closed 6 years ago

yunus89 commented 6 years ago

In a spring boot application running google cloud dataflow code. The dataflow takes data from google PubSub, transform incoming data and output result to bigquery for storage. The code does not have any syntax errors. The problem is when the application is run, the following exception is thrown. 

Exception in thread "main" java.lang.RuntimeException: Failed to construct instance from factory method DataflowRunner#fromOptions(interface org.apache.beam.sdk.options.PipelineOptions)
at org.apache.beam.sdk.util.InstanceBuilder.buildFromMethod(InstanceBuilder.java:233)
at org.apache.beam.sdk.util.InstanceBuilder.build(InstanceBuilder.java:162)
at org.apache.beam.sdk.PipelineRunner.fromOptions(PipelineRunner.java:52)
at org.apache.beam.sdk.Pipeline.create(Pipeline.java:142)
at com.trackers.exlon.ExlonApplication.main(ExlonApplication.java:69)

Caused by: java.lang.reflect.InvocationTargetException
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.beam.sdk.util.InstanceBuilder.buildFromMethod(InstanceBuilder.java:222)
... 4 more

Caused by: java.lang.NoClassDefFoundError: com/google/api/services/cloudresourcemanager/CloudResourceManager$Builder
at org.apache.beam.sdk.extensions.gcp.options.GcpOptions$GcpTempLocationFactory.newCloudResourceManagerClient(GcpOptions.java:369)
at org.apache.beam.sdk.extensions.gcp.options.GcpOptions$GcpTempLocationFactory.create(GcpOptions.java:240)
at org.apache.beam.sdk.extensions.gcp.options.GcpOptions$GcpTempLocationFactory.create(GcpOptions.java:228)
at org.apache.beam.sdk.options.ProxyInvocationHandler.returnDefaultHelper(ProxyInvocationHandler.java:592)
at org.apache.beam.sdk.options.ProxyInvocationHandler.getDefault(ProxyInvocationHandler.java:533)
at org.apache.beam.sdk.options.ProxyInvocationHandler.invoke(ProxyInvocationHandler.java:156)
at com.sun.proxy.$Proxy85.getGcpTempLocation(Unknown Source)
at org.apache.beam.runners.dataflow.DataflowRunner.fromOptions(DataflowRunner.java:223)
... 9 more

Caused by: java.lang.ClassNotFoundException: com.google.api.services.cloudresourcemanager.CloudResourceManager$Builder
at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
... 17 more

Maven dependency tree:

[INFO] +- org.springframework.boot:spring-boot-starter-webflux:jar:2.0.0.RC1:compile
[INFO] |  +- org.springframework.boot:spring-boot-starter:jar:2.0.0.RC1:compile
[INFO] |  |  +- org.springframework.boot:spring-boot:jar:2.0.0.RC1:compile
[INFO] |  |  |  \- org.springframework:spring-context:jar:5.0.3.RELEASE:compile
[INFO] |  |  |     +- org.springframework:spring-aop:jar:5.0.3.RELEASE:compile
[INFO] |  |  |     \- org.springframework:spring-expression:jar:5.0.3.RELEASE:compile
[INFO] |  |  +- org.springframework.boot:spring-boot-autoconfigure:jar:2.0.0.RC1:compile
[INFO] |  |  +- org.springframework.boot:spring-boot-starter-logging:jar:2.0.0.RC1:compile
[INFO] |  |  |  +- ch.qos.logback:logback-classic:jar:1.2.3:compile
[INFO] |  |  |  |  \- ch.qos.logback:logback-core:jar:1.2.3:compile
[INFO] |  |  |  +- org.apache.logging.log4j:log4j-to-slf4j:jar:2.10.0:compile
[INFO] |  |  |  |  \- org.apache.logging.log4j:log4j-api:jar:2.10.0:compile
[INFO] |  |  |  \- org.slf4j:jul-to-slf4j:jar:1.7.25:compile
[INFO] |  |  +- javax.annotation:javax.annotation-api:jar:1.3.1:compile
[INFO] |  |  \- org.yaml:snakeyaml:jar:1.19:runtime
[INFO] |  +- org.springframework.boot:spring-boot-starter-json:jar:2.0.0.RC1:compile
[INFO] |  |  +- com.fasterxml.jackson.datatype:jackson-datatype-jdk8:jar:2.9.2:compile
[INFO] |  |  +- com.fasterxml.jackson.datatype:jackson-datatype-jsr310:jar:2.9.2:compile
[INFO] |  |  \- com.fasterxml.jackson.module:jackson-module-parameter-names:jar:2.9.2:compile
[INFO] |  +- org.springframework.boot:spring-boot-starter-reactor-netty:jar:2.0.0.RC1:compile
[INFO] |  |  \- io.projectreactor.ipc:reactor-netty:jar:0.7.3.RELEASE:compile
[INFO] |  |     \- io.netty:netty-transport-native-epoll:jar:4.1.20.Final:compile
[INFO] |  |        +- io.netty:netty-common:jar:4.1.20.Final:compile
[INFO] |  |        \- io.netty:netty-transport-native-unix-common:jar:4.1.20.Final:compile
[INFO] |  +- org.hibernate.validator:hibernate-validator:jar:6.0.7.Final:compile
[INFO] |  |  +- javax.validation:validation-api:jar:2.0.1.Final:compile
[INFO] |  |  +- org.jboss.logging:jboss-logging:jar:3.3.1.Final:compile
[INFO] |  |  \- com.fasterxml:classmate:jar:1.3.4:compile
[INFO] |  +- org.springframework:spring-web:jar:5.0.3.RELEASE:compile
[INFO] |  |  \- org.springframework:spring-beans:jar:5.0.3.RELEASE:compile
[INFO] |  +- org.springframework:spring-webflux:jar:5.0.3.RELEASE:compile
[INFO] |  \- org.synchronoss.cloud:nio-multipart-parser:jar:1.1.0:compile
[INFO] |     \- org.synchronoss.cloud:nio-stream-storage:jar:1.1.3:compile
[INFO] +- org.springframework.cloud:spring-cloud-gcp-starter-pubsub:jar:1.0.0.BUILD-SNAPSHOT:compile
[INFO] |  +- org.springframework.cloud:spring-cloud-gcp-starter:jar:1.0.0.BUILD-SNAPSHOT:compile
[INFO] |  |  +- org.springframework.cloud:spring-cloud-gcp-core:jar:1.0.0.BUILD-SNAPSHOT:compile
[INFO] |  |  \- org.springframework.cloud:spring-cloud-gcp-autoconfigure:jar:1.0.0.BUILD-SNAPSHOT:compile
[INFO] |  +- org.springframework.cloud:spring-cloud-gcp-pubsub:jar:1.0.0.BUILD-SNAPSHOT:compile
[INFO] |  |  +- com.google.cloud:google-cloud-pubsub:jar:0.33.0-beta:compile
[INFO] |  |  \- org.springframework:spring-messaging:jar:5.0.3.RELEASE:compile
[INFO] |  \- org.slf4j:slf4j-api:jar:1.7.25:compile
[INFO] +- org.apache.beam:beam-sdks-java-core:jar:2.2.0:compile
[INFO] |  +- com.google.code.findbugs:jsr305:jar:3.0.1:compile
[INFO] |  +- com.fasterxml.jackson.core:jackson-core:jar:2.9.2:compile
[INFO] |  +- com.fasterxml.jackson.core:jackson-annotations:jar:2.9.0:compile
[INFO] |  +- com.fasterxml.jackson.core:jackson-databind:jar:2.9.2:compile
[INFO] |  +- org.apache.avro:avro:jar:1.8.2:compile
[INFO] |  |  +- org.codehaus.jackson:jackson-core-asl:jar:1.9.13:compile
[INFO] |  |  +- org.codehaus.jackson:jackson-mapper-asl:jar:1.9.13:compile
[INFO] |  |  +- com.thoughtworks.paranamer:paranamer:jar:2.7:compile
[INFO] |  |  +- org.apache.commons:commons-compress:jar:1.8.1:compile
[INFO] |  |  \- org.tukaani:xz:jar:1.5:compile
[INFO] |  +- org.xerial.snappy:snappy-java:jar:1.1.4:compile
[INFO] |  \- joda-time:joda-time:jar:2.9.9:compile
[INFO] +- com.google.cloud:google-cloud-bigquery:jar:0.33.0-beta:compile
[INFO] |  +- com.google.cloud:google-cloud-core:jar:1.15.0:compile
[INFO] |  |  +- com.google.api:api-common:jar:1.2.0:compile
[INFO] |  |  +- com.google.api:gax:jar:1.16.0:compile
[INFO] |  |  |  +- com.google.auto.value:auto-value:jar:1.2:compile
[INFO] |  |  |  \- org.threeten:threetenbp:jar:1.3.3:compile
[INFO] |  |  +- com.google.protobuf:protobuf-java-util:jar:3.5.1:compile
[INFO] |  |  |  \- com.google.code.gson:gson:jar:2.8.2:compile
[INFO] |  |  +- com.google.api.grpc:proto-google-common-protos:jar:1.0.4:compile
[INFO] |  |  \- com.google.api.grpc:proto-google-iam-v1:jar:0.1.28:compile
[INFO] |  +- com.google.cloud:google-cloud-core-http:jar:1.15.0:compile
[INFO] |  |  +- com.google.oauth-client:google-oauth-client:jar:1.23.0:compile
[INFO] |  |  +- com.google.http-client:google-http-client-appengine:jar:1.23.0:compile
[INFO] |  |  +- com.google.http-client:google-http-client-jackson:jar:1.23.0:compile
[INFO] |  |  \- com.google.api:gax-httpjson:jar:0.33.0:compile
[INFO] |  \- com.google.apis:google-api-services-bigquery:jar:v2-rev347-1.22.0:compile
[INFO] +- io.grpc:grpc-core:jar:1.9.0:compile
[INFO] |  +- io.grpc:grpc-context:jar:1.9.0:compile
[INFO] |  +- com.google.guava:guava:jar:20.0:compile
[INFO] |  +- com.google.errorprone:error_prone_annotations:jar:2.1.2:compile
[INFO] |  +- com.google.instrumentation:instrumentation-api:jar:0.4.3:compile
[INFO] |  +- io.opencensus:opencensus-api:jar:0.10.0:compile
[INFO] |  \- io.opencensus:opencensus-contrib-grpc-metrics:jar:0.10.0:compile
[INFO] +- io.grpc:grpc-netty:jar:1.9.0:compile
[INFO] |  +- io.netty:netty-codec-http2:jar:4.1.20.Final:compile
[INFO] |  |  +- io.netty:netty-codec-http:jar:4.1.20.Final:compile
[INFO] |  |  |  \- io.netty:netty-codec:jar:4.1.20.Final:compile
[INFO] |  |  \- io.netty:netty-handler:jar:4.1.20.Final:compile
[INFO] |  |     \- io.netty:netty-buffer:jar:4.1.20.Final:compile
[INFO] |  \- io.netty:netty-handler-proxy:jar:4.1.20.Final:compile
[INFO] |     +- io.netty:netty-transport:jar:4.1.20.Final:compile
[INFO] |     |  \- io.netty:netty-resolver:jar:4.1.20.Final:compile
[INFO] |     \- io.netty:netty-codec-socks:jar:4.1.20.Final:compile
[INFO] +- org.apache.beam:beam-runners-google-cloud-dataflow-java:jar:2.2.0:compile
[INFO] |  +- org.apache.beam:beam-sdks-java-extensions-google-cloud-platform-core:jar:2.2.0:compile
[INFO] |  |  +- com.google.cloud.bigdataoss:gcsio:jar:1.4.5:compile
[INFO] |  |  \- com.google.apis:google-api-services-cloudresourcemanager:jar:v1beta1-rev10-1.21.0:compile
[INFO] |  +- org.apache.beam:beam-sdks-common-runner-api:jar:2.2.0:compile
[INFO] |  |  +- com.google.protobuf:protobuf-java:jar:3.5.1:compile
[INFO] |  |  +- io.grpc:grpc-protobuf:jar:1.9.0:compile
[INFO] |  |  |  \- io.grpc:grpc-protobuf-lite:jar:1.9.0:compile
[INFO] |  |  \- io.grpc:grpc-stub:jar:1.9.0:compile
[INFO] |  +- org.apache.beam:beam-sdks-java-io-google-cloud-platform:jar:2.2.0:compile
[INFO] |  |  +- org.apache.beam:beam-sdks-java-extensions-protobuf:jar:2.2.0:compile
[INFO] |  |  +- com.google.api:gax-grpc:jar:1.16.0:compile
[INFO] |  |  +- com.google.cloud:google-cloud-core-grpc:jar:1.15.0:compile
[INFO] |  |  +- com.google.apis:google-api-services-pubsub:jar:v1-rev10-1.22.0:compile
[INFO] |  |  +- com.google.api.grpc:grpc-google-cloud-pubsub-v1:jar:0.1.28:compile
[INFO] |  |  +- com.google.api.grpc:proto-google-cloud-pubsub-v1:jar:0.1.28:compile
[INFO] |  |  +- com.google.cloud.datastore:datastore-v1-proto-client:jar:1.6.0:compile
[INFO] |  |  |  +- com.google.api.grpc:proto-google-cloud-datastore-v1:jar:0.1.28:compile
[INFO] |  |  |  \- com.google.http-client:google-http-client-protobuf:jar:1.20.0:compile
[INFO] |  |  +- com.google.cloud.datastore:datastore-v1-protos:jar:1.3.0:compile
[INFO] |  |  |  \- com.google.api.grpc:grpc-google-common-protos:jar:1.0.4:compile
[INFO] |  |  +- io.grpc:grpc-auth:jar:1.9.0:compile
[INFO] |  |  +- io.grpc:grpc-all:jar:1.2.0:runtime
[INFO] |  |  |  +- io.grpc:grpc-okhttp:jar:1.2.0:runtime
[INFO] |  |  |  |  +- com.squareup.okhttp:okhttp:jar:2.5.0:runtime
[INFO] |  |  |  |  \- com.squareup.okio:okio:jar:1.6.0:runtime
[INFO] |  |  |  \- io.grpc:grpc-protobuf-nano:jar:1.2.0:runtime
[INFO] |  |  |     \- com.google.protobuf.nano:protobuf-javanano:jar:3.0.0-alpha-5:runtime
[INFO] |  |  +- com.google.cloud:google-cloud-spanner:jar:0.33.0-beta:compile
[INFO] |  |  |  +- com.google.api.grpc:proto-google-cloud-spanner-v1:jar:0.1.28:compile
[INFO] |  |  |  +- com.google.api.grpc:proto-google-cloud-spanner-admin-instance-v1:jar:0.1.28:compile
[INFO] |  |  |  +- com.google.api.grpc:grpc-google-cloud-spanner-v1:jar:0.1.28:compile
[INFO] |  |  |  +- com.google.api.grpc:grpc-google-cloud-spanner-admin-database-v1:jar:0.1.28:compile
[INFO] |  |  |  +- com.google.api.grpc:grpc-google-cloud-spanner-admin-instance-v1:jar:0.1.28:compile
[INFO] |  |  |  \- io.opencensus:opencensus-contrib-grpc-util:jar:0.10.0:compile
[INFO] |  |  +- com.google.cloud.bigtable:bigtable-protos:jar:1.0.0-pre3:compile
[INFO] |  |  +- com.google.cloud.bigtable:bigtable-client-core:jar:1.0.0-pre3:compile
[INFO] |  |  |  +- commons-logging:commons-logging:jar:1.2:compile
[INFO] |  |  |  +- com.google.auth:google-auth-library-appengine:jar:0.7.0:compile
[INFO] |  |  |  \- io.dropwizard.metrics:metrics-core:jar:3.2.6:compile
[INFO] |  |  +- io.netty:netty-tcnative-boringssl-static:jar:2.0.7.Final:compile
[INFO] |  |  \- com.google.api.grpc:proto-google-cloud-spanner-admin-database-v1:jar:0.1.28:compile
[INFO] |  +- com.google.api-client:google-api-client:jar:1.23.0:compile
[INFO] |  +- com.google.http-client:google-http-client:jar:1.23.0:compile
[INFO] |  |  \- org.apache.httpcomponents:httpclient:jar:4.5.5:compile
[INFO] |  |     +- org.apache.httpcomponents:httpcore:jar:4.4.9:compile
[INFO] |  |     \- commons-codec:commons-codec:jar:1.11:compile
[INFO] |  +- com.google.http-client:google-http-client-jackson2:jar:1.23.0:compile
[INFO] |  +- com.google.apis:google-api-services-dataflow:jar:v1b3-rev213-1.22.0:compile
[INFO] |  +- com.google.apis:google-api-services-clouddebugger:jar:v2-rev8-1.22.0:compile
[INFO] |  +- com.google.apis:google-api-services-storage:jar:v1-rev114-1.23.0:compile
[INFO] |  +- com.google.auth:google-auth-library-credentials:jar:0.9.0:compile
[INFO] |  +- com.google.auth:google-auth-library-oauth2-http:jar:0.7.1:compile
[INFO] |  \- com.google.cloud.bigdataoss:util:jar:1.4.5:compile
[INFO] |     +- com.google.api-client:google-api-client-java6:jar:1.20.0:compile
[INFO] |     +- com.google.api-client:google-api-client-jackson2:jar:1.20.0:compile
[INFO] |     \- com.google.oauth-client:google-oauth-client-java6:jar:1.20.0:compile
[INFO] +- net.sf.marineapi:marineapi:jar:0.10.0:compile
[INFO] +- org.projectlombok:lombok:jar:1.16.20:compile (optional)
[INFO] +- org.springframework.boot:spring-boot-starter-test:jar:2.0.0.RC1:test
[INFO] |  +- org.springframework.boot:spring-boot-test:jar:2.0.0.RC1:test
[INFO] |  +- org.springframework.boot:spring-boot-test-autoconfigure:jar:2.0.0.RC1:test
[INFO] |  +- com.jayway.jsonpath:json-path:jar:2.4.0:test
[INFO] |  |  \- net.minidev:json-smart:jar:2.3:test
[INFO] |  |     \- net.minidev:accessors-smart:jar:1.2:test
[INFO] |  |        \- org.ow2.asm:asm:jar:5.0.4:test
[INFO] |  +- junit:junit:jar:4.12:compile
[INFO] |  +- org.assertj:assertj-core:jar:3.9.0:test
[INFO] |  +- org.mockito:mockito-core:jar:2.13.0:test
[INFO] |  |  +- net.bytebuddy:byte-buddy:jar:1.7.9:test
[INFO] |  |  +- net.bytebuddy:byte-buddy-agent:jar:1.7.9:test
[INFO] |  |  \- org.objenesis:objenesis:jar:2.6:test
[INFO] |  +- org.hamcrest:hamcrest-core:jar:1.3:compile
[INFO] |  +- org.hamcrest:hamcrest-library:jar:1.3:test
[INFO] |  +- org.skyscreamer:jsonassert:jar:1.5.0:test
[INFO] |  |  \- com.vaadin.external.google:android-json:jar:0.0.20131108.vaadin1:test
[INFO] |  +- org.springframework:spring-core:jar:5.0.3.RELEASE:compile
[INFO] |  |  \- org.springframework:spring-jcl:jar:5.0.3.RELEASE:compile
[INFO] |  +- org.springframework:spring-test:jar:5.0.3.RELEASE:test
[INFO] |  \- org.xmlunit:xmlunit-core:jar:2.5.1:test
[INFO] \- io.projectreactor:reactor-test:jar:3.1.3.RELEASE:test
[INFO]    \- io.projectreactor:reactor-core:jar:3.1.3.RELEASE:compile
[INFO]       \- org.reactivestreams:reactive-streams:jar:1.0.2:compile
artembilan commented 6 years ago

You have this stack trace:

Caused by: java.lang.NoClassDefFoundError: com/google/api/services/cloudresourcemanager/CloudResourceManager$Builder
at org.apache.beam.sdk.extensions.gcp.options.GcpOptions$GcpTempLocationFactory.newCloudResourceManagerClient(GcpOptions.java:369)
at org.apache.beam.sdk.extensions.gcp.options.GcpOptions$GcpTempLocationFactory.create(GcpOptions.java:240)
at org.apache.beam.sdk.extensions.gcp.options.GcpOptions$GcpTempLocationFactory.cre

And according your dependencies you have this:

org.apache.beam:beam-runners-google-cloud-dataflow-java:jar:2.2.0:compile
[INFO] |  +- org.apache.beam:beam-sdks-java-extensions-google-cloud-platform-core:jar:2.2.0:compile
[INFO] |  |  +- com.google.cloud.bigdataoss:gcsio:jar:1.4.5:compile
[INFO] |  |  \- com.google.apis:google-api-services-cloudresourcemanager:jar:v1beta1-rev10-1.21.0:compile

So, technically you have everything to make it work, but looks like there is some versions incompatibility.

This project doesn’t provide (yet) any DataFlow Integration . Please consider to move your question to Apache Beam project. Or really prove that exactly this Spring Cloud GCP causes the problem and point us, please, what is wrong.

Thank you for understanding

yunus89 commented 6 years ago

Apache Beam project has no Issues tab/new Issue area at Github, if there is anyone able to help, i will appreciate.

artembilan commented 6 years ago

There is Contact Us in their README: https://github.com/apache/beam/blob/master/README.md#contact-us and clear link to JIRA in that section:

Report issues on JIRA.

meltsufin commented 6 years ago

Note the same post on StackOverflow.

meltsufin commented 6 years ago

Try the latest version of the GCP beam runner that was released on Feb 10.

<dependency>
    <groupId>org.apache.beam</groupId>
    <artifactId>beam-runners-google-cloud-dataflow-java</artifactId>
    <version>2.3.0</version>
</dependency>
yunus89 commented 6 years ago

I added this

    <dependency>
        <groupId>io.grpc</groupId>
        <artifactId>grpc-core</artifactId>
        <version>1.9.0</version>
    </dependency>

    <dependency>
        <groupId>io.grpc</groupId>
        <artifactId>grpc-netty</artifactId>
        <version>1.9.0</version>
    </dependency>

    <dependency>
        <groupId>org.apache.beam</groupId>
        <artifactId>beam-runners-google-cloud-dataflow-java</artifactId>
        <version>2.3.0</version>
        <exclusions>
            <exclusion>
                <groupId>io.grpc</groupId>
                <artifactId>grpc-core</artifactId>
            </exclusion>
            <exclusion>
                <groupId>io.grpc</groupId>
                <artifactId>grpc-netty</artifactId>
            </exclusion>
        </exclusions>
    </dependency>

Still the same exception is thrown

meltsufin commented 6 years ago

The reason you're seeing CloudResourceManager not found is because of the conflicting version of com.google.apis:google-api-services-cloudresourcemanager dependency. I see two versions of it in play v1beta1-rev10-1.21.0 and v1-rev6-1.22.0. The older version has the class Cloudresourcemanager and the newer version has CloudResourceManager. beam-runners-google-cloud-dataflow-java really needs the newer version. However, the older version is being managed by the Google Cloud Java Libraries BOM com.google.cloud:google-cloud-bom, which Spring Cloud GCP BOM uses. I will file an issue against Google Cloud Java Libraries, but in the meantime simply add the newer google-api-services-cloudresourcemanager to your POM.

        <dependency>
            <groupId>com.google.apis</groupId>
            <artifactId>google-api-services-cloudresourcemanager</artifactId>
            <version>v1-rev6-1.22.0</version>
        </dependency>
joaoandremartins commented 6 years ago

Please reopen if still an issue.