logstash-plugins / logstash-input-google_pubsub

Logstash input for pulling events from Google Pub/Sub service
Apache License 2.0
19 stars 33 forks source link

java.lang.NoClassDefFoundError: org/threeten/bp/Duration when running with jdbc #59

Closed llermaly closed 2 years ago

llermaly commented 2 years ago

Logstash information:

Please include the following information:

  1. Logstash version : 8.5.2
  2. Logstash installation source: docker
  3. How is Logstash being run : k8s
  4. How was the Logstash Plugin installed: via docker image

Description of the problem including expected versus actual behavior:

We have 2 Logstash docker images, one that installs gcp_pubsub input plugin, and another one that copies jar files for jdbc_static filter.

Both images work separately, but when I combine both in one stops working. if I remove the gcp_pubsub input the image works correctly.

This is the error:

java.lang.NoClassDefFoundError: org/threeten/bp/Duration\n\tat com.google.api.gax.retrying.RetrySettings.newBuilder(com/google/api/gax/retrying/RetrySettings.java:146) ~[postgres-socket-factory-1.6.4-SNAPSHOT-jar-with-dependencies.jar:?]\n\tat com.google.cloud.pubsub.v1.stub.SubscriberStubSettings$Builder.<clinit>(com/google/cloud/pubsub/v1/stub/SubscriberStubSettings.java:560) ~[google-cloud-pubsub-1.120.13.jar:1.120.13]\n\tat com.google.cloud.pubsub.v1.stub.SubscriberStubSettings.newBuilder(com/google/cloud/pubsub/v1/stub/SubscriberStubSettings.java:453) ~[google-cloud-pubsub-1.120.13.jar:1.120.13]\n\tat com.google.cloud.pubsub.v1.Subscriber.<init>(com/google/cloud/pubsub/v1/Subscriber.java:189) ~[google-cloud-pubsub-1.120.13.jar:1.120.13]\n\tat com.google.cloud.pubsub.v1.Subscriber.<init>(com/google/cloud/pubsub/v1/Subscriber.java:91) ~[google-cloud-pubsub-1.120.13.jar:1.120.13]\n\tat com.google.cloud.pubsub.v1.Subscriber$Builder.build(com/google/cloud/pubsub/v1/Subscriber.java:707) ~[google-cloud-pubsub-1.120.13.jar:1.120.13]\n\tat jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:?]\n\tat jdk.internal.reflect.NativeMethodAccessorImpl.invoke(jdk/internal/reflect/NativeMethodAccessorImpl.java:77) ~[?:?]\n\tat jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(jdk/internal/reflect/DelegatingMethodAccessorImpl.java:43) ~[?:?]\n\tat java.lang.reflect.Method.invoke(java/lang/reflect/Method.java:568) ~[?:?]\n\tat org.jruby.javasupport.JavaMethod.invokeDirectWithExceptionHandling(org/jruby/javasupport/JavaMethod.java:442) ~[jruby.jar:?]\n\tat org.jruby.javasupport.JavaMethod.invokeDirect(org/jruby/javasupport/JavaMethod.java:306) ~[jruby.jar:?]\n\tat RUBY.run(/usr/share/logstash/vendor/bundle/jruby/2.6.0/gems/logstash-input-google_pubsub-1.2.2/lib/logstash/inputs/google_pubsub.rb:274) ~[?:?]\n\tat RUBY.inputworker(/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:411) ~[?:?]\n\tat RUBY.start_input(/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:402) ~[?:?]\n\tat org.jruby.RubyProc.call(org/jruby/RubyProc.java:309) ~[jruby.jar:?]\n\tat java.lang.Thread.run(java/lang/Thread.java:833) [?:?]\nCaused by: java.lang.ClassNotFoundException: org.threeten.bp.Duration\n\tat jdk.internal.loader.BuiltinClassLoader.loadClass(BuiltinClassLoader.java:641) ~[?:?]\n\tat jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(ClassLoaders.java:188) ~[?:?]\n\tat java.lang.ClassLoader.loadClass(ClassLoader.java:520) ~[?:?]\n\tat com.google.api.gax.retrying.RetrySettings.newBuilder(RetrySettings.java:146) ~[postgres-socket-factory-1.6.4-SNAPSHOT-jar-with-dependencies.jar:?]\n\tat com.google.cloud.pubsub.v1.stub.SubscriberStubSettings$Builder.<clinit>(SubscriberStubSettings.java:560) ~[google-cloud-pubsub-1.120.13.jar:1.120.13]\n\tat com.google.cloud.pubsub.v1.stub.SubscriberStubSettings.newBuilder(SubscriberStubSettings.java:453) ~[google-cloud-pubsub-1.120.13.jar:1.120.13]\n\tat com.google.cloud.pubsub.v1.Subscriber.<init>(Subscriber.java:189) ~[google-cloud-pubsub-1.120.13.jar:1.120.13]\n\tat com.google.cloud.pubsub.v1.Subscriber.<init>(Subscriber.java:91) ~[google-cloud-pubsub-1.120.13.jar:1.120.13]\n\tat com.google.cloud.pubsub.v1.Subscriber$Builder.build(Subscriber.java:707) ~[google-cloud-pubsub-1.120.13.jar:1.120.13]\n\tat jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:?]\n\tat jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77) ~[?:?]\n\tat jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:?]\n\tat java.lang.reflect.Method.invoke(Method.java:568) ~[?:?]\n\tat org.jruby.javasupport.JavaMethod.invokeDirectWithExceptionHandling(JavaMethod.java:442) ~[jruby.jar:?]\n\tat org.jruby.javasupport.JavaMethod.invokeDirect(JavaMethod.java:306) ~[jruby.jar:?]\n\tat org.jruby.java.invokers.InstanceMethodInvoker.call(InstanceMethodInvoker.java:32) ~[jruby.jar:?]\n\tat org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(CachingCallSite.java:351) ~[jruby.jar:?]\n\tat org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:144) ~[jruby.jar:?]\n\tat org.jruby.ir.interpreter.InterpreterEngine.processCall(InterpreterEngine.java:345) ~[jruby.jar:?]\n\tat org.jruby.ir.interpreter.StartupInterpreterEngine.interpret(StartupInterpreterEngine.java:72) ~[jruby.jar:?]\n\tat org.jruby.ir.interpreter.InterpreterEngine.interpret(InterpreterEngine.java:86) ~[jruby.jar:?]\n\tat org.jruby.internal.runtime.methods.MixedModeIRMethod.INTERPRET_METHOD(MixedModeIRMethod.java:201) ~[jruby.jar:?]\n\tat org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:188) ~[jruby.jar:?]\n\tat org.jruby.internal.runtime.methods.DynamicMethod.call(DynamicMethod.java:218) ~[jruby.jar:?]\n\tat org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(CachingCallSite.java:372) ~[jruby.jar:?]\n\tat org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:175) ~[jruby.jar:?]\n\tat org.jruby.ir.interpreter.InterpreterEngine.processCall(InterpreterEngine.java:316) ~[jruby.jar:?]\n\tat org.jruby.ir.interpreter.StartupInterpreterEngine.interpret(StartupInterpreterEngine.java:72) ~[jruby.jar:?]\n\tat org.jruby.ir.interpreter.InterpreterEngine.interpret(InterpreterEngine.java:86) ~[jruby.jar:?]\n\tat org.jruby.internal.runtime.methods.MixedModeIRMethod.INTERPRET_METHOD(MixedModeIRMethod.java:201) ~[jruby.jar:?]\n\tat org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:188) ~[jruby.jar:?]\n\tat org.jruby.internal.runtime.methods.DynamicMethod.call(DynamicMethod.java:218) ~[jruby.jar:?]\n\tat org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(CachingCallSite.java:372) ~[jruby.jar:?]\n\tat org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:175) ~[jruby.jar:?]\n\tat org.jruby.ir.interpreter.InterpreterEngine.processCall(InterpreterEngine.java:316) ~[jruby.jar:?]\n\tat org.jruby.ir.interpreter.StartupInterpreterEngine.interpret(StartupInterpreterEngine.java:72) ~[jruby.jar:?]\n\tat org.jruby.ir.interpreter.Interpreter.INTERPRET_BLOCK(Interpreter.java:116) ~[jruby.jar:?]\n\tat org.jruby.runtime.MixedModeIRBlockBody.commonYieldPath(MixedModeIRBlockBody.java:136) ~[jruby.jar:?]\n\tat org.jruby.runtime.IRBlockBody.call(IRBlockBody.java:66) ~[jruby.jar:?]\n\tat org.jruby.runtime.IRBlockBody.call(IRBlockBody.java:58) ~[jruby.jar:?]\n\tat org.jruby.runtime.Block.call(Block.java:143) ~[jruby.jar:?]\n\tat org.jruby.RubyProc.call(RubyProc.java:309) ~[jruby.jar:?]\n\tat org.jruby.internal.runtime.RubyRunnable.run(RubyRunnable.java:107) ~[jruby.jar:?]\n\tat java.lang.Thread.run(Thread.java:833) ~[?:?]

This is the dockerfile:

FROM docker.elastic.co/logstash/logstash:8.5.2
COPY postgres-socket-factory-1.6.4-SNAPSHOT-jar-with-dependencies.jar /usr/share/logstash/logstash-core/lib/jars/postgres-socket-factory-1.6.4-SNAPSHOT-jar-with-dependencies.jar
COPY postgres-socket-factory-1.6.4-SNAPSHOT.jar /usr/share/logstash/logstash-core/lib/jars/postgres-socket-factory-1.6.4-SNAPSHOT.jar
COPY postgresql-42.5.0.jar /usr/share/logstash/logstash-core/lib/jars/postgresql-42.5.0.jar
RUN bin/logstash-plugin install logstash-input-google_pubsub

Steps to reproduce:

Please include a minimal but complete recreation of the problem, including (e.g.) pipeline definition(s), settings, locale, etc. The easier you make for us to reproduce it, the more likely that somebody will take the time to look at it.

  1. Download/build the image
  2. Create a gcp pubsub input, and a jdbc_static filter
  3. Run

Example conf file:

input {
    google_pubsub {
        project_id => "cosmic-keep-148903"
        subscription => "slm-sub"
        topic => "slm"
    }
}

filter {
    jdbc_streaming {
        jdbc_driver_class => "org.postgresql.Driver"
        jdbc_connection_string => "jdbc:postgresql:///postgres?cloudSqlInstance=cosmic-keep-148903:us-central1:keybank&socketFactory=com.google.cloud.sql.postgres.SocketFactory&user=postgres&password=postgres"
        jdbc_user => "postgres"
        jdbc_password => "postgres"
        statement => "select * from people"
        target => "country_details"
    }
}

output {
    stdout {
        codec => rubydebug
    }
}

Provide logs (if relevant):

llermaly commented 2 years ago

Just found out the jar file I'm building for the socketfactory:

https://github.com/GoogleCloudPlatform/cloud-sql-jdbc-socket-factory

Includes some exclusions, one of them threeten. If I remove it then the error changes and complains for a different one. If I remove all the exclusions building fails.

 <dependency>
      <groupId>com.google.api</groupId>
      <artifactId>gax</artifactId>
      <version>2.19.5</version>
      <exclusions>
        <!-- Native image users only need GAX's native-image metadata. GAX's
        dependencies are unnecessary for normal users and native image users -->
        <exclusion>
          <groupId>com.google.api</groupId>
          <artifactId>api-common</artifactId>
        </exclusion>
        <exclusion>
          <groupId>com.google.api.grpc</groupId>
          <artifactId>proto-google-common-protos</artifactId>
        </exclusion>
        <exclusion>
          <groupId>com.google.auth</groupId>
          <artifactId>google-auth-library-credentials</artifactId>
        </exclusion>
        <exclusion>
          <groupId>com.google.auth</groupId>
          <artifactId>google-auth-library-oauth2-http</artifactId>
        </exclusion>
        <exclusion>
          <groupId>com.google.code.findbugs</groupId>
          <artifactId>jsr305</artifactId>
        </exclusion>
        <exclusion>
          <groupId>com.google.guava</groupId>
          <artifactId>guava</artifactId>
        </exclusion>
        <exclusion>
          <groupId>com.google.protobuf</groupId>
          <artifactId>protobuf-java</artifactId>
        </exclusion>
        <exclusion>
          <groupId>io.opencensus</groupId>
          <artifactId>opencensus-api</artifactId>
        </exclusion>
        <exclusion>
          <groupId>org.threeten</groupId>
          <artifactId>threetenbp</artifactId>
        </exclusion>
      </exclusions>
    </dependency>

How can I make both libraries work together?

llermaly commented 2 years ago

I disabled the enforcer when building and now it works.