search

logstash-plugins / logstash-input-google_pubsub

Logstash input for pulling events from Google Pub/Sub service
Apache License 2.0
19 stars 33 forks source link

Sometimes uses the Guava shipped with Logstash and reach a NoSuchMethodError #57

Closed andsel closed 2 years ago

andsel commented 2 years ago

Logstash information:

Please include the following information:

  1. Logstash version (e.g. bin/logstash --version) : 8.3.3 but started since Logstash 8.2.0
  2. Logstash installation source (e.g. built from source, with a package manager: DEB/RPM, expanded from tar or zip archive, docker) : any
  3. How is Logstash being run (e.g. as a service/service manager: systemd, upstart, etc. Via command line, docker/kubernetes): any
  4. How was the Logstash Plugin installed: plugin-install

JVM (e.g. java -version):

If the affected version of Logstash is 7.9 (or earlier), or if it is NOT using the bundled JDK or using the 'no-jdk' version in 7.10 (or higher), please provide the following information:

  1. JVM version (java -version)
  2. JVM installation source (e.g. from the Operating System's package manager, from source, etc).
  3. Value of the JAVA_HOME environment variable if set.

OS version (uname -a if on a Unix-like system): N/A

Description of the problem including expected versus actual behavior: Sometimes Logstash starts and report the error:

[2022-08-17T22:59:46,910][ERROR][logstash.inputs.googlepubsub][main][69466b68efba17b190dd16bcd12466cd5488773b860c9a0f79fa6c7c65f5ec61] Expected the service InnerService [FAILED] to be RUNNING, but the service has FAILED
Aug 17, 2022 10:59:46 PM com.google.common.util.concurrent.ListenerCallQueue$PerListenerQueue run
SEVERE: Exception while executing callback: com.google.api.core.AbstractApiService$1@724362ac failed({from = STARTING, cause = java.lang.IllegalStateException: Expected the service InnerService [FAILED] to be RUNNING, but the service has FAILED})
java.lang.IllegalStateException: Expected the service InnerService [FAILED] to be RUNNING, but the service has FAILED
        at com.google.common.util.concurrent.AbstractService.checkCurrentState(AbstractService.java:381)
        at com.google.common.util.concurrent.AbstractService.awaitRunning(AbstractService.java:305)
        at com.google.api.core.AbstractApiService.awaitRunning(AbstractApiService.java:96)
        at com.google.cloud.pubsub.v1.Subscriber.startConnections(Subscriber.java:390)
        at com.google.cloud.pubsub.v1.Subscriber.startStreamingConnections(Subscriber.java:355)
        at com.google.cloud.pubsub.v1.Subscriber.access$200(Subscriber.java:93)
        at com.google.cloud.pubsub.v1.Subscriber$3.run(Subscriber.java:299)
        at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: java.lang.NoSuchMethodError: 'void com.google.common.util.concurrent.Futures.addCallback(com.google.common.util.concurrent.ListenableFuture, com.google.common.util.concurrent.FutureCallback)'
        at com.google.cloud.pubsub.v1.StreamingSubscriberConnection.initialize(StreamingSubscriberConnection.java:218)
        at com.google.cloud.pubsub.v1.StreamingSubscriberConnection.doStart(StreamingSubscriberConnection.java:110)
        at com.google.api.core.AbstractApiService$InnerService.doStart(AbstractApiService.java:148)
        at com.google.common.util.concurrent.AbstractService.startAsync(AbstractService.java:251)
        at com.google.api.core.AbstractApiService.startAsync(AbstractApiService.java:120)
        at com.google.cloud.pubsub.v1.Subscriber.startConnections(Subscriber.java:387)
        ... 4 more

Sometimes it works.

Steps to reproduce:

Please include a minimal but complete recreation of the problem, including (e.g.) pipeline definition(s), settings, locale, etc. The easier you make for us to reproduce it, the more likely that somebody will take the time to look at it.

1. 2. 3.

Provide logs (if relevant):

andsel commented 2 years ago

With PR https://github.com/elastic/logstash/pull/13700 Google Guava shipped with Logstash switched from 24.1.1 to 31.0.1 and this happened in Logstash 8.2.0

In Guava 24.1.1 the method was present but deprecated and result removed in 26.0.0.

Now the gems with java libraries inside, comes with all the classes they needs, just check this by downloading the pubsub plugin gem and navigate the data.tar.gz/vendor/jar-dependencies folder. There you can find all the classes, including the Guava 24.

The problem is that the JVM classpath is not versioned and when the classpath contains 2 classes in same package with same name, there is no way to control which is loaded first.

So in this case, sometime it works because the Guava 24 class is loaded first, while some other times the classloader loads Guava 31 class and generates the error.

How to solve Adapt the plugin to use the new Guava class in version 31.0.1 (the same used since Logstash 8.2.0) and put a strong requirement in gemspec file on Logstash 8.2+.

SystemZ commented 2 years ago

I encountered this and seems like easy temp workaround is to use logstash 8.1.3 instead of current newest 8.3.3

andsel commented 2 years ago

Hi @SystemZ I've create a PR to update the version of the Google pub sub client in #58 , please could you test that against a Logstash >= 8.2 ?