Support Pub/Sub Lite for input/output

[dpcollins-google]

Pub/Sub Lite is a lower-cost, zonal version of google's Pub/Sub service that has the same user-facing API while using a different low level API for efficiency reasons.

I would like to contribute input and output logstash plugins for this service, however, Pub/Sub Lite does not support Ruby. Would it be possible to set up a java plugin repo for both an input and an output plugin so that I can contribute the implementation required?

[kares]

Hey, thanks for your interest in building an alternative to the existing pubsub input/output plugins.

First of all the existing plugins while looking like Ruby are relatively thin wrappers and are using JRuby's Java scripting capabilities, for example the pubsub input :

• relies on a ton of Java dependencies (through gradle)
• gradle generated a .rb script that loads these jars (and their deps) at runtime (when the plugin is being loaded)
• the plugin includes a bunch of Java packages in it's name space
• there's a com.google.cloud.pubsub.v1.MessageReceiver interface implementation which is the gist of the plugin's run entry-point

Thus despite looking like a lot of Ruby the plugin mostly relies on Java libraries to do the heavy lifting and Ruby is mainly used to glue everything together.

---

The second option is similar to the first but also includes Java sources e.g. the Beats input, the plugin entry point is still Ruby scripted - the implementation simply delegates to the Java part.

---

The third option is to have a pure Java plugin, such as seen in the example input.

One limiting factor for a pure Java plugin is that there's no way to use Ruby codecs since the codecs currently shipped with Logstash are implemented in Ruby wout a Java adapter interface. This is really a bug (something that got forgotten when the Java plugin API was designed) that will eventually get fixed in the long term.

---

Would it be possible to set up a java plugin repo for both an input and an output plugin so that I can contribute the implementation required?

There's no need to setup an initial repository under the logstash-plugins organization, most 3rd party plugins exists outside the GH organization. Historically a central organization was used to list plugins, today a lot of the repos, under logstash-plugins org, are non-active either community contributed or abandoned by the team.

Once the new pub-sub input plugin is production ready and has documentation we could list it under the plugins documentation, at that point the Logstash team would review the plugin and provide feedback. Again no need to have it under the logstash-plugins organization (although we could move the repository as well at that point).

---

Let us know if there's anything else that might be unclear or if you need help setting up your plugin development environment.

[dpcollins-google]

Hi Karol,

Thank you for all the info! In particular, I was unaware that the Cloud Pub/Sub plugin was ruby-wrapped-java, given that, I think I could add Pub/Sub Lite support to the existing plugins given that all the end-user interfaces are shared. I'll make an attempt at that instead of setting up a new repo.

Thanks!

-Daniel

[dpcollins-google]

So, I'm having issues building the plugin to validate that it works- I'm getting the following error saying logstash-devutils is missing:

An error occurred while loading ./spec/inputs/google_pubsub_spec.rb.
Hint: Install the `did_you_mean` gem in order to provide suggestions for similarly named files.
Failure/Error: require "logstash/devutils/rspec/spec_helper"

but running:

jgem install logstash-devutils

Results in the following error:

jgem install logstash-devutils                                                                                                                                      
ERROR:  Could not find a valid gem 'logstash-core' (= 5.1.2.snapshot1) (required by 'logstash-devutils' (>= 0)) in any repository
ERROR:  Possible alternatives: logstash-core

Any ideas?

[kares]

Sorry about the mess at some point Logstash core stopped pushing it's internals to RubyGems.org and some of the plugin dependencies expect a fairly recent logstash-core gem, these are always available if you have a local Logstash installation.

I've setup a PR for the plugin to reflect the local LOGSTASH_PATH development setup as well as be able to build against Java 11: https://github.com/logstash-plugins/logstash-input-google_pubsub/pull/56

[LPacione]

Hi! I'm trying to move data from PubSub to PubSub Lite in Logstash, do you know if this is possible? Because i'm reading your conversation and i don't know what was the result. Thank you very much.