Initial Google Implementation

josephlewis42 commented 6 years ago

We've wrapped up our initial implementation and it's been approved by Google for public release. Thanks @acchen97 for getting the repo set up for us.

This initial release was built on work @erjohnso did that I ported over to using our Java clients after the other plugins started experiencing SSL issues with their ruby versions.

josephlewis42 commented 6 years ago

Additional info:

Travis CI: https://travis-ci.org/josephlewis42/logstash-output-google_pubsub/branches
Pre-release gemfile and documentation for testing.

erjohnso commented 6 years ago

Great work @josephlewis42 - this LGTM

fkoclas commented 6 years ago

Hi,

I can't get the plugin to initialize properly due to a type error: TypeError: cannot convert instance of class org.jruby.RubyString to class com.google.pubsub.v1.TopicName

If I remove the topic from the configuration it complains that topic => # SETTING MISSING as you would expect.

I tried with both Logstash 5.6.10 and 6.3.1 and get the same error.

Here is the complete stacktrace:

[ERROR][logstash.agent ] Pipeline aborted due to error {:exception=>#, :backtrace=>["/opt/logstash/vendor/local_gems/510e3a38/logstash-output-google_pubsub-1.0.0-java/lib/logstash/outputs/pubsub/client.rb:80:in initialize_google_client'", "/opt/logstash/vendor/local_gems/510e3a38/logstash-output-google_pubsub-1.0.0-java/lib/logstash/outputs/pubsub/client.rb:30:ininitialize'", "/opt/logstash/vendor/local_gems/510e3a38/logstash-output-google_pubsub-1.0.0-java/lib/logstash/outputs/google_pubsub.rb:114:in register'", "/opt/logstash/logstash-core/lib/logstash/output_delegator_strategies/shared.rb:9:inregister'", "/opt/logstash/logstash-core/lib/logstash/output_delegator.rb:43:in register'", "/opt/logstash/logstash-core/lib/logstash/pipeline.rb:290:inregister_plugin'", "/opt/logstash/logstash-core/lib/logstash/pipeline.rb:301:in register_plugins'", "org/jruby/RubyArray.java:1613:ineach'", "/opt/logstash/logstash-core/lib/logstash/pipeline.rb:301:in register_plugins'", "/opt/logstash/logstash-core/lib/logstash/pipeline.rb:310:instart_workers'", "/opt/logstash/logstash-core/lib/logstash/pipeline.rb:235:in run'", "/opt/logstash/logstash-core/lib/logstash/agent.rb:408:instart_pipeline'"]}

josephlewis42 commented 6 years ago

@robbavey thanks for the feedback!

I changed a few things:

Bumped the version of the Pub/Sub library to the most recent version.
Added initial docs with examples.
Cleaned up the tests.
Changed receive to multi_receive_encoded (and tests around it) because it's really not acceptable to release a new plugin without it

@fkoclas would you mind posting your config file? I can't seem to reproduce it in my setup. Mine looks like this:

input { stdin { } }

output {
    google_pubsub {
        project_id => "[redacted]"

        topic => "ls-input-google-pubsub-test-topic"
        attributes => {
            "attr_key" => "attr_value"
            "attr_key2" => "attrval2"
        }

        json_key_file => "/home/[redacted]/key.json"
    }
}

fkoclas commented 6 years ago

@josephlewis42 pretty much the same config:

input {
  stdin {
  }
}

output {
  google_pubsub {
    project_id => "<snip>"
    topic => "elasticsearch_indexing"
    json_key_file => "/etc/secrets/serviceaccount.json"
  }
}

Interestingly I did some more testing, and it turns out that it works on the openjdk:8-jre docker image (debian-based), but not on the openjdk:8-jre-alpine docker image which is what I have been using to run logstash for nearly the past two years.

robbavey commented 6 years ago

@karenzone Can you please review the documentation additions here?

josephlewis42 commented 6 years ago

@karenzone I merged those files and changed LogStash -> Logstash

@robbavey License headers are gone and updated the jars include file

Thank you both for taking the time to help review this! :-)

robbavey commented 6 years ago

@josephlewis42 I'm happy with this from a code perspective, but please wait for the OK from @karenzone on the doc front before submission.

josephlewis42 commented 6 years ago

@robbavey and @karenzone thank you so much for the reviews, I'll merge this in and push a 1.0.0 gem tonight when I get home.

@fkoclas I'll open an issue and mention you in it so we can dig into the specific problem with the Alpine combo.

karenzone commented 6 years ago

@josephlewis42 Thanks for your contribution and for the kind words. It's been a pleasure working with you on this. FYI, the plugin is in the Versioned Plugin doc now (https://www.elastic.co/guide/en/logstash-versioned-plugins/current/v1.0.0-plugins-outputs-google_pubsub.html). I have a few more steps to complete, and then it will appear in the Logstash Reference. Thanks again!

karenzone commented 6 years ago

Hi @josephlewis42. I'll be pulling your doc into the Logstash Reference Guide shortly. While I was testing, I noticed that the https://cloud.google.com/pubsub/pricing-summary/[Cloud Pub/Sub Pricing] link under Additional Resources is not resolving. I can merge the PR to include the plugin, but thought you'd want to know about the broken link so that we can get it fixed.

logstash-plugins / logstash-output-google_pubsub

Initial Google Implementation #5