logstash-plugins / logstash-input-google_pubsub

Logstash input for pulling events from Google Pub/Sub service
Apache License 2.0
19 stars 33 forks source link

Use codec to decode messages #23

Closed jakubsacha closed 6 years ago

jakubsacha commented 6 years ago

For some reason this plugin assumes that data in the pubsub is always json. It is not possible to decode for example protobuf coming from pubsub.

Please have a look at the proposed fix. I have awareness that this is bc for most of the configs, since the plugin do json decoding by default.

I'm not sure but maybe additional change of default :codec, "plain" to default :codec, "json" would help here.

Just for completeness. This is logstash config that works for me (pubsub + protobuf) when that PR is applied:

input {
    google_pubsub {
        project_id => "xxxx"
        topic => "xxxx"
        subscription => "xxxx"
    codec => protobuf
    {
        class_name => "MyProtoClass"
        include_path => ['path_to_my_proto_class_pb.rb']
        protobuf_version_3 => true
    }
    }
}

output {
    stdout { codec => rubydebug }
}
jakubsacha commented 6 years ago

Hey @jsvd, i just signed CLA. I'm not sure how to re-trigger the check.

jakubsacha commented 6 years ago

@jsvd as requested. Please have a look.