logstash-plugins / logstash-codec-protobuf

Codec plugin for parsing Protobuf messages
Apache License 2.0
26 stars 16 forks source link

Nested protobuf messages #19

Closed d0nski closed 4 years ago

d0nski commented 6 years ago

Hi,

Is there a way to get this working with a nested protobuf message i'm getting from JMS using the JMS plugin? I have an "envelope" protobuf message, within this there are several data fields including a message_data field which contains another protobuf message which is a Report protobuf message. I have both the pb.rb files, and can decode the envelope fine, but am so far unable to figure out how to decode the message_data field within.

Envelope.pb.rb

#!/usr/bin/env ruby
# Generated by the protocol buffer compiler. DO NOT EDIT!

require 'protocol_buffers'

# forward declarations
class Envelope < ::ProtocolBuffers::Message; end

class Envelope < ::ProtocolBuffers::Message
  set_fully_qualified_name "Envelope"

  optional :string, :type_name, 1
  optional :int32, :type_version, 2
  optional :bytes, :message_data, 3
  optional :int64, :timeStamp, 4
end

proto file for message_data field:

#!/usr/bin/env ruby
# Generated by the protocol buffer compiler. DO NOT EDIT!

require 'protocol_buffers'

begin; require 'Types.pb'; rescue LoadError; end
begin; require 'EnumTypes.pb'; rescue LoadError; end

module Protos
  # forward declarations
  class Report < ::ProtocolBuffers::Message; end
  class ReportNew < ::ProtocolBuffers::Message; end

  class Report < ::ProtocolBuffers::Message
    set_fully_qualified_name "protos.Report"

    optional ::Protos::ControlType, :control, 1
    optional :string, :orderID, 2
    optional :string, :secondaryOrderID, 3
    optional :string, :secondaryClOrdID, 4
    optional :string, :secondaryExecID, 5
.
.
.

I can decode the envelope message with this conf below and it works in bringing me back the four fields specified in envelope

input {
    jms {
        include_header => true
        include_properties => false
        include_body => true
    pub_sub => true
        use_jms_timestamp => false
        interval => 1
        destination => ">"
        yaml_file => "solace.yml"
        yaml_section => "solace"
    runner => "async"
    codec => protobuf
     {
       class_name => "Envelope"
         include_path => ['event/Envelope.pb.rb']
         }
    }
}

output {
  stdout { codec => rubydebug }
}

I was trying to get the message_data data with the below conf, but get the error "Couldn't decode protobuf: #<ProtocolBuffers::DecodeError: incorrect wire type for tag: 10, expected 2 but got 0"

input {
    jms {
        include_header => true
        include_properties => false
        include_body => true
    pub_sub => true
        use_jms_timestamp => false
        interval => 1
        destination => ">"
        yaml_file => "solace.yml"
        yaml_section => "solace"
    runner => "async"
    codec => protobuf
     {
       class_name => "Envelope::Protos::Report"
       include_path => ['event/Envelope.pb.rb', 'event/EnumTypes.pb.rb', 'event/Types.pb.rb',  'event/MessageEvents.pb.rb']
       }
    }
}

output {
  stdout { codec => rubydebug }
}

I'm new to protobuf - does what I am trying to do make sense? And if so is there a way to do it or how can I look at achieving this? (I have no control over the protobuf messages themselves)

Thanks

d0nski commented 6 years ago

I've modified the plugin to do this myself - closing

IngaFeick commented 6 years ago

Hi @d0nski , glad to hear that you fixed it! Would you mind sharing the solution with us for improving the codec, or sending a patch upstream? Would be much appreciated :) Thank you!

3fr61n commented 3 years ago

@d0nski Hi!!

Could you please let us know what was the fix/workaround? I'm running similar issues and I don't know what else to do.

Regards