The plugin can't load requirements on google's definitions

[tomtom3383]

Hye, i'm using logstash 5.6.2 and version 1.0.3 of logstash protobuf codec. I have a proto3 class definition that references google objetcs such as google.protobuf.Value. Once compiled to ruby, my myClass_pb.rb file contains a "require 'google/protobuf'". I use the folowing input configuration :

input {
    kafka {
     zk_connect => "myServer:9092"
     topic_id => "myTopicName"
     group_id => "myGroupId"
     codec => protobuf
     {
       class_name => "myClass"
       include_path => [
       'D:/test/myClass_pb.rb'
       ]
     }
    }
}

When i launch logstash it always fails with this error :

LoadError: no such file to load -- google/protobuf
                         require at org/jruby/RubyKernel.java:1040
                         require at D:/logstash-5.6.2/vendor/bundle/jruby/1.9/gems/polyglot-0.3.5/lib/polyglot.rb:65
                          (root) at D:/test/myClass_pb.rb:3

where line 3 is "require 'google/protobuf'".

I tried to copy ruby google protobuf definitions in a google folder next to my myClass_pb.rb file and added 'D:/test/google/protobuf' as first item of "include_path" parameter but it then fails with the same LoadError on another "require 'google/protobuf'" line in a google definition file (any_pb.rb), the first it found i guess. I also tried passing explicitly the ruby definition file for google.protobuf.Value but it also includes a "require google/protobuf". I have installed the ruby-protocol-buffers-1.6.1.gem in my local ruby installation but i have the impression that logstash's jruby does not take this into account. Has anyone successfully used this codec with proto defined like mine ? Thanks.

[IngaFeick]

Hi @tomtom3383 , could you please provide the full myClass_pb.rb ? Thank you

[tomtom3383]

Hi, here is the file. I compiled it with google protoc tool. Does the codec handle proto3 syntax ? it's not clear in the documentation as it talks about new Google official proto3 ruby implementation in ruby-protocol-buffer documentation. Thanks

# Generated by the protocol buffer compiler.  DO NOT EDIT!
# source: MyClass.proto

require 'google/protobuf'

require 'google/protobuf/struct_pb'

Google::Protobuf::DescriptorPool.generated_pool.build do
  add_message "MyClass" do
    optional :tmst, :int64, 1
    optional :agency_id, :string, 2
    optional :customer_id, :string, 3
    map :data, :string, :message, 4, "google.protobuf.Value"
  end
end
MyClass = Google::Protobuf::DescriptorPool.generated_pool.lookup("MyClass").msgclass

[IngaFeick]

Hi, sadly no, the library that we are currently using supports version 2 only. Might that be the source of the problem? We have support for v3 on the roadmap for the not so far future. If you need it quickly let me know so that I can bump up the priority.

[tomtom3383]

I'm trying to get a workaround by writing a deserializer class for the kafka input plugin but it would be neater to use directly your proto codec.

[xxx7xxxx]

@IngaFeick Look forward to supporting protobuf3 which is mainstream and also supports ruby_out in official repo. Moreover, the development of protobuf2 seems not active anymore. Thanks, guys.

[aliluyko89]

@IngaFeick, Do you have some news about supporting protobuf3 or workaround?

[IngaFeick]

@aliluyko89 I do! We are currently testing a new version which supports v3, you might test it in a couple of days maybe. I will give you the details at the beginning of the next week.

[aliluyko89]

@IngaFeick, It's cool! Just let me know when you have some working version, I will test it with pleasure))

[kporter13]

@IngaFeick Sorry to hijack this issue slightly, but I cloned your latest code for supporting proto3, and successfully built and installed it. When I try to start logstash however, I get exactly the same error as described above LoadError: no such file to load -- google/protobuf. My knowledge of Ruby is completely nil, and given I had to fiddle slightly with the gemspec dependencies to get it installed, I may have messed it up, but thought I'd let you know just in case! We're using logstash 6.1.2. Thank you!

[IngaFeick]

@aliluyko89 Hi, we now have a protobuf 3 supporting version ready for testing. It's not officially released yet so I have uploaded a preliminary version of it to dropbox. You should be able to download it here and then you install it by running

logstash-plugin install  /your_downloads/logstash-codec-protobuf-1.1.0.gem

Please note also the changed configuration section, it's in the documentation of the pull request. Let me know if it works, thank you!

[IngaFeick]

@kporter13 looking into your issue now.

[IngaFeick]

Hi @kporter13 I have built and tested with logstash 6.1.2 and I was not able to reproduce the problem that you described. There are two things you can do now: 1) either you test drive the precompiled version that I mentioned 2 comments above or 2) you give me a detailed description of what you did exactly when building the gem, or I can give you my bash history for building for you to run on your machine, if you want?

[kporter13]

@IngaFeick I'll just give the precompiled gem a run now that it's available. Thank you so much for your work on this, we've been really excited about proto3 support!

EDIT: I'd changed the protobuf dependency in the gemspec to 3.5.1.2. I've just checked the PR comments, and the problems I was having are explained.

[jsvd]

solved by #18