Imported proto can't be loaded

[jvmlet]

Hello It looks like the imported proto (the corresponding rb file) can't be found. Here are my proto files:

child.proto :

syntax = "proto3";
message Child {
    string name=1;
}

parent.proto :

syntax = "proto3";
import "child.proto";
message Parent {
    Child child =1;
}

Generated parent_pb.rb:

# Generated by the protocol buffer compiler.  DO NOT EDIT!
# source: parent.proto

require 'google/protobuf'

require 'child_pb'
Google::Protobuf::DescriptorPool.generated_pool.build do
  add_message "Parent" do
    optional :child, :message, 1, "Child"
  end
end

Parent = Google::Protobuf::DescriptorPool.generated_pool.lookup("Parent").msgclass

Note the require 'child_pb'

Generated child_pb.rb:

# Generated by the protocol buffer compiler.  DO NOT EDIT!
# source: child.proto

require 'google/protobuf'

Google::Protobuf::DescriptorPool.generated_pool.build do
  add_message "Child" do
    optional :name, :string, 1
  end
end

Child = Google::Protobuf::DescriptorPool.generated_pool.lookup("Child").msgclass

Both child_pb.rb and parent_pb.rb reside under C:/test/ directory.

The proto-pipeline.conf:

input {
    tcp  {
        port => "8082"
        type => "entity"
        codec => protobuf   {
            class_name => "Parent"
            include_path => ['file://C:/test/child_pb.rb','file://C:/test/parent_pb.rb']
             protobuf_version => 3
        }
    }
}

output {
   elasticsearch {
        hosts => [ "localhost:9200" ]
    }
}

Running bin/logstash -f proto-pipeline.conf --config.test_and_exit gives me below output :

[2019-04-03T14:22:31,592][DEBUG][logstash.config.source.local.configpathloader] Reading config file {:config_file=>"C:/XXX/logstash-6.7.0/proto-pipeline.conf"}
[2019-04-03T14:22:36,276][DEBUG][logstash.plugins.registry] On demand adding plugin to the registry {:name=>"protobuf", :type=>"codec", :class=>LogStash::Codecs::Protobuf}
[2019-04-03T14:22:36,394][DEBUG][logstash.codecs.protobuf ] config LogStash::Codecs::Protobuf/@protobuf_version = 3
[2019-04-03T14:22:36,395][DEBUG][logstash.codecs.protobuf ] config LogStash::Codecs::Protobuf/@id = "0505178b-e211-4e8f-b4f7-d7bab19dc699"
[2019-04-03T14:22:36,397][DEBUG][logstash.codecs.protobuf ] config LogStash::Codecs::Protobuf/@include_path = ["file://C:/test/child_pb.rb", "file://C:/test/parent_pb.rb"]
[2019-04-03T14:22:36,401][DEBUG][logstash.codecs.protobuf ] config LogStash::Codecs::Protobuf/@class_name = "Parent"
[2019-04-03T14:22:36,402][DEBUG][logstash.codecs.protobuf ] config LogStash::Codecs::Protobuf/@enable_metric = true
[2019-04-03T14:22:36,402][DEBUG][logstash.codecs.protobuf ] config LogStash::Codecs::Protobuf/@stop_on_error = false
[2019-04-03T14:22:36,576][ERROR][org.logstash.Logstash    ] java.lang.IllegalStateException: Logstash stopped processing because of an error: (LoadError) no such file to load -- child_pb

Note the (LoadError) no such file to load -- child_pb - probably caused by require 'child_pb' in parent_pb.rb file.

Please advise. Thanks

[IngaFeick]

Hi @jvmlet Can you remove the file:// from the beginning of the pathes and let me know if that changes the behaviour? Thank you.

[rimvydas-pranciulis]

I had same issue, but not sure if it was caused by same thing. My problem was that I got installed newest version of logstash-codec-protobuf plugin (1.2.2), but oficial documentation there was for older version (v1.2.1) https://www.elastic.co/guide/en/logstash/current/plugins-codecs-protobuf.html. I didn't notice that versions were different at the beginning.

After going through code I found two new properties that are in new version but were not documented. So in order to solve issue I replaced include_path with two other properties:

• protobuf_root_directory - Absolute path to the root directory that contains all referenced/used dependencies
• class_file - Relative path to the ruby file that contains class_name

[jorgelbg]

@rimvydas-pranciulis strange, the documentation should have been updated already since we released 1.2.2. Also, although we added those new configuration options the old include_path should continue to work (if all the dependencies of the class_name were specified as absolute paths in the include_path).

We will look into that. FYI @IngaFeick

[rimvydas-pranciulis]

@jorgelbg include_path only worked if everything was in one file. When I had two files I was getting error that require fails (I put file without require as first in include_path). I'm on Windows, using proto3 and used absolute file paths like c:/logstash/proto-files/test_pb.rb