Support encoding timestamp as google.protobuf.Timestamp

[willbuckner]

It appears that currently, all fields in the protobuf must be strings. Is it currently possible to encode @timestamp as a google.protobuf.Timestamp timestamp in the protobuf? Unless there is currently some way to do this that I'm missing, please consider this a feature request :) Thanks!

[IngaFeick]

Hi @willbuckner, the fields in the protobuf definition don't have to be strings. You can use all the scalar data types that google has defined, and all complex data types that you import or define yourself. Which protobuf version and programming language are you using? I assume v3 and java? Could you please provide a full test case, including your protobuf definition, the compiler call (or name of the IDE you're using), any error messages you might have gotten from the compiler, and the expected output/behaviour (if applicable)? Thank you

[IngaFeick]

Please also be aware that the @timestamp field is used by logstash for the event date, so the type of that field will be set by logstash itself, unrelated to the protobuf codec.

[IngaFeick]

Hi @willbuckner Do you still need anything or can we close this?

[dhanashri-pitre]

Hi, I am using the codec along with ruby-protoc compiler. In my proto2 message I have google.protobuf.Timestamp and I get following error Reason: # while loading the proto file. Can we please look into this. Thanks.

[IngaFeick]

Hi @dhanashri-pitre could you please provide your protobuf definition and the protobuf codec section of your logstash config? Thank you

[dhanashri-pitre]

Here is the proto msg syntax = "proto2"; package b; option java_package = "com.xyz.b"; option java_outer_classname = "Example"; import "google/protobuf/timestamp.proto"; message I { required google.protobuf.Timestamp i_time = 1; required int64 a_id = 2; required int64 c_id = 3; }

And this is the conf input { kafka { bootstrap_servers => "localhost:9092" client_id => "logstash-test-c1" group_id => "logstash-test-g1" topics => "imp" auto_offset_reset => "earliest" key_deserializer_class => "org.apache.kafka.common.serialization.ByteArrayDeserializer" value_deserializer_class => "org.apache.kafka.common.serialization.ByteArrayDeserializer" codec => protobuf { class_name => "B::I" class_file => "/com/xyz/b/example.pb.rb" protobuf_root_directory => "/com/xyz/" } } } output { file { path => "/com/xyz/Kafka-sink.txt" } }

Error encountered [2020-04-02T18:34:41,921][ERROR][logstash.codecs.protobuf ] Unable to load file: /com/xyz/b/example.pb.rb. Reason: # [2020-04-02T18:34:41,928][FATAL][logstash.runner ] The given configuration is invalid. Reason: Unable to configure plugins: (NameError) uninitialized constant Google::Protobuf::Timestamp [2020-04-02T18:34:41,933][ERROR][org.logstash.Logstash ] java.lang.IllegalStateException: Logstash stopped processing because of an error: (SystemExit) exit

[IngaFeick]

Thanks @dhanashri-pitre, I will look into this

[jorgelbg]

@dhanashri-pitre how did you compile the .pb.rb file? Can you share the ruby compiled version of your proto message?

[dhanashri-pitre]

Hi, I compiled the file using ruby-protocol-buffers Here is the example.pb.rb

require 'protocol_buffers' begin; require 'google/protobuf/timestamp.pb'; rescue LoadError; end

module B class I < ::ProtocolBuffers::Message; end

class I < ::ProtocolBuffers::Message set_fully_qualified_name "b.I"

required ::Google::Protobuf::Timestamp, :i_time, 1
required :int64, :a_id, 2
required :int64, :c_id, 3

end

end

[IngaFeick]

@dhanashri-pitre did you install any gems related to the timestamp prior to this? When I try to do this with the same compiler I run into google/protobuf/timestamp.proto: File not found. GoogleTimestampTest.proto: Import "google/protobuf/timestamp.proto" was not found or had errors. GoogleTimestampTest.proto:7:9: "google.protobuf.Timestamp" is not defined. which is to be expected. Could you kindly also provide a list of the installed gems on your machine? Thanks

[dhanashri-pitre]

I did not install any new gem, only the ruby-protocol-buffers gem And here is the list of gem available on my machine : bigdecimal (default: 1.4.1) bundler (default: 1.17.2) CFPropertyList (2.3.6) cmath (default: 1.0.0) csv (default: 3.0.9) date (default: 2.0.0) dbm (default: 1.0.0) did_you_mean (1.3.0) e2mmap (default: 0.1.0) etc (default: 1.0.1) fcntl (default: 1.0.0) fiddle (default: 1.0.0) fileutils (default: 1.1.0) forwardable (default: 1.2.0) io-console (default: 0.4.7) ipaddr (default: 1.2.2) irb (default: 1.0.0) json (default: 2.1.0) libxml-ruby (3.1.0) logger (default: 1.3.0) matrix (default: 0.1.0) mini_portile2 (2.4.0) minitest (5.11.3) mutex_m (default: 0.1.0) net-telnet (0.2.0) nokogiri (1.10.1) openssl (default: 2.1.2) ostruct (default: 0.1.0) power_assert (1.1.3) prime (default: 0.1.0) psych (default: 3.1.0) rake (12.3.2) rdoc (default: 6.1.0) rexml (default: 3.1.9) rss (default: 0.2.7) ruby-protocol-buffers (1.6.1) scanf (default: 1.0.0) sdbm (default: 1.0.0) shell (default: 0.7) sqlite3 (1.3.13) stringio (default: 0.0.2) strscan (default: 1.0.0) sync (default: 0.5.0) test-unit (3.2.9) thwait (default: 0.1.0) tracer (default: 0.1.0) webrick (default: 1.4.2) xmlrpc (0.3.0) zlib (default: 1.0.0)

[jorgelbg]

@dhanashri-pitre google/protobuf/timestamp.proto seems to be compatible with protobuf v3 only. Even so, it seems just to be a .proto definition offered only by the official protobuf library for convenience and not a core feature of the language. I don't see any type of support for that particular timestamp in the ruby-protocol-buffers gem.

You should be able to adapt the google/protobuf/timestamp.proto to use it with proto2 and import it as a local dependency of your main definition. TBH I'm not sure how it even compiled, as @IngaFeick pointed out the expected behavior is to fail in the .rb generation step.

[dhanashri-pitre]

@jorgelbg Thank You, adapting timestamp.proto and using the local dependency worked for me.