search

revpoint / logstash-codec-avro_schema_registry

A logstash codec plugin for decoding and encoding Avro records
Other
26 stars 23 forks source link

expected type double, got BigDecimal with value 0.0 #20

Open gergof opened 5 years ago

gergof commented 5 years ago

We wan't to use this lib to use logstash with schemas from our schema-registry. But when deserializing the data we get this error:

Avro::SchemaParseError: Error validating default for workday_hours: at . expected type double, got BigDecimal with value 0.0

The relevant part of the schema:

{
  "name" : "workday_hours",
  "type" : {
    "type" : "double",
    "connect.default" : 0.0
  },
  "default" : 0.0
}

The column in our database was float(10, 2) (imported using debezium).

The call stack:

[2019-08-14T10:59:25,376][ERROR][logstash.javapipeline ] A plugin had an unrecoverable error. Will restart this plugin.
Pipeline_id:main
Plugin: <LogStash::Inputs::Kafka value_deserializer_class=>"org.apache.kafka.common.serialization.ByteArrayDeserializer", codec=><LogStash::Codecs::AvroSchemaRegistry endpoint=>"http://kafka.local:8081", id=>"9ec6aef1-7f4a-4db3-98e9-d4273a5cebbc", enable_metric=>true, check_compatibility=>false, register_schema=>false, binary_encoded=>true, tag_on_failure=>false, verify_mode=>"verify_peer">, auto_offset_reset=>"earliest", topics_pattern=>"^paymodb\\.paymoapp\\..*", id=>"kafka_paymodb", bootstrap_servers=>"kafka.local:9092", client_id=>"logstash", enable_metric=>true, auto_commit_interval_ms=>"5000", consumer_threads=>1, enable_auto_commit=>"true", group_id=>"logstash", key_deserializer_class=>"org.apache.kafka.common.serialization.StringDeserializer", topics=>["logstash"], poll_timeout_ms=>100, ssl_endpoint_identification_algorithm=>"https", security_protocol=>"PLAINTEXT", sasl_mechanism=>"GSSAPI", decorate_events=>false>
Error: Error validating default for workday_hours: at . expected type double, got BigDecimal with value 0.0
Exception: Avro::SchemaParseError
Stack: /usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/avro-1.9.0/lib/avro/schema.rb:403:in `validate_default!'
/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/avro-1.9.0/lib/avro/schema.rb:377:in `initialize'
/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/avro-1.9.0/lib/avro/schema.rb:216:in `block in make_field_objects'
org/jruby/RubyArray.java:1792:in `each'
org/jruby/RubyEnumerable.java:1194:in `each_with_index'
/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/avro-1.9.0/lib/avro/schema.rb:209:in `make_field_objects'
/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/avro-1.9.0/lib/avro/schema.rb:240:in `initialize'
/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/avro-1.9.0/lib/avro/schema.rb:72:in `real_parse'
/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/avro-1.9.0/lib/avro/schema.rb:161:in `subparse'
/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/avro-1.9.0/lib/avro/schema.rb:295:in `block in initialize'
org/jruby/RubyArray.java:1792:in `each'
org/jruby/RubyEnumerable.java:1204:in `each_with_object'
/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/avro-1.9.0/lib/avro/schema.rb:294:in `initialize'
/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/avro-1.9.0/lib/avro/schema.rb:90:in `real_parse'
/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/avro-1.9.0/lib/avro/schema.rb:161:in `subparse'
/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/avro-1.9.0/lib/avro/schema.rb:372:in `initialize'
/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/avro-1.9.0/lib/avro/schema.rb:216:in `block in make_field_objects'
org/jruby/RubyArray.java:1792:in `each'
org/jruby/RubyEnumerable.java:1194:in `each_with_index'
/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/avro-1.9.0/lib/avro/schema.rb:209:in `make_field_objects'
/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/avro-1.9.0/lib/avro/schema.rb:240:in `initialize'
/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/avro-1.9.0/lib/avro/schema.rb:72:in `real_parse'
/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/avro-1.9.0/lib/avro/schema.rb:38:in `parse'
/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-codec-avro_schema_registry-1.1.1/lib/logstash/codecs/avro_schema_registry.rb:158:in `get_schema'
/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-codec-avro_schema_registry-1.1.1/lib/logstash/codecs/avro_schema_registry.rb:228:in `decode'
/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-input-kafka-9.0.0/lib/logstash/inputs/kafka.rb:256:in `block in thread_runner'
/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-input-kafka-9.0.0/lib/logstash/inputs/kafka.rb:255:in `block in thread_runner'
[2019-08-14T10:59:25,405][FATAL][logstash.runner ] An unexpected error occurred! {:error=>#<Avro::SchemaParseError: Error validating default for workday_hours: at . expected type double, got BigDecimal with value 0.0>, :backtrace=>["/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/avro-1.9.0/lib/avro/schema.rb:403:in `validate_default!'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/avro-1.9.0/lib/avro/schema.rb:377:in `initialize'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/avro-1.9.0/lib/avro/schema.rb:216:in `block in make_field_objects'", "org/jruby/RubyArray.java:1792:in `each'", "org/jruby/RubyEnumerable.java:1194:in `each_with_index'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/avro-1.9.0/lib/avro/schema.rb:209:in `make_field_objects'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/avro-1.9.0/lib/avro/schema.rb:240:in `initialize'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/avro-1.9.0/lib/avro/schema.rb:72:in `real_parse'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/avro-1.9.0/lib/avro/schema.rb:161:in `subparse'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/avro-1.9.0/lib/avro/schema.rb:295:in `block in initialize'", "org/jruby/RubyArray.java:1792:in `each'", "org/jruby/RubyEnumerable.java:1204:in `each_with_object'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/avro-1.9.0/lib/avro/schema.rb:294:in `initialize'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/avro-1.9.0/lib/avro/schema.rb:90:in `real_parse'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/avro-1.9.0/lib/avro/schema.rb:161:in `subparse'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/avro-1.9.0/lib/avro/schema.rb:372:in `initialize'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/avro-1.9.0/lib/avro/schema.rb:216:in `block in make_field_objects'", "org/jruby/RubyArray.java:1792:in `each'", "org/jruby/RubyEnumerable.java:1194:in `each_with_index'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/avro-1.9.0/lib/avro/schema.rb:209:in `make_field_objects'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/avro-1.9.0/lib/avro/schema.rb:240:in `initialize'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/avro-1.9.0/lib/avro/schema.rb:72:in `real_parse'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/avro-1.9.0/lib/avro/schema.rb:38:in `parse'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-codec-avro_schema_registry-1.1.1/lib/logstash/codecs/avro_schema_registry.rb:158:in `get_schema'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-codec-avro_schema_registry-1.1.1/lib/logstash/codecs/avro_schema_registry.rb:228:in `decode'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-input-kafka-9.0.0/lib/logstash/inputs/kafka.rb:256:in `block in thread_runner'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-input-kafka-9.0.0/lib/logstash/inputs/kafka.rb:255:in `block in thread_runner'"]}
jj76lee commented 4 years ago

@gergof I have a same problem when I use this codec with a double default value within a schema registry. You should fix some code like below @ avro-1.9.2/lib/avro/schema_validator.rb

130           fail TypeMismatchError unless datum.is_a?(Float) || datum.is_a?(Integer) || datum.is_a?(BigDecimal)

I hope it will help.

eperry commented 2 years ago

Same issue here too. Did anyone have a way of working around this?