search

theangryangel / logstash-output-jdbc

JDBC output for Logstash
MIT License
255 stars 101 forks source link

mariadb float error #69

Closed Matt-Dee closed 7 years ago

Matt-Dee commented 7 years ago

Expected & Actual Behavior

when trying to insert a field that is a float into a column that is defined as a float, it will fail and not insert the value. If I try to insert a string into a column that is defined as a float, it succeeds.

so 7.7 into column defined as float. fails. "7.7" into column defined as float. succeeds.

This is problematic for me, because I'm trying to define 2 endpoints in my flow. es and mysql. I need the values to be floats for elasticsearch so that I can plot graphs correctly in kibana.

Thanks! Matt

Your Environment

theangryangel commented 7 years ago

2 questions -

  1. Can you be more specific on which version of the 2.x series plugin you're running? There were some fixes later in the 2.x series that may have already addressed this. If you check the Gemfile.lock it should tell you.
  2. Does it fail to insert silently, or does it log anything st all?
Matt-Dee commented 7 years ago

Thanks for getting back to me.

1.) I checked out the branch v2.x for the output plugin and have been working with that. branch

I checked for any tags that might correspond to the version of Logstash that I'm currently running, but didn't see anything. tags

2.) It's inserting and failing silently.

Thanks, Matt

theangryangel commented 7 years ago

I'll try to take a look at it after work tonight and see if I can figure out whats going on.

There aren't necessarily specific tags for specific versions of logstash. Older versions of the plugin are forward compatible, but newer versions of the plugin aren't guaranteed to work with older versions of logstash. As each new version of logstash is released I usually try and release a new version and take advantage of any new features that are applicable, but don't necessarily release for every version of logstash as I don't need to.

Tags 0.2.0.rc1 - 0.3.2 all apply to the logstash v2 releases, but anything after 0.3.0 should only work on 2.3 and newer due to elastic introducing logstash-core-plugin-api for plugins to depend on.

Honestly, unless you're wanting to contribute patches, I'd really suggest relying on bin/logstash-plugin (or bin/plugin for older versions) from the logstash package to install the plugin as it'll just figure out the right version of the gem to pull from rubygems.org.

I'd also suggest that you're on at least logstash 2.3, if you must use v2 of logstash :) There are a pile of fixes that I didn't backport to the <2.3 versions (caused by the introduction of logstash-core-plugin-api).

theangryangel commented 7 years ago

Sorry @Matt-Dee I've not had anytime at all this week to look at this. I've been absolutely slammed at work. I'm not ignoring it, I will address it, but I just haven't had 10 minutes to look at it yet :/

theangryangel commented 7 years ago

Hi @Matt-Dee I've tried reproducing this issue but I'm having no luck at the moment.

If I pass a float to the event it works fine. If it's a string like "7.7", that doesn't work... but that's kinda expected.

If this is what's happening, there are 2 options, as things stand:

  1. Use a ruby filter in Logstash convert the event field into an actual float
  2. Cast'ing - i.e. statement => [ "INSERT INTO TABLE (float_field) values(CAST(? as float))", "event_float_field" ]

If you can give me a way to reproduce this I'm happy to carry on looking into it...