fangli / fluent-plugin-influxdb

A buffered output plugin for fluentd and InfluxDB
MIT License
111 stars 65 forks source link

Multiple bugs report #89

Open ash0080 opened 6 years ago

ash0080 commented 6 years ago
  1. if record contains a NULL as value, no Error, has POST log, no records, just gone

  2. timeStamp type conversion Error, should be INTEGER or a DATE? but FLOAT with an Error now

  3. auto_tags almost not work, in most time, cause fail.

repeatedly commented 6 years ago

2 seems https://github.com/fangli/fluent-plugin-influxdb/issues/85 1 and 3 are not clear for me. Could you write reproducible step here? Configuration and log example are needed to debug.

ash0080 commented 6 years ago

For 1, That's because influxdb doesn't accept null like value now, for example { title: '', message: 'hello' }, but I think this plugin should throw an Error but not just POST like everything is ok , or there should be a http response from influxDB, better to catch that if possible

For 3, just try a full String record like 1, this feature never worked for me, so not sure the limit very well.

repeatedly commented 6 years ago

I see.

there should be a http response from influxDB, better to catch that if possible

This seems impossible because influxdb seems to not return the error for the client.

https://github.com/influxdata/influxdb-ruby/blob/ab77a703f5fef39fb05957672f8afb0b42ebd37e/lib/influxdb/client/http.rb#L24

influxdb-ruby checks HTTP response and throw an exception when got error. Or does influxdb-ruby return some information in OK response?

BTW, we can skip '' value in fluentd side. Honestly, influxdb-ruby should check invalid values...

For 3, just try a full String record like 1, this feature never worked for me, so not sure the limit very well.

Yeah, this is auto_tags limitation. You need to specify tag_keys parameter because fluentd can't judge which is the value for all string fields case.

ash0080 commented 6 years ago

Confused, The link you post is a client api, so why do you care a client Error? the response data structure could be found here, it's clear that it returns a "writes" number in response.body at least

If auto_tags feature is not ready, I think it's better to be removed from the released version. I am not a ruby coder, but seems fluent provides a 'delimiter' to parse the key=>value pairs. so, not be sure, but maybe this could solve the problem you mentioned.

Now my works have been done. but not elegant, I did most formatting & checking works even the ugly time transforming in javascript before sending. so it's hard to say this plugin is good enough for production. unfortunately, this' the only 'verified" one influxdb plugin now

repeatedly commented 6 years ago

The link you post is a client api, so why do you care a client Error?

Because this plugin uses influxdb-ruby client, not call influxdb HTTP API directly.

If auto_tags feature is not ready, I think it's better to be removed from the released version.

auto_tags is used on some user's production so we can't remove this parameter. Again, this feature is for non all string records. Please use tag_keys for all string case. You know which fields are tags.

I am not a ruby coder, but seems fluent provides a 'delimiter' to parse the key=>value pairs. so, not be sure, but maybe this could solve the problem you mentioned.

This sentence is unclear for me. out_influxdb is output plugin so this plugin doesn't parse incoming data.

repeatedly commented 6 years ago

For 1, partial fix is here: https://github.com/fangli/fluent-plugin-influxdb/pull/91

ash0080 commented 6 years ago

Because this plugin uses influxdb-ruby client, not call influxdb HTTP API directly.

Isn't it a wrap of RESTful? I thought you can get response here

This sentence is unclear for me. out_influxdb is output plugin so this plugin doesn't parse incoming data.

I see, I got it wrong.

cosmo0920 commented 6 years ago

@ash0080 fluent-plugin-influxdb does not use HTTP API explictly. Instead, it uses #write_points.

ash0080 commented 6 years ago

@ash0080 fluent-plugin-influxdb does not use HTTP API explictly. Instead, it uses #write_points.

なるほど