Make the plugin compatible with Influxdb 0.9

contentfree commented 9 years ago

I've got a working version in my fork of the repo (which also includes some other fixes and enhancements like the ability to use the event data directly instead of requiring the explicit setting of data_points).

brupm commented 9 years ago

:+1:

contentfree commented 9 years ago

There's a philosophical change in Influxdb 0.9. The Influxdb team asserts that metadata about the values in the series (things like host, username, IP, etc) should go into Influxdb as tags instead of fields. Values should only be a field if it changes over time.

Logstash also has tags and fields — and thought of in a similar way — however, we'll probably need to decide the method of mapping Logstash fields to Influxdb fields & tags.

I don't think we can just say "just use add_tag and pass through those tags" because things like the grok filter match values into fields and not tags.

Should we use a config setting to provide "value_fields" (an array of fields names that will be used as the field values sent to Influxdb) and everything else would go to Influxdb as tags.

Thoughts @ph, @jsvd?

GaryRogers commented 9 years ago

Yeah, trying to bring up a new InfluxDB instance, fed by Logstash 1.5 and I ran right into this issue.

gdlx commented 9 years ago

:+1: but why don't you provide a PR if you did it on your fork ?

brupm commented 9 years ago

@contentfree I like that idea and separation strategy, but it may make sense to have 2 arrays one for value_fields and one for tag_fields or something like that so we can control what data is inserted into the series as values and which data is inserted as tags.

gdlx commented 9 years ago

@contentfree I've tried your fork, which works fine except when it inserts multiple points at the same time (same as this issue: https://github.com/influxdb/influxdb/issues/2984). Would you open issues on your repo to help testing and reporting problems ? Anyway, I'll take a look to the code to eventually submit a PR.

contentfree commented 9 years ago

@gauthier-delacroix I've just done so.

I was unable to get logstash -> influxdb working reliably enough to move to production with it, so my immediate needs have waned and I'm fine adding folks as collaborators.

skynet commented 9 years ago

:+1:

tgermain commented 9 years ago

Few note on making this plugin compatible with influxdb 0.9 : the json protocol will be deprecated at some point in favor of the line protocole, which achieve better performance anyway.

It might be good to adapt how data are formatted before being sent.

see https://github.com/influxdb/influxdb/pull/2696

rainerhahnekamp commented 9 years ago

I was able to successfully setup the fork of @contentfree with approximately 200 data points per second. Currently it runs in production for one week without any issues. Neither performance issues, reboots or crashes of some kind - it just runs fine.

Since that fork is already available since May and InfluxDB in version 9 even longer, I really would like to get an answer or statement from @ph or @jsvd on the status quo. The official plugin is more or less not the very useful.

If some work needs to be done, I can spare time to work on issues, etc.

rvrignaud commented 9 years ago

This is also problematic for me. I'd rather avoid running a fork on production. So a big +1 if this could be merged.

bcubk commented 9 years ago

+1 for merge

contentfree commented 9 years ago

Happy to add folks as collaborators. I stopped pursuing influxdb (& logstash) a while ago, though I might come back around to it again (since graphite is being such a PITA to get installed…)