Send to Kafka (to simplify InfluxDB maintenance)

[zeugfr]

Hi,

I use Influx Capacitor for several monthes and I am very happy with it. Thanks for this great piece of software !!!

Since our organization if growing, we got some problems with InfluxDB and have sometimes to put it offline for several minutes/hours for maintenance. One of our engineers who is also working in ELK proposed us to implement Kafka (see http://kafka.apache.org/) to put it between our multiple Influx-Capacitors and our InlfuxDBs. It mainly allows us to be able to "bufferize" perfmon data streams before they reach InfluxDB (and, when everything is working well, have the smallest possible delay to reach InfluxDB). It is highly scalable and has very high throughput.

Do you think it is possible to implement a second backend type (to Kafka) in Influx-Capacitor ? Our engineer said that it takes not that much time to write this extension on usual tools (we have others internal projects that we rewrote to ship to Kafka instead of directly to InfluxDB or Elastic). I'am sorry I can't help because I am an op with to few knowledge in development ...

Format submitted could be a standard JSON. Only time and value should be float, others should be strings. For example :

 [{
                "measurement": "sys.cpu",
                "time": 1443085314552,
                "value": 10.0,
                "tags": {
                    "collector": "influx-capacitor",
                    "fqdn": "my-test-hostname",
                    "ecosystem": "myapp",
                    "role": "mssql",
                    "os": "Windows",
                    "osrelease": "2012R2",
                    "unit": "percent",
                    "category1": "user",
                     "core": "0"
                }
},
{
                "measurement": "sys.cpu",
                "time": 1443085314552,
                "value": 15.43,
                "tags": {
                    "collector": "influx-capacitor",
                    "fqdn": "my-test-hostname",
                    "ecosystem": "myapp",
                    "role": "mssql",
                    "os": "Windows",
                    "osrelease": "2012R2",
                    "unit": " percent ",
                    "category1": "idle",
                     "core": "1"
                }
}]

Tell me what you think of that idea ! Thanks.

[poxet]

It sounds like a good idea. I am not at all familliar with kafka, can you send json-strings in any format to kafka to be relayed?

[poxet]

I just found a github project; https://github.com/Jroland/kafka-net with a nuget package; https://www.nuget.org/packages/kafka-net/ and after reading about kafka it seems very straight forward. Perhaps this will be a good project for the holidays. :)

[poxet]

Now there is a first version for Kafka. Download it here and try it out. http://influx-capacitor.com/Resources/Continuous/Influx-Capacitor.1.0.15.61.msi

[zeugfr]

Great ! I'll have a look at it later this week ... Many thanks, i'll keep you informed !

[poxet]

It does not post json data, it uses the same format as influx. If needed, json output can be added too.

[framiere]

Hi, thanks for this feature.

as of for now using Influx-Capacitor.1.0.17.msi I can see it working only for the first batch.

Here's is my configuration

<Application>
    <FlushSecondsInterval>5</FlushSecondsInterval>
    <MaxQueueSize>100</MaxQueueSize>
</Application>

With this database

<Database Type="Kafka" Enabled="true">
    <Url>http://kafka:9092</Url>
</Database>

When the logs are activated I can see something that looks like it's all good

DEBUG - Enqueueing 7 points to server http://kafka:9092 database Kafka.
...snip snip...
DEBUG - Enqueueing 2 points to server http://kafka:9092 database Kafka.
DEBUG - Enqueueing 7 points to server http://kafka:9092 database Kafka.
DEBUG - Starting to send 100 points to server http://kafka:9092 database Kafka.
DEBUG - Done sending 100 points to server http://kafka:9092 database Kafka. Now 0 items in queue.
DEBUG - Starting to send 1 points to server http://kafka:9092 database Kafka.
DEBUG - Enqueueing 5 points to server http://kafka:9092 database Kafka.
DEBUG - Enqueueing 5 points to server http://kafka:9092 database Kafka.
DEBUG - Done sending 1 points to server http://kafka:9092 database Kafka. Now 0 items in queue.
DEBUG - Enqueueing 7 points to server http://kafka:9092 database Kafka.
DEBUG - Enqueueing 7 points to server http://kafka:9092 database Kafka.
...snip snip...
DEBUG - Enqueueing 2 points to server http://kafka:9092 database Kafka.
DEBUG - Enqueueing 7 points to server http://kafka:9092 database Kafka.
DEBUG - Starting to send 99 points to server http://kafka:9092 database Kafka.
DEBUG - Done sending 99 points to server http://kafka:9092 database Kafka. Now 0 items in queue.
...

However when I consume the kafka topic I can see only the first batch arriving

./kafka-console-consumer.sh --zookeeper kafka/kafka --topic InfluxCapacitor

This behavior is the same when I fiddle around with the MaxQueueSize and FlushSecondsInterval.

Do you have an idea what I am missing ?

Thanks !

[poxet]

I am not sure what the problem could be. I will try to have a look at it. Please get back to me if you find any other clues.