[Feature Request] Support writing metrics to Kafka

bk-khaled commented 6 years ago

Hi,

I think it would be great if we can send metrics from snmpcollector to Kafka. Influx is great but this way we can have more flexibility and control over the data flow ...

Best regards

toni-moreno commented 6 years ago

Hi @bk-khaled we would like to add this kind of new features but I would like to understand your use case.

why do you need kafka between snmpcollector and influxdb? how many data ingestion are you currently generating that influxdb is not supporting them?

I think you would be able to do this kind of data transformation with telegraf , did you test it before?

lerela commented 6 years ago

Hi @toni-moreno. After discussing this with @bk-khaled, goal isn't necessarily to address a data ingestion issue. For instance, a few use cases:

need for clustering & redundancy (without InfluxDB Entreprise): Kafka can be clustered & ensure reliable persistence to subscribers,
storing the data elsewhere: the overall architecture would be simpler if the flow is snmpcollector -> kafka -> logstash -> ... (for instance), instead of having snmpcollector -> influxdb -> telegraf -> kafka -> logstash -> ...,
more broadly, Kafka offers interesting features so being able to directly push measurements to topics could allow finer control of the data flow (and of course data could also be pushed to influxdb after kafka, making kafka the message broker & influx a consumer among others)

What are your thoughts?

lerela commented 6 years ago

@toni-moreno another use case is that we are collecting many other, non-snmp metrics and need a message broker to isolate the DB or to feed the stream processing engine (as in most metrics infrastructures I know of); it would make much sense to feed all our metrics into the same solution and process them with the same pipeline.

I understand that you don't have time to devote to this task right now so we might try and implement it, but could you share some insights or implementation tips so that I can assess the feasibility of adding a Kafka sink? eg, what in your opinion would need to be done so that both InfluxDB & Kafka can be supported the "clean" way?

Many thanks.

toni-moreno commented 6 years ago

Hi @lerela . You are right, I'm sorry but I have not time enough , but you are welcomed to add this new feature if you need.

This new feature will need a lot of redesign and refactor the following steps would be a way to do it.

1.- Refactor output configuration config and store.

This means the way as snmpcollector will store config data and let users config them from the web ui the influxdb/kafka or any other kind of backend.

1.1- Refactor the way to store generic other ouput configuration into the database.

right now influxdb is stored with this struct https://github.com/toni-moreno/snmpcollector/blob/master/pkg/config/dbconfig.go#L50-L68

And has I/O API to de database here. https://github.com/toni-moreno/snmpcollector/blob/master/pkg/config/influxcfg.go

1.2- Refactor the webui Angular&Golang

This go file connects with the config.InfluxCFG to load/save data to the database https://github.com/toni-moreno/snmpcollector/blob/master/pkg/webui/apicfg-influxerver.go This directory should be renamed as GenericOutput perhaps. https://github.com/toni-moreno/snmpcollector/tree/master/src/influxserver

2.- Refactor/Design for Output Interfaces

2.1- create an output.Backend interface

 "pkg/agent/output/interface.go"
 "pkg/agent/output/Backend.go"

Be careful , this module has internal statistics that should be maintained for all kind of output backends.

2.2- Adapt the influxdb output to the new interface

 "pkg/agent/output/influxdb.go"

2.2.- Modify the agent to store an array of generic output.Backend instead of output.InfluxDB devices.

https://github.com/toni-moreno/snmpcollector/blob/master/pkg/agent/agent.go#L64

3.- Create your new Kafka Output

 "pkg/agent/output/kalka.go"

Let me know if you need more info to begin to work in this new feature.

lerela commented 6 years ago

Thanks for your detailed answer @toni-moreno. Will look into this and let you know if I have additional questions.

steffenschumacher commented 3 years ago

So I would also like to support the development of this feature, and we will be in touch on the specifics - did anything ever get done? @lerela ?

lerela commented 3 years ago

Hi @steffenschumacher, no. snmpcollector is great but we ended up writing a new tool more suited to our use case (https://github.com/kosctelecom/horus)

toni-moreno / snmpcollector