search

bpaquet / node-logstash

Simple logstash implmentation in nodejs : file log collection, sent with zeromq
Other
517 stars 141 forks source link

add kafka input/output plugins using kafka-node #96

Open whitedr opened 9 years ago

whitedr commented 9 years ago

I know more is needed for this (like testing...), but this is the beginnings of a kafka input/output plugin. This is using https://github.com/SOHU-Co/kafka-node/ and the HighLevelProducer/HighLevelConsumer interfaces to produce/consume messages to kafka topics. On the output side, I also created a kafka_river output that is the same as the kafka output but formats the messages published to kafka with an JSON envelope that can be indexed into ElasticSearch with the https://github.com/clippPR/elasticsearch-river-kafka ES river. An example of this JSON envelope structure can be seen here: https://github.com/clippPR/elasticsearch-river-kafka/blob/master/src/main/java/org/elasticsearch/river/kafka/JsonMessageHandler.java#L35

One thing I wasn't completely sure of was whether to have the kafka_river as a separate output type or have it be a filter. I ended up following the example of the output_elasticsearch_zeromq.

bpaquet commented 8 years ago

Can you add documentation for this great plugin ?