search

mguenther / kafka-sampler

Showcases how to build Java applications using Apache Kafka and Kafka Streams
1 stars 0 forks source link

Integrate elasticsearch-river-kafka to push data from Kafka to Elasticsearch #13

Closed mguenther closed 7 years ago

mguenther commented 7 years ago

We need to connect our existing streaming architecture with Elasticsearch. In order to do this, we will use elasticsearch-river-kafka and integrate it with our stream processors / filters.

mguenther commented 7 years ago

Downgraded Elasticsearch to 2.x strain and Kibana to 4.x strain. Both in 5.1.2 did not work properly with the Java client API in the same version. For whatever reason, I was unable to connect with Elasticsearch from Java, although the server is running properly at the correct address (NoNodeAvailableException).

mguenther commented 7 years ago

We will not use elasticsearch-river-kafka since it is outdated as hell. The approach we use now simply uses a Kafka consumer using the Consumer abstraction, pulls data out of a topic and uses a processor implementation that feeds analyzed tweets to Elasticsearch.

mguenther commented 7 years ago

Merged back into development.