search

OpenSOC / opensoc-ui

User interface for OpenSOC
Apache License 2.0
100 stars 55 forks source link

Kafka integration #6

Closed jamilbk closed 9 years ago

jamilbk commented 10 years ago

Backend component of real-time alerts

jamilbk commented 10 years ago

The way this could work is as follows:

  1. Register an ES JSON query to a mock devo http api
  2. Kafka topic will be id or name of this query
  3. Expose a websockets endpoint through Node referencing the kafka topic

So the Kibana will consume alerts from this endpoint with a socket.io connection to something like http://localhost:5000/search?realtime=true&query=foobar. The query parameter will uniquely reference which kafka topic to stream alerts from. We can probably persistent this in Kafka itself, so when a dashboard is created, the query is converted to Elasticsearch JSON on the fly, a kafka topic is created or used (if exists), then the websockets endpoint is created and the Kibana will be able to stream results from it.

rammie commented 9 years ago

Stale.