search

robcowart / elastiflow

Network flow analytics (Netflow, sFlow and IPFIX) with the Elastic Stack
Other
2.48k stars 596 forks source link

Input from Kafka topic #686

Closed punisherVX closed 3 years ago

punisherVX commented 3 years ago

I was going to ask this on the slack channel, but I can't get connected because it states you need an @elastiflow.com email address. So, asking it here: Is there, either in the Logstash or new Unified Collector, a way to read IPFIX from a kafka topic rather than having it streamed directly to logstash/EUC respectively? I have looked everywhere I can think of and it doesn't seem to be the case, but wanted to verify.

Thanks.

robcowart commented 3 years ago

First, can you please retry the Slack invite link. It should be fixed now.

The new ElastiFlow collector does not currently support a Kafka input. However our future plans, will include Kafka and other streaming platform options, to further increase scale (millions of flows per second) and enable more advanced analytics.

Logstash has a Kafka input however to get any data that might be consumed from Kafka to be processed by the rest of the ElastiFlow pipeline will require a bit of normalization/conversion.

Can you explain your use-case a bit more? Why do you need to consume from Kafka? How did the data get into Kafka to begin with?

punisherVX commented 3 years ago

Thanks for the quick reply.

The use case for Kafka is we are standardizing on pushing all streaming data from our endpoints to Kafka so that we have one place for all consumers to retrieve data as well as have some historical archive for replay, data mining, etc. I can get our syslog in through Logstash/Filebeat using the kafka input plug-in but didn't think I could do it with Elastiflow.
One of the main reasons for asking is we are trying to figure out what form it needs to be in when/if it is pulled by a generic consumer (i.e. logstash kafka plug-in) and what we can do to strip off any metadata that is added. In our example this would probably be the JSON format from CloudEvents spec: https://github.com/cloudevents/spec/blob/v1.0.1/json-format.md

I have retried the slack login, but still get the same error:
image

robcowart commented 3 years ago

Please try this link... https://join.slack.com/t/elastiflowcommunity/shared_invite/zt-lv54rhcx-7esE8r8cqggE5mQlShftpA

The readme here is now updated with the new link. Sorry about that.

punisherVX commented 3 years ago

Please try this link... https://join.slack.com/t/elastiflowcommunity/shared_invite/zt-lv54rhcx-7esE8r8cqggE5mQlShftpA

The readme here is now updated with the new link. Sorry about that.

No problem. The new link worked!!

Would you like me to take this to slack in the #feature_requests channel or a different one? We can also keep it here if you prefer.

robcowart commented 3 years ago

I sent you a DM on Slack, but I am also happy to discuss in the #feature_requests channel there.