search

robcowart / elastiflow

Network flow analytics (Netflow, sFlow and IPFIX) with the Elastic Stack
Other
2.49k stars 595 forks source link

ELK 6.7 breaks Elastiflow #284

Closed alfredosola closed 5 years ago

alfredosola commented 5 years ago

Hello,

We had a perfectly well behaved Elastic 6.6 stack with Elastiflow.

Upon upgrading everything to 6.7, the netflow data seems to have become corrupted somehow, with a mix of real looking data and obviously corrupted data. See this screenshot for an example. Same happens with IP addresses, etc.

Netflow exporters are Mikrotik boxes and these havent't been touched.

I did upgrade to the latest Elastiflow after upgrading the stack, with the same result.

I think I could be hitting a bug either on Elastiflow or Logstash. Does anybody have an idea on how to dig further?

Captura de pantalla 2019-04-02 a las 10 46 54

alfredosola commented 5 years ago

6.7.1 is out, but we are seeing the same behaviour.

borjam commented 5 years ago

Interestingly I run it on a cluster that went through 6.6.1 and 6.7.0. I updated Elasticsearch and Kibana but I didn't update Logstash.

Everything working fine so far and my Netflow sources are Mikrotik as well.

bluefangs commented 5 years ago

FYI - This seems very similar to the issue I had reported. https://github.com/robcowart/elastiflow/issues/281

However, I run nProbe in order to generate netflow traffic. Some IPv6 traffic used to cause very absurd looking data (HPOPT, invalid IP versions like IPv0, IPv666, jumbled application names, jumbled protocol names etc) I had to put in a filter in logstash to drop ipv6 traffic all together - as it was not required in my use case.

This isn't got anything to do with elastiflow directly. I suspect that if you ingest all the netflow directly into elastic without all the middle level processing, you will still end up with these abnormal flows.

Further, it looks like nprobe has fixed things at it's source which I'm yet to test.

robcowart commented 5 years ago

@alfredosola I would have to see PCAPS of the different flows that you are receiving in order investigate.

alfredosola commented 5 years ago

I have upgraded to 7.0.1 and Elastiflow 3.5 and still see the same. Enclosing captures, please let me know if you need them bigger or filtered.

deleted capture after downloading

robcowart commented 5 years ago

I will have a look. It will likely be this coming weekend.

robcowart commented 5 years ago

Please review this known issue... https://github.com/robcowart/elastiflow/blob/master/KNOWN_ISSUES.md#1-template-conflicts-between-devices

This is often the cause of such issue.