philhagen
commented
7 months ago in what is possibly the worst recorded response time on record (I am sorry for that), I finally have time to re-approach this project and its backlog in earnest.
I think the problem you're seeing had to do with some regression issues involving netflow v9 and its logstash input handler.
- Good news: there is a totally new netflow handler that's part of filebeat, making things faster, easier, and more comprehensive!!!
- Bad news: this means we'll need to take netflow handling to full ECS compatibility, which will also require a re-engineered dashboard, and some added bridge features to allow cross-compatibility between ECS and non-ECS indices.
That all said, this was an eventual necessity and we're just seeing it now instead of later. So I'll get cranking on this, aiming to release it as part of the next public VM in a few weeks.
While looking at the pcap you sent me at the time you reported the issue, I did find a few oddities:
- no port numbers
- duration is always in full seconds - no mills
- there are no tcp flags (for tcp flow records, obviously they'll be zero for UDP)
I'm not sure why this is, but it will definitely lead to some strange results in the index. It's not a blocker to the ECS work noted above, but I wanted to mention it.
I was attending the 6 Jun class at Fort Gordon with Mr. Hagen as our instructor. I set up SOF-ELK to receive netflow v9 from my pfsense Firewall and conducted a tcpdump to view logs coming across. SO-ELK was seeing the logs, however, there was no data being parsed. SOF-ELK only picked up flows and that is it. Please help