search

CSIRT-MU / Stream4Flow

A framework for the real-time network traffic analysis based on world-leading technologies for distributed stream processing, network traffic monitoring, and visualization.
https://csirt.muni.cz/?lang=en
MIT License
99 stars 36 forks source link

How does the collector work? #96

Closed oortega closed 5 years ago

oortega commented 5 years ago

Hello

What collector are you using to send the data to kafka?

I currently use https://github.com/bitkeks/python-netflow-v9-softflowd but the CPU is quite high when I run more than 4 collectors

thanks for your project

tomjirsa commented 5 years ago

Hi, more info about the all individual components of our framework can be found at https://stream4flow.ics.muni.cz/.

Regarding the collector, we use https://github.com/CESNET/ipfixcol collector.

oortega commented 5 years ago

thanks @tomjirsa another question, Is there any way to know the IP of the netflow exporter?

tomjirsa commented 5 years ago

know the IP of the netflow exporter

I do not understand the question. Which exporter?

oortega commented 5 years ago

for example IP router that send data to collector. See the image: https://upload.wikimedia.org/wikipedia/commons/9/9e/NetFlow_Architecture_2012.png

tomjirsa commented 5 years ago

You should know the IP of the exporter by default - it is your data source and you have to configure it to send the monitored records to the collector.

oortega commented 5 years ago

yes I know the IP of exporter, but I have 4 exporters and from the collector I would like to know what information comes each exporter

do you know what is called the parameter or variable for the IP exporter? I can not find it in the protocol description, see page 18 https://tools.ietf.org/html/rfc3954

Thanks

tomjirsa commented 5 years ago

Hi, I see your problem. The exporter need to fill in observation domain ID element into flow record (see IANA information elements assignments number 149) Most exporters do this by default. By this element you can differentiate data on collector based on exporter.

oortega commented 5 years ago

Is there something similar in netflow V9?

tomjirsa commented 5 years ago

I am not so familiar with Netflow v9, nevertheless after a quick google search I found, that field Source ID is the field you are looking for. Search RFC3954 for Source ID string.

oortega commented 5 years ago

many thanks @tomjirsa

oortega commented 5 years ago

I am reviewing the document https://www.ietf.org/rfc/rfc3954.txt , but in point 8. Field Type Definitions, I can not find the field Source ID

Do you know if there is a Source ID?

tomjirsa commented 5 years ago

it is in the Netflow v9 packet header. If you need to have it in flow record, you have to use IPFIX instead of Netflow v9.