search

giros-dit / semantic-data-aggregator

A semantic monitoring framework for aggregating data from heterogeneous sources.
Apache License 2.0
4 stars 0 forks source link

Modification of GoFlow2 Netflow Collector #58

Closed cristinapmz closed 2 years ago

cristinapmz commented 2 years ago

Description

The original GoFlow2 collector outputs only those fields that are common to all the protocols it is capable of processing, such as SFlow, NetFlow-v5, NetFlow-v9 and IPFIX. We need to collect more fields to adapt them to the CSV schema used in PALANTIR for NetFlow-v9 data collected. Therefore, for this purpose, we have to slightly modify and change the code of the original repository.

cristinapmz commented 2 years ago

GoFlow2 original repository has been modified to adapt the output data to the CSV Schema currently in use in PALANTIR. An example of a NetFlow record collected by the new GoFlow2 is shown below:

{"Type":"NETFLOW_V9","TimeReceived":1646383748,"SamplerAddress":"192.168.165.138","SamplingRate":1,"SequenceNum":1156048,"Count":424,"SystemUptime":564854470,"UnixSeconds":1646382841,"SourceId":0,"FlowDirection":0,"TimeFlowStart":1646382486,"TimeFlowEnd":1646382486,"BytesIn":76,"PacketsIn":1,"BytesOut":0,"PacketsOut":0,"Etype":2048,"Proto":17,"SrcAddr":"185.132.136.116","DstAddr":"10.0.27.14","NextHop":"","SrcPort":123,"DstPort":123,"InIf":0,"OutIf":0,"SrcMacIn":"00:00:00:00:00:00","DstMacIn":"00:00:00:00:00:00","SrcMacOut":"00:00:00:00:00:00","DstMacOut":"00:00:00:00:00:00","SrcVlan":0,"DstVlan":0,"VlanId":0,"SrcTos":0,"DstTos":0,"ForwardingStatus":0,"MinTTL":0,"MaxTTL":0,"TCPFlags":0,"IcmpType":0,"IcmpCode":0,"IPv6FlowLabel":0,"IPv6OptionHeaders":0,"FragmentId":0,"FragmentOffset":0,"BiFlowDirection":0,"SrcAS":0,"DstAS":0,"BgpNextHop":"","SrcNet":0,"DstNet":0,"SrcPrefix":"","DstPrefix":"","MPLSPalRd":0,"MPLSPrefixLen":0,"MPLSTopLabelType":0,"MPLSTopLabelIP":"","MPLS1Label":0,"MPLS2Label":0,"MPLS3Label":0,"MPLS4Label":0,"MPLS5Label":0,"MPLS6Label":0,"MPLS7Label":0,"MPLS8Label":0,"MPLS9Label":0,"MPLS10Label":0,"EngineType":0,"EngineId":0,"EtypeName":"IPv4","ProtoName":"UDP","IcmpName":""}
cristinapmz commented 2 years ago

File location of the modified GoFlow2 collector

All files corresponding to the modified Goflow2 collector have been copied to the semantic-data-aggregator repository in the docker directory.

cristinapmz commented 2 years ago

Information about softflowd

Hugo has configured the machine running the softflowd (i.e. the NetFlow exporter) to send Netflow traffic filtered by different transport protocols to the following addresses:

Machine: 192.168.165.36:9995 Protocol: UDP
Machine: 192.168.165.36:9996 Protocol: TCP
Machine: 192.168.159.8:31000 Protocol:UDP

The IP address 192.168.165.36 corresponds to the Openstack VM named VM1-Cristina within the Spider project. Meanwhile, the IP address 192.168.159.8 refers to the IP address of the TID k8s cluster worker node.