Support Netflow v5 - Githubissues

As it stands, we have implemented a subset of netflow v9 and IPFIX.

We would like to go forward with implementing netflow v5.

In collector/logstash/pipeline/ipfix.conf, we do not specify the versions to be processed by logstash, so netflow v5 data should make it into the MongoDB buffer collection. As it stands, we should be seeing the converter write an error out to the log whenever it tries to parse a netflow v5 record. This error should contain the text "unspported netflow version: 5". This error is thrown from FillFromBSONMap in ipfix-rita/converter/input/mgologstash/flow.go.

An investigation of the logstash source code reveals that each of the records produced should have the following fields

    ip4_addr :ipv4_src_addr
    ip4_addr :ipv4_dst_addr
    ip4_addr :ipv4_next_hop
    uint16   :input_snmp
    uint16   :output_snmp
    uint32   :in_pkts
    uint32   :in_bytes
    uint32   :first_switched
    uint32   :last_switched
    uint16   :l4_src_port
    uint16   :l4_dst_port
    uint8    :tcp_flags
    uint8    :protocol
    uint8    :src_tos
    uint16   :src_as
    uint16   :dst_as
    uint8    :src_mask
    uint8    :dst_mask
    uint32   :uptime
    uint32   :unix_sec
    uint32   :unix_nsec
    uint32   :flow_seq_num
    uint8    :engine_type
    uint8    :engine_id
    bit2     :sampling_algorithm
    bit14    :sampling_interval
    uint16   :version

type Flow interface {
    //SourceIPAddress returns the source IPv4 or IPv6 address
    SourceIPAddress() string
    //SourcePort returns the source transport port
    SourcePort() uint16
    //DestinationIPAddress returns the destination IPv4 or IPv6 address
    DestinationIPAddress() string
    //DestinationPort returns the destination transport port
    DestinationPort() uint16
    //ProtocolIdentifier returns which transport protocol was used
    ProtocolIdentifier() protocols.Identifier
    //FlowStartMilliseconds is the time the flow started as a Unix timestamp
    FlowStartMilliseconds() (int64, error)
    //FlowEndMilliseconds is the time the flow ended as a Unix timestamp
    FlowEndMilliseconds() (int64, error)
    //OctetTotalCount returns the total amount of bytes sent (including IP headers and payload)
    OctetTotalCount() int64
    //PacketTotalCount returns the number of packets sent from the source to the destination
    PacketTotalCount() int64
    //FlowEndReason returns why the metering process stopped recording the flow
    FlowEndReason() FlowEndReason
    //Version returns the IPFIX/Netflow version
    Version() uint8
    //Exporter returns the address of the exporting process for this flow
    Exporter() string
}

The mapping between these two looks something like this:

ipv4_src_addr   -> SourceIPAddress
l4_src_port     -> SourcePort
ipv4_dst_addr   -> DestinationIPAddress
l4_dst_port     -> DestinationPort
protocol        -> ProtocolIdentifier # We need to check if netflow v5 uses the same protocol numbers
first_switched  -> FlowStartMilliseconds
last_switched   -> FlowEndMilliseconds
in_bytes        -> OctectTotalCount
in_pkts         -> PacketTotalCount
NilEndReason    -> FlowEndReason # Netflow v5 does not support this information
version         -> Version
host            -> Exporter # Logstash tells us what machine sent it data in the host field of each record

We will need to add a method fillFromNetflowv5BSONMap to ipfix-rita/converter/input/mgologstash/flow.go and dispatch to it based on the version value in the method FillFromBSONMap. It should be rather to base this new method off of fillFromNetflowv9BSONMap.

At the moment, I do not see any reason other changes need to be made. Once we map the incoming BSON data to flow records to flow objects, the data should pass easily through the rest of the program.

activecm / ipfix-rita

Support Netflow v5 #47