cloudflare / goflow

The high-scalability sFlow/NetFlow/IPFIX collector used internally at Cloudflare.
BSD 3-Clause "New" or "Revised" License
859 stars 172 forks source link

Question: Flow field support? #19

Closed 5tingray closed 5 years ago

5tingray commented 5 years ago

Hi

Please do correct me if I'm wrong but it doesn't appear I can export NetFlow fields that aren't available across all the NetFlow protocols. For example, in my case I'm dealing with IPFIX biflows and require fields specific to IPFIX such as 'biflowDirection'. Is there a way to request theses fields in the protobuff definition or is this a feature i could request?

Thanks!

lspgn commented 5 years ago

Hello, yes only some fields are converted into the protobuf. Feel free to fork it in the meantime. It is possible to add them. Ideally it's better if they are present in multiple versions.

Out of curiosity, what is the use case of this field? What equipments are sending it?

5tingray commented 5 years ago

Thanks for getting back to me so quickly :) It's for a security project in which it's important to know who initiated the connection. I'll be receiving flows from traditional routers and switches but they'll be configured to export in biflow format (rfc5103).

lspgn commented 5 years ago

I don't think I have a compatible devices for testing, do you have samples (pcap)?

5tingray commented 5 years ago

For testing I'm using nprobe to generate IPFIX flows. There is a free version of nprobe available.

sudo nprobe -i eth0 -n 192.168.1.102:2055 -V 10 -T "%PROTOCOL %IN_SRC_MAC %OUT_DST_MAC %IP_PROTOCOL_VERSION %IPV4_SRC_ADDR %IPV4_SRC_MASK %IPV4_DST_ADDR %IPV4_DST_MASK %L4_SRC_PORT %L4_DST_PORT %TOTAL_PKTS_EXP %TOTAL_BYTES_EXP %SRC_TOS %DIRECTION %FLOW_START_MILLISECONDS %FLOW_END_MILLISECONDS %BIFLOW_DIRECTION"

Here's a pcap: https://file.io/ZeNZoM

lspgn commented 5 years ago

@5tingray sorry for the delay, did not have time to look into it, could you send the file again?

lspgn commented 5 years ago

I just made a pre-release https://github.com/cloudflare/goflow/releases/tag/v3.0.0.0 which integrate biflow direction

5tingray commented 5 years ago

Thank you that's awesome! Sorry for not getting back to you before. I'll get testing this now and get back to you.