search

netobserv / flowlogs-pipeline

Transform flow logs into metrics
Apache License 2.0
69 stars 21 forks source link

IPFIX fields missing for newer netobserv features (Connection RTT, DNS decode etc) #544

Open rupertgregoryibm opened 7 months ago

rupertgregoryibm commented 7 months ago

Hello team,

As the main agent has been leaping forward, it seems the IPFIX export feature of FLP needs an update to support export of these fields -

DnsFlags DnsFlagsResponseCode DnsId DnsLatencyMs RTT DstK8S_HostIP DstK8S_HostName DstK8S_Name DstK8S_Namespace DstK8S_OwnerName DstK8S_OwnerType, DstK8S_Type

....there may be (read: probably are) more fields, I've just done brief scan of various MRs over the past few months.

Please can we get the IPFIX export updated. I'm happy to allocate new IPFIX entries if someone can help or point me to the complete schema for all the possible fields and also what each one means.

Many thanks. R.

jotak commented 7 months ago

Some of these fields are indeed probably missing but some of them are not expected: all the "DstK8S..." stuff comes from kube enrichment stage that happens in flowlogs-pipeline ie downstream of the agent.

@msherif1234 we should look for a way to assert that all exporters are always on par with their exported fields ... Like in my current PR here I'm adding assertions that the grpc or direct-flp export produce the same result; maybe we can try to generalize to include ipfix too

rupertgregoryibm commented 7 months ago

Thanks for the quick response @jotak - yes, should have been clearer, theres some upstream fields from the eBPF agent and some FLP enriched fields - all should be consumable irrespective of the export method.. which you've highlighted 👍

I have also let @dushyantbehl know as well as some of this he worked on.

jotak commented 7 months ago

Oh I didn't realize you posted on flowlogs-pipeline repo I thought it was the ebpf agent repo ... it is a bit confusing because both the agent and FLP have the ability to export as ipfix ; so you're using IPFIX export from FLP and not from the Agent, so indeed you should get the kube enriched fields. On the new fields such as DNS / RTT etc. I guess yes they need to be added somewhere on this file https://github.com/netobserv/flowlogs-pipeline/blob/main/pkg/pipeline/write/write_ipfix.go

rupertgregoryibm commented 7 months ago

@jotak is there a master field schema anywhere in the ebpf agent? I assume it would be buried in a protobuf def somewhere?

jotak commented 7 months ago

@rupertgregoryibm yes indeed, here's the protobuf defintion https://github.com/netobserv/netobserv-ebpf-agent/blob/main/proto/flow.proto And here's where it's converted in FLP format: https://github.com/netobserv/netobserv-ebpf-agent/blob/main/pkg/decode/decode_protobuf.go#L44