lspgn
commented
1 year ago Hello @balioune Network flow data can be considered as event/logs whereas Prometheus data is more metrics gauges and counters. In order to represent events as metrics, an aggregation step is required but GoFlow2 does not support it. You need to use additional tools like ClickHouse (aggregation at query time) or write an aggregation layer (GoFlow2 could pipe into it).
The aggregation step is just a bucket for specific fields that sums the bytes of each flow. It's a good practice to often limit the cardinality: if you sum by Source IP address, this often leads to cardinality explosion and tools like Prometheus will not work well. Because GoFlow2 does not always map fields (eg: prefixes or ASNs) that could reduce cardinality, aggregation is not a feature that has been implemented. Additionally, more and more tools now offer good performance when diving into the logs (eg: NewRelic, Grafana Cloud), you may want to explore those options.
Hi all, I have an issue with the goflow2 binary, on the shell output I get full description of the flow but I can't see the same description on Prometheus exporter running on port 8080. Is the a way to have the same description of labels in goflow2 prometheus exporter ?
BR,
Shell Output {"type":"NETFLOW_V9","time_received_ns":1686367216889342072,"sequence_num":2831871748,"sampling_rate":0,"sampler_address":"::ffff.0.70.1","time_flow_start_ns":1686365392000000000,"time_flow_end_ns":1686365492000000000,"bytes":5647,"packets":13,"src_addr":"106.30.36","dst_addr":"10.70.248.42","etype":"IPv4","proto":"TCP","src_port":443,"dst_port":58594,"in_if":64,"out_if":0,"src_mac":"00:00:00:00:00","dst_mac":"00:00:00:00:00:00","src_vlan":0,"dst_vlan":0,"vlan_id":0,"ip_tos":0,"forwarding_status":0,"ip_ttl":0,"_flags":0,"icmp_type":0,"icmp_code":0,"ipv6_flow_label":0,"fragment_id":0,"fragment_offset":0,"src_as":0,"dst_as":0,"next_hop":"1.70.82","next_hop_as":0,"src_net":"0.0.0.0/0","dst_net":"0.0.0.0/0","bgp_next_hop":"","bgp_communities":[],"as_path":[],"mpls_ttl],"mpls_label":[],"mpls_ip":[],"observation_domain_id":0,"observation_point_id":0}
Output from Prometheus Exporter goflow2_flow_traffic_size_bytes{local_ip="::",local_port="2055",remote_ip="::ffff:10.70.57.21",type="netflow",quantile="0.5"} 1393