search

neptune-networks / flow-exporter

Export network flows from Kafka to Prometheus
https://brooks.sh/2019/11/17/network-flow-analysis-with-prometheus/
MIT License
123 stars 20 forks source link

src_port, dst_port, protocol details #13

Open vishnubraj opened 4 years ago

vishnubraj commented 4 years ago

Hi,

Is it possible to add src_port, dst_port, protocol labels also to the metrics?

bswinnerton commented 4 years ago

Hi @vishnubraj,

It's definitely possible, but not something that I would recommend. By adding source and destination ports, it would result in very high cardinality metrics (you'd have a unique combination of every label type). This not only would slow down queries but would significantly increase the amount of data stored in Prometheus.

If you were to do this, you would need to first export the data from pmacct, with something like this in your pmacctd.conf

aggregate: src_host, dst_host, src_port, dst_port, src_as, dst_as, label, src_port, dst_port

(other options are laid out here)

And then the following sections of the Kafka consumer would need to be updated:

https://github.com/neptune-networks/flow-exporter/blob/e8e2114af60536460f836e2130f2ad50c654a9d3/internal/consumer/consumer.go#L17-L33

https://github.com/neptune-networks/flow-exporter/blob/e8e2114af60536460f836e2130f2ad50c654a9d3/internal/consumer/consumer.go#L35-L42

https://github.com/neptune-networks/flow-exporter/blob/e8e2114af60536460f836e2130f2ad50c654a9d3/internal/consumer/consumer.go#L127-L147

vishnubraj commented 4 years ago

Thanks for the clear explanation @bswinnerton I agree with the increase in the data storage part. I will check if I can use it without any performance degradation.

tf3t commented 3 years ago

It would also be a nice addition (I'm testing it) to add tags like iface_in, iface_out - this would open up more granularity of distinguishing what provider the traffic is coming from (e.g. peering vs transit) I'm looking into if the tags can be used in pmacct. and also multi-as, in my use case I'm monitoring two different AS'es. I solved it by running two exporters - but I'm going to try to modify so multiple AS'es are supported. (perhaps a special case).