search

cloudflare / goflow

The high-scalability sFlow/NetFlow/IPFIX collector used internally at Cloudflare.
BSD 3-Clause "New" or "Revised" License
859 stars 172 forks source link

Add gonprobe #61

Closed synfinatic closed 4 years ago

synfinatic commented 4 years ago

ntopng is a nice NetFlow reporting tool, but they want 199Euro for their collector which is super awesome but not really reasonable when you just want stats from your $140USD Ubiquiti USG.

This adds support for submitting NetFlow stats over ZMQ using (optionally zlib compressed) JSON for ntopng as well as protobufs since that was trivial.

Tested with ntopng 3.8.2 https://www.ntop.org/products/traffic-analysis/ntop/

Refs: #60

lspgn commented 4 years ago

Hi @synfinatic, I would like to thank you for contributing to GoFlow. While I always appreciate extending the code, I don't think these features should be added to the core, I'm sorry: The way I wrote GoFlow was originally for Kafka+protobuf. After a few iterations, I decoupled the transport + serialization from the actual decoding. Now GoFlow is a flow decoding library, Kafka transport, Protobuf serialization of the packets and a few CLI tools integrating those. The goal of this tool is to provide a simple abstracted format of "flow" over many protocols (sFlow, IPFIX...).

I maintain this project as it's what we use internally. Unfortunately, I don't have time to test, integrate and maintain a complete different framework. I'm happy my code allowed you to create a complete new CLI using ZMQ and JSON (which was the purpose of the new architecture and decoupling the code). But I hope you understand I cannot embed every type of formats and protocols in this repository. It also seems very branded ntopng and restricted to NetFlow.

I believe you have great content, but I would be suggesting to put it in its own repo, while still using the GoFlow libraries.

synfinatic commented 4 years ago

SGTM