search

netsampler / goflow2

High performance sFlow/IPFIX/NetFlow Collector
BSD 3-Clause "New" or "Revised" License
487 stars 112 forks source link

how scale goflow2 for high throughput #314

Closed majid-darvishfard closed 6 months ago

majid-darvishfard commented 6 months ago

hi i use Goflow2 version 1 how can i scale goflow2 for high throughput ?

Currently, 0.1% of the traffic drops I have also set the kernel related queue parameters

i use this command: -metrics.addr ":8084" -reuseport -listen "netflow://10.69.75.162:5678/?count=9,netflow://10.69.75.162:5678/?count=9,netflow://10.69.75.162:5678/?count=9" -transport.kafka.brokers "10.1.48.132:9092,10.1.48.133:9092,10.1.48.134:9092" -transport.kafka.topic "netflow-topici" -transport=kafka -transport=kafka -format.selector TimeFlowStart,TimeFlowEnd,SrcAddr,DstAddr,SrcPort,DstPort,Bytes,Proto -format.hash TimeFlowStart,SrcAddr -transport.kafka.hashing -transport.kafka.compression "gzip"

What is meant by count here? Does it affect performance?

lspgn commented 6 months ago

Hi, Count is the number of sockets open Have you looked at the performance docs? https://github.com/netsampler/goflow2/blob/main/docs/performance.md

majid-darvishfard commented 6 months ago

Hi, Count is the number of sockets open Have you looked at the performance docs? https://github.com/netsampler/goflow2/blob/main/docs/performance.md

I increased the count value to 20, but I did not see any effect The following command listed only one socket cat /proc/net/udp

Should I check somewhere else?

lspgn commented 6 months ago

I'm missing a lot of information to be able to help Did you increase the number of workers? Does htop/uptime show heavy CPU usage?

lspgn commented 6 months ago

A small bug with the count of workers is being fixed https://github.com/netsampler/goflow2/pull/323#issue-2304699338

majid-darvishfard commented 6 months ago

hi @lspgn I checked the COUNT works correctly with this change you made Thank