Closed jcdaniel14 closed 5 years ago
What hardware resources have you given to Logstash?
The whole system (kibana-elastic-logstash) is within the same VM which has 24 GB of RAM 12vCPU cores (Xeon 2.7Ghz), I've assigned 6GB RAM to logstash JVM, and 10GB to elastic (since elastic was getting stalled when I set time range above 4 hours), I'm also considering disabling reverse dns resolution since we've had to wait around 15 minutes for flows to be correctly indexed and start appearing on visualizations.
As for storage its 1TB SSD
You are undoubtedly dropping packets because Logstash can't keep up. There are a few things which you can do that will help.
Run sudo sysctl -w net.core.rmem_max=33554432
to provide a bit more buffers for the incoming packets. To make this persistent across reboots, switch to the the ElastiFlow v3.x-dev
branch and grab sysctl.d/87-elastiflow.conf
and put it in /etc/sysctl.d
Edit /etc/systemd/system/logstash.service
, finding this line Nice=19
and changing it to Nice=0
. By default Logstash is started as a low priority process. This KILLS throughput. A nice value of 0 will give it the same priority as normal processes.
Increase the environment variable ELASTIFLOW_NETFLOW_UDP_WORKERS
(or ELASTIFLOW_IPFIX_UDP_WORKERS
if applicable) from the default of 4 to 6 or even 8.
This should greatly increase your throughput (disabling name lookups is also a good idea for now).
Having said all of this, I still think you will struggle to decode 20K flows/sec. https://github.com/robcowart/elastiflow/issues/244
First of all, I really appreciate this tool has helped alot I work for a ISP and I'm receiving around 15k to 20k flows/sec, and currently I'm facing a problem with traffic graphs, so a traffic graph that we know should be around 80Gbps (verified on Cacti), shows ~14 Mbps on Kibana default graphs (no filters).
I know that netflow works by sending samples and I think it's configured to be around 1 out of 1000 samples right now, and I've seen the graphs are calculating traffic aggregating the bytes received on the packets. So I want to know how should I handle them, should I just put a multiplier to compensate or calculate moving average, I've tried both these but resulting graphs seem unreliable. Maybe there's something else I could apply?
As an example I'm showing last day traffic as presented via netflow![image](https://user-images.githubusercontent.com/13785650/52436410-3879e980-2ae2-11e9-85d1-830a6a72a120.png)
vs. same traffic collected via SNMP (Cacti)![image](https://user-images.githubusercontent.com/13785650/52436470-5ba49900-2ae2-11e9-8be8-0343119e50d6.png)