search

javadmohebbi / goNfCollector

A set of tool to collect & analyze netflow & export them to many DBs & Apps like InfluxDB time-series DB
94 stars 20 forks source link

How to properly configure RAM and CPU limits for containers #6

Closed rodrigocastrillon closed 3 years ago

rodrigocastrillon commented 3 years ago

Hello! First I need to say, this version is far superior than the old one. Thanks for putting effort in this project. I really hope it goes as far as possible.

I have your application in a container processing 161K flows average for each 15min. I have removed the CPU limit from docker and gave a 4vCPU/8G RAM VM to the container. The behavior I see is that the app will use as much as RAM it can. With 2G RAM I get OOM and Docker kills the container. Also, it will between 1.5CPU to 2.5CPU cores with that flow count.

Is there a recommendation for CPU and RAM limits versus flow count being processed by the application?

rodrigocastrillon commented 3 years ago

Hi again. I've just upgraded to 16G RAM, and the app continuously increase RAM usage until the system starts to swap RAM to the disk which causes iowait on the CPU. So far I am monitoring it to compare how many flows it can process with 16G vs 8G.

rodrigocastrillon commented 3 years ago

Hi,

So after some investigation, I updated the default RHEL docker, from 1.13.1, build 64e9980 to 1.13.1, build 7d71120 and also tweaked the IPFIX sample rate, idle and active timers. On the NSX system feeding your app I set the idle timer to 30s, Active to 120s and the sampling-rate to 512. This caused the CPU usage to drop considerably (check the second image). Also, the memory usage seems much better. You can see in image 1 that RAM was quickly consumed causing the iowait in the CPU after some time. However after the changes, RAM is being consumed slower over time.

Despite the changes in both docker and IPFIX config, the docker update immediately reduced the resources usage, while the IPFIX change just reduced the CPU usage a little bit.

Still, the app seems to consume RAM until it stinguishes... Right now I have limited the CPU to 1.5 and RAM to 8G in the container, so docker will probably restart the container once it hits 8G (blue line in the first image.

gonfcol1 First arrow: Docker update; Second: container restart after removing CPU and RAM limits.

gonfcol2 Arrow: Docker update

javadmohebbi commented 3 years ago

Hi @rodrigocastrillon

First of all, I am sorry for the delayed response. And thank you for the great investigations and information.

These recent weeks, I have meetings with some Banks & other organization to test the product to investigate the mentioned RAM usage issue more. As long as I have tested the product on a limited environment, I could not improve it more. But your test is really amazing & I am trying to find alternate environment to provide much better app. In the mean time, I will do my best to provide more updates & improve the features.

New release will support POSTGRES db to store flow metrics for Forensics Investigations. During the test period of postgres netflow-collector app performed better & less resources consumed. I will let you know when the version is fully-ready.

Again, Thank you for the provided information, They will help me to improve the app.

javadmohebbi commented 3 years ago

Hi @rodrigocastrillon,

Again, Thank you for the provided information. I have fix the mentioned IO-wait problem and it cause by a 3rd party Go package which I use to fetch the ASN and AS information. In the next release I will temporary remove this feature to fix the performance issue and I will find another way to fetch Autonomous System information without scarifying system performance.

Thanks again

javadmohebbi commented 3 years ago

Hi @rodrigocastrillon

please update the docker image and the mentioned behavior will be resolved. docker pull javadmohebbi/gonfcollector