Sampler and Scaling issue

ntop / ntopng

Web-based Traffic and Security Network Traffic Monitoring

http://www.ntop.org

GNU General Public License v3.0

6.03k stars 639 forks source link

Sampler and Scaling issue #2181

Closed rickguise closed 5 years ago

rickguise commented 5 years ago

Hello. Due to performance issues, I separated my nProbe/nTopng into 2 physical machines. My nProbe is receiving unsampled data, running in promiscuous mode with a packet capture sampling of 1000. My config file is as follows: --zmq="tcp://*:5556" --flow-version 9 -n=none -i=enp0s25 -S=1000:1:1

The problem is that the traffic is showing up as kbits capture_ntopng

as opposed to mbits with no sampling capture_ntopng2

In the nProbe manual it says "All the statistics, including total bytes and packets, will be automatically up-scaled by nprobe to reflect the sample rate" but it is not being reflected in my nTopng. Am I not understanding this or is there another way to send the sampling rate to nTopng?

Thank you for your help.

emanuele-f commented 5 years ago

Hi, you should use the same nprobe configuration as before (#2137) . Only change the --zmq option to send the traffic to the remote ntopng. What is your ntopng config?

rickguise commented 5 years ago

Hi, thanks for the response.

My ntopng config is

-i=tcp://192.168.69.14:5556 -m=192.168.0.0/16,10.0.0.0 -F "mysql;localhost;ntopng;flows;ntopng;xxx"

But isn't the poll mode the better architecture according to the link? https://www.ntop.org/nprobe/advanced-flow-collection-with-ntopng-and-nprobe/

Also, the nProbe guide states that -S=1:1:100 rate is effective when nprobe exports NetFlow towards a downstream collector, that is, when option -n is used.

emanuele-f commented 5 years ago

Using poll or push mode depends on your network config, it's ok like you configured it. In bug #2147 you solved the issue by setting -S 1:100:100, does it fix your issue in this case?

rickguise commented 5 years ago

With -S=1:1000:1000 I'm getting: capture_ntopng3

with -S=1:100:100 capture_ntopng4

and with -S=1:1:1 capture_ntopng5

so the upscalling is not being applied as well.

It's strange because by following the manual it should be -S=1000:1:1

simonemainardi commented 5 years ago

Thanks for reporting. I've fixed an issue on nProbe that was preventing it from properly upscaling sampled traffic as shown in the example.

A new build is in progress. Please, hold on a couple of hours and then update nProbe to the latest 8.7 dev version. Also make sure to have the latest ntopng 3.7 dev.

Correct configuration is the one you've used in the first place:

--zmq="tcp://*:5556"
--flow-version 9
-n=none
-i=enp0s25
-S=1000:1:1

rickguise commented 5 years ago

OK, thank you for the fix! It is working now. I have -S=1000 : 1 : 1 and my ntopng is showing the correct values: capture_ntopng6

simonemainardi commented 5 years ago

thanks for reporting!