search

ntop / ntopng

Web-based Traffic and Security Network Traffic Monitoring
http://www.ntop.org
GNU General Public License v3.0
6.26k stars 656 forks source link

Ntopng 2.4.160818-1304.x86_64 - Filtering Not Working #714

Closed garth1985 closed 8 years ago

garth1985 commented 8 years ago

I cant seem to get the filtering at layer 7 to work at all on a Centos 7.2 server. I start ntop with the simplest of options to try eliminate possible issues.

untitled

ntopng -i bridge:p1p1,p1p2 -m "10.0.0.0/24" -G=/var/run/ntopng.pid -e

p1p1 being the external interface. p1p2 being the local LAN interface. The filtering layer 7 options all come up but no matter the protocol, nothing is ever blocked. I've watched the Luca Skype video as well.

Is there something simple that most people miss on install? Please let me know what to provide to assist here.

simonemainardi commented 8 years ago

please, inspect the ntopng logs and post here any possible error you may get. if you can't find the logs, run it in foreground without -e

garth1985 commented 8 years ago

Hi, thanks for the reply. The only errors I see are the following: 25/Aug/2016 15:19:00 [main.cpp:261] ERROR: Unable to store PID in file =/var/run/ntopng.pid 25/Aug/2016 15:19:04 [PacketBridge.cpp:252] ERROR: DISCARD [p1p1] 30:F7:0D:96:52:C1 -> 30:F7:0D:96:52:C1: do you have a loop in the network?

The loop error repeats every 10-15 seconds. A technician is going to check the cables but we are quite sure there is only one cable from the cisco device to the centos box. Could the reporting loop be causing this?

simonemainardi commented 8 years ago

attach the contents of the interface statistics page. I want to make sure that there's actually traffic flowing through the interfaces. That page also reports statistics on the policed packets. Check that out.

garth1985 commented 8 years ago

Hi. Please see below. I feel that it is almost like the system is not showing all the stats or it does not know how to identify the traffic. bridge protocols bridge stats

simonemainardi commented 8 years ago

hi, thanks for posting the screenshots. Apparently, everything looks OK and the bridge is working properly. The fact that shaped and dropped packet counters are at zero seems to suggest that no traffic matching the filtering/shaping criteria has gone through the interfaces. This statement is also enforced by the pie diagrams that don't show any protocol you've blacklisted. Please check that.

garth1985 commented 8 years ago

Hi There My first post shows blacklisting though? Is there another way to check/show this? How come there is so much "unidentified" traffic, is there a way to update the layer 7 protocols? Why does it seem like not all the flows/packets are being displayed/captured? Is there a limit to the number of flows? We have confirmed, there is no way to connect out other than via the bridge of the Ntop. Thanks Garth

simonemainardi commented 8 years ago

Hi,

On Mon, Aug 29, 2016 at 3:27 PM, garth1985 notifications@github.com wrote:

Hi There My first post shows blacklisting though? Is there another way to check/show this?

you can check rules from the interfaces page, menu 'traffic filtering' and 'traffic shaping'

How come there is so much "unidentified" traffic, is there a way to update the layer 7 protocols?

layer7 protocols are detected via nDPI. That is pretty accurate so a large part of unidentified traffic may suggest that there's something to check on the traffic that is directed to ntopng. Also there's a 'no traffic yet' on the TCP so this is unlikely, too. Please check.

Why does it seem like not all the flows/packets are being displaced/captured? Is there a limit to the number of flows?

there's no limit on the number of flows.

We have confirmed, there is no way to connect out other than via the bridge of the Ntop. Thanks Garth

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/ntop/ntopng/issues/714#issuecomment-243123234, or mute the thread https://github.com/notifications/unsubscribe-auth/ADPYH_TAGMLwbepCViGoq4GudKVmtVWhks5qkt44gaJpZM4Js__p .

davewkzn commented 8 years ago

Good day Simone,

Thank you for your feedback to Garth. Sorry for only coming back to you now!

How are you keeping?

I am seeing inconsistent statistics in ntopng but perhaps it is my lack of understanding? Please may I ask for your urgent guidance/assistance? Our setup is very simple. We have ntopng installed on a Linux firewall (CentOS-7.x) that has interface "p1p1" connected to the Internet and interface "p1p2" connected to the customer's LAN. ntopng is running as "ntopng -i bridge:p1p1,p1p2 -m 10.0.0.0/24 -G=/var/run/ntopng.pid -e"

The areas that I require assistance with please are as follows:

Traffic stats - Chart screen When I select "bridge: p1p1,p1p2" from the Interfaces menu, the live traffic stats seem to indicate the correct Ingress and Egress packets. As can been seen in the "interfaces.png" screenshot, there is currently 4.48Mb and 2.08Mb indicated under the "Ingress Packets" column. What is strange is that the traffic on the Interfaces "Chart" page never seems to reflect the same amount of bandwidth. The stats on the Chart page seem to always indicate way less traffic than on the screen from the "Interface" menu. Please see the "chart.png" screenshot attached.

Traffic profiles I've created a couple of traffic profiles. Please see the screenshot "traffic_profiles.png" attached. What is strange is that I am seeing little to no traffic for these traffic profiles - despite there being lots of such traffic going through the system. Please see the screenshot named "traffic_profile_stats" of the traffic profile stats from the "Interfaces-> bridge:p1p1,p1p2"

Traffic filtering I've created a Traffic Filtering Policy for host "10.0.0.252/32" that prevent access to POP3 - please see the "traffic_policy.png" screenshot attached. It seems that this traffic policy is not working as I am able to still establish an outbound POP3 connection to the internet from the host 10.0.0.252. It seems as if the Traffic Filtering Policy is simply not applied.

In summary - it seems that ntopng is aware of all the traffic and is reflecting it on the initial interface summary screen, but seems to lose this traffic when drilling down to the Chart and Traffic Profile stats pages?

Many thanks. Eagerly awaiting your response. traffic_policy traffic_profile_stats traffic_profiles chart interfaces

simonemainardi commented 8 years ago

hi @davewkzn do you think it is possible to have credential for remotely connect to the host? we would like to check that. You can send the credential to mainardi ntop org

thank you

simonemainardi commented 8 years ago

Hi, so I logged into the box and checked that the filtering is working properly: image

I noticed that you set traffic policies for hosts that were not currently active, this is why you thought filtering was not working. I set policies for an active hosts as well as for the whole network 10.0.0.0/8 and everything works as expected.

image

also counter are steadily increasing image

Please, make sure to configure properly

davewkzn commented 8 years ago

Good day Simone.

How are you keeping?

I appreciate your feedback but the changes that you made have sadly made no difference. As mentioned before, the issues are still as follows - please may I ask for your urgent assistance?

1.) Traffic Profiles are still not showing any matched traffic. Ntopng is not showing any SMTP or POP3 traffic in the "Traffic Profiles" page: profiles

2.) The total traffic indicated on the "Chart" page is way less than the traffic shown on the interface bridge summary screen: chart1 summary1

simonemainardi commented 8 years ago

I checked again (for the second time) your system. I can confirm that everything works as expected. I also used tcpdump with the same traffic profiles: not POP3 or SMTP is detected neither by ntopng or by tcpdump.

I checked interfaces configuration and the routing table: it turned out that you assigned IP addresses to the interfaces p1p1 and p1p2 and, in addition, you also configured routes for packets entering/exiting them. So it is very likely that the traffic it is not following the paths expected.

In order to make sure ntopng properly bridges the traffic, do NOT assign ip addresses and routes for the interfaces that are part of the bridge. Ntopng will take care of receiving the packets and forwarding them. Use a third management interface to reach the ntopng host.

davewkzn commented 8 years ago

Good morning Simone,

Thank you for your reply.

I appreciate your assistance.

I checked again (for the second time) your system. I can confirm that everything works as expected. I also used tcpdump with the same traffic profiles: not POP3 or SMTP is detected neither by ntopng or by tcpdump. I am not sure how this is possible. I see lots of SMTP traffic:

[root@mail log]# tcpdump -i p1p1 port 25 -n tcpdump: verbose output suppressed, use -v or -vv for full protocol decode listening on p1p1, link-type EN10MB (Ethernet), capture size 65535 bytes 07:56:42.907476 IP 41.216.129.159.4271 > 10.0.2.2.smtp: Flags [S], seq 3052841299, win 5840, options [mss 1460,sackOK,TS val 828733608 ecr 0,nop,wscale 1], length 0 07:56:42.907497 IP 10.0.2.2.smtp > 41.216.129.159.4271: Flags [S.], seq 1166001665, ack 3052841300, win 28960, options [mss 1460,sackOK,TS val 43389068 ecr 828733608,nop,wscale 7], length 0 07:56:42.928507 IP 41.216.129.159.4271 > 10.0.2.2.smtp: Flags [.], ack 1, win 2920, options [nop,nop,TS val 828733610 ecr 43389068], length 0 07:56:42.997252 IP 10.0.2.2.smtp > 41.216.129.159.4271: Flags [P.], seq 1:47, ack 1, win 227, options [nop,nop,TS val 43389157 ecr 828733610], length 46 07:56:43.018340 IP 41.216.129.159.4271 > 10.0.2.2.smtp: Flags [.], ack 47, win 2920, options [nop,nop,TS val 828733619 ecr 43389157], length 0

I checked interfaces configuration and the routing table: it turned out that you assigned IP addresses to the interfaces p1p1 and p1p2 and, in addition, you also configured routes for packets entering/exiting them. So it is very likely that the traffic it is not following the paths expected. These are our external and internal interfaces respectively - hence the fact that there are IP addresses bound to these interfaces. The p1p1 interface is the default gateway, so all internet-related traffic will be visible on this interface. There are a number of other internal networks and interfaces configured on this system but for troubleshooting purposes we've limited the number of interfaces to p1p1 and p1p2 and are only interested in monitoring traffic that traverses these interfaces.

In order to make sure ntopng properly bridges the traffic, do NOT assign ip addresses and routes for the interfaces that are part of the bridge. Ntopng will take care of receiving the packets and forwarding them. Use a third management interface to reach the ntopng host. Ntopng was installed by my colleague Garth who is no longer at the company - please excuse me if there is something that we haven't configured correctly but I would be most grateful if you could assist me with getting this resolved. Please may I ask you to point me to the appropriate pages in the ntopng user manual that explain what I should be doing here with the interfaces and bridge? I had a quick look at the ntopng user guide but don't see any mention of not assigning IP addresses etc.

Let's assume that I only have two interfaces p1p1 (External internet): 10.0.2.2 and p1p2 (Internal LAN): 10.0.0.254. What do you recommend is the best way for us to configure the ntopng bridge, to monitor all traffic passing through these two interfaces and to also make use of ntopng's shaping and policy functionality ?

Thank you. I promise that I'll stop bugging you as soon as this is resolved :)

is the world's leader in email security and is your best bet against email-based malware and ransomware.

Kind regards,

David Wilson CNS, CLS, LINUX+, CLA, DCTS, LPIC3, RHCSA LinuxTech CC t/a DcData CK number: 2001/058368/23 Website: http://www.dcdata.co.za Support: +27(0)860-1-LINUX Mobile: +27(0)824147413 Tel: +27(0)333446100 Fax: +27(0)866878971

From: "notifications" notifications@github.com To: "ntopng" ntopng@noreply.github.com Cc: "David Wilson" dave@dcdata.co.za, "Mention" mention@noreply.github.com Sent: Monday, 26 September, 2016 18:29:57 Subject: Re: [ntop/ntopng] Ntopng 2.4.160818-1304.x86_64 - Filtering Not Working (#714)

I checked again (for the second time) your system. I can confirm that everything works as expected. I also used tcpdump with the same traffic profiles: not POP3 or SMTP is detected neither by ntopng or by tcpdump.

I checked interfaces configuration and the routing table: it turned out that you assigned IP addresses to the interfaces p1p1 and p1p2 and, in addition, you also configured routes for packets entering/exiting them. So it is very likely that the traffic it is not following the paths expected.

In order to make sure ntopng properly bridges the traffic, do NOT assign ip addresses and routes for the interfaces that are part of the bridge. Ntopng will take care of receiving the packets and forwarding them. Use a third management interface to reach the ntopng host.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub , or mute the thread .

simonemainardi commented 8 years ago

Hi, see below,

On Tue, Sep 27, 2016 at 8:10 AM, davewkzn notifications@github.com wrote:

Good morning Simone,

Thank you for your reply.

I appreciate your assistance.

I checked again (for the second time) your system. I can confirm that everything works as expected. I also used tcpdump with the same traffic profiles: not POP3 or SMTP is detected neither by ntopng or by tcpdump. I am not sure how this is possible. I see lots of SMTP traffic:

[root@mail log]# tcpdump -i p1p1 port 25 -n tcpdump: verbose output suppressed, use -v or -vv for full protocol decode listening on p1p1, link-type EN10MB (Ethernet), capture size 65535 bytes 07:56:42.907476 IP 41.216.129.159.4271 > 10.0.2.2.smtp: Flags [S], seq 3052841299, win 5840, options [mss 1460,sackOK,TS val 828733608 ecr 0,nop,wscale 1], length 0 07:56:42.907497 IP 10.0.2.2.smtp > 41.216.129.159.4271: Flags [S.], seq 1166001665, ack 3052841300, win 28960, options [mss 1460,sackOK,TS val 43389068 ecr 828733608,nop,wscale 7], length 0 07:56:42.928507 IP 41.216.129.159.4271 > 10.0.2.2.smtp: Flags [.], ack 1, win 2920, options [nop,nop,TS val 828733610 ecr 43389068], length 0 07:56:42.997252 IP 10.0.2.2.smtp > 41.216.129.159.4271: Flags [P.], seq 1:47, ack 1, win 227, options [nop,nop,TS val 43389157 ecr 828733610], length 46 07:56:43.018340 IP 41.216.129.159.4271 > 10.0.2.2.smtp: Flags [.], ack 47, win 2920, options [nop,nop,TS val 828733619 ecr 43389157], length 0

I checked interfaces configuration and the routing table: it turned out that you assigned IP addresses to the interfaces p1p1 and p1p2 and, in addition, you also configured routes for packets entering/exiting them. So it is very likely that the traffic it is not following the paths expected. These are our external and internal interfaces respectively - hence the fact that there are IP addresses bound to these interfaces. The p1p1 interface is the default gateway, so all internet-related traffic will be visible on this interface.

if p1p1 is the default GW, this means that it can send also traffic coming from interfaces different that p1p2 as well as from the host.

The bridge you have set up with ntopng is between p1p1 and p1p2. That is, if a packet enters p1p2 ntopng forwards it to p1p1. If a packet enters p1p1, ntopng forwards it to p1p2. ntopng in bridge mode does not policy/shape any other traffic that is different from that. So if the kernel choses p1p2 as default route for traffic originating from the host or from another interface, then ntopng won't see that (in bridge mode)

There are a number of other internal networks and interfaces configured on this system but for troubleshooting purposes we've limited the number of interfaces to p1p1 and p1p2 and are only interested in monitoring traffic that traverses these interfaces.

This is perfectly fine. If you are interested in monitoring use ntopng -i p1p1 -i p1p2 and you will see all the traffic. Bridging is another story as already explained.

In order to make sure ntopng properly bridges the traffic, do NOT assign ip addresses and routes for the interfaces that are part of the bridge. Ntopng will take care of receiving the packets and forwarding them. Use a third management interface to reach the ntopng host. Ntopng was installed by my colleague Garth who is no longer at the company

  • please excuse me if there is something that we haven't configured correctly but I would be most grateful if you could assist me with getting this resolved. Please may I ask you to point me to the appropriate pages in the ntopng user manual that explain what I should be doing here with the interfaces and bridge? I had a quick look at the ntopng user guide but don't see any mention of not assigning IP addresses etc.

Let's assume that I only have two interfaces p1p1 (External internet): 10.0.2.2 and p1p2 (Internal LAN): 10.0.0.254. What do you recommend is the best way for us to configure the ntopng bridge, to monitor all traffic passing through these two interfaces and to also make use of ntopng's shaping and policy functionality ?

I already told you: In order to make sure ntopng properly bridges the traffic, do NOT assign ip addresses and routes for the interfaces that are part of the bridge.

ntopng will do the job of forwarding packets from one interface to the other.

Thank you. I promise that I'll stop bugging you as soon as this is resolved :)

is the world's leader in email security and is your best bet against email-based malware and ransomware.

Kind regards,

David Wilson CNS, CLS, LINUX+, CLA, DCTS, LPIC3, RHCSA LinuxTech CC t/a DcData CK number: 2001/058368/23 Website: http://www.dcdata.co.za Support: +27(0)860-1-LINUX Mobile: +27(0)824147413 Tel: +27(0)333446100 Fax: +27(0)866878971

From: "notifications" notifications@github.com To: "ntopng" ntopng@noreply.github.com Cc: "David Wilson" dave@dcdata.co.za, "Mention" < mention@noreply.github.com> Sent: Monday, 26 September, 2016 18:29:57 Subject: Re: [ntop/ntopng] Ntopng 2.4.160818-1304.x86_64 - Filtering Not Working (#714)

I checked again (for the second time) your system. I can confirm that everything works as expected. I also used tcpdump with the same traffic profiles: not POP3 or SMTP is detected neither by ntopng or by tcpdump.

I checked interfaces configuration and the routing table: it turned out that you assigned IP addresses to the interfaces p1p1 and p1p2 and, in addition, you also configured routes for packets entering/exiting them. So it is very likely that the traffic it is not following the paths expected.

In order to make sure ntopng properly bridges the traffic, do NOT assign ip addresses and routes for the interfaces that are part of the bridge. Ntopng will take care of receiving the packets and forwarding them. Use a third management interface to reach the ntopng host.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub , or mute the thread .

— You are receiving this because you modified the open/close state. Reply to this email directly, view it on GitHub https://github.com/ntop/ntopng/issues/714#issuecomment-249775020, or mute the thread https://github.com/notifications/unsubscribe-auth/ADPYH8_wno1oO696NqsySW6GBjldYWIOks5quLNFgaJpZM4Js__p .

davewkzn commented 8 years ago

Thank you Simone.

I appreciate your feedback. Based on your responses - it sounds like ntopng's "bridging" functionality does actually create an actual layer-2 Ethernet bridge. Is that correct?

is the world's leader in email security and is your best bet against email-based malware and ransomware.

Kind regards,

David Wilson CNS, CLS, LINUX+, CLA, DCTS, LPIC3, RHCSA LinuxTech CC t/a DcData CK number: 2001/058368/23 Website: http://www.dcdata.co.za Support: +27(0)860-1-LINUX Mobile: +27(0)824147413 Tel: +27(0)333446100 Fax: +27(0)866878971

From: "notifications" notifications@github.com To: "ntopng" ntopng@noreply.github.com Cc: "David Wilson" dave@dcdata.co.za, "Mention" mention@noreply.github.com Sent: Tuesday, 27 September, 2016 09:52:10 Subject: Re: [ntop/ntopng] Ntopng 2.4.160818-1304.x86_64 - Filtering Not Working (#714)

Hi, see below,

On Tue, Sep 27, 2016 at 8:10 AM, davewkzn notifications@github.com wrote:

Good morning Simone,

Thank you for your reply.

I appreciate your assistance.

I checked again (for the second time) your system. I can confirm that everything works as expected. I also used tcpdump with the same traffic profiles: not POP3 or SMTP is detected neither by ntopng or by tcpdump. I am not sure how this is possible. I see lots of SMTP traffic:

[root@mail log]# tcpdump -i p1p1 port 25 -n tcpdump: verbose output suppressed, use -v or -vv for full protocol decode listening on p1p1, link-type EN10MB (Ethernet), capture size 65535 bytes 07:56:42.907476 IP 41.216.129.159.4271 > 10.0.2.2.smtp: Flags [S], seq 3052841299, win 5840, options [mss 1460,sackOK,TS val 828733608 ecr 0,nop,wscale 1], length 0 07:56:42.907497 IP 10.0.2.2.smtp > 41.216.129.159.4271: Flags [S.], seq 1166001665, ack 3052841300, win 28960, options [mss 1460,sackOK,TS val 43389068 ecr 828733608,nop,wscale 7], length 0 07:56:42.928507 IP 41.216.129.159.4271 > 10.0.2.2.smtp: Flags [.], ack 1, win 2920, options [nop,nop,TS val 828733610 ecr 43389068], length 0 07:56:42.997252 IP 10.0.2.2.smtp > 41.216.129.159.4271: Flags [P.], seq 1:47, ack 1, win 227, options [nop,nop,TS val 43389157 ecr 828733610], length 46 07:56:43.018340 IP 41.216.129.159.4271 > 10.0.2.2.smtp: Flags [.], ack 47, win 2920, options [nop,nop,TS val 828733619 ecr 43389157], length 0

I checked interfaces configuration and the routing table: it turned out that you assigned IP addresses to the interfaces p1p1 and p1p2 and, in addition, you also configured routes for packets entering/exiting them. So it is very likely that the traffic it is not following the paths expected. These are our external and internal interfaces respectively - hence the fact that there are IP addresses bound to these interfaces. The p1p1 interface is the default gateway, so all internet-related traffic will be visible on this interface.

if p1p1 is the default GW, this means that it can send also traffic coming from interfaces different that p1p2 as well as from the host.

The bridge you have set up with ntopng is between p1p1 and p1p2. That is, if a packet enters p1p2 ntopng forwards it to p1p1. If a packet enters p1p1, ntopng forwards it to p1p2. ntopng in bridge mode does not policy/shape any other traffic that is different from that. So if the kernel choses p1p2 as default route for traffic originating from the host or from another interface, then ntopng won't see that (in bridge mode)

There are a number of other internal networks and interfaces configured on this system but for troubleshooting purposes we've limited the number of interfaces to p1p1 and p1p2 and are only interested in monitoring traffic that traverses these interfaces.

This is perfectly fine. If you are interested in monitoring use ntopng -i p1p1 -i p1p2 and you will see all the traffic. Bridging is another story as already explained.

In order to make sure ntopng properly bridges the traffic, do NOT assign ip addresses and routes for the interfaces that are part of the bridge. Ntopng will take care of receiving the packets and forwarding them. Use a third management interface to reach the ntopng host. Ntopng was installed by my colleague Garth who is no longer at the company

  • please excuse me if there is something that we haven't configured correctly but I would be most grateful if you could assist me with getting this resolved. Please may I ask you to point me to the appropriate pages in the ntopng user manual that explain what I should be doing here with the interfaces and bridge? I had a quick look at the ntopng user guide but don't see any mention of not assigning IP addresses etc.

Let's assume that I only have two interfaces p1p1 (External internet): 10.0.2.2 and p1p2 (Internal LAN): 10.0.0.254. What do you recommend is the best way for us to configure the ntopng bridge, to monitor all traffic passing through these two interfaces and to also make use of ntopng's shaping and policy functionality ?

I already told you: In order to make sure ntopng properly bridges the traffic, do NOT assign ip addresses and routes for the interfaces that are part of the bridge.

ntopng will do the job of forwarding packets from one interface to the other.

Thank you. I promise that I'll stop bugging you as soon as this is resolved :)

is the world's leader in email security and is your best bet against email-based malware and ransomware.

Kind regards,

David Wilson CNS, CLS, LINUX+, CLA, DCTS, LPIC3, RHCSA LinuxTech CC t/a DcData CK number: 2001/058368/23 Website: http://www.dcdata.co.za Support: +27(0)860-1-LINUX Mobile: +27(0)824147413 Tel: +27(0)333446100 Fax: +27(0)866878971

From: "notifications" notifications@github.com To: "ntopng" ntopng@noreply.github.com Cc: "David Wilson" dave@dcdata.co.za, "Mention" < mention@noreply.github.com> Sent: Monday, 26 September, 2016 18:29:57 Subject: Re: [ntop/ntopng] Ntopng 2.4.160818-1304.x86_64 - Filtering Not Working (#714)

I checked again (for the second time) your system. I can confirm that everything works as expected. I also used tcpdump with the same traffic profiles: not POP3 or SMTP is detected neither by ntopng or by tcpdump.

I checked interfaces configuration and the routing table: it turned out that you assigned IP addresses to the interfaces p1p1 and p1p2 and, in addition, you also configured routes for packets entering/exiting them. So it is very likely that the traffic it is not following the paths expected.

In order to make sure ntopng properly bridges the traffic, do NOT assign ip addresses and routes for the interfaces that are part of the bridge. Ntopng will take care of receiving the packets and forwarding them. Use a third management interface to reach the ntopng host.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub , or mute the thread .

— You are receiving this because you modified the open/close state. Reply to this email directly, view it on GitHub https://github.com/ntop/ntopng/issues/714#issuecomment-249775020, or mute the thread https://github.com/notifications/unsubscribe-auth/ADPYH8_wno1oO696NqsySW6GBjldYWIOks5quLNFgaJpZM4Js__p .

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub , or mute the thread .

simonemainardi commented 8 years ago

On Tue, Sep 27, 2016 at 6:49 PM, davewkzn notifications@github.com wrote:

Thank you Simone.

I appreciate your feedback. Based on your responses - it sounds like ntopng's "bridging" functionality does actually create an actual layer-2 Ethernet bridge. Is that correct?

correct

is the world's leader in email security and is your best bet against email-based malware and ransomware.

Kind regards,

David Wilson CNS, CLS, LINUX+, CLA, DCTS, LPIC3, RHCSA LinuxTech CC t/a DcData CK number: 2001/058368/23 Website: http://www.dcdata.co.za Support: +27(0)860-1-LINUX Mobile: +27(0)824147413 Tel: +27(0)333446100 Fax: +27(0)866878971

From: "notifications" notifications@github.com To: "ntopng" ntopng@noreply.github.com Cc: "David Wilson" dave@dcdata.co.za, "Mention" < mention@noreply.github.com> Sent: Tuesday, 27 September, 2016 09:52:10 Subject: Re: [ntop/ntopng] Ntopng 2.4.160818-1304.x86_64 - Filtering Not Working (#714)

Hi, see below,

On Tue, Sep 27, 2016 at 8:10 AM, davewkzn notifications@github.com wrote:

Good morning Simone,

Thank you for your reply.

I appreciate your assistance.

I checked again (for the second time) your system. I can confirm that everything works as expected. I also used tcpdump with the same traffic profiles: not POP3 or SMTP is detected neither by ntopng or by tcpdump. I am not sure how this is possible. I see lots of SMTP traffic:

[root@mail log]# tcpdump -i p1p1 port 25 -n tcpdump: verbose output suppressed, use -v or -vv for full protocol decode listening on p1p1, link-type EN10MB (Ethernet), capture size 65535 bytes 07:56:42.907476 IP 41.216.129.159.4271 > 10.0.2.2.smtp: Flags [S], seq 3052841299, win 5840, options [mss 1460,sackOK,TS val 828733608 ecr 0,nop,wscale 1], length 0 07:56:42.907497 IP 10.0.2.2.smtp > 41.216.129.159.4271: Flags [S.], seq 1166001665, ack 3052841300, win 28960, options [mss 1460,sackOK,TS val 43389068 ecr 828733608,nop,wscale 7], length 0 07:56:42.928507 IP 41.216.129.159.4271 > 10.0.2.2.smtp: Flags [.], ack 1, win 2920, options [nop,nop,TS val 828733610 ecr 43389068], length 0 07:56:42.997252 IP 10.0.2.2.smtp > 41.216.129.159.4271: Flags [P.], seq 1:47, ack 1, win 227, options [nop,nop,TS val 43389157 ecr 828733610], length 46 07:56:43.018340 IP 41.216.129.159.4271 > 10.0.2.2.smtp: Flags [.], ack 47, win 2920, options [nop,nop,TS val 828733619 ecr 43389157], length 0

I checked interfaces configuration and the routing table: it turned out that you assigned IP addresses to the interfaces p1p1 and p1p2 and, in addition, you also configured routes for packets entering/exiting them. So it is very likely that the traffic it is not following the paths expected. These are our external and internal interfaces respectively - hence the fact that there are IP addresses bound to these interfaces. The p1p1 interface is the default gateway, so all internet-related traffic will be visible on this interface.

if p1p1 is the default GW, this means that it can send also traffic coming from interfaces different that p1p2 as well as from the host.

The bridge you have set up with ntopng is between p1p1 and p1p2. That is, if a packet enters p1p2 ntopng forwards it to p1p1. If a packet enters p1p1, ntopng forwards it to p1p2. ntopng in bridge mode does not policy/shape any other traffic that is different from that. So if the kernel choses p1p2 as default route for traffic originating from the host or from another interface, then ntopng won't see that (in bridge mode)

There are a number of other internal networks and interfaces configured on this system but for troubleshooting purposes we've limited the number of interfaces to p1p1 and p1p2 and are only interested in monitoring traffic that traverses these interfaces.

This is perfectly fine. If you are interested in monitoring use ntopng -i p1p1 -i p1p2 and you will see all the traffic. Bridging is another story as already explained.

In order to make sure ntopng properly bridges the traffic, do NOT assign ip addresses and routes for the interfaces that are part of the bridge. Ntopng will take care of receiving the packets and forwarding them. Use a third management interface to reach the ntopng host. Ntopng was installed by my colleague Garth who is no longer at the company

  • please excuse me if there is something that we haven't configured correctly but I would be most grateful if you could assist me with getting this resolved. Please may I ask you to point me to the appropriate pages in the ntopng user manual that explain what I should be doing here with the interfaces and bridge? I had a quick look at the ntopng user guide but don't see any mention of not assigning IP addresses etc.

Let's assume that I only have two interfaces p1p1 (External internet): 10.0.2.2 and p1p2 (Internal LAN): 10.0.0.254. What do you recommend is the best way for us to configure the ntopng bridge, to monitor all traffic passing through these two interfaces and to also make use of ntopng's shaping and policy functionality ?

I already told you: In order to make sure ntopng properly bridges the traffic, do NOT assign ip addresses and routes for the interfaces that are part of the bridge.

ntopng will do the job of forwarding packets from one interface to the other.

Thank you. I promise that I'll stop bugging you as soon as this is resolved :)

is the world's leader in email security and is your best bet against email-based malware and ransomware.

Kind regards,

David Wilson CNS, CLS, LINUX+, CLA, DCTS, LPIC3, RHCSA LinuxTech CC t/a DcData CK number: 2001/058368/23 Website: http://www.dcdata.co.za Support: +27(0)860-1-LINUX Mobile: +27(0)824147413 Tel: +27(0)333446100 Fax: +27(0)866878971

From: "notifications" notifications@github.com To: "ntopng" ntopng@noreply.github.com Cc: "David Wilson" dave@dcdata.co.za, "Mention" < mention@noreply.github.com> Sent: Monday, 26 September, 2016 18:29:57 Subject: Re: [ntop/ntopng] Ntopng 2.4.160818-1304.x86_64 - Filtering Not Working (#714)

I checked again (for the second time) your system. I can confirm that everything works as expected. I also used tcpdump with the same traffic profiles: not POP3 or SMTP is detected neither by ntopng or by tcpdump.

I checked interfaces configuration and the routing table: it turned out that you assigned IP addresses to the interfaces p1p1 and p1p2 and, in addition, you also configured routes for packets entering/exiting them. So it is very likely that the traffic it is not following the paths expected.

In order to make sure ntopng properly bridges the traffic, do NOT assign ip addresses and routes for the interfaces that are part of the bridge. Ntopng will take care of receiving the packets and forwarding them. Use a third management interface to reach the ntopng host.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub , or mute the thread .

— You are receiving this because you modified the open/close state. Reply to this email directly, view it on GitHub https://github.com/ntop/ntopng/issues/714#issuecomment-249775020, or mute the thread https://github.com/notifications/unsubscribe-auth/ADPYH8_ wno1oO696NqsySW6GBjldYWIOks5quLNFgaJpZM4Js__p .

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub , or mute the thread .

— You are receiving this because you modified the open/close state. Reply to this email directly, view it on GitHub https://github.com/ntop/ntopng/issues/714#issuecomment-249925270, or mute the thread https://github.com/notifications/unsubscribe-auth/ADPYHzer9NZoWnDxGzJZi5VBg2ccn5S9ks5quUkLgaJpZM4Js__p .

davewkzn commented 8 years ago

Thank you Simone.

I appreciate you clarifying this and we will modify our client's setup accordingly.

Perhaps we missed it in the Ntopng User Manual. Is it explained in the User Manual or other documentation?

On 27 September 2016 7:26:31 PM SAST, simonemainardi notifications@github.com wrote:

On Tue, Sep 27, 2016 at 6:49 PM, davewkzn notifications@github.com wrote:

Thank you Simone.

I appreciate your feedback. Based on your responses - it sounds like ntopng's "bridging" functionality does actually create an actual layer-2 Ethernet bridge. Is that correct?

correct

is the world's leader in email security and is your best bet against email-based malware and ransomware.

Kind regards,

David Wilson CNS, CLS, LINUX+, CLA, DCTS, LPIC3, RHCSA LinuxTech CC t/a DcData CK number: 2001/058368/23 Website: http://www.dcdata.co.za Support: +27(0)860-1-LINUX Mobile: +27(0)824147413 Tel: +27(0)333446100 Fax: +27(0)866878971

From: "notifications" notifications@github.com To: "ntopng" ntopng@noreply.github.com Cc: "David Wilson" dave@dcdata.co.za, "Mention" < mention@noreply.github.com> Sent: Tuesday, 27 September, 2016 09:52:10 Subject: Re: [ntop/ntopng] Ntopng 2.4.160818-1304.x86_64 - Filtering Not Working (#714)

Hi, see below,

On Tue, Sep 27, 2016 at 8:10 AM, davewkzn notifications@github.com wrote:

Good morning Simone,

Thank you for your reply.

I appreciate your assistance.

I checked again (for the second time) your system. I can confirm that everything works as expected. I also used tcpdump with the same traffic profiles: not POP3 or SMTP is detected neither by ntopng or by tcpdump. I am not sure how this is possible. I see lots of SMTP traffic:

[root@mail log]# tcpdump -i p1p1 port 25 -n tcpdump: verbose output suppressed, use -v or -vv for full protocol decode listening on p1p1, link-type EN10MB (Ethernet), capture size 65535 bytes 07:56:42.907476 IP 41.216.129.159.4271 > 10.0.2.2.smtp: Flags [S], seq 3052841299, win 5840, options [mss 1460,sackOK,TS val 828733608 ecr 0,nop,wscale 1], length 0 07:56:42.907497 IP 10.0.2.2.smtp > 41.216.129.159.4271: Flags [S.], seq 1166001665, ack 3052841300, win 28960, options [mss 1460,sackOK,TS val 43389068 ecr 828733608,nop,wscale 7], length 0 07:56:42.928507 IP 41.216.129.159.4271 > 10.0.2.2.smtp: Flags [.], ack 1, win 2920, options [nop,nop,TS val 828733610 ecr 43389068], length 0 07:56:42.997252 IP 10.0.2.2.smtp > 41.216.129.159.4271: Flags [P.], seq 1:47, ack 1, win 227, options [nop,nop,TS val 43389157 ecr 828733610], length 46 07:56:43.018340 IP 41.216.129.159.4271 > 10.0.2.2.smtp: Flags [.], ack 47, win 2920, options [nop,nop,TS val 828733619 ecr 43389157], length 0

I checked interfaces configuration and the routing table: it turned out that you assigned IP addresses to the interfaces p1p1 and p1p2 and, in addition, you also configured routes for packets entering/exiting them. So it is very likely that the traffic it is not following the paths expected. These are our external and internal interfaces respectively - hence the fact that there are IP addresses bound to these interfaces. The p1p1 interface is the default gateway, so all internet-related traffic will be visible on this interface.

if p1p1 is the default GW, this means that it can send also traffic coming from interfaces different that p1p2 as well as from the host.

The bridge you have set up with ntopng is between p1p1 and p1p2. That is, if a packet enters p1p2 ntopng forwards it to p1p1. If a packet enters p1p1, ntopng forwards it to p1p2. ntopng in bridge mode does not policy/shape any other traffic that is different from that. So if the kernel choses p1p2 as default route for traffic originating from the host or from another interface, then ntopng won't see that (in bridge mode)

There are a number of other internal networks and interfaces configured on this system but for troubleshooting purposes we've limited the number of interfaces to p1p1 and p1p2 and are only interested in monitoring traffic that traverses these interfaces.

This is perfectly fine. If you are interested in monitoring use ntopng -i p1p1 -i p1p2 and you will see all the traffic. Bridging is another story as already explained.

In order to make sure ntopng properly bridges the traffic, do NOT assign ip addresses and routes for the interfaces that are part of the bridge. Ntopng will take care of receiving the packets and forwarding them. Use a third management interface to reach the ntopng host. Ntopng was installed by my colleague Garth who is no longer at the company

  • please excuse me if there is something that we haven't configured correctly but I would be most grateful if you could assist me with getting this resolved. Please may I ask you to point me to the appropriate pages in the ntopng user manual that explain what I should be doing here with the interfaces and bridge? I had a quick look at the ntopng user guide but don't see any mention of not assigning IP addresses etc.

Let's assume that I only have two interfaces p1p1 (External internet): 10.0.2.2 and p1p2 (Internal LAN): 10.0.0.254. What do you recommend is the best way for us to configure the ntopng bridge, to monitor all traffic passing through these two interfaces and to also make use of ntopng's shaping and policy functionality ?

I already told you: In order to make sure ntopng properly bridges the traffic, do NOT assign ip addresses and routes for the interfaces that are part of the bridge.

ntopng will do the job of forwarding packets from one interface to the other.

Thank you. I promise that I'll stop bugging you as soon as this is resolved :)

is the world's leader in email security and is your best bet against email-based malware and ransomware.

Kind regards,

David Wilson CNS, CLS, LINUX+, CLA, DCTS, LPIC3, RHCSA LinuxTech CC t/a DcData CK number: 2001/058368/23 Website: http://www.dcdata.co.za Support: +27(0)860-1-LINUX Mobile: +27(0)824147413 Tel: +27(0)333446100 Fax: +27(0)866878971

From: "notifications" notifications@github.com To: "ntopng" ntopng@noreply.github.com Cc: "David Wilson" dave@dcdata.co.za, "Mention" < mention@noreply.github.com> Sent: Monday, 26 September, 2016 18:29:57 Subject: Re: [ntop/ntopng] Ntopng 2.4.160818-1304.x86_64 - Filtering Not Working (#714)

I checked again (for the second time) your system. I can confirm that everything works as expected. I also used tcpdump with the same traffic profiles: not POP3 or SMTP is detected neither by ntopng or by tcpdump.

I checked interfaces configuration and the routing table: it turned out that you assigned IP addresses to the interfaces p1p1 and p1p2 and, in addition, you also configured routes for packets entering/exiting them. So it is very likely that the traffic it is not following the paths expected.

In order to make sure ntopng properly bridges the traffic, do NOT assign ip addresses and routes for the interfaces that are part of the bridge. Ntopng will take care of receiving the packets and forwarding them. Use a third management interface to reach the ntopng host.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub , or mute the thread .

— You are receiving this because you modified the open/close state. Reply to this email directly, view it on GitHub https://github.com/ntop/ntopng/issues/714#issuecomment-249775020, or mute the thread https://github.com/notifications/unsubscribe-auth/ADPYH8_ wno1oO696NqsySW6GBjldYWIOks5quLNFgaJpZM4Js__p .

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub , or mute the thread .

— You are receiving this because you modified the open/close state. Reply to this email directly, view it on GitHub https://github.com/ntop/ntopng/issues/714#issuecomment-249925270, or mute the thread

https://github.com/notifications/unsubscribe-auth/ADPYHzer9NZoWnDxGzJZi5VBg2ccn5S9ks5quUkLgaJpZM4Js__p .

You are receiving this because you were mentioned. Reply to this email directly or view it on GitHub: https://github.com/ntop/ntopng/issues/714#issuecomment-249936038

Kind Regards, David Wilson DcData 0333446100 0824147413

davewkzn commented 8 years ago

Hi Simone,

I hope you are well.

I've removed the ntopng bridge and configured ntopng to monito the interfaces by specifying them in the configuration file. So far my monitoring is working much better - thank you for your assistance!

From what I understand, I still need to enable ntopng's bridging functionality so that I can make use of ntopng's filtering and traffic management functionality. Understanding that the existing Linux firewall that ntopng is currently installed on is acting as an IP router - what do you recommend is the best way for me to incorporate ntopng's bridge on the existing system so that I may back use of ntopng's filtering and traffic management?

From what you've told me it seems to indicate that ntopng is best installed on a separate physical device that will work as a transparent bridge - is there anyway that I can still implement ntopng's bridge feature on the existing Linux router/firewall without losing the existing routing functionality of the existing server?

simonemainardi commented 8 years ago

On Thu, Sep 29, 2016 at 5:47 PM, davewkzn notifications@github.com wrote:

Hi Simone,

I hope you are well.

I've removed the ntopng bridge and configured ntopng to monito the interfaces by specifying them in the configuration file. So far my monitoring is working much better - thank you for your assistance!

From what I understand, I still need to enable ntopng's bridging functionality so that I can make use of ntopng's filtering and traffic management functionality. Understanding that the existing Linux firewall that ntopng is currently installed on is acting as an IP router - what do you recommend is the best way for me to incorporate ntopng's bridge on the existing system so that I may back use of ntopng's filtering and traffic management?

From what you've told me it seems to indicate that ntopng is best installed on a separate physical device that will work as a transparent bridge - is there anyway that I can still implement ntopng's bridge feature on the existing Linux router/firewall without losing the existing routing functionality of the existing server?

If you have a pair of interfaces that you want to bridge using ntopng, you can't let the kernel use these interfaces for the routing. We've already commented on that.

— You are receiving this because you modified the open/close state. Reply to this email directly, view it on GitHub https://github.com/ntop/ntopng/issues/714#issuecomment-250506048, or mute the thread https://github.com/notifications/unsubscribe-auth/ADPYHx2Nfkb6RK2r-VFbMrWUy19mfTWxks5qu92JgaJpZM4Js__p .