Server freezing

[melicherm]

Hi Guys, got 2 servers in the role of an firewall (mainly FORWARD chain filtering)

server 1: debian jessie 8.6 - kernel 3.16.0-4-amd64

server 2: debian stretch 9.0 - kernel 4.8.0-1-amd64

The server 1 is an production server, server 2 is for testing purposes.

The thing is, both the servers insta freez (sometimes within 5min, sometimes it lasts some hours -> applying NDPI rules on 20:00 server dead on 04:00) in random intervals (mainly the production one, where the traffic is 200Mbit/s 5min average to 500Mbit/s 5min average). Through the production server goes the traffic of around 500 - 700 client PC's.

1Gbit uplink

The server 1 has much more complicated iptables rules ( ca 1170 in the FORWARD, PREROUTING chain)..., but the server 2 has only few, so the debug is here possible (37 rules in the FORWARD chain, no PREROUTING, some INPUT chain rules just for security reasons).

The test server is much more stable, because the minimal traffic, but freezes are also happening.

_If i apply the dpi_check rules:_ iptables -t mangle -A PREROUTING -m ndpi --dpi_check iptables -t mangle -A POSTROUTING -m ndpi --dpi_check

the servers are insta dead.... nothing in the console, the server is unresponsive, nothing on the screen, nothing in the logs. The only possible thing is to reboot the servers.

Any idea how to debug this behavior? Got the NDPI kernel module build from this GIT.

ii iptables 1.6.0+snapshot20161117-5 ii iptables-dev 1.6.0+snapshot20161117-4 ii iptables-persistent 1.0.4+nmu1 ii xtables-addons-common 2.11-3 ii xtables-addons-dkms 2.11-3

the rules are:

-A FORWARD -m ndpi --applejuice --directconnect --gnutella --edonkey --bittorrent --soulseek -m comment --comment "ndpi checker - dropper" -j DROP

The loopback is enabled: -A INPUT -i lo -j ACCEPT

[rightkick]

Have you tried to accept lo on mangle also?

Example: iptables -t mangle -I PREROUTING -i lo -j ACCEPT iptables -t mangle -I POSTROUTING -i lo -j ACCEPT

Alex

On February 27, 2017 6:05:06 PM EET, melicherm notifications@github.com wrote:

Hi Guys, got 2 servers in the role of an firewall (mainly FORWARD chain filtering)

server 1: debian jessie 8.6 - kernel 3.16.0-4-amd64

server 2: debian sid 9.0 - kernel 4.8.0-1-amd64

The server 1 is an production server, server 2 is for testing purposes.

The thing is, both the servers insta freez (sometimes within 5min, sometimes it lasts some hours -> applying NDPI rules on 20:00 server dead on 04:00) in random intervals (mainly the production one, where the traffic is 200Mbit/s 5min average to 500Mbit/s 5min average). Through the production server goes the traffic of around 500 - 700 client PC's.

1Gbit uplink

The server 1 has much more complicated iptables rules ( ca 1170 in the FORWARD, PREROUTING chain)..., but the server 2 has only few, so the debug is here possible (37 rules in the FORWARD chain, no PREROUTING, some INPUT chain rules just for security reasons).

The test server is much more stable, because the minimal traffic, but freezes are also happening.

_If i apply the dpi_check rules:_ iptables -t mangle -A PREROUTING -m ndpi --dpi_check iptables -t mangle -A POSTROUTING -m ndpi --dpi_check

the servers are insta dead.... nothing in the console, the server is unresponsive, nothing on the screen, nothing in the logs. The only possible thing is to reboot the servers.

Any idea how to debug this behavior? Got the NDPI kernel module build from this GIT.

ii iptables 1.6.0+snapshot20161117-5 amd64
administration tools for packet filtering and NAT ii iptables-dev 1.6.0+snapshot20161117-4 all
transitional dummy package ii iptables-persistent 1.0.4+nmu1 all
boot-time loader for netfilter rules, iptables plugin ii xtables-addons-common 2.11-3 amd64
Extensions targets and matches for iptables [tools, libs] ii xtables-addons-dkms 2.11-3 all
Extensions targets and matches for iptables

the rules are:

-A FORWARD -m ipp2p --edk --dc --gnu --kazaa --bit --apple --soul --winmx --ares -m comment --comment "ipp2p checker - dropper" -j DROP -A FORWARD -m ndpi --applejuice --directconnect --gnutella --edonkey --bittorrent --soulseek -m comment --comment "ndpi checker - dropper" -j DROP

got the IPP2P module from the xtables-addons, and after the IPP2P the NDPI module to DROP the communication that IPP2P cannot detect.

The loopback is enabled: -A INPUT -i lo -j ACCEPT

-- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: https://github.com/betolj/ndpi-netfilter/issues/50

-- To go quickly go alone. To reach far go together

[melicherm]

Going to try it out.

rigth now:

iptables -t filter -S -P INPUT ACCEPT -P FORWARD ACCEPT -P OUTPUT ACCEPT

INPUT (default policy is allow): -A INPUT -i lo -j ACCEPT ... --> some allowing rules ... -A INPUT -j DROP (on the end drop)

FORWARD (default policy is allow): --> some allowing rules ... -A FORWARD -j DROP (on the end drop)

OUTPUT (default policy is allow): nothing here...

iptables -t mangle -S -P PREROUTING ACCEPT -P INPUT ACCEPT -P FORWARD ACCEPT -P OUTPUT ACCEPT -P POSTROUTING ACCEPT -A PREROUTING -i lo -j ACCEPT -A PREROUTING -m ndpi --dpi_check -A POSTROUTING -o lo -j ACCEPT -A POSTROUTING -m ndpi --dpi_check

[melicherm]

@rightkick Alex, but the default policy is accept (the whole mangle table)

Also if i just had the ndpi on the production server (just counting the packets like this):

iptables -I FORWARD -m ndpi --applejuice --directconnect --gnutella --edonkey --bittorrent --soulseek -m comment --comment "ndpi checker"

The server also freezes ( the -m ndpi --dpi_check rules are not applied... just this one rule )

[melicherm]

Ok, tried right now to speed it up:

added:

iptables -I FORWARD -m ndpi --youtube -j DROP

watched youtube maybe 2 minutes and the server crashes over ipmi the login screen was there, no response to keys, going to check the logs after the reboot...

It's the testing server 2 2 posts up are the iptables rules .... dpi_check applied, also the loopback allowed, but default policy on mangle is allow.

Only droping is happening on the FORWARD and INPUT chain in the filter table.

A.D.: Nothing in the logs

[elico]

@melicherm This is the same issue I was having since I tried using this repo. I tried to use i but everytime the machine froze so eventually I started other options.

[melicherm]

@elico - what were your other options? Curious about it.

[elico]

@melicherm zeroshell FW, Ubiquiti EdgeMAX router, and https://github.com/vel21ripn/nDPI. All of the options are not perfect but I didn't managed to make the server iresponsive. I have also seen that ClearOS and\or NethServer have nDPI support but yet to try it.

[rightkick]

Regardless of policy on mangle, all packets are processed. What I am suspecting is that too many packets are checked when including lo traffic. Amend mangle chains to exlude that traffic from ndpi checks. I had encountered similar freezes and was able to resolve them with checking only forwarded traffic.

Alex

On February 28, 2017 11:35:45 AM EET, melicherm notifications@github.com wrote:

@rightkick Alex, but the default policy is accept (the whole mangle table)

-- You are receiving this because you were mentioned. Reply to this email directly or view it on GitHub: https://github.com/betolj/ndpi-netfilter/issues/50#issuecomment-282989676

-- Sent from my Android device with K-9 Mail. Please excuse my brevity.

[melicherm]

What do you mean with that line:

Amend mangle chains to exlude that traffic from ndpi checks. I had encountered similar freezes and was able to resolve them with checking only forwarded traffic.

what i want to have in the -t filter table in the FORWARD chain this rule: iptables -I FORWARD -m ndpi --applejuice --directconnect --gnutella --edonkey --bittorrent --soulseek -m comment --comment "ndpi checker" -j DROP

That is my only wish :).. without freezes, what are you suggesting. Please be specific.

Thank you, Markus.

[rightkick]

Hi,

I thought you mentioned mangle and not filter. In that case put a separate rule per each type of traffic:

-I FORWARD -j ndpi-filter -A FORWARD -j filter-forward -A ndpi-filter -m ndpi --applejuice -j DROP -A ndpi-filter -m ndpi --edonkey -j DROP ... I assume you have a stateful firewall, accepting established sessions. What amount of traffic are you testing? I recommend also to put the rules one by one and testing each one separately.

Check also issue #29 on https://github.com/betolj/ndpi-netfilter, for a similar discussion .

Cheers

On February 28, 2017 11:40:23 PM EET, melicherm notifications@github.com wrote:

What do you mean with that line:

Amend mangle chains to exlude that traffic from ndpi checks. I had encountered similar freezes and was able to resolve them with checking only forwarded traffic.

what i want to have in the -t filter table in the FORWARD chain this rule: iptables -I FORWARD -m ndpi --applejuice --directconnect --gnutella --edonkey --bittorrent --soulseek -m comment --comment "ndpi checker" -j DROP

That is my only wish :).. without freezes, what are you suggesting. Please be specific.

Thank you, Markus.

-- You are receiving this because you were mentioned. Reply to this email directly or view it on GitHub: https://github.com/betolj/ndpi-netfilter/issues/50#issuecomment-283168381

-- Sent from my Android device with K-9 Mail. Please excuse my brevity.

[melicherm]

New info... tried right now:

@rightkick - separating rules like you wrote -t filter iptables -A FORWARD -s XXX.XXX.XXX.XXX/21 -j ndpi_filter iptables -A FORWARD -d XXX.XXX.XXX.XXX/21 -j ndpi_filter

iptables -A FORWARD -s XXX.XXX.XXX.XXX/21 -j ndpi_dropper iptables -A FORWARD -d XXX.XXX.XXX.XXX/21 -j ndpi_dropper

-A ndpi_dropper -m ndpi --gnutella -j DROP -A ndpi_dropper -m ndpi --edonkey -j DROP -A ndpi_dropper -m ndpi --bittorrent -j DROP -A ndpi_filter -m ndpi --gnutella -j SET --add-set torrents src,dst --timeout 7200 -A ndpi_filter -m ndpi --edonkey -j SET --add-set torrents src,dst --timeout 7200 -A ndpi_filter -m ndpi --bittorrent -j SET --add-set torrents src,dst --timeout 7200

-t mangle iptables -A PREROUTING -m ndpi --dpi_check iptables -A POSTROUTING -m ndpi --dpi_check

also specifiying the network in the rules?:

iptables -A PREROUTING -s XXX.XXX.XXX.XXX/21 -m ndpi --dpi_check iptables -A PREROUTING -d XXX.XXX.XXX.XXX/21 -m ndpi --dpi_check iptables -A POSTROUTING -s XXX.XXX.XXX.XXX/21 -m ndpi --dpi_check iptables -A POSTROUTING -d XXX.XXX.XXX.XXX/21 -m ndpi --dpi_check

tried every of this (with DROPing or just counting), also separating the rules like only the FB rule nothing more, only YT, only bittorent, only edonkey, etc... -A ndpi_dropper -m ndpi --gnutella -j DROP -A ndpi_dropper -m ndpi --edonkey -j DROP -A ndpi_dropper -m ndpi --bittorrent -j DROP -A ndpi_dropper -m ndpi --facebook -j DROP -A ndpi_dropper -m ndpi --youtube -j DROP freezes just with facebook && youtube && ndpi_check in mangle

also tried none of the rules, so just the -m ndpi --dpi_check in the mangle table...

Something with the module & kernel somewhere...

the server freezes... @betolj any help from you?

[rightkick]

Hi,

What kernel are you running? I had many panics with 3.2 on Debian 7. I am using now 3.18.36 and has been stable. Not much traffic though is passing through. My device is indended for max 2Mbps. I might be able to simulate high traffic at some point to see stability at that rates.

Alex

On March 5, 2017 2:57:44 PM GMT+02:00, melicherm notifications@github.com wrote:

New info... tried right now:

@rightkick - separating rules

-t mangle iptables -A PREROUTING -m ndpi --dpi_check iptables -A POSTROUTING -m ndpi --dpi_check

also specifiying the network in the rules?:

iptables -A PREROUTING -s XXX.XXX.XXX.XXX/21 -m ndpi --dpi_check iptables -A PREROUTING -d XXX.XXX.XXX.XXX/21 -m ndpi --dpi_check iptables -A POSTROUTING -s XXX.XXX.XXX.XXX/21 -m ndpi --dpi_check iptables -A POSTROUTING -d XXX.XXX.XXX.XXX/21 -m ndpi --dpi_check

tried every of this (with DROPing or just counting), also separate... -A ndpi_dropper -m ndpi --gnutella -j DROP -A ndpi_dropper -m ndpi --edonkey -j DROP -A ndpi_dropper -m ndpi --bittorrent -j DROP -A ndpi_dropper -m ndpi --facebook -j DROP -A ndpi_dropper -m ndpi --youtube -j DROP freezes just with facebook && youtube && ndpi_check in mangle

also tried none of the rules, so just the -m ndpi --dpi_check in the mangle table...

the server freezes... @betolj any help from you?

-- You are receiving this because you were mentioned. Reply to this email directly or view it on GitHub: https://github.com/betolj/ndpi-netfilter/issues/50#issuecomment-284226121

-- Sent from my Android device with K-9 Mail. Please excuse my brevity.

[melicherm]

server 1: debian jessie 8.6 - kernel 3.16.0-4-amd64

server 2: the main tests are done here, the first one is the production one debian stretch 9.0 - kernel 4.8.0-1-amd64

The server should handle up to 1Gbps traffic, but the 5min average right now is 250Mbps

[rightkick]

Have you tested on 3.16.0-4-amd64 kernel?

Also the mangle rules as provided mean that packets are checked twice (pre, post) against ndpi filters. This puts more stress. I would suggest to try first only on filter table, forwarding chain, and on a test system with 3.16 or 3.18.

Alex

On March 6, 2017 7:34:39 PM GMT+02:00, melicherm notifications@github.com wrote:

server 1: debian jessie 8.6 - kernel 3.16.0-4-amd64

server 2: the main tests are done here, the first one is the production one debian stretch 9.0 - kernel 4.8.0-1-amd64

The server should handle up to 1Gbps traffic, but 5min average 250Mbps

-- You are receiving this because you were mentioned. Reply to this email directly or view it on GitHub: https://github.com/betolj/ndpi-netfilter/issues/50#issuecomment-284470760

-- Sent from my Android device with K-9 Mail. Please excuse my brevity.

[melicherm]

On the production one i tested:

only filter table, forward chain, 1 rule... no mangle

iptables -I FORWARD -m ndpi --applejuice --directconnect --gnutella --edonkey --bittorrent --soulseek -m comment --comment "ndpi checker"

Just counting, not dropping or anything -> server freezes 3.16 kernel

applied rule on 20.00 server dead on 04:00 ca, after that i don't do any tests on the production one :D

Will try on the test one different kernel then.

[rightkick]

After testing with persistent high traffic (~ 100Mbit/s) I also encountered kernel panic with 3.18.36.

Using https://github.com/vel21ripn/nDPI/tree/netfilter for 30+hours with same traffic did not panic.