search

betolj / ndpi-netfilter

GNU General Public License v2.0
126 stars 69 forks source link

bittorrent maching is not working properly #25

Closed rsweerarathna closed 8 years ago

rsweerarathna commented 8 years ago

Hi, I need to mark or classify all bittorrent traffic and apply a tc class to shape the traffic. At least, i couldn't drop all the bittorrent packets. Some packets are identified and Dropped, but most of them are cannot be identified and torrent downloads continuously. my rule is as follows.

suddo iptables -A FORWARD -m ndpi --bittorrent -j DROP

kong156 commented 8 years ago

This matter was already discussed. Please check the older topics.

kong156 commented 8 years ago

Please see "kernel panic and build issue w/ current" #8

rsweerarathna commented 8 years ago

yes i read them, for some torrents, it's working fine, but some are goes as unknown traffic. is there anyway to block unknown traffic?

kong156 commented 8 years ago

Please test this particular rule iptables -A FORWARD -m ndpi --bittorrent -j ACCEPT and check if the counters are matching most of the traffic and with what ratio.

rsweerarathna commented 8 years ago

I applied this rule to a particular ip and started downloading a torrent size of 9.82MB. But rule matched very less amount of packets as follows.

Chain INPUT (policy ACCEPT 3294 packets, 215K bytes) pkts bytes target prot opt in out source destination

Chain FORWARD (policy ACCEPT 205K packets, 135M bytes) pkts bytes target prot opt in out source destination
4216 5955K ACCEPT all -- * * 0.0.0.0/0 192.168.200.15 protocol BITTORRENT 3451 214K ACCEPT all -- * * 192.168.200.15 0.0.0.0/0 protocol BITTORRENT

Chain OUTPUT (policy ACCEPT 3281 packets, 543K bytes) pkts bytes target prot opt in out source destination

rsweerarathna commented 8 years ago

It's nearly half of the full size of torrent.

dwipujono commented 8 years ago

Same issue here, only matched 78MB of 744MB, downloading torrent using aria2. But match all with rtorrent.

Using ubuntu 14.04 server, kernel 3.19.0-51.

kong156 commented 8 years ago

Same results with ntopng?

kong156 commented 8 years ago

Please try to connmark the bittorent connections on POSTROUTING, restore on PREROUTING and match your marks on FORWARDING. You could do the same thing with a local bt client and match on INPUT and please compare the results. thanks...

dwipujono commented 8 years ago

Hi Kong,

ntopng shows most of the protocol is Unknown (TCP) when using aria2 (BT 13.3%, Unknown 86.6%). Using rtorrent: BT 97,3%, Others: 2.7%.

I will try connmark later.

dwipujono commented 8 years ago

I test with transmission-cli, ntopng result: BT 54,8%, Unknown: 45,2%

dwipujono commented 8 years ago

Below are the result of connmark bittorrent connections

***** Download torrent using aria2 *****

Chain PREROUTING (policy ACCEPT 571K packets, 842M bytes) pkts bytes target prot opt in out source destination 571K 842M CONNMARK all -- * * 0.0.0.0/0 0.0.0.0/0 CONNMARK restore

Chain POSTROUTING (policy ACCEPT 411K packets, 25M bytes) pkts bytes target prot opt in out source destination 425K 26M CONNMARK all -- * * 0.0.0.0/0 0.0.0.0/0 CONNMARK restore 14162 994K ACCEPT all -- * * 0.0.0.0/0 0.0.0.0/0 mark match ! 0x0 754 95787 MARK all -- * * 0.0.0.0/0 0.0.0.0/0 mark match 0x0protocol BITTORRENT MARK set 0x1 411K 25M CONNMARK all -- * * 0.0.0.0/0 0.0.0.0/0 CONNMARK save

Chain INPUT (policy ACCEPT 555K packets, 819M bytes) pkts bytes target prot opt in out source destination 16021 23M ACCEPT all -- * * 0.0.0.0/0 0.0.0.0/0 mark match 0x1

***** Download torrent using rtorrent *****

Chain PREROUTING (policy ACCEPT 539K packets, 818M bytes) pkts bytes target prot opt in out source destination 539K 818M CONNMARK all -- * * 0.0.0.0/0 0.0.0.0/0 CONNMARK restore

Chain POSTROUTING (policy ACCEPT 17748 packets, 1164K bytes) pkts bytes target prot opt in out source destination 489K 30M CONNMARK all -- * * 0.0.0.0/0 0.0.0.0/0 CONNMARK restore 472K 28M ACCEPT all -- * * 0.0.0.0/0 0.0.0.0/0 mark match ! 0x0 103 12841 MARK all -- * * 0.0.0.0/0 0.0.0.0/0 mark match 0x0protocol BITTORRENT MARK set 0x1 17748 1164K CONNMARK all -- * * 0.0.0.0/0 0.0.0.0/0 CONNMARK save

Chain INPUT (policy ACCEPT 19196 packets, 27M bytes) pkts bytes target prot opt in out source destination 520K 792M ACCEPT all -- * * 0.0.0.0/0 0.0.0.0/0 mark match 0x1

kong156 commented 8 years ago

Same torrent? If so, It looks like ndpi cannot deal with the way aria2 downloads the torrent. Could you set a default DROP rule on INPUT chain and accept only the marked ones?

rsweerarathna commented 8 years ago

I tried with default DROP and allow ndpi marked torrent traffic. But it drops all the packets, because DROP rule is in layer 4 and ndpi marking is at layer7. As Drop rule cannot identify layer 7 marks it DRO everything, So it's not possible to mark torrent traffic at layer4.

kong156 commented 8 years ago

Could you dump the input chain? also make sure that you have a related, established rule set on input chain.

kong156 commented 8 years ago

I've just tried aria2c and yes, this client passes ndpi inspection successfully. I think this is a matter of ndpi library protocol dissection. You could solve this by sending a pcap file to ndpi developers in order to fix this issue, if possible...

rsweerarathna commented 8 years ago

I installed the latest ndpi version from "https://github.com/ntop/nDPI" and run the ndpiReader example. it seems that torrent maching is works fine. Can i compile latest ndpi version with ndpi-netfilter?

freeyoung commented 8 years ago

Same question as @rsweerarathna. I used -j ACCEPT just to make sure if ndpi-netfilter could recognise the BT traffic but no luck. The counters were always 0.

With ndpiReader, torrent matching works perfectly fine.

rsweerarathna commented 8 years ago

Hi bitolj and kong156, nDPI version 1.8 has released. But ndpi-netfilter is still using old version. Are you not going to update the code? I like to contribute to update it with latest nDPI library. I tried to compile with new library but couldn't. It needs some code change, right? If you are not going to update the code, Please give some necessary instructions me to update it.

betolj commented 8 years ago

Hi,

I'm working on it in this week. I can build the binaries now, but I'm reviewing the spinlocks calls.

2016-06-16 3:05 GMT-04:00 rsweerarathna notifications@github.com:

Hi bitolj and kong156, nDPI version 1.8 has released. But ndpi-netfilter is still using old version. Are you not going to update the code? I like to contribute to update it with latest nDPI library. I tried to compile with new library but couldn't. It needs some code change, right? If you are not going to update the code, Please give some necessary instructions me to update it.

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/betolj/ndpi-netfilter/issues/25#issuecomment-226404716, or mute the thread https://github.com/notifications/unsubscribe/AEBma2oSVcuKF4b1oqeW7cXTQrGSTrNaks5qMPW_gaJpZM4HtcWS .

rsweerarathna commented 8 years ago

Good to hear that, im eagerly waiting for it. What are the improvments made in ndpi-netfilter? I mean max no of rules, performance,etc. New nDPI has some new protocols and improved decectors. But what about the performance and scalability?

elico commented 8 years ago

@betolj I was wondering if there is a version of OS and kernel that is not affected by: https://github.com/betolj/ndpi-netfilter/issues/11#issuecomment-226617515 ?

betolj commented 8 years ago

Try the lastest ndpi-netfilter update (v3.0).

betolj commented 8 years ago

The encrypted bittorrent traffic detection does not work well.

betolj commented 8 years ago

Good results with ktorrent without encrypt option. No kernel panics (spin_lock fix).