Detection of SSL traffic

Sacriyana commented 7 years ago

Hi everyone

I have tried to detect SSL traffic but ndpi detect nothing when I go on https website. Yet nDPI in userland detect it. I have tried to debug the SSL files (whith some pr_debug) but I have nothing in my kern.log whereas I have the debug that i added in /src/main.c I have tired to understand how ndpi work to launch the function in each protocol dissector but I don't find. But, the specific SSL like Skype, Microsoft, Youtube, etc ... work. Do you have any ideas to debug the SSL detection (end so to have the host name server) ?

Moreover, do we need to compil and install nDPI ? Because we have to re-compile it for kernel module.

betolj commented 7 years ago

Hi Sacriyana,

In which chain you wrote the firewall filter? To detect some protocols, like SSL, it's necessary that all packets are inspected by ndpi (in all directions - inbound and outbound flows).

Sacriyana commented 7 years ago

Hi,

I have tried this : iptables -t mangle -A PREROUTING -m ndpi --dpi_check iptables -t mangle -A POSTROUTING -m ndpi --dpi_check
iptables -A INPUT -m ndpi --ssl -j ACCEPT iptables -A OUTPUT -m ndpi --ssl -j ACCEPT

And i confirm, I went on website which use https, and it didn't detect it, (server port 443). Yet it detected few packet in SSL for the same website.

I'm working on a 16.04

betolj commented 7 years ago

Strange ... it's works for me. There are some other iptables rules prior the DPI inspection?

Try debug xt_ndpi mode

Flush your firewall rules to remove xt_ndpi module
Remove modules with rmmod command: rmmod xt_ndpi
Load the firewall module with debug option: modprobe xt_ndpi debug_dpi=1
Define your firewall rules again

tail -f /var/log/syslog Oct 14 12:53:04 humberto-XPS-8300 kernel: [ 844.662670] xt_ndpi: flow detected GOOGLE ( dst 74.125.141.189 ) Oct 14 12:53:04 humberto-XPS-8300 kernel: [ 844.662736] xt_ndpi: flow detected GOOGLE ( dst 74.125.141.189 ) Oct 14 12:53:05 humberto-XPS-8300 kernel: [ 845.206317] xt_ndpi: flow detected SSL ( dst 151.101.56.133 ) Oct 14 12:53:05 humberto-XPS-8300 kernel: [ 845.380578] xt_ndpi: flow detected SSL ( dst 151.101.56.133 ) Oct 14 12:53:05 humberto-XPS-8300 kernel: [ 845.380596] xt_ndpi: flow detected SSL ( dst 151.101.56.133 ) Oct 14 12:53:15 humberto-XPS-8300 kernel: [ 855.381917] xt_ndpi: flow detected SSL ( dst 151.101.56.133 ) Oct 14 12:53:15 humberto-XPS-8300 kernel: [ 855.529882] xt_ndpi: flow detected SSL ( dst 151.101.56.133 ) Oct 14 12:53:15 humberto-XPS-8300 kernel: [ 855.529901] xt_ndpi: flow detected SSL ( dst 151.101.56.133 ) Oct 14 12:53:17 humberto-XPS-8300 kernel: [ 857.478079] xt_ndpi: flow detected SSL ( dst 151.101.56.133 ) Oct 14 12:53:25 humberto-XPS-8300 kernel: [ 865.297458] xt_ndpi: flow detected GOOGLE ( dst 173.194.217.118 ) Oct 14 12:53:25 humberto-XPS-8300 kernel: [ 865.697510] xt_ndpi: flow detected GOOGLE ( dst 173.194.217.118 )

Sacriyana commented 7 years ago

Hi, sometimes I have the behavior, with specific protocol like GOOGLE or MICROSOFT, I haven't any trouble, but with some SSL website, sometime it doesn't categorize the flow :+1:

(I change the destination IP because it's my IP's server), I have a gitlab (in https) and I went on this page for the test.

I put the "protocol detected XXX" after ndpi_process_packet. I add a id (incremental) for each flow and add the associated port.

I think there is a problem with the condition of the end of a flow, particular when the end comes from a TCP->FIN or TCP->RST, because we lost the information about the SSL certificate that we found at the beginning of the session. I try to see the problem on Wireshark. I think that when we saw a TCP-RST or TCP->FIN, it can't find anymore the flow the answer of the server is categorized as Unknown.

nf_conntrack version 0.5.0 (65536 buckets, 262144 max) xt_ndpi 3.0 (nDPI wrapper module). xt_ndpi: TCP->SYN | | protocol detected Unknown (dst SAME_DST_IP) xt_ndpi: TCP->SYN | | protocol detected Unknown (dst SAME_DST_IP) xt_ndpi: TCP->SYN | | protocol detected Unknown (dst SAME_DST_IP) xt_ndpi: TCP->SYN | | protocol detected Unknown (dst SAME_DST_IP) xt_ndpi: flow detected id=30 (58990 - 443) | | protocol detected Unknown (dst SAME_DST_IP) xt_ndpi: flow detected id=30 (58990 - 443) | | protocol detected Unknown (dst SAME_DST_IP) xt_ndpi: flow detected id=30 (58990 - 443) | | protocol detected SSL (dst SAME_DST_IP) xt_ndpi: flow detected id=30 (58990 - 443) | | protocol detected SSL (dst SAME_DST_IP) xt_ndpi: flow detected id=30 (443 - 58990) | | protocol detected SSL (dst SAME_DST_IP) xt_ndpi: flow detected id=30 (443 - 58990) | | protocol detected SSL (dst SAME_DST_IP) xt_ndpi: flow detected id=30 (58990 - 443) | | protocol detected SSL (dst SAME_DST_IP) xt_ndpi: TCP->SYN | | protocol detected Unknown (dst SAME_DST_IP) xt_ndpi: TCP->SYN | | protocol detected Unknown (dst SAME_DST_IP) xt_ndpi: TCP->SYN | | protocol detected Unknown (dst SAME_DST_IP) xt_ndpi: TCP->SYN | | protocol detected Unknown (dst SAME_DST_IP) xt_ndpi: TCP->SYN | | protocol detected Unknown (dst SAME_DST_IP) xt_ndpi: flow detected id=43 (58992 - 443) | | protocol detected Unknown (dst SAME_DST_IP) xt_ndpi: flow detected id=43 (58992 - 443) | | protocol detected Unknown (dst SAME_DST_IP) xt_ndpi: flow detected id=30 (443 - 58990) | | protocol detected SSL (dst SAME_DST_IP) xt_ndpi: flow detected id=30 (443 - 58990) | | protocol detected SSL (dst SAME_DST_IP) xt_ndpi: flow detected id=43 (58992 - 443) | | protocol detected SSL (dst SAME_DST_IP) xt_ndpi: flow detected id=43 (58992 - 443) | | protocol detected SSL (dst SAME_DST_IP) xt_ndpi: TCP->SYN | | protocol detected Unknown (dst SAME_DST_IP) xt_ndpi: TCP->SYN | | protocol detected Unknown (dst SAME_DST_IP) xt_ndpi: flow detected id=44 (58994 - 443) | | protocol detected Unknown (dst SAME_DST_IP) xt_ndpi: flow detected id=44 (58994 - 443) | | protocol detected Unknown (dst SAME_DST_IP) xt_ndpi: TCP->SYN | | protocol detected Unknown (dst SAME_DST_IP) xt_ndpi: TCP->SYN | | protocol detected Unknown (dst SAME_DST_IP) xt_ndpi: flow detected id=44 (58994 - 443) | | protocol detected SSL (dst SAME_DST_IP) xt_ndpi: flow detected id=44 (58994 - 443) | | protocol detected SSL (dst SAME_DST_IP) xt_ndpi: flow detected id=43 (443 - 58992) | | protocol detected SSL (dst SAME_DST_IP) xt_ndpi: flow detected id=43 (443 - 58992) | | protocol detected SSL (dst SAME_DST_IP) xt_ndpi: TCP->SYN | | protocol detected Unknown (dst SAME_DST_IP) xt_ndpi: TCP->SYN | | protocol detected Unknown (dst SAME_DST_IP) xt_ndpi: flow detected id=45 (58996 - 443) | | protocol detected Unknown (dst SAME_DST_IP) xt_ndpi: flow detected id=45 (58996 - 443) | | protocol detected Unknown (dst SAME_DST_IP) xt_ndpi: flow detected id=45 (58996 - 443) | | protocol detected SSL (dst SAME_DST_IP) xt_ndpi: flow detected id=45 (58996 - 443) | | protocol detected SSL (dst SAME_DST_IP) xt_ndpi: flow detected id=43 (443 - 58992) | | protocol detected SSL (dst SAME_DST_IP) xt_ndpi: flow detected id=43 (443 - 58992) | | protocol detected SSL (dst SAME_DST_IP) xt_ndpi: flow detected id=43 (58992 - 443) | | protocol detected SSL (dst SAME_DST_IP) xt_ndpi: flow detected id=43 (58992 - 443) | | protocol detected SSL (dst SAME_DST_IP) xt_ndpi: flow detected id=44 (443 - 58994) | | protocol detected SSL (dst SAME_DST_IP) xt_ndpi: flow detected id=44 (443 - 58994) | | protocol detected SSL (dst SAME_DST_IP) xt_ndpi: flow detected id=43 (58992 - 443) | | protocol detected SSL (dst SAME_DST_IP) xt_ndpi: flow detected id=43 (58992 - 443) | | protocol detected SSL (dst SAME_DST_IP) xt_ndpi: flow detected id=43 (58992 - 443) | | protocol detected SSL (dst SAME_DST_IP) xt_ndpi: flow detected id=44 (58994 - 443) | | protocol detected SSL (dst SAME_DST_IP) xt_ndpi: flow detected id=44 (58994 - 443) | | protocol detected SSL (dst SAME_DST_IP) xt_ndpi: flow detected id=30 (443 - 58990) | | protocol detected SSL (dst SAME_DST_IP) xt_ndpi: flow detected id=30 (443 - 58990) | | protocol detected SSL (dst SAME_DST_IP) xt_ndpi: flow detected id=30 (58990 - 443) | | protocol detected SSL (dst SAME_DST_IP) xt_ndpi: flow detected id=44 (443 - 58994) | | protocol detected SSL (dst SAME_DST_IP) xt_ndpi: flow detected id=44 (443 - 58994) | | protocol detected SSL (dst SAME_DST_IP) xt_ndpi: flow detected id=44 (443 - 58994) | | protocol detected SSL (dst SAME_DST_IP) xt_ndpi: flow detected id=44 (58994 - 443) | | protocol detected SSL (dst SAME_DST_IP) xt_ndpi: flow detected id=30 (443 - 58990) | | protocol detected SSL (dst SAME_DST_IP) xt_ndpi: flow detected id=30 (443 - 58990) | | protocol detected SSL (dst SAME_DST_IP) xt_ndpi: flow detected id=30 (58990 - 443) | | protocol detected SSL (dst SAME_DST_IP) xt_ndpi: flow detected id=30 (58990 - 443) | | protocol detected SSL (dst SAME_DST_IP) xt_ndpi: flow detected id=43 (58992 - 443) | | protocol detected SSL (dst SAME_DST_IP) xt_ndpi: flow detected id=43 (58992 - 443) | | protocol detected SSL (dst SAME_DST_IP) xt_ndpi: flow detected id=44 (443 - 58994) | | protocol detected SSL (dst SAME_DST_IP) xt_ndpi: flow detected id=44 (443 - 58994) | | protocol detected SSL (dst SAME_DST_IP) xt_ndpi: flow detected id=44 (58994 - 443) | | protocol detected SSL (dst SAME_DST_IP) xt_ndpi: flow detected id=44 (58994 - 443) | | protocol detected SSL (dst SAME_DST_IP) xt_ndpi: flow detected id=45 (58996 - 443) | | protocol detected SSL (dst SAME_DST_IP) xt_ndpi: flow detected id=45 (58996 - 443) | | protocol detected SSL (dst SAME_DST_IP) xt_ndpi: flow detected id=43 (58992 - 443) | | protocol detected SSL (dst SAME_DST_IP) xt_ndpi: flow detected id=43 (58992 - 443) | | protocol detected SSL (dst SAME_DST_IP) xt_ndpi: (TCP->FIN or TCP->RST) flow terminated id=45 (58996 - 443) xt_ndpi: flow detected id=46 (58996 - 443) | | protocol detected Unknown (dst SAME_DST_IP) xt_ndpi: (TCP->FIN or TCP->RST) flow terminated id=43 (58992 - 443) xt_ndpi: flow detected id=47 (58992 - 443) | | protocol detected Unknown (dst SAME_DST_IP) xt_ndpi: flow detected id=47 (443 - 58992) | | protocol detected Unknown (dst SAME_DST_IP) xt_ndpi: flow detected id=47 (443 - 58992) | | protocol detected Unknown (dst SAME_DST_IP) xt_ndpi: (TCP->FIN or TCP->RST) flow terminated id=47 (58992 - 443) xt_ndpi: flow detected id=48 (58992 - 443) | | protocol detected Unknown (dst SAME_DST_IP) xt_ndpi: (TCP->FIN or TCP->RST) flow terminated id=48 (443 - 58992) xt_ndpi: flow detected id=49 (443 - 58992) | | protocol detected Unknown (dst SAME_DST_IP) xt_ndpi: (TCP->FIN or TCP->RST) flow terminated id=49 (58992 - 443) xt_ndpi: flow detected id=50 (58992 - 443) | | protocol detected Unknown (dst SAME_DST_IP) xt_ndpi: flow detected id=46 (443 - 58996) | | protocol detected Unknown (dst SAME_DST_IP) xt_ndpi: flow detected id=46 (443 - 58996) | | protocol detected Unknown (dst SAME_DST_IP) xt_ndpi: flow detected id=46 (443 - 58996) | | protocol detected Unknown (dst SAME_DST_IP) xt_ndpi: flow detected id=46 (443 - 58996) | | protocol detected Unknown (dst SAME_DST_IP) xt_ndpi: (TCP->FIN or TCP->RST) flow terminated id=46 (58996 - 443) xt_ndpi: flow detected id=51 (58996 - 443) | | protocol detected Unknown (dst SAME_DST_IP) xt_ndpi: (TCP->FIN or TCP->RST) flow terminated id=51 (443 - 58996) xt_ndpi: flow detected id=52 (443 - 58996) | | protocol detected Unknown (dst SAME_DST_IP) xt_ndpi: (TCP->FIN or TCP->RST) flow terminated id=52 (58996 - 443) xt_ndpi: flow detected id=53 (58996 - 443) | | protocol detected Unknown (dst SAME_DST_IP) xt_ndpi: flow detected id=50 (443 - 58992) | | protocol detected Unknown (dst SAME_DST_IP) xt_ndpi: flow detected id=50 (443 - 58992) | | protocol detected Unknown (dst SAME_DST_IP) xt_ndpi: (TCP->FIN or TCP->RST) flow terminated id=50 (58992 - 443) xt_ndpi: flow detected id=54 (58992 - 443) | | protocol detected Unknown (dst SAME_DST_IP) xt_ndpi: flow detected id=30 (443 - 58990) | | protocol detected SSL (dst SAME_DST_IP) xt_ndpi: flow detected id=30 (443 - 58990) | | protocol detected SSL (dst SAME_DST_IP) xt_ndpi: (TCP->FIN or TCP->RST) flow terminated id=30 (443 - 58990) xt_ndpi: flow detected id=55 (443 - 58990) | | protocol detected Unknown (dst SAME_DST_IP) xt_ndpi: (TCP->FIN or TCP->RST) flow terminated id=55 (58990 - 443) xt_ndpi: flow detected id=56 (58990 - 443) | | protocol detected Unknown (dst SAME_DST_IP) xt_ndpi: flow detected id=56 (443 - 58990) | | protocol detected Unknown (dst SAME_DST_IP) xt_ndpi: flow detected id=56 (443 - 58990) | | protocol detected Unknown (dst SAME_DST_IP) xt_ndpi: flow detected id=44 (443 - 58994) | | protocol detected SSL (dst SAME_DST_IP) xt_ndpi: flow detected id=44 (443 - 58994) | | protocol detected SSL (dst SAME_DST_IP) xt_ndpi: flow detected id=44 (58994 - 443) | | protocol detected SSL (dst SAME_DST_IP) xt_ndpi: flow detected id=44 (58994 - 443) | | protocol detected SSL (dst SAME_DST_IP) xt_ndpi: (TCP->FIN or TCP->RST) flow terminated id=44 (443 - 58994) xt_ndpi: flow detected id=57 (443 - 58994) | | protocol detected Unknown (dst SAME_DST_IP) xt_ndpi: (TCP->FIN or TCP->RST) flow terminated id=57 (58994 - 443) xt_ndpi: flow detected id=58 (58994 - 443) | | protocol detected Unknown (dst SAME_DST_IP) xt_ndpi: flow detected id=58 (58994 - 443) | | protocol detected Unknown (dst SAME_DST_IP) xt_ndpi: flow detected id=58 (58994 - 443) | | protocol detected Unknown (dst SAME_DST_IP) xt_ndpi: flow detected id=58 (443 - 58994) | | protocol detected Unknown (dst SAME_DST_IP) xt_ndpi: flow detected id=58 (443 - 58994) | | protocol detected Unknown (dst SAME_DST_IP)

But after many try I have noticed that HTTP doesn't work : xt_ndpi: flow detected id=1511 (45328 - 80) | | protocol detected Unknown (dst 23.55.155.27) xt_ndpi: flow detected id=1511 (45328 - 80) | | protocol detected Unknown (dst 23.55.155.27)

But when I inspect the packet on Wireshark, we see "HTTP/1.1......", is it normal ?

Sorry, there many questions.

Sacriyana commented 7 years ago

After investigation for the HTTP detection, I force the detection because even if I add detection of HTTP in iptables, the traffic isn't detected.

betolj / ndpi-netfilter

Detection of SSL traffic #41