Traffic from Google sites not being detected

patrick-dev237 commented 3 months ago

Describe the bug

After updating to the latest nDPI hash, it seems like the traffic from any Google application website (Google Drive, Docs and Maps) isn't being properly classified and these sites aren't being blocked as a result.

The latest hash that seems to work is 761993583265cf9a022be1ad2d701e446bb91d42 from March 20th 2024. The issue seems to occur on versions later than this at some point. Perhaps there was a change made around this time that could've impacted this functionality? Other websites are being detected without issue, it just seems to be any Google related ones that aren't.

Expected behavior

Traffic from sites like Google Drive, Docs and Maps is classified properly.

Obtained behavior

Traffic from sites like Google Drive, Docs and Maps isn't being properly classified

Screenshot (287)

nDPI Environment (please complete the following information):

nDPI version or commit hash: d824fc1ffc4b148f2f545c365790f897c220963f

How to reproduce the reported bug

Visit one of the following websites and check reported traffic:

drive.google.com
docs.google.com
maps.google.com

If your bug is reproducible using a pcap, please attach a pcap file (or a valid link to download it)

Examples: google_traffic_examples.zip

vel21ripn commented 3 months ago

nDPI has two protocols: Google Maps and Google Docs. You need to check both. I don't see a line in your rules that contains "-m ndpi --all" It would be much easier if the rules were in the form of text, and not a picture.

patrick-dev237 commented 3 months ago

I'm currently checking for Google Drive, Google Maps and Google Docs separately. Here are all the rules I'm using:

Chain PREROUTING (policy ACCEPT 52928 packets, 8176K bytes)
 pkts bytes target     prot opt in     out     source               destination
53041 8212K ia_mangle_pre_hook  all  --  *      *       0.0.0.0/0            0.0.0.0/0            /* instance-agent: mangle pre hook */
 2262  355K app_control_pre  all  --  wg0    *       0.0.0.0/0            0.0.0.0/0            /* app-control: mangle pre hook */

Chain INPUT (policy ACCEPT 49136 packets, 4937K bytes)
 pkts bytes target     prot opt in     out     source               destination

Chain FORWARD (policy ACCEPT 3592 packets, 3202K bytes)
 pkts bytes target     prot opt in     out     source               destination

Chain OUTPUT (policy ACCEPT 51472 packets, 18M bytes)
 pkts bytes target     prot opt in     out     source               destination

Chain POSTROUTING (policy ACCEPT 55064 packets, 22M bytes)
 pkts bytes target     prot opt in     out     source               destination
55177   22M ia_mangle_post_hook  all  --  *      *       0.0.0.0/0            0.0.0.0/0            /* instance-agent: mangle post hook */
 1712 2909K app_control_pre  all  --  *      wg0     0.0.0.0/0            0.0.0.0/0            /* app-control: mangle post hook */

Chain app_control_classify (1 references)
 pkts bytes target     prot opt in     out     source               destination
    0     0 MARK       all  --  *      *       0.0.0.0/0            0.0.0.0/0            /* app-control: match unmarked and mark block */ mark match 0x0/0xf0 ndpi proto googledocs match-set policy-1776 src,dst MARK or 0x40
    0     0 MARK       all  --  *      *       0.0.0.0/0            0.0.0.0/0            /* app-control: match unmarked and mark block */ mark match 0x0/0xf0 ndpi proto googledrive match-set policy-1776 src,dst MARK or 0x40
    0     0 MARK       all  --  *      *       0.0.0.0/0            0.0.0.0/0            /* app-control: match unmarked and mark block */ mark match 0x0/0xf0 ndpi proto googlemaps match-set policy-1776 src,dst MARK or 0x40

Chain app_control_pre (2 references)
 pkts bytes target     prot opt in     out     source               destination
 3974 3264K CONNMARK   all  --  *      *       0.0.0.0/0            0.0.0.0/0            mark match 0x0 CONNMARK restore
 3126 2967K RETURN     all  --  *      *       0.0.0.0/0            0.0.0.0/0            connbytes 10 connbytes mode packets connbytes direction both
  848  297K app_control_classify  all  --  *      *       0.0.0.0/0            0.0.0.0/0            /* app-control: match unmarked */ mark match 0x0/0xf0
  848  297K CONNMARK   all  --  *      *       0.0.0.0/0            0.0.0.0/0            CONNMARK save

Chain ia_ipsec_bound (0 references)
 pkts bytes target     prot opt in     out     source               destination

Chain ia_mangle_post_hook (1 references)
 pkts bytes target     prot opt in     out     source               destination
   67  3460 TCPMSS     tcp  --  *      wg0     0.0.0.0/0            0.0.0.0/0            tcp flags:0x06/0x02 TCPMSS set 1240

Chain ia_mangle_pre_hook (1 references)
 pkts bytes target     prot opt in     out     source               destination
   55  2860 TCPMSS     tcp  --  wg0    *       0.0.0.0/0            0.0.0.0/0            tcp flags:0x06/0x02 TCPMSS set 1240

vel21ripn commented 3 months ago

There are too many conditions in the app_control_classify chain. To understand where the problem is, I would add "-m ndpi --all" without any conditions as the first rule in this chain and check the protocols of interest with the following rules. This way you will see what gets into ndpi (see the file /proc/net/xt_ndpi/proto). Reminder: All network packets from the client to the server and back should get into ndpi.

vel21ripn / nDPI