protocols supported by nDPI (3.4 stable) but not identified by nDPI sample reader app

[subhajit-cdot]

Hi,

Below are the protocols which are supported by nDPI (3.4 stable) but not identified by nDPI sample app using the attached pcap files corresponding to each protocol listed:

1. Kontiki
2. ZATTOO
3. SHOUTCAST
4. TVUPLAYER
5. QQLIVE
6. AIMINI
7. WARCRAFT3
8. CITRIX
9. PPStream
10. Hotmail
11. FASTTRACK
12. Soulseek
13. Bittorent (TCP sessions)
14. UPNP
15. TOR
16. Skype Chat/Videocall (some versions)
17. Git
18. FIX
19. QUIC
20. TFTP
21. DOFUS
22. Whatsapp (chat)
23. CORBA
24. Telegram
25. kakaotalk
26. Wechat
27. Hangout Duo
28. Hotspot shield
29. TRUPHONE
30. Lotus Notes
31. RSYNC 

Please let me know if there is any issue in the pcap files or are these application signatures different from the existing dissector?

Download PCAP files: https://gofile.io/d/LkB0uw Thanks & Regards, Subhajit

[IvanNardi]

A quick comment on the QUIC traces. Packet 4 of client.pcap can't be the first QUIC packet sent from the client. Therefore, or that is not a QUIC session, or the initial handshake is missing, IMO; nDPI needs the initial packets to properly detect QUIC traffic. Could you double check, please?

[utoni]

I've just looked at the aimini traces and I see two issues here.

The first issue is that the aimini protocol classification needs an update here. The second issue is the nDPI core which stops after detecting the first protocol and does not care about possible subprotocols of the detected protocol (mentioned in #1141). If my understanding of subprotocols is correct; nDPI protocol detection routines should not stop after detecting a protocol which can have subprotocols e.g. HTTP and TLS. At the moment, the subprotocol flag is only used if a protocol classification decides to change the detected protocol which makes IMHO only sense if the protocol classification (in that case: aimini) callback gets called (which is not the case for aimini).

I'm working on a PR which is not done yet (needs some testing).

[subhajit-cdot]

@lnslbrty Hi Toni, Can you please brief when a protocol decides to change detected protocol as you stated "the subprotocol flag is only used if a protocol classification decides to change the detected protocol". My understanding is also the same in case of sub-protocol detection, however can you please elaborate why have you picked HTTP and TLS for example? Is there any specific case which you have observed?

[subhajit-cdot]

@IvanNardi I have just compared the quic pcap given here with ndpi pcap folder's quic-23, quic-24 etc. and can't able to trace any initial exchange packets. Can you please elaborate how to check initial exchange packets in quic (udp) protocol. I can able to see initial packets in the case of gquic (google quic) but not here.

[IvanNardi]

@IvanNardi I have just compared the quic pcap given here with ndpi pcap folder's quic-23, quic-24 etc. and can't able to trace any initial exchange packets. Can you please elaborate how to check initial exchange packets in quic (udp) protocol. I can able to see initial packets in the case of gquic (google quic) but not here.

From the initail packets POV, the main difference between QUIC versions is

• for earlier GQUIC versions (< Q50) the initial packet is sent in cleartext, while
• for later GQUIC versions (>= Q50 and Txx) and for all IETF-QUIC versions the initial packet is "kind of" encrypted.

Fortunately, you are able to always see the Initail Packets in cleartext with lately Wireshark versions ( >= 3.4.2).

About the traces in nDPI repository. As a general rule "quic-NUMBER.pcap" traces contain IETF-QUIC traffic (the NUMBER is the draft version), while "quic-qNUMBER.pcap" and "quic-tNUMBER" traces contain GQUIC data.

As example in the attached screenshot you can see that quic-23.pcap contains the initial TLS handshake.

In the QUIC pcap you provided, the UDP session definitely lacks of initial packets.

Let me know if you need further information.

[subhajit-cdot]

@IvanNardi Actually It was showing little different in my wireshark (2.6.10), that's why I was bit confused. Also seeing your output, it seems some initial packets are missing in the pcap provided. I'll check and get back to you.

[IvanNardi]

@IvanNardi Actually It was showing little different in my wireshark (2.6.10),

Yeah, 2.6 is VERY old; to correctly handle QUIC traffic you need 3.4.2 at least.

[utoni]

@lnslbrty Hi Toni, Can you please brief when a protocol decides to change detected protocol as you stated "the subprotocol flag is only used if a protocol classification decides to change the detected protocol". My understanding is also the same in case of sub-protocol detection, however can you please elaborate why have you picked HTTP and TLS for example?

That is simply because most of the classifications are TLS based, followed by HTTP. ;)

Is there any specific case which you have observed?

For example: aimini Uses HTTP for it's TCP detection part, but it's detection routine is never ever called, because HTTP has a lower index in the callback buffer and thus get's called earlier. After HTTP was detected, nDPI does not care about possible subprotocols e.g. the HTTP part of aimini.

[subhajit-cdot]

Update:

1. I have tested Whatsapp real traffic between two phones. nDPI was failed to detect Whatsapp Chat whereas accessing whatsapp from web can be easily detected. Can anyone please look at the given Whatsapp pcap shared and comment why it was not detected.
2. Checked facebook app real traffic between two phones, nDPI was able to detect it, but not facebook messenger.

[subhajit-cdot]

Can we use destination port 5222 to detect whatsapp as mentioned in the https://developers.facebook.com/docs/whatsapp/guides/network-requirements ? Whatsapp also uses 5223, 5228 in some cases. (refer https://github.com/ukanth/afwall/wiki/HOWTO-blocking-WhatsApp)

Also please clarify why ndpi is using exact match for url in ndpi_content_match.c.inc foe whatsapp when we have a broader entry for whatsapp. Doesn't it mean it will work for "*.whatsapp.*\". @IvanNardi @lucaderi Can you please comment/explain?

1. { ".whatsapp.", "WhatsApp", NDPI_PROTOCOL_WHATSAPP, NDPI_PROTOCOL_CATEGORY_CHAT, NDPI_PROTOCOL_ACCEPTABLE },
2. { "g.whatsapp.net", "WhatsApp", NDPI_PROTOCOL_WHATSAPP, NDPI_PROTOCOL_CATEGORY_CHAT, NDPI_PROTOCOL_ACCEPTABLE },
3. { "v.whatsapp.net", "WhatsApp", NDPI_PROTOCOL_WHATSAPP, NDPI_PROTOCOL_CATEGORY_CHAT, NDPI_PROTOCOL_ACCEPTABLE },
4. { "mmg.whatsapp.net", "WhatsApp", NDPI_PROTOCOL_WHATSAPP, NDPI_PROTOCOL_CATEGORY_CHAT, NDPI_PROTOCOL_ACCEPTABLE },

Thanks

[subhajit-cdot]

Update :

1. Initially with the dissector we were only able to block whatsapp file uploading/downloading. Whatsapp was working fine.

I have added the below line in protocols.cfg to make whatsapp work.

host:".whatsapp."@WhatsApp

I don't know why the below rule from ndpi_content_match.c.inc didn't work. { ".whatsapp.", "WhatsApp", NDPI_PROTOCOL_WHATSAPP, NDPI_PROTOCOL_CATEGORY_CHAT, NDPI_PROTOCOL_ACCEPTABLE },

Any idea on this?

2. GTP in the above added pcap didn't work because the below segment is hitting, Not sure if this is required. The packet is GTP for sure but this is an invalid GTP. Mostly we put these checks in GTP firewall.

`static void ndpi_check_gtp(struct ndpi_detection_module_struct ndpi_struct, struct ndpi_flow_struct flow) {

    if(message_len <= (payload_len-sizeof(struct gtp_header_generic))) {
      NDPI_LOG_INFO(ndpi_struct, "found gtp\n");
      ndpi_set_detected_protocol(ndpi_struct, flow, NDPI_PROTOCOL_GTP, NDPI_PROTOCOL_UNKNOWN);
      return;
    }

} ` Thanks Subhajit

[IvanNardi]

1. GTP in the above added pcap didn't work because the below segment is hitting,

What GTP pcap are you referring to? I didn't find any GTP traffic in the archive you attached to your first message. Have I missed something?

[subhajit-cdot]

Sorry forgot to attach that file, in some cases it was failed due to the upper code segment's checking. gtp.zip

[IvanNardi]

Sorry forgot to attach that file, in some cases it was failed due to the upper code segment's checking. gtp.zip

Thanks for the trace, but I am not getting your point abot incorrect GTP classification... In the trace there are 4 (perfectly valid) GTP packets and nDPI classified them as GTP (as expected, IMO).

ivan@ivan-Latitude-E6540:~/svnrepos/nDPI(dev)$ ./example/ndpiReader -t -i ./gtp/client.pcap -v2 
[...]
    1   UDP 10.101.0.2:1024 <-> 10.102.0.2:2123 [proto: 152/GTP][cat: Network/14][2 pkts/365 bytes <-> 2 pkts/319 bytes][Goodput ratio: 77/73][0.00 sec][PLAIN TEXT (internet)][Plen Bins: 25,25,0,0,0,0,25,25,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0]

What am I missing?

[subhajit-cdot]

@IvanNardi This works because protocol guessing is enabled de-facto in ndpiReader application. If you try without protocol guessing you will see it will fail to detect GTP and the reason is that the GTP dissector checks for invalid GTP packet. Even with protocol guessing, you can put breakpoint in the below dissector function to see what's happening.

static void ndpi_check_gtp(struct ndpi_detection_module_struct ndpi_struct, struct ndpi_flow_struct flow) {

if(message_len <= (payload_len-sizeof(struct gtp_header_generic))) {
  NDPI_LOG_INFO(ndpi_struct, "found gtp\n");
  ndpi_set_detected_protocol(ndpi_struct, flow, NDPI_PROTOCOL_GTP, NDPI_PROTOCOL_UNKNOWN);
  return;
}

}

Thanks

[IvanNardi]

@subhajit-cdot , #1161 should have fixed GTP checks. Could you check, please?

[subhajit-cdot]

Thanks @IvanNardi for the fix. It's working fine!