search

ntop / nDPI

Open Source Deep Packet Inspection Software Toolkit
http://www.ntop.org
GNU Lesser General Public License v3.0
3.69k stars 882 forks source link

Support for popular VPN application Detection #1610

Closed mmanoj closed 1 year ago

mmanoj commented 2 years ago

Hi,

As per the current test, it's observed most of the free and popular VPN apps are not detected.Most detected as either cloudflare or cloudfront as it's use many dynamic DNS addresses.

Some examples are below; hotspot shield Tunnelbear *Ultrasurf

What is the strategy to support such application detection?

utoni commented 2 years ago

Could you please provide some pcaps?

mmanoj commented 2 years ago

Hi @utoni

Please find the attached PCAPS. I also like to contribute to creating the signature. PktCaptures.zip

mmanoj commented 2 years ago

CloudFlare WARP-VPN (1.1.1.1) cloudFlare-Warp.pcap.zip

mmanoj commented 2 years ago

@utoni Any findings based on shared PCAPS? if any support is required please let me know.

utoni commented 2 years ago

@mmanoj I am working on it. But for now I can not promise that the detection of every VPN you've mentioned will be possible.

mmanoj commented 2 years ago

Hi @utoni ,

Thanks for the feedback, which VPN you're currently working on? so I can share my analysis also for other VPNs. We can work out to create the signatures together. Please let me know.

utoni commented 2 years ago

I am working on the UltraSurf PCAP. But besides some background traffic to third party services, there is nothing interesting in it.

mmanoj commented 2 years ago

@utoni

Thanks for the update, Do you want me to get new capture? So I can arrange it soon.

mmanoj commented 2 years ago

@utoni I'm working on new captures and will share them once completed.

utoni commented 2 years ago

PR #1615 may detect TunnelBear VPN. The rest of the PCAPs did not contain any interesting patterns.

mmanoj commented 2 years ago

@utoni

I'm working on capturing Ultrasurf again, a bit challenging to get the correct handshake message. I will upload some captures I got yesterday. Please check the CloudFlare WARP-VPN (1.1.1.1), hope it contains the required info.

mmanoj commented 2 years ago

Ultraserf-02072022.zip

Please check the latest capture for Ultraserf

utoni commented 2 years ago

I think something went wrong. There are TLS flows with missing handshakes. They could be of interest.

mmanoj commented 2 years ago

I saw a communication to Dest Port 50053 and TLS communication having below bytes pattern

cc 1c 30 41 5b a4 38 66 99 6f eb a3 ff c3 dd 23

utoni commented 2 years ago

I am unsure if this pattern is stable. Can you try #1618 and check if it does detect UltraSurf VPN more or less reliable?

There are two TLSv1.3 flows with a high data throughput. They are probably part of UltraSurf VPN and I think that there is a way to detect those (if you really need this).

mmanoj commented 2 years ago

@utoni

Thanks for the feedback,I will check and update the results. Also, I found some research papers which help to detect Ultrasurf. I will email you. Please share your email.

mmanoj commented 2 years ago

Detected protocols: Unknown packets: 705 bytes: 76642 flows: 31
DNS packets: 9 bytes: 1239 flows: 5
Yahoo packets: 45 bytes: 22758 flows: 1
ICMP packets: 67 bytes: 5153 flows: 5
RTP packets: 3 bytes: 252 flows: 1
TLS packets: 37467 bytes: 28264124 flows: 104
YouTube packets: 78 bytes: 59710 flows: 3
Google packets: 1378 bytes: 689035 flows: 101
WhatsApp packets: 188 bytes: 31218 flows: 5
DoH_DoT packets: 624 bytes: 221625 flows: 18
Microsoft packets: 27 bytes: 10646 flows: 1
Cloudflare packets: 17 bytes: 8875 flows: 1
GoogleServices packets: 589 bytes: 258365 flows: 23
AmazonAWS packets: 230 bytes: 98873 flows: 13
GoogleCloud packets: 33 bytes: 15572 flows: 1
UltraSurf packets: 2971 bytes: 2991918 flows: 1

mmanoj commented 2 years ago

Reading packets from pcap file ../../../PktCaptures/UltraSurf_rx_02.pcap... UltraSurf packets: 2971 bytes: 2991918 flows: 1
5 TCP 65.49.68.25:50053 <-> 10.132.0.23:37898 [VLAN: 200][proto: 301/UltraSurf][Encrypted][Confidence: DPI][cat: VPN/2][1802 pkts/2867775 bytes <-> 1169 pkts/124143 bytes][Goodput ratio: 96/19][46.77 sec][bytes ratio: 0.917 (Upload)][IAT c2s/s2c min/avg/max/stddev: 0/0 24/31 438/290 32/43][Pkt Len c2s/s2c min/avg/max/stddev: 70/60 1591/106 2646/1900 592/121][PLAIN TEXT (OFdfbY)][Plen Bins: 0,10,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,60,0,0,0,0,0,0,28]

utoni commented 2 years ago

Does the UltraSurf detection work as expected?

mmanoj commented 2 years ago

Yes, Post PCAPS was detected. I'm testing with fresh capture and update you the results asap. Also, I'm reverse engineering the apk to check the communications, and IP addresses will update once completed.

mmanoj commented 2 years ago

Pls find the latest captures, it didn't detect ultra_rx_20220707_052251_86723.zip

I saw some pattern 17 03 03 but in random locations.

utoni commented 2 years ago

Pls find the latest captures, it didn't detect ultra_rx_20220707_052251_86723.zip

It is apparently not so easy to reliably detect UltraSurf.

I saw some pattern 17 03 03 but in random locations.

This pattern is part of TLS (application data).

mmanoj commented 2 years ago

Hi @utoni , Please find my latest analysis below with attached dumps with various PCAPS. I found the below pattern more reliable.

16 03 01 02 00 01 00 01 fc 03 03 3b b6 68 fc 96

mmanoj commented 2 years ago

ultraSurf_pcapDumpAnalysys.zip

utoni commented 2 years ago

16 03 01 02 00 01 00 01 fc 03 03 3b b6 68 fc 96

The pattern belongs to a TLSv1.3 client hello.

mmanoj commented 2 years ago

I can see it's in most of the communication happened, any other suggestions to detect?

I'm working on sequence diagram of communication flow of the ultradutf to more understand.i also send an email to you with some research papers as well .

mmanoj commented 2 years ago

@utoni

Any other suggestions to detect the Ultrasurf? I'm still analyzing possible methods and trying to figure out the communication pattern/domain fronting etc.

utoni commented 2 years ago

Those two papers are quite old. The newer one from 2016 uses a combination of IP address matching together with caching DNS requests and responses and doing some port/service scanning (nDPI is a passive DPI, so we won't do that). The PCAPs you've provided seem to not use the aforementioned approaches anymore (at least not in way that would help to reliably detect UltraSurf). I guess I am lost here.

The UltraSurf dev's did a great job. Censorship circumvention is more important than ever, especially nowadays..

mmanoj commented 2 years ago

Hi Toni, Thanks for the feedback, yes those papers were old,just share for info.So it mean we need to figure out new method to detect Ultrasurf right? Can you advice any starting point?

utoni commented 2 years ago

I'm afraid there won't be much that can I do.

mmanoj commented 2 years ago

@utoni

Please find captures of Ultrasurf Chrome extension captures. Also, look for the below IP communication (which I break and restart the VPN communication)

64.62.219.190 74.82.60.91 65.49.68.22

Ulreasurf-chrome0.zip

utoni commented 2 years ago

We can add those IP addresses for flow guessing. But I am not sure how frequent they might change.

mmanoj commented 2 years ago

Hi@toni,

Those IP communication having useful info? If so I'm working on some detail analysis with DNS communication,I will share it with in couple of days.hope it having more visibility to move further.

mmanoj commented 2 years ago

@utoni Please find the DNS analysis of 74.82.60.91 Please let me know is this info useful. So I can dig more. Ultrasurf-Chrome-DNS-Analysis-01.pdf

mmanoj commented 2 years ago

Mostly I observed No DNS information if it's connected site via VPN

mmanoj commented 2 years ago

@utoni I'm preparing an IP list with CIDR info, and will share it ASAP. Hope we can detect the service much more reliable way.

mmanoj commented 1 year ago

Hi Toni,

Please find the up-to-now verified IP list from captured traffic, I'm attaching the suspected list also. 45.95.98.0/24 64.62.128.0/17 64.62.128.0/18 64.71.128.0/18 65.49.0.0/17 65.49.68.0/24 72.52.64.0/18 72.52.92.0/24 74.82.0.0/18 74.82.46.0/24 74.82.48.0/22 UltraSurfIPList.xlsx

utoni commented 1 year ago

Hi @mmanoj, the prefixes you've provided are too generic and will result in false-positive Ultrasurf guesses.

IvanNardi commented 1 year ago

Hi @mmanoj, the prefixes you've provided are too generic and will result in false-positive Ultrasurf guesses.

+1

I am not against using an ip list to detect Ultrasurf but, at very least:

mmanoj commented 1 year ago

@IvanNardi

Thanks for the feedback and concerns raised. It's fully understood. Please find my comments below.

1. this list should not be too generic

The IP list was prepared based on captured traffic samples and various info collected from the internet.I will check to improve the list much more specific in my next update.

2.a script to easily update that list MUST be provided

For this we need some community support and more traffic samples to generate the list. If any suggestions are always welcome.

3.a configuration knob should allow avoiding loading this list

Hope this need to be developed with protocol Dissector

Please suggest any other detection techniques, so I can explore them.

lucaderi commented 1 year ago

@mmanoj Why do we need t use IPs instead of DPI?

mmanoj commented 1 year ago

Hi @lucaderi

We not find any reliable bit pattern from communication captures so far, we just think of using IP address for flow guessing.As per analysis most communication happen with random DNS and servers. Your advice is highly appreciate for way forward few other mechanisms to detect more reliably.So we can explore it.Thank You for the support and advice.

mmanoj commented 1 year ago

Hi All,

Any plan for a way forward? so we can work together to mitigate this challenge.

mmanoj commented 1 year ago

Ultrasurf_8.pcap.zip Ultrasurf_9.pcap (1).zip

utoni commented 1 year ago

Unfortunately, this is again TLSv1.3 traffic w/o any useful detection pattern e.g. the string of a SNI extension. I think you are wasting your time trying to detect Ultrasurf with conventional methods.

mmanoj commented 1 year ago

@utoni Thanks for the feedback and I agreed with you, It's not possible to detect bytes patterns or IP lists. Can you suggest a good strategy so we can work it out?

ML/AI may be a good option, need to have a plan if we going to work in that direction. Or evaluate available options which we can integrate with nDPI.

Appreciate your advice.

utoni commented 1 year ago

@mmanoj For the ML approach, I would recommend https://github.com/nfstream/nfstream. I am using it from time to time and there is also a ML example using sklearn.

mmanoj commented 1 year ago

@utoni

Thanks for the feedback, Do you/the nDPI team have any other approach to this issue? If yes we can put collective effort, otherwise I will check the ML approach.

Please advise before starting.