search

fooinha / nginx-ssl-ja3

nginx module for SSL/TLS ja3 fingerprint.
BSD 2-Clause "Simplified" License
195 stars 48 forks source link

JA3 hashes do not match #29

Closed migolovanov closed 2 years ago

migolovanov commented 2 years ago

Client:

# lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description:    Ubuntu 20.04.4 LTS
Release:        20.04
Codename:       focal
# curl -V
curl 7.68.0 (x86_64-pc-linux-gnu) libcurl/7.68.0 OpenSSL/1.1.1f zlib/1.2.11 brotli/1.0.7 libidn2/2.2.0 libpsl/0.21.0 (+libidn2/2.2.0) libssh/0.9.3/openssl/zlib nghttp2/1.40.0 librtmp/2.3
Release-Date: 2020-01-08
Protocols: dict file ftp ftps gopher http https imap imaps ldap ldaps pop3 pop3s rtmp rtsp scp sftp smb smbs smtp smtps telnet tftp
Features: AsynchDNS brotli GSS-API HTTP2 HTTPS-proxy IDN IPv6 Kerberos Largefile libz NTLM NTLM_WB PSL SPNEGO SSL TLS-SRP UnixSockets

Destinations: https://check.ja3.zone - HAProxy (built-in JA3 module) https://check.ja3.zone:8443 - nginx with nginx-ssl-ja3 module, built with docker-compose.yml from this repository

nginx-ssl-ja3 says hash is eaa1a9e1db47ffcca16305566a6efba4 HAProxy, wirehsark built-in JA3 and lua plugin fullylegit/ja3, python implementation from salesforce/ja3 say hash is f436b9416f37d134cadd04886327d3e8, so I believe this one is correct

Details:

# curl https://check.ja3.zone
{"hash":"f436b9416f37d134cadd04886327d3e8","fingerprint":"771,4866-4867-4865-49196-49200-159-52393-52392-52394-49195-49199-158-49188-49192-107-49187-49191-103-49162-49172-57-49161-49171-51-157-156-61-60-53-47-255,0-11-10-13172-16-22-23-49-13-43-45-51-21,29-23-30-25-24,0-1-2","ciphers":"TLS_AES_256_GCM_SHA384,TLS_CHACHA20_POLY1305_SHA256,TLS_AES_128_GCM_SHA256,ECDHE-ECDSA-AES256-GCM-SHA384,ECDHE-RSA-AES256-GCM-SHA384,DHE-RSA-AES256-GCM-SHA384,ECDHE-ECDSA-CHACHA20-POLY1305,ECDHE-RSA-CHACHA20-POLY1305,DHE-RSA-CHACHA20-POLY1305,ECDHE-ECDSA-AES128-GCM-SHA256,ECDHE-RSA-AES128-GCM-SHA256,DHE-RSA-AES128-GCM-SHA256,ECDHE-ECDSA-AES256-SHA384,ECDHE-RSA-AES256-SHA384,DHE-RSA-AES256-SHA256,ECDHE-ECDSA-AES128-SHA256,ECDHE-RSA-AES128-SHA256,DHE-RSA-AES128-SHA256,ECDHE-ECDSA-AES256-SHA,ECDHE-RSA-AES256-SHA,DHE-RSA-AES256-SHA,ECDHE-ECDSA-AES128-SHA,ECDHE-RSA-AES128-SHA,DHE-RSA-AES128-SHA,AES256-GCM-SHA384,AES128-GCM-SHA256,AES256-SHA256,AES128-SHA256,AES256-SHA,AES128-SHA,TLS_EMPTY_RENEGOTIATION_INFO_SCSV","curves":"001D:0017:001E:0019:0018","protocol":"TLSv1.3","user_agent":"curl/7.68.0"}

# curl https://check.ja3.zone:8443
{"hash":"eaa1a9e1db47ffcca16305566a6efba4","fingerprint":"771,4866-4867-4865-49196-49200-159-52393-52392-52394-49195-49199-158-49188-49192-107-49187-49191-103-49162-49172-57-49161-49171-51-157-156-61-60-53-47-255,0-11-10-13172-16-22-23-49-13-43-45-51-21,29-23-1035-25-24,0-1-2","ciphers":"TLS_AES_256_GCM_SHA384:TLS_CHACHA20_POLY1305_SHA256:TLS_AES_128_GCM_SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:DHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-CHACHA20-POLY1305:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA384:DHE-RSA-AES256-SHA256:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA256:DHE-RSA-AES128-SHA256:ECDHE-ECDSA-AES256-SHA:ECDHE-RSA-AES256-SHA:DHE-RSA-AES256-SHA:ECDHE-ECDSA-AES128-SHA:ECDHE-RSA-AES128-SHA:DHE-RSA-AES128-SHA:AES256-GCM-SHA384:AES128-GCM-SHA256:AES256-SHA256:AES128-SHA256:AES256-SHA:AES128-SHA:TLS_EMPTY_RENEGOTIATION_INFO_SCSV","curves":"X25519:prime256v1:X448:secp521r1:secp384r1","protocol":"TLSv1.2","user_agent":"curl/7.68.0"}

# python3 ja3.py -a -j /mnt/d/ja3.pcap
[
    {
        "destination_ip": "212.237.16.237",
        "destination_port": 443,
        "ja3": "771,4866-4867-4865-49196-49200-159-52393-52392-52394-49195-49199-158-49188-49192-107-49187-49191-103-49162-49172-57-49161-49171-51-157-156-61-60-53-47-255,0-11-10-13172-16-22-23-49-13-43-45-51-21,29-23-30-25-24,0-1-2",
        "ja3_digest": "f436b9416f37d134cadd04886327d3e8",
        "source_ip": "172.26.210.215",
        "source_port": 59528,
        "timestamp": 1653503728.020719
    },
    {
        "destination_ip": "212.237.16.237",
        "destination_port": 8443,
        "ja3": "771,4866-4867-4865-49196-49200-159-52393-52392-52394-49195-49199-158-49188-49192-107-49187-49191-103-49162-49172-57-49161-49171-51-157-156-61-60-53-47-255,0-11-10-13172-16-22-23-49-13-43-45-51-21,29-23-30-25-24,0-1-2",
        "ja3_digest": "f436b9416f37d134cadd04886327d3e8",
        "source_ip": "172.26.210.215",
        "source_port": 42284,
        "timestamp": 1653503730.283344
    }
]

ja3_pcap.zip

fooinha commented 2 years ago

thanx @migolovanov .

I'll take a look, when possible

migolovanov commented 2 years ago

Thank you very much for quick response. This fix worked for curl and now its hash is same as other tools say. But I have checked Chrome and Burp and their hashes still differ:

First packet in dump: Chrome 102.0.5005.63 nginx-ssl-ja3 says hash is b32309a26951912be7dba376398abc3b wireshark internal plugin seems to be wrong too with completely different value, but lua plugin and JA3 python script says it is cd08e31494f9531f560d64c695473da9

Second packet in dump: Burp 2022.3.9 nginx-ssl-ja3 says hash is 5deced51aefff5efaba914e5cd11a2ca wireshark lua plugin and JA3 python script says it is dfd0bc4a6d8c21e500d6be9121fd44d7

Third packet in dump: Burp 2022.3.9 nginx-ssl-ja3 says hash is 6989d3afab0e0fcaaf6a951909b9780e wireshark lua plugin and JA3 python script says it is 1875aec3e9d2d51ffde08cc376aedcff

Details: ja3_2.zip

# python3 ja3.py -a -j /mnt/d/ja3_2.pcap
[
    {
        "destination_ip": "212.237.16.237",
        "destination_port": 8443,
        "ja3": "771,4865-4866-4867-49195-49199-49196-49200-52393-52392-49171-49172-156-157-47-53,0-23-65281-10-11-35-16-5-13-18-51-45-43-27-17513-21,29-23-24,0",
        "ja3_digest": "cd08e31494f9531f560d64c695473da9",
        "source_ip": "192.168.101.60",
        "source_port": 63449,
        "timestamp": 1653864095.257029
    },
    {
        "destination_ip": "212.237.16.237",
        "destination_port": 8443,
        "ja3": "771,4866-4865-4867-49196-49195-52393-49200-52392-49199-159-52394-163-158-162-49188-49192-49187-49191-107-106-103-64-49198-49202-49197-49201-49190-49194-49189-49193-49162-49172-49161-49171-57-56-51-50-49157-49167-49156-49166-157-156-61-60-53-47-49160-49170-22-19-49155-49165-10-255,0-5-10-11-16-17-23-35-13-43-45-50-51,29-23-24-25-30-256-257-258-259-260,0",
        "ja3_digest": "dfd0bc4a6d8c21e500d6be9121fd44d7",
        "source_ip": "192.168.101.60",
        "source_port": 63471,
        "timestamp": 1653864112.860739
    },
    {
        "destination_ip": "212.237.16.237",
        "destination_port": 8443,
        "ja3": "771,4866-4865-4867-49196-49195-52393-49200-52392-49199-159-52394-163-158-162-49188-49192-49187-49191-107-106-103-64-49198-49202-49197-49201-49190-49194-49189-49193-49162-49172-49161-49171-57-56-51-50-49157-49167-49156-49166-157-156-61-60-53-47-49160-49170-22-19-49155-49165-10-255,0-5-10-11-17-23-35-13-43-45-50-51,29-23-24-25-30-256-257-258-259-260,0",
        "ja3_digest": "1875aec3e9d2d51ffde08cc376aedcff",
        "source_ip": "192.168.101.60",
        "source_port": 63472,
        "timestamp": 1653864114.747529
    }
]
fooinha commented 2 years ago

@migolovanov , please try again, I've just updated it.

migolovanov commented 2 years ago

The first one (for chrome) is fine now, for burp the issue still persist.

fooinha commented 2 years ago

@migolovanov , the last few changes may help to make the hashes better . Thanx again for testing this.

migolovanov commented 2 years ago

It already works great for quite all tools I checked (python libs, nodejs, go, curl, wget). But somehow it still shows wrong hash for burp, 2-3 packets from dump (and maybe other Java applications, have not tested yet). Are the latest changes you mentioned already present in master? Don't worry, I'm ready to test it while you continue to provide fixes, since am personally interested in this one (trying to build ja3 fingerprint database and it's kind of crucial to calculate hashes correctly).

fooinha commented 2 years ago

@migolovanov

So, all the changes are already in master.

Issue 1:

I've disabled the session resumption/cache from ssl ticket extension. That could be the reason as the following requests had different values from the first request.

Do you get different hashes for burp, for a sequence of requests? Or, now are they all the same?

Issue 2:

We expect requests coming from burp to have a correct hash.

This approach via nginx uses openssl as the library. Openssl must be updated or patched in order to get and understand TLS fields from connection. If something, usually an TLS extension, is not supported or understood by Openssl, those values would not be accessible to library callers, and these values will not be included in the ja3 fingerprint ( Making the final value differ from others) .

This approach is active.

Other passive approaches based on PCAPs will not to all validations and values for the fingerprint are retrieved directly from packets data.

Just added "fake support" for extension status_request_v2, that is used by Burp ( with MacOS ). The lack of support was producing diff hash values, but, now I get ...

771,4866-4865-4867-49196-49195-52393-49200-52392-49199-159-52394-163-158-162-49188-49192-49187-49191-107-106-103-64-49198-49202-49197-49201-49190-49194-49189-49193-49162-49172-49161-49171-57-56-51-50-49157-49167-49156-49166-157-156-61-60-53-47-49160-49170-22-19-49155-49165-10-255,5-10-11-17-23-13-43-45-50-51,29-23-24-25-30-256-257-258-259-260,0

67c09d64ab06c98cd22800a65012987b

Which is the same value as running python's ja3.py script. 

python ja3.py /Users/paulopacheco/Downloads/burp.pcap
[127.0.0.1:443] JA3: 771,4866-4865-4867-49196-49195-52393-49200-52392-49199-159-52394-163-158-162-49188-49192-49187-49191-107-106-103-64-49198-49202-49197-49201-49190-49194-49189-49193-49162-49172-49161-49171-57-56-51-50-49157-49167-49156-49166-157-156-61-60-53-47-49160-49170-22-19-49155-49165-10-255,5-10-11-17-23-13-43-45-50-51,29-23-24-25-30-256-257-258-259-260,0 --> 67c09d64ab06c98cd22800a65012987b
migolovanov commented 2 years ago

Issue 1: yes, it is "a thing" of burp modern versions - first request has different hash compared to the following ones. Issue 2: thank you very much for the clarifications, I didn't know most of the things you mentioned. Can confirm that with latest patches, hashes are the same. Thank you very much for the patience and quick fix of the issue.