search

internetarchive / warcprox

WARC writing MITM HTTP/S proxy
371 stars 54 forks source link

Randomize TLS fingerprint #175

Closed vbanos closed 2 years ago

vbanos commented 2 years ago

Create a random TLS fingerprint per HTTPS connection to avoid TLS fingerprinting.

vbanos commented 2 years ago

We use Ubuntu 20.04.4 LTS with OpenSSL 1.1.1f 31 Mar 2020. We use this excellent tool to see our TLS fingerprint https://tools.scrapfly.io/api/fp/ja3?extended=1

The master TLS fingerprint is:

{"digest":"40adfd923eb82b89d8836ba37a19bca1","ja3":"771,4866-4867-4865-49196-49200-159-52393-52392-52394-49195-49199-158-49188-49192-107-49187-49191-103-49162-49172-57-49161-49171-51-157-156-61-60-53-47-255,0-11-10-35-22-23-13-43-45-51,29-23-30-25-24,0-1-2","tls":{"version":"0x303 - TLS 1.2","ciphers":["TLS_AES_256_GCM_SHA384","TLS_CHACHA20_POLY1305_SHA256","TLS_AES_128_GCM_SHA256","TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384","TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384","TLS_DHE_RSA_WITH_AES_256_GCM_SHA384","TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305_SHA256","TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305_SHA256","TLS_DHE_RSA_WITH_CHACHA20_POLY1305_SHA256","TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256","TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256","TLS_DHE_RSA_WITH_AES_128_GCM_SHA256","TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA384","TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384","TLS_DHE_RSA_WITH_AES_256_CBC_SHA256","TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256","TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256","TLS_DHE_RSA_WITH_AES_128_CBC_SHA256","TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA","TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA","TLS_DHE_RSA_WITH_AES_256_CBC_SHA","TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA","TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA","TLS_DHE_RSA_WITH_AES_128_CBC_SHA","TLS_RSA_WITH_AES_256_GCM_SHA384","TLS_RSA_WITH_AES_128_GCM_SHA256","TLS_RSA_WITH_AES_256_CBC_SHA256","TLS_RSA_WITH_AES_128_CBC_SHA256","TLS_RSA_WITH_AES_256_CBC_SHA","TLS_RSA_WITH_AES_128_CBC_SHA","0x00FF"],"curves":["X25519 (29)","secp256r1 (23)","X448 (30)","secp521r1 (25)","secp384r1 (24)"],"extensions":["server_name (0) (IANA)","ec_point_formats (11) (IANA)","supported_groups (10) (IANA)","session_ticket (35) (IANA)","encrypt_then_mac (22) (IANA)","extended_master_secret (23) (IANA)","signature_algorithms (13) (IANA)","supported_versions (43) (IANA)","psk_key_exchange_modes (45) (IANA)","key_share (51) (IANA)","padding (21) (IANA)"],"points":["0","1","2"],"protocols":null,"versions":["0x303 - TLS 1.2","0x302 - TLS 1.1","0x301 - TLS 1.0","0x300 - SSL 3.0"]}}'

Using this MR, the TLS fingerprint changes with every request. I make the same HTTP request 4 times and display the results:

{"digest":"c5a3dcb4dc4437886fd4c9ba6e1a3904","ja3":"771,4866-4867-4865-159-158-107-103-57-51-49196-49200-49195-49199-49188-49192-49187-49191-49162-49172-49161-49171-52394-157-156-61-60-53-47-52393-52392-255,0-11-10-35-22-23-13-43-45-51,29-23-30-25-24,0-1-2","tls":{"version":"0x303 - TLS 1.2","ciphers":["TLS_AES_256_GCM_SHA384","TLS_CHACHA20_POLY1305_SHA256","TLS_AES_128_GCM_SHA256","TLS_DHE_RSA_WITH_AES_256_GCM_SHA384","TLS_DHE_RSA_WITH_AES_128_GCM_SHA256","TLS_DHE_RSA_WITH_AES_256_CBC_SHA256","TLS_DHE_RSA_WITH_AES_128_CBC_SHA256","TLS_DHE_RSA_WITH_AES_256_CBC_SHA","TLS_DHE_RSA_WITH_AES_128_CBC_SHA","TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384","TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384","TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256","TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256","TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA384","TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384","TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256","TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256","TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA","TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA","TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA","TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA","TLS_DHE_RSA_WITH_CHACHA20_POLY1305_SHA256","TLS_RSA_WITH_AES_256_GCM_SHA384","TLS_RSA_WITH_AES_128_GCM_SHA256","TLS_RSA_WITH_AES_256_CBC_SHA256","TLS_RSA_WITH_AES_128_CBC_SHA256","TLS_RSA_WITH_AES_256_CBC_SHA","TLS_RSA_WITH_AES_128_CBC_SHA","TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305_SHA256","TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305_SHA256","0x00FF"],"curves":["X25519 (29)","secp256r1 (23)","X448 (30)","secp521r1 (25)","secp384r1 (24)"],"extensions":["server_name (0) (IANA)","ec_point_formats (11) (IANA)","supported_groups (10) (IANA)","session_ticket (35) (IANA)","encrypt_then_mac (22) (IANA)","extended_master_secret (23) (IANA)","signature_algorithms (13) (IANA)","supported_versions (43) (IANA)","psk_key_exchange_modes (45) (IANA)","key_share (51) (IANA)","padding (21) (IANA)"],"points":["0","1","2"],"protocols":null,"versions":["0x303 - TLS 1.2","0x302 - TLS 1.1","0x301 - TLS 1.0","0x300 - SSL 3.0"]}}'

{"digest":"b564d96f5dbf9a532ae9cae5c8f68b2d","ja3":"771,4866-4867-4865-52394-49196-49200-49195-49199-157-156-52393-52392-159-158-61-60-53-47-107-103-57-51-49188-49192-49187-49191-49162-49172-49161-49171-255,0-11-10-35-22-23-13-43-45-51,29-23-30-25-24,0-1-2","tls":{"version":"0x303 - TLS 1.2","ciphers":["TLS_AES_256_GCM_SHA384","TLS_CHACHA20_POLY1305_SHA256","TLS_AES_128_GCM_SHA256","TLS_DHE_RSA_WITH_CHACHA20_POLY1305_SHA256","TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384","TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384","TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256","TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256","TLS_RSA_WITH_AES_256_GCM_SHA384","TLS_RSA_WITH_AES_128_GCM_SHA256","TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305_SHA256","TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305_SHA256","TLS_DHE_RSA_WITH_AES_256_GCM_SHA384","TLS_DHE_RSA_WITH_AES_128_GCM_SHA256","TLS_RSA_WITH_AES_256_CBC_SHA256","TLS_RSA_WITH_AES_128_CBC_SHA256","TLS_RSA_WITH_AES_256_CBC_SHA","TLS_RSA_WITH_AES_128_CBC_SHA","TLS_DHE_RSA_WITH_AES_256_CBC_SHA256","TLS_DHE_RSA_WITH_AES_128_CBC_SHA256","TLS_DHE_RSA_WITH_AES_256_CBC_SHA","TLS_DHE_RSA_WITH_AES_128_CBC_SHA","TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA384","TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384","TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256","TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256","TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA","TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA","TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA","TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA","0x00FF"],"curves":["X25519 (29)","secp256r1 (23)","X448 (30)","secp521r1 (25)","secp384r1 (24)"],"extensions":["server_name (0) (IANA)","ec_point_formats (11) (IANA)","supported_groups (10) (IANA)","session_ticket (35) (IANA)","encrypt_then_mac (22) (IANA)","extended_master_secret (23) (IANA)","signature_algorithms (13) (IANA)","supported_versions (43) (IANA)","psk_key_exchange_modes (45) (IANA)","key_share (51) (IANA)","padding (21) (IANA)"],"points":["0","1","2"],"protocols":null,"versions":["0x303 - TLS 1.2","0x302 - TLS 1.1","0x301 - TLS 1.0","0x300 - SSL 3.0"]}}

{"digest":"087f29263c587636197b8a23f6d0af25","ja3":"771,4866-4867-4865-52394-52393-52392-49196-49200-49195-49199-157-156-61-60-53-47-159-158-49188-49192-49187-49191-49162-49172-49161-49171-107-103-57-51-255,0-11-10-35-22-23-13-43-45-51,29-23-30-25-24,0-1-2","tls":{"version":"0x303 - TLS 1.2","ciphers":["TLS_AES_256_GCM_SHA384","TLS_CHACHA20_POLY1305_SHA256","TLS_AES_128_GCM_SHA256","TLS_DHE_RSA_WITH_CHACHA20_POLY1305_SHA256","TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305_SHA256","TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305_SHA256","TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384","TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384","TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256","TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256","TLS_RSA_WITH_AES_256_GCM_SHA384","TLS_RSA_WITH_AES_128_GCM_SHA256","TLS_RSA_WITH_AES_256_CBC_SHA256","TLS_RSA_WITH_AES_128_CBC_SHA256","TLS_RSA_WITH_AES_256_CBC_SHA","TLS_RSA_WITH_AES_128_CBC_SHA","TLS_DHE_RSA_WITH_AES_256_GCM_SHA384","TLS_DHE_RSA_WITH_AES_128_GCM_SHA256","TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA384","TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384","TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256","TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256","TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA","TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA","TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA","TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA","TLS_DHE_RSA_WITH_AES_256_CBC_SHA256","TLS_DHE_RSA_WITH_AES_128_CBC_SHA256","TLS_DHE_RSA_WITH_AES_256_CBC_SHA","TLS_DHE_RSA_WITH_AES_128_CBC_SHA","0x00FF"],"curves":["X25519 (29)","secp256r1 (23)","X448 (30)","secp521r1 (25)","secp384r1 (24)"],"extensions":["server_name (0) (IANA)","ec_point_formats (11) (IANA)","supported_groups (10) (IANA)","session_ticket (35) (IANA)","encrypt_then_mac (22) (IANA)","extended_master_secret (23) (IANA)","signature_algorithms (13) (IANA)","supported_versions (43) (IANA)","psk_key_exchange_modes (45) (IANA)","key_share (51) (IANA)","padding (21) (IANA)"],"points":["0","1","2"],"protocols":null,"versions":["0x303 - TLS 1.2","0x302 - TLS 1.1","0x301 - TLS 1.0","0x300 - SSL 3.0"]}}

{"digest":"3eaf8f467b80fc6537e1a70a7ec72f2c","ja3":"771,4866-4867-4865-49196-49200-49195-49199-159-158-157-156-61-60-53-47-52393-52392-49188-49192-49187-49191-49162-49172-49161-49171-52394-107-103-57-51-255,0-11-10-35-22-23-13-43-45-51,29-23-30-25-24,0-1-2","tls":{"version":"0x303 - TLS 1.2","ciphers":["TLS_AES_256_GCM_SHA384","TLS_CHACHA20_POLY1305_SHA256","TLS_AES_128_GCM_SHA256","TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384","TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384","TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256","TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256","TLS_DHE_RSA_WITH_AES_256_GCM_SHA384","TLS_DHE_RSA_WITH_AES_128_GCM_SHA256","TLS_RSA_WITH_AES_256_GCM_SHA384","TLS_RSA_WITH_AES_128_GCM_SHA256","TLS_RSA_WITH_AES_256_CBC_SHA256","TLS_RSA_WITH_AES_128_CBC_SHA256","TLS_RSA_WITH_AES_256_CBC_SHA","TLS_RSA_WITH_AES_128_CBC_SHA","TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305_SHA256","TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305_SHA256","TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA384","TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384","TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256","TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256","TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA","TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA","TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA","TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA","TLS_DHE_RSA_WITH_CHACHA20_POLY1305_SHA256","TLS_DHE_RSA_WITH_AES_256_CBC_SHA256","TLS_DHE_RSA_WITH_AES_128_CBC_SHA256","TLS_DHE_RSA_WITH_AES_256_CBC_SHA","TLS_DHE_RSA_WITH_AES_128_CBC_SHA","0x00FF"],"curves":["X25519 (29)","secp256r1 (23)","X448 (30)","secp521r1 (25)","secp384r1 (24)"],"extensions":["server_name (0) (IANA)","ec_point_formats (11) (IANA)","supported_groups (10) (IANA)","session_ticket (35) (IANA)","encrypt_then_mac (22) (IANA)","extended_master_secret (23) (IANA)","signature_algorithms (13) (IANA)","supported_versions (43) (IANA)","psk_key_exchange_modes (45) (IANA)","key_share (51) (IANA)","padding (21) (IANA)"],"points":["0","1","2"],"protocols":null,"versions":["0x303 - TLS 1.2","0x302 - TLS 1.1","0x301 - TLS 1.0","0x300 - SSL 3.0"]}}

vbanos commented 2 years ago

The ideal solution would be to have a TLS fingerprint identical to the one used by a popular web browser but this isn't possible because from my understanding they are using their own SSL libraries and not openssl.

Also it use important to highlight that: TLS 1.3 cipher suites are managed through a different interface not exposed by CPython (yet!) and are enabled by default if they're available.

vbanos commented 2 years ago

@galgeek as I wrote in my MR:

Ref and detailed description about cipher selection at https://github.com/urllib3/urllib3/blob/f070ec2e6f6c545f40d9196e5246df10c72e48e1/src/urllib3/util/ssl_.py#L170

galgeek commented 2 years ago

Yes — I noticed that your list in the MR differs from the urllib3 list, though looking harder, I see it's likely a difference only in ordering.