Can't proxy / replay netflix.com

Corb3nik commented 9 months ago

Community Note

Please vote on this issue by adding a 👍 reaction to the original issue to help us know what to work on
Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request

Bug description

Visit https://www.netflix.com/ca/ while proxying through Caido.
Refresh a few times

Notice that you'll get a mix of 403s, 503s, no responses

This is also reproducible when sending requests to netflix.com through replay

Environment & setup

OS:  OS version:  Caido client:  Caido version:

Sytten commented 9 months ago

This seems like it could be TLS fingerprinting / Bot Detection, thus we probably need https://github.com/caido/caido/issues/523 to fix it. The inconsistent behaviour looks like typical anti-scrapping strategies to mess with clients. Curl seems whitelisted for some reason.

Sytten commented 9 months ago

Here is the client hello of curl

Frame 62: 388 bytes on wire (3104 bits), 388 bytes captured (3104 bits) on interface en7, id 0
Ethernet II, Src: 80:6d:97:2c:10:57 (80:6d:97:2c:10:57), Dst: SagemcomBroa_c5:70:c6 (0c:ac:8a:c5:70:c6)
Internet Protocol Version 4, Src: 192.168.10.55, Dst: 3.137.75.83
Transmission Control Protocol, Src Port: 57008, Dst Port: 443, Seq: 1, Ack: 1, Len: 322
Transport Layer Security
    TLSv1.3 Record Layer: Handshake Protocol: Client Hello
        Content Type: Handshake (22)
        Version: TLS 1.0 (0x0301)
        Length: 317
        Handshake Protocol: Client Hello
            Handshake Type: Client Hello (1)
            Length: 313
            Version: TLS 1.2 (0x0303)
            Random: 35afbdea692385f1c03652c2275d93146db81cda425152e0d9d047e64fa61a96
            Session ID Length: 32
            Session ID: ae04d30de5929cfc005a321b449ddbd495c2f6793836b44622a271792095ea17
            Cipher Suites Length: 98
            Cipher Suites (49 suites)
            Compression Methods Length: 1
            Compression Methods (1 method)
                Compression Method: null (0)
            Extensions Length: 142
            Extension: supported_versions (len=9) TLS 1.3, TLS 1.2, TLS 1.1, TLS 1.0
                Type: supported_versions (43)
                Length: 9
                Supported Versions length: 8
                Supported Version: TLS 1.3 (0x0304)
                Supported Version: TLS 1.2 (0x0303)
                Supported Version: TLS 1.1 (0x0302)
                Supported Version: TLS 1.0 (0x0301)
            Extension: key_share (len=38) x25519
                Type: key_share (51)
                Length: 38
                Key Share extension
                    Client Key Share Length: 36
                    Key Share Entry: Group: x25519, Key Exchange length: 32
                        Group: x25519 (29)
                        Key Exchange Length: 32
                        Key Exchange: 476a85a6abb52d0f5441bb2b94545189c073f61e31ef6ecae4fc190485c29c03
            Extension: server_name (len=20) name=www.netflix.com
                Type: server_name (0)
                Length: 20
                Server Name Indication extension
                    Server Name list length: 18
                    Server Name Type: host_name (0)
                    Server Name length: 15
                    Server Name: www.netflix.com
            Extension: ec_point_formats (len=2)
                Type: ec_point_formats (11)
                Length: 2
                EC point formats Length: 1
                Elliptic curves point formats (1)
                    EC point format: uncompressed (0)
            Extension: supported_groups (len=10)
                Type: supported_groups (10)
                Length: 10
                Supported Groups List Length: 8
                Supported Groups (4 groups)
                    Supported Group: x25519 (0x001d)
                    Supported Group: secp256r1 (0x0017)
                    Supported Group: secp384r1 (0x0018)
                    Supported Group: secp521r1 (0x0019)
            Extension: signature_algorithms (len=24)
                Type: signature_algorithms (13)
                Length: 24
                Signature Hash Algorithms Length: 22
                Signature Hash Algorithms (11 algorithms)
                    Signature Algorithm: rsa_pss_rsae_sha512 (0x0806)
                        Signature Hash Algorithm Hash: Unknown (8)
                        Signature Hash Algorithm Signature: Unknown (6)
                    Signature Algorithm: rsa_pkcs1_sha512 (0x0601)
                        Signature Hash Algorithm Hash: SHA512 (6)
                        Signature Hash Algorithm Signature: RSA (1)
                    Signature Algorithm: ecdsa_secp521r1_sha512 (0x0603)
                        Signature Hash Algorithm Hash: SHA512 (6)
                        Signature Hash Algorithm Signature: ECDSA (3)
                    Signature Algorithm: rsa_pss_rsae_sha384 (0x0805)
                        Signature Hash Algorithm Hash: Unknown (8)
                        Signature Hash Algorithm Signature: Unknown (5)
                    Signature Algorithm: rsa_pkcs1_sha384 (0x0501)
                        Signature Hash Algorithm Hash: SHA384 (5)
                        Signature Hash Algorithm Signature: RSA (1)
                    Signature Algorithm: ecdsa_secp384r1_sha384 (0x0503)
                        Signature Hash Algorithm Hash: SHA384 (5)
                        Signature Hash Algorithm Signature: ECDSA (3)
                    Signature Algorithm: rsa_pss_rsae_sha256 (0x0804)
                        Signature Hash Algorithm Hash: Unknown (8)
                        Signature Hash Algorithm Signature: SM2 (4)
                    Signature Algorithm: rsa_pkcs1_sha256 (0x0401)
                        Signature Hash Algorithm Hash: SHA256 (4)
                        Signature Hash Algorithm Signature: RSA (1)
                    Signature Algorithm: ecdsa_secp256r1_sha256 (0x0403)
                        Signature Hash Algorithm Hash: SHA256 (4)
                        Signature Hash Algorithm Signature: ECDSA (3)
                    Signature Algorithm: rsa_pkcs1_sha1 (0x0201)
                        Signature Hash Algorithm Hash: SHA1 (2)
                        Signature Hash Algorithm Signature: RSA (1)
                    Signature Algorithm: ecdsa_sha1 (0x0203)
                        Signature Hash Algorithm Hash: SHA1 (2)
                        Signature Hash Algorithm Signature: ECDSA (3)
            Extension: application_layer_protocol_negotiation (len=11)
                Type: application_layer_protocol_negotiation (16)
                Length: 11
                ALPN Extension Length: 9
                ALPN Protocol
                    ALPN string length: 8
                    ALPN Next Protocol: http/1.1
            [JA4: t13d4907h1_0d8feac7bc37_7395dae3b2f3]
            [JA4_r [truncated]: t13d4907h1_0004,0005,000a,0016,002f,0033,0035,0039,003c,003d,0041,0045,0067,006b,0081,0084,0088,009c,009d,009e,009f,00ba,00be,00c0,00c4,00ff,1301,1302,1303,c007,c008,c009,c00a,c011,c012,c013,c014,c023,c024,c027,c028,c02b]
            [JA3 Fullstring [truncated]: 771,4867-4866-4865-52393-52392-52394-49200-49196-49192-49188-49172-49162-159-107-57-65413-196-136-129-157-61-53-192-132-49199-49195-49191-49187-49171-49161-158-103-51-190-69-156-60-47-186-65-49169-49159-5-4-4917]
            [JA3: 375c6162a492dfbf2795909110ce8424]

caido / caido