Using JA3 to build Spec results in errors

[Harusakii]

When using already preset ClientSpecs (like HelloChrome_105), it works fine as intended. Then getting the ja3 String (771,4865-4866-4867-49195-49199-49196-49200-52393-52392-49171-49172-156-157-47-53,0-23-65281-10-11-35-16-5-13-18-51-45-43-27-21,29-23-24,0 the one I get for Chrome105) and trying to make a CustomClient with that doesnt work.

Getting the error local error: tls: bad record MAC, and on further investigation it happens here at conn.go. The "real" error is this: cipher: message authentication failed

What I did see is that the JA3 string, when both (Chrome105 and CustomClient from JA3 of Chrome105) parsed at https://tls.peet.ws/api/tls both have the same ja3 and ja3 hash, the only difference is one works for sites and one doesnt and I dont know why (without or with grease, both dont work in the ja3 customclient).

[bogdanfinn]

@Harusakii the issue here is that the JA3 string does not define every information which is needed to build a full functional tls client profile. So when building a tls client profile out of the ja3 string i have to "make assumptions" on some extensions values which are currently hardcoded (see here https://github.com/bogdanfinn/tls-client/blob/master/ja3.go#L96). For example the SignatureAlgorithms Extension.

Obviously my assumptions are not right in every case. i need to revisit the part of the code to change the "preset" used when building the client out of the ja3 string.

[Harusakii]

@Harusakii the issue here is that the JA3 string does not define every information which is needed to build a full functional tls client profile. So when building a tls client profile out of the ja3 string i have to "make assumptions" on some extensions values which are currently hardcoded (see here https://github.com/bogdanfinn/tls-client/blob/master/ja3.go#L96). For example the SignatureAlgorithms Extension.

Obviously my assumptions are not right in every case. i need to revisit the part of the code to change the "preset" used when building the client out of the ja3 string.

yeah, I know that the ja3 doesnt contain everything, but the thing is even when I do this:

return &tls.ClientHelloSpec{CipherSuites: []uint16{
        tls.GREASE_PLACEHOLDER,
        tls.TLS_AES_128_GCM_SHA256,
        tls.TLS_AES_256_GCM_SHA384,
        tls.TLS_CHACHA20_POLY1305_SHA256,
        tls.TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,
        tls.TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,
        tls.TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,
        tls.TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,
        tls.TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,
        tls.TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305,
        tls.TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA,
        tls.TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA,
        tls.TLS_RSA_WITH_AES_128_GCM_SHA256,
        tls.TLS_RSA_WITH_AES_256_GCM_SHA384,
        tls.TLS_RSA_WITH_AES_128_CBC_SHA,
        tls.TLS_RSA_WITH_AES_256_CBC_SHA,
    },
        CompressionMethods: []uint8{
            uint8(0),
        },
        Extensions: []tls.TLSExtension{
            &tls.UtlsGREASEExtension{},
            &tls.SNIExtension{},
            &tls.UtlsExtendedMasterSecretExtension{},
            &tls.RenegotiationInfoExtension{Renegotiation: tls.RenegotiateOnceAsClient},
            &tls.SupportedCurvesExtension{[]tls.CurveID{
                tls.CurveID(tls.GREASE_PLACEHOLDER),
                tls.X25519,
                tls.CurveP256,
                tls.CurveP384,
            }},
            &tls.SupportedPointsExtension{SupportedPoints: []byte{
                uint8(0),
            }},
            &tls.SessionTicketExtension{},
            &tls.ALPNExtension{AlpnProtocols: []string{"h2", "http/1.1"}},
            &tls.StatusRequestExtension{},
            &tls.SignatureAlgorithmsExtension{SupportedSignatureAlgorithms: []tls.SignatureScheme{
                tls.ECDSAWithP256AndSHA256,
                tls.PSSWithSHA256,
                tls.PKCS1WithSHA256,
                tls.ECDSAWithP384AndSHA384,
                tls.PSSWithSHA384,
                tls.PKCS1WithSHA384,
                tls.PSSWithSHA512,
                tls.PKCS1WithSHA512,
            }},
            &tls.SCTExtension{},
            &tls.KeyShareExtension{[]tls.KeyShare{
                {Group: tls.CurveID(tls.GREASE_PLACEHOLDER), Data: []byte{0}},
                {Group: tls.X25519},
            }},
            &tls.PSKKeyExchangeModesExtension{[]uint8{
                tls.PskModeDHE,
            }},
            &tls.SupportedVersionsExtension{[]uint16{
                //tls.GREASE_PLACEHOLDER,
                tls.VersionTLS13,
                tls.VersionTLS12,
                tls.VersionTLS11,
                tls.VersionTLS10,
            }},
            &tls.UtlsCompressCertExtension{[]tls.CertCompressionAlgo{
                tls.CertCompressionBrotli,
            }},
            &tls.ApplicationSettingsExtension{SupportedProtocols: []string{"h2"}},
            &tls.UtlsGREASEExtension{},
            &tls.UtlsPaddingExtension{GetPaddingLen: tls.BoringPaddingStyle},
        },
    }, nil

in the ja3.go in springToSpec() (just to test it out) (its the exact copy from u_parrots.go), it gives me this error unsupported Curve in KeyShareExtension: 14906.To mimic it, fill the Data(key) field manually. and when I comment out the GREASE stuff, the previously mentioned error comes up, even though the client specs are literally 1:1 to u_parrots (HelloClient105)

thats the reason I am a bit confused lol, the normal HelloChrome105 works fine (with or without Grease), but even when copying the exact Spec and setting it there seems to do sth wrong :/

//update the problem with the unsupported Curve in KeyShareExtension comes from using GREASE in the KeyShareExtension with using the client from ja3,

when using the Chrome105, those are the keyshare values (printing them in the func ApplyPreset) and it works fine:

[{2570 [0]} {29 []}]
[{2570 [0]} {29 []}]

using the same Spec from Chrome105 in the ja3 converting func (basically not building spec by ja3 but hardcoded) it doesnt work:

[{2570 [0]} {29 []}]
[{27242 [0]} {29 [197 117 50 36 33 121 20 64 175 125 144 185 101 48 55 3 174 24 236 73 209 42 115 108 57 91 157 244 142 158 237 28]}]

so it seems that the grease Curve ID changed (idk why), also the X25519 suddenly has data in it. So when just ignoring the changed GREASE (cause the Data stay at 0 either way), we then again get the local error: tls: bad record MAC error as previously mentioned

will keep trying to figure it out tho

//update 2 all those errors only come when TLS1.3 is activated, so basically theres sth going on when it is activated that I do not understand, what the normal spec does differently

using TLS1.2 for now works

[bogdanfinn]

@Harusakii can you maybe name the site you are testing against? and also give me your full tls.ClientHelloID{} not only the return value of the ClientHelloSpecFactory

i cant reproduce your issues when i use the above given values and adjust the code (ja3.go) according to your details

[Harusakii]

Well sure, those are some sites that I randomly tested and know it doesnt work on (for the custom Client):

https://malwarebytes.com

https://microsoft.com

https://adobe.com

and this is the ClientProfile from the custom client: {{Custom 1 nil 0x1110180} map[1:65536 3:1000 4:6291456 6:262144] [1 3 4 6] [:method :authority :scheme :path] 15663105 []}

for reference, this is the Chrome105 Client: {{Chrome 105 nil 0x663960} map[1:65536 3:1000 4:6291456 6:262144] [1 3 4 6] [:method :authority :scheme :path] 15663105 []}

Those sites all work with the normal chrome105 profile, but using the ja3 client builder even with returning the exact Spec copied from u_parrots.go to the SpecFactory, it just wont work on TLS 1.3 (local error: tls: bad record MAC, and unsupported Curve in KeyShareExtension if you use GREASE in the custom client )

[Harusakii]

Well sure, those are some sites that I randomly tested and know it doesnt work on (for the custom Client):

https://malwarebytes.com

https://microsoft.com

https://adobe.com

and this is the ClientProfile from the custom client: {{Custom 1 nil 0x1110180} map[1:65536 3:1000 4:6291456 6:262144] [1 3 4 6] [:method :authority :scheme :path] 15663105 []}

for reference, this is the Chrome105 Client: {{Chrome 105 nil 0x663960} map[1:65536 3:1000 4:6291456 6:262144] [1 3 4 6] [:method :authority :scheme :path] 15663105 []}

Those sites all work with the normal chrome105 profile, but using the ja3 client builder even with returning the exact Spec copied from u_parrots.go to the SpecFactory, it just wont work on TLS 1.3 (local error: tls: bad record MAC, and unsupported Curve in KeyShareExtension if you use GREASE in the custom client )

do those sites work for you when doing it like that @bogdanfinn ?

[bogdanfinn]

@Harusakii sorry for not replying that often ... i was sick the last week and was not able to work a lot on your issues. only replying time to time to comments

nevertheless i looked into the issue again and yeah you spotted an issue with the GetSpecFactorFromJa3String function.

Nice catch!

When i created a test scenario according to your description here i was also receiving unsupported Curve in KeyShareExtension: 14906.To mimic it, fill the Data(key) field manually error messages.

Now they should be gone.

See the change / fix here: https://github.com/bogdanfinn/tls-client/commit/9ccc0d2227345137a62668370b826cd537799c8b

There is some comment in the utls package which helps understanding the actual issue:

// ApplyPreset should only be used in conjunction with HelloCustom to apply custom specs.
// Fields of TLSExtensions that are slices/pointers are shared across different connections with
// same ClientHelloSpec. It is advised to use different specs and avoid any shared state.
func (uconn *UConn) ApplyPreset(p *ClientHelloSpec) error {

Nevertheless i want to rework parts of the custom client creation (not using predefined profiles) anyway because of the mentioned sentence in the beginning:

the issue here is that the JA3 string does not define every information which is needed to build a full functional tls client profile.

Please let me know if in terms of this open issue is still something left todo - if not lets close this and focus on the other open issues :-) if there is still some task left please quickly point me in that direction ...

[Harusakii]

sorry for not replying that often ... i was sick the last week and was not able to work a lot on your issues. only replying time to time to comments

Dont stress yourself out haha!

and yes, it fixed the Curve Problem and also the following local bad mac error.

Nevertheless i want to rework parts of the custom client creation (not using predefined profiles) anyway...

I think that would be great haha <3

Thank you very much for helping, hope it goes a long way here ^^