tls Handshake bug

[lz520520]

Tls.UClient uses Chrome, edge and other browser fingerprints. After Handshake(), the server may actively send "application data", but it will not be judged as a handshake process, causing the client to receive extra useless data, such as this host 42.194.227.196:443

[gaukas]

Hi @lz520520,

the server may actively send "application data"

You may set KeyLogWriter in Config then decrypt & inspect the extra (alleged) Application Data pkt received.

such as this host 42.194.227.196:443

uTLS comes with no warranty and we just could not provide support for any specific TLS server.

[lz520520]

But this is a CDN. The client needs to communicate with the server through the CDN. The IP I provided can be directly used for testing

[gaukas]

So possibly multiple retry/roundtrips?

Btw don't post contents in your keylog file, it is not safe.

You may want to see this link for instructions to use the keylog file with wireshark: https://resources.infosecinstitute.com/topic/decrypting-ssl-tls-traffic-with-wireshark/

[gaukas]

But this is a CDN. The client needs to communicate with the server through the CDN.

Just to confirm your question here: connecting to CDN node 42.194.227.196:443 with uTLS sees an extra Application Data pkt, while connecting to the origin server does not see the extra Application Data pkt. Is that correct?

[lz520520]

But this is a CDN. The client needs to communicate with the server through the CDN.

Just to confirm your question here: connecting to CDN node 42.194.227.196:443 with uTLS sees an extra Application Data pkt, while connecting to the origin server does not see the extra Application Data pkt. Is that correct?

Yes, you are right. I will also try to decrypt tls for analysis, but because I am not familiar with utls, I may need to spend time reviewing the code to debug

[gaukas]

As a follow up on this issue, could you or anyone else possibly provide a decrypted pcap (or as a screenshot) showing the content of the "extra" application data record?

[lz520520]

I will provide you with a demo. It is normal to use HelloChrome_102 to access www.microsoft.com, but an error will be reported when accessing 42.194.227.196:443, while using HelloGolang to access 42.194.227.196:443. I tested and found that accessing github also has this problem

package main

import (
    "bufio"
    "fmt"
    "net"
    "net/http"
    "net/http/httputil"
    "net/url"
    "time"

    tls "github.com/refraction-networking/utls"
    "golang.org/x/net/http2"
)

var (
    dialTimeout   = time.Duration(15) * time.Second
    sessionTicket = []uint8(`Here goes phony session ticket: phony enough to get into ASCII range
Ticket could be of any length, but for camouflage purposes it's better to use uniformly random contents
and common length. See https://tlsfingerprint.io/session-tickets`)
)

func HttpGetByHelloID(hostname string, addr string, helloID tls.ClientHelloID) (*http.Response, error) {
    config := tls.Config{InsecureSkipVerify: true}
    dialConn, err := net.DialTimeout("tcp", addr, dialTimeout)
    if err != nil {
        return nil, fmt.Errorf("net.DialTimeout error: %+v", err)
    }
    uTlsConn := tls.UClient(dialConn, &config, helloID)
    defer uTlsConn.Close()

    err = uTlsConn.Handshake()
    if err != nil {
        return nil, fmt.Errorf("uTlsConn.Handshake() error: %+v", err)
    }

    return httpGetOverConn(uTlsConn, hostname, "http/1.1")
}

func main() {
    var response *http.Response
    var err error

    response, err = HttpGetByHelloID("www.microsoft.com", "www.microsoft.com:443", tls.HelloChrome_102)
    if err != nil {
        fmt.Printf("#>www.microsoft.com:443 HttpGetByHelloID(HelloChrome_Auto) failed: %+v\n", err)
    } else {
        fmt.Printf("#>www.microsoft.com:443 HttpGetByHelloID(HelloChrome_Auto) response: %+s\n", dumpResponseNoBody(response))
    }

    response, err = HttpGetByHelloID("42.194.227.196:443", "42.194.227.196:443", tls.HelloChrome_102)
    if err != nil {
        fmt.Printf("#>42.194.227.196:443 HttpGetByHelloID(HelloChrome_Auto) failed: %+v\n", err)
    } else {
        fmt.Printf("#>42.194.227.196:443 HttpGetByHelloID(HelloChrome_Auto) response: %+s\n", dumpResponseNoBody(response))
    }

    response, err = HttpGetByHelloID("42.194.227.196:443", "42.194.227.196:443", tls.HelloGolang)
    if err != nil {
        fmt.Printf("#>42.194.227.196:443 HttpGetByHelloID(HelloGolang) failed: %+v\n", err)
    } else {
        fmt.Printf("#>42.194.227.196:443 HttpGetByHelloID(HelloGolang) response: %+s\n", dumpResponseNoBody(response))
    }

    response, err = HttpGetByHelloID("github.com:443", "github.com:443", tls.HelloChrome_102)
    if err != nil {
        fmt.Printf("#>github.com:443 HttpGetByHelloID(HelloChrome_Auto) failed: %+v\n", err)
    } else {
        fmt.Printf("#>github.com:443 HttpGetByHelloID(HelloChrome_Auto) response: %+s\n", dumpResponseNoBody(response))
    }

    response, err = HttpGetByHelloID("github.com:443", "github.com:443", tls.HelloGolang)
    if err != nil {
        fmt.Printf("#>github.com:443 HttpGetByHelloID(HelloGolang) failed: %+v\n", err)
    } else {
        fmt.Printf("#>github.com:443 HttpGetByHelloID(HelloGolang) response: %+s\n", dumpResponseNoBody(response))
    }
    return
}

func httpGetOverConn(conn net.Conn, hostname string, alpn string) (*http.Response, error) {
    req := &http.Request{
        Method: "GET",
        URL:    &url.URL{Host: hostname},
        Header: make(http.Header),
        Host:    hostname,
    }

    switch alpn {
    case "h2":
        req.Proto = "HTTP/2.0"
        req.ProtoMajor = 2
        req.ProtoMinor = 0

        tr := http2.Transport{}
        cConn, err := tr.NewClientConn(conn)
        if err != nil {
            return nil, err
        }
        return cConn.RoundTrip(req)
    case "http/1.1", "":
        req.Proto = "HTTP/1.1"
        req.ProtoMajor = 1
        req.ProtoMinor = 1

        err := req.Write(conn)
        if err != nil {
            return nil, err
        }
        return http.ReadResponse(bufio.NewReader(conn), req)
    default:
        return nil, fmt.Errorf("unsupported ALPN: %v", alpn)
    }
}

func dumpResponseNoBody(response *http.Response) string {
    resp, err := httputil.DumpResponse(response, false)
    if err != nil {
        return fmt.Sprintf("failed to dump response: %v", err)
    }
    return string(resp)
}

[gaukas]

The error you encountered at 42.194.227.196:443 is perfectly normal. It only indicates H2 is selected by the server thus your HTTP/1 client won't be able to decode the stream.

However the unexpected EOF at GitHub is concerning. Further investigation needed on that one.

[gaukas]

Just confirmed that it is also the case for github.com. H2 is the result of ALPN negotiation but you were hardcoding HTTP/1.1. Seemingly 42.194.227.196 fails differently (sending back another H2 response, probably indicating an error) from github.com, which unexpectedly closes the connection.

[gaukas]

You might want to use a better HTTP client with more flexibility in terms of the ALPN handling.

I would recommend using imroc/req:

• https://req.cool/docs/tutorial/tls-fingerprinting/
• https://github.com/imroc/req/discussions/198

I will close this issue for now since it is solved. If there are any other issues, let me know by opening another.