cloudwego / hertz

Go HTTP framework with high-performance and strong-extensibility for building micro-services.
https://www.cloudwego.io
Apache License 2.0
5.04k stars 491 forks source link

work wrong when use client.NewClient() to fetch a https url, even if setted tlsConfig: InsecureSkipVerify=true #1145

Closed karliehgf closed 1 month ago

karliehgf commented 1 month ago

Describe the bug i use client.NewClient() to fetch a https request, if the target server will enforce HTTP to HTTPS redirection, i will fetch the wrong http response. example code:

    tlsConfig := &tls.Config{
        InsecureSkipVerify: true,
    }
    c, _ := client.NewClient(client.WithTLSConfig(tlsConfig), client.WithDialer(standard.NewDialer()))
    req, resp := protocol.AcquireRequest(), protocol.AcquireResponse()
    req.SetRequestURI("https://www.baidu.com")

    req.SetMethod("GET")
    err := c.Do(context.Background(), req, resp)

    if err != nil {
        fmt.Printf("%s", err)
    }
    // fmt.Println(code)
    fmt.Printf("%s", resp.Body())

result: image

however, it worked well if i change the request uri to "http://www.baidu.com". And the result is: image

Hertz version:

hertz v0.9.1

Environment:

go version: go1.21.8

Additional context

Add any other context about the problem here.

FGYFFFF commented 1 month ago

问题原因:hertz client 发请求默认会带一个 user-agent header;可能 baidu 的接口认为这个 user-agent 是一个爬虫或者什么的,就返回了一个 js 重定向页面(具体原因目前不清楚)。 解决方法:配置下不带默认user-agent就可以了

c, _ := NewClient(WithDialer(standard.NewDialer()), WithNoDefaultUserAgentHeader(true))