strange timeout use *fasthttp.Client do request

[soulseen]

I have a gateway service that forwards incoming requests to another backend service(65 pod) ~ 40 rps, but after it comes online it occasionally times out requests (more than 30s), but the CPU/MEM is smooth and very low! here is my code example:

Client := NewClient()

func Send() {
    ctx := &fasthttp.RequestCtx{}

    response := fasthttp.AcquireResponse()
    defer fasthttp.ReleaseResponse(response)

    req := fasthttp.AcquireRequest()
    defer fasthttp.ReleaseRequest(req)

    //set header
    req.Header.SetMethod(fasthttp.MethodPost)
    req.Header.Set("Content-Type", "application/json; charset=UTF-8")

    req.SetBodyRaw(jsonData)

        // do request
        log.Info("start send")
        err = client.Do(req, response)
        log.Info("send end")
        if err != nil {
            log.Error("error: %s", err.Error())
        }
}

func NewClient() *fasthttp.Client {
    return &fasthttp.Client{
        ReadTimeout:                   60000 * time.Millisecond,
        WriteTimeout:                  60000 * time.Millisecond,
        MaxIdleConnDuration:           30000 * time.Millisecond,
        NoDefaultUserAgentHeader:      true, // Don't send: User-Agent: fasthttp
        DisableHeaderNamesNormalizing: true, // If you set the case on your headers correctly you can enable this
        DisablePathNormalizing:        true,
        MaxConnsPerHost:               65536,
        // increase DNS cache time to an hour instead of default minute
        Dial: (&fasthttp.TCPDialer{
            Concurrency:      4096,
            DNSCacheDuration: time.Hour,
        }).Dial,
    }
}

Additional information: fasthttp version: v1.48.0

When a timeout occurs, I found: Gateway Log: "ts":"2023-09-04 15:00:00", "start send" "ts":"2023-09-04 15:30:00", "start end" "ts":"2023-09-04 15:30:00", "error: timeout" Backend service log: "ts":"2023-09-04 15:43:00", "received request"

Strangely, No warnings, no instance restart, no memory / cpu peaks, no memory leak or alloc problems. And I can confirm the backend service is working well and the network is fine, If replace it to Nginx, the timeout does not occur! And I can not reproduce in test env. What can it be?

[soulseen]

@erikdubbelboer can you help to take a look? And how can I dig it? thanks a lot.

[erikdubbelboer]

What was it?

[soulseen]

it is the backend app problem