lookup <some.example.com>: i/o timeout

[nohackjustnoobb]

I am building a proxy server with fasthttp. I want the server to fetch images from multiple sources and only return the fastest one to the client. It works perfectly on my MacBook. However, when I moved the app to my home server, it became buggy. When handling multiple requests at the same time, some of the requests will raise an error. "lookup : i/o timeout" is what I get when I print the error.

here is some example code:

func shuffle(array []string) {
    for i := range array {
        j := rand.Intn(i + 1)
        array[i], array[j] = array[j], array[i]
    }
}

func fetchImage(driver string, destination string, genre string) (contentType string, body []byte) {
       // This part can be ignored. Basically, I am generating the URLs by replacing the server address. 
    var urls []string
    desUrl, _ := url.Parse(destination)

    for _, host := range settings[driver].(map[string]any)["genre"].(map[string]any)[genre].([]interface{}) {
        u, _ := url.Parse(host.(string))
        u.Path = desUrl.Path
        u.RawQuery = desUrl.RawQuery

        urls = append(urls, u.String())
    }

    if len(urls) == 0 {
        urls = append(urls, destination)
    }

    if len(urls) > maxClientEachRequest {
        shuffle(urls)
        urls = urls[:maxClientEachRequest]
    }

    var wg sync.WaitGroup
    var mu sync.Mutex
    wg.Add(1)

        // **** main part ****
    for _, url := range urls {
        go func(url string) {

            req := fasthttp.AcquireRequest()
            req.Header.SetMethod(fasthttp.MethodGet)

            // set headers
            for key, value := range settings[driver].(map[string]any)["headers"].(map[string]interface{}) {
                req.Header.Add(key, value.(string))
            }

            // send the request
            req.SetRequestURI(url)
            resp := fasthttp.AcquireResponse()
            err := client.Do(req, resp) // <-- error here
            fasthttp.ReleaseRequest(req)

            if err == nil {
                contentType = string(resp.Header.ContentType())
                body = resp.Body()

                // lock the mutex
                if mu.TryLock() {
                    wg.Done()
                }

            } else {
                fmt.Println(err)
            }

            fasthttp.ReleaseResponse(resp)

        }(url)
    }

    wg.Wait()

    return contentType, body
}

the client:

var client = &fasthttp.Client{
        NoDefaultUserAgentHeader: true,
        DisablePathNormalizing:   true,
    }

I don't think it is a problem with the server part so I am not giving the server code. The full project is here: https://github.com/nohackjustnoobb/Better-Manga-Proxy

additional information: CPU of my Macbook: M2 CPU of my home server: Ryzen 7 5700G There is no firewall enabled and it is running Debian.

(I am new to Golang and fasthttp so the code may look bad.)

[erikdubbelboer]

The DNS on your server doesn't seem to be working correctly. DefaultDialTimeout is set to 3 seconds and it doesn't seem to respond within that time. On your macbook you are probably using a different DNS server.

One BIG bug is that you are using the return value of resp.Body() after fasthttp.ReleaseResponse(resp). That is going to lead to undefined behavior.

[nohackjustnoobb]

Thank you for replying! I managed to fix the problem by using "tcp" instead of "udp" to access the DNS server as following:

client = &fasthttp.Client{
        NoDefaultUserAgentHeader: true,
        DisablePathNormalizing:   true,
        Dial: (&fasthttp.TCPDialer{
            Resolver: &net.Resolver{
                Dial: func(ctx context.Context, network, address string) (net.Conn, error) {
                    d := net.Dialer{}
                    return d.DialContext(ctx, "tcp", "192.168.2.1:53") // by default, it uses udp here.
                },
            },
        }).Dial,
    }

I don't know why it only happens on my home server but not on my MacBook (I also tried to use udp on my Macbook and it is fine). (Edited: The problem is gone after I changed the DNS server to Google One even using udp. Not sure if it is the problem with my original DNS server, adGuard home)

About the bug you mentioned, I changed the code like this:

// original "body = resp.Body()" only
body = make([]byte, len(resp.Body()))
copy(body, resp.Body())

I use a deep copy of the body instead of a shallow copy. Is it safe to release the response after this?

[erikdubbelboer]

Yes making a copy and returning that is good.