andelf / go-curl

golang curl(libcurl) binding.
Apache License 2.0
478 stars 129 forks source link

ERROR: curl: Out of memory #54

Closed ivan-kiselev closed 8 months ago

ivan-kiselev commented 7 years ago

Hi there! Thank you for opensourcing!

My go version: go version go1.7.4 linux/amd64 My libcurl version: 7.52.1-2

I have such function:

func curlIt(url string, host string)  {
    start := time.Now()
    easy := curl.EasyInit()
    defer easy.Cleanup()

    callBackDataFunc := func(buf []byte, userdata interface{}) bool {
        elapsed := time.Since(start)
        log := fmt.Sprintf("\n=>URL:%v  Time spent:%v, size=>%v", url, elapsed, len(buf))
        logger.Output(1, log)
        return true
    }

    optResolve := fmt.Sprintf("somehost:443:%v", host)

    easy.Setopt(curl.OPT_WRITEFUNCTION, callBackDataFunc)
    easy.Setopt(curl.OPT_RESOLVE, []string{optResolve})
    easy.Setopt(curl.OPT_URL, url)
    if err := easy.Perform(); err != nil {
        fmt.Printf("ERROR: %v\n", err)
    }
}

Which I run inside goroutine in for loop, where url is valid url I want to fetch and host is existing ipv4 address I want curllib resolve instead of host inside url. Like this:

    for {
        <-time.After(time.Duration(period) * time.Second)
        for _, square := range squaresSlice {
            //dunno how to do it better than []int{1,2,3}
            for range []int{1,2,3} {
                url := generateUrl(square)
                go func() {
                    for _, host := range hosts {
                                                // !Here!
                        curlIt(url, host)
                    }
                }()
            }
        }
    }

I want you to pay attention to such string as easy.Setopt(curl.OPT_RESOLVE, []string{optResolve})

When I added this option to curl - I get strange result: for all, exept of one urls in the ONLY FIRST iteration of for _, square := range squaresSlice I get:

ERROR: curl: Out of memory
ERROR: curl: Out of memory
logger: main.go:96: 
=>URL:https://url_address  Time spent:19.594906ms, size=>16045

In all follow iterations of for _, square := range squaresSlice I get no Out of memory error at all, never. Only first start, only all, except one of urls fails with out of memory.

And when I remove this option, like this:

        // optResolve := fmt.Sprintf("somehost:443:%v", host)
        ....
    // easy.Setopt(curl.OPT_RESOLVE, []string{optResolve})

I have no such problem at all.

Do you have any suggestions what could be cause?

ivan-kiselev commented 7 years ago

Tcpdump shows me that when it fails with curl: Out of memory - it actually doesnt apply OPT_RESOLVE, but tries to send request to real host instead of what I defer in optResolve