ChimeraCoder / anaconda

A Go client library for the Twitter 1.1 API
MIT License
1.14k stars 247 forks source link

Increasing Go-routines & Memory #160

Closed ameykpatil closed 7 years ago

ameykpatil commented 7 years ago

This is my func -

func GetFollowersIds(userID, cursorStr, accessToken, accessTokenSecret string) (anaconda.Cursor, error) {
    api := anaconda.NewTwitterApi(accessToken, accessTokenSecret)
    api.ReturnRateLimitError(true)
    defer api.Close()
    if cursorStr == "" {
        cursorStr = "-1"
    }
    v := url.Values{}
    v.Set("user_id", userID)
    v.Set("cursor", cursorStr)
    result, err := api.GetFollowersIds(v)
    if err != nil {
        return result, err
    }
    return result, nil
}

This method gets called for many users in my long running application/service. I have observed that the no. of go-routines & memory increases continuously, I have confirmed this through multiple tools.

screen shot 2016-10-29 at 10 29 06 pm

And all these boils down to the above method. I introduced pprof & took output for /debug/pprof/goroutine In the output, it shows there are more than 300 instances of go-routines regarding throttledQuery Full stack trace is as follows -

goroutine 62852 [runnable]:
net/http.(*http2ClientConn).RoundTrip(0xc421568340, 0xc424b3c690, 0x0, 0x0, 0x0)
    /usr/local/go/src/net/http/h2_bundle.go:5570 +0xa02
net/http.(*http2Transport).RoundTripOpt(0xc4300ee0f0, 0xc422b3c690, 0x800000000f24d00, 0x130, 0xffffffffffffffff, 0x40bb3c)
    /usr/local/go/src/net/http/h2_bundle.go:5160 +0x18e
net/http.(*http2Transport).RoundTrip(0xc4100ee0f0, 0xc423b3c690, 0xffffffff, 0x0, 0x54c479)
    /usr/local/go/src/net/http/h2_bundle.go:5130 +0x40
net/http.http2noDialH2RoundTripper.RoundTrip(0xc4200ee0f0, 0xc423b3c690, 0xc4264c5020, 0x5, 0xc420118088)
    /usr/local/go/src/net/http/h2_bundle.go:343 +0x39
net/http.(*Transport).RoundTrip(0xc4200ba000, 0xc423b3c690, 0xc4200ba000, 0x0, 0xc400000000)
    /usr/local/go/src/net/http/transport.go:338 +0x87c
net/http.send(0xc423b3c690, 0xed4fe0, 0xc4200ba000, 0x0, 0x0, 0x0, 0x8, 0xc42c275bc0, 0x410468)
    /usr/local/go/src/net/http/client.go:256 +0x15f
net/http.(*Client).send(0xf07d60, 0xc423b3c690, 0x0, 0x0, 0x0, 0xc42c275bc0, 0x0, 0x1)
    /usr/local/go/src/net/http/client.go:146 +0x102
net/http.(*Client).doFollowingRedirects(0xf07d60, 0xc423b3c690, 0xbaf940, 0x3, 0x1, 0x0)
    /usr/local/go/src/net/http/client.go:528 +0x5e5
net/http.(*Client).Do(0xf07d60, 0xc423b3c690, 0x4e8, 0xb48133, 0x3)
    /usr/local/go/src/net/http/client.go:184 +0x1ea
github.com/<my-project>/vendor/github.com/garyburd/go-oauth/oauth.(*Client).Get(0xeff980, 0xf07d60, 0xc424a32500, 0xc4264c5020, 0x2d, 0xc4264c4ff0, 0xc427e0a360, 0xc4513d3770, 0xc46d6ccf70)
    /go/src/github.com/<my-project>/vendor/github.com/garyburd/go-oauth/oauth/oauth.go:466 +0x251
github.com/<my-project>/vendor/github.com/ChimeraCoder/anaconda.TwitterApi.apiGet(0xc424a32500, 0xc427e0a360, 0x0, 0x1, 0xf07d60, 0xee1560, 0xf24d78, 0xb5b24e, 0x1b, 0xc4264c5020, ...)
    /go/src/github.com/<my-project>/vendor/github.com/ChimeraCoder/anaconda/twitter.go:192 +0x87
github.com/<my-project>/vendor/github.com/ChimeraCoder/anaconda.TwitterApi.execQuery(0xc424a32500, 0xc427e0a360, 0x0, 0x1, 0xf07d60, 0xee1560, 0xf24d78, 0xb5b24e, 0x1b, 0xc4264c5020, ...)
    /go/src/github.com/<my-project>/vendor/github.com/ChimeraCoder/anaconda/twitter.go:244 +0x1b6
github.com/<my-project>/vendor/github.com/ChimeraCoder/anaconda.(*TwitterApi).throttledQuery(0xc4514bbb30)
    /go/src/github.com/<my-project>/vendor/github.com/ChimeraCoder/anaconda/twitter.go:268 +0x1f0
created by github.com/<my-project>/vendor/github.com/ChimeraCoder/anaconda.NewTwitterApi
    /go/src/github.com/<my-project>/vendor/github.com/ChimeraCoder/anaconda/twitter.go:120 +0x17b

I want to know if I am using anaconda correctly? Do I have to handle or set throttledQuery value? How does throttledQuery works in anaconda? Insights?

carbocation commented 7 years ago

I have also noticed continuously increasing memory use via throttledQuery in a long-running instance. I see that you closed this issue, so I'm curious how you solved it.