Rate Limitation not reported

[kcphysics]

Describe the bug Rate Limitation is not correctly being detected and reported

To Reproduce Run the snippet of code below multiple times until you hit your rate limit:

        API_KEY := os.Getenv("POLYGON_API_KEY")
    client := polygon.New(API_KEY)
    adjusted := true
    limit := 50000
    from := time.Now().AddDate(-2, 0, 0)
    to := time.Now()
    params := models.ListAggsParams{
        Ticker:     "AAPL",
        From:       models.Millis(from),
        To:         models.Millis(to),
        Limit:      &limit,
        Adjusted:   &adjusted,
        Timespan:   models.Day,
        Multiplier: 1,
    }
    iter := client.ListAggs(context.TODO(), &params, models.WithTrace(true))
    count := 0
    for iter.Next() {
        if iter.Err() != nil {
            log.Fatal(iter.Err())
        }
        count += 1
    }
    log.Printf("Found %d results", count)

Expected behavior When rate limitation is reached, iter.Err contains an error stating that the rate limitation has been encountered

Screenshots Output from the snipped above:

Additional context I've tried this several different ways, but there's not "good" way to detect the rate limitation, and we cannot inspect the response inside of the Polygon.io client.

[justinpolygon]

Thanks for reporting this @kcphysics. I'll check it out.

[justinpolygon]

Hey @kcphysics, there is a couple things happening here.

There was a bug in the tracing code where we were not reporting the trace when there was an error. Obviously, this information is most needed when there is an API error so you can debug things. Thank you for reporting this as it'll really help people going forward.

Here's what the trace will look like now (with status code):

Request URL: /v2/aggs/ticker/AAPL/range/1/day/1643037491376/1706109491376?adjusted=true&limit=50000
Request Headers: map[Accept-Encoding:[gzip] Authorization:[REDACTED] User-Agent:[Polygon.io GoClient/v1.16.0]]
Response Status Code: 429
Response Headers: map[Content-Length:[206] Content-Type:[application/json] Date:[Wed, 24 Jan 2024 15:18:11 GMT] Server:[nginx/1.19.2] Strict-Transport-Security:[max-age=15724800; includeSubDomains] X-Request-Id:[60a62c691e2f820a45ba7213e3f6b40d]]

There was a second issue though with your example in that the iter.Err() check was nested under for iter.Next() {. So, if there wasn't any results you'd never print the error message.

    for iter.Next() {
        if iter.Err() != nil {
            log.Fatal(iter.Err())
        }
        count += 1
    }

You can change the code to look like this you'll capture the error.

        for iter.Next() {
                count += 1
        }
        if iter.Err() != nil {
                log.Fatal(iter.Err())
        }

With the added trace fix you'll see something like this following (although you don't need the trace fix to still see the error):

Request URL: /v2/aggs/ticker/AAPL/range/1/day/1643037491376/1706109491376?adjusted=true&limit=50000
Request Headers: map[Accept-Encoding:[gzip] Authorization:[REDACTED] User-Agent:[Polygon.io GoClient/v1.16.0]]
Response Status Code: 429
Response Headers: map[Content-Length:[206] Content-Type:[application/json] Date:[Wed, 24 Jan 2024 15:18:11 GMT] Server:[nginx/1.19.2] Strict-Transport-Security:[max-age=15724800; includeSubDomains] X-Request-Id:[60a62c691e2f820a45ba7213e3f6b40d]]
2024/01/24 07:18:11 bad status with code '429': message 'You've exceeded the maximum requests per minute, please wait or upgrade your subscription to continue. https://polygon.io/pricing': request ID '60a62c691e2f820a45ba7213e3f6b40d': internal status: 'ERROR'
exit status 1

Again, thanks again for reporting this.

[kcphysics]

Thanks! I appreciate the clarity and action!