http.Client rate limited http.Transport

[dougnukem]

Overview

It'd be nice if this library supported a way for http.Client outgoing requests to utilizing limiter.Limit to rate limit based on configured API rate limits useful for things like:

• Github API rate limits
• Twitter API rate limits

  Features

• Filter and intercept http.Client requests and map to a limit.Limiter which could then have some policy callback to let the client
  • block and wait until reset time
  • abort request and send error back
  • maybe fake an HTTP rate-limit response (but avoid hitting actual servers)
• HTTP API's usually return with headers like X-RateLimit-* it'd be useful if there was a way to call limiter.Limiter.Set(key string, c *Context) to attempt to keep in sync with server state

I think you'd want to be able to configure a limit.Limiter per URL request path e.g.:

• /1.1/users/user_timeline.json

Similar to how server-side HTTP handlers are setup:

mux.HandleFunc("/1.1/statuses/user_timeline.json", func(r *http.Request) error {
    c, err := userTimelineLimiter.Get("user_timeline.json")
    if err != nil {
      return err
    }

    if c.Reached {
      // client policy:
      // 1. wait and retry
      // 2. error rate limit
      // 3. simulate http response error with rate limit
      err := handleRatelimitReached(r)
    }
})

Maybe even use http.NewServeMux to client-side proxy rate limit responses?

There's some example of it here:

• https://github.com/mmcloughlin/ghr/blob/0a008d3cb57649c35a574d26eaace5f9b6f3a2d0/ratelimitedtransport.go

// RateLimitedTransport is used to conform to rate limits that are
// communicated through "X-RateLimit-" headers, like GitHub's API. It
// implements http.RoundTripper and can be used for configuring a http.Client.
type RateLimitedTransport struct {
    Base http.RoundTripper
}

func (t *RateLimitedTransport) RoundTrip(req *http.Request) (*http.Response, error) {
    res, err := t.base().RoundTrip(req)
    if err != nil {
        return res, err
    }

    // Fetch headers
    remStr := res.Header.Get("X-RateLimit-Remaining")
    if remStr == "" {
        return res, err
    }
    resetStr := res.Header.Get("X-RateLimit-Reset")
    if resetStr == "" {
        return res, err
    }

    rem, err := strconv.Atoi(remStr)
    if err != nil {
        return res, err
    }
    epoch, err := strconv.ParseInt(resetStr, 10, 64)
    if err != nil {
        return res, err
    }
    reset := time.Unix(epoch, 0)

    // Determine sleep time
    untilReset := reset.Sub(time.Now())
    delay := time.Duration(float64(untilReset) / (float64(rem) + 1))
    time.Sleep(delay)

    return res, err
}

[dougnukem]

Looking at httpcontrol

Seems like there's a good example of intercepting an HTTP request and redirecting the response or also using the real response (e.g. if Rate limit is exceeded).

• https://github.com/facebookgo/httpcontrol/blob/master/httpcache/httpcache.go

// A cache enabled RoundTrip.
func (t *Transport) RoundTrip(req *http.Request) (res *http.Response, err error) {
    key := t.Config.Key(req)
    var entry cacheEntry

    // from cache
    if key != "" {
        raw, err := t.ByteCache.Get(key)
        if err != nil {
            return nil, err
        }

        if raw != nil {
            if err = json.Unmarshal(raw, &entry); err != nil {
                return nil, err
            }

            // setup fake http.Response
            res = entry.Response
            res.Body = ioutil.NopCloser(bytes.NewReader(entry.Body))
            res.Request = req
            return res, nil
        }
    }

    // real request
    res, err = t.Transport.RoundTrip(req)
    if err != nil {
        return nil, err
    }

    // no caching required
    if key == "" {
        return res, nil
    }

    // fully buffer response for caching purposes
    body, err := ioutil.ReadAll(res.Body)
    res.Body.Close()
    if err != nil {
        return nil, err
    }

    // remove properties we want to skip in serialization
    res.Body = nil
    res.Request = nil

    // serialize the cache entry
    entry.Response = res
    entry.Body = body
    raw, err := json.Marshal(&entry)
    if err != nil {
        return nil, err
    }

    // put back non serialized properties
    res.Body = ioutil.NopCloser(bytes.NewReader(body))
    res.Request = req

    // determine timeout & put it in cache
    timeout := t.Config.MaxAge(res)
    if timeout != 0 {
        if err = t.ByteCache.Store(key, raw, timeout); err != nil {
            return nil, err
        }
    }

    // reset body in case the config.Timeout logic consumed it
    res.Body = ioutil.NopCloser(bytes.NewReader(body))
    return res, nil
}