search

guacsec / guac

GUAC aggregates software security metadata into a high fidelity graph database.
https://guac.sh
Apache License 2.0
1.29k stars 176 forks source link

[feature] Rate limit outgoing http requests #2011

Closed jeffmendoza closed 2 months ago

jeffmendoza commented 4 months ago

Is your feature request related to a problem? Please describe. GUAC pulls data from deps.dev, osv.dev, and soon ClearlyDefined. With a large graph, an instance might pull a lot of data. It would be best to rate limit outgoing requests.

Describe the solution you'd like This blog covers a simple design: https://medium.com/mflow/rate-limiting-in-golang-http-client-a22fba15861a using https://pkg.go.dev/golang.org/x/time/rate

We should make an http.RoundTripper to hande this and stack with other middleware for all GUAC outgoing clients. Similar to the existing user agent middleware: https://github.com/guacsec/guac/blob/82e3f8030ecfe6276c965d9a8ab225cee432d944/pkg/version/version.go

pxp928 commented 4 months ago

Note: ClearlyDefined has a 2K request per minute limit: https://docs.clearlydefined.io/docs/get-involved/using-data