search

hyperlane-xyz / hyperlane-registry

A collection of configs, artifacts, and schemas for Hyperlane
https://www.hyperlane.xyz
MIT License
43 stars 45 forks source link

Mitigate GithubRegistry rate limit issues #163

Closed jmrossy closed 3 days ago

jmrossy commented 2 weeks ago

When multiple users connect from the same IP, or possibly a single user with aggressive usage, the GithubRegistry requests may start to fail due to API rate limits.

Rate limits on raw.githubusercontent.com requests are quite high (> 1000s per hour) but the unauthenticated github API requests (which we do also use currently) are more strict.

For the apps (explorer, warp ui), there's a range of solutions. But the CLI solutions are limited. The current plan is to stand up a proxy service to replace authenticated github API requests with authenticated ones. This has costs and risks but would help solve the problem globally for all users (up to a certain scale at least).

Relevant threads:

1: https://discord.com/channels/935678348330434570/1261371685337042965

2: https://discord.com/channels/935678348330434570/1270458661822664775

jmrossy commented 2 weeks ago

@ltyu There are a some open questions here that we should narrow down before we make any moves:

  1. Do we use an off-the-shelf proxy product or write a little one ourselves in TS
  2. Where should we host the proxy (Cloudflare fn, Vercel fn, GCP with our K8S cluster)?
  3. Should we cache requests and if so, how (Redis, in-memory)?
  4. Is one API key sufficient or should we have a few that round-robin?
  5. How do we monitor the proxy for failures?
  6. How do we protect against abuse (DDoS)
jmrossy commented 2 weeks ago

@ltyu I guess my recommendations for these would be as follows but I don't have strong opinions about any of them:

  1. Write our own
  2. Cloudflare
  3. Redis cache
  4. Start with one
  5. Sentry error monitoring
  6. Protection via Cloudflare