Restrict access to S3

[kalabukdima]

Our Cloudflare R2 cluster is accesses under the same API key from all workers, which makes it vulnerable to DoS attacks.

Goal

Spamming requests from one worker (or hand-crafted) doesn't affect all the other workers.

Solution

We can use Cloudflare's firewall (WAF) to restrict requests (both check signatures and rate limit). However, to use it with the R2 cluster, we have to bind it to a domain, which leaves us only with the HTTP API (get object by key, no listing operations). This implies that we have to send all the required file names to the workers together with the assignment. We can also include signed headers in the assignment, that would authenticate the worker to the firewall.

[kalabukdima]

Summary:

• The assignments contain the lists of files with the direct download URLs now.
• The scheduler signs worker peer IDs and sends the signature together with the assignment to the active (registered on chain) workers.
• Cloudflare WAF checks the signatures and blocks invalid requests. It also ratelimits requests by the IP address.

The current Cloudflare subscription (Pro) doesn't allow ratelimiting by any other values than IP. With the Business plan we can ratelimit by the worker id. However, nobody will probably risk DOSing us when it requires staking SQD and exposing your peer id, so we agreed that it should be enough for now. We can switch to the Business and change the rule in 2 clicks at any moment if we need it.

Here is the WAF blocking rule:

(
  not is_timed_hmac_valid_v0(
    "<secret>",
    concat(
      http.request.headers["worker-id"][0],
      http.request.headers["worker-signature"][0]),
    86400, http.request.timestamp.sec, 0
  )
)

The secret is shared with WAF and both testnet and mainnet schedulers.