Add `Retry-After` and `Cache-Control` headers

[masih]

In cases where 404s can potentially be satisfied in a later time, e.g. max internal timeout was hit, or 5 minutes caching of 404s, add extra headers to aid http clients make better decisions on retries.

See:

• https://docs.aws.amazon.com/AmazonCloudFront/latest/DeveloperGuide/HTTPStatusCodes.html

[lidel]

• Cache-Control: max-age=5, stale-while-revalidate=300provides a hint for Cloudflont and end HTTP clients on how long they should cache responsefor a specific CID

  • Allows setting different max-age for 404s and 200s.
  • 200s: If you have IPNI provider records that are not DHT, and never change, you could return 200 with way longer max-age than DHT ones.
  • 404s: minimize caching of false-negatives, but avoid processing CIDs that fail all the time
  • Returning explicit Cache-Control with 404s allows IPNI to have more control over how long 404s should be cached by Cloudfront (see their docs).
  • It no longer has to be static for all CIDs across all failed attempts.
  • This allows us to start with very small cache window, and then, if the same CID is requested over and over again, and fails all the time, grow cache's max-age window geometrically (helps with UX problem described here):
    • first failure: Cache-Control: max-age=5, stale-while-revalidate=300
    • second failure: Cache-Control: max-age=30, stale-while-revalidate=300
    • third failure: Cache-Control: max-age=90, stale-while-revalidate=300
    • etc (capping at something sensible, like 5m)
  • stale-while-revalidate means check for updated result happens asynchronously, so re-checking 404s will not introduce latency to end user
  • Lassie could read Cache-Control and Age headers and make informed decision

• Retry-After is usually only used for 429 and 503 as explicit hint for automated HTTP clients to perform a cooldown and wait between retries. It could be added in addition to Cache-Control, but I feel both Cloudfront and Lassie care about the same max-age, so we probably can do without this header in 404s.

My suggestion would be to start small:

• Add Cache-Control: max-age=n, stale-while-revalidate=300 to 404s, and make n grow from 5 seconds to 300 (geometric or exponential)
• Ask Lassie (cc @hannahhoward) to check if Cache-Control header is present in IPNI response, and if so, cache IPNI result for max-age number of seconds, to avoid sending requests that would produce the same response from Cloudfront.

If feel if we have this, we don't need Retry-After.