edgi-govdata-archiving / web-monitoring-db

An HTTP API for tracking and annotating changes to a set of web pages.
https://api.monitoring.envirodatagov.org/
GNU General Public License v3.0
17 stars 26 forks source link

Output CloudFront-friendly headers for diffs #1098

Closed Mr0grog closed 1 year ago

Mr0grog commented 1 year ago

This is part of web-monitoring#168 — the goal here is to make our diff cache less important so we can shrink it or even remove it (in favor of just having CloudFront do the job).

CloudFront needs a Date header to go with the Last-Modified and ETag headers from the stale? method, and works better still if we give it an actual cache age. This adds the expires_in helper which sets all the headers. I've also taken the strategy of setting a relatively short cache time but a much longer revalidation window when a stale response can be used, to balance concerns about updating the diff algorithm and effective long-term caching.