When we successfully request a page, put its html in a cache with a one hour timeout or something. Also send appropriate cache headers on requests. Speeds up the scrapers and is nicer to the facilities. Also means that it's nbd if a scraper requests the same page 100 times because it's stupid. Also makes the logic much cleaner if we want to do finer grained, more frequent scrapes in response to a game change since we can run the whole facility without actually anything but the page that changed.
When we successfully request a page, put its html in a cache with a one hour timeout or something. Also send appropriate cache headers on requests. Speeds up the scrapers and is nicer to the facilities. Also means that it's nbd if a scraper requests the same page 100 times because it's stupid. Also makes the logic much cleaner if we want to do finer grained, more frequent scrapes in response to a game change since we can run the whole facility without actually anything but the page that changed.