Mr0grog
commented
6 years ago Useful addendum from @ericnost here — some pages are also persistent 403, e.g. https://www.epa.gov/ghgreporting/subpart-w-basic-information, and that would be nice to flag, too.
Not sure on the best way to handle these needs, but some thoughts:
- A page-level status that follows the same rules (200 unless we are getting some 4xx or 5xx repeatedly for 14 days)
- A page-level status that lists the most recent status, plus the
removedboolean flag - Multiple flags, e.g.
removedandlimited_accessand others as we come up with them - Something else?
stale[bot]
One really useful feature for analysts would be to flag pages that have been removed. There’s no objective or even canonical way to monitor this (after all, we’ve seen plenty of servers that respond erroneously with 404s when they are under heavy load and unable to return a page that exists, as well as intermittent 404s when a page disappears for only a few days). So for now we are thinking:
A removed page is one that has responded with a 404 continuously for 14 days.
How might we measure this?
A script that repeatedly checks Wayback on a regular schedule to get the status of all the snapshots for a page over the past 14 days. Similar to the generalized ETL script we’re currently working on. We’d run it on a regular schedule, but probably not daily since it’s a big query. This would also require adding the ability to POST updates to a page in -db.
Seeing no new, different versions added to -db over a 14 day period when the most recent version was a 404. -db could easily manage its own async job to check and update this on a regular schedule.
Neither of these is perfect — (1) would be reliable, but only for URLs that we have Wayback checking for us on a regular basis. For other URLs, all bets are off. (2) Would be much easier to implement and maintain, but doesn’t give us a way to distinguish between pages that are persistently 404s and pages that our sources (like Versionista, Wayback, and Walk) have stopped monitoring and whose last snapshot was a 404 (because a page could always come back, but if we’ve stopped monitoring, we won’t know that).
I’d tend towards (2). Even with its imperfections, I think it’s probably good enough, is way lower impact to measure, and way lower effort to implement. In the future we could even make it more reliable by making
HEADrequests for the pages we think are removed to make sure they are still 404s. (That said, depending how Walk works out, it might make doing this easier.)@danielballan @jsnshrmn any thoughts here?