The current check for whether the CDX is up to date uses the BBC robots.txt file as a check, but (I think) Wayback is configured to omit revisits, so this only reports the date correctly if the robots.txt is changed. It would make more sense to use https://www.bbc.co.uk/news as the sensor URL.
The current check for whether the CDX is up to date uses the BBC
robots.txt
file as a check, but (I think) Wayback is configured to omit revisits, so this only reports the date correctly if therobots.txt
is changed. It would make more sense to usehttps://www.bbc.co.uk/news
as the sensor URL.