ukwa / ukwa-monitor

Dashboard and monitoring system for the UK Web Archive
0 stars 5 forks source link

Fix problem with CDX-up-to-date check #26

Closed anjackson closed 3 years ago

anjackson commented 3 years ago

The current check for whether the CDX is up to date uses the BBC robots.txt file as a check, but (I think) Wayback is configured to omit revisits, so this only reports the date correctly if the robots.txt is changed. It would make more sense to use https://www.bbc.co.uk/news as the sensor URL.