Currently, we hard-code some web archives we might hit so we don't bother re-crawling them. Can we do this a more intelligent way? e.g. using the Memento registry http://timetravel.mementoweb.org/guide/api/#registry to create a block list to integrate with the nevercrawl list in #36
Currently, we hard-code some web archives we might hit so we don't bother re-crawling them. Can we do this a more intelligent way? e.g. using the Memento registry http://timetravel.mementoweb.org/guide/api/#registry to create a block list to integrate with the
nevercrawllist in #36