JustAnotherArchivist
commented
4 years ago - New York Times videos:
^https?://vp\.nyt\.com/and^https?://video1\.nytimes\.com/ - Videos hosted on JW Player:
^https?://cdn\.jwplayer\.com/videos/(Note, these are different from the content.jwplatform.com one above. cdn.jwplayer.com serves various things by customers, content.jwplatform.com seems to only have FastCo videos.)
An alternative solution would be to dedupe based on data type or size, but that would require a new download every time and might slow down some crawls massively. If we go down this road, we should write revisit records for those; wpull already has support for that, it would just have to be activated and the remote calls implemented through a custom URLTable.
While global deduplication for everything in ArchiveBot is not feasible, we should consider adding something for certain URLs that waste a lot of disk space, probably shouldn't be ignored entirely, but are regrabbed needlessly and repeatedly. Two examples come to mind:
^https?://mp3\.cbc\.ca/and^https?://podcast-a\.akamaihd\.net/mp3/(pending further investigation whether the latter also has non-CBC content)^https?://content\.jwplatform\.com/videos/Currently, these ignores are typically manually added when someone sees it. I know we've grabbed some of those URLs thousands of times, but others were never covered before. Because the contents on these hosts don't change with time, ignoring them if they've ever been grabbed before by some AB job should be fine. However, job starting URLs should not be checked against the dedupe list so that they can be saved again if needed – specifically, this means that URL table entries with
level = 0would always be retrieved.An implementation would probably keep the dedupe DB and the list of URL patterns to be checked against it on the control node. The latter is pushed to the pipelines (and updated if it changes), then the pipeline queries the DB on encountering a matching URL. TBD is whether the pipeline should be able to directly add entries to the DB or whether they should come from the CDXs in the AB collection. The latter is more trustworthy (and also covers the unfortunate case when archives are lost between retrieval and IA upload) but adds a delay which can still lead to repeated retrieval. Alternatively, pipelines could add a temporary entry which gets dropped after a few days if it isn't confirmed by the CDXs.