Some web pages are crawler traps that generate an abnormal number of mementos for the same URL. For example, on some of our crawls the google script for ads had over one million mementos because so many websites use this script.(https://pagead2.googlesyndication.com/pagead/js/adsbygoogle.js)
We created a workaround with a filtering script but other users can have problems while crawling traps.
Some web pages are crawler traps that generate an abnormal number of mementos for the same URL. For example, on some of our crawls the google script for ads had over one million mementos because so many websites use this script.(https://pagead2.googlesyndication.com/pagead/js/adsbygoogle.js)
We created a workaround with a filtering script but other users can have problems while crawling traps.