I am trying to crawl a big website (arezzo.com.br), however, after ~1700 URLs crawled, it simply stopped. No errors printed, and also the finished callback wasn't called.
Can someone give me a north regarding what can happen in this case?
I think you can reproduce the issue by trying to crawl www.arezzo.com.br with my configs.
I am trying to crawl a big website (arezzo.com.br), however, after ~1700 URLs crawled, it simply stopped. No errors printed, and also the
finishedcallback wasn't called.Can someone give me a north regarding what can happen in this case?
I think you can reproduce the issue by trying to crawl www.arezzo.com.br with my configs.