During the crawl of unreadable directories, which can exists in the days
before a release, the crawler tries to count the number of unreadable
directories in the stats dict. The key 'unreadable' does not exist which
means that the crawlers just stops crawling a host once the first
unreadable directory has been found. This means that the crawler will
error out on most hosts just before a release.
pypingou
commented
8 years ago
During the crawl of unreadable directories, which can exists in the days before a release, the crawler tries to count the number of unreadable directories in the stats dict. The key 'unreadable' does not exist which means that the crawlers just stops crawling a host once the first unreadable directory has been found. This means that the crawler will error out on most hosts just before a release.