Closed sebastian-nagel closed 7 years ago
I would look at the seed URLs from #12 ; this looks like the 3-field CSV of [URL,Language,Region] was treated as a single URL.
Thanks, @john-hewitt! That explains how these URLs went into the crawl. A quick fix would be to just remove them.
However, they are currently re-fetched every 3-5 minutes which is definitely a bug. Better to analyze while it's hot. It's an unnecessary waste of resources, and what's worse, some of them are repeatedly written into the WARC files.
Ok, got it: it's a record with a trailing space
"_source" : {
"url" : "http://www.topix.com/world/burkina-faso/,ENG,Foreign ",
"status" : "DISCOVERED",
"metadata" : { },
"hostname" : "topix.com",
"nextFetchDate" : "2016-12-21T16:14:40.784Z"
}
The trailing is lost in the topology when FetcherBolt converts the URL string into a java.net.URL object. In consequence, the original record never gets updated.
Removed all troublesome URLs from Elasticsearch.
Certain URLs following a quite specific pattern are continuously re-refetched, here the counts from a couple of hours log files:
The status index of these URLs seems correct, e.g.:
or