search

seantanwh / crawler4j

Automatically exported from code.google.com/p/crawler4j
0 stars 0 forks source link

Fatal Transport Error when crawling robots.txt #298

Closed GoogleCodeExporter closed 8 years ago

GoogleCodeExporter commented 8 years ago 
Apparently the crawler tries to get the robots.txt of any site.

Some sites don't have this file thus a Fatal Transport Error is being thrown.

It shouldn't

Original issue reported on code.google.com by avrah...@gmail.com on 1 Sep 2014 at 8:22

GoogleCodeExporter commented 8 years ago

Original comment by avrah...@gmail.com on 1 Sep 2014 at 8:22

GoogleCodeExporter commented 8 years ago 
Fixed in revision: b77ca34d7d14

Original comment by avrah...@gmail.com on 1 Sep 2014 at 8:23