Closed chasepd closed 3 years ago
Crawler still occasionally hit situations where it would be fed a non-http link, causing a crash. Added a check to verify that the url is an http link
Crawler still occasionally hit situations where it would be fed a non-http link, causing a crash. Added a check to verify that the url is an http link