search

stewartmckee / cobweb

Web crawler with very flexible crawling options. Can either use standalone or can be used with resque to perform clustered crawls.
MIT License
226 stars 45 forks source link

cobweb http requests should not include the fragment. #3

Closed rojotek closed 12 years ago

rojotek commented 12 years ago

Ensured that http requests do not include the fragment (section from the #). Includes tests around cobweb to verify this.