Change to ensure that utf8 content can be crawled.

stewartmckee / cobweb

Web crawler with very flexible crawling options. Can either use standalone or can be used with resque to perform clustered crawls.

MIT License

226 stars 45 forks source link

Change to ensure that utf8 content can be crawled. #5

Closed rojotek closed 12 years ago

rojotek commented 12 years ago

...ncode content before it is given to resque, wrapping the user specified processing_queue in a special queue named "encoding_safe_process_job", which will decode the content before giving it back to the processing_queue to handle. Ensure that there is a encoding_safe_process_job worker started.