search

gt-big-data / QDoc

Quick & Dirty Operating Crawler
4 stars 1 forks source link

Trim article URLs before crawling. #26

Closed supersam654 closed 8 years ago

supersam654 commented 8 years ago

Currently a few articles can't be parsed and print the following error:

Could not download the article:
            http://vegasmagazine.com/culture-post/20446/5-Ways-to-Incorporate-Gorgeous-Spring-Florals-Into-Your-Vegas-Wedding

No connection adapters were found for '
            http://vegasmagazine.com/culture-post/20446/5-Ways-to-Incorporate-Gorgeous-Spring-Florals-Into-Your-Vegas-Wedding
        '

I'm pretty sure those articles would parse if there wasn't a ton of whitespace around the link.

vpradeep3 commented 8 years ago

I am working on this.

vpradeep3 commented 8 years ago

could not reproduce

supersam654 commented 8 years ago

Fixed now.