search

fredwu / crawler

A high performance web crawler / scraper in Elixir.
945 stars 91 forks source link

Links with space are not url encoded and return 404 #12

Closed julienmarie closed 6 years ago

julienmarie commented 6 years ago

Example:

"https://www.city-discovery.com/continent/North America" => 404
"https://www.city-discovery.com/continent/North%20America" => 200
jpyamamoto commented 6 years ago

@julienmarie Problem solved. You can now call Crawler.crawl("https://www.city-discovery.com/continent/North America", encode_uri: true) to have your URL encoded. I think this issue can be closed.

fredwu commented 6 years ago

👏 Thanks @JPYamamoto

julienmarie commented 6 years ago

Awesome!