search

Anonyfox / elixir-scrape

Scrape any website, article or RSS/Atom Feed with ease!
https://github.com/Anonyfox/elixir-scrape
GNU Lesser General Public License v3.0
327 stars 43 forks source link

Spoof/Change User-Agent #28

Open jonzlin95 opened 7 years ago

jonzlin95 commented 7 years ago

Some sites don't play well without User-Agents (e.g. ESPN).

HTTPoison by default sends hackney which is blocked, so it could be useful to set it to

["User-Agent": "Elixir"] or some similar string to bypass these problems.

Anonyfox commented 7 years ago

Why not faking a typical iPhone signature? I think some sites whitelist on UA strings

jonzlin95 commented 7 years ago

That could work too. Maybe it would be useful to add a way to pass in a User Agent to the Scrape.X function calls. I notice that you already have fetch(url, html_headers \ []), and that could just be passed onto the next set of functions.

I can make a simple PR for this.