search

fabienvauchelles / scrapoxy

Scrapoxy is a super proxy aggregator, allowing you to manage all proxies in one place 🎯, rather than spreading it across multiple scrapers 🕸️. It also smartly handles traffic routing 🔀 to minimize bans and increase success rates 🚀.
http://scrapoxy.io
MIT License
1.94k stars 232 forks source link

Support mobile User-Agents and the ability to switch between desktop/mobile #160

Open Stealthmate opened 5 years ago

Stealthmate commented 5 years ago

Hello,

Expected Behavior

Generally, when scraping one would want to use the mobile version of a website since it is smaller and and thus saves on both speed and network traffic. However, sometimes you have information that is only available on desktop (or mobile) and so you want to switch. Therefore it would be nice if Scrapoxy allowed the user to specify what type of UA to use (via a header or something).

Actual Behavior

Scrapoxy just gets a random UA from a list.

I checked the code and this seems fairly trivial to implement. Since I need this feature, I'm going to fork and implement it, so I can make a PR later as well :)

Thanks, Stealthmate

fabienvauchelles commented 11 months ago

Good idea for the next version