search

alecxe / scrapy-fake-useragent

Random User-Agent middleware based on fake-useragent
MIT License
686 stars 98 forks source link

Random browser user agents or random user agents? #10

Closed japborst closed 7 years ago

japborst commented 7 years ago

Hi,

I got a question. The user agents from useragentstring.com also includes crawlers and bots. Does that mean that those are included in the random generation of user agents or are only browsers included?

Thanks!

alecxe commented 7 years ago

@japborst good question. Since we use fake-useragent package and it uses the real-world stats (available here), the random user agent you get would only contain the browsers presented on this page which are Chrome, IE, Firefox, Safari or Opera. The user agent strings can though be not only 'desktop' but 'mobile' as well. Thanks.

japborst commented 7 years ago

Thanks for the update @alecxe. With some auto throttling it works perfectly btw! :-)