Fallback causing it always use Fallback user agent even though there is no error

alecxe / scrapy-fake-useragent

Random User-Agent middleware based on fake-useragent

MIT License

686 stars 98 forks source link

Fallback causing it always use Fallback user agent even though there is no error #22

Closed vionemc closed 5 years ago

vionemc commented 5 years ago

I tried to check what user agents are used by the requests since it seems it still gets blocked, even after using Crawlera. It turned out, it always uses Fallback even though there is no problem in downloading the user agents database from the internet. It's really not the intention of the Fallback. We'd need to address it seriously.

vionemc commented 5 years ago

Sorry, false alarm. It was caused by me turning on RANDOM_UA_PER_PROXY.