search

alecxe / scrapy-fake-useragent

Random User-Agent middleware based on fake-useragent
MIT License
687 stars 98 forks source link

User Agent not updating #26

Closed yilu1015 closed 4 years ago

yilu1015 commented 4 years ago

I had a successful pip installation. I put the correct configuration code in the SETTINGS.py for the global project. I thought it was as easy as that. But when I asked my scrapy shell, it told me that I am still, yours sincerely, 'Scrapy/2.1.0 (+https://scrapy.org)'}.

Is there anything I am missing here? Relevant sections of my debugging code are shown as below.

'DOWNLOADER_MIDDLEWARES': {'scrapy.downloadermiddlewares.retry.RetryMiddleware': None,
                            'scrapy.downloadermiddlewares.useragent.UserAgentMiddleware': None,
                            'scrapy_fake_useragent.middleware.RandomUserAgentMiddleware': 400,
                            'scrapy_fake_useragent.middleware.RetryUserAgentMiddleware': 401},
 'DOWNLOADER_MIDDLEWARES_BASE': {'scrapy.downloadermiddlewares.ajaxcrawl.AjaxCrawlMiddleware': 560,
                                 'scrapy.downloadermiddlewares.cookies.CookiesMiddleware': 700,
                                 'scrapy.downloadermiddlewares.defaultheaders.DefaultHeadersMiddleware': 400,
                                 'scrapy.downloadermiddlewares.downloadtimeout.DownloadTimeoutMiddleware': 350,
                                 'scrapy.downloadermiddlewares.httpauth.HttpAuthMiddleware': 300,
                                 'scrapy.downloadermiddlewares.httpcache.HttpCacheMiddleware': 900,
                                 'scrapy.downloadermiddlewares.httpcompression.HttpCompressionMiddleware': 590,
                                 'scrapy.downloadermiddlewares.httpproxy.HttpProxyMiddleware': 750,
                                 'scrapy.downloadermiddlewares.redirect.MetaRefreshMiddleware': 580,
                                 'scrapy.downloadermiddlewares.redirect.RedirectMiddleware': 600,
                                 'scrapy.downloadermiddlewares.retry.RetryMiddleware': 550,
                                 'scrapy.downloadermiddlewares.robotstxt.RobotsTxtMiddleware': 100,
                                 'scrapy.downloadermiddlewares.stats.DownloaderStats': 850,
                                 'scrapy.downloadermiddlewares.useragent.UserAgentMiddleware': 500},
yilu1015 commented 4 years ago

I think I found the issue: when I removed RANDOM_UA_TYPE = 'mobile' from settings.py, it worked. BUT, what is wrong with my parameter?

alecxe commented 4 years ago

With 1.3.0 and above there is now more control over the user agent providers and fallback options. Please check it out.