search

alecxe / scrapy-fake-useragent

Random User-Agent middleware based on fake-useragent
MIT License
686 stars 98 forks source link

Didn't work when used to scrape linkedin #2

Closed vionemc closed 8 years ago

vionemc commented 8 years ago

I set user-agent normally in scrapy, and I succeed to scrape linkedin.

I thought this library would help me bypass 999 response from linkedin even better, but I got 999 response like I used to.

vionemc commented 8 years ago

It seemed like it didn't change the user-agent at all

vionemc commented 8 years ago

I checked the user-agent used when running it. Instead of using a good and useful user-agent, it uses ['Scrapy/1.0.0rc1 (+http://scrapy.org)']

alecxe commented 8 years ago

I think you might've just not followed the setup and configuration instructions. Most likely, the middleware was not enabled or the scrapy built-in middleware was not disabled.

Make sure you have:

DOWNLOADER_MIDDLEWARES = {
    'scrapy.contrib.downloadermiddleware.useragent.UserAgentMiddleware': None,
    'scrapy_fake_useragent.middleware.RandomUserAgentMiddleware': 400,
}

in your settings.