alecxe / scrapy-fake-useragent

Random User-Agent middleware based on fake-useragent
MIT License
686 stars 98 forks source link

Custom folder for local Cache DB #12

Open bezkos opened 7 years ago

bezkos commented 7 years ago

I did a small tweak so we can have custom folder for local cache DB.

` class RandomUserAgentMiddleware(object): def init(self, crawler): super(RandomUserAgentMiddleware, self).init()

    loc = crawler.settings.get('LOCATION')
    if loc is None: 
        self.ua = UserAgent()           
    else:
        self.ua = UserAgent(path=crawler.settings.get('LOCATION'))        
    self.per_proxy = crawler.settings.get('RANDOM_UA_PER_PROXY', False)        
    self.proxy2ua = {}`

In settings file, u can have an option like: LOCATION = '/home/user/fake_useragent%s.json' % fake_useragent.VERSION

alecxe commented 7 years ago

@bezkos good idea, thanks!