search

TeamHG-Memex / scrapy-rotating-proxies

use multiple proxies with Scrapy
MIT License
738 stars 158 forks source link

Possibility to use proxies from DB? #39

Open fparareda opened 4 years ago

fparareda commented 4 years ago

Hello, I would like to ask you if you have a way to connect your marvelous middleware with a DB (sql or no-sql). Do you think that it is possible and useful?

Thank you in advance

StasDeep commented 4 years ago

You can get proxies from wherever you want, like this:

def get_proxy_list():
    # Request your DB here
    return proxy_list

ROTATING_PROXY_LIST = get_proxy_list()