TeamHG-Memex / scrapy-rotating-proxies

use multiple proxies with Scrapy
MIT License
733 stars 156 forks source link

What about getting proxy list from url? #31

Open aidiss opened 4 years ago

aidiss commented 4 years ago

It could work as something like this

ROTATING_PROXY_LIST_URL = 'http://..../get_proxies.json'

Or, what would be the correct way of implementing this?

heilwood commented 4 years ago
  def getProxyList():
          resp = requests.get('https:/name/api/proxylist.txt')
          return resp.text.split()

  ROTATING_PROXY_LIST = getProxyList()

Same with json, you need to parse response of request as json

hamcher commented 1 year ago

How to update proxy list every month, without stop the scrapper app ??