search

TeamHG-Memex / scrapy-rotating-proxies

use multiple proxies with Scrapy
MIT License
733 stars 156 forks source link

Add dynamic signaling #38

Open n1kn4x opened 4 years ago

n1kn4x commented 4 years ago

Adding and removing proxies is now possible through signals "ADD_PROXY" and "REMOVE_PROXY".

Additionally when a proxy is marked as dead/good a signal "DEAD_PROXY"/"GOOD_PROXY" is sent.

This enables the middleware to adapt proxies and dynamically spawn/kill proxies if the user wants.

codecov-io commented 4 years ago

Codecov Report

Merging #38 into master will increase coverage by 1.34%. The diff coverage is 66.66%.

@@            Coverage Diff             @@
##           master      #38      +/-   ##
==========================================
+ Coverage   51.68%   53.03%   +1.34%     
==========================================
  Files           5        5              
  Lines         238      264      +26     
  Branches       39       43       +4     
==========================================
+ Hits          123      140      +17     
- Misses        109      115       +6     
- Partials        6        9       +3
Impacted Files Coverage Δ
rotating_proxies/middlewares.py 24.61% <0.00%> (-0.39%) :arrow_down:
rotating_proxies/expire.py 77.19% <72.00%> (-1.70%) :arrow_down:
StasDeep commented 4 years ago

Hi! Nice PR, thanks a lot! What do you think about using constants as signals instead of strings? The way it's done in Scrapy:

# scrapy/signals.py

spider_opened = object()
spider_idle = object()
spider_closed = object()