jhao104 / proxy_pool

Python ProxyPool for web spider
https://jhao104.github.io/proxy_pool/
MIT License
20.85k stars 5.05k forks source link

自定义fetcher方法验证通过但是不往redis存入 #759

Open ubadly opened 11 months ago

ubadly commented 11 months ago

2023-07-26 11:55:47,572 fetch.py[line:39] INFO ProxyFetch - ju_ip_proxy12: 221.231.77.45:50750 ok 2023-07-26 11:55:47,572 fetch.py[line:39] INFO ProxyFetch - ju_ip_proxy12: 27.150.167.210:42601 ok 2023-07-26 11:55:57,942 fetch.py[line:39] INFO ProxyFetch - ju_ip_proxy12: 122.230.52.72:50114 ok 2023-07-26 11:55:57,942 fetch.py[line:39] INFO ProxyFetch - ju_ip_proxy12: 114.231.46.102:37743 ok

我的是付费的,搞了几百个一个都没有往里面上传成功!

 @staticmethod
    def ju_ip_proxy12():
        while True:
            r = WebRequest().get("http://v2.api.juliangip.com/dynamic/getips?xxxxx", timeout=10)
            ips = r.text.split("\r\n")

            for ip in ips:
                print("代理ip", [ip])
                yield ip
            sleep(10)
jhao104 commented 11 months ago

要看proxyCheck的日志,有没有认证或者白名单,看认证是否通过 另外不要在fetch里面while True。 调度会自己定时去调