pingc0y / go_proxy_pool

无环境依赖开箱即用的代理IP池
749 stars 111 forks source link

请教一个使用的问题 #9

Open gaoyong06 opened 1 year ago

gaoyong06 commented 1 year ago

我用这个工具,能正常得到一些代理ip,类似:

http://127.0.0.1:8080/get?type=HTTP&count=2&anonymity=all

[
    {
        "Ip": "117.160.250.133",
        "Port": "9999",
        "Country": "中国",
        "Province": "河南",
        "City": "安阳",
        "Isp": "移动",
        "Type": "HTTP",
        "Anonymity": "透明",
        "Time": "2023-03-21 20:25:57",
        "Speed": "85.7072ms",
        "SuccessNum": 5,
        "RequestNum": 5,
        "Source": "scrape代理"
    },
    {
        "Ip": "117.159.15.99",
        "Port": "9091",
        "Country": "中国",
        "Province": "河南",
        "City": "郑州",
        "Isp": "移动",
        "Type": "HTTP",
        "Anonymity": "普匿",
        "Time": "2023-03-21 20:25:57",
        "Speed": "130.7455ms",
        "SuccessNum": 5,
        "RequestNum": 5,
        "Source": "scrape代理"
    }
]

我将得到的这些ip 和 port 放到我的程序中,我使用了colly 这个工具, 代码类似:

        ...
    // 获取可被抓取的域名
    domains := strings.Split(spiders.Domains, ",")

    // 使用go_proxy_pool 得到的代理ip和port
    proxyUrls := []string{
        "http://117.160.250.133:9999",
        "http://117.159.15.99:9091",
        "http://116.113.68.130:9091",
        "http://221.6.215.202:9091",
        "http://117.160.250.138:8080",
        "http://117.160.250.134:8081",
        "http://117.160.250.131:8899",
        "http://27.15.232.197:9091",
        "http://222.139.221.185:9091",
        "http://42.228.61.245:9091",
    }

    rp, err := proxy.RoundRobinProxySwitcher(proxyUrls...)
    if err != nil {
        log.Fatal(err)
    }

    // Collector
    collector := colly.NewCollector(
        colly.AllowedDomains(domains...),
        colly.AllowURLRevisit(),
    )
    collector.SetRequestTimeout(120 * time.Second)
    collector.SetProxyFunc(rp)
        ...

但是启动后,总是报下面Access Denied的错误,类似:


2023/03/22 15:10:36 (D:/work/aaa/main.go:133 main.Run.func1) [info] OnRequest: Req.ID: 2, urlType:listPage, Req.URL: https://www.baidu.com/list/, q.IsEmpty: true, q.Size: 0, q.Threads: 20
2023/03/22 15:10:46 (D:/work/aaa/main.go:300 main.Run.func4) [info] OnError: [2]listPage, https://www.baidu.com/list/, Get "https://www.baidu.com/list/": Access Denied

请问是哪里,搞错了?

pingc0y commented 1 year ago

我不清楚你的代码是否有问题,但可能是免费代理的质量问题,如果报错可以更换ip尝试。(可以通过不断使用verify去验证代理,来筛选出高可用的代理ip)