Open gaoyong06 opened 1 year ago
我用这个工具,能正常得到一些代理ip,类似:
http://127.0.0.1:8080/get?type=HTTP&count=2&anonymity=all
[ { "Ip": "117.160.250.133", "Port": "9999", "Country": "中国", "Province": "河南", "City": "安阳", "Isp": "移动", "Type": "HTTP", "Anonymity": "透明", "Time": "2023-03-21 20:25:57", "Speed": "85.7072ms", "SuccessNum": 5, "RequestNum": 5, "Source": "scrape代理" }, { "Ip": "117.159.15.99", "Port": "9091", "Country": "中国", "Province": "河南", "City": "郑州", "Isp": "移动", "Type": "HTTP", "Anonymity": "普匿", "Time": "2023-03-21 20:25:57", "Speed": "130.7455ms", "SuccessNum": 5, "RequestNum": 5, "Source": "scrape代理" } ]
我将得到的这些ip 和 port 放到我的程序中,我使用了colly 这个工具, 代码类似:
... // 获取可被抓取的域名 domains := strings.Split(spiders.Domains, ",") // 使用go_proxy_pool 得到的代理ip和port proxyUrls := []string{ "http://117.160.250.133:9999", "http://117.159.15.99:9091", "http://116.113.68.130:9091", "http://221.6.215.202:9091", "http://117.160.250.138:8080", "http://117.160.250.134:8081", "http://117.160.250.131:8899", "http://27.15.232.197:9091", "http://222.139.221.185:9091", "http://42.228.61.245:9091", } rp, err := proxy.RoundRobinProxySwitcher(proxyUrls...) if err != nil { log.Fatal(err) } // Collector collector := colly.NewCollector( colly.AllowedDomains(domains...), colly.AllowURLRevisit(), ) collector.SetRequestTimeout(120 * time.Second) collector.SetProxyFunc(rp) ...
但是启动后,总是报下面Access Denied的错误,类似:
2023/03/22 15:10:36 (D:/work/aaa/main.go:133 main.Run.func1) [info] OnRequest: Req.ID: 2, urlType:listPage, Req.URL: https://www.baidu.com/list/, q.IsEmpty: true, q.Size: 0, q.Threads: 20 2023/03/22 15:10:46 (D:/work/aaa/main.go:300 main.Run.func4) [info] OnError: [2]listPage, https://www.baidu.com/list/, Get "https://www.baidu.com/list/": Access Denied
请问是哪里,搞错了?
我不清楚你的代码是否有问题,但可能是免费代理的质量问题,如果报错可以更换ip尝试。(可以通过不断使用verify去验证代理,来筛选出高可用的代理ip)
我用这个工具,能正常得到一些代理ip,类似:
http://127.0.0.1:8080/get?type=HTTP&count=2&anonymity=all
我将得到的这些ip 和 port 放到我的程序中,我使用了colly 这个工具, 代码类似:
但是启动后,总是报下面Access Denied的错误,类似:
请问是哪里,搞错了?