-
I found this wonderful snippet in `gocolly`, this file could make random headers rather simple: https://github.com/gocolly/colly/blob/master/extensions/random_user_agent.go
-
- Just like pyspider, how about add a web UI for Colly which can help user to control their spider.
-
I've run into a few sites that are compressing using Brotli over gzip.
https://chromestatus.com/feature/5420797577396224
Native support for Brotli would be a nice enhancement.
Also, if th…
-
when I scrapping data, page return http status 404 but result still have html response. I want get response. But in colly, if OnError occurred then onHTML do not occurre. How can I get response when e…
-
Scraproxy accepts requests as HTTP but the HTTPS URL must be in the Location header, source:
http://docs.scrapoxy.io/en/master/advanced/understand/index.html#can-scrapoxy-relay-https-requests
go-c…
-
After I clone a Collector, I'm not sure if I need to use the same storage and queue...
I referenced
http://go-colly.org/docs/examples/redis_backend/
and
http://go-colly.org/docs/examples/courser…
-
Visiting https://go-colly.org/ shows a different website (omnom) instead of colly's.
If it's not intended, when you'll find the time to fix it you could also enable automatic `https` redirection.
…
-
Hello,
Colly is a great product, thanks so much for sharing.
I'd like to ask that is it possible for me to use colly to take screenshot of scraped page? Or, I need to use another Selenium / headless…
-
(for a future release) using something like http://go-colly.org/ to be able to input a URL and have it scrape the images of certain dimensions into a web gallery? thinking about if a studio releases a…
-
我用这个工具,能正常得到一些代理ip,类似:
http://127.0.0.1:8080/get?type=HTTP&count=2&anonymity=all
```
[
{
"Ip": "117.160.250.133",
"Port": "9999",
"Country": "中国",
"Provin…