-
User would like flexible tiers like the ones on Webpack.
This is the collective URL:
https://opencollective.com/colly
-
I wrote [check](https://github.com/kamilsk/check) tool and found some problems in the **awesome list**:
```
├─── [301] https://github.com/ahmetalpbalkan/govvv -> https://github.com/ahmetb/govv…
-
Since I need to visit an unsafe website over https, the Post / Get method will return an error: x509: certificate has expired or is not yet valid.
I know I can use `InsecureSkipVerify: true` when sta…
-
package main
import (
"github.com/gocolly/colly"
"github.com/gocolly/colly/debug"
"time"
)
func main() {
urls := []string{"https://weibo.cn/repost/FBrYpiw8h?uid=1153760245&rl=…
-
From about a week ago I've started getting 403 errors from making a GET to a users json endpoint `https://www.instagram.com/${username}/?__a=1`, one which never gave errors previously (and doesn't req…
-
Hi,
Thank you for this wonderful library as it is one of building blocks of the go-colly crawler, which I use.
I have been running into a corner case of late. Consider this piece of html.
`T…
-
I'm doing some timing during a crawl recording the start time in Ctx in OnRequest and calculating a duration in OnResponse. It all works very well until I try to throttle the crawler with a Limit wit…
-
package main
import (
"fmt"
"github.com/gocolly/colly"
)
func main() {
c := colly.NewCollector()
// Find and visit all links
c.OnHTML("a[href]", func(e *colly.HTMLElement) {
li…
-
When there is a page redirect, colly automatically follows the redirect. In that case, I get a Request object in the OnHTML callback. It seems that colly provides the original Request and not the …
-
Hi,
It seems that `sync.WaitGroup` is not used correctly in `func (c *Collector) scrape(...) error` method.
```
// colly.go:307
func (c *Collector) scrape(...) error {
c.wg.Add(1)
de…