-
I wrote [check](https://github.com/kamilsk/check) tool and found some problems in the **awesome list**:
```
├─── [301] https://github.com/ahmetalpbalkan/govvv -> https://github.com/ahmetb/govv…
-
package main
import (
"github.com/gocolly/colly"
"github.com/gocolly/colly/debug"
"time"
)
func main() {
urls := []string{"https://weibo.cn/repost/FBrYpiw8h?uid=1153760245&rl=…
-
Since I need to visit an unsafe website over https, the Post / Get method will return an error: x509: certificate has expired or is not yet valid.
I know I can use `InsecureSkipVerify: true` when sta…
-
package main
import (
"fmt"
"github.com/gocolly/colly"
)
func main() {
c := colly.NewCollector()
// Find and visit all links
c.OnHTML("a[href]", func(e *colly.HTMLElement) {
li…
-
Now that https://github.com/gocolly/colly/pull/82 has been merged we should probably update the documentation to match.
@asciimoo I'm happy to do it if you don't want to :)
## Pages that should …
-
`ID` is part of the list of [common initialisms](https://github.com/golang/lint/blob/f635bddafc7154957bd70209ee858a4b97e64a9b/lint.go#L740-L782) used by the golint tool and thus we get warnings like t…
-
When there is a page redirect, colly automatically follows the redirect. In that case, I get a Request object in the OnHTML callback. It seems that colly provides the original Request and not the …
-
Currently, you can put anything in the context. Which was allowed after the fix for [this](https://github.com/gocolly/colly/issues/47) issue.
But it will type assert any value to a string [neverthel…
-
I'm doing some timing during a crawl recording the start time in Ctx in OnRequest and calculating a duration in OnResponse. It all works very well until I try to throttle the crawler with a Limit wit…
-
Is it possible to create randomized delays, i.e., per-request delays selected from some range or based on some random factor? I couldn't think of a good way to do this, other than maybe cycling t…