-
Hello,
Colly is a great product, thanks so much for sharing.
I'd like to ask that is it possible for me to use colly to take screenshot of scraped page? Or, I need to use another Selenium / headless…
-
The sample program provided will result in a panic
> panic: interface conversion: interface {} is map[string]interface {}, not *main.node
The code only results in a panic when using a queue that…
-
Will v3 have context cancellation as a standard part of the API?
pjebs updated
3 years ago
-
After I clone a Collector, I'm not sure if I need to use the same storage and queue...
I referenced
http://go-colly.org/docs/examples/redis_backend/
and
http://go-colly.org/docs/examples/courser…
-
At the moment the maximum number of redirects is hardcoded and is equal to 10. It would be great if there would be a possibility to change this value via configuration.
-
```go
type InMemoryStorage struct {
// TTL
visitedURLsTTL time.Duration
}
```
I think handling TTL for visited URLs internally is quite useful,
and we can make `redisstorage` to support TTL…
gsoec updated
5 years ago
-
when the high performance situation ,the colly chan will be blocked, cause the signal in channel …
-
I need to login and then make a search call.
```
import (
"fmt"
"github.com/gocolly/colly"
"log"
)
const baseUrl = "https://xxx.com/"
const url = "https://xxx.com/login"
const queryUrl…
-
just wanted to try this out, but got stuck. Did not have time to dig into code yet, so making an issue for now.
```
go install github.com/VonC/ghchangelog@latest
go: downloading github.com/VonC/g…
-
I did make Scraping for Amazon Product Titles but Amazon captcha catches my scraper. I tried 10 times with go run main.go(8 times catches me - 2 times I scrapped the product title)
I researched thi…