-
I was wondering how to use the proxy functionality with Basic Auth? I've modified the callback function in insert the base64 encoded username and password with the `Proxy-Authorization` header. I …
-
For my given website after the ```colly.Visit``` function, the ```colly.OnError``` shows **Get "{website}": EOF**. When I check the `StatusCode` it shows `0` **Why?**. Is I'm missing anything?
Note…
-
**the example:**
```
func main() {
url := "https://httpbin.org/delay/1"
// Instantiate default collector
c := colly.NewCollector(colly.AllowURLRevisit())
// create a request queue wi…
-
```go
package main
import (
"log"
"net/http"
"github.com/gocolly/colly/v2"
)
func main() {
// we have to disable Accept-Encoding: gzip,
// because the remote server might send comp…
-
If a server is broken and returns an invalid or missing content-length, colly reports "unexpected EOF", despite having received data.
Repro:
1. use colly to visit https://regexr.com/39p0t
2. r…
-
The default setting is not suitable for long runs
-
The headers module should get a feature that allows for randomized user-agents per request.
The feature should be enabled when the following configuration is provided.
```js
headers: {
"User…
-
I did make Scraping for Amazon Product Titles but Amazon captcha catches my scraper. I tried 10 times with go run main.go(8 times catches me - 2 times I scrapped the product title)
I researched thi…
-
When a server requests an upgrade to a websocket, colly will hang indefinitely regardless of timeout settings.
Here is an example program and domain that demonstrates this problem:
```go
packag…
-
Hi there,
thank you for your amazing job, it's really a great framework!
I'm scratching my head several days but can't understand what's wrong ...
I'm using HTTP Forward Proxy (Squid) and User-Age…