-
I have setup an OnRequest callback like so:
```
c.OnRequest(func(r *colly.Request) {
for header, value := range headers {
r.Headers.Set(header, value)
}
})
```
One of these headers is the …
-
# 新闻页 #
https://www.iema.net/resources/news/
# 问题 #
1. 脚本不运行(见下图)
2. 网站翻页的href为“#”(见下图)
# 脚本代码 #
```go
package dev
import (
"megaCrawler/crawlers"
"megaCrawler/extractors"
…
-
Any plan to add JavaScript engines to this framework. A few projects that might help are
https://github.com/robertkrimen/otto
https://github.com/lazytiger/go-v8
https://github.com/dop251/goja
…
-
I'm using an array of HTTP proxies and setting up the collector as described in the example:
```go
c := colly.NewCollector(
colly.MaxDepth(cfg.MaxDepth),
colly.URLFilters(
…
-
**Is your feature request related to a problem? Please describe.**
Currently the crawler sequentially fetches each paper details, parses it and downloads the paper. This can be made lot faster using …
-
Hi, I'm trying to web scrapping [YouTube charts](https://charts.youtube.com/?hl=pt), unsuccessfully because they use polymer / shadow DOM. With Geziyor, could I do that? I'm using colly, and they don'…
-
I crawl website like http://127.0.0.1:8080,AllowedDomains is 127.0.0.1:8080,crawl is not work.
in code github.com/gocolly/colly/v2.(*Collector).isDomainAllowed at colly.go:752
if d2 == domain { /…
-
I'm attempting to use colly to write a program that first logs in to a site, and then obtains some information. The site uses 2 different domains to handle authentication, say `www.site.com` and `auth…
-
Hi, I'm newbie and I have a question.
First of all, thanks to make and share this cooool project!
I'm trying to do some examples but the 'cryptocoinmarketcap.go' file doesn't work.
I think …
-
single url#1
```
> go run cmd/xscrap/main.go -urls https://kreuzwerker.de/post/aws-summit-2022-berlin-have-some-pie -tags Elasticsearch
url: https://kreuzwerker.de/post/aws-summit-2022-berlin-hav…