search

gocolly / colly

Elegant Scraper and Crawler Framework for Golang
https://go-colly.org/
Apache License 2.0
22.97k stars 1.75k forks source link

[Question] Filename extensions blacklist #686

Open mxrch opened 2 years ago

mxrch commented 2 years ago

Hi, thanks for your tool it's pretty cool :) I wanted to know, does colly has an extensions blacklist to avoid checking ?

For example, it should check .js, .xml, .php, .html etc, but not .css, .exe, .jar, .zip etc.

mozhenshuang commented 1 year ago

好像目前并没有，甚至连colly.LimitRule的DomainGlob都已经失效，我是自己在外部实现了一个黑名单策略