search

Qianlitp / crawlergo

A powerful browser crawler for web vulnerability scanners
GNU General Public License v3.0
2.82k stars 469 forks source link

是否支持多域名绑定同一ip的去重方式？ #5

Open maybe-why-not opened 4 years ago

maybe-why-not commented 4 years ago

像tuchong这样一个用户给一个子域名的，https://shenan.tuchong.com/work，https://seatory.tuchong.com/posts。

djerryz commented 4 years ago

可能类似问题: 爬虫循环投入了: A.xxx.com/ A.xxx.com/1.html A.xxx.com/2.html 这三个目标,然后参数为--fuzz-path, 是否会存在大量重复请求的情况, 请问如何以A.xxx.com为主键, 避免重复请求.

djerryz commented 4 years ago

是要把1.html 2.html,提取出来放到字典里面嘛