search

Qianlitp / crawlergo

A powerful browser crawler for web vulnerability scanners
GNU General Public License v3.0
2.85k stars 480 forks source link

Crawlergo not working #134

Closed byposeidon closed 1 year ago

byposeidon commented 1 year ago

All of the websites I am trying are giving the following error:

"level=warning msg="navigate timeout" You can try Amazon, eBay, or any other website you want.

On the same sites, Netsparker or Burp Suite can crawl thousands of pages within one minute. However, Crawlergo only gives a timeout error. What is the problem?

Is this application only properly crawling HTTP sites from 10 years ago?

"Crawlergo is useless unless this problem is fixed. And according to my determination, this problem has been continuing for 2 years."

byposeidon commented 1 year ago

选择规模较小的网站可正常爬取，大规模网站则出现上述报错

Qianlitp commented 1 year ago

或许，你只需要设置超时参数，并使用一个网络状况良好的代理即可。

image

byposeidon commented 1 year ago

I set up a server on AWS with 8 CPUs and 32 GB of RAM to retest. The internet download speed is 500 mbit. I made the query as follows.

./crawlergo -c /usr/bin/chromium-browser -t 2 --tab-run-timeout=60s --wait-dom-content-loaded-timeout=60s https://www.amazon.com/

Result: navigate timeout.

It first crawling, but when the timeout period is up it gives a "navigate timeout" error. The timeout is also written in the picture you shared. If there were more to the picture, I'm sure we would see more timeout errors.

Your suggested solution didn't solve the problem, but thank you for your interest.