search

GraveHag / CyberspaceSpider

CyberspaceSpider is a visualization-based web crawling project that maps the path a web crawler takes as it navigates through the internet. With CyberspaceSpider, you can gain insights into the structure of the web and the relationships between different sites. It is a simple and intuitive tool that provides a unique perspective on web crawling.
1 stars 0 forks source link

WebWalker configuration #14

Closed GraveHag closed 1 year ago

GraveHag commented 1 year ago

WebWalker configuration

A task that represents load configuration for web walker could be to set the maximum number of concurrent requests or threads that the web walker can make at a time.

This configuration can help optimize the web walker's performance and prevent it from overloading the server or network. Other load configurations for a web walker might include setting a maximum request rate, defining a timeout period for requests, or specifying a user-agent string to identify the web walker to web servers.

GraveHag commented 1 year ago

[StartDomain], [MaxDepth on Domain], [?MaxCrawler], [TimeToComplete], [MaxCountOfDomain], [?MaxRequestPerCrawler]