Depth limit to set how levels deep links to be crawled.

omkarcloud / botasaurus

The All in One Framework to build Awesome Scrapers.

https://www.omkar.cloud/botasaurus/

MIT License

1.14k stars 103 forks source link

Depth limit to set how levels deep links to be crawled. #143

Open sonupandey228 opened 1 week ago

sonupandey228 commented 1 week ago

Is there any way to set the depth limit of links to be crawled ? Similar feature present in scrapy that controls how many levels deep the spider will go when following links from the initial URLs provided.