Depth Setting for More Comprehensive Crawling

[aloysius-tim]

Hello,

I've been using Cariddi to crawl various websites and noticed that while it successfully extracts many links, it appears not to capture all possible links. This observation leads me to believe that Cariddi might be limiting the crawl depth or not providing an option to configure the depth of the crawl.

Issue/Feature Request Description:

• While crawling with Cariddi, the tool does not seem to explore links beyond a certain depth.
• I could not find any settings or command-line options to adjust the crawl depth to ensure a more exhaustive search of the website's links.

Expected Behavior:

• An option to configure the depth of the crawl, allowing users to specify how deep Cariddi should go into a website's link structure.

Actual Behavior:

• The tool extracts many links but seems to miss out on deeper links that are only reachable by navigating through several layers of the website.

Steps to Reproduce:

1. Run Cariddi with the default settings on a target website.
2. Notice the output and compare it with a manual check of the website's link depth.
3. Observe that links beyond a certain depth are not included in Cariddi's output.

Possible Solution:

• It would be beneficial if Cariddi included a feature to adjust the crawl depth, such as a --depth flag that users could set according to their needs.

Thank you for developing Cariddi, and I look forward to any guidance or updates regarding this request.

Best regards, Tim

[edoardottt]

Hi Tim, thanks for your contribution. Can you provide me an input to test this scenario? I'm not sure the problem is the depth level of the links.

thanks, edo

[aloysius-tim]

Hi Edo, Here is one of the domain I've tried https://kretzrealestate.com/fr I'm getting only 58 links Thanks, Tim

[edoardottt]

Hi Tim, sorry for the late reply.

I've ran a test with the URL you supplied. I'm getting 58 links too, but in my opinion there aren't more links to crawl.

The last URL I get is https://kretzrealestate.com/fr/annonce/1460y/appartement/ (note that your last one could be different) and on that page there are just already crawled links or links belonging to different domains. Hence, crawling 58 links is the correct behavior here.

What were you expecting? Maybe you have something in mind I can't think of.

p.s.: I've tried using other crawlers too specifying the depth level (something incredibly high, like 1000) and I get the same ~60 links too.

[edoardottt]

Hi @aloysius-tim , is everything working fine?

If yes I would like to close this issue. Let me know, otherwise I'll close the issue as stale

[aloysius-tim]

Hi @edoardottt, Yes all good, thanks a lot for your help ! Cheers !