feat: custom link parser

[Velka-DEV]

Feature Overview

Purpose

This update addresses the current limitation where only <a href="..."> links are discoverable. Previously, there was no support for parsing elements like sitemaps and iframes.

Modifications

• The LinkAdder class has been refashioned into the LinkUrlParser, now implementing the UrlParser interface.
• Introduced a new SitemapUrlParser capable of parsing <loc> nodes within sitemaps.
• The LinkUrlParser retains its original functionality and implementation, mirroring the erstwhile LinkAdder.
• Added a new method to the Crawler class: public function setUrlParserClass(string $urlParserClass): self. This method allows for the dynamic assignment of the URL parser class.

Testing

• Added new unit tests in the SitemapUrlParserTest file to validate the functionality of the SitemapUrlParser.

[freekmurze]

Very nice! Could you also document how to crawl a sitemap using the new functionality? A clear example will be helpful for most users 👍

[Velka-DEV]

Very nice! Could you also document how to crawl a sitemap using the new functionality? A clear example will be helpful for most users 👍

I've added a quick guide on using SitemapUrlParser for sitemap crawling in the docs. I also added some tests for sitemap indexes.

Also, Happy New Year 🎉 !

[freekmurze]

🥳 Thanks, happy new year to you too!

[freekmurze]

Thanks!