Closed Velka-DEV closed 8 months ago
Very nice! Could you also document how to crawl a sitemap using the new functionality? A clear example will be helpful for most users 👍
Very nice! Could you also document how to crawl a sitemap using the new functionality? A clear example will be helpful for most users 👍
I've added a quick guide on using SitemapUrlParser for sitemap crawling in the docs. I also added some tests for sitemap indexes.
Also, Happy New Year 🎉 !
🥳 Thanks, happy new year to you too!
Thanks!
Feature Overview
Purpose
This update addresses the current limitation where only
<a href="...">
links are discoverable. Previously, there was no support for parsing elements like sitemaps and iframes.Modifications
LinkAdder
class has been refashioned into theLinkUrlParser
, now implementing theUrlParser
interface.SitemapUrlParser
capable of parsing<loc>
nodes within sitemaps.LinkUrlParser
retains its original functionality and implementation, mirroring the erstwhileLinkAdder
.Crawler
class:public function setUrlParserClass(string $urlParserClass): self
. This method allows for the dynamic assignment of the URL parser class.Testing
SitemapUrlParserTest
file to validate the functionality of theSitemapUrlParser
.