search

atomicptr / crab

🦀 A versatile tool to crawl dozens of URLs from a given source, like a sitemap or an URL list.
MIT License
20 stars 6 forks source link

sitemap request doesn't use --header flag #7

Closed Smadarl closed 1 year ago

Smadarl commented 1 year ago

When running crab crawl:sitemap https://www.domain.com/sitemap.xml --header user-agent=some-whitelisted-user-agent the request for the sitemap doesn't use the user agent header.

atomicptr commented 1 year ago

Fixed with v1.3.2, thanks for reporting.