feat(crawl): Similar URL deduplication

mendableai / firecrawl

🔥 Turn entire websites into LLM-ready markdown or structured data. Scrape, crawl and extract with a single API.

https://firecrawl.dev

GNU Affero General Public License v3.0

19.26k stars 1.5k forks source link

Closed mogery closed 2 weeks ago

mogery commented 3 weeks ago

This PR adds some important features:

If a crawl's scrape was redirected, the target URL is also locked, preventing it from being scraped again later.
lockURL now generates similar URLs that will presumably return the same content, and it will lock all of them, preventing the same content from being scraped twice. This can be disabled with the deduplicateSimilarURLs crawl parameter (on by default).
Added missing parameters to the JS SDK.

nickscamara commented 2 weeks ago

Worth adding tests to the dedup stuff