search

mendableai / firecrawl

🔥 Turn entire websites into LLM-ready markdown or structured data. Scrape, crawl and extract with a single API.
https://firecrawl.dev
GNU Affero General Public License v3.0
17.39k stars 1.26k forks source link

疑问 #770

Closed zhk0603 closed 1 week ago

zhk0603 commented 1 week ago

https://github.com/mendableai/firecrawl/blob/5ab52854b9d49983f1e2ce10e4b4fb8585c8de42/apps/api/src/scraper/WebScraper/crawler.ts#L352

为什么第一个链接才使用 scrapSingleUrl 方法进行爬取，后续通过 axios，在我使用过程中，许多网站使用 axion 爬取不了，导致无法找到链接

nickscamara commented 1 week ago

This is the old crawler implementation. We have moved away from that quite a few months ago. Most of crawler.ts isn't used anymore