Closed zhk0603 closed 1 week ago
https://github.com/mendableai/firecrawl/blob/5ab52854b9d49983f1e2ce10e4b4fb8585c8de42/apps/api/src/scraper/WebScraper/crawler.ts#L299
This is the old crawler implementation. We have moved away from that quite a few months ago. Most of crawler.ts isn't used anymore.
https://github.com/mendableai/firecrawl/blob/5ab52854b9d49983f1e2ce10e4b4fb8585c8de42/apps/api/src/scraper/WebScraper/crawler.ts#L299