[BUG] Issue crawling swift documentation

mendableai / firecrawl

🔥 Turn entire websites into LLM-ready markdown or structured data. Scrape, crawl and extract with a single API.

https://firecrawl.dev

GNU Affero General Public License v3.0

7.09k stars 507 forks source link

[BUG] Issue crawling swift documentation #286

Open calebpeffer opened 2 weeks ago

calebpeffer commented 2 weeks ago

Issue scraping the swift documentation:

Bug described by user on the /crawl endpoint

_Trying to scrape the documentation for Swift UI https://developer.apple.com/documentation/swiftui/ but it keeps stopping. It only does two pages, even with limit set to 50 and Include Only Paths=/documentation/swiftui/*

It got https://developer.apple.c...swiftui/view-fundamentals https://developer.apple.c...ntation/swiftui/clipboard_

calebpeffer commented 1 week ago

Notify @ gregschwartz when fixed

snippet commented 3 days ago

This issue was fixed by running the #325 service as a PLAYWRIGHT_MICROSERVICE_URL.