🔥 Turn entire websites into LLM-ready markdown or structured data. Scrape, crawl and extract with a single API.
GNU Affero General Public License v3.0
19.26k
stars
1.5k
forks
source link
fix(sitemap): scrape with tlsclient #891
Closed
mogery closed 2 weeks ago
Still needs testing on whether this is stable (whether tlsclient has enough capacity to handle all the sitemaps)