search

unclecode / crawl4ai

🔥🕷️ Crawl4AI: Open-source LLM Friendly Web Crawler & Scrapper
Apache License 2.0
16.58k stars 1.23k forks source link

How to crawl current webpage only? #242

Open OpEnD17 opened 2 weeks ago

OpEnD17 commented 2 weeks ago

Whenever I crawl a webpage, this tool not only scrapes the content of the webpage but also automatically visits the URLs on the page. This process is very time-consuming. How can I make the tool scrape only the content of the given webpage?

unclecode commented 2 weeks ago

@OpEnD17 I'd like to see your code because our library doesn't work that way. When you pass a URL, it only crawls that URL and extracts links, but doesn't crawl the external links. Please share your code so I can help.