langgenius / dify

Dify is an open-source LLM app development platform. Dify's intuitive interface combines AI workflow, RAG pipeline, agent capabilities, model management, observability features and more, letting you quickly go from prototype to production.
https://dify.ai
Other
51.88k stars 7.53k forks source link

add Crawl4AI to tools and for sync data from website in Knowledge #8905

Open chankwongyin opened 1 month ago

chankwongyin commented 1 month ago

Self Checks

1. Is this request related to a challenge you're experiencing? Tell me about your story.

I found from Crawl4AI who states that they outperforms Firecrawl significantly:

Simple crawl: Crawl4AI is over 4 times faster than Firecrawl. With JavaScript execution: Even when executing JavaScript to load more content (doubling the number of images found), Crawl4AI is still faster than Firecrawl's simple crawl.

I would suggest to add this tool to dify to allow different ways to crawl the data from websites.

2. Additional context or comments

No response

3. Can you help us with this feature?

chankwongyin commented 1 month ago

For your convenience: https://github.com/unclecode/crawl4ai