search

open-webui / pipelines

Pipelines: Versatile, UI-Agnostic OpenAI-Compatible Plugin Framework
MIT License
768 stars 232 forks source link

feat: GPT Crawler RAG pipeline #12

Closed zd1990 closed 2 months ago

zd1990 commented 8 months ago

Thank you for your work and hope to integrate GPT Crawler into RAG's functionality

justinh-rahb commented 8 months ago

Do you mean RAG from web URLs? If so that's already implemented, if you type a # followed by a URL and then click it, it'll be added as a RAG "document" to your chat.

sahajamit commented 7 months ago

But it only scraps the content of given page URL. Is it possible to scan all the links on the page which takes to other section of the website?

justinh-rahb commented 4 months ago

https://github.com/mendableai/firecrawl

I've tested this, it works.