This repository contains an AI-powered knowledge base that utilizes the LLMs model to answer questions based on a given website's content and provide sources as links to the relevant pages.
The system:
It also provides a Streamlit-based web interface for an easy-to-use experience.
knowledge_base.py
: The main module that creates the KnowledgeBase class. This class is responsible for loading and processing the website content, creating the document index, and querying the LLM model for answers.app.py
: A Streamlit web application that provides a user interface for querying the AI-powered knowledge base.git clone git@github.com:bitswired/website-to-knowledge-base.git
poetry install
To use the KnowledgeBase class, follow these steps:
from knowledge_base import KnowledgeBase
kb = KnowledgeBase(
sitemap_url="https://nextjs.org/sitemap.xml",
pattern="docs/api-refe",
chunk_size=8000,
chunk_overlap=3000,
)
result = kb.ask("How do I deploy my Next.js app?")
print(result)
To run the Streamlit web application, execute the following command in your terminal:
streamlit run app.py
The web app will open in your default browser. Enter the URL to the website's sitemap, an optional filter pattern for the URLs, and your question. The AI-powered knowledge base will return an answer based on the content of the website.
This project is licensed under the MIT License.