Knowledge API

Standalone Knowledge Tool to be used with GPTScript and GPTStudio

Build

Requires Go 1.22+

make build

Run

The knowledge tool can run in two modes: server and client, where client can be standalone or referring to a remote server.

You can find a full gptscript-generated documentation in the CLI documentation.

Client - Standalone

knowledge create-dataset foobar
knowledge ingest -d foobar README.md
knowledge retrieve -d foobar "Which filetypes are supported?"
knowledge delete-dataset foobar

Server & Client - Server Mode

knowledge server

export KNOW_SERVER_URL=http://localhost:8000/v1
knowledge create-dataset foobar
knowledge ingest -d foobar README.md
knowledge retrieve -d foobar "Which filetypes are supported?"
knowledge delete-dataset foobar

Supported File Types

• .pdf
• .html
• .md
• .txt
• .docx
• .odt
• .rtf
• .csv
• .ipynb
• .json

OpenAPI / Swagger

The API is documented using OpenAPI 2.0 (Swagger), automatically generated using swaggo/swag (make openapi).

GPTScript Examples

Note: The examples in the examples/ directory expect the knowledge binary to be in your $PATH.

Run

gptscript examples/client.gpt

Architecture & Components

The knowledge tool is composed of the following components, which are all run from the same executable:

• knowledge client
  • can run in two modes:
  • standalone (client/standalone): manages its own vector and knowledge database locally
  • server/remote (client/default): interacts with a knowledge server over the network
• knowledge server (server)
  • lets you run a REST API server that interacts with the below databases, so the client is stateless and sends/receives data over the network
• datastore (datastore)
  • responsible for handling data ingestion and retrieval
  • ingestion includes
    • loading documents (extracting text)
    • splitting text into chunks
    • pre-processing (e.g. metadata extraction, content enrichment)
    • requesting embeddings (part of the vectorstore implementation)
    • storing embeddings and metadata in the vector database
    • registering the document in the knowledge database (index)
  • retrieval includes
    • query embedding
    • querying the vector database for embeddings - similarity search
    • mapping the retrieved embeddings to document contents
    • (optional) post-processing (e.g. filtering, sorting, summarization)
    • returning document contents alongside their similarity scores
  • consists of two databases:
  • vector database (vectorstore)
    • Current choice: chromem-go
    • used for storing and retrieving embeddings alongside the document contents
    • the implementation is responsible for
    • requesting the embeddings from a model (e.g. OpenAI's text-embeddings-ada-002)
    • storing the embeddings together with metadata and document contents
    • doing similarity searches to retrieve embeddings
    • returning document contents alongside their similarity scores
  • knowledge database (index
    • Current choice: sqlite3
    • used for
    • indexing knowledge bases (datasets): dataset <(1:n)> files <(1:n)> documents
      • this is useful for deleting specific documents or files from a dataset and to get quick overviews over datasets without having to query the vector database (which holds this information in the metadata)
    • storing knowledge base metadata and e.g. attached ingestion flows