Explore Gaia tools for generating embeddings

[santteegt]

Explore available tools/node settings to make Gaia the primary provider for generating embeddings.

• [x] The aim to find the most lightweight solution to use an embeddings model for text-to-vector-embeddings using Gaia tooling.

• [x] List the pro/cons of potential solutions

• Available tools:

  • https://github.com/GaiaNet-AI/embedding-tools (docs)
  • https://github.com/GaiaNet-AI/rag-tool

[santteegt]

Cons of using current Gaianet rag tools:

• Rust-based projects: need to rethink the pipeline architecture to integrate with Rust
• Doesn't provide an API Server: the pipeline is designed to generate embeddings via an OpenAI-like API endpoint (/v1/embeddings)

[santteegt]

Work done:

• Deep dive into GaiaNet install script and gaianet execution script
• Deep dive into LlamaEdge and the llama-api-server component
  • Setup and built locally for debugging and potential customization purposes
  • https://github.com/LlamaEdge/LlamaEdge/issues/236 has some convo and solutions found when trying to build the component
• Deep dive into LlamaEdge llama-core package

Solution:

• After deep diving into the GaiaNet node architecture and LlamaEdge components, I figured out that there's no need to implement any extra components to spin up a lightweight LLM API server that only provides embedding generation capabilities (just like Ollama). We only need to provide docs on how to start the llama-api-server (included in the gaianet node) in embeddings-only mode (a.k.a embeddings running mode):

cd $HOME/gaianet
wasmedge --dir .:./dashboard --env NODE_VERSION=0.4.7 \
  --nn-preload embedding:GGML:AUTO:nomic-embed-text-v1.5.f16.gguf llama-api-server.wasm \
  --model-name Nomic-embed-text-v1.5 --model-alias embedding --prompt-template embedding \
  --ctx-size 8192 --batch-size 768 --web-ui ./ --socket-addr 0.0.0.0:8080

• Under the hood, GaiaNet uses WasmEdge WebAssembly runtime with the wasi-nn plugin for performing AI inference with LLM interfaces. According to the docs, it does support macOS arm64 platform and it will use the built-in GPU by default. We need to do a few performance tests between Gaia embeddings-only mode vs Ollama