Nexa SDK

[![MacOS][MacOS-image]][release-url] [![Linux][Linux-image]][release-url] [![Windows][Windows-image]][release-url] [](https://github.com/NexaAI/nexa-sdk/releases/latest) [](https://github.com/NexaAI/nexa-sdk/actions/workflows/ci.yaml?query=branch%3Amain)  [](https://discord.gg/thRu2HaK4D) [On-device Model Hub](https://model-hub.nexa4ai.com/) / [Nexa SDK Documentation](https://docs.nexaai.com/) [release-url]: https://github.com/NexaAI/nexa-sdk/releases [Windows-image]: https://img.shields.io/badge/windows-0078D4?logo=windows [MacOS-image]: https://img.shields.io/badge/-MacOS-black?logo=apple [Linux-image]: https://img.shields.io/badge/-Linux-333?logo=ubuntu

Nexa SDK is a comprehensive toolkit for supporting ONNX and GGML models. It supports text generation, image generation, vision-language models (VLM), and text-to-speech (TTS) capabilities. Additionally, it offers an OpenAI-compatible API server with JSON schema mode for function calling and streaming support, and a user-friendly Streamlit UI. Users can run Nexa SDK in any device with Python environment, and GPU acceleration is supported.

Features

• Model Support:

  • ONNX & GGML models
  • Conversion Engine
  • Inference Engine:
  • Text Generation
  • Image Generation
  • Vision-Language Models (VLM)
  • Text-to-Speech (TTS)

Detailed API documentation is available here.

• Server:
  • OpenAI-compatible API
  • JSON schema mode for function calling
  • Streaming support
• Streamlit UI for interactive model deployment and testing

Below is our differentiation from other similar tools:

Installation

We have released pre-built wheels for various Python versions, platforms, and backends for convenient installation on our index page.

[!NOTE]

1. If you want to use ONNX model, just replace pip install nexaai with pip install "nexaai[onnx]" in provided commands.
2. For Chinese developers, we recommend you to use Tsinghua Open Source Mirror as extra index url, just replace --extra-index-url https://pypi.org/simple with --extra-index-url https://pypi.tuna.tsinghua.edu.cn/simple in provided commands.

CPU

pip install nexaai --prefer-binary --index-url https://nexaai.github.io/nexa-sdk/whl/cpu --extra-index-url https://pypi.org/simple --no-cache-dir

GPU (Metal)

For the GPU version supporting Metal (macOS):

CMAKE_ARGS="-DGGML_METAL=ON -DSD_METAL=ON" pip install nexaai --prefer-binary --index-url https://nexaai.github.io/nexa-sdk/whl/metal --extra-index-url https://pypi.org/simple --no-cache-dir

GPU (CUDA)

For Linux:

CMAKE_ARGS="-DGGML_CUDA=ON -DSD_CUBLAS=ON" pip install nexaai --prefer-binary --index-url https://nexaai.github.io/nexa-sdk/whl/cu124 --extra-index-url https://pypi.org/simple --no-cache-dir

For Windows PowerShell:

$env:CMAKE_ARGS="-DGGML_CUDA=ON -DSD_CUBLAS=ON"; pip install nexaai --prefer-binary --index-url https://nexaai.github.io/nexa-sdk/whl/cu124 --extra-index-url https://pypi.org/simple --no-cache-dir

For Windows Command Prompt:

set CMAKE_ARGS="-DGGML_CUDA=ON -DSD_CUBLAS=ON" & pip install nexaai --prefer-binary --index-url https://nexaai.github.io/nexa-sdk/whl/cu124 --extra-index-url https://pypi.org/simple --no-cache-dir

For Windows Git Bash:

CMAKE_ARGS="-DGGML_CUDA=ON -DSD_CUBLAS=ON" pip install nexaai --prefer-binary --index-url https://nexaai.github.io/nexa-sdk/whl/cu124 --extra-index-url https://pypi.org/simple --no-cache-dir

GPU (ROCm)

For Linux:

CMAKE_ARGS="-DGGML_HIPBLAS=on" pip install nexaai --prefer-binary --index-url https://nexaai.github.io/nexa-sdk/whl/rocm602 --extra-index-url https://pypi.org/simple --no-cache-dir

Local Build

How to clone this repo

git clone --recursive https://github.com/NexaAI/nexa-sdk

If you forget to use --recursive, you can use below command to add submodule

git submodule update --init --recursive

Then you can build and install the package

pip install -e .

Docker Usage

Note: Docker doesn't support GPU acceleration

docker pull nexa4ai/sdk:latest

replace following placeholder with your path and command

docker run -v <your_model_dir>:/model -it nexa4ai/sdk:latest [nexa_command] [your_model_relative_path]

Example:

docker run -v /home/ubuntu/.cache/nexa/hub/official:/model -it nexa4ai/sdk:latest nexa gen-text /model/Phi-3-mini-128k-instruct/q4_0.gguf

will create an interactive session with text generation

Supported Models

CLI Reference

Here's a brief overview of the main CLI commands:

• nexa run: Run inference for various tasks using GGUF models.
• nexa onnx: Run inference for various tasks using ONNX models.
• nexa server: Run the Nexa AI Text Generation Service.
• nexa pull: Pull a model from official or hub.
• nexa remove: Remove a model from local machine.
• nexa clean: Clean up all model files.
• nexa list: List all models in the local machine.
• nexa login: Login to Nexa API.
• nexa whoami: Show current user information.
• nexa logout: Logout from Nexa API.

For detailed information on CLI commands and usage, please refer to the CLI Reference document.

Start Local Server

To start a local server using models on your local computer, you can use the nexa server command. For detailed information on server setup, API endpoints, and usage examples, please refer to the Server Reference document.

Acknowledgements

We would like to thank the following projects:

• llama.cpp
• stable-diffusion.cpp
• optimum