Alpaca Core Local SDK

[!NOTE] This project is still in in an alpha stage of development. Significant changes are very likely and backwards compatibility is disregarded.

The Alpaca Core Local SDK, or AC Local for short, is a multi-platform SDK for local AI Inference.

"Local" here means running on the device which executes the code. This could be a server, a desktop, or a mobile device.

It provides a unified API for doing inference with multiple models. The API itself can be split into two layers:

• Programming language specific (Language API): The API which one calls writing code in a specific programming language. It's just a means to call the:
• Inference API: A JSON/CBOR/POJO-like API which is used to communicate with the underlying inference engines following their specific API schema.

Read the full introduction here.

Supported models

• Multiple LLM-s through llama.cpp
• Whisper through whisper.cpp

This list will be updated as new models are added.

Minimal Example

ac::local::ModelFactory factory;
ac::local::addLlamaInference(factory);

auto model = factory.createModel(
    {
        .inferenceType = "llama.cpp",
        .assets = {
            {.path = "/path/to/model.gguf"}
        }
    }, {}, {}
);

auto instance = model->createInstance("general", {});

auto result = instance->runOp("run",
    {{"prompt", "If you could travel faster than light,"}}, {});

std::cout << result << "\n";

Bindings, Wrappers, and Integrations

• C wrapper
• Java wrapper
• Swift wrapper
• Cocoa DictConverter - Convert NSDictionary to ac::Dict and back in Objective-C++

Demos

There are multiple examples in this repo. Look for the example directories throughout the tree. Besides them, there are also are several larger demos in separate repositories:

• AI Chat (LLM-based chatbot)
  • Android: alpaca-core/demo-android-ai-chat
  • IOS: alpaca-core/demo-ios-ai-chat

Build

The repo has submodules. Don't forget to fetch them.

Use CMake. Works as a root or as a subdirectory. Some useful presets are provided in the repo.

Detailed build instructions can be found in the documentation.

License

This software is distributed under the MIT Software License. See accompanying file LICENSE or copy here.

Copyright © 2024 Alpaca Core, Inc

Third Party Libraries

A list of the third party libraries used here. Please consider supporting them.