search

filipw / strathweb-phi-engine

A cross platform library with C#/Swift/Kotlin/Python bindings for running Phi-3 inference
MIT License
11 stars 1 forks source link

readme

Strathweb Phi Engine

A cross-platform library for running Microsoft's Phi-3 locally using candle in GGUF format. Safetensors support is on the way.

Supported platforms

Swift .NET Kotlin Python
Package Swift Package Nuget N/A N/A
Manual Integration Bindings + XCFramework Bindings + native library Bindings + native library Bindings + native library Bindings + native library
Platforms macOS arm64
iOS		 Windows x64
Windows arm64 (not via Nuget)
Linux x64
Linux arm64 (not via Nuget)
macOS arm64		 Windows x64
Linux x64
macOS arm64		 Windows x64
Windows arm64
Linux x64
Linux arm64
macOS arm64

Building instructions

Swift

Build the Swift Package (arm64 Mac required).

./build-swift.sh

This builds:

Now open samples/io/phi.engine.sample/phi.engine.sample.xcodeproj and build the SwiftUI app (iOS), or go to samples/swift and run ./run.sh (macOS) to launch the Swift console app.

C

Install UniFFI C# bindings generator

cargo install uniffi-bindgen-cs --git https://github.com/NordSecurity/uniffi-bindgen-cs --tag v0.8.0+v0.25.0

Build the Nuget package for your platform:

./build-dotnet.sh

or (on Windows)

build-dotnet.bat

or

cargo build --release --manifest-path strathweb-phi-engine/Cargo.toml
dotnet build packages/csharp -c Release
dotnet pack packages/csharp -c Release -o artifacts/csharp

Nuget package will be in artifacts/csharp/Strathweb.Phi.Engine.0.1.0.nupkg. (Optional) Run the sample console app:

cd samples/csharp/console
dotnet run -c Release

Kotlin

Run the sample console app:

cd samples/kotlin
./run.sh

Python

Run the sample console app:

cd samples/python/console
./run.sh # or run.bat on Windows

or use the Jupyter Notebooks

cd samples/python/jupyter
./init.sh # or init.bat on Windows

Now open the Notebook and run the cells.

Compatibility notes

.NET

✅ Tested on Windows arm64

✅ Tested on Windows x64

✅ Tested on Linux arm64

✅ Tested on Linux x64

✅ Tested on macOS arm64. Supports Metal.

Swift

✅ Tested on macOS arm64. Supports Metal.

✅ Tested on iPad Air M1 8GB RAM

✅ Should work on 6GB RAM iPhones too

❌ Will not work on 4GB RAM iPhones

However, for 4GB RAM iPhones, it's possible to use the (very) low fidelity Q2_K quantized model. Such model is not included in the official Phi-3 release, but I tested this one from HuggingFace on an iPhone 12 mini successfully.

Kotlin

✅ Tested on macOS arm64. Supports Metal.

Python

✅ Tested on Windows arm64

✅ Tested on macOS arm64. Supports Metal.

AutoGen

The repository also contains a C# integration library for AutoGen, called Strathweb.Phi.Engine.AutoGen. There is an example in the samples/csharp/autogen folder. It allows creating a local Phi-3 agent and integrating it into the other typical AutoGen workflows.

The NuGet package is built when running ./build-dotnet.sh (build-dotnet.bat on Windows).

Blog post

For an announcement post, go here.