search

EricLBuehler / mistral.rs

Blazingly fast LLM inference.
MIT License
3.59k stars 254 forks source link

Add C api and provide shared and static libraries. #258

Open maximus2600 opened 5 months ago

maximus2600 commented 5 months ago

It would be nice to a stable (or versioned) C api and provide a way to compiled shared and static libraries so one can created bindings for various othe languages.

The advantage is one can use these to embed inference engine into their own programs to eliminate the network/http latencies, which might be crucial for low compute tasks when network latency is greater than the actual compute.

Other is that once there are bindings for various languages, a lot more people can write micro services/web GUI/..., which will broaden the usage of this project.

EricLBuehler commented 4 months ago

Hi @maximus2600!

I have begun work on a C shared library FFI binding for mistral.rs. Initially, it will support the plain model type, and gguf is planned to be next.