EricLBuehler / mistral.rs

Blazingly fast LLM inference.
MIT License
4.4k stars 304 forks source link

Compiled wheels on PyPI would be really useful #864

Open simonw opened 2 weeks ago

simonw commented 2 weeks ago

The PyPI builds of mistralrs - https://pypi.org/project/mistralrs/ and https://pypi.org/project/mistralrs-metal/ and suchlike - currently only ship a .tar.gz file.

This means the user must have a Rust toolchain installed in order for pip install mistralrs-metal to work.

It would be great if you could publish compiled wheels for macOS and Linux (and Windows too if that works) to PyPI.

Here's a macOS Apple Silicon Python 3.10 wheel I built when I ran pip install mistralrs-metal just now:

https://static.simonwillison.net/static/2024/mistralrs_metal-0.3.1-cp310-cp310-macosx_11_0_arm64.whl

Anyone with Python 3.10 on macOS Apple Silicon should be able to install that like this:

pip install https://static.simonwillison.net/static/2024/mistralrs_metal-0.3.1-cp310-cp310-macosx_11_0_arm64.whl

GitHub Actions has macOS runners for Apple Silicon now so it should be possible to do this entirely in a GitHub Actions workflow.

I believe https://github.com/pypa/cibuildwheel is the best way to do this kind of thing, but I've not used it myself.

simonw commented 2 weeks ago

It looks like this repo is using both Rust and cibuildwheel in a GitHub Actions workflow to build wheels across multiple operating systems: https://github.com/google/flax/blob/c1046ad08f885e3ff35e6fdec2bd0b6495e04e22/.github/workflows/flaxlib_publish.yml#L11-L33

simonw commented 2 weeks ago

This is the simplest example I've found of cibuildwheel in action: https://github.com/adamchainz/time-machine/blob/2.15.0/.github/workflows/build.yml - it produces 55 wheels: https://pypi.org/project/time-machine/#files