search

abetlen / llama-cpp-python

Python bindings for llama.cpp
https://llama-cpp-python.readthedocs.io
MIT License
8.22k stars 980 forks source link

OS package system builds? (Apt, yum, etc) #1480

Open walterheck opened 6 months ago

walterheck commented 6 months ago

Is your feature request related to a problem? Please describe. I am using llama-cpp-python in some projects and the difference in build time between using and not using llama-cpp-python is 15-20 minutes when building.

Describe the solution you'd like It occurs to me that rebuilding this package everytime you pip install is a little much, and it should be preventable if this package provided apt packages so there's no rebuild needed on every single pip install.

Describe alternatives you've considered Anything to lower the processing time would help

metal3d commented 5 months ago

There are now binaries packages (see the README) and you can build, also, your own "wheel" packages. I personally don't think that apt/dnf/emerge packages are OK for this kind of package. Because there are many build possibilities (CUDA, Vulkan,...)

IMHO, "wheel" packages are a better option.

BTW: it's a problem when CI build the binaries as AVX support is not compiled AFAIK, and I need it. There is only AVX2 support.