search

prefix-dev / pixi

Package management made easy
https://pixi.sh
BSD 3-Clause "New" or "Revised" License
3.02k stars 166 forks source link

docs: Added LLM example #1545

Closed ytjhai closed 2 months ago

ytjhai commented 3 months ago

I noticed there had been a few issues like #206 and #261 for demoing Pixi with an ML stack. This example shows it's possible with the current feature set to run Llama-index with llama.cpp on a local machine with a Mistral LLM, also running locally with GPU support on an ARM M1 Mac. Hopefully this helps someone else get started with LLM inference.

tdejager commented 3 months ago

Hi @ytjhai, thanks a lot for the PR.

Would you be open to adding

# linux-64 to the platforms
platforms = ["osx-arm64", "linux-64"]

# setting this avoids building scikit_learn from source (on arm)
[system-requirements]
macos = "12.0"
tdejager commented 3 months ago

Hi @ytjhai, thanks a lot for the PR.

Would you be open to adding

# linux-64 to the platforms
platforms = ["osx-arm64", "linux-64"]

# setting this avoids building scikit_learn from source (on arm)
[system-requirements]
macos = "12.0"

Even better would be having separate features and environments. Where the system-requirement for osx-arm64 and osx are different. This would allow older non-arm osx versions.

ytjhai commented 3 months ago

I don't currently have access to a linux machine, so can't 100% verify if this lock file would work on linux-64. But I could put a test environment together when I have some time? It would have an AMD GPU, which is a bit off the beaten path (similar to the idea behind this build).

Regarding older OSX environments, GPU strength was very patchy and my personal use case was running some LLMs locally since M1 has good GPU support in even base models (like the Mac Mini). Even llama.cpp doesn't support that architecture on Macs AFAIK.

Hofer-Julian commented 3 months ago

I don't currently have access to a linux machine, so can't 100% verify if this lock file would work on linux-64. But I could put a test environment together when I have some time? It would have an AMD GPU, which is a bit off the beaten path (similar to the idea behind this build).

I tested it with an AMD GPU (I assume it is using the CPU though) and it seems to work fine :)

ytjhai commented 3 months ago

Great! I made the above requested changes, adding linux-64 as a supported environment

tdejager commented 2 months ago

Last, tiny small thing, could you maybe add a pixi run start task that runs the python command :)?

tdejager commented 2 months ago

Last, tiny small thing, could you maybe add a pixi run start task that runs the python command :)?

Don't worry I've added it :)