[Feature Suggest] Support for AVX instruction set

b4rtaz / distributed-llama

Tensor parallelism is all you need. Run LLMs on an AI cluster at home using any device. Distribute the workload, divide RAM usage, and increase inference speed.

MIT License

1.53k stars 109 forks source link

[Feature Suggest] Support for AVX instruction set #107

Open mrlicmi opened 4 months ago

mrlicmi commented 4 months ago

I have couple of VMs but only plain AVX instruction set not AVX2. Is this project compatible with AVX ?

b4rtaz commented 4 months ago

From 0.10.1 version all CPUs are supported but Distributed Llama is optimized only for ARM and AVX2. AVX is within range but with a low priority.