search

cztomsik / ava

All-in-one desktop app for running LLMs locally.
https://avapls.com
Other
419 stars 15 forks source link

enable avx and avx2 if supported when GPU doesn't exist #18

Closed prabirshrestha closed 6 months ago

prabirshrestha commented 8 months ago

Noticed that ollama recently added support for avx and avx2 instructions.

Fixed performance issues on Windows without GPU acceleration. Systems with AVX and AVX2 instruction sets should be 2-4x faster.

https://github.com/ollama/ollama/releases/tag/v0.1.27

cztomsik commented 8 months ago

According to this it might be just adding option to ./src/windows/create_zip.sh and/or compiling linux with similar flags

https://github.com/ggerganov/llama.cpp/blob/cb49e0f8c906e5da49e9f6d64a57742a9a241c6a/README.md?plain=1#L298

cztomsik commented 6 months ago

Superseded by https://github.com/cztomsik/ava/issues/23

cztomsik commented 6 months ago

Closing in favor of #23