Documentation - configure Llama.cpp server with parallel running and GPU acceleration, AMD GPUs - Githubissues

eliranwong / toolmate

ToolMate AI, developed by Eliran Wong, is a cutting-edge AI companion that seamlessly integrates agents, tools, and plugins to excel in conversations, generative work, and task execution. Supports custom workflow and plugins to automate multi-step actions.

https://letmedoit.ai

GNU Affero General Public License v3.0

126 stars 13 forks source link

Documentation - configure Llama.cpp server with parallel running and GPU acceleration, AMD GPUs #22

Closed eliranwong closed 5 months ago

eliranwong commented 5 months ago

Documentation - configure Llama.cpp server with parallel running and GPU acceleration

eliranwong commented 5 months ago

Screenshot 2024-06-06 at 11 32 54

eliranwong commented 5 months ago

in progress at https://github.com/eliranwong/freegenius/wiki/Llama.cpp-Server-for-GPU-Acceleration

eliranwong commented 5 months ago

Integrated AMD GPU; No Discrete GPU

Tested device: Ryzen 9 6900HX CPU + integrated Radeon 680M GPU

-ngl 33

Screenshot from 2024-06-08 00-39-04

eliranwong commented 5 months ago

Done: https://github.com/eliranwong/freegenius/wiki/Llama.cpp-Server-with-GPU-Acceleration

eliranwong commented 4 months ago

llamacpp_with_gpu_offloading_compressed