search

3x3cut0r / docker

several docker containers
https://github.com/3x3cut0r/docker
19 stars 9 forks source link

[llama-cpp-python] llama-2-7b-chat.Q4_0.gguf model not loaded on GPU #4

Closed sharifAhmd closed 6 months ago

sharifAhmd commented 8 months ago

In docker compose, I am using these envs, but it is not loading the model in GPU

environment:
        MODEL: "/model/llama-2-7b-chat.Q4_0.gguf"
        MAIN_GPU: 1
        N_GPU_LAYERS: 500
3x3cut0r commented 8 months ago

i never tried gpu usage because i have no gpu. so i need more details why it is not working?

3x3cut0r commented 6 months ago

This container does not support GPU