yhyu13
commented
11 months ago I am new to this project, too. It looks like you need to set up gpu_layer in the config somewhere, but I don't know how.
Open noblerboy2004 opened 11 months ago
yhyu13
commented
11 months ago I am new to this project, too. It looks like you need to set up gpu_layer in the config somewhere, but I don't know how.
noblerboy2004
commented
11 months ago I am new to this project, too. It looks like you need to set up gpu_layer in the config somewhere, but I don't know how.
I also tried to search over internet. however, I could not find out solution :) Hope any experts could read this thread.
localai-bot
commented
11 months ago Hi! I'm a bot running with LocalAI ( a crazy experiment of @mudler ) - please beware that I might hallucinate sometimes!
_but.... I can also be funny or helpful :smilecat: and I can provide generally speaking good tips or places where to look after in the documentation or in the code based on what you wrote in the issue.
Don't engage in conversation with me, I don't support (yet) replying!
I see that you are using the LocalAI container image from quay.io, which should have CUDA support enabled by default. However, it seems like the GPU is not being used for some reason. To troubleshoot this issue, I would recommend checking the following:
nvidia-cuda-toolkit to install the driver if needed.CUDA_DEVICE_POOL_GPU_OVERRIDE is set to 1 and CUDA_VISIBLE_DEVICES is set to 0-1. You can check this by looking at the logs when starting LocalAI in debug mode (--debug or DEBUG=true).localai models list to see a list of available models and their compatibility information.Sources:
noblerboy2004
commented
11 months ago ⚠️⚠️⚠️⚠️⚠️
Hi! I'm a bot running with LocalAI ( a crazy experiment of @mudler ) - please beware that I might hallucinate sometimes!
but.... I can also be funny or helpful 😸 and I can provide generally speaking good tips or places where to look after in the documentation or in the code based on what you wrote in the issue.
Don't engage in conversation with me, I don't support (yet) replying!
⚠️⚠️⚠️⚠️⚠️
I see that you are using the LocalAI container image from quay.io, which should have CUDA support enabled by default. However, it seems like the GPU is not being used for some reason. To troubleshoot this issue, I would recommend checking the following:
- Ensure that the GPU is properly connected and detected by the system. You can use a tool like NVIDIA X Server to check this.
- Check that the CUDA driver is installed and up-to-date. You can use the command
nvidia-cuda-toolkitto install the driver if needed.- Ensure that the environment variables are properly set when running LocalAI. Make sure that
CUDA_DEVICE_POOL_GPU_OVERRIDEis set to1andCUDA_VISIBLE_DEVICESis set to0-1. You can check this by looking at the logs when starting LocalAI in debug mode (--debugorDEBUG=true).- Check that the model you are using is compatible with CUDA and has been compiled with the correct settings. You can use the command
localai models listto see a list of available models and their compatibility information.- Ensure that the container image is up-to-date and has the latest CUDA drivers installed. You can try updating the image if necessary. I hope this helps! Let me know if you have any further questions or need more assistance.
Sources:
Hi localai-bot,
Thank you for your support. I tried above guides but still failed.
Ensure that the GPU is properly connected and detected by the system. You can use a tool like NVIDIA X Server to check this
Check that the CUDA driver is installed and up-to-date. Same above.
Ensure that the environment variables are properly set
Check that the model you are using is compatible with CUDA. I did downloaded and tried three models below. But still failed.
I pull latest LocalAI but still not working with CUDA.
Please help to process further. Thank you very much.
lunamidori5
commented
11 months ago @noblerboy2004 please post your models yaml file for better review
lunamidori5
commented
11 months ago I am new to this project, too. It looks like you need to set up gpu_layer in the config somewhere, but I don't know how.
noblerboy2004
commented
11 months ago @noblerboy2004 please post your models yaml file for better review
Hi Lunamidori5,
Thank you for your action.
Here is folder of downloaded models:
gpt4all-j-groovy working ok with CPU. No GPU usage -CUDO 0% backend: gpt4all-j context_size: 1024 name: gpt4all-j-groovy parameters: model: ggml-gpt4all-j-v1.3-groovy.bin temperature: 0.2 top_k: 80 top_p: 0.7 template: chat: gpt4all-chat completion: gpt4all-completion main_gpu: "0"
I Tried open-llama-3b-q4_0
backend: llama context_size: 1024 name: openllama f16: true ## If you are using cpu set this to false gpu_layers: 4 batch: 512 parameters: model: open-llama-3b-q4_0.bin temperature: 0.2 top_k: 80 top_p: 0.7 template: chat: openllama-chat completion: openllama-completion roles: assistant: 'ASSISTANT:' system: 'SYSTEM:' user: 'USER:'
Content of openllama-chat.tmpl: Q: {{.Input}}\nA: Content of openllama-completion.tmpl: Q: Complete the following text: {{.Input}}\nA:
And error occured with log:
2023-09-27 06:35:37 11:35PM DBG Request received:
2023-09-27 06:35:37 11:35PM DBG Configuration read: &{PredictionOptions:{Model:open-llama-3b-q4_0.bin Language: N:0 TopP:0.7 TopK:80 Temperature:0.9 Maxtokens:0 Echo:false Batch:0 F16:false IgnoreEOS:false RepeatPenalty:0 Keep:0 MirostatETA:0 MirostatTAU:0 Mirostat:0 FrequencyPenalty:0 TFZ:0 TypicalP:0 Seed:0 NegativePrompt: RopeFreqBase:0 RopeFreqScale:0 NegativePromptScale:0 UseFastTokenizer:false ClipSkip:0 Tokenizer:} Name:openllama F16:true Threads:32 Debug:true Roles:map[assistant:ASSISTANT: system:SYSTEM: user:USER:] Embeddings:false Backend:llama TemplateConfig:{Chat:openllama-chat ChatMessage: Completion:openllama-completion Edit: Functions:} PromptStrings:[] InputStrings:[] InputToken:[] functionCallString: functionCallNameString: FunctionsConfig:{DisableNoAction:false NoActionFunctionName: NoActionDescriptionName:} FeatureFlag:map[] LLMConfig:{SystemPrompt: TensorSplit: MainGPU: RMSNormEps:0 NGQA:0 PromptCachePath: PromptCacheAll:false PromptCacheRO:false MirostatETA:0 MirostatTAU:0 Mirostat:0 NGPULayers:4 MMap:false MMlock:false LowVRAM:false Grammar: StopWords:[] Cutstrings:[] TrimSpace:[] ContextSize:1024 NUMA:false LoraAdapter: LoraBase: NoMulMatQ:false DraftModel: NDraft:0 Quantization:} AutoGPTQ:{ModelBaseName: Device: Triton:false UseFastTokenizer:false} Diffusers:{PipelineType: SchedulerType: CUDA:false EnableParameters: CFGScale:0 IMG2IMG:false ClipSkip:0 ClipModel: ClipSubFolder:} Step:0 GRPC:{Attempts:0 AttemptsSleepTime:0} VallE:{AudioPath:}}
2023-09-27 06:35:37 11:35PM DBG Parameters: &{PredictionOptions:{Model:open-llama-3b-q4_0.bin Language: N:0 TopP:0.7 TopK:80 Temperature:0.9 Maxtokens:0 Echo:false Batch:0 F16:false IgnoreEOS:false RepeatPenalty:0 Keep:0 MirostatETA:0 MirostatTAU:0 Mirostat:0 FrequencyPenalty:0 TFZ:0 TypicalP:0 Seed:0 NegativePrompt: RopeFreqBase:0 RopeFreqScale:0 NegativePromptScale:0 UseFastTokenizer:false ClipSkip:0 Tokenizer:} Name:openllama F16:true Threads:32 Debug:true Roles:map[assistant:ASSISTANT: system:SYSTEM: user:USER:] Embeddings:false Backend:llama TemplateConfig:{Chat:openllama-chat ChatMessage: Completion:openllama-completion Edit: Functions:} PromptStrings:[] InputStrings:[] InputToken:[] functionCallString: functionCallNameString: FunctionsConfig:{DisableNoAction:false NoActionFunctionName: NoActionDescriptionName:} FeatureFlag:map[] LLMConfig:{SystemPrompt: TensorSplit: MainGPU: RMSNormEps:0 NGQA:0 PromptCachePath: PromptCacheAll:false PromptCacheRO:false MirostatETA:0 MirostatTAU:0 Mirostat:0 NGPULayers:4 MMap:false MMlock:false LowVRAM:false Grammar: StopWords:[] Cutstrings:[] TrimSpace:[] ContextSize:1024 NUMA:false LoraAdapter: LoraBase: NoMulMatQ:false DraftModel: NDraft:0 Quantization:} AutoGPTQ:{ModelBaseName: Device: Triton:false UseFastTokenizer:false} Diffusers:{PipelineType: SchedulerType: CUDA:false EnableParameters: CFGScale:0 IMG2IMG:false ClipSkip:0 ClipModel: ClipSubFolder:} Step:0 GRPC:{Attempts:0 AttemptsSleepTime:0} VallE:{AudioPath:}}
2023-09-27 06:35:37 11:35PM DBG Prompt (before templating): USER: How are you?
2023-09-27 06:35:37 11:35PM DBG Template found, input modified to: Q: USER: How are you?\nA:
2023-09-27 06:35:37
2023-09-27 06:35:37 11:35PM DBG Prompt (after templating): Q: USER: How are you?\nA:
2023-09-27 06:35:37
2023-09-27 06:35:37 11:35PM DBG Loading model llama from open-llama-3b-q4_0.bin
2023-09-27 06:35:37 11:35PM DBG Loading model in memory from file: /models/open-llama-3b-q4_0.bin
2023-09-27 06:35:37 11:35PM DBG Loading GRPC Model llama: {backendString:llama model:open-llama-3b-q4_0.bin threads:32 assetDir:/tmp/localai/backend_data context:{emptyCtx:{}} gRPCOptions:0xc000503ba0 externalBackends:map[autogptq:/build/extra/grpc/autogptq/autogptq.py bark:/build/extra/grpc/bark/ttsbark.py diffusers:/build/extra/grpc/diffusers/backend_diffusers.py exllama:/build/extra/grpc/exllama/exllama.py huggingface-embeddings:/build/extra/grpc/huggingface/huggingface.py vall-e-x:/build/extra/grpc/vall-e-x/ttsvalle.py vllm:/build/extra/grpc/vllm/backend_vllm.py] grpcAttempts:20 grpcAttemptsDelay:2 singleActiveBackend:false}
2023-09-27 06:35:37 11:35PM DBG Loading GRPC Process: /tmp/localai/backend_data/backend-assets/grpc/llama
2023-09-27 06:35:37 11:35PM DBG GRPC Service for open-llama-3b-q4_0.bin will be running at: '127.0.0.1:40367'
2023-09-27 06:35:37 11:35PM DBG GRPC Service state dir: /tmp/go-processmanager3853150060
2023-09-27 06:35:37 11:35PM DBG GRPC Service Started
2023-09-27 06:35:37 rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial tcp 127.0.0.1:40367: connect: connection refused"
2023-09-27 06:35:37 11:35PM DBG GRPC(open-llama-3b-q4_0.bin-127.0.0.1:40367): stderr 2023/09/26 23:35:37 gRPC Server listening at 127.0.0.1:40367
2023-09-27 06:35:39 11:35PM DBG GRPC Service Ready
2023-09-27 06:35:39 11:35PM DBG GRPC: Loading model with options: {state:{NoUnkeyedLiterals:{} DoNotCompare:[] DoNotCopy:[] atomicMessageInfo:
lunamidori5
commented
11 months ago @noblerboy2004 You have GPU layers set to 0, So 0% of your GPU will be used... Here is a fixed yaml for your easy copy and paste make sure to RESTART localai after changing a yaml file
backend: llama-stable
context_size: 1024
name: openllama
f16: true
gpu_layers: 30
parameters:
model: open-llama-3b-q4_0.bin
temperature: 0.2
top_k: 80
top_p: 0.7
template:
chat: openllama-chat
completion: openllama-completion
roles:
assistant: 'ASSISTANT:'
system: 'SYSTEM:'
user: 'USER:'backend: llama-stable
context_size: 1024
f16: true
gpu_layers: 30
name: gpt4all-j-groovy
parameters:
model: ggml-gpt4all-j-v1.3-groovy.bin
temperature: 0.2
top_k: 80
top_p: 0.7
template:
chat: gpt4all-chat
completion: gpt4all-completionYou will need to fix the formatting of the yaml files before restarting localai
lunamidori5
commented
11 months ago As a note, gpt4all is not fully supported at this time, and the open-llama model uses llama-stable not llama
If you would like more info on setting up a model
https://localai.io/howtos/ https://localai.io/howtos/easy-model-import-downloaded/ https://localai.io/advanced/
noblerboy2004
commented
11 months ago backend: llama-stable context_size: 1024 name: openllama f16: true gpu_layers: 30 parameters: model: open-llama-3b-q4_0.bin temperature: 0.2 top_k: 80 top_p: 0.7 template: chat: openllama-chat completion: openllama-completion roles: assistant: 'ASSISTANT:' system: 'SYSTEM:' user: 'USER:'
Hi luminadori5,
Thank you for your quick action. I did above guide. However, still failed with openllama with log:
2023-09-27 07:04:18 [127.0.0.1]:59474 200 - GET /readyz
2023-09-27 07:04:20 12:04AM DBG Request received:
2023-09-27 07:04:20 12:04AM DBG Configuration read: &{PredictionOptions:{Model:open-llama-3b-q4_0.bin Language: N:0 TopP:0.7 TopK:80 Temperature:0.9 Maxtokens:0 Echo:false Batch:0 F16:false IgnoreEOS:false RepeatPenalty:0 Keep:0 MirostatETA:0 MirostatTAU:0 Mirostat:0 FrequencyPenalty:0 TFZ:0 TypicalP:0 Seed:0 NegativePrompt: RopeFreqBase:0 RopeFreqScale:0 NegativePromptScale:0 UseFastTokenizer:false ClipSkip:0 Tokenizer:} Name:openllama F16:true Threads:32 Debug:true Roles:map[assistant:ASSISTANT: system:SYSTEM: user:USER:] Embeddings:false Backend:llama TemplateConfig:{Chat:openllama-chat ChatMessage: Completion:openllama-completion Edit: Functions:} PromptStrings:[] InputStrings:[] InputToken:[] functionCallString: functionCallNameString: FunctionsConfig:{DisableNoAction:false NoActionFunctionName: NoActionDescriptionName:} FeatureFlag:map[] LLMConfig:{SystemPrompt: TensorSplit: MainGPU: RMSNormEps:0 NGQA:0 PromptCachePath: PromptCacheAll:false PromptCacheRO:false MirostatETA:0 MirostatTAU:0 Mirostat:0 NGPULayers:1000 MMap:false MMlock:false LowVRAM:false Grammar: StopWords:[] Cutstrings:[] TrimSpace:[] ContextSize:1024 NUMA:false LoraAdapter: LoraBase: NoMulMatQ:false DraftModel: NDraft:0 Quantization:} AutoGPTQ:{ModelBaseName: Device: Triton:false UseFastTokenizer:false} Diffusers:{PipelineType: SchedulerType: CUDA:false EnableParameters: CFGScale:0 IMG2IMG:false ClipSkip:0 ClipModel: ClipSubFolder:} Step:0 GRPC:{Attempts:0 AttemptsSleepTime:0} VallE:{AudioPath:}}
2023-09-27 07:04:20 12:04AM DBG Parameters: &{PredictionOptions:{Model:open-llama-3b-q4_0.bin Language: N:0 TopP:0.7 TopK:80 Temperature:0.9 Maxtokens:0 Echo:false Batch:0 F16:false IgnoreEOS:false RepeatPenalty:0 Keep:0 MirostatETA:0 MirostatTAU:0 Mirostat:0 FrequencyPenalty:0 TFZ:0 TypicalP:0 Seed:0 NegativePrompt: RopeFreqBase:0 RopeFreqScale:0 NegativePromptScale:0 UseFastTokenizer:false ClipSkip:0 Tokenizer:} Name:openllama F16:true Threads:32 Debug:true Roles:map[assistant:ASSISTANT: system:SYSTEM: user:USER:] Embeddings:false Backend:llama TemplateConfig:{Chat:openllama-chat ChatMessage: Completion:openllama-completion Edit: Functions:} PromptStrings:[] InputStrings:[] InputToken:[] functionCallString: functionCallNameString: FunctionsConfig:{DisableNoAction:false NoActionFunctionName: NoActionDescriptionName:} FeatureFlag:map[] LLMConfig:{SystemPrompt: TensorSplit: MainGPU: RMSNormEps:0 NGQA:0 PromptCachePath: PromptCacheAll:false PromptCacheRO:false MirostatETA:0 MirostatTAU:0 Mirostat:0 NGPULayers:1000 MMap:false MMlock:false LowVRAM:false Grammar: StopWords:[] Cutstrings:[] TrimSpace:[] ContextSize:1024 NUMA:false LoraAdapter: LoraBase: NoMulMatQ:false DraftModel: NDraft:0 Quantization:} AutoGPTQ:{ModelBaseName: Device: Triton:false UseFastTokenizer:false} Diffusers:{PipelineType: SchedulerType: CUDA:false EnableParameters: CFGScale:0 IMG2IMG:false ClipSkip:0 ClipModel: ClipSubFolder:} Step:0 GRPC:{Attempts:0 AttemptsSleepTime:0} VallE:{AudioPath:}}
2023-09-27 07:04:20 12:04AM DBG Prompt (before templating): USER: How are you?
2023-09-27 07:04:20 12:04AM DBG Template found, input modified to: Q: USER: How are you?\nA:
2023-09-27 07:04:20
2023-09-27 07:04:20 12:04AM DBG Prompt (after templating): Q: USER: How are you?\nA:
2023-09-27 07:04:20
2023-09-27 07:04:20 12:04AM DBG Loading model llama from open-llama-3b-q4_0.bin
2023-09-27 07:04:20 12:04AM DBG Loading model in memory from file: /models/open-llama-3b-q4_0.bin
2023-09-27 07:04:20 12:04AM DBG Loading GRPC Model llama: {backendString:llama model:open-llama-3b-q4_0.bin threads:32 assetDir:/tmp/localai/backend_data context:{emptyCtx:{}} gRPCOptions:0xc000681040 externalBackends:map[autogptq:/build/extra/grpc/autogptq/autogptq.py bark:/build/extra/grpc/bark/ttsbark.py diffusers:/build/extra/grpc/diffusers/backend_diffusers.py exllama:/build/extra/grpc/exllama/exllama.py huggingface-embeddings:/build/extra/grpc/huggingface/huggingface.py vall-e-x:/build/extra/grpc/vall-e-x/ttsvalle.py vllm:/build/extra/grpc/vllm/backend_vllm.py] grpcAttempts:20 grpcAttemptsDelay:2 singleActiveBackend:false}
2023-09-27 07:04:20 12:04AM DBG Loading GRPC Process: /tmp/localai/backend_data/backend-assets/grpc/llama
2023-09-27 07:04:20 12:04AM DBG GRPC Service for open-llama-3b-q4_0.bin will be running at: '127.0.0.1:34155'
2023-09-27 07:04:20 12:04AM DBG GRPC Service state dir: /tmp/go-processmanager1578684964
2023-09-27 07:04:20 12:04AM DBG GRPC Service Started
2023-09-27 07:04:20 rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial tcp 127.0.0.1:34155: connect: connection refused"
2023-09-27 07:04:21 12:04AM DBG GRPC(open-llama-3b-q4_0.bin-127.0.0.1:34155): stderr 2023/09/27 00:04:21 gRPC Server listening at 127.0.0.1:34155
2023-09-27 07:04:22 12:04AM DBG GRPC Service Ready
2023-09-27 07:04:22 12:04AM DBG GRPC: Loading model with options: {state:{NoUnkeyedLiterals:{} DoNotCompare:[] DoNotCopy:[] atomicMessageInfo:
For gpt4all-j-groovy, when changing backend to llama-stable, get similar problem above.
THank you.
noblerboy2004
commented
11 months ago When i tried to set gpu-layer for gpt4all-j-groovy with gpt4all backend.
LocalAi use CPU instead of GPU.
noblerboy2004
commented
11 months ago I am new to this project, too. It looks like you need to set up gpu_layer in the config somewhere, but I don't know how.
Hi Lunamidori5,yhyu13,
I tried with the following steps below and working now for lunademo. CUDA Not working (Note: when docker compose again, all previous install will be erased)
Follow the link: https://localai.io/howtos/easy-model-import-downloaded/
Download model: https://huggingface.co/TheBloke/Luna-AI-Llama2-Uncensored-GGML/blob/main/luna-ai-llama2-uncensored.ggmlv3.q5_K_M.bin
Create 3 file In the "lunademo-chat.tmpl" file add {{.Input}} ASSISTANT: In the "lunademo-completion.tmpl" file add Complete the following sentence: {{.Input}} In the "lunademo.yaml" file (If you want to see advanced yaml configs - Link) backend: llama-stable context_size: 2000 f16: true ## If you are using cpu set this to false gpu_layers: 30 batch: 512 name: lunademo parameters: model: luna-ai-llama2-uncensored.ggmlv3.q5_K_M.bin temperature: 0.2 top_k: 40 top_p: 0.65 roles: assistant: 'ASSISTANT:' system: 'SYSTEM:' user: 'USER:' template: chat: lunademo-chat completion: lunademo-completion
Edit .env file: Notice to remove -DLLAMA_AVX=OFF out of the string CMAKE. Because our CPU support AVX (not support AVX2, AVX512)
CMAKE_ARGS="-DLLAMA_F16C=OFF -DLLAMA_AVX512=OFF -DLLAMA_AVX2=OFF -DLLAMA_FMA=OFF -DLLAMA_F16C=OFF" CUDA_VISIBLE_DEVICES=0-1 CUDA_DEVICE_POOL_GPU_OVERRIDE=1
THREADS=32
#
/models/availableGALLERIES=[{"name":"model-gallery", "url":"github:go-skynet/model-gallery/index.yaml"}, {"url": "github:go-skynet/model-gallery/huggingface.yaml","name":"huggingface"}]
# MODELS_PATH=/models
DEBUG=true
BUILD_TYPE=cublas
REBUILD=true
#
Edit dockercompose file version: '3.6'
services: api: deploy: resources: reservations: devices:
Hope that helpfull.
Thank you and have a nice day.
qingfenghcy
commented
5 months ago Hello, I also had a problem when using gpu version. Have you solved your problem?
longunmin
commented
3 months ago I am having the same issue, despite having gpu set in docker-compose and setting gpu_layers in the yaml. Could it be a docker issue?
noblerboy2004
commented
2 weeks ago Hello, I also had a problem when using gpu version. Have you solved your problem?
yes. The comment above show the way fixing my problem.
LocalAI version:
1.30.0 Latest.
Environment, CPU architecture, OS, and Version:
Window server 2022. Xeon E5 2670v2. GPU Geforece GTX 1070 Describe the bug
LocalAI using CPU instead of GPU. CUDA remains 0% When calling chat completeion.
To Reproduce
Expected behavior
Logs
Additional context
Configure in env:
Docker Compose: version: '3.6'
services: api: deploy: resources: reservations: devices: