Closed ronaldpetty closed 7 months ago
The --explain
flag uses the AI provider configured by the user to run an analysis by communicating with the AI through the API keys. Localai doesn't use any API keys or AI and this is why you're facing this error. However, I think that the output could be improved to help the user understand the real problem here.
@ronaldpetty FYI
If model is proper working on
Here is the result... Even though it could not support in Korean :)
root@cp-k8s:~# k8sgpt analyze --explain -b localai -l Korean
100% |███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| (2/2, 52 it/hr)
AI Provider: localai
0 default/llama-2-7b-chat(llama-2-7b-chat)
- Error: Service has not ready endpoints, pods: [Pod/llama-2-7b-chat-55f89d58df-bwlzt], expected 1
Error: Service has not ready endpoints, pods: [Pod/llama-2-7b-chat-55f89d58df-bwlzt], expected 1.
Solution:
1. Check if the service is running by running `kubectl get svc llama-2-7b-chat`
2. If the service is not running, run `kubectl start sfllama-2-7b-chat` to start it.
3. Wait for the service to be ready by running `kubectl wait --for=service=llama-2-7b-chat-ready`
4. Once the service is ready, check the endpoints by running `kubectl get ep llama-2-7b-chat`
1 default/llama-2-7b-chat-55f89d58df-bwlzt(Deployment/llama-2-7b-chat)
- Error: failed to generate container "288eac52810fb2072ef983c70dd57bba05cdf181524a92ff51e5ca55d4da8084" spec: failed to generate spec: failed to stat "/var/lib/kubelet/pods/60c6876c-2a05-4499-92e8-5f326a9e046e/volumes/kubernetes.io~nfs/pvc-d82ae098-9e08-4dc7-90cf-09e4b9dd79cd": stat /var/lib/kubelet/pods/60c6876c-2a05-4499-92e8-5f326a9e046e/volumes/kubernetes.io~nfs/pvc-d82ae098-9e08-4dc7-90cf-09e4b9dd79cd: stale NFS file handle
Error: failed to stat "/var/lib/kubelet/pods/60c6876c-2a05-4499-92e8-5f326a9e046e/volumes/kubernetes.io~nfs/pvc-d82ae098-9e08-4dc7-90cf-09e4b9dd79cd": stale NFS file handle
Solution:
1. Check and update the Kubernetes version on your cluster.
2. Run `kubectl cleanup` to remove any stale state left over from previous runs.
3. Restart the kubelet to refresh the file handles.
4. Try running the `kubectl create` command again.
@sysnet4admin I am very new to this. Per @VaibhavMalik4187 it sounds like this would never work (unless local-ai started accepting tokens). Can you share how you set is done? I used the k8sgpt operator and localai.
@ronaldpetty I use just binary k8sgpt + ollama(https://github.com/ollama/ollama) Hopefully it would understand further.
You could leverage this one for loading models in local env easily. (text-generation-webui)
how did you set your localai
?
@ronaldpetty I use just binary k8sgpt + ollama(https://github.com/ollama/ollama) Hopefully it would understand further.
@sysnet4admin may you kindly share your backend configuration. I try to use ollama, but could not find any docs on how to set it up. Thanks a lot!
@JuHyung-Son @liyimeng https://yozm.wishket.com/magazine/detail/2516/ (writer is me @sysnet4admin )
Hopefully it will help to understand for setup & moving further. (unfortunately this article wrote in Korean, but content may understand due to command is in EN).
Thanks for instantly response! @sysnet4admin I will give it a shot!
Thank you @sysnet4admin . I think I am starting to understand based on your tutorial. I was using this tutorial originally:
https://itnext.io/k8sgpt-localai-unlock-kubernetes-superpowers-for-free-584790de9b65
I believe these issues were causing me trouble.
Once I updated the cluster service port, the errors went away. However, I am still confused on basic operations of k8sgpt.
~$ k8sgpt analyze
AI Provider: AI not used; --explain not set
0 default/broken-pod(broken-pod)
- Error: Back-off pulling image "nginx:1.a.b.c"
What does that even mean? (AI Provider not used?) Does that mean k8sgpt still does analyse even if there are no backends?
Another thing I noticed. If you have no k8sgpt resource, nothing happens (e.g. submit broken pod and never get a result). If you do have a k8sgpt, even with a broken baseURL (like I had), it still creates a result. Again, that is confusing, it implies no AI is needed. Maybe it says that somewhere but I must be missing it.
I think my issue title might need to be changed to local-ai and localai.
Final thing, I installed the operator via helm. The logs look as such:
...
Finished Reconciling k8sGPT
Creating new client for 10.43.58.77:8080
Connection established between 10.43.58.77:8080 and localhost with time out of 1 seconds.
Remote Address : 10.43.58.77:8080
K8sGPT address: 10.43.58.77:8080
Checking if defaultbrokenpod is still relevant
It would be helpful if the operator logged who it was talking to (maybe its not talking to anyone, but then I don't understand this configuration:
~$ cat k8sgpt-localai.yaml
apiVersion: core.k8sgpt.ai/v1alpha1
kind: K8sGPT
metadata:
name: k8sgpt-local-ai
namespace: default
spec:
ai:
enabled: true
model: ggml-gpt4all-j
backend: localai
baseUrl: http://local-ai.default.svc.cluster.local:80/v1
noCache: false
repository: ghcr.io/k8sgpt-ai/k8sgpt
version: v0.3.8
Here are the svc's I have.
~$ kubectl get svc
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
kubernetes ClusterIP 10.43.0.1 <none> 443/TCP 156m
local-ai ClusterIP 10.43.90.122 <none> 80/TCP 122m
k8sgpt-local-ai ClusterIP 10.43.58.77 <none> 8080/TCP 19m
I made some more progress. I got k8sgpt CLI tool working with OpenAI. I went back to LocalAI, and made a little more progress. I tried the k8sgpt CLI in a Pod, since I installed LocalAI there. It seems that I am misusing LocalAI.
root@testing:/# k8sgpt auth add --backend localai --model tinyllama --baseurl http://local-ai.default.svc.cluster.local:80/v1
localai added to the AI backend provider list
root@testing:/# k8sgpt analyze --explain -b localai --kubeconfig=.kube/config
0% | | (0/1, 0 it/hr) [0s:0s]
Error: failed while calling AI provider localai: error, status code: 500, message: could not load model - all backends returned error: 23 errors occurred:
* could not load model: rpc error: code = Canceled desc =
* could not load model: rpc error: code = Unknown desc = failed loading model
* could not load model: rpc error: code = Unknown desc = failed loading model
* could not load model: rpc error: code = Unknown desc = failed loading model
* could not load model: rpc error: code = Unavailable desc = error reading from server: EOF
* could not load model: rpc error: code = Unknown desc = stat /models/add: no such file or directory
* could not load model: rpc error: code = Unknown desc = stat /models/add: no such file or directory
* grpc process not found: /tmp/localai/backend_data/backend-assets/grpc/tinydream. some backends(stablediffusion, tts) require LocalAI compiled with GO_TAGS
* could not load model: rpc error: code = Unknown desc = unsupported model type /models/add (should end with .onnx)
* grpc process not found: /tmp/localai/backend_data/backend-assets/grpc/build/backend/python/transformers/run.sh. some backends(stablediffusion, tts) require LocalAI compiled with GO_TAGS
* grpc process not found: /tmp/localai/backend_data/backend-assets/grpc/build/backend/python/vllm/run.sh. some backends(stablediffusion, tts) require LocalAI compiled with GO_TAGS
* grpc process not found: /tmp/localai/backend_data/backend-assets/grpc/build/backend/python/transformers-musicgen/run.sh. some backends(stablediffusion, tts) require LocalAI compiled with GO_TAGS
* grpc process not found: /tmp/localai/backend_data/backend-assets/grpc/build/backend/python/sentencetransformers/run.sh. some backends(stablediffusion, tts) require LocalAI compiled with GO_TAGS
* grpc process not found: /tmp/localai/backend_data/backend-assets/grpc/build/backend/python/autogptq/run.sh. some backends(stablediffusion, tts) require LocalAI compiled with GO_TAGS
* grpc process not found: /tmp/localai/backend_data/backend-assets/grpc/build/backend/python/sentencetransformers/run.sh. some backends(stablediffusion, tts) require LocalAI compiled with GO_TAGS
* grpc process not found: /tmp/localai/backend_data/backend-assets/grpc/build/backend/python/coqui/run.sh. some backends(stablediffusion, tts) require LocalAI compiled with GO_TAGS
* grpc process not found: /tmp/localai/backend_data/backend-assets/grpc/build/backend/python/exllama2/run.sh. some backends(stablediffusion, tts) require LocalAI compiled with GO_TAGS
* grpc process not found: /tmp/localai/backend_data/backend-assets/grpc/build/backend/python/bark/run.sh. some backends(stablediffusion, tts) require LocalAI compiled with GO_TAGS
* grpc process not found: /tmp/localai/backend_data/backend-assets/grpc/build/backend/python/mamba/run.sh. some backends(stablediffusion, tts) require LocalAI compiled with GO_TAGS
* grpc process not found: /tmp/localai/backend_data/backend-assets/grpc/build/backend/python/diffusers/run.sh. some backends(stablediffusion, tts) require LocalAI compiled with GO_TAGS
* grpc process not found: /tmp/localai/backend_data/backend-assets/grpc/build/backend/python/vall-e-x/run.sh. some backends(stablediffusion, tts) require LocalAI compiled with GO_TAGS
* grpc process not found: /tmp/localai/backend_data/backend-assets/grpc/build/backend/python/petals/run.sh. some backends(stablediffusion, tts) require LocalAI compiled with GO_TAGS
* grpc process not found: /tmp/localai/backend_data/backend-assets/grpc/build/backend/python/exllama/run.sh. some backends(stablediffusion, tts) require LocalAI compiled with GO_TAGS
Below are the logs from LocalAI. I at least know its hitting it, but seems to be failing due to some configuration issue. I guess maybe will close this ticket and focus on LocalAI.
4:17AM INF Trying to load the model 'add' with all the available backends: llama-cpp, llama-ggml, gpt4all, bert-embeddings, rwkv, whisper, stablediffusion, tinydream, piper, /build/backend/python/transformers/run.sh, /build/backend/python/vllm/run.sh, /build/backend/python/transformers-musicgen/run.sh, /build/backend/python/sentencetransformers/run.sh, /build/backend/python/autogptq/run.sh, /build/backend/python/sentencetransformers/run.sh, /build/backend/python/coqui/run.sh, /build/backend/python/exllama2/run.sh, /build/backend/python/bark/run.sh, /build/backend/python/mamba/run.sh, /build/backend/python/diffusers/run.sh, /build/backend/python/vall-e-x/run.sh, /build/backend/python/petals/run.sh, /build/backend/python/exllama/run.sh
4:17AM INF [llama-cpp] Attempting to load
4:17AM INF Loading model 'add' with backend llama-cpp
4:17AM INF [llama-cpp] Fails: could not load model: rpc error: code = Canceled desc =
4:17AM INF [llama-ggml] Attempting to load
4:17AM INF Loading model 'add' with backend llama-ggml
4:17AM INF [llama-ggml] Fails: could not load model: rpc error: code = Unknown desc = failed loading model
4:17AM INF [gpt4all] Attempting to load
4:17AM INF Loading model 'add' with backend gpt4all
4:17AM INF [gpt4all] Fails: could not load model: rpc error: code = Unknown desc = failed loading model
4:17AM INF [bert-embeddings] Attempting to load
4:17AM INF Loading model 'add' with backend bert-embeddings
4:17AM INF [bert-embeddings] Fails: could not load model: rpc error: code = Unknown desc = failed loading model
4:17AM INF [rwkv] Attempting to load
4:17AM INF Loading model 'add' with backend rwkv
4:17AM INF [rwkv] Fails: could not load model: rpc error: code = Unavailable desc = error reading from server: EOF
4:17AM INF [whisper] Attempting to load
4:17AM INF Loading model 'add' with backend whisper
4:17AM INF [whisper] Fails: could not load model: rpc error: code = Unknown desc = stat /models/add: no such file or directory
4:17AM INF [stablediffusion] Attempting to load
4:17AM INF Loading model 'add' with backend stablediffusion
4:17AM INF [stablediffusion] Fails: could not load model: rpc error: code = Unknown desc = stat /models/add: no such file or directory
4:17AM INF [tinydream] Attempting to load
4:17AM INF Loading model 'add' with backend tinydream
4:17AM INF [tinydream] Fails: grpc process not found: /tmp/localai/backend_data/backend-assets/grpc/tinydream. some backends(stablediffusion, tts) require LocalAI compiled with GO_TAGS
4:17AM INF [piper] Attempting to load
4:17AM INF Loading model 'add' with backend piper
4:17AM INF [piper] Fails: could not load model: rpc error: code = Unknown desc = unsupported model type /models/add (should end with .onnx)
4:17AM INF [/build/backend/python/transformers/run.sh] Attempting to load
4:17AM INF Loading model 'add' with backend /build/backend/python/transformers/run.sh
4:17AM INF [/build/backend/python/transformers/run.sh] Fails: grpc process not found: /tmp/localai/backend_data/backend-assets/grpc/build/backend/python/transformers/run.sh. some backends(stablediffusion, tts) require LocalAI compiled with GO_TAGS
4:17AM INF [/build/backend/python/vllm/run.sh] Attempting to load
4:17AM INF Loading model 'add' with backend /build/backend/python/vllm/run.sh
4:17AM INF [/build/backend/python/vllm/run.sh] Fails: grpc process not found: /tmp/localai/backend_data/backend-assets/grpc/build/backend/python/vllm/run.sh. some backends(stablediffusion, tts) require LocalAI compiled with GO_TAGS
4:17AM INF [/build/backend/python/transformers-musicgen/run.sh] Attempting to load
4:17AM INF Loading model 'add' with backend /build/backend/python/transformers-musicgen/run.sh
4:17AM INF [/build/backend/python/transformers-musicgen/run.sh] Fails: grpc process not found: /tmp/localai/backend_data/backend-assets/grpc/build/backend/python/transformers-musicgen/run.sh. some backends(stablediffusion, tts) require LocalAI compiled with GO_TAGS
4:17AM INF [/build/backend/python/sentencetransformers/run.sh] Attempting to load
4:17AM INF Loading model 'add' with backend /build/backend/python/sentencetransformers/run.sh
4:17AM INF [/build/backend/python/sentencetransformers/run.sh] Fails: grpc process not found: /tmp/localai/backend_data/backend-assets/grpc/build/backend/python/sentencetransformers/run.sh. some backends(stablediffusion, tts) require LocalAI compiled with GO_TAGS
4:17AM INF [/build/backend/python/autogptq/run.sh] Attempting to load
4:17AM INF Loading model 'add' with backend /build/backend/python/autogptq/run.sh
4:17AM INF [/build/backend/python/autogptq/run.sh] Fails: grpc process not found: /tmp/localai/backend_data/backend-assets/grpc/build/backend/python/autogptq/run.sh. some backends(stablediffusion, tts) require LocalAI compiled with GO_TAGS
4:17AM INF [/build/backend/python/sentencetransformers/run.sh] Attempting to load
4:17AM INF Loading model 'add' with backend /build/backend/python/sentencetransformers/run.sh
4:17AM INF [/build/backend/python/sentencetransformers/run.sh] Fails: grpc process not found: /tmp/localai/backend_data/backend-assets/grpc/build/backend/python/sentencetransformers/run.sh. some backends(stablediffusion, tts) require LocalAI compiled with GO_TAGS
4:17AM INF [/build/backend/python/coqui/run.sh] Attempting to load
4:17AM INF Loading model 'add' with backend /build/backend/python/coqui/run.sh
4:17AM INF [/build/backend/python/coqui/run.sh] Fails: grpc process not found: /tmp/localai/backend_data/backend-assets/grpc/build/backend/python/coqui/run.sh. some backends(stablediffusion, tts) require LocalAI compiled with GO_TAGS
4:17AM INF [/build/backend/python/exllama2/run.sh] Attempting to load
4:17AM INF Loading model 'add' with backend /build/backend/python/exllama2/run.sh
4:17AM INF [/build/backend/python/exllama2/run.sh] Fails: grpc process not found: /tmp/localai/backend_data/backend-assets/grpc/build/backend/python/exllama2/run.sh. some backends(stablediffusion, tts) require LocalAI compiled with GO_TAGS
4:17AM INF [/build/backend/python/bark/run.sh] Attempting to load
4:17AM INF Loading model 'add' with backend /build/backend/python/bark/run.sh
4:17AM INF [/build/backend/python/bark/run.sh] Fails: grpc process not found: /tmp/localai/backend_data/backend-assets/grpc/build/backend/python/bark/run.sh. some backends(stablediffusion, tts) require LocalAI compiled with GO_TAGS
4:17AM INF [/build/backend/python/mamba/run.sh] Attempting to load
4:17AM INF Loading model 'add' with backend /build/backend/python/mamba/run.sh
4:17AM INF [/build/backend/python/mamba/run.sh] Fails: grpc process not found: /tmp/localai/backend_data/backend-assets/grpc/build/backend/python/mamba/run.sh. some backends(stablediffusion, tts) require LocalAI compiled with GO_TAGS
4:17AM INF [/build/backend/python/diffusers/run.sh] Attempting to load
4:17AM INF Loading model 'add' with backend /build/backend/python/diffusers/run.sh
4:17AM INF [/build/backend/python/diffusers/run.sh] Fails: grpc process not found: /tmp/localai/backend_data/backend-assets/grpc/build/backend/python/diffusers/run.sh. some backends(stablediffusion, tts) require LocalAI compiled with GO_TAGS
4:17AM INF [/build/backend/python/vall-e-x/run.sh] Attempting to load
4:17AM INF Loading model 'add' with backend /build/backend/python/vall-e-x/run.sh
4:17AM INF [/build/backend/python/vall-e-x/run.sh] Fails: grpc process not found: /tmp/localai/backend_data/backend-assets/grpc/build/backend/python/vall-e-x/run.sh. some backends(stablediffusion, tts) require LocalAI compiled with GO_TAGS
4:17AM INF [/build/backend/python/petals/run.sh] Attempting to load
4:17AM INF Loading model 'add' with backend /build/backend/python/petals/run.sh
4:17AM INF [/build/backend/python/petals/run.sh] Fails: grpc process not found: /tmp/localai/backend_data/backend-assets/grpc/build/backend/python/petals/run.sh. some backends(stablediffusion, tts) require LocalAI compiled with GO_TAGS
4:17AM INF [/build/backend/python/exllama/run.sh] Attempting to load
4:17AM INF Loading model 'add' with backend /build/backend/python/exllama/run.sh
4:17AM INF [/build/backend/python/exllama/run.sh] Fails: grpc process not found: /tmp/localai/backend_data/backend-assets/grpc/build/backend/python/exllama/run.sh. some backends(stablediffusion, tts) require LocalAI compiled with GO_TAGS
@ronaldpetty are you using vllm for your local ai test? Thanks
Hi @gyliu513 Its been a long time since this ticket. I am not using VLLM lately. I will try again and if I get that working will update.
@ronaldpetty no worries, I got a blog for k8sgpt and vllm https://medium.com/@panpan0000/empower-kubernetes-with-k8sgpt-using-open-source-llm-1b3fa021abd6 , thanks!
Checklist
Affected Components
K8sGPT Version
0.3.24
Kubernetes Version
v1.26.9
Host OS and its Version
Ubuntu 22.04
Steps to reproduce
Install k8sgpt-operator and localai (sorry can add steps, but I had so issues and its not trivial to do).
~$ k8sgpt analyse -b localai
works but adding--explain
has auth issue.Expected behaviour
I assume
--explain
gives more information (I am new here, so not certain, but didn't expect error).Actual behaviour
Additional Information
No response