Closed AlexsJones closed 1 year ago
Hi, i'm experiencing the same - working also with latest version , installed with the helm chart. i can see the requests from k8sgpt, and local-ai response in local-ai log, but for some reason the results comes back empty:
DBG Prompt (before templating): Simplify the following Kubernetes error message delimited by triple dashes written in --- english --- language; --- Deployment WTUzQHs3JQ==/ezpCQDtvdWhk has 1 replicas but 2 are available ---. Error: {Explain error here} Simplify the following Kubernetes error message delimited by triple dashes written in --- english --- language; --- Deployment WTUzQHs3JQ==/ezpCQDtvdWhk has 1 replicas but 2 are available ---. Error: {Explain error here} Simplify the following Kubernetes error message delimited by triple dashes written in --- english --- language; --- Deployment WTUzQHs3JQ==/ezpCQDtvdWhk has 1 replicas but 2 are available ---. Error: {Explain error here} 10:42AM DBG Response: {"object":"chat.completion","model":"ggml-gpt4all-j","choices":[{"index":0,"finish_reason":"stop","message":{"role":"assistant","content":": #\nThe error message \"Deploymment has 1 replica but 2 are available\" indicates that there is a mismatch between the number of replicas and available replica instances. To resolve this, you can try the following steps:\n1. Check if there are any pending or failed deployments in the current namespace.\n2. Check if there are any replicas that have not been deployed or updated.\n3. Check if there are any replicas that have been deleted or moved to another namespace.\n4. Check if there are any replicas that have not been created or added to the deployment.\n5. If there are any pending or failed deployments, delete them and try deploying the deployment again.\n6. If there are any replicas that have not been deployed or updated, create a new replica and update it with the latest data.\n7. If there are any replicas that have been deleted or moved to another namespace, move them back and update the deployment.\n8. If there are any replicas that have not been created or added to the deployment, create a new replica and add it to the deployment.\n9. If there are any issues with the deployment, check if all the required resources are available and try again.\n10. If there are any issues with the deployment, check if all the required resources are available and try again.\n11. If there are any issues with the deployment, check if all the required resources are available and try again.\n12. If there are any issues with the deployment, check if all the required resources are available and try again.\n13. If there are any issues with the deployment, check if all the required resources are available and try again.\n14. If there are any issues with the deployment, check if all the required resources are available and try again.\n15. If there are any issues with the deployment, check if all the required resources are available and try again.\n16. If there are any issues with the deployment, check if all the required resources are available and try again.\n17. If there are any issues with the deployment, check if all the required resources are available and try again.\n18. If there are any issues with the deployment, check if all the required resources are available and try again.\n19. If there are any issues with the deployment, check if all the required resources are available and try again.\n20. If there are any issues with the deployment, check if all the required resources"}}],"usage":{"prompt_tokens":0,"completion_tokens":0,"total_tokens":0}}
and when i run the results command:
kubectl get results -o json | jq .
i get empty results:
{ "apiVersion": "v1", "items": [], "kind": "List", "metadata": { "resourceVersion": "" } }
This doesn't look an issue with LocalAI
by itself - but rather model configuration and the model being used.
The actual reply from the LLM is there:
local-ai-966dbbfff-khf5v local-ai 11:01AM DBG Response: {"object":"chat.completion","model":"ggml-gpt4all-j","choices":[{"index":0,"finish_reason":"stop","message":{"role":"assistant","content":", which should be updated with new data on a regular basis.\nAs an AI language model, I do not have access to the specific details of your project or environment. However, in general terms:\n1) You should consider the following options for creating and managing your models:\n- Creating a separate repository for each model, which could be managed independently and updated on a regular basis.\n- Creating an API endpoint that exposes your models as a RESTful interface, which could be accessed by clients that need to download or update the models.\n- Using a database to store your model data, which could be updated with new models on a regular basis.\n- You could also consider using a cloud storage solution, such as AWS S3 or Google Cloud Storage to store and share your models with other applications or services.\n2) It's also important to consider the following factors when choosing a cloud provider:\n- Security and compliance requirements (e.g., HIPAA, GDPR)\n- Scalability and performance capabilities (e.g., availability, pricing)\n- Integration and collaboration capabilities (e.g., APIs, webhooks)\n- Disaster recovery and backup solutions (e.g., AWS Backup, Google Cloud Storage)\n- Support and documentation resources (e.g., AWS Knowledge Center, Google Cloud Platform documentation)\n- Cost and usage limits (e.g., cost-effective pricing, usage limits)\nUltimately, the best solution will depend on your specific project requirements and constraints."}}],"usage":{"prompt_tokens":0,"completion_tokens":0,"total_tokens":0}}
@AlexsJones did you changed k8sgpt lately to ask to return data in a specific format from the LLM? It's the first time I see this:
local-ai-966dbbfff-khf5v local-ai Simplify the following Kubernetes error message delimited by triple dashes written in --- english --- language; --- Service has no endpoints, expected label app.kubernetes.io/component=webhook Service has no endpoints, expected label app.kubernetes.io/instance=cert-manager Service has no endpoints, expected label app.kubernetes.io/name=webhook ---.
local-ai-966dbbfff-khf5v local-ai Provide the most possible solution in a step by step style in no more than 280 characters. Write the output in the following format:
local-ai-966dbbfff-khf5v local-ai Error: {Explain error here}
local-ai-966dbbfff-khf5v local-ai Solution: {Step by step solution here}
A note: this probably will break most of the small LLM models, which are not really good at formatting output
We did evolve to structure the response which works well on all the hosted API. To eliminate that I have removed it locally and am using this:
simple_prompt = "Simplify the following Kubernetes error message written in --- %s --- language; --- %s ---. Provide the most possible solution in a step by step style in no more than 280 characters"
I can see localai is generating responses in the pod, but it seems to not come back to my request
In the pod for localai
local-ai-966dbbfff-khf5v local-ai 12:17PM DBG Request received: {"model":"ggml-gpt4all-j","language":"","n":0,"top_p":0,"top_k":0,"temperature":0,"max_tokens":0,"echo":false,"batch":0,"f16":false,"ignore_eos":false,"repeat_penalty":0,"n_keep":0,"mirostat_eta":0,"mirostat_tau":0,"mirostat":0,"frequency_penalty":0,"tfz":0,"typical_p":0,"seed":0,"file":"","response_format":"","size":"","prompt":null,"instruction":"","input":null,"stop":null,"messages":[{"role":"user","content":"Simplify the following Kubernetes error message written in --- english --- language; --- Service has no endpoints, expected label app.kubernetes.io/instance=cert-manager Service has no endpoints, expected label app.kubernetes.io/name=webhook Service has no endpoints, expected label app.kubernetes.io/component=webhook ---. Provide the most possible solution in a step by step style in no more than 280 characters"}],"functions":null,"function_call":null,"stream":false,"mode":0,"step":0,"grammar":"","grammar_json_functions":null}
local-ai-966dbbfff-khf5v local-ai 12:17PM DBG Configuration read: &{PredictionOptions:{Model:ggml-gpt4all-j Language: N:0 TopP:0.7 TopK:80 Temperature:0.9 Maxtokens:512 Echo:false Batch:0 F16:false IgnoreEOS:false RepeatPenalty:0 Keep:0 MirostatETA:0 MirostatTAU:0 Mirostat:0 FrequencyPenalty:0 TFZ:0 TypicalP:0 Seed:0} Name: StopWords:[] Cutstrings:[] TrimSpace:[] ContextSize:512 F16:false NUMA:false Threads:4 Debug:true Roles:map[] Embeddings:false Backend: TemplateConfig:{Chat: ChatMessage: Completion: Edit: Functions:} MirostatETA:0 MirostatTAU:0 Mirostat:0 NGPULayers:0 MMap:false MMlock:false LowVRAM:false TensorSplit: MainGPU: ImageGenerationAssets: PromptCachePath: PromptCacheAll:false PromptCacheRO:false Grammar: PromptStrings:[] InputStrings:[] InputToken:[] functionCallString: functionCallNameString: FunctionsConfig:{DisableNoAction:false NoActionFunctionName: NoActionDescriptionName:} SystemPrompt:}
local-ai-966dbbfff-khf5v local-ai 12:17PM DBG Parameters: &{PredictionOptions:{Model:ggml-gpt4all-j Language: N:0 TopP:0.7 TopK:80 Temperature:0.9 Maxtokens:512 Echo:false Batch:0 F16:false IgnoreEOS:false RepeatPenalty:0 Keep:0 MirostatETA:0 MirostatTAU:0 Mirostat:0 FrequencyPenalty:0 TFZ:0 TypicalP:0 Seed:0} Name: StopWords:[] Cutstrings:[] TrimSpace:[] ContextSize:512 F16:false NUMA:false Threads:4 Debug:true Roles:map[] Embeddings:false Backend: TemplateConfig:{Chat: ChatMessage: Completion: Edit: Functions:} MirostatETA:0 MirostatTAU:0 Mirostat:0 NGPULayers:0 MMap:false MMlock:false LowVRAM:false TensorSplit: MainGPU: ImageGenerationAssets: PromptCachePath: PromptCacheAll:false PromptCacheRO:false Grammar: PromptStrings:[] InputStrings:[] InputToken:[] functionCallString: functionCallNameString: FunctionsConfig:{DisableNoAction:false NoActionFunctionName: NoActionDescriptionName:} SystemPrompt:}
local-ai-966dbbfff-khf5v local-ai 12:17PM DBG Prompt (before templating): Simplify the following Kubernetes error message written in --- english --- language; --- Service has no endpoints, expected label app.kubernetes.io/instance=cert-manager Service has no endpoints, expected label app.kubernetes.io/name=webhook Service has no endpoints, expected label app.kubernetes.io/component=webhook ---. Provide the most possible solution in a step by step style in no more than 280 characters
local-ai-966dbbfff-khf5v local-ai 12:17PM DBG Template found, input modified to: The prompt below is a question to answer, a task to complete, or a conversation to respond to; decide which and write an appropriate response.
local-ai-966dbbfff-khf5v local-ai ### Prompt:
local-ai-966dbbfff-khf5v local-ai Simplify the following Kubernetes error message written in --- english --- language; --- Service has no endpoints, expected label app.kubernetes.io/instance=cert-manager Service has no endpoints, expected label app.kubernetes.io/name=webhook Service has no endpoints, expected label app.kubernetes.io/component=webhook ---. Provide the most possible solution in a step by step style in no more than 280 characters
local-ai-966dbbfff-khf5v local-ai ### Response:# Models to download at runtime
local-ai-966dbbfff-khf5v local-ai 12:17PM DBG Prompt (after templating): The prompt below is a question to answer, a task to complete, or a conversation to respond to; decide which and write an appropriate response.
local-ai-966dbbfff-khf5v local-ai ### Prompt:
local-ai-966dbbfff-khf5v local-ai Simplify the following Kubernetes error message written in --- english --- language; --- Service has no endpoints, expected label app.kubernetes.io/instance=cert-manager Service has no endpoints, expected label app.kubernetes.io/name=webhook Service has no endpoints, expected label app.kubernetes.io/component=webhook ---. Provide the most possible solution in a step by step style in no more than 280 characters
local-ai-966dbbfff-khf5v local-ai ### Response:# Models to download at runtime
local-ai-966dbbfff-khf5v local-ai 12:17PM DBG Model already loaded in memory: ggml-gpt4all-j
local-ai-966dbbfff-khf5v local-ai 12:17PM DBG Model 'ggml-gpt4all-j' already loaded
local-ai-966dbbfff-khf5v local-ai 12:18PM DBG Response: {"object":"chat.completion","model":"ggml-gpt4all-j","choices":[{"index":0,"finish_reason":"stop","message":{"role":"assistant","content":"for Kuberentes endpoints\nTo solve the error message, you need to add endpoints for the Kubernetes service in your app.yaml file or specify the endpoints in your client configuration. \nYou can use the `--server` option to specify a Kubernetes API server URL and use it in your client configuration. \nYou can also use the `--server-config` option to specify a configuration file that contains the Kubernetes API server URL and other configuration options. \nIn addition, you need to make sure that the Kubernetes API server is running and accessible from your application."}}],"usage":{"prompt_tokens":0,"completion_tokens":0,"total_tokens":0}}
local-ai-966dbbfff-khf5v local-ai [127.0.0.1]:50378 200 - POST /v1/chat/completions
Locally k8sgpt ( via port forward )
~/Code/k8sgpt main ⇣1 go run ./main.go analyze --no-cache=true --explain -b localai
0% | | (0/4, 0 it/hr) [0s:0s]
Error: failed while calling AI provider localai: Post "http://localhost:8080/v1/chat/completions": EOF
exit status 1
I will try with curl also to validate
@mudler - can you verify the preload_models and promptTemplates config are correct?
preload_models: '[{ "url": "github:go-skynet/model-gallery/gpt4all-j.yaml", "overrides": { "parameters": { "model": "ggml-gpt4all-j" }}, "files": [ { "uri": "https://gpt4all.io/models/ggml-gpt4all-j.bin", "sha256": "acd54f6da1cad7c04c48b785178d686c720dcbe549903032a0945f97b1a43d20", "filename": "ggml-gpt4all-j" }]}]'
promptTemplates: ggml-gpt4all-j.tmpl: | The prompt below is a question to answer, a task to complete, or a conversation to respond to; decide which and write an appropriate response.
Prompt:
{{.Input}}
Response:# Models to download at runtime
Response:# Models to download at runtime
### Response:# Models to download at runtime
doesn't look right
I will try with curl also to validate
can you try curling from the localai pod? maybe the proxy is getting in between?
I will try with curl also to validate
can you try curling from the localai pod? maybe the proxy is getting in between?
root@local-ai-966dbbfff-khf5v:/build# curl http://localhost:8080/v1/completions -H "Content-Type: application/json" -d '{
"model": "ggml-gpt4all-j.bin",
"prompt": "A long time ago in a galaxy far, far away",
"temperature": 0.7
}'
{"object":"text_completion","model":"ggml-gpt4all-j.bin","choices":[{"index":0,"finish_reason":"stop","text":"…\nThere was an alien race known as the X-Men. They were a group of mutants who had the ability to control and manipulate their powers. They were often feared by humans, but they also had a strong sense of community and were known for their kind hearts."}],"usage":{"prompt_tokens":0,"completion_tokens":0,"total_tokens":0}}root@local-ai-966dbbfff-khf5v:/build#
root@local-ai-966dbbfff-khf5v:/build#
So this is good, we can curl inside the pod and it works - however, proxy or via k8sgpt externally we see it hangs. 🤔
I will try with curl also to validate
can you try curling from the localai pod? maybe the proxy is getting in between?
root@local-ai-966dbbfff-khf5v:/build# curl http://localhost:8080/v1/completions -H "Content-Type: application/json" -d '{ "model": "ggml-gpt4all-j.bin", "prompt": "A long time ago in a galaxy far, far away", "temperature": 0.7 }' {"object":"text_completion","model":"ggml-gpt4all-j.bin","choices":[{"index":0,"finish_reason":"stop","text":"…\nThere was an alien race known as the X-Men. They were a group of mutants who had the ability to control and manipulate their powers. They were often feared by humans, but they also had a strong sense of community and were known for their kind hearts."}],"usage":{"prompt_tokens":0,"completion_tokens":0,"total_tokens":0}}root@local-ai-966dbbfff-khf5v:/build# root@local-ai-966dbbfff-khf5v:/build#
So this is good, we can curl inside the pod and it works - however, proxy or via k8sgpt externally we see it hangs. thinking
when trying from outside, add stream: true
so you can see it live:
curl <endpoint>/v1/completions -H "Content-Type: application/json" -d '{
"model": "ggml-gpt4all-j.bin",
"stream": true,
"prompt": "A long time ago in a galaxy far, far away",
"temperature": 0.7
}'
In this way we can tell if it's just slow to respond to the full answer, or there are issues reaching out the service
Thanks I will give it a go in tomorrow This blocks me rn
│ local-ai 7:48PM ERR error: Get "https://raw.githubusercontent.com/go-skynet/model-gallery/ma │
│ in/gpt4all-j.yaml": dial tcp: lookup raw.githubusercontent.com: i/o timeout
``` 💀
LocalAI version: latest
Environment, CPU architecture, OS, and Version: amd64 thinkpad + kind
Describe the bug We can see localai receives the prompts buts fails to respond to the request
To Reproduce
k8sgpt auth add -b localai -m ggml-gpt4all-j -u http://localhost:8080/v1 -b localai
k8sgpt analyze --no-cache=true --explain -b localai
Expected behavior The expectation is the requests do not time out but return the inferenceAPI response.
Logs
Additional context