search

k8sgpt-ai / k8sgpt

Giving Kubernetes Superpowers to everyone
http://k8sgpt.ai
Apache License 2.0
5.69k stars 654 forks source link

[Question]: "An empty namespace may not be set during creation" error #1107

Closed electrocucaracha closed 4 months ago

electrocucaracha commented 5 months ago

Checklist

Affected Components

K8sGPT Version

v0.3.30

Kubernetes Version

v1.28.6

Host OS and its Version

Ubuntu 18.04.6 LTS

Steps to reproduce

  1. Deploy LocalAI services
  2. Deploy the K8sGPT Operator v1.4
  3. Wait for Operator readiness 
    $ kubectl logs -n k8sgpt-operator-system k8sgpt-operator-controller-manager-695d646b5d-bffth -f
2024-05-10T00:54:35Z    INFO    controller-runtime.metrics      Metrics server is starting to listen    {"addr": "127.0.0.1:8080"}
2024-05-10T00:54:35Z    INFO    setup   starting manager                                                                                                     
2024-05-10T00:54:35Z    INFO    Starting server {"kind": "health probe", "addr": "[::]:8081"}       
2024-05-10T00:54:35Z    INFO    starting server {"path": "/metrics", "kind": "metrics", "addr": "127.0.0.1:8080"}
I0510 00:54:35.546330       1 leaderelection.go:250] attempting to acquire leader lease k8sgpt-operator-system/ea9c19f7.k8sgpt.ai...                         
I0510 00:54:35.595158       1 leaderelection.go:260] successfully acquired lease k8sgpt-operator-system/ea9c19f7.k8sgpt.ai
2024-05-10T00:54:35Z    INFO    Starting EventSource    {"controller": "k8sgpt", "controllerGroup": "core.k8sgpt.ai", "controllerKind": "K8sGPT", "source": "kind source: *v1alpha1.K8sGPT"}                                                                                                                               
2024-05-10T00:54:35Z    DEBUG   events  k8sgpt-operator-controller-manager-695d646b5d-bffth_a11702aa-4a27-4657-9cdd-b33168857e8b became leader  {"type": "Normal", "object": {"kind":"Lease","namespace":"k8sgpt-operator-system","name":"ea9c19f7.k8sgpt.ai","uid":"811341dd-5fe9-48fd-b744-5a4fe6245690","apiVersion":"co
ordination.k8s.io/v1","resourceVersion":"26610"}, "reason": "LeaderElection"}                                                                                                                                                                                                                                              
2024-05-10T00:54:35Z    INFO    Starting Controller     {"controller": "k8sgpt", "controllerGroup": "core.k8sgpt.ai", "controllerKind": "K8sGPT"}                                                                                                                                                                          
2024-05-10T00:54:35Z    INFO    Starting workers        {"controller": "k8sgpt", "controllerGroup": "core.k8sgpt.ai", "controllerKind": "K8sGPT", "worker count": 1}
  4. Deploy the following K8sGPT CR: 
    apiVersion: core.k8sgpt.ai/v1alpha1
kind: K8sGPT
metadata:
name: k8sgpt-local
namespace: k8sgpt-operator-system
spec:
ai:
model: gpt-4
backend: localai
baseUrl: http://local-ai.local-ai-system.svc.cluster.local:8080/v1
enabled: true
version: v0.3.30
noCache: false
  5. Wait for CR readiness checking operator's logs: 
    Finished Reconciling k8sGPT                                                                                                                                  
Finished Reconciling k8sGPT                                                                                                                                  
Creating new client for 10.233.1.108:8080                                                                                                                    
Connection established between 10.233.1.108:8080 and localhost with time out of 1 seconds.          
Remote Address : 10.233.1.108:8080                                                                                                                           
K8sGPT address: 10.233.1.108:8080
  6. Deploy a broken pod: 
    apiVersion: v1
kind: Pod
metadata:
name: broken-pod
spec:
containers:
- name: broken-pod
  image: nginx:1.a.b.c
  livenessProbe:
    httpGet:
      path: /
      port: 90
    initialDelaySeconds: 3
    periodSeconds: 3
  7. Getting error in Operator's logs: 
    Finished Reconciling k8sGPT with error: an empty namespace may not be set during creation                                                                                                                                                                                                                                  
2024-05-10T00:59:16Z    ERROR   Reconciler error        {"controller": "k8sgpt", "controllerGroup": "core.k8sgpt.ai", "controllerKind": "K8sGPT", "K8sGPT": {"name":"k8sgpt-local","namespace":"k8sgpt-operator-system"}, "namespace": "k8sgpt-operator-system", "name": "k8sgpt-local", "reconcileID": "4f9ace39-1e48-4211
-8a56-e7f78a37bebb", "error": "an empty namespace may not be set during creation"}                                                                                                                                                                                                                                         
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler                                                                        
    /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.15.0/pkg/internal/controller/controller.go:324
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem            
    /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.15.0/pkg/internal/controller/controller.go:265
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2                  
    /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.15.0/pkg/internal/controller/controller.go:226
Creating new client for 10.233.1.108:8080                                                                                                                    
Connection established between 10.233.1.108:8080 and localhost with time out of 1 seconds.                                                                   
Remote Address : 10.233.1.108:8080                                                                                                                           
K8sGPT address: 10.233.1.108:8080                                                                                                                            
Finished Reconciling k8sGPT with error: an empty namespace may not be set during creation

Expected behaviour

I expect some results resources

Actual behaviour

$ kubectl get results.core.k8sgpt.ai -A
No resources found

Additional Information

No response

calebyyy commented 5 months ago

Hi I am getting this issue too did u manage to resolve it?

JuHyung-Son commented 4 months ago

would you re install k8sgpt and operator?

electrocucaracha commented 4 months ago

@JuHyung-Son I don't mind giving it a try but could you explain what was changed on the release version that I tested?

electrocucaracha commented 4 months ago

@JuHyung-Son I suspect that this error could be caused by PodAnalyzer sending an empty namespace value to the Analyzer

Nitishmane commented 4 months ago

I am seeing a similar issue. However, providing a 'targetNamespace' generates results for only that particular namespace.

electrocucaracha commented 4 months ago

I am seeing a similar issue. However, providing a 'targetNamespace' generates results for only that particular namespace.

It seems like that is the missing part, thanks @Nitishmane for the solution.

This is the Operator CR:

apiVersion: core.k8sgpt.ai/v1alpha1
kind: K8sGPT
metadata:
  name: k8sgpt-local
  namespace: k8sgpt-operator-system
spec:
  ai:
    model: gpt-4
    backend: localai
    baseUrl: http://local-ai.local-ai-system.svc.cluster.local:8080/v1
    enabled: true
  version: v0.3.30
  noCache: false
  targetNamespace: default

and this is the pod broken example

apiVersion: v1
kind: Pod
metadata:
  name: broken-pod
  namespace: default
spec:
  containers:
    - name: broken-pod
      image: nginx:1.a.b.c
      livenessProbe:
        httpGet:
          path: /
          port: 90
        initialDelaySeconds: 3
        periodSeconds: 3

Which generates the following result:

$ kubectl describe results.core.k8sgpt.ai 
Name:         defaultbrokenpod
Namespace:    default
Labels:       k8sgpts.k8sgpt.ai/backend=localai
              k8sgpts.k8sgpt.ai/name=k8sgpt-local
              k8sgpts.k8sgpt.ai/namespace=k8sgpt-operator-system
Annotations:  <none>
API Version:  core.k8sgpt.ai/v1alpha1
Kind:         Result
Metadata:
  Creation Timestamp:  2024-05-15T20:58:20Z
  Generation:          1
  Resource Version:    3550
  UID:                 d34c1a0b-ee29-477e-b721-0f29bef9d7e7
Spec:
  Backend:  localai
  Details:  Error: Back-off pulling image "nginx:1.a.b.c" indicates that Kubernetes is unable to pull the specified Docker image due to network issues or registry unavailability.
Solution: 
1. Check the internet connection: Ensure that the Kubernetes cluster has stable network access to the Docker registry hosting the image.
2. Verify image URL: Double-check the image name and tag "nginx:1.a.b.c" for any typos or missing information.
3. Pull image manually: Run the command "docker pull nginx:1.a.b.c" outside of Kubernetes to ensure that the image can be pulled successfully.
4. Check image availability: Confirm that the specified image version "1.a.b.c" is available on the registry.
5. Update image pull policy: If the image is available, consider updating the image pull policy in the pod's specification to "IfNotPresent" or "Always" to force Kubernetes to pull the image.
  Error:
    Text:         Back-off pulling image "nginx:1.a.b.c"
  Kind:           Pod
  Name:           default/broken-pod
  Parent Object:  
Events:           <none>
g3david405 commented 4 months ago

electrocucaracha so k8sgpt operator can analyze for only one namespace now?

electrocucaracha commented 4 months ago

electrocucaracha so k8sgpt operator can analyze for only one namespace now?

I'm not sure if the operator was designed to work in that way, but it looks like that's the intention.

MartyB-007 commented 4 months ago

I'm also seeing the same issue on v0.3.32, it doesn't feel like the intent is to force a target namespace, otherwise we'd have to run one K8sGPT per namespace on the cluster to get a clear view in the results crd.. seems like a maintenance nightmare. Unless I'm missing it the documentation has never alluded to this need

JuHyung-Son commented 4 months ago

i guess this is bug, i will take a look

g3david405 commented 4 months ago

@JuHyung-Son thank you!

JuHyung-Son commented 4 months ago

this will be fixed soon. https://github.com/k8sgpt-ai/k8sgpt-operator/pull/449

JuHyung-Son commented 4 months ago

it's done. Test with a new version https://github.com/k8sgpt-ai/k8sgpt-operator/pull/449

MartyB-007 commented 4 months ago

Hey @JuHyung-Son thanks for the fast reply and fast fix! I just tested and it is working now. Thank you :)