search

opea-project / GenAIInfra

Containerization and cloud native suite for OPEA
Apache License 2.0
30 stars 58 forks source link

GMC manager panic during CI GMC e2e test #394

Closed lianhao closed 2 months ago

lianhao commented 2 months ago

Using the latest code(commit id: 59ce50506), during the GMC e2e CI test, I found the GMC manager restarted several times due to panic:

$ kubectl -n opea-system get pod
NAME                             READY   STATUS    RESTARTS        AGE
gmc-controller-f67965865-nz89w   1/1     Running   2 (3m36s ago)   7m13s
$ kubectl -n opea-system logs gmc-controller-f67965865-nz89w -p
2024-09-04T12:56:34Z    INFO    WebhookConfig   Creating or updating the validatingwebhookconfiguration {"webhook name": "validating-webhook-configuration"}
2024-09-04T12:56:34Z    INFO    WebhookConfig   Updated the validatingWebhookConfiguration      {"webhookConfigName": "validating-webhook-configuration"}
2024-09-04T12:56:34Z    INFO    controller-runtime.builder      skip registering a mutating webhook, object does not implement admission.Defaulter or WithDefaulter wasn't called       {"GVK": "gmc.opea.io/v1alpha3, Kind=GMConnector"}
2024-09-04T12:56:34Z    INFO    controller-runtime.builder      Registering a validating webhook        {"GVK": "gmc.opea.io/v1alpha3, Kind=GMConnector", "path": "/validate-gmc-opea-io-v1alpha3-gmconnector"}
2024-09-04T12:56:34Z    INFO    controller-runtime.webhook      Registering webhook     {"path": "/validate-gmc-opea-io-v1alpha3-gmconnector"}
2024-09-04T12:56:34Z    INFO    setup   starting manager
2024-09-04T12:56:34Z    INFO    controller-runtime.metrics      Starting metrics server
2024-09-04T12:56:34Z    INFO    starting server {"kind": "health probe", "addr": "[::]:8081"}
2024-09-04T12:56:34Z    INFO    controller-runtime.metrics      Serving metrics server  {"bindAddress": ":8080", "secure": false}
2024-09-04T12:56:34Z    INFO    controller-runtime.webhook      Starting webhook server
2024-09-04T12:56:34Z    INFO    controller-runtime.webhook      Serving webhook server  {"host": "", "port": 9443}
2024-09-04T12:56:34Z    INFO    Starting EventSource    {"controller": "gmconnector", "controllerGroup": "gmc.opea.io", "controllerKind": "GMConnector", "source": "kind source: *v1alpha3.GMConnector"}
2024-09-04T12:56:34Z    INFO    Starting EventSource    {"controller": "gmconnector", "controllerGroup": "gmc.opea.io", "controllerKind": "GMConnector", "source": "kind source: *v1.Deployment"}
2024-09-04T12:56:34Z    INFO    Starting Controller     {"controller": "gmconnector", "controllerGroup": "gmc.opea.io", "controllerKind": "GMConnector"}
2024-09-04T12:56:34Z    INFO    Starting workers        {"controller": "gmconnector", "controllerGroup": "gmc.opea.io", "controllerKind": "GMConnector", "worker count": 1}
-----------------Reconciling GMConnector opea-app-audioqa/asr-svc-deployment -------------------------
-----------------Reconciling GMConnector opea-app-audioqa/llm-svc-deployment -------------------------

| spec changed false | meta changed: false |
failed to get graph before update status audioqa <nil>
-----------------Reconciling GMConnector local-path-storage/local-path-provisioner -------------------------
-----------------Reconciling GMConnector opea-app-audioqa/router-service-deployment -------------------------
-----------------Reconciling GMConnector opea-app-audioqa/speecht5-svc-deployment -------------------------
-----------------Reconciling GMConnector opea-app-audioqa/tgi-svc-deployment -------------------------
-----------------Reconciling GMConnector opea-app-audioqa/tts-svc-deployment -------------------------
-----------------Reconciling GMConnector opea-app-audioqa/whisper-svc-deployment -------------------------
-----------------Reconciling GMConnector opea-system/gmc-controller -------------------------
-----------------Reconciling GMConnector kube-system/coredns -------------------------
-----------------Reconciling GMConnector opea-app-audioqa/audioqa -------------------------
Reconciling kind  GMConnector apiVersion  gmc.opea.io/v1alpha3  graph  audioqa

reconcile resource for node: Asr
trying to reconcile internal service [ asr-svc ] in namespace
get resource config: {Asr { {asr-svc  map[ASR_ENDPOINT:whisper-svc endpoint:/v1/audio/transcriptions] false} }    }
The raw yaml file has been split into 4 yaml files
Success to reconcile ConfigMap: asr-config
Success to reconcile Service: asr-svc
find downstream service for Asr with name whisper-svc
Success to reconcile Deployment: asr-svc-deployment
the service URL is: http://asr-svc.opea-app-audioqa.svc.cluster.local:9099/v1/audio/transcriptions

reconcile resource for node: Whisper
trying to reconcile internal service [ whisper-svc ] in namespace
get resource config: {Whisper { {whisper-svc  map[endpoint:/v1/asr] true} }    }
The raw yaml file has been split into 4 yaml files
Success to reconcile ConfigMap: whisper-config
Success to reconcile Service: whisper-svc
Success to reconcile Deployment: whisper-svc-deployment
the service URL is: http://whisper-svc.opea-app-audioqa.svc.cluster.local:7066/v1/asr

reconcile resource for node: Llm
trying to reconcile internal service [ llm-svc ] in namespace
get resource config: {Llm { {llm-svc  map[TGI_LLM_ENDPOINT:tgi-svc endpoint:/v1/chat/completions] false} } $response   }
The raw yaml file has been split into 4 yaml files
Success to reconcile ConfigMap: llm-uservice-config
Success to reconcile Service: llm-svc
find downstream service for Llm with name tgi-svc
Success to reconcile Deployment: llm-svc-deployment
the service URL is: http://llm-svc.opea-app-audioqa.svc.cluster.local:9000/v1/chat/completions

reconcile resource for node: Tgi
trying to reconcile internal service [ tgi-svc ] in namespace
get resource config: {Tgi { {tgi-svc  map[endpoint:/generate] true} }    }
The raw yaml file has been split into 6 yaml files
Success to reconcile ConfigMap: tgi-config
Success to reconcile Service: tgi-svc
Success to reconcile Deployment: tgi-svc-deployment
the service URL is: http://tgi-svc.opea-app-audioqa.svc.cluster.local:80/generate

reconcile resource for node: Tts
trying to reconcile internal service [ tts-svc ] in namespace
get resource config: {Tts { {tts-svc  map[TTS_ENDPOINT:speecht5-svc endpoint:/v1/audio/speech] false} } $response   }
The raw yaml file has been split into 4 yaml files
Success to reconcile ConfigMap: tts-config
Success to reconcile Service: tts-svc
find downstream service for Tts with name speecht5-svc
Success to reconcile Deployment: tts-svc-deployment
the service URL is: http://tts-svc.opea-app-audioqa.svc.cluster.local:9088/v1/audio/speech

reconcile resource for node: SpeechT5
trying to reconcile internal service [ speecht5-svc ] in namespace
get resource config: {SpeechT5 { {speecht5-svc  map[endpoint:/v1/tts] true} }    }
The raw yaml file has been split into 4 yaml files
Success to reconcile ConfigMap: speecht5-config
Success to reconcile Service: speecht5-svc
Success to reconcile Deployment: speecht5-svc-deployment
the service URL is: http://speecht5-svc.opea-app-audioqa.svc.cluster.local:7055/v1/tts

user config {opea-app-audioqa router-service router-service-deployment    '{"metadata":{"creationTimestamp":null},"spec":{"nodes":{"root":{"routerType":"Sequence","steps":[{"name":"Asr","internalService":{"serviceName":"asr-svc","config":{"ASR_ENDPOINT":"whisper-svc","endpoint":"/v1/audio/transcriptions"}},"serviceUrl":"http://asr-svc.opea-app-audioqa.svc.cluster.local:9099/v1/audio/transcriptions"},{"name":"Whisper","internalService":{"serviceName":"whisper-svc","config":{"endpoint":"/v1/asr"},"isDownstreamService":true},"serviceUrl":"http://whisper-svc.opea-app-audioqa.svc.cluster.local:7066/v1/asr"},{"name":"Llm","internalService":{"serviceName":"llm-svc","config":{"TGI_LLM_ENDPOINT":"tgi-svc","endpoint":"/v1/chat/completions"}},"data":"$response","serviceUrl":"http://llm-svc.opea-app-audioqa.svc.cluster.local:9000/v1/chat/completions"},{"name":"Tgi","internalService":{"serviceName":"tgi-svc","config":{"endpoint":"/generate"},"isDownstreamService":true},"serviceUrl":"http://tgi-svc.opea-app-audioqa.svc.cluster.local:80/generate"},{"name":"Tts","internalService":{"serviceName":"tts-svc","config":{"TTS_ENDPOINT":"speecht5-svc","endpoint":"/v1/audio/speech"}},"data":"$response","serviceUrl":"http://tts-svc.opea-app-audioqa.svc.cluster.local:9088/v1/audio/speech"},{"name":"SpeechT5","internalService":{"serviceName":"speecht5-svc","config":{"endpoint":"/v1/tts"},"isDownstreamService":true},"serviceUrl":"http://speecht5-svc.opea-app-audioqa.svc.cluster.local:7055/v1/tts"}]}},"routerConfig":{"name":"router","serviceName":"router-service","nameSpace":"","config":null}},"status":{"condition":{"lastUpdateTime":null}}}'}
Success to reconcile Deployment: router-service-deployment
Success to reconcile Service: router-service
the router URL is: http://router-service.opea-app-audioqa.svc.cluster.local:8080
-----------------Reconciling GMConnector opea-app-audioqa/client-test -------------------------
| opea-app-audioqa:tgi-svc-deployment: status changed from : False to True|
-----------------Reconciling GMConnector opea-app-audioqa/tgi-svc-deployment -------------------------

| spec changed false | meta changed: false |
-----------------Reconciling GMConnector opea-app-audioqa/audioqa -------------------------
resource audioqa not found or deleted, Deployment.apps "audioqa" not found, ignore
-----------------Reconciling GMConnector opea-app-audioqa/asr-svc-deployment -------------------------
resource asr-svc-deployment not found or deleted, Deployment.apps "asr-svc-deployment" not found, ignore
-----------------Reconciling GMConnector opea-app-audioqa/llm-svc-deployment -------------------------
resource llm-svc-deployment not found or deleted, Deployment.apps "llm-svc-deployment" not found, ignore
-----------------Reconciling GMConnector opea-app-audioqa/whisper-svc-deployment -------------------------
resource whisper-svc-deployment not found or deleted, Deployment.apps "whisper-svc-deployment" not found, ignore
-----------------Reconciling GMConnector opea-app-audioqa/tts-svc-deployment -------------------------
resource tts-svc-deployment not found or deleted, Deployment.apps "tts-svc-deployment" not found, ignore
-----------------Reconciling GMConnector opea-app-audioqa/router-service-deployment -------------------------
resource router-service-deployment not found or deleted, Deployment.apps "router-service-deployment" not found, ignore
-----------------Reconciling GMConnector opea-app-audioqa/speecht5-svc-deployment -------------------------
resource speecht5-svc-deployment not found or deleted, Deployment.apps "speecht5-svc-deployment" not found, ignore
-----------------Reconciling GMConnector opea-app-audioqa/tgi-svc-deployment -------------------------
resource tgi-svc-deployment not found or deleted, Deployment.apps "tgi-svc-deployment" not found, ignore
2024-09-04T12:59:21Z    INFO    validating-webhook      validate create {"name": "chatqa"}
-----------------Reconciling GMConnector opea-app-chatqna/chatqa -------------------------
Reconciling kind  GMConnector apiVersion  gmc.opea.io/v1alpha3  graph  chatqa

reconcile resource for node: Embedding
trying to reconcile internal service [ embedding-svc ] in namespace
get resource config: {Embedding { {embedding-svc  map[TEI_EMBEDDING_ENDPOINT:tei-embedding-svc endpoint:/v1/embeddings] false} }    }
The raw yaml file has been split into 4 yaml files
Success to reconcile ConfigMap: embedding-usvc-config
Success to reconcile Service: embedding-svc
find downstream service for Embedding with name tei-embedding-svc
Success to reconcile Deployment: embedding-svc-deployment
the service URL is: http://embedding-svc.opea-app-chatqna.svc.cluster.local:6000/v1/embeddings

reconcile resource for node: TeiEmbedding
trying to reconcile internal service [ tei-embedding-svc ] in namespace
get resource config: {TeiEmbedding { {tei-embedding-svc  map[] true} }    }
The raw yaml file has been split into 6 yaml files
Success to reconcile ConfigMap: tei-config
Success to reconcile Service: tei-embedding-svc
Success to reconcile Deployment: tei-embedding-svc-deployment
the service URL is: http://tei-embedding-svc.opea-app-chatqna.svc.cluster.local:80

reconcile resource for node: Retriever
trying to reconcile internal service [ retriever-svc ] in namespace
get resource config: {Retriever { {retriever-svc  map[REDIS_URL:redis-vector-db TEI_EMBEDDING_ENDPOINT:tei-embedding-svc endpoint:/v1/retrieval] false} } $response   }
The raw yaml file has been split into 4 yaml files
Success to reconcile ConfigMap: retriever-usvc-config
Success to reconcile Service: retriever-svc
find downstream service for Retriever with name tei-embedding-svc
find downstream service for Retriever with name redis-vector-db
Success to reconcile Deployment: retriever-svc-deployment
the service URL is: http://retriever-svc.opea-app-chatqna.svc.cluster.local:7000/v1/retrieval

reconcile resource for node: VectorDB
trying to reconcile internal service [ redis-vector-db ] in namespace
get resource config: {VectorDB { {redis-vector-db  map[] true} }    }
The raw yaml file has been split into 3 yaml files
Success to reconcile Service: redis-vector-db
Success to reconcile Deployment: redis-vector-db-deployment
the service URL is: http://redis-vector-db.opea-app-chatqna.svc.cluster.local:6379

reconcile resource for node: Reranking
trying to reconcile internal service [ reranking-svc ] in namespace
get resource config: {Reranking { {reranking-svc  map[TEI_RERANKING_ENDPOINT:tei-reranking-svc endpoint:/v1/reranking] false} } $response   }
The raw yaml file has been split into 4 yaml files
Success to reconcile ConfigMap: reranking-usvc-config
Success to reconcile Service: reranking-svc
find downstream service for Reranking with name tei-reranking-svc
Success to reconcile Deployment: reranking-svc-deployment
the service URL is: http://reranking-svc.opea-app-chatqna.svc.cluster.local:8000/v1/reranking

reconcile resource for node: TeiReranking
trying to reconcile internal service [ tei-reranking-svc ] in namespace
get resource config: {TeiReranking { {tei-reranking-svc  map[endpoint:/rerank] true} }    }
The raw yaml file has been split into 6 yaml files
Success to reconcile ConfigMap: teirerank-config
Success to reconcile Service: tei-reranking-svc
Success to reconcile Deployment: tei-reranking-svc-deployment
the service URL is: http://tei-reranking-svc.opea-app-chatqna.svc.cluster.local:80/rerank

reconcile resource for node: Llm
trying to reconcile internal service [ llm-svc ] in namespace
get resource config: {Llm { {llm-svc  map[TGI_LLM_ENDPOINT:tgi-service-m endpoint:/v1/chat/completions] false} } $response   }
The raw yaml file has been split into 4 yaml files
Success to reconcile ConfigMap: llm-uservice-config
Success to reconcile Service: llm-svc
find downstream service for Llm with name tgi-service-m
Success to reconcile Deployment: llm-svc-deployment
the service URL is: http://llm-svc.opea-app-chatqna.svc.cluster.local:9000/v1/chat/completions

reconcile resource for node: Tgi
trying to reconcile internal service [ tgi-service-m ] in namespace
get resource config: {Tgi { {tgi-service-m  map[endpoint:/generate] true} }    }
The raw yaml file has been split into 6 yaml files
Success to reconcile ConfigMap: tgi-config
Success to reconcile Service: tgi-service-m
Success to reconcile Deployment: tgi-service-m-deployment
the service URL is: http://tgi-service-m.opea-app-chatqna.svc.cluster.local:80/generate

user config {opea-app-chatqna router-service router-service-deployment    '{"metadata":{"creationTimestamp":null},"spec":{"nodes":{"root":{"routerType":"Sequence","steps":[{"name":"Embedding","internalService":{"serviceName":"embedding-svc","config":{"TEI_EMBEDDING_ENDPOINT":"tei-embedding-svc","endpoint":"/v1/embeddings"}},"serviceUrl":"http://embedding-svc.opea-app-chatqna.svc.cluster.local:6000/v1/embeddings"},{"name":"TeiEmbedding","internalService":{"serviceName":"tei-embedding-svc","isDownstreamService":true},"serviceUrl":"http://tei-embedding-svc.opea-app-chatqna.svc.cluster.local:80"},{"name":"Retriever","internalService":{"serviceName":"retriever-svc","config":{"REDIS_URL":"redis-vector-db","TEI_EMBEDDING_ENDPOINT":"tei-embedding-svc","endpoint":"/v1/retrieval"}},"data":"$response","serviceUrl":"http://retriever-svc.opea-app-chatqna.svc.cluster.local:7000/v1/retrieval"},{"name":"VectorDB","internalService":{"serviceName":"redis-vector-db","isDownstreamService":true},"serviceUrl":"http://redis-vector-db.opea-app-chatqna.svc.cluster.local:6379"},{"name":"Reranking","internalService":{"serviceName":"reranking-svc","config":{"TEI_RERANKING_ENDPOINT":"tei-reranking-svc","endpoint":"/v1/reranking"}},"data":"$response","serviceUrl":"http://reranking-svc.opea-app-chatqna.svc.cluster.local:8000/v1/reranking"},{"name":"TeiReranking","internalService":{"serviceName":"tei-reranking-svc","config":{"endpoint":"/rerank"},"isDownstreamService":true},"serviceUrl":"http://tei-reranking-svc.opea-app-chatqna.svc.cluster.local:80/rerank"},{"name":"Llm","internalService":{"serviceName":"llm-svc","config":{"TGI_LLM_ENDPOINT":"tgi-service-m","endpoint":"/v1/chat/completions"}},"data":"$response","serviceUrl":"http://llm-svc.opea-app-chatqna.svc.cluster.local:9000/v1/chat/completions"},{"name":"Tgi","internalService":{"serviceName":"tgi-service-m","config":{"endpoint":"/generate"},"isDownstreamService":true},"serviceUrl":"http://tgi-service-m.opea-app-chatqna.svc.cluster.local:80/generate"}]}},"routerConfig":{"name":"router","serviceName":"router-service","nameSpace":"","config":null}},"status":{"condition":{"lastUpdateTime":null}}}'}
Success to reconcile Deployment: router-service-deployment
Success to reconcile Service: router-service
the router URL is: http://router-service.opea-app-chatqna.svc.cluster.local:8080
2024-09-04T12:59:24Z    INFO    Observed a panic in reconciler: runtime error: index out of range [1] with length 1     {"controller": "gmconnector", "controllerGroup": "gmc.opea.io", "controllerKind": "GMConnector", "GMConnector": {"name":"embedding-svc-deployment","namespace":"opea-app-chatqna"}, "namespace": "opea-app-chatqna", "name": "embedding-svc-deployment", "reconcileID": "f94a57cb-bae8-49c1-aa41-e545db826013"}
-----------------Reconciling GMConnector opea-app-chatqna/embedding-svc-deployment -------------------------
panic: runtime error: index out of range [1] with length 1 [recovered]
        panic: runtime error: index out of range [1] with length 1

goroutine 344 [running]:
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile.func1()
        /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.3/pkg/internal/controller/controller.go:116 +0x1e5
panic({0x19d33e0?, 0xc000a8b8c0?})
        /usr/local/go/src/runtime/panic.go:914 +0x21f
github.com/opea-project/GenAIInfra/microservices-connector/internal/controller.(*GMConnectorReconciler).collectResourceStatus(0xc0005b1bf0, 0xc0004fee00, {0x1d72248, 0xc000987a40})
        /workspace/internal/controller/gmconnector_controller.go:513 +0xa5f
github.com/opea-project/GenAIInfra/microservices-connector/internal/controller.(*GMConnectorReconciler).handleStatusUpdate(0xc0005b1bf0, {0x1d72248, 0xc000987a40}, 0xc000a2a000)
        /workspace/internal/controller/gmconnector_controller.go:444 +0x188
github.com/opea-project/GenAIInfra/microservices-connector/internal/controller.(*GMConnectorReconciler).Reconcile(0xc0005b1bf0, {0x1d72248, 0xc000987a40}, {{{0xc000b1f310, 0x10}, {0xc0005de828, 0x18}}})
        /workspace/internal/controller/gmconnector_controller.go:342 +0x5f0
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile(0x1d75280?, {0x1d72248?, 0xc000987a40?}, {{{0xc000b1f310?, 0xb?}, {0xc0005de828?, 0x0?}}})
        /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.3/pkg/internal/controller/controller.go:119 +0xb7
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler(0xc0006bbf40, {0x1d72280, 0xc0006ecaa0}, {0x1939140?, 0xc000708d20?})
        /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.3/pkg/internal/controller/controller.go:316 +0x3cc
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem(0xc0006bbf40, {0x1d72280, 0xc0006ecaa0})
        /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.3/pkg/internal/controller/controller.go:266 +0x1af
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2()
        /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.3/pkg/internal/controller/controller.go:227 +0x79
created by sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2 in goroutine 307
        /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.3/pkg/internal/controller/controller.go:223 +0x565

Here is more restart with more panic logs:

......
reconcile resource for node: Tgi
trying to reconcile internal service [ tgi-service ] in namespace
get resource config: {Tgi { {tgi-service  map[MODEL_ID:meta-llama/CodeLlama-7b-hf endpoint:/generate] true} }    }
The raw yaml file has been split into 6 yaml files
Success to reconcile ConfigMap: tgi-config
Success to reconcile Service: tgi-service
Success to reconcile Deployment: tgi-service-deployment
the service URL is: http://tgi-service.opea-app-modstep.svc.cluster.local:80/generate

user config {opea-app-modstep router-service router-service-deployment    '{"metadata":{"creationTimestamp":null},"spec":{"nodes":{"root":{"routerType":"Sequence","steps":[{"name":"Llm","internalService":{"serviceName":"llm-service","config":{"TGI_LLM_ENDPOINT":"tgi-service","endpoint":"/v1/chat/completions"}},"data":"$response","serviceUrl":"http://llm-service.opea-app-modstep.svc.cluster.local:9000/v1/chat/completions"},{"name":"Tgi","internalService":{"serviceName":"tgi-service","config":{"MODEL_ID":"meta-llama/CodeLlama-7b-hf","endpoint":"/generate"},"isDownstreamService":true},"serviceUrl":"http://tgi-service.opea-app-modstep.svc.cluster.local:80/generate"}]}},"routerConfig":{"name":"router","serviceName":"router-service","nameSpace":"","config":null}},"status":{"condition":{"lastUpdateTime":null}}}'}
Success to reconcile Deployment: router-service-deployment
Success to reconcile Service: router-service
the router URL is: http://router-service.opea-app-modstep.svc.cluster.local:8080
-----------------Reconciling GMConnector opea-app-modstep/llm-service-deployment -------------------------
2024-09-04T13:08:21Z    INFO    Observed a panic in reconciler: runtime error: index out of range [1] with length 1     {"controller": "gmconnector", "controllerGroup": "gmc.opea.io", "controllerKind": "GMConnector", "GMConnector": {"name":"llm-service-deployment","namespace":"opea-app-modstep"}, "namespace": "opea-app-modstep", "name": "llm-service-deployment", "reconcileID": "9651baa5-8f9f-40b3-99b5-fb728c25cb3e"}
panic: runtime error: index out of range [1] with length 1 [recovered]
        panic: runtime error: index out of range [1] with length 1

goroutine 360 [running]:
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile.func1()
        /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.3/pkg/internal/controller/controller.go:116 +0x1e5
panic({0x19d33e0?, 0xc000b09b60?})
        /usr/local/go/src/runtime/panic.go:914 +0x21f
github.com/opea-project/GenAIInfra/microservices-connector/internal/controller.(*GMConnectorReconciler).collectResourceStatus(0xc00082d878, 0xc000aec1c0, {0x1d72248, 0xc000a0f7d0})
        /workspace/internal/controller/gmconnector_controller.go:513 +0xa5f
github.com/opea-project/GenAIInfra/microservices-connector/internal/controller.(*GMConnectorReconciler).handleStatusUpdate(0xc00082d878, {0x1d72248, 0xc000a0f7d0}, 0xc000af0000)
        /workspace/internal/controller/gmconnector_controller.go:444 +0x188
github.com/opea-project/GenAIInfra/microservices-connector/internal/controller.(*GMConnectorReconciler).Reconcile(0xc00082d878, {0x1d72248, 0xc000a0f7d0}, {{{0xc000e89300, 0x10}, {0xc000e8ab58, 0x16}}})
        /workspace/internal/controller/gmconnector_controller.go:342 +0x5f0
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile(0x1d75280?, {0x1d72248?, 0xc000a0f7d0?}, {{{0xc000e89300?, 0xb?}, {0xc000e8ab58?, 0x0?}}})
        /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.3/pkg/internal/controller/controller.go:119 +0xb7
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler(0xc0002dce60, {0x1d72280, 0xc000418f50}, {0x1939140?, 0xc0005a53c0?})
        /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.3/pkg/internal/controller/controller.go:316 +0x3cc
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem(0xc0002dce60, {0x1d72280, 0xc000418f50})
        /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.3/pkg/internal/controller/controller.go:266 +0x1af
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2()
        /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.3/pkg/internal/controller/controller.go:227 +0x79
created by sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2 in goroutine 328
        /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.3/pkg/internal/controller/controller.go:223 +0x565
irisdingbj commented 2 months ago

@KfreeZ plz take a look at the controller.(*GMConnectorReconciler).collectResourceStatus