cis pod log:[ERROR] [CORE] Ingress class resource not found.

guardian0916 commented 2 years ago

Setup Details

CIS Version : 2.6.1 Build: f5networks/k8s-bigip-ctlr:2.6.1 BIGIP Version: BIG-IP 15.1.4 Build 0.0.47 Final AS3 Version:3.28.0 Agent Mode: Custom Resource Definitions Orchestration: K8S Pool Mode: Cluster

Description

When the CIS is created,after running the following command: [root@choerodon-master1 f5ve]# kubectl logs k8s-bigip-ctlr-deployment-c-b77f6dc87-28cth -n kube-system

Output the following information: 2021/12/30 16:58:16 [ERROR] [CORE] Ingress class resource not found. 2021/12/30 16:58:16 [ERROR] [CORE] Ingress class resource not found. 2021/12/30 16:58:16 [ERROR] [CORE] Ingress class resource not found. 2021/12/30 16:58:16 [ERROR] [CORE] Ingress class resource not found. 2021/12/30 16:58:16 [ERROR] [CORE] Ingress class resource not found. 2021/12/30 16:58:16 [ERROR] [CORE] Ingress class resource not found. 2021/12/30 16:58:16 [ERROR] [CORE] Ingress class resource not found. 2021/12/30 16:58:16 [ERROR] [CORE] Ingress class resource not found. 2021/12/30 16:58:16 [ERROR] [CORE] Ingress class resource not found. 2021/12/30 16:58:16 [ERROR] [CORE] Ingress class resource not found. 2021/12/30 16:58:16 [ERROR] [CORE] Ingress class resource not found. 2021/12/30 16:58:16 [ERROR] [CORE] Ingress class resource not found. 2021/12/30 16:58:16 [ERROR] [CORE] Ingress class resource not found. 2021/12/30 16:58:16 [ERROR] [CORE] Ingress class resource not found. 2021/12/30 16:58:16 [ERROR] [CORE] Ingress class resource not found. 2021/12/30 16:58:16 [ERROR] [CORE] Ingress class resource not found. 2021/12/30 16:58:16 [ERROR] [CORE] Ingress class resource not found. 2021/12/30 16:58:16 [ERROR] [CORE] Ingress class resource not found. 2021/12/30 16:58:16 [ERROR] [CORE] Ingress class resource not found. 2021/12/30 16:58:16 [ERROR] [CORE] Ingress class resource not found. 2021/12/30 16:58:16 [ERROR] [CORE] Ingress class resource not found. 2021/12/30 16:58:16 [ERROR] [CORE] Ingress class resource not found. 2021/12/30 16:58:46 [ERROR] [CORE] Ingress class resource not found. 2021/12/30 16:58:46 [ERROR] [CORE] Ingress class resource not found. 2021/12/30 16:58:46 [ERROR] [CORE] Ingress class resource not found. 2021/12/30 16:58:46 [ERROR] [CORE] Ingress class resource not found. 2021/12/30 16:58:46 [ERROR] [CORE] Ingress class resource not found. 2021/12/30 16:58:46 [ERROR] [CORE] Ingress class resource not found. 2021/12/30 16:58:46 [ERROR] [CORE] Ingress class resource not found. 2021/12/30 16:58:46 [ERROR] [CORE] Ingress class resource not found. 2021/12/30 16:58:46 [ERROR] [CORE] Ingress class resource not found. 2021/12/30 16:58:46 [ERROR] [CORE] Ingress class resource not found. 2021/12/30 16:58:46 [ERROR] [CORE] Ingress class resource not found. 2021/12/30 16:58:46 [ERROR] [CORE] Ingress class resource not found. 2021/12/30 16:58:46 [ERROR] [CORE] Ingress class resource not found. 2021/12/30 16:58:46 [ERROR] [CORE] Ingress class resource not found. 2021/12/30 16:58:46 [ERROR] [CORE] Ingress class resource not found. 2021/12/30 16:58:46 [ERROR] [CORE] Ingress class resource not found. 2021/12/30 16:58:46 [ERROR] [CORE] Ingress class resource not found. 2021/12/30 16:58:46 [ERROR] [CORE] Ingress class resource not found. 2021/12/30 16:58:46 [ERROR] [CORE] Ingress class resource not found. 2021/12/30 16:58:46 [ERROR] [CORE] Ingress class resource not found. 2021/12/30 16:58:46 [ERROR] [CORE] Ingress class resource not found. 2021/12/30 16:58:46 [ERROR] [CORE] Ingress class resource not found.

the cis yaml as follow: apiVersion: apps/v1 kind: Deployment metadata: name: k8s-bigip-ctlr-deployment-c namespace: kube-system spec: replicas: 1 selector: matchLabels: app: k8s-bigip-ctlr template: metadata: name: k8s-bigip-ctlr labels: app: k8s-bigip-ctlr spec: serviceAccountName: bigip-ctlr containers:

name: k8s-bigip-ctlr image: "devops-harbor.sany.com.cn/f5/k8s-bigip-ctlr:2.6.1" env:
- name: TZ value: Asia/Singapore
- name: BIGIP_USERNAME valueFrom: secretKeyRef: name: bigip-login key: username
- name: BIGIP_PASSWORD valueFrom: secretKeyRef: name: bigip-login key: password command: ["/app/bin/k8s-bigip-ctlr"] args: [ "--bigip-username=$(BIGIP_USERNAME)", "--bigip-password=$(BIGIP_PASSWORD)", "--bigip-url=10.0.10.230", "--bigip-partition=k8s-choerondon-uat", "--pool-member-type=cluster", "--insecure=true", "--namespace-label=cpaas.io/project=choerodon" ]

Expected Result

the cis pod has no error is displayed.

Actual Result

the cis pod has output error information:

Question:

How the cis pod has no error is display ?

trinaths commented 2 years ago

@guardian0916 please create the IngressClass and restart CIS deployment.

apiVersion: networking.k8s.io/v1
kind: IngressClass
metadata:
  name: f5
  annotations:
    # Set the annotation as true to make CIS default ingress controller of k8s cluster
    ingressclass.kubernetes.io/is-default-class: "true"
spec:
  # Provide the controller name as "f5.com/cntr-ingress-svcs"
  # Warning: don't change the controller name as cis verify the controller name while processing the ingress resource
  controller: f5.com/cntr-ingress-svcs

trinaths commented 2 years ago

@guardian0916 - Please see documentation for more info - https://clouddocs.f5.com/containers/latest/userguide/ingress.html

gazhuchao commented 2 years ago

I have try to create ingress class in my K8S cluster according the user guide. But there are many error logs appeared as the attachment. Is there any other ingress that is not released by F5 causing the problem?

.

trinaths commented 2 years ago

@gazhuchao please give VS IP in ingress annotation or in the CIS config arg. this resolves the issues.

gazhuchao commented 2 years ago

@trinaths My K8S version is 1.19.9, I have tried to set CIS arg 'manage-ingress' to false. But when I deployed, the CIS pod cannot startup and CIS error log as below, what's wrong with CIS? And how can I solved it, thanks!

2022/01/04 16:23:38 [INFO] [INIT] Starting: Container Ingress Services - Version: 2.6.1, BuildInfo: azure-1230-afb08130e1e718d2f78f9a838f461b02a9a4a36b 2022/01/04 16:23:39 [INFO] ConfigWriter started: 0xc00047a2d0 2022/01/04 16:23:39 [INFO] Started config driver sub-process at pid: 16 2022/01/04 16:23:39 [INFO] [INIT] Creating Agent for as3 2022/01/04 16:23:39 [INFO] [AS3] Initializing AS3 Agent 2022/01/04 16:23:40 [INFO] [CORE] NodePoller (0xc000242750) registering new listener: 0x1797560 2022/01/04 16:23:40 [INFO] [CORE] NodePoller started: (0xc000242750) 2022/01/04 16:23:40 [INFO] [CORE] Registered BigIP Metrics 2022/01/04 16:23:40 [INFO] [CORE] Not watching Ingress resources. 2022/01/04 16:23:40 [INFO] [CORE] Watching ConfigMap resources. 2022/01/04 16:23:40 [INFO] [CORE] Handling ConfigMap resource events. 2022/01/04 16:23:40 [INFO] [CORE] Not handling Ingress resource events. 2022/01/04 16:23:40 [INFO] [CORE] Not watching Ingress resources. 2022/01/04 16:23:40 [INFO] [CORE] Watching ConfigMap resources. 2022/01/04 16:23:40 [INFO] [CORE] Handling ConfigMap resource events. 2022/01/04 16:23:40 [INFO] [CORE] Not handling Ingress resource events. 2022/01/04 16:23:40 [INFO] [CORE] Not watching Ingress resources. 2022/01/04 16:23:40 [INFO] [CORE] Watching ConfigMap resources. 2022/01/04 16:23:40 [INFO] [CORE] Handling ConfigMap resource events. 2022/01/04 16:23:40 [INFO] [CORE] Not handling Ingress resource events. E0104 16:23:40.386817 1 runtime.go:78] Observed a panic: "invalid memory address or nil pointer dereference" (runtime error: invalid memory address or nil pointer dereference) goroutine 181 [running]: k8s.io/apimachinery/pkg/util/runtime.logPanic(0x19480c0, 0x28bcc10) /go/src/github.com/F5Networks/k8s-bigip-ctlr/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:74 +0x95 k8s.io/apimachinery/pkg/util/runtime.HandleCrash(0x0, 0x0, 0x0) /go/src/github.com/F5Networks/k8s-bigip-ctlr/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:48 +0x89 panic(0x19480c0, 0x28bcc10) /usr/local/go/src/runtime/panic.go:969 +0x1b9 github.com/F5Networks/k8s-bigip-ctlr/pkg/appmanager.(Manager).getSecretServiceQueueKeyForIngress(0xc00056f500, 0xc000acac40, 0x1b6d28b, 0x7, 0xc0007244c0) /go/src/github.com/F5Networks/k8s-bigip-ctlr/pkg/appmanager/validateResources.go:192 +0x75 github.com/F5Networks/k8s-bigip-ctlr/pkg/appmanager.(Manager).checkValidSecrets(0xc00056f500, 0x1b4aba0, 0xc000acac40, 0xc00093170e, 0x2, 0x0, 0x0) /go/src/github.com/F5Networks/k8s-bigip-ctlr/pkg/appmanager/validateResources.go:277 +0xe5 github.com/F5Networks/k8s-bigip-ctlr/pkg/appmanager.(Manager).enqueueSecrets(0xc00056f500, 0x1b4aba0, 0xc000acac40, 0x1b6c6be, 0x6) /go/src/github.com/F5Networks/k8s-bigip-ctlr/pkg/appmanager/appManager.go:832 +0x46 github.com/F5Networks/k8s-bigip-ctlr/pkg/appmanager.(Manager).newAppInformer.func14(0x1b4aba0, 0xc000acac40) /go/src/github.com/F5Networks/k8s-bigip-ctlr/pkg/appmanager/appManager.go:757 +0x53 k8s.io/client-go/tools/cache.ResourceEventHandlerFuncs.OnAdd(...) /go/src/github.com/F5Networks/k8s-bigip-ctlr/vendor/k8s.io/client-go/tools/cache/controller.go:231 k8s.io/client-go/tools/cache.(processorListener).run.func1() /go/src/github.com/F5Networks/k8s-bigip-ctlr/vendor/k8s.io/client-go/tools/cache/shared_informer.go:777 +0xc2 k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1(0xc0008daf60) /go/src/github.com/F5Networks/k8s-bigip-ctlr/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:155 +0x5f k8s.io/apimachinery/pkg/util/wait.BackoffUntil(0xc00093df60, 0x1d5d280, 0xc00091b830, 0x1907f01, 0xc0005c31a0) /go/src/github.com/F5Networks/k8s-bigip-ctlr/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:156 +0xad k8s.io/apimachinery/pkg/util/wait.JitterUntil(0xc0008daf60, 0x3b9aca00, 0x0, 0x1, 0xc0005c31a0) /go/src/github.com/F5Networks/k8s-bigip-ctlr/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:133 +0x98 k8s.io/apimachinery/pkg/util/wait.Until(...) /go/src/github.com/F5Networks/k8s-bigip-ctlr/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:90 k8s.io/client-go/tools/cache.(processorListener).run(0xc000919e00) /go/src/github.com/F5Networks/k8s-bigip-ctlr/vendor/k8s.io/client-go/tools/cache/shared_informer.go:771 +0x95 k8s.io/apimachinery/pkg/util/wait.(Group).Start.func1(0xc00020f2c0, 0xc00021db30) /go/src/github.com/F5Networks/k8s-bigip-ctlr/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:73 +0x51 created by k8s.io/apimachinery/pkg/util/wait.(Group).Start /go/src/github.com/F5Networks/k8s-bigip-ctlr/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:71 +0x65 panic: runtime error: invalid memory address or nil pointer dereference [recovered] panic: runtime error: invalid memory address or nil pointer dereference [signal SIGSEGV: segmentation violation code=0x1 addr=0x38 pc=0x16e1215]

gazhuchao commented 2 years ago

I have found the same issue in https://github.com/F5Networks/k8s-bigip-ctlr/issues/2062.

gazhuchao commented 2 years ago

In the customer's environment, they have installed the KIC in the cluster. The info as below:

controller: Image: devops-harbor.sany.com.cn/devops/ingress-nginx-controller:v0.45.0 Ports: 80/TCP, 443/TCP, 8443/TCP Host Ports: 80/TCP, 443/TCP, 8443/TCP Args: /nginx-ingress-controller --publish-service=$(POD_NAMESPACE)/ingress-nginx-controller --election-id=ingress-controller-leader --ingress-class=nginx --configmap=$(POD_NAMESPACE)/ingress-nginx-controller --validating-webhook=:8443 --validating-webhook-certificate=/usr/local/certificates/cert --validating-webhook-key=/usr/local/certificates/key

trinaths commented 2 years ago

@gazhuchao are there any secrets used with ingress resources ?

gazhuchao commented 2 years ago

Yes, in some ingresses, HTTPS use TLS as below:

gazhuchao commented 2 years ago

In issue #2062, Kkfinkkfin has tried to do a testing which using clear HTTP with no encryption and the result was the same.

trinaths commented 2 years ago

@gazhuchao we tried to test this in like env with CIS and KIC and manage-ingress to false. However, we haven't seen any crash.

trinaths commented 2 years ago

@gazhuchao Please try with CIS 2.7 and share your findings for further investigation.

trinaths commented 2 years ago

@gazhuchao - Please share your findings with upgrade to 2.7. Please see changes to EDNS CRD before upgrade . https://github.com/F5Networks/k8s-bigip-ctlr/blob/master/docs/RELEASE-NOTES.rst#270

trinaths commented 2 years ago

@gazhuchao - Please share your findings with upgrade to 2.7.

gazhuchao commented 2 years ago

I set manage-ingress to false, and there is not any error in CIS 2.7. But if the customer want to use ingress, how can I do it?

trinaths commented 2 years ago

@gazhuchao set manage-ingress to true.

trinaths commented 2 years ago

No reply from authors for the recommendations. Hope the suggestions worked well. Closing this issue.

F5Networks / k8s-bigip-ctlr